# Temporal Platform Documentation > This file is the complete text of Temporal's documentation, intended for bulk ingestion. Temporal is an open-source platform for building crash-proof applications that resume exactly where they left off after failures. This file contains the full text of every documentation page, in one document. Pages are separated by a horizontal rule and each carries a `Source:` URL. For a structured index instead, see https://docs.temporal.io/llms.txt. --- # What is a Temporal Activity? Source: https://docs.temporal.io/activities This guide provides a comprehensive overview of Temporal Activities including [Activity Definition](/activity-definition), [Activity Type](/activity-definition#activity-type), [Activity Execution](/activity-execution), [Local Activity](/local-activity), and [Standalone Activity](/standalone-activity). > **💡 Tip:** > > Watch a short overview of what an Activity is in Temporal: > > [Watch: What is an Activity in Temporal?](https://www.youtube.com/watch?v=rtWrzjnKlSQ) > An Activity is a normal function or method that executes a single, well-defined action (either short or long running), such as calling another service, transcoding a media file, or sending an email message. Activity code can be non-deterministic. We recommend that it be [idempotent](/activity-definition#idempotency). Activities are the most common Temporal primitive and encompass small units of work such as: - Single write operations, like updating user information or submitting a credit card payment - Batches of similar writes, like creating multiple orders or sending multiple messages - One or more read operations followed by a write operation, like checking a product status and user address before updating an order status - A read that should be memoized, like an LLM call, a large download, or a slow-polling read Larger pieces of functionality should be broken up into multiple activities. This makes it easier to do failure recovery, have short timeouts, and be idempotent. Workflow code orchestrates the execution of Activities, persisting the results. If an Activity Execution fails, any future attempt will start from the initial state, unless your code uses ([Heartbeat details payloads](/encyclopedia/detecting-activity-failures#activity-heartbeat)) for checkpointing (storing state on the server, and using it when resuming subsequent attempts). Activity Functions are executed by Worker Processes. When the Activity Function returns, the Worker sends the results back to the Temporal Service as part of the [ActivityTaskCompleted](/references/events#activitytaskcompleted) Event. The Event is added to the Workflow Execution's Event History. For other Activity-related Events, see [Activity Events](/workflow-execution/event#activity-events). If you only want to execute one Activity Function, then you don't need to use a Workflow: you can use your SDK Client to invoke it directly as a [Standalone Activity](/standalone-activity). --- # Activity Definition Source: https://docs.temporal.io/activity-definition > Learn about defining Temporal Activities, including Activity Types, parameters, and implementation details. This page discusses the following: - [Activity Definition](#activity-definition) - [Idempotency](#idempotency) - [Constraints](#activity-constraints) - [Parameters](#activity-parameters) - [Activity Type](#activity-type) In day-to-day conversation, the term _Activity_ denotes an [Activity Definition](/activity-definition), [Activity Type](/activity-definition#activity-type), or [Activity Execution](/activity-execution). Temporal documentation aims to be explicit and differentiate between them. ## What is an Activity Definition? An Activity Definition is the code that defines the constraints of an [Activity Task Execution](/tasks#activity-task-execution). Activities encapsulate business logic that is prone to failure, allowing for automatic retries when issues occur. Below are examples of basic Activity Definitions across supported SDKs. **Go** **[Activity Definition in Go](/develop/go/activities/basics)** ```go import ( "context" "go.temporal.io/sdk/activity" ) func YourSimpleActivity(ctx context.Context) error { return nil } ``` **Java** **[Activity Definition in Java (Interface)](/develop/java/activities/basics)** ```java @ActivityInterface public interface GreetingActivities { @ActivityMethod String composeGreeting(String greeting, String language); } ``` **[Activity Definition in Java (Implementation)](/develop/java/activities/basics)** ```java static class GreetingActivitiesImpl implements GreetingActivities { @Override public String composeGreeting(String greeting, String name) { return greeting + " " + name + "!"; } } ``` **PHP** **[Activity Definition in PHP (Interface)](/develop/php/activities/basics)** ```php #[ActivityInterface] interface GreetingActivities { public function composeGreeting(string $greeting, string $name): string; } ``` **[Activity Definition in PHP (Implementation)](/develop/php/activities/basics)** ```php class GreetingActivitiesImpl implements GreetingActivities { public function composeGreeting(string $greeting, string $name): string { return $greeting . ' ' . $name; } } ``` **Python** **[Activity Definition in Python](/develop/python/activities/basics)** ```python from temporalio import activity @activity.defn(name="your_activity") async def your_activity(input: YourParams) -> str: return f"{input.greeting}, {input.name}!" ``` **TypeScript** **[Activity Definition in TypeScript](/develop/typescript/activities/basics)** ```ts export async function greet(name: string): Promise { return `Hello, ${name}!`; } ``` **.NET** **[Activity Definition in C# and .NET](/develop/dotnet/activities/basics)** ```csharp using Temporalio.Activities; public class MyActivities { [Activity] public string MyActivity(MyActivityParams input) => $"{input.Greeting}, {input.Name}!"; } ``` **Rust** **[Activity Definition in Rust](/develop/rust/activities/basics)** ```rust use temporalio_sdk::activities::{ActivityContext, ActivityError}; use temporalio_macros::activities; pub struct GreetingActivities; #[activities] impl GreetingActivities { #[activity] pub async fn greet(_ctx: ActivityContext, name: String) -> Result { Ok(format!("Hello, {}!", name)) } } ``` For full SDK-specific guides, see: - [How to develop an Activity Definition using the .NET SDK](/develop/dotnet/activities/basics) - [How to develop an Activity Definition using the Go SDK](/develop/go/activities/basics) - [How to develop an Activity Definition using the Java SDK](/develop/java/activities/basics) - [How to develop an Activity Definition using the PHP SDK](/develop/php/activities/basics) - [How to develop an Activity Definition using the Python SDK](/develop/python/activities/basics) - [How to develop an Activity Definition using the Ruby SDK](/develop/ruby/activities/basics) - [How to develop an Activity Definition using the Rust SDK](/develop/rust/activities/basics) - [How to develop an Activity Definition using the TypeScript SDK](/develop/typescript/activities/basics) The term 'Activity Definition' is used to refer to the full set of primitives in any given language SDK that provides an access point to an Activity Function Definition——the method or function that is invoked for an [Activity Task Execution](/tasks#activity-task-execution). Therefore, the terms Activity Function and Activity Method refer to the source of an instance of an execution. Activity Definitions are named and referenced in code by their [Activity Type](/activity-definition#activity-type). ![Activity Definition](/diagrams/activity-definition.svg) ### Idempotency Temporal recommends that Activities be idempotent. Idempotence means that performing an operation multiple times has the same result as performing it once. In the context of Temporal, Activities should be designed to be safely executed multiple times without causing unexpected or undesired side effects. Consider the power button on your laptop. When you press it, the machine is changed from one state to the other, from on to off, and vice versa. This is not an idempotent operation. Each invocation leads to a different state. However, imagine that you modified your laptop to have separate on and off buttons. Pressing the On button multiple times would have no effect beyond the initial invocation as the laptop is already on. This action is considered idempotent. ![](/diagrams/idempotence-image.png) Idempotency is an important design consideration in software applications as well. You have probably encountered idempotent operations in your work already. A few examples where idempotent operations are vital would be: - **Infrastructure-as-Code (IaC) tool** - Conserving resources is important when you're provisioning infrastructure in the cloud. An IaC system that was not designed with idempotence in mind could lead to high costs if the function to provision a new server was accidentally invoked multiple times. An IaC tool that is designed with idempotence in mind ensures that multiple invocations of the tool doesn't lead to unintended instances being created. - **Payment processing system** - A payment processing system must charge the customer only once for a given purchase. If the system was not designed to be idempotent, duplicate requests would result in extra charges and unhappy customers. A payment processing system that is designed to be idempotent ensures customers are not charged multiple times for the same transaction, preventing financial discrepancies. > **ℹ️ Info:** > > By design, completed Activities will not re-execute as part of a [Workflow Replay](/workflow-execution#replay). However, Activities won’t record to the [Event History](/encyclopedia/retry-policies#event-history) until they return or produce an error. If an Activity fails to report to the server at all, it will be retried. Designing for idempotence, especially if you have a [Global Namespace](/global-namespace), will improve reusability and reliability. > An Activity is idempotent if multiple [Activity Task Executions](/tasks#activity-task-execution) do not change the state of the system beyond the first Activity Task Execution. The lack of idempotency might affect the correctness of your application but does not affect the Temporal Platform. In other words, lack of idempotency doesn't lead to a platform error. In some cases, whether something is idempotent doesn't affect the correctness of an application. For example, if you have a monotonically incrementing counter, you might not care that retries increment the counter because you don't care about the actual value, only that the current value is greater than a previous value. You should always make your business logic Activities idempotent in Temporal. Because Activities may be retried, these functions may be executed more than once. A non-idempotent Activity could adversely affect the state of the system. Activities are an atomic unit of execution within Temporal. They are invoked and either complete successfully or not. Take this into consideration when you design your Activities. For example, consider an Activity that has the following three steps: 1. Perform a database lookup 2. Make a call to a microservice with parameters retrieved from the database 3. Write the result of the microservice call to the filesystem Imagine that the first two steps succeed, but the third step fails due to a permissions issue. During retry, the entire Activity—and therefore each of the three steps—is executed again. To maintain idempotency, design your Activities to be more granular. In this case, you could have three Activities, one for each step. This way, only the step that failed will be executed again. However, you must balance this against the potential for a larger Event History, since there would now be three Activity Executions instead of one. Idempotence for Activities is also important due to a particular edge case inherent in distributed computing. Consider a scenario in which a Worker polls the Temporal Service, accepts the Activity Task, and begins executing the Activity. The Activity function completes successfully, but the Worker crashes just before it notifies the Temporal Service. In this case, the Event History won’t reflect the successful completion of the Task, so the Activity will be retried. If the Activity is not idempotent, this could have negative consequences, such as duplicate charges in a payment processing scenario. You can achieve idempotency in your application through the use of unique identifiers, known as idempotency keys, which are used to detect duplicate requests. These are enforced by the service you are calling from your Activity, not by the Activity itself. For example, the APIs provided by most payment processors allow the client to include an idempotency key with the request. When the payment service receives a request, it checks a database to determine whether there has already been a request with this key. If so, the duplicate request is ignored and does not result in another charge. If not, then it writes a new record to the database with this key, allowing it to identify duplicate requests in the future. In Temporal, the request to the payment service would be made from within an Activity. You can use a combination of the Workflow Run ID and the Activity ID as an idempotency key since this is guaranteed to be consistent across retry attempts but unique among Workflow Executions. For more information about idempotency in Temporal, see the following post: [Idempotency and Durable Execution](https://temporal.io/blog/idempotency-and-durable-execution) ### Activity retry policy The Activity retry mechanism gives applications the benefits of durable execution. For example, Temporal will keep track of the [exponential backoff delay](/encyclopedia/retry-policies#backoff-coefficient) even if the Worker crashes. Since Temporal can’t tell when a Worker crashes, Workflows rely on the [start_to_close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) to know how long to wait before assuming that an Activity is inactive. For an Activity with a [Retry Policy](/encyclopedia/retry-policies) that allows retries, Temporal guarantees that the Activity will be observed as completed exactly once. However, the Activity may be executed multiple times and may even partially complete more than once during this process. This could lead to a scenario where certain parts of the Activity are executed multiple times before a successful execution is completed. > **⚠️ Caution:** > Be cautious when doing retries within your Activity because it lengthens the needed Activity timeout. Such internal retries also prevent users from counting failure metrics and make it harder for users to debug in Temporal UI when something is wrong. ### Constraints Activity Definitions are executed as normal functions. In the event of failure, the function begins at its initial state when retried (except when Activity Heartbeats are established). Therefore, an Activity Definition has no restrictions on the code it contains. ### Parameters An Activity Definition can support as many parameters as needed. All values passed through these parameters are recorded in the [Event History](/workflow-execution/event#event-history) of the Workflow Execution. Return values are also captured in the Event History for the calling Workflow Execution. Activity Definitions must contain the following parameters: - Context: an optional parameter that provides Activity context within multiple APIs. - Heartbeat: a notification from the Worker to the Temporal Service that the Activity Execution is progressing. Cancelations are allowed only if the Activity Definition permits Heartbeating. - Timeouts: intervals that control the execution and retrying of Activity Task Executions. Other parameters, such as [Retry Policies](/encyclopedia/retry-policies) and return values, can be seen in the implementation guides, listed in the next section. ## What is an Activity Type? An Activity Type is the mapping of a name to an Activity Definition. Activity Types are scoped through Task Queues. ## Best practices for defining Activities Here are some best practices you can use when you are creating Activities for your Workflow: - Activity arguments and return values should be serializable. - Activities that perform writes should be idempotent. - Activities have [timeouts](/develop/python/activities/timeouts#activity-heartbeats) and [retry policies](/encyclopedia/retry-policies). For Activities, your operation should either complete within a few minutes or it should support the ability to heartbeat or poll for a result. This way it will be clear to the Workflow when the Activity is still making progress. - You need to specify at least one timeout, typically the [start_to_close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). Keep in mind that the shorter the timeout, the faster Temporal will retry upon failure. See the [Activity retry policy section](#activity-retry-policy) to learn more. --- # Activity Execution Source: https://docs.temporal.io/activity-execution > Understand how Activity Executions work in Temporal, including retries, timeouts, and failure handling. This page discusses the following: - [Activity Execution](#activity-execution) - [Cancellation](#cancellation) - [Activity Id](#activity-id) - [Asynchronous Activity Completion](#asynchronous-activity-completion) - [Task Token](#task-token) ## What is an Activity Execution? An Activity Execution is the full chain of [Activity Task Executions](/tasks#activity-task-execution). > **ℹ️ Info:** > > - [How to start an Activity Execution using the Go SDK](/develop/go/activities/execution) > - [How to start an Activity Execution using the Java SDK](/develop/java/activities/execution) > - [How to start an Activity Execution using the PHP SDK](/develop/php/activities/execution) > - [How to start an Activity Execution using the Python SDK](/develop/python/activities/execution) > - [How to start an Activity Execution using the TypeScript SDK](/develop/typescript/activities/execution) > - [How to start an Activity Execution using the .NET SDK](/develop/dotnet/activities/execution) > - [How to start an Activity Execution using the Ruby SDK](/develop/ruby/activities/execution) > - [How to start an Activity Execution using the Rust SDK](/develop/rust/activities/execution) > ![Activity Execution](/diagrams/activity-execution.svg) You can customize [Activity Execution timeouts](/encyclopedia/detecting-activity-failures#start-to-close-timeout) and [retry policies](/encyclopedia/retry-policies). If an Activity Execution fails (because it exhausted all retries, threw a [non-retryable error](/encyclopedia/retry-policies#non-retryable-errors), or was canceled), the error is returned to your [Workflow](/workflows) code when it attempts to fetch the Activity result. For [Standalone Activities](/standalone-activity) the error is returned to the Client when you attempt to fetch the Activity result. > **📝 Note:** > > Temporal guarantees that an Activity Task either runs or timeouts. There are multiple failure scenarios when an Activity > Task is lost. It can be lost during delivery to a Worker or after the Activity Function is called and the Worker > crashed. > > Temporal doesn't detect task loss directly. It relies on > [Start-To-Close timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). If the Activity Task times > out, the Activity Execution will be retried according to the Activity Execution Retry Policy. > > In scenarios where the Activity Execution Retry Policy is set to `1` and a Timeout occurs, the Activity Execution will > not be tried. > ## Cancellation Activity Cancellation: - lets the Activity know it doesn't need to keep doing work, and - gives the Activity time to clean up any resources it has created. Activities must heartbeat to receive cancellations from a Temporal Service. An Activity may receive Cancellation if: - The Activity was requested to be Cancelled. This can often cascade from Workflow Cancellation, but not always—SDKs have ways to stop Cancellation from cascading. - The Activity was considered failed by the Server because any of the Activity timeouts have triggered (for example, the Server didn't receive a heartbeat within the Activity's Heartbeat timeout). The [Cancelled Failure](/references/failures#cancelled-failure) that the Activity receives will have `message: 'TIMED_OUT'`. - The Workflow Run reached a [Closed state](/workflow-execution#workflow-execution-status), in which case the Cancelled Failure will have `message: 'NOT_FOUND'`. - In some SDKs: - The Worker is shutting down. - An Activity sends a Heartbeat but the Heartbeat details can't be converted by the Worker's configured [Data Converter](/dataconversion). This fails the Activity Task Execution with an Application Failure. - The Activity timed out on the Worker side and is not Heartbeating or the Temporal Service hasn't relayed a Cancellation. There are different ways to receive Cancellation depending on the SDK. An Activity may accept or ignore Cancellation: - To allow Cancellation to happen, let the Cancellation Failure propagate. - To ignore Cancellation, catch it and continue executing. Some SDKs have ways to shield tasks from being stopped while still letting the Cancellation propagate. The Workflow can also decide if it wants to wait for the Activity Cancellation to be accepted or to proceed without waiting. Cancellation can only be requested a single time. If you try to cancel your Activity Execution more than once, it will not receive more than one Cancellation request. ## What is an Activity Id? The identifier for an [Activity Execution](#activity-execution). The identifier can be generated by the system, or it can be provided by the Workflow code that spawns the Activity Execution. The identifier is unique among the open Activity Executions of a [Workflow Run](/workflow-execution/workflowid-runid#run-id). (A single Workflow Run may reuse an Activity Id if an earlier Activity Execution with the same Id has closed.) An Activity Id can be used to [complete the Activity asynchronously](#asynchronous-activity-completion). [Standalone Activities](/standalone-activity) have a separate ID space from Workflows and other Temporal primitives. This means use of conflict policy (`USE_EXISTING`, …) and reuse policy (`REJECT_DUPLICATES`, …) will only observe the Standalone Activity ID space. ## What is Asynchronous Activity Completion? Asynchronous Activity Completion is a feature that enables an Activity Function to return without causing the Activity Execution to complete. The Temporal Client can then be used from anywhere to both Heartbeat Activity Execution progress and eventually complete the Activity Execution and provide a result. How to complete an Activity Asynchronously in: - [.NET](/develop/dotnet/activities/asynchronous-activity) - [Go](/develop/go/activities/asynchronous-activity) - [Java](/develop/java/activities/asynchronous-activity) - [PHP](/develop/php/activities/asynchronous-activity) - [Python](/develop/python/activities/asynchronous-activity) - [Ruby](/develop/ruby/activities/asynchronous-activity) - [TypeScript](/develop/typescript/activities/asynchronous-activity) ### When to use Async Completion When an external system has the final result of a computation that is started by an Activity, there are three main ways of getting the result to the Workflow: 1. The external system uses Async Completion to complete the Activity with the result. 2. The Activity completes normally, without the result. Later, the external system sends a Signal to the Workflow with the result. 3. A subsequent Activity [polls the external system](https://community.temporal.io/t/what-is-the-best-practice-for-a-polling-activity/328/2) for the result. If you don't have control over the external system — that is, you can't add Async Completion or a Signal to its code — then: - you can poll (#3), or - if the external system can reliably call a webhook (and retry calling in the case of failure), you can write a webhook handler that sends a Signal to the Workflow (#2). The decision between using #1 vs #2 involves a few factors. Use Async Completion if: - the external system is unreliable and might fail to Signal, or - you want the external process to Heartbeat or receive Cancellation. Otherwise, if the external system can reliably be trusted to do the task and Signal back with the result, and it doesn't need to Heartbeat or receive Cancellation, then you may want to use Signals. The benefit to using Signals has to do with the timing of failure retries. For example, consider an external process that is waiting for a human to review something and respond, and they could take up to a week to do so. If you use Async Completion (#1), you would: - set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) of one week on the Activity, - in the Activity, notify the external process you need the human review, and - have the external process Asynchronously Complete the Activity when the human responds. If the Activity fails on the second step to notify the external system and doesn't throw an error (for example, if the Worker dies), then the Activity won't be retried for a week, when the Start-To-Close Timeout is hit. If you use Signals, you would: - set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) of one minute on the Activity, - in the Activity, notify the external process you need the human review, - complete the Activity without the result, and - have the external process Signal the Workflow when the human responds. If the Activity fails on the second step to notify the external system and doesn't throw an error, then the Activity will be retried in a minute. In the second scenario, the failure is retried sooner. This is particularly helpful in scenarios like this in which the external process might take a long time. ### What is a Task Token? A Task Token is a unique identifier for an [Activity Task Execution](/tasks#activity-task-execution). [Asynchronous Activity Completion](#asynchronous-activity-completion) calls take either of the following as arguments: - a Task Token, or - an [Activity Id](#activity-id), a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id), and optionally a [Run Id](/workflow-execution/workflowid-runid#run-id). Since a Task Token is unique for an Activity execution, retries can cause a remote service using the Task Token to end up with an invalid one. For example, an Activity might fail after passing its current Task Token to a remote service, but before returning the complete async error, leaving that service with a Task Token that's no longer valid. To avoid this risk, you can provide the Activity Id and Workflow Id to the remote service instead of the Task Token. --- # Activity Operations Source: https://docs.temporal.io/activity-operations > Operations you can perform on an Activity - Pause, Unpause, Reset, and Update Options. This page discusses the following: - [Pause](#pause) - [Unpause](#unpause) - [Reset](#reset) - [Update Options](#update-options) - [Observability](#observability) Activity Operations are deliberate actions you perform on a specific [Activity Execution](/activity-execution), as opposed to lifecycle behaviors like [retries](/encyclopedia/retry-policies) and [timeouts](/encyclopedia/detecting-activity-failures) which happen automatically. You can perform Activity Operations through the [CLI](/cli/command-reference/activity), the UI, or directly via the gRPC API. Activity Operations don't apply to [Local Activities](/local-activity) or [Standalone Activities](/standalone-activity). > **📝 Note:** > Public Preview > > Activity Operations are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview). > Pause, Unpause, and Reset are available in Server v1.28.0+. Self-hosted UI requires v2.47.0+. > > Activity Operations aren't available as SDK client methods. They're operational controls designed for the CLI, UI, and > gRPC API - not for programmatic use in Workflow or Activity code. > ## Operations summary | Operation | What it does | CLI | | --------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | [Pause](#pause) | Stops retries. In-flight execution continues unless the Activity uses Heartbeat. | [`temporal activity pause`](/cli/command-reference/activity#pause) | | [Unpause](#unpause) | Resumes a Paused Activity. The next execution starts immediately. | [`temporal activity unpause`](/cli/command-reference/activity#unpause) | | [Reset](#reset) | Clears retry state (attempts, backoff) and schedules a new execution. | [`temporal activity reset`](/cli/command-reference/activity#reset) | | [Update Options](#update-options) | Changes timeouts, Retry Policy, or Task Queue without restarting the Activity. | [`temporal activity update-options`](/cli/command-reference/activity#update-options) | ## Pause Pause stops the Temporal Service from scheduling new retries of an [Activity Execution](/activity-execution). ### When to Pause - An Activity is calling an external service that's experiencing issues, and you want to stop retries until the service recovers. - You need to inspect or change configuration before the Activity retries. - You're rolling out a new Worker version and want to hold specific Activities until the deploy is complete. ### What happens when you Pause an Activity - **Pausing an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and [Signals, Queries, and Updates](/encyclopedia/workflow-message-passing) on the parent Workflow are unaffected. - **No further retries are scheduled.** The Temporal Service stops scheduling retries. This is enforced server-side, not by the SDK. - **Workflow code has no visibility into Activity Operations.** Pause doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). - **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether the in-flight execution is interrupted:** - **Activities with Heartbeat** are interrupted on their next Heartbeat. The SDK raises a Pause-specific error, and the Activity can catch this to clean up resources before exiting. - **Activities without Heartbeat** continue running to completion. If the execution succeeds, the result is delivered to the Workflow normally. If it fails, no retry is scheduled. Pause takes effect after the in-flight execution ends. - **Pause is idempotent.** Pausing an already-Paused Activity has no effect. Pausing a completed Activity returns an error. ### CLI usage ```bash temporal activity pause \ --workflow-id my-workflow \ --activity-id my-activity \ --reason "Downstream API is down, pausing until recovery" ``` See the [CLI reference for `temporal activity pause`](/cli/command-reference/activity#pause) for all options. ### Detect Pause in Activity code Activities with Heartbeat can detect that an interruption was caused by Pause rather than a timeout or Workflow Cancellation. A Paused Activity resumes later. A Cancelled Activity doesn't. Your Activity code may need to handle these cases differently, for example releasing held resources on Pause while preserving them on Cancellation, or vice versa. | SDK | Version | How to detect Pause | | ---------- | -------- | -------------------------------------------------------------------- | | Go | v1.34.0+ | Catch `activity.ErrActivityPaused` | | Java | v1.29.0+ | Catch `ActivityPausedException` | | TypeScript | v1.12.3+ | Check `cancellationDetails.paused === true` | | Python | v1.12.0+ | Check `cancellation_details().paused` on `asyncio.CancelledError` | | .NET | v1.7.0+ | Check `CancellationDetails.IsPaused` on `OperationCanceledException` | ### Interaction with Workflow Pause [Workflow Pause](/encyclopedia/workflow/workflow-pause) and Activity Pause are independent. Both stop Activity retries, but they must be Unpaused separately. - Workflow Pause blocks retries but doesn't interrupt in-flight executions via Heartbeat. Activity Pause does. - If both are active, both must be Unpaused before the Activity resumes. ### Important considerations - **A Paused Activity can still time out.** Pause doesn't stop or extend the [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout). Use [`update-options`](#update-options) to adjust the timeout if needed. - **Pause won't interrupt an Activity that doesn't Heartbeat.** The current execution runs to completion, which could take up to the full [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). ### Limitations - **Pause operates on individual Activities by ID within a single Workflow.** Unlike Unpause, Reset, and Update Options, there's no `--query` flag. To pause multiple Activities, issue separate commands for each Activity ID. - **No Namespace-wide query for Paused Activities.** You must know the Workflow Id. See [Observability](#observability). ## Unpause Unpause resumes a Paused Activity Execution. ### When to Unpause - The downstream service or dependency that caused you to Pause has recovered. - A code deploy or configuration change is complete and the Activity is safe to retry. - You Paused an Activity for investigation and are ready to let it proceed. ### What happens when you Unpause an Activity - **The Activity is rescheduled immediately.** Any remaining retry backoff is discarded. The next execution starts right away. - **Attempt count and Heartbeat data are preserved by default.** The Activity resumes from where it left off. Use `--reset-attempts` or `--reset-heartbeats` on the CLI to clear these, or use [Reset](#reset) to restart from attempt 1. Unpause is idempotent. Unpausing an Activity that isn't Paused has no effect. Unpausing an Activity that has already completed returns an error. ### CLI usage ```bash temporal activity unpause \ --workflow-id my-workflow \ --activity-id my-activity ``` See the [CLI reference for `temporal activity unpause`](/cli/command-reference/activity#unpause) for all options, including `--reset-attempts` and `--reset-heartbeats` to clear state on resume. ### Important considerations - **Unpausing many Activities at once can overwhelm downstream services.** If you Paused multiple Activities because a service was down, Unpausing them all at the same time sends all retries simultaneously. Consider Unpausing in batches to avoid overwhelming a recovering service. - **Unpausing doesn't override Workflow Pause.** If the parent Workflow is also Paused, Unpausing the Activity alone isn't enough. Both must be Unpaused before the Activity resumes. See [Interaction with Workflow Pause](#interaction-with-workflow-pause). - **Unpausing doesn't reset the attempt count.** The Activity retries from its current attempt number. Use [Reset](#reset) to restart from attempt 1. - **A Paused Activity can time out before you Unpause it.** The [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) isn't stopped or extended while Paused. Use [`update-options`](#update-options) to extend the timeout before Unpausing if needed. - **Unpause doesn't interrupt or duplicate an in-flight execution.** If an Activity without Heartbeat is still running when you Unpause, it continues to completion. The Temporal Service doesn't schedule a concurrent execution. If the in-flight execution fails, the next retry proceeds normally. ## Reset Reset clears an Activity's retry state and schedules a fresh execution. ### When to Reset - An Activity has exhausted most of its retries, and you want to give it a fresh set after fixing the underlying issue. - A Paused Activity needs to start clean after a configuration change or code deploy. - You want to clear accumulated retry backoff and retry immediately instead of waiting for the next backoff interval. - A batch of Activities failed due to a transient issue and you want to restart them all with staggered jitter. ### What happens when you Reset an Activity - **The attempt count resets to 1.** The Activity gets a full set of retry attempts regardless of how many it had used. - **Retry backoff is discarded.** If the Activity was in a backoff wait, it's rescheduled to run immediately. - **If the Activity is Paused, Reset also Unpauses it.** Use `--keep-paused` to Reset the attempt count without resuming execution. With `--keep-paused`, the attempt count and Heartbeat data (if `--reset-heartbeats`) are reset, but the Activity stays Paused. No retry is scheduled until you [Unpause](#unpause) separately. - **Resetting an Activity doesn't affect the parent Workflow.** The Workflow continues Running, and Signals, Queries, and Updates on the parent Workflow are unaffected. - **Workflow code has no visibility into Activity Operations.** Reset doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). - **[Heartbeating](/encyclopedia/detecting-activity-failures#activity-heartbeat) determines whether an in-flight execution is interrupted:** - **Activities with Heartbeat** are interrupted on their next Heartbeat. The SDK may raise a Reset-specific error so the Activity can clean up before exiting. The next execution starts at attempt 1. - **Activities without Heartbeat** continue running to completion. Reset doesn't cancel, interrupt, or schedule a concurrent execution. If the Activity was already retrying, the Temporal Service rejects the current execution's result because Reset changed the expected attempt number, and a fresh execution is scheduled after the [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) expires. If it was on its first execution, a successful result is still delivered to the Workflow normally. - **Reset is idempotent.** Resetting an Activity that's already at attempt 1 with no backoff has no effect. Resetting a completed Activity returns an error. ### CLI usage ```bash temporal activity reset \ --workflow-id my-workflow \ --activity-id my-activity # Reset retry state but don't resume yet temporal activity reset \ --workflow-id my-workflow \ --activity-id my-activity \ --keep-paused ``` See the [CLI reference for `temporal activity reset`](/cli/command-reference/activity#reset) for all options, including `--reset-heartbeats` and bulk mode via `--query`. ### Detect Reset in Activity code Activities with Heartbeat can detect that an interruption was caused by Reset rather than a timeout or Workflow Cancellation. A Reset Activity is retried from attempt 1. A Cancelled Activity isn't. Your Activity code may need to handle these cases differently, for example saving partial progress on Reset while discarding it on Cancellation. | SDK | How to detect Reset | | ---------- | ---------------------------------------------------------------------------------- | | Go | `activity.GetCancellationDetails(ctx).Cause()` returns `activity.ErrActivityReset` | | Java | Catch `ActivityResetException` | | TypeScript | Catch `ApplicationFailure` with `error.type === "ActivityReset"` | | Python | Check `cancellation_details().reset` on `asyncio.CancelledError` | | .NET | Check `CancellationDetails.IsReset` on `OperationCanceledException` | ### Important considerations - **A Reset Activity can still time out.** Reset doesn't restart the [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout). The deadline is calculated from when the Activity was originally scheduled. Use [`update-options`](#update-options) to extend the timeout before or after Reset. - **Heartbeat details are preserved by default.** If your Activity uses Heartbeat details for progress tracking and you want a clean restart, pass `--reset-heartbeats`. - **Reset won't interrupt an Activity that doesn't Heartbeat.** The current execution runs to completion, which could take up to the full [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). If the Activity had already retried (attempt > 1), the Temporal Service rejects the current execution's result because Reset changed the expected attempt number. The Activity waits for its Start-To-Close Timeout to expire before a new execution is scheduled. - **`--restore-original-options` restores the Activity's original configuration.** It reverts timeouts, Retry Policy, and Task Queue to the values from when the Activity was first scheduled. - **Bulk Reset can overwhelm downstream services.** When using `--query` to Reset Activities across many Workflows, use `--jitter` to stagger the restart times. ## Update Options Update Options changes an Activity's runtime configuration without restarting it. ### When to Update Options - The [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) is about to expire on a Paused Activity, and you need to extend it before Unpausing. - An Activity's [Retry Policy](/encyclopedia/retry-policies) needs tuning based on observed failure patterns (for example, increasing the backoff interval or maximum attempts). - You want to move an Activity to a different [Task Queue](/task-queue) to route it to a specific set of [Workers](/workers). - You need to restore an Activity's original configuration after a temporary override. ### What happens when you Update an Activity's Options You can change [timeouts](/encyclopedia/detecting-activity-failures) (Schedule-To-Close, Start-To-Close, Schedule-To-Start, Heartbeat), Retry Policy (initial interval, maximum interval, backoff coefficient, maximum attempts), and Task Queue. Only the fields you specify are changed. All other options remain unchanged. - **If the Activity is waiting for retry (scheduled),** the new options take effect immediately. Any pending retry timer is regenerated with the updated configuration. - **If the Activity is currently running,** the new options are stored but take effect on the next execution. The in-flight execution isn't interrupted. - **If the Activity is Paused,** the new options are stored immediately. They take effect when the Activity is Unpaused and the next execution starts. - **Workflow code has no visibility into Activity Operations.** Update Options doesn't produce an Event History event, so the Workflow can't detect or react to it. See [Observability](#observability). Update Options is idempotent. Updating an Activity with the same values it already has produces no change. Updating options on an Activity that has already completed returns an error. ### CLI usage ```bash temporal activity update-options \ --workflow-id my-workflow \ --activity-id my-activity \ --schedule-to-close-timeout 24h ``` See the [CLI reference for `temporal activity update-options`](/cli/command-reference/activity#update-options) for all options, including Retry Policy, Task Queue, and bulk mode via `--query`. ### Important considerations - **Changes to a running Activity take effect on the next execution, not the current one.** If you need the change to apply immediately, the Activity must finish or fail its current execution first. - **`--restore-original-options` is batch-only.** This flag only works with `--query`. It's silently ignored in single-workflow mode. It can't be combined with other option changes in the same command. ## Observability Activity Operations have a limited audit trail because they are not recorded in a Workflow's Event History. However, you can use the CLI and the UI to check Activity state and find Paused Activities for running Workflows. ### Check Activity state `temporal workflow describe` shows the current state of each pending Activity, including whether it's Paused, its current attempt count, and last failure. The UI shows who performed an operation, when, and why (if a `--reason` was provided). ### Find Paused Activities The `TemporalPauseInfo` [Search Attribute](/search-attribute) is filterable within a Workflow. There's no Namespace-wide query to find all Paused Activities across Workflows. You must know the Workflow Id. ### Audit trail Activity Operations don't produce Event History events. There is no record of a Pause, Reset, or option change in the Workflow's [Event History](/workflow-execution/event#event-history). Nothing that reads the Event History - Workflow code, Replays, or external tooling - will see that an Operation occurred. Evidence of an Operation is gone when the Activity completes or the Workflow closes. There's no persistent record that an Activity was Paused, Reset, or had its options changed. The only way to confirm the current state of an Activity is `temporal workflow describe` or the UI. --- # Durable AI Source: https://docs.temporal.io/ai > Build durable AI agents and systems on Temporal, with runnable cookbook recipes, SDK integrations, and design patterns for agent workloads. Temporal gives AI applications and agents Durable Execution: a Workflow resumes automatically after a crash, a network timeout, or a multi-day wait for a human to approve a step. Temporal shows up in four recurring types of AI system: **Agents.** Long-running, stateful agent loops that call LLMs and tools, wait on humans, and pick up exactly where they left off after a failure. Start with the [AI Cookbook](/ai/cookbook) and the [Approval](/design-patterns/approval) and [Entity Workflow](/design-patterns/entity-workflow) patterns. **Processing pipelines.** Multi-step data and document pipelines, such as extraction, embedding, or batch inference, that need to fan out, retry failed steps in isolation, and resume without reprocessing completed work. When pipelines share Workers across models, tenants, or urgency levels, use [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness) to run urgent work ahead of bulk work and keep one tenant from starving the others. See the [batch processing](/design-patterns#batch-processing-patterns) and [QoS and throughput](/design-patterns/qos-throughput-patterns) patterns. **Internal agent platforms.** Teams building a shared runtime for many agents reuse Temporal's Worker and Task Queue primitives instead of building their own scheduler. See the [worker configuration patterns](/design-patterns#worker-configuration-patterns) for routing and isolating agent workloads. **Model training.** Long-running training and fine-tuning jobs coordinated across GPU resources, with checkpointing and recovery handled by Temporal's Event History instead of custom orchestration code. Start with the [long-running activity](/design-patterns/long-running-activity) and [parallel-execution](/design-patterns/parallel-execution) patterns. > Looking to use an AI coding assistant to write Temporal code instead? See [Develop with AI](/with-ai). ## AI Cookbook Runnable, step-by-step recipes for building AI systems and agents with Temporal: tool calling, MCP, structured output, human-in-the-loop, and more. - [Hello world](/ai/cookbook/hello-world-openai-responses-python) — Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library. - [Hello world with LiteLLM](/ai/cookbook/hello-world-litellm-python) — Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers. - [Durable agent with tools using the AI SDK by Vercel](/ai/cookbook/ai-sdk-by-vercel-typescript) — Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions. - [Structured outputs with Temporal and OpenAI](/ai/cookbook/structured-output-openai-responses-python) — Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure. [Browse all recipes](/ai/cookbook) ## Agent framework integrations Temporal integrations for the SDKs and frameworks teams use to build agents. This view is pre-filtered to agent frameworks — browse [every integration](/integrations) for the full catalog. - [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk) — Build AI-powered applications with Durable Execution using the Vercel AI SDK. _(TypeScript · Agent framework)_ - [Deep Agents](/develop/python/integrations/deepagents) — Make LangChain Deep Agents durable with Temporal Workflows and Activities. _(Python · Agent framework)_ - [Google ADK](/develop/python/integrations/google-adk) — Orchestrate Google ADK agents with durable Temporal Workflows. _(Python · Agent framework)_ - [Google ADK](/develop/go/integrations/google-adk) — Run Google ADK agents with durable execution using the Temporal Go SDK. _(Go · Agent framework)_ - [Google GenAI](/develop/python/integrations/google-genai) — Call Google Gemini models durably from Temporal Workflows with the Google Gen AI SDK. _(Python · Agent framework)_ - [LangGraph](/develop/python/integrations/langgraph) — Run LangGraph agent graphs as durable, resumable Temporal Workflows. _(Python · Agent framework)_ - [Mastra](https://mastra.ai/guides/deployment/temporal) — Build durable AI agents and workflows with the Mastra TypeScript framework. _(TypeScript · Agent framework)_ - [OpenAI Agents SDK](/develop/python/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(Python · Agent framework)_ - [OpenAI Agents SDK](/develop/typescript/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(TypeScript · Agent framework)_ - [Pydantic AI](https://ai.pydantic.dev/durable_execution/temporal/) — Build type-safe AI agents with Durable Execution through Pydantic AI. _(Python · Agent framework)_ - [Spring AI](/develop/java/integrations/spring-ai) — Build AI-powered Java applications with durable Spring AI tool calls. _(Java · Agent framework)_ - [Strands Agents](/develop/python/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(Python · Agent framework)_ - [Strands Agents](/develop/typescript/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(TypeScript · Agent framework)_ ## Featured from the Code Exchange A hand-picked look at samples built with Temporal and AI. Browse the full [Code Exchange](https://temporal.io/code-exchange) for more. - [AI enhanced e-commerce application](https://temporal.io/code-exchange/ai-enhanced-e-commerce-application): A sample e-commerce gift shop with hybrid full-text and vector search plus an AI-powered chat shopping assistant, built with Stripe and Temporal Workflows. _(Dotnet · Hybrid search)_ - [Temporal AI Question Planetarium](https://temporal.io/code-exchange/ai-question-planetarium): Runs a Hugging Face model inside Temporal Activities and Workers, streaming updates to the browser over WebSockets in real time. _(Python · Demo)_ - [Document Processing w/ AI](https://temporal.io/code-exchange/document-processing-w-ai): A mortgage underwriting demo that uses Gemini OCR and policy-grounded AI analysis in deterministic Workflows, with human-in-the-loop review and full traceability. _(Python · Gemini · Mortgage)_ - [Rust Confessional: a durable AI agent demo](https://temporal.io/code-exchange/rust-confessional): A live demo where an AI agent judges audience programming confessions, its progress surviving a Worker crash mid-task. _(Rust · Demo)_ ## Design patterns for AI agents - [Approval](/design-patterns/approval): Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata. - [Saga Pattern](/design-patterns/saga-pattern): Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail. - [Long-Running Activity](/design-patterns/long-running-activity): Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support. - [Entity Workflow](/design-patterns/entity-workflow): Models long-lived business entities as individual Workflows that persist for the entity's entire lifetime, handling all state transitions through Signals and Updates. - [Local Activities](/design-patterns/local-activities): Run Activity functions in-process inside the Workflow Task, eliminating all server scheduling round-trips. Best for short, idempotent Activities on a latency-sensitive path. Browse the full [Design Patterns catalog](/design-patterns) for more, or jump straight into the [AI Cookbook](/ai/cookbook) for runnable code. --- # AI Cookbook Source: https://docs.temporal.io/ai/cookbook > Step-by-step recipes for building reliable AI systems with Temporal, covering LLM integrations, agentic loops, tool calling, and production patterns. - [Hello world](/ai/cookbook/hello-world-openai-responses-python) — Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library. - [Hello world with LiteLLM](/ai/cookbook/hello-world-litellm-python) — Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers. - [Durable agent with tools using the AI SDK by Vercel](/ai/cookbook/ai-sdk-by-vercel-typescript) — Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions. - [Structured outputs with Temporal and OpenAI](/ai/cookbook/structured-output-openai-responses-python) — Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure. - [Retry policy from HTTP responses](/ai/cookbook/http-retry-enhancement-python) — Extract retry information from HTTP response headers and pass it to Temporal's retry mechanisms in Python. - [Basic agentic loop with Claude and tool calling](/ai/cookbook/agentic-loop-tool-call-claude-python) — Build a durable agentic loop in Python with Claude tool calling and Temporal. - [Basic agentic loop with OpenAI and tool calling](/ai/cookbook/agentic-loop-tool-call-openai-python) — Build a durable agentic loop in Python that calls a dynamic set of tools with Temporal and the OpenAI Responses API. - [Durable MCP weather server](/ai/cookbook/hello-world-durable-mcp-server) — Build a durable MCP server in Python that runs weather tools reliably with Temporal Workflows. - [Tool calling agent](/ai/cookbook/tool-call-openai-python) — Build a simple, non-looping Python agent that lets the LLM choose tools and then invokes the chosen tools with Temporal and OpenAI. - [Durable agent with MCP and Activity-backed tools using the Strands Agents SDK](/ai/cookbook/strands-agents-python) — Build a durable AI agent in Python with Temporal and the Strands Agents SDK plugin, combining an MCP server tool with an Activity-backed tool that calls a live HTTP feed. - [Durable agent with tools using the OpenAI Agents SDK](/ai/cookbook/openai-agents-sdk-python) — Build a durable AI agent with the OpenAI Agents SDK and Temporal that chooses tools to answer user questions. - [Human-in-the-loop AI agent](/ai/cookbook/human-in-the-loop-python) — Add human-in-the-loop approval to a durable AI agent using Temporal Signals in Python. - [Post-LLM guardrail with hard-rule overrides](/ai/cookbook/guardrails-hard-rules-python) — Build a durable content-moderation guardrail in Python with Temporal and Claude that layers deterministic hard rules over an LLM's verdict for auditable overrides. - [Claim check pattern with Temporal](/ai/cookbook/claim-check-pattern-python) — Use the Claim Check pattern with Temporal to keep large payloads out of Event History by offloading them to S3. - [Deep research](/ai/cookbook/basic-openai-python) — Build a multi-agent deep research system in Python with Temporal and the OpenAI Responses API. --- # Basic agentic loop with Claude and tool calling Source: https://docs.temporal.io/ai/cookbook/agentic-loop-tool-call-claude-python > Build a durable agentic loop in Python with Claude tool calling and Temporal. This example implements an agentic loop using Claude (Anthropic) that has a set of tools available. If the agent determines that no tools are needed to satisfy a user request, it will return the response directly. If Claude determines a tool should be used, it will return with the name of the chosen tool and any needed parameters. The agent then invokes the appropriate tool. Tools are supplied to Claude's Messages API through the `tools` parameter. The `tools` parameter is in JSON format and includes a description of the function as well as descriptions of each of the arguments using Claude's `input_schema` format. Being external API calls, invoking Claude and invoking any functions/tools are done within a Temporal Activity. This recipe highlights the following key design decisions: - We use dynamic Activities to allow the agent to be loosely coupled from specific tools. This sample isolates the tools in the `tools` directory; changing the tools requires NO changes to the agent implementation. - Because there is an agentic loop, each Claude invocation is passed the accumulated *conversation history* in a structured messages array with role alternation (user/assistant). - Claude can return multiple tool calls in a single response, and can mix text with tool calls in the same response. - A generic Activity for invoking Claude's Messages API; instructions and other parameters are passed into the Activity making it appropriate for use in a variety of different use cases. - Retries are handled by Temporal and not by the Anthropic client library. This is important because client retries can interfere with correct and durable error handling and recovery. Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python). ## Application components This example includes the following components: - The [Workflow](#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation. - The Activities for [invoking Claude](#create-the-activity-for-claude-invocations) and for [invoking tools](#create-the-activity-for-the-tool-invocation). - A [helper function](#create-the-helper-function) that creates tool definitions in Claude's format. - Sample [tools](#create-tool-definitions). - The [Worker](#create-the-worker) that manages the Workflow and the Activities. - An application that [initiates an interaction](#initiate-an-interaction-with-the-agent) with the agent. ## Create the agent (agentic loop) ### Create the main agentic loop The agent is implemented as a Temporal Workflow that implements an agentic loop. The loop will continue until the agent responds with no tool calls. Each time through the loop: - Claude is called with the accumulated conversation history that is made up of the initial user input and any previous assistant responses and tool outputs. - The Workflow checks if Claude returned any tool calls (content blocks with `type: "tool_use"`). - If tool calls are present, the assistant's complete response (including all content blocks) is appended to the messages array, then all tools are executed, and their results are added as a user message. - If no tool has been called, the text response is returned. *File: workflows/agent.py* ```python from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import claude_responses from helpers import tool_helpers from tools import get_tools @workflow.defn class AgentWorkflow: @workflow.run async def run(self, input: str) -> str: # Initialize messages list with user input messages = [{"role": "user", "content": input}] print(f"\n[User] {input}") # The agentic loop while True: # Consult Claude result = await workflow.execute_activity( claude_responses.create, claude_responses.ClaudeResponsesRequest( model="claude-sonnet-4-5-20250929", system=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS, messages=messages, tools=get_tools(), max_tokens=4096, ), start_to_close_timeout=timedelta(seconds=30), ) # Claude returns content blocks - check if any are tool_use tool_use_blocks = [block for block in result.content if block.type == "tool_use"] if tool_use_blocks: # We have tool calls to handle # First, add the assistant's response to messages # Convert content blocks to dictionaries for serialization assistant_content = [] for block in result.content: if block.type == "text": assistant_content.append({"type": "text", "text": block.text}) elif block.type == "tool_use": print(f"[Agent] Calling tool: {block.name}") assistant_content.append({ "type": "tool_use", "id": block.id, "name": block.name, "input": block.input }) messages.append({"role": "assistant", "content": assistant_content}) # Execute all tool calls and collect results tool_results = [] for block in tool_use_blocks: # Execute the tool tool_result = await self._execute_tool(block.name, block.input) # Add tool result in Claude's expected format tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": str(tool_result) }) # Add tool results as a user message. Claude has only two message roles: user and assistant, # and the user message role is used to send tool results back to Claude; in this case the content # block includes the tool result. messages.append({"role": "user", "content": tool_results}) else: # No tool calls - extract the text response and return text_blocks = [block for block in result.content if block.type == "text"] if text_blocks: response_text = text_blocks[0].text print(f"[Agent] Final response: {response_text}\n") return response_text else: return "No text response from Claude" ``` ### Create the tool execution handler The tool execution handler is invoked by the main agentic loop when Claude has chosen tools. Because the Activity implementation is dynamic, the arguments are passed to the Activity as a dictionary. The Activity invocation is the same as any non-dynamic Activity invocation, passing the name of the Activity, the arguments, and any Activity configurations. *File: workflows/agent.py* ```python async def _execute_tool(self, tool_name: str, tool_input: dict) -> str: """ Execute a tool dynamically. Args: tool_name: Name of the tool to execute tool_input: Dictionary of input parameters """ # Execute dynamic activity with the tool name and arguments result = await workflow.execute_activity( tool_name, tool_input, start_to_close_timeout=timedelta(seconds=30), ) return result ``` ## Create the Activity for Claude invocations We create a wrapper for the `create` method of the `AsyncAnthropic` client object. This is a generic Activity that invokes Claude's Messages API. We set `max_retries=0` when creating the `AsyncAnthropic` client. This moves the responsibility for retries from the Anthropic client to Temporal. The Activity then has to interpret the errors coming from Claude's API so the Workflow knows whether to retry. Errors that can never succeed on a retry, such as a 404 for a retired model identifier or a 401 for a bad API key, are re-raised as a non-retryable `ApplicationError`. Without this they would be retried under the default retry policy until the Workflow timed out, which hides the real problem. Transient errors such as rate limits and 5xx responses propagate unchanged so that Temporal retries them. In this implementation, we allow for the model, system instructions, messages, list of tools, and max_tokens (required) to be passed in. *File: activities/claude_responses.py* ```python from dataclasses import dataclass from typing import Any import anthropic from anthropic import AsyncAnthropic from anthropic.types import Message from temporalio import activity from temporalio.exceptions import ApplicationError # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class ClaudeResponsesRequest: model: str system: str messages: list[dict[str, Any]] tools: list[dict[str, Any]] max_tokens: int = 4096 @activity.defn async def create(request: ClaudeResponsesRequest) -> Message: # We disable retry logic in Anthropic API client library so that Temporal can handle retries. client = AsyncAnthropic(max_retries=0) try: resp = await client.messages.create( model=request.model, system=request.system, messages=request.messages, tools=request.tools, max_tokens=request.max_tokens, ) return resp except ( anthropic.BadRequestError, anthropic.AuthenticationError, anthropic.PermissionDeniedError, anthropic.NotFoundError, anthropic.UnprocessableEntityError, ) as exc: # These errors will never succeed on a retry: a retired model identifier, for # example, returns 404. Retrying them under the default policy would loop # forever instead of surfacing the problem. Everything else, such as rate # limits and 5xx responses, propagates so Temporal can retry it. raise ApplicationError( str(exc), type=exc.__class__.__name__, non_retryable=True, ) from exc finally: await client.close() ``` ## Create the Activity for the tool invocation Implement a single tool invocation Activity, as a dynamic Activity (note the `@activity.defn(dynamic=True)` annotation) that acts as a broker to the right tool function. The name of the Activity is drawn from the `activity.info()` and the property bag of arguments from the Activity payload. The `handler` is the function that maps to the `tool_name` (see [Create Tool Definitions](#create-tool-definitions) for more details) and that function is then called with the supplied arguments. *File: activities/tool_invoker.py* ```python import inspect from collections.abc import Sequence from pydantic import BaseModel from temporalio import activity from temporalio.common import RawValue # We use dynamic activities to allow the agent to be defined independently of the tools it can call. @activity.defn(dynamic=True) async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict: from tools import get_handler # the name of the tool to execute - this is passed in via the execute_activity call in the workflow tool_name = activity.info().activity_type tool_args = activity.payload_converter().from_payload(args[0].payload, dict) activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}") handler = get_handler(tool_name) # in dynamic activity sig = inspect.signature(handler) params = list(sig.parameters.values()) if len(params) == 0: call_args = [] else: ann = params[0].annotation if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel): call_args = [ann(**tool_args)] # or ann.model_validate(tool_args) on Pydantic v2 else: call_args = [tool_args] if not inspect.iscoroutinefunction(handler): raise TypeError("Tool handler must be async (awaitable).") result = await handler(*call_args) # Optionally log or augment the result activity.logger.info(f"Tool '{tool_name}' result: {result}") return result ``` ## Create the helper function The `claude_tool_from_model` function accepts a tool name and description, as well as a Pydantic model for the parameters, and returns JSON that is in the format expected for tool definitions in Claude's Messages API. *File: helpers/tool_helpers.py* ```python def claude_tool_from_model(name: str, description: str, model: type[BaseModel] | None) -> dict[str, Any]: """ Convert a Pydantic model to Claude's tool format. Claude's tool format structure: { "name": "tool_name", "description": "Tool description", "input_schema": { "type": "object", "properties": {...}, "required": [...] } } """ if model is None: # For tools without parameters return { "name": name, "description": description, "input_schema": { "type": "object", "properties": {}, "required": [] } } # Get the JSON schema from the Pydantic model schema = model.model_json_schema() # Claude expects an input_schema field instead of parameters return { "name": name, "description": description, "input_schema": schema } ``` This file also holds the system instruction for the agent. ```python HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """ You are a helpful agent that can use tools to help the user. You will be given input from the user and a list of tools to use. You may or may not need to use the tools to satisfy the user ask. If no tools are needed, respond in haikus. """ ``` ## Create tool definitions Tools are defined in the `tools` directory and should be thought of as independent from the agent implementation; as described above, dynamic Activities are used for this loose coupling. The `__init__.py` file holds tools for providing location (`get_location_info`), IP address (`get_ip_address`), and weather alerts (`get_weather_alerts`). - The `get_tools` method returns the set of tool definitions that will be passed to Claude. - The `get_handler` method captures the mapping from tool name to tool function. *File: tools/\_\_init\_\_.py* ```python from typing import Any, Awaitable, Callable from . import get_location, get_weather # Location and weather related tools from .get_location import get_ip_address, get_location_info from .get_weather import get_weather_alerts ToolHandler = Callable[..., Awaitable[Any]] def get_handler(tool_name: str) -> ToolHandler: if tool_name == "get_location_info": return get_location_info if tool_name == "get_ip_address": return get_ip_address if tool_name == "get_weather_alerts": return get_weather_alerts raise ValueError(f"Unknown tool name: {tool_name}") def get_tools() -> list[dict[str, Any]]: return [ get_weather.WEATHER_ALERTS_TOOL_CLAUDE, get_location.GET_LOCATION_TOOL_CLAUDE, get_location.GET_IP_ADDRESS_TOOL_CLAUDE ] ``` The tool descriptions and functions are defined in `tools/get_location.py`, `tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains: - data structures for function arguments - tool definitions (in JSON form using Claude's `input_schema` format) - the function definitions. `tools/get_location.py` ```python # get_location.py from typing import Any import httpx from pydantic import BaseModel, Field from helpers import tool_helpers # For the location finder we use Pydantic to create a structure that encapsulates the input parameter # (an IP address). # This is used for both the location finding function and to craft the tool definitions that # are passed to Claude. class GetLocationRequest(BaseModel): ipaddress: str = Field(description="An IP address") # Build the tool definitions for Claude GET_LOCATION_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model( "get_location_info", "Get the location information for an IP address. This includes the city, state, and country.", GetLocationRequest) GET_IP_ADDRESS_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model( "get_ip_address", "Get the IP address of the current machine.", None) # The functions async def get_ip_address() -> str: async with httpx.AsyncClient() as client: response = await client.get("https://icanhazip.com") response.raise_for_status() return response.text.strip() async def get_location_info(req: GetLocationRequest) -> str: async with httpx.AsyncClient() as client: response = await client.get(f"http://ip-api.com/json/{req.ipaddress}") response.raise_for_status() result = response.json() return f"{result['city']}, {result['regionName']}, {result['country']}" ``` ## Create the Worker The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for Claude and tool invocations. *File: worker.py* ```python import asyncio from concurrent.futures import ThreadPoolExecutor from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.worker import Worker from activities import claude_responses, tool_invoker from workflows.agent import AgentWorkflow async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-invoking-agent-claude-python-task-queue", workflows=[ AgentWorkflow, ], activities=[ claude_responses.create, tool_invoker.dynamic_tool_activity, ], activity_executor=ThreadPoolExecutor(max_workers=10), ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Initiate an interaction with the agent To interact with this simple AI agent, we create a Temporal client and execute a Workflow. *File: start_workflow.py* ```python import asyncio import sys import uuid from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from workflows.agent import AgentWorkflow async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion" # Submit the agent workflow for execution result = await client.execute_workflow( AgentWorkflow.run, query, id=f"agentic-loop-claude-id-{uuid.uuid4()}", task_queue="tool-invoking-agent-claude-python-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Running the app In the terminal where you run the agent Worker, set an Anthropic API key: ``` export ANTHROPIC_API_KEY=sk-ant-... ``` ``` uv sync ``` Start the agent Worker: ```bash uv run worker.py ``` Make request to the agent: ```bash uv run start_workflow.py "are there any weather alerts for where I am?" ``` Try a number of different user prompts: ```bash uv run start_workflow.py "where am I?" uv run start_workflow.py "what is my ip address?" uv run start_workflow.py "tell me about recursion" ``` ## Troubleshooting **`not_found_error` naming the model**: Anthropic retires older model identifiers, and this recipe pins one in `workflows/agent.py`. Check the [list of current models](https://docs.claude.com/en/docs/about-claude/models/overview) and update the identifier. The Activity classifies this error as non-retryable, so the Workflow fails with the API message rather than retrying. **`authentication_error`**: `ANTHROPIC_API_KEY` is unset or invalid in the terminal running the Worker. The Activity, not `start_workflow`, makes the API call, so the key has to be set where the Worker runs. **Workflow runs but no tool is invoked**: Claude decides whether a tool is needed. Prompts like "tell me about recursion" are answered directly. Check the Event History in the [Temporal UI](http://localhost:8233) to see which Activities were scheduled. --- # Basic agentic loop with OpenAI and tool calling Source: https://docs.temporal.io/ai/cookbook/agentic-loop-tool-call-openai-python > Build a durable agentic loop in Python that calls a dynamic set of tools with Temporal and the OpenAI Responses API. This example implements a basic agentic loop that has a set of tools available. If the agent determines that no tools are needed to satisfy a user request, it will respond directly. If the LLM determines a tool should be used it will return with the name of the chosen tool and any needed parameters. The agent then invokes the appropriate tool. Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools). The `tools` parameter is in `json` format and includes a description of the function as well as descriptions of each of the arguments. > **⚠️ Caution:** > The API used to generate the tools `json` is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. Being external API calls, invoking the LLM and invoking any functions/tools are done within a Temporal Activity. This recipe highlights the following key design decisions: - We use dynamic Activities to allow the agent to be loosely coupled from specific tools. This sample isolates the tools in the `tools` directory; changing the tools requires no changes to the agent implementation. - Because there is an agentic loop, each LLM invocation is passed the accumulated *conversation history*, that includes the initial user input as well as LLM and tool calls. - A generic Activity for invoking an LLM API; that is, instructions and other `responses` arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases. - Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python). ## Application components This example includes the following components: - The [Workflow](#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation. - The activities for [invoking the LLM](#create-the-activity-for-llm-invocations) and for [invoking tools](#create-the-activity-for-the-tool-invocation). - A [helper function](#create-the-helper-function) that creates tool definitions of the appropriate form. - Sample [tools](#create-tool-definitions). - The [worker](#create-the-worker) that manages the Workflow and the Activities. - An application that [initiates an interaction](#initiate-an-interaction-with-the-agent) with the agent. ## Create the agent (agentic loop) ### Create the main agentic loop The agent is implemented as a Temporal Workflow that: - implements an agentic loop. The loop will continue until the agent responds with no tool calls. Each time through the loop: - the LLM is called with the accumulated conversation history that is made up of the initial user input and any previous LLM responses and tool outputs. - the invocation of the function, if the LLM has chosen one - if a function is called the function result is added to the conversation history - if no tool has been called, the LLM response is returned. This example demonstrates a most simple UX where the user provides single shot input. Note however that the agent is not single shot. *File: workflows/agent.py* ```python from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import openai_responses from helpers import tool_helpers from tools import get_tools @workflow.defn class AgentWorkflow: @workflow.run async def run(self, input: str) -> str: input_list = [{"type": "message", "role": "user", "content": input}] # The agentic loop while True: print(80 * "=") # consult the LLM result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS, input=input_list, tools=get_tools(), ), start_to_close_timeout=timedelta(seconds=30), ) # For this simple example, we only have one item in the output list # Either the LLM will have chosen a single function call or it will # have chosen to respond with a message. item = result.output[0] # Now process the LLM output to either call a tool or respond with a message. # if the result is a tool call, call the tool if item.type == "function_call": result = await self._handle_function_call(item, result, input_list) # add the tool call result to the input list for context input_list.append({"type": "function_call_output", "call_id": item.call_id, "output": result}) # if the result is not a tool call we will just respond with a message else: print(f"No tools chosen, responding with a message: {result.output_text}") return result.output_text ``` ### Create the function call handler The function call handler is invoked by the main agentic loop when an LLM has chosen a tool. Because the activty implementation is dynamic, the arguments are passed to the Activity in a property bag; the `args` variable is appropriately set. Otherwise, the Activity invocation is the same as any non-dynamic Activity invocation passing the name of the Activity, the arguments and any Activity configurations. *File: workflows/agent.py* ```python async def _handle_function_call(self, item, result, input_list): # serialize the LLM output - the decision the LLM made to call a tool i = result.output[0] input_list += [ i.model_dump() if hasattr(i, "model_dump") else i ] # execute dynamic activity with the tool name chosen by the LLM # and the arguments crafted by the LLM args = json.loads(item.arguments) if isinstance(item.arguments, str) else item.arguments result = await workflow.execute_activity( item.name, args, start_to_close_timeout=timedelta(seconds=30), ) print(f"Made a tool call to {item.name}") return result ``` ## Create the Activity for LLM invocations We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. This means that the Activity should interpret any errors coming from the OpenAI API call and return the appropriate error type so that the workflow knows if it should retry the Activity or not. In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools. *File: activities/openai_responses.py* ```python from dataclasses import dataclass from typing import Any from openai import AsyncOpenAI from openai.types.responses import Response from temporalio import activity # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class OpenAIResponsesRequest: model: str instructions: str input: object tools: list[dict[str, Any]] @activity.defn async def create(request: OpenAIResponsesRequest) -> Response: # We disable retry logic in OpenAI API client library so that Temporal can handle retries. # In a real setting, you would need to handle any errors coming back from the OpenAI API, # so that Temporal can appropriately retry in the manner that OpenAI API would. # See the `http_retry_enhancement_python` example for inspiration. client = AsyncOpenAI(max_retries=0) try: resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, tools=request.tools, timeout=30, ) return resp finally: await client.close() ``` ## Create the Activity for the tool invocation Implement a single tool invocation Activity, as a dynamic Activity (note the `@activity.defn(dynamic=True)` annotation) that acts as a broker to the right tool function. The name of the Activity is drawn from the `activity.info()` and the property bag of arguments from the Activity payload. The `handler` is the function that maps to the `tool_name` (see [Create Tool Definitions](#create-tool-definitions) for more details) and that function is then called with the supplied arguments. *File: activities/tool_invoker.py* ```python import inspect from collections.abc import Sequence from pydantic import BaseModel from temporalio import activity from temporalio.common import RawValue # We use dynamic activities to allow the agent to be defined independently of the tools it can call. @activity.defn(dynamic=True) async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict: from tools import get_handler # the name of the tool to execute - this is passed in via the execute_activity call in the Workflow tool_name = activity.info().activity_type tool_args = activity.payload_converter().from_payload(args[0].payload, dict) activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}") handler = get_handler(tool_name) # in dynamic activity sig = inspect.signature(handler) params = list(sig.parameters.values()) if len(params) == 0: call_args = [] else: ann = params[0].annotation if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel): call_args = [ann(**tool_args)] # or ann.model_validate(tool_args) on Pydantic v2 else: call_args = [tool_args] if not inspect.iscoroutinefunction(handler): raise TypeError("Tool handler must be async (awaitable).") result = await handler(*call_args) # Optionally log or augment the result activity.logger.info(f"Tool '{tool_name}' result: {result}") return result ``` ## Create the helper function The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API. > **⚠️ Caution:** > The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. *File: helpers/tool_helpers.py* ```python from openai.lib._pydantic import to_strict_json_schema # private API; may change # there currently is no public API to generate the tool definition from a Pydantic model # or a function signature. from pydantic import BaseModel def oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]): return { "type": "function", "name": name, "description": description, # OpenAI Responses strict tools require a JSON Schema object where # additionalProperties is explicitly false. For tools without # parameters, supply an empty object schema. "parameters": ( to_strict_json_schema(model) if model else {"type": "object", "properties": {}, "required": [], "additionalProperties": False} ), "strict": True, } ``` This file also holds the system instruction for the agent. ```python HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """ You are a helpful agent that can use tools to help the user. You will be given a input from the user and a list of tools to use. You may or may not need to use the tools to satisfy the user ask. If no tools are needed, respond in haikus. """ ``` ## Create tool definitions Tools are defined in the `tools` directory and should be thought of as independent from the agent implementation; as described above, dynamic Activities are used for this loose coupling. The `__init__.py` file holds two examples of tool sets, one providing location and weather tools, the other a simple random number generating tool; comment and uncomment sets you would like to include (or combine them by updating the `get_tools` and `get_handler` methods). - The `get_tools` method returns the set of tool definitions that will be passed to the LLM. - The `get_handler` method captures the mapping from tool name to tool function *File: tools/\_\_init\_\_.py* ```python # Uncomment and comment out the tools you want to use from typing import Any, Awaitable, Callable # Location and weather related tools from .get_location import ( GET_IP_ADDRESS_TOOL_OAI, GET_LOCATION_TOOL_OAI, get_ip_address, get_location_info, ) from .get_weather import WEATHER_ALERTS_TOOL_OAI, get_weather_alerts ToolHandler = Callable[..., Awaitable[Any]] def get_handler(tool_name: str) -> ToolHandler: if tool_name == "get_location_info": return get_location_info if tool_name == "get_ip_address": return get_ip_address if tool_name == "get_weather_alerts": return get_weather_alerts raise ValueError(f"Unknown tool name: {tool_name}") def get_tools() -> list[dict[str, Any]]: return [WEATHER_ALERTS_TOOL_OAI, GET_LOCATION_TOOL_OAI, GET_IP_ADDRESS_TOOL_OAI] # Random number tool # from .random_stuff import get_random_number, RANDOM_NUMBER_TOOL_OAI # def get_handler(tool_name: str) -> ToolHandler: # if tool_name == "get_random_number": # return get_random_number # raise ValueError(f"Unknown tool name: {tool_name}") # def get_tools() -> list[dict[str, Any]]: # return [RANDOM_NUMBER_TOOL_OAI] ``` The tool descriptions and functions are defined in `tools/get_location.py`, `tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains: - data structures for function arguments - tool definitions (in `json` form) - the function definitions. `tools/get_location.py` ```python # get_location.py from typing import Any import httpx from pydantic import BaseModel, Field from helpers import tool_helpers # For the location finder we use Pydantic to create a structure that encapsulates the input parameter # (an IP address). # This is used for both the location finding function and to craft the tool definitions that # are passed to the OpenAI Responses API. class GetLocationRequest(BaseModel): ipaddress: str = Field(description="An IP address") # Build the tool definitions for the OpenAI Responses API. GET_LOCATION_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_location_info", "Get the location information for an IP address. This includes the city, state, and country.", GetLocationRequest) GET_IP_ADDRESS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_ip_address", "Get the IP address of the current machine.", None) # The functions async def get_ip_address() -> str: async with httpx.AsyncClient() as client: response = await client.get("https://icanhazip.com") response.raise_for_status() return response.text.strip() async def get_location_info(req: GetLocationRequest) -> str: async with httpx.AsyncClient() as client: response = await client.get(f"http://ip-api.com/json/{req.ipaddress}") response.raise_for_status() result = response.json() return f"{result['city']}, {result['regionName']}, {result['country']}" ``` See files in GitHub for more tool definitions. ## Create the Worker The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for the LLM and tool invocations. *File: worker.py* ```python import asyncio from concurrent.futures import ThreadPoolExecutor from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.envconfig import ClientConfig from temporalio.worker import Worker from activities import openai_responses, tool_invoker from workflows.agent import AgentWorkflow async def main(): config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") client = await Client.connect( **config, data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-invoking-agent-python-task-queue", workflows=[ AgentWorkflow, ], activities=[ openai_responses.create, tool_invoker.dynamic_tool_activity, ], activity_executor=ThreadPoolExecutor(max_workers=10), ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Initiate an interaction with the agent To interact with this simple AI agent, we create a Temporal client and execute a Workflow. *File: start_workflow.py* ```python import asyncio import sys import uuid from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from workflows.agent import AgentWorkflow async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion" # Submit the the agent workflow for execution result = await client.execute_workflow( AgentWorkflow.run, query, id=f"agentic-loop-id-{uuid.uuid4()}", task_queue="tool-invoking-agent-python-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Running the app In the terminal where you run the agent worker, set an OpenAI API key: ``` export OPENAI_API_KEY=sk... ``` ``` uv sync ``` Start the agent worker: ```bash uv run worker.py ``` Make request to the agent: ```bash uv run start_workflow.py "are there any weather alerts for where I am?" ``` Try a number of different user prompts: ```bash uv run start_workflow.py "where am I?" uv run start_workflow.py "what is my ip address?" uv run start_workflow.py "can I please have a random number?" ``` --- # Durable agent with tools using the AI SDK by Vercel Source: https://docs.temporal.io/ai/cookbook/ai-sdk-by-vercel-typescript > Build a durable AI agent with the AI SDK by Vercel and Temporal that chooses tools to answer user questions. In this example, we show you how to build a durable agent using the [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk#provide-your-durable-agent-with-tools). The agent calls tools backed by Temporal Activities to answer user questions, and it can determine which tools to use based on the input it receives. This recipe highlights key implementation patterns: - **AI SDK client integration**: The Workflow uses `generateText` from `ai` and `temporalProvider` from `@temporalio/ai-sdk/workflow`. This automatically wraps the LLM invocation as an Activity, so it's retried and tracked like any other durable step. `temporalProvider` is configured for `gpt-4o-mini` here, but you can point it at any model the AI SDK supports. - **Tools-as-Activities**: `proxyActivities` wires the `getWeather` and `calculateAreaOfCircle` Activities into the Workflow so `toolsAgent` can offer tool schemas to the model, wait for results durably, and retry a tool call if it fails. Unlike some other Temporal AI integrations — for example, the OpenAI Agents SDK's `activity_as_tool` helper, which generates a tool schema from a Python function's type hints — the Vercel AI SDK's `tool()` has no equivalent auto-generation from a TypeScript function signature. Each tool's `inputSchema` is written by hand as a Zod schema. ## Create the Activity Temporal Activities provide the tools that `toolsAgent` can call. `getWeather` demonstrates an Activity that wraps an unreliable external call: it geocodes the city name (via the free [Open-Meteo geocoding API](https://open-meteo.com/en/docs/geocoding-api)) and then queries the [National Weather Service API](https://www.weather.gov/documentation/services-web-api), both of which are free and require no API key. Because the NWS API only covers the United States, use a US city in your prompt. `calculateAreaOfCircle` shows the opposite case — a tool that runs entirely locally with no external I/O. *File: src/activities.ts* ```typescript const USER_AGENT = '(temporal-ai-cookbook, cookbook@temporal.io)'; async function geocode(location: string): Promise<{ name: string; latitude: number; longitude: number }> { const url = `https://geocoding-api.open-meteo.com/v1/search?name=${encodeURIComponent(location)}&count=1&countryCode=US`; const response = await fetch(url); if (!response.ok) { throw new Error(`Geocoding request failed for "${location}": ${response.status}`); } const data = (await response.json()) as { results?: Array<{ name: string; latitude: number; longitude: number }>; }; const match = data.results?.[0]; if (!match) { throw new Error(`No location found for "${location}". The National Weather Service only covers the US.`); } return match; } // The National Weather Service API only covers the US and requires a two-step // lookup: resolve coordinates to a forecast grid, then fetch that grid's forecast. export async function getWeather(input: { location: string; }): Promise<{ city: string; temperatureRange: string; conditions: string }> { const { name, latitude, longitude } = await geocode(input.location); const pointsResponse = await fetch(`https://api.weather.gov/points/${latitude.toFixed(4)},${longitude.toFixed(4)}`, { headers: { 'User-Agent': USER_AGENT }, }); if (!pointsResponse.ok) { throw new Error(`National Weather Service points lookup failed: ${pointsResponse.status}`); } const points = (await pointsResponse.json()) as { properties: { forecast: string } }; const forecastResponse = await fetch(points.properties.forecast, { headers: { 'User-Agent': USER_AGENT } }); if (!forecastResponse.ok) { throw new Error(`National Weather Service forecast lookup failed: ${forecastResponse.status}`); } const forecast = (await forecastResponse.json()) as { properties: { periods: Array<{ temperature: number; temperatureUnit: string; shortForecast: string }> }; }; const current = forecast.properties.periods[0]; return { city: name, temperatureRange: `${current.temperature}${current.temperatureUnit}`, conditions: current.shortForecast, }; } export async function calculateAreaOfCircle(input: { radius: number }): Promise<{ area: number }> { return { area: Math.PI * input.radius * input.radius }; } ``` ## Create the Workflow The Workflow registers both Activities as tools with a Zod schema so the model can call them when appropriate. *File: src/workflows.ts* ```typescript import type * as activities from './activities'; import { generateText, stepCountIs, tool } from 'ai'; import { temporalProvider } from '@temporalio/ai-sdk/workflow'; import { proxyActivities } from '@temporalio/workflow'; import z from 'zod'; const { getWeather, calculateAreaOfCircle } = proxyActivities({ startToCloseTimeout: '1 minute', retry: { maximumAttempts: 3, }, }); export async function toolsAgent(question: string): Promise { const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt: question, system: 'You are a helpful agent.', tools: { getWeather: tool({ description: 'Get the weather for a given city', inputSchema: z.object({ location: z.string().describe('The location to get the weather for'), }), execute: getWeather, }), calculateCircleArea: tool({ description: 'Calculate the area of a circle', inputSchema: z.object({ radius: z.number().describe('The radius of the circle'), }), execute: calculateAreaOfCircle, }), }, stopWhen: stepCountIs(5), }); return result.text; } ``` ## Create the Worker Create the process for executing Activities and Workflows. The Worker uses `AiSdkPlugin` to configure the OpenAI provider and to keep Workflow code isolated from the Activity environment. *File: src/worker.ts* ```typescript import { NativeConnection, Worker } from '@temporalio/worker'; import * as activities from './activities'; import { AiSdkPlugin } from '@temporalio/ai-sdk'; import { openai } from '@ai-sdk/openai'; async function run() { const connection = await NativeConnection.connect({ address: 'localhost:7233' }); const worker = await Worker.create({ plugins: [new AiSdkPlugin({ modelProvider: openai })], connection, namespace: 'default', taskQueue: 'ai-sdk', workflowsPath: require.resolve('./workflows'), activities, }); await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); ``` ## Create the Workflow Starter The starter (`src/client.ts`) takes the question to ask as a command-line argument, spins up a Temporal client, and starts `toolsAgent` with a new Workflow Id. *File: src/client.ts* ```typescript import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { toolsAgent } from './workflows'; import { nanoid } from 'nanoid'; async function run() { const question = process.argv.slice(2).join(' ') || 'What is the weather in Seattle right now?'; const config = loadClientConnectConfig(); const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection }); const handle = await client.workflow.start(toolsAgent, { taskQueue: 'ai-sdk', args: [question], workflowId: 'workflow-' + nanoid(), }); console.log(`Started workflow ${handle.workflowId}`); console.log(await handle.result()); } run().catch((err) => { console.error(err); process.exit(1); }); ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Install all dependencies: ```bash npm install ``` Set your API Key based on your preferred model provider: ```bash export OPENAI_API_KEY= ``` Run the worker: ```bash npm run start.watch ``` Start execution with the default question, or supply your own: ```bash npm run workflow npm run workflow "What is the weather in Chicago?" npm run workflow "Calculate the area of a circle with radius 5" ``` ## Example interactions Try asking the agent questions like: - "What is the weather in Seattle right now?" - "Calculate the area of a circle with radius 5" - "What is the weather in Chicago and calculate the area of a circle with radius 3" The agent decides which tools to use. Open the [Temporal UI](http://localhost:8233) to see the `invokeModel`, `getWeather`, and `calculateAreaOfCircle` Activities recorded in the Event History. --- # Deep research Source: https://docs.temporal.io/ai/cookbook/basic-openai-python > Build a multi-agent deep research system in Python with Temporal and the OpenAI Responses API. Deep research systems combine multiple agents with information retrieval from the web or other sources to produce evidence-based reports on specific topics. Commercial implementations include [Anthropic Research](https://www.anthropic.com/engineering/multi-agent-research-system), [OpenAI Deep Research](https://openai.com/index/introducing-deep-research/), and [Google Gemini Deep Research](https://gemini.google/overview/deep-research/). This recipe demonstrates a simple deep research system embodying the standard deep research architecture. Deep research spans the following four phases: - **Planning**. Task decomposition and research strategy formulation. This involves identifying separate aspects of the research problem that can be worked on independently. - **Question Development/Query Generation**. Designing queries for each of the research questions. - **Web Exploration/Information Retrieval**. Searching the web to retrieve documents relevant to the research question. Extracting and summarizing relevant information. - **Report Generation/Synthesis**. Synthesizing findings into comprehensive, well-cited reports. Deep research tasks can involve dozens of searches and process hundreds of documents. This creates many possible failure modes that durable execution helps protect against. This recipe uses OpenAI's Responses API, which includes a tool for web search. It also uses OpenAI's [Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs), which asks the model to generate outputs corresponding to desired data structures. ## Create the data structures We will use Python classes to ensure information passes between agents in a structured way. The Planning Agent creates a `ResearchPlan`, which includes a research question, a list of `ResearchAspects`, expected sources, a search strategy, and success criteria. ResearchAspects include an aspect name, a priority, and a description. ```python class ResearchPlan(BaseModel): research_question: str key_aspects: List[ResearchAspect] expected_sources: List[str] search_strategy: str success_criteria: List[str] ``` ```python class ResearchAspect(BaseModel): aspect: str priority: int description: str ``` The Query Generation Agent creates a `QueryPlan`, and generates a list of `SearchQueries`. ```python class QueryPlan(BaseModel): queries: List[SearchQuery] ``` ```python class SearchQuery(BaseModel): query: str rationale: str expected_info_type: str priority: int ``` The Web Search Agent creates a `SearchResult`, which includes a query, a list of sources, a key finding, a relevance score, and a list of citations. ```python class SearchResult(BaseModel): query: str sources: List[str] key_findings: str relevance_score: float citations: List[str] ``` Finally, the Report Synthesis Agent creates a `ResearchReport`, which includes an executive summary, a detailed analysis, a list of key findings, a confidence assessment, a list of citations, and a list of follow-up questions. ```python class ResearchReport(BaseModel): executive_summary: str detailed_analysis: str key_findings: List[str] confidence_assessment: str citations: List[str] follow_up_questions: List[str] ``` ## Create the agents The deep research system uses four specialized agents, each implemented as Temporal Activities. In this implementation, each agent is implemented as a single call to the OpenAI Responses API. This is possible because we are using structured outputs, which guarantee the response will be in the correct format, eliminating the need for retries. The web search agent also requires only a single API call because OpenAI integrates the web search tool into the Responses API. These agents run in the Workflow and use the `invoke_model` Activity to make OpenAI API calls. It is critical to set the `start_to_close_timeout` for these Activities to a value that is long enough to complete the task. If it is too short, the Activity will fail with a timeout error, causing a retry loop that never completes. Response times for reasoning models such as `GPT-5` can vary significantly depending on the nature of the request. Web search times also vary depending on the size and content of the documents located by the search. Each agent imports the `invoke_model` Activity inside `workflow.unsafe.imports_passed_through()`. That Activity module pulls in the OpenAI client and, through it, `httpx`, which touches modules that the Workflow sandbox restricts at import time. Only the Activity import is passed through, so the agent code itself stays sandboxed and keeps its determinism checks. Each agent also appends the current date to its instructions with `with_today()`. The date is read from the Workflow clock through `workflow.now()`, which requires a running Workflow. Building the instruction string at module import time would instead freeze the date when the Worker starts. ### Research Planning Agent Analyzes research queries and creates comprehensive research strategies. Takes an unstructured question and decomposes it into specific research aspects with priorities, identifies expected source types, and defines success criteria. *File: agents/research_planning.py* ```python from datetime import timedelta from temporalio import workflow from .config import COMPLEX_REASONING_MODEL from .shared import ResearchPlan, with_today with workflow.unsafe.imports_passed_through(): from activities.invoke_model import InvokeModelRequest, invoke_model RESEARCH_PLANNING_INSTRUCTIONS = """ You are a research planning specialist who creates focused research strategies. CORE RESPONSIBILITIES: 1. Decompose the user's question into 3-7 key research aspects 2. Identify required sources and evidence types 3. Design a practical search strategy 4. Set clear success criteria OUTPUT REQUIREMENTS: - research_question: Clarified version of the original query - key_aspects: Specific areas requiring investigation, each with: - aspect: The research area name - priority: 1-5 ranking (5 highest priority) - description: What needs to be investigated - expected_sources: Types of sources likely to contain relevant information - search_strategy: High-level approach for information gathering - success_criteria: Specific indicators of research completeness """ async def plan_research(query: str) -> ResearchPlan: result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=COMPLEX_REASONING_MODEL, instructions=with_today(RESEARCH_PLANNING_INSTRUCTIONS), input=f"Research query: {query}", response_format=ResearchPlan, ), start_to_close_timeout=timedelta(seconds=300), summary="Planning research", ) return result.response ``` ### Query Generation Agent Converts research plans into optimized web search queries. Creates 3-5 diverse queries that target different information types (factual data, expert analysis, case studies, recent news) with varied search styles and temporal modifiers. *File: agents/research_query_generation.py* ```python from datetime import timedelta from temporalio import workflow from .config import EFFICIENT_PROCESSING_MODEL from .shared import QueryPlan, ResearchPlan, with_today with workflow.unsafe.imports_passed_through(): from activities.invoke_model import InvokeModelRequest, invoke_model QUERY_GENERATION_INSTRUCTIONS = """ You are a search query specialist who crafts effective web searches. CORE RESPONSIBILITIES: 1. Generate 3-5 diverse search queries based on the research plan 2. Balance specificity with discoverability 3. Target different information types (factual, analytical, recent, historical) APPROACH: - Vary query styles: direct questions, topic + keywords, source-specific searches - Include temporal modifiers when relevant (recent, 2024, historical) - Use domain-specific terminology appropriately OUTPUT REQUIREMENTS: - queries: Search queries, each with: - query: The actual search string - rationale: Why this query addresses research needs - expected_info_type: One of "factual_data", "expert_analysis", "case_studies", "recent_news" - priority: 1-5 (5 highest priority) """ async def generate_queries(research_plan: ResearchPlan) -> QueryPlan: # Prepare input with research plan context plan_context = f""" Research Question: {research_plan.research_question} Key Aspects to Research: {chr(10).join([f"- {aspect.aspect} (Priority: {aspect.priority}): {aspect.description}" for aspect in research_plan.key_aspects])} Expected Sources: {", ".join(research_plan.expected_sources)} Search Strategy: {research_plan.search_strategy} Success Criteria: {", ".join(research_plan.success_criteria)} """ result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=EFFICIENT_PROCESSING_MODEL, instructions=with_today(QUERY_GENERATION_INSTRUCTIONS), input=plan_context, response_format=QueryPlan, ), start_to_close_timeout=timedelta(seconds=300), summary="Generating search queries", ) return result.response ``` ### Web Search Agent Executes searches using OpenAI's web search tool and analyzes results. Prioritizes authoritative sources, extracts key findings, assesses relevance, and provides proper citations with reliability assessments. *File: agents/research_web_search.py* ```python from datetime import timedelta from temporalio import workflow from .config import EFFICIENT_PROCESSING_MODEL from .shared import SearchQuery, SearchResult, with_today with workflow.unsafe.imports_passed_through(): from activities.invoke_model import InvokeModelRequest, invoke_model WEB_SEARCH_INSTRUCTIONS = """ You are a web research specialist who finds and evaluates information from web sources. CORE RESPONSIBILITIES: 1. Execute web searches using the web search tool 2. Prioritize authoritative sources: academic, government, established research organizations, prominent news outlets, primary sources 3. Extract key information relevant to the research question 4. Provide proper citations and assess reliability APPROACH: - Focus on information directly relevant to the research question - Extract specific facts, data points, and evidence - Note conflicting information and limitations - Flag questionable or unverified claims OUTPUT REQUIREMENTS: - query: The search query that was executed - sources: URLs and source descriptions consulted - key_findings: Synthesized information relevant to research question (2-4 paragraphs) - relevance_score: 0.0-1.0 assessment of how well results address the query - citations: Formatted sources with URLs """ async def search_web(query: SearchQuery) -> SearchResult: search_input = f""" Search Query: {query.query} Query Rationale: {query.rationale} Expected Information Type: {query.expected_info_type} Priority Level: {query.priority} Please search for information using the provided query and analyze the results according to the instructions. """ result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=EFFICIENT_PROCESSING_MODEL, instructions=with_today(WEB_SEARCH_INSTRUCTIONS), input=search_input, response_format=SearchResult, tools=[{"type": "web_search"}], ), start_to_close_timeout=timedelta(seconds=300), summary="Searching web for information", ) return result.response ``` ### Report Synthesis Agent Directs the agent to synthesize all research findings into comprehensive, well-cited reports. These should include structured narratives with executive summaries, detailed analysis, key findings, confidence assessments, and follow-up research questions. *File: agents/research_report_synthesis.py* ```python from datetime import timedelta from typing import List from temporalio import workflow from .config import COMPLEX_REASONING_MODEL from .shared import ResearchPlan, ResearchReport, SearchResult, with_today with workflow.unsafe.imports_passed_through(): from activities.invoke_model import InvokeModelRequest, invoke_model REPORT_SYNTHESIS_INSTRUCTIONS = """ You are a research synthesis expert who creates comprehensive research reports. CORE RESPONSIBILITIES: 1. Synthesize all research into a coherent narrative 2. Structure information logically with evidence support 3. Provide comprehensive citations 4. Assess confidence levels and acknowledge limitations 5. Generate follow-up questions for deeper research REPORT STRUCTURE: 1. **Executive Summary**: Core findings and conclusions (1-2 paragraphs) 2. **Detailed Analysis**: Examination organized by themes with evidence 3. **Key Findings**: Bullet-point list of important discoveries 4. **Confidence Assessment**: Rate findings as High/Medium/Low/Uncertain 5. **Citations**: Complete source list with URLs 6. **Follow-up Questions**: Up to 5 areas for additional research, as warranted APPROACH: - Address contradictory findings transparently - Weight authoritative sources more heavily - Distinguish facts from expert opinions - Be explicit about information limitations OUTPUT REQUIREMENTS: - executive_summary: 1-2 paragraph summary of core findings - detailed_analysis: Multi-paragraph analysis organized by themes - key_findings: Bullet-point discoveries - confidence_assessment: Assessment of finding reliability - citations: All sources referenced - follow_up_questions: 3-5 specific questions for further research """ async def generate_synthesis( original_query: str, research_plan: ResearchPlan, search_results: List[SearchResult] ) -> ResearchReport: # Prepare comprehensive input with all research context synthesis_input = f""" ORIGINAL RESEARCH QUERY: {original_query} RESEARCH PLAN: Research Question: {research_plan.research_question} Key Aspects Investigated: { ", ".join([aspect.aspect for aspect in research_plan.key_aspects]) } Search Strategy Used: {research_plan.search_strategy} Success Criteria: {", ".join(research_plan.success_criteria)} SEARCH RESULTS TO SYNTHESIZE: { chr(10).join( [ f"Query: {result.query}{chr(10)}Findings: {result.key_findings}{chr(10)}Relevance: {result.relevance_score}{chr(10)}Sources: {', '.join(result.sources)}{chr(10)}Citations: {', '.join(result.citations)}{chr(10)}" for result in search_results ] ) } Please synthesize all this information into a comprehensive research report following the specified structure and quality standards. """ result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=COMPLEX_REASONING_MODEL, instructions=with_today(REPORT_SYNTHESIS_INSTRUCTIONS), input=synthesis_input, response_format=ResearchReport, ), start_to_close_timeout=timedelta(seconds=300), summary="Generating research report synthesis", ) return result.response ``` ## Create the Workflow The `DeepResearchWorkflow` orchestrates the four-phase research process with built-in resilience and error handling: First, planning and query generation agents are run sequentially. Then, the workflow executes searches concurrently. For robustness, the workflow continues with partial results if some searches fail. Finally, the report synthesis agent pulls together the findings into a comprehensive report. *File: workflows/deep_research_workflow.py* ```python import asyncio from typing import List from temporalio import workflow from temporalio.exceptions import ApplicationError from agents.research_planning import plan_research from agents.research_query_generation import generate_queries from agents.research_report_synthesis import generate_synthesis from agents.research_web_search import search_web from agents.shared import SearchResult @workflow.defn class DeepResearchWorkflow: @workflow.run async def run(self, query: str) -> str: # Step 1: Research Planning research_plan = await plan_research(query) # Step 2: Query Generation query_plan = await generate_queries(research_plan) # Step 3: Web Search (parallel execution with resilience) search_results = await self._execute_searches(query_plan.queries) # Ensure we have at least one successful search result if not search_results: raise ApplicationError( "All web searches failed - cannot generate report", "NO_SEARCH_RESULTS", non_retryable=True, ) # Step 4: Report Synthesis final_report = await generate_synthesis(query, research_plan, search_results) # Format the final output formatted_report = self._format_final_report(query, final_report) return formatted_report async def _execute_searches(self, search_queries) -> List[SearchResult]: """Execute web searches in parallel with resilience to individual failures""" # Create individual search coroutines async def execute_single_search(search_query): try: return await search_web(search_query) except Exception as e: workflow.logger.exception( f"Search failed for query '{search_query.query}': {e}" ) return None # Execute all searches in parallel search_tasks = [execute_single_search(query) for query in search_queries] results = await asyncio.gather(*search_tasks) # Filter out None results return [result for result in results if result is not None] def _format_final_report(self, original_query, report) -> str: """Format the final report for display""" return f""" # Deep Research Report **Research Query:** {original_query} ## Executive Summary {report.executive_summary} ## Detailed Analysis {report.detailed_analysis} ## Key Findings {chr(10).join([f"• {finding}" for finding in report.key_findings])} ## Confidence Assessment {report.confidence_assessment} ## Sources and Citations {chr(10).join([f"• {citation}" for citation in report.citations])} ## Recommended Follow-up Questions {chr(10).join([f"• {question}" for question in report.follow_up_questions])} """ ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Run the worker: ```bash uv run worker.py ``` Start execution: ```bash uv run start_workflow.py ``` To start execution with a specific query: ```bash uv run start_workflow.py "What is the latest news on the stock market?" ``` --- # Claim check pattern with Temporal Source: https://docs.temporal.io/ai/cookbook/claim-check-pattern-python > Use the Claim Check pattern with Temporal to keep large payloads out of Event History by offloading them to S3. This recipe demonstrates how to use the Claim Check pattern to offload data from Temporal Server's Event History to external storage. This can be useful in conversational AI applications that include the full conversation history with each LLM call, creating large Event History that can exceed server size limits. This recipe includes: - A `PayloadCodec` ([docs](/payload-codec)) that stores large payloads in S3 and replaces them with keys - A client [plugin](/develop/plugins-guide) that wires the codec into the Temporal data converter - A lightweight codec server for a better Web UI experience - An AI/RAG example workflow that demonstrates the pattern end-to-end ## Temporal's built-in external storage feature Since Python SDK 1.25, Temporal offers a built-in [external storage](/develop/python/data-handling/external-storage) feature that implements the same claim check pattern without a custom `PayloadCodec`. It's configured directly on the `DataConverter` via an `ExternalStorage` option, and Temporal provides an `S3StorageDriver` out of the box. The feature is in Public Preview, so its API may change before General Availability. Use the built-in feature first if S3 (or a self-hosted equivalent) is a good fit and you don't need custom encode/decode logic. Follow the codec-based approach in this recipe when you need a storage backend other than what the built-in drivers support, want to combine claim check with other codec logic (such as encryption) in a single codec, or want full control over the encode/decode implementation. ## How the Claim Check pattern works Each Temporal Workflow has an associated Event History that is stored in Temporal Server and used to provide durable execution. When using the Claim Check pattern, we store the payload content of the Event in a separate storage system, then store a reference to that storage in the Temporal Event History instead. The Claim Check Recipe implements a `PayloadCodec` that: 1. Encode: Replaces large payloads with unique keys and stores the original data in external storage (S3, Database, etc.) 2. Decode: Retrieves the original payload using the key when needed Workflows operate with small, lightweight keys while maintaining transparent access to full data through automatic encoding/decoding. ## Claim Check codec implementation The `ClaimCheckCodec` implements `PayloadCodec` and adds an inline threshold to keep small payloads inline. This avoids the latency costs of uploading/downloading the payload externally when it's not required. *File: codec/claim_check.py* ```python import logging import uuid from typing import Iterable, List import aioboto3 from botocore.exceptions import ClientError from temporalio.api.common.v1 import Payload from temporalio.converter import PayloadCodec logger = logging.getLogger(__name__) class ClaimCheckCodec(PayloadCodec): """PayloadCodec that implements the Claim Check pattern using S3 storage. This codec stores large payloads in S3 and replaces them with unique keys, allowing Temporal workflows to operate with lightweight references instead of large payload data. """ def __init__( self, bucket_name: str = "temporal-claim-check", endpoint_url: str = None, region_name: str = "us-east-1", max_inline_bytes: int = 20 * 1024, ): """Initialize the claim check codec with S3 connection details. Args: bucket_name: S3 bucket name for storing claim check data endpoint_url: S3 endpoint URL (for MinIO or other S3-compatible services) region_name: AWS region name max_inline_bytes: Payloads up to this size will be left inline """ self.bucket_name = bucket_name self.endpoint_url = endpoint_url self.region_name = region_name self.max_inline_bytes = max_inline_bytes self.session = aioboto3.Session() self._bucket_created = False async def _ensure_bucket_exists(self): """Ensure the S3 bucket exists, creating it if necessary.""" if self._bucket_created: return async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: try: await s3_client.head_bucket(Bucket=self.bucket_name) except ClientError as e: error_code = e.response['Error']['Code'] if error_code in ['404', 'NoSuchBucket']: try: await s3_client.create_bucket(Bucket=self.bucket_name) except ClientError as create_error: # Handle bucket already exists race condition if create_error.response['Error']['Code'] not in ['BucketAlreadyExists', 'BucketAlreadyOwnedByYou']: raise create_error elif error_code not in ['403', 'Forbidden']: raise e self._bucket_created = True async def encode(self, payloads: Iterable[Payload]) -> List[Payload]: """Replace large payloads with keys and store original data in S3. Args: payloads: Iterable of payloads to encode Returns: List of encoded payloads (keys for claim-checked payloads) """ await self._ensure_bucket_exists() out: List[Payload] = [] for payload in payloads: # Leave small payloads inline to improve debuggability and avoid unnecessary indirection data_size = len(payload.data or b"") if data_size <= self.max_inline_bytes: out.append(payload) continue encoded = await self.encode_payload(payload) out.append(encoded) return out async def decode(self, payloads: Iterable[Payload]) -> List[Payload]: """Retrieve original payloads from S3 using stored keys. Args: payloads: Iterable of payloads to decode Returns: List of decoded payloads (original data retrieved from S3) Raises: ValueError: If a claim check key is not found in S3 """ await self._ensure_bucket_exists() out: List[Payload] = [] for payload in payloads: if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1": # Not a claim-checked payload, pass through unchanged out.append(payload) continue s3_key = payload.data.decode("utf-8") stored_data = await self.get_payload_from_s3(s3_key) if stored_data is None: raise ValueError(f"Claim check key not found in S3: {s3_key}") original_payload = Payload.FromString(stored_data) out.append(original_payload) return out async def encode_payload(self, payload: Payload) -> Payload: """Store payload in S3 and return a key-based payload. Args: payload: Original payload to store Returns: Payload containing only the S3 key """ await self._ensure_bucket_exists() key = str(uuid.uuid4()) serialized_data = payload.SerializeToString() # Store the original payload data in S3 async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: await s3_client.put_object( Bucket=self.bucket_name, Key=key, Body=serialized_data ) # Return a lightweight payload containing only the key return Payload( metadata={ "encoding": b"claim-checked", "temporal.io/claim-check-codec": b"v1", }, data=key.encode("utf-8"), ) async def get_payload_from_s3(self, s3_key: str) -> bytes: """Retrieve payload data from S3. Args: s3_key: S3 object key Returns: Raw payload data bytes, or None if not found """ try: async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: response = await s3_client.get_object( Bucket=self.bucket_name, Key=s3_key ) return await response['Body'].read() except ClientError as e: if e.response['Error']['Code'] == 'NoSuchKey': return None raise e ``` ### Inline payload threshold - Default: 20KB - Where configured: `ClaimCheckCodec(max_inline_bytes=20 * 1024)` in `codec/claim_check.py` - Change by passing a different `max_inline_bytes` when constructing `ClaimCheckCodec` ### Choosing the right threshold The `max_inline_bytes` threshold controls which payloads are offloaded to S3 and which stay inline in Event History. Here is how to choose the right value for your use case. #### Temporal Service size limits Temporal enforces size limits at several levels ([self-hosted defaults](/self-hosted-guide/defaults), [Temporal Cloud limits](/cloud/limits)): | Limit | Warning | Error / Termination | |-------|---------|---------------------| | Single payload (blob) size | 256 KB | 2 MB | | Event History total size | 10 MB | 50 MB (Workflow terminated) | | Event History event count | 10,240 events | 51,200 events (Workflow terminated) | | gRPC message size | — | 4 MB per message | | Event History transaction size | — | 4 MB per transaction | The single payload limit applies to each serialized Activity input, Activity output, or Workflow argument individually. The gRPC limit applies to the full request, so scheduling several Activities with moderate-sized inputs in the same Workflow Task can exceed 4 MB even when each payload is under 2 MB. These are the defaults for both self-hosted and Temporal Cloud. The blob size thresholds are configurable on self-hosted deployments. Temporal Cloud limits are not configurable. #### Sizing recommendations | Threshold | Good for | Trade-off | |-----------|----------|-----------| | **2 KB** | Chatty Workflows with many small Activities | More S3 round-trips, higher latency per call | | **20 KB** (default) | Most AI/RAG Workflows | Balances debuggability with Event History size | | **128 KB** | Low-Activity Workflows with moderate payloads | Fewer S3 calls, but Event History grows faster | | **256 KB+** | Workflows with few, large payloads | Right at the blob warning threshold — payloads near this size risk triggering warnings, and Event History grows fast | #### How to decide 1. **Estimate your payload sizes.** LLM conversation histories grow with each turn. A 10-turn conversation with tool calls can reach 50–100 KB. RAG chunks with embeddings can be several megabytes. 2. **Count your Activities.** Each Activity input and output is a separate payload in Event History. A Workflow with 20 Activity calls at 100 KB each adds 4 MB to Event History from payloads alone. 3. **Start with the default (20 KB).** This offloads anything that would meaningfully impact Event History size while keeping small, debuggable payloads visible in the Web UI. 4. **Stay well under 256 KB.** Payloads above 256 KB trigger a warning from the Temporal Service. If your payloads regularly exceed this size, Claim Check is strongly recommended. 5. **Lower the threshold** if your Workflows are long-running (many Activity calls over time) or if you run many concurrent Workflows on the same Temporal Service. 6. **Raise the threshold** if S3 latency is a concern and your Workflows have few Activities with moderate payloads. #### Monitoring Event History size Use the Web UI or `temporal workflow describe` to check a Workflow's Event History size and event count. If you see the 10 MB / 10,240 event warning in Temporal Service logs, lower your `max_inline_bytes` threshold or review which payloads should use Claim Check instead of staying inline. ## Claim Check plugin The `ClaimCheckPlugin` integrates the codec with the Temporal client configuration. *File: codec/plugin.py* ```python import os from temporalio.converter import DataConverter from temporalio.plugin import SimplePlugin from .claim_check import ClaimCheckCodec class ClaimCheckPlugin(SimplePlugin): """Temporal plugin that integrates the Claim Check codec with client configuration.""" def __init__(self): """Initialize the plugin with S3 connection configuration.""" super().__init__( name="claim-check", data_converter=DataConverter( payload_codec=ClaimCheckCodec( bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"), endpoint_url=os.getenv("S3_ENDPOINT_URL"), region_name=os.getenv("AWS_REGION", "us-east-1"), ), ), ) ``` ## Example: AI/RAG Workflow using Claim Check This example ingests a large text, performs lightweight lexical retrieval, and answers a question with an LLM. Large intermediates (chunks, scores) are kept out of Temporal payloads via the Claim Check codec. Only the small final answer is returned inline. ### Shared models *File: shared/models.py* ```python from dataclasses import dataclass from typing import Any, Dict, List @dataclass class IngestRequest: document_bytes: bytes filename: str mime_type: str chunk_size: int = 1500 chunk_overlap: int = 200 embedding_model: str = "text-embedding-3-large" @dataclass class IngestResult: chunk_texts: List[str] metadata: Dict[str, Any] @dataclass class RagRequest: question: str top_k: int = 4 generation_model: str = "gpt-4o-mini" @dataclass class RagAnswer: answer: str sources: List[Dict[str, Any]] ``` ### Activities *File: activities/ai_claim_check.py* ```python from typing import List from temporalio import activity from shared.models import IngestRequest, IngestResult, RagAnswer, RagRequest def _split_text(text: str, chunk_size: int, overlap: int) -> List[str]: chunks: List[str] = [] start = 0 n = len(text) while start < n: end = min(n, start + chunk_size) chunks.append(text[start:end]) if end >= n: break start = max(end - overlap, start + 1) return chunks @activity.defn async def ingest_document(req: IngestRequest) -> IngestResult: # Convert bytes to text. For PDFs/audio/images, integrate proper extractors. if req.mime_type != "text/plain": raise ValueError(f"Unsupported MIME type: {req.mime_type}") text = req.document_bytes.decode("utf-8", errors="ignore") chunks = _split_text(text, req.chunk_size, req.chunk_overlap) return IngestResult( chunk_texts=chunks, metadata={ "filename": req.filename, "mime_type": req.mime_type, "chunk_count": len(chunks), }, ) @activity.defn async def rag_answer(req: RagRequest, ingest_result: IngestResult) -> RagAnswer: # Import heavy dependencies inside the function, not at module level # This prevents NumPy from being loaded into the workflow sandbox from openai import AsyncOpenAI from rank_bm25 import BM25Okapi client = AsyncOpenAI(max_retries=0) # Lexical retrieval using BM25 over chunk texts # Simple whitespace tokenization tokenized_corpus: List[List[str]] = [chunk.split() for chunk in ingest_result.chunk_texts] bm25 = BM25Okapi(tokenized_corpus) tokenized_query = req.question.split() scores = bm25.get_scores(tokenized_query) # Get top-k indices by score top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[: max(1, req.top_k)] contexts = [ingest_result.chunk_texts[i] for i in top_indices] sources = [{"chunk_index": i, "score": float(scores[i])} for i in top_indices] prompt = ( "Use the provided context chunks to answer the question.\n\n" f"Question: {req.question}\n\n" "Context:\n" + "\n---\n".join(contexts) + "\n\nAnswer:" ) chat = await client.chat.completions.create( model=req.generation_model, messages=[{"role": "user", "content": prompt}], temperature=0.2, ) answer = chat.choices[0].message.content.strip() return RagAnswer(answer=answer, sources=sources) ``` ### Workflow *File: workflows/ai_rag_workflow.py* ```python from datetime import timedelta from temporalio import workflow from activities.ai_claim_check import ingest_document, rag_answer from shared.models import IngestRequest, IngestResult, RagAnswer, RagRequest @workflow.defn class AiRagWorkflow: @workflow.run async def run(self, document_bytes: bytes, filename: str, mime_type: str, question: str) -> RagAnswer: ingest: IngestResult = await workflow.execute_activity( ingest_document, IngestRequest( document_bytes=document_bytes, filename=filename, mime_type=mime_type, ), start_to_close_timeout=timedelta(minutes=10), summary="Ingest and embed large document", ) answer: RagAnswer = await workflow.execute_activity( rag_answer, args=[ RagRequest(question=question), ingest, ], start_to_close_timeout=timedelta(minutes=5), summary="RAG answer using embedded chunks", ) return answer ``` ## Running ### Prerequisites - MinIO server (for local testing) or AWS S3 access (for production) - Temporal dev server - Python 3.10+ ### Configuration Set environment variables to configure S3 and OpenAI: ```bash # For MinIO (recommended for local testing) export S3_ENDPOINT_URL=http://localhost:9000 export S3_BUCKET_NAME=temporal-claim-check export AWS_ACCESS_KEY_ID=minioadmin export AWS_SECRET_ACCESS_KEY=minioadmin export AWS_REGION=us-east-1 # For production AWS S3 # export S3_BUCKET_NAME=your-bucket-name # export AWS_REGION=us-east-1 # you may use access keys or... # export AWS_ACCESS_KEY_ID=your-access-key # export AWS_SECRET_ACCESS_KEY=your-secret-key # ... sso # export AWS_profile=your-profile # aws sso login --profile your-profile export OPENAI_API_KEY=your_key_here ``` ### Option 1: MinIO (Recommended for Testing) 1. Start MinIO: ```bash docker run -d -p 9000:9000 -p 9001:9001 \ --name minio \ -e "MINIO_ROOT_USER=minioadmin" \ -e "MINIO_ROOT_PASSWORD=minioadmin" \ quay.io/minio/minio server /data --console-address ":9001" ``` The bucket will be auto-created by the code. You can view stored objects in the MinIO web console at http://localhost:9001 (credentials: minioadmin/minioadmin). 2. Start Temporal dev server: ```bash temporal server start-dev ``` 3. Run the worker: ```bash uv run worker.py ``` 4. Start execution: ```bash uv run start_workflow.py ``` ### Option 2: AWS S3 (Production) 1. Create an S3 bucket in your AWS account 2. Configure AWS credentials (via AWS CLI, environment variables, IAM roles or sso) 3. Set the environment variables for your bucket 4. Follow steps 2-4 from Option 1 ### Toggle Claim Check (optional) To demonstrate payload size failures without claim check, you can disable it in your local wiring (e.g., omit the plugin/codec) and re-run. With claim check disabled, large payloads may exceed Temporal's default payload size limits and fail. ## Codec server for Web UI When claim check is enabled, the Web UI would otherwise show opaque keys. This codec server shows helpful text with a link to view the raw data on demand. *File: codec/codec_server.py* ```python import json import os from functools import partial from typing import Awaitable, Callable, Iterable, List from aiohttp import hdrs, web from claim_check import ClaimCheckCodec from google.protobuf import json_format from temporalio.api.common.v1 import Payload, Payloads def build_codec_server() -> web.Application: # Create codec with environment variable configuration (same as plugin) codec = ClaimCheckCodec( bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"), endpoint_url=os.getenv("S3_ENDPOINT_URL"), region_name=os.getenv("AWS_REGION", "us-east-1") ) # Configure Web UI endpoint temporal_web_url = os.getenv("TEMPORAL_WEB_URL", "http://localhost:8233") # Configure codec server endpoint for viewing raw data codec_server_url = os.getenv("CODEC_SERVER_URL", "http://localhost:8081") # CORS handler - needed because Temporal Web UI runs on a different port/domain # and the browser blocks cross-origin requests by default; CORS headers allow these requests async def cors_options(req: web.Request) -> web.Response: resp = web.Response() if req.headers.get(hdrs.ORIGIN) == temporal_web_url: resp.headers[hdrs.ACCESS_CONTROL_ALLOW_ORIGIN] = temporal_web_url resp.headers[hdrs.ACCESS_CONTROL_ALLOW_METHODS] = "POST" resp.headers[hdrs.ACCESS_CONTROL_ALLOW_HEADERS] = "content-type,x-namespace" return resp # Custom decode function that provides URLs to view raw data async def decode_with_urls(payloads: Iterable[Payload]) -> List[Payload]: """Decode claim check payloads and provide URLs to view the raw data.""" out: List[Payload] = [] for payload in payloads: if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1": # Not a claim-checked payload, pass through unchanged out.append(payload) continue # Get the S3 key s3_key = payload.data.decode("utf-8") # Return simple text with link - no data reading link_text = f"Claim check data (key: {s3_key}) - View at: {codec_server_url}/view/{s3_key}" summary_payload = Payload( metadata={"encoding": b"json/plain"}, data=json.dumps({"text": link_text}).encode("utf-8") ) out.append(summary_payload) return out # Endpoint to view raw payload data async def view_raw_data(req: web.Request) -> web.Response: """View the raw payload data for a given S3 key.""" s3_key = req.match_info['key'] try: stored_data = await codec.get_payload_from_s3(s3_key) if stored_data is None: return web.Response( text=json.dumps({"error": f"Key not found: {s3_key}"}), content_type="application/json", status=404 ) # Parse and return the original payload original_payload = Payload.FromString(stored_data) # Try to decode as text, fall back to base64 for binary data try: data_text = original_payload.data.decode("utf-8") return web.Response( text=data_text, content_type="text/plain" ) except UnicodeDecodeError: import base64 data_b64 = base64.b64encode(original_payload.data).decode("utf-8") return web.Response( text=f"Binary data (base64):\n{data_b64}", content_type="text/plain" ) except Exception as e: return web.Response( text=json.dumps({"error": f"Failed to retrieve data: {str(e)}"}), content_type="application/json", status=500 ) # General purpose payloads-to-payloads async def apply( fn: Callable[[Iterable[Payload]], Awaitable[List[Payload]]], req: web.Request ) -> web.Response: # Read payloads as JSON assert req.content_type == "application/json" data = await req.read() payloads = json_format.Parse(data.decode("utf-8"), Payloads()) # Apply payloads = Payloads(payloads=await fn(payloads.payloads)) # Apply CORS and return JSON resp = await cors_options(req) resp.content_type = "application/json" resp.text = json_format.MessageToJson(payloads) return resp # Build app app = web.Application() app.add_routes( [ web.post("/encode", partial(apply, codec.encode)), web.post("/decode", partial(apply, decode_with_urls)), web.get("/view/{key}", view_raw_data), web.options("/decode", cors_options), ] ) return app if __name__ == "__main__": web.run_app(build_codec_server(), host="127.0.0.1", port=8081) ``` ### Running the codec server ```bash uv run codec/codec_server.py ``` Then [configure the Web UI to use the codec server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli). ### What it shows Instead of raw keys: ``` abc123-def4-5678-9abc-def012345678 ``` You will see text like: ``` "Claim check data (key: abc123-def4-5678-9abc-def012345678) - View at: http://localhost:8081/view/abc123-def4-5678-9abc-def012345678" ``` ### Endpoints - `POST /encode`: Encodes payloads using the claim check codec - `POST /decode`: Returns helpful text with S3 key and view URL (no data reads) - `GET /view/{key}`: Serves raw payload data for inspection - `OPTIONS /decode`: Handles CORS preflight requests The server also includes CORS handling for the local Web UI. --- # Post-LLM guardrail with hard-rule overrides Source: https://docs.temporal.io/ai/cookbook/guardrails-hard-rules-python > >- This recipe shows how to combine an LLM classifier with a deterministic guardrail layer. The LLM provides nuanced judgment for ambiguous cases; hard rules act as a safety net for unambiguous policy violations, overriding the LLM's verdict regardless of what it concluded. The pattern answers a real problem: LLMs can be manipulated via prompt injection or hallucinate outright. For any decision with real consequences — content moderation, access control, transaction approval — you shouldn't rely on the LLM alone. Hard rules catch clear-cut cases deterministically; the LLM handles everything in the grey zone. Critically, when a hard rule fires, the LLM's original reasoning is preserved inside the override so every decision remains auditable. The recipe uses a content moderation scenario: user-submitted text is classified as `safe`, `review`, or `block`. Hard rules override to `block` when contact information or banned keywords are detected, regardless of what the LLM concluded. ## Prerequisites - Python 3.10+ - [uv](https://docs.astral.sh/uv/) - A running Temporal server: `temporal server start-dev` - `ANTHROPIC_API_KEY` environment variable set ## Run it ```bash uv sync # Terminal 1 — start the worker uv run python -m worker # Terminal 2 — submit two example workflows uv run python -m start_workflow ``` ## Expected output ``` --- Example 1: Hard rule override --- Input: 'Great product! Contact me at john.doe@example.com for a special deal.' Classification: block Overridden by hard rule: True Reasoning: Hard rule: contains email address (privacy policy violation). [LLM classified as 'safe' — reasoning: The message is promotional but does not appear harmful.] --- Example 2: LLM verdict stands --- Input: 'I really enjoyed the hiking trail last weekend. The views were amazing!' Classification: safe Overridden by hard rule: False Reasoning: Positive personal experience with no policy concerns. ``` In Example 1, the LLM's classification and reasoning are preserved inside brackets — the override is fully auditable. ## Architecture - **Models** (`models/`): - `signals.py`: `ContentSignals` — the text and metadata being classified - `verdict.py`: `LLMVerdict` (the LLM's raw classification, also used as the tool's input schema) and `Verdict` (adds `overridden_by_hard_rule`) - **Guardrails** (`guardrails/hard_rules.py`): pure functions that check content against banned keywords, phone numbers, and email addresses, and escalate the verdict to `block` when one matches - **Activity** (`activities/classify.py`): calls Claude via a forced tool call to get a structured `LLMVerdict`, then applies the hard rules - **Workflow** (`workflows/classify_workflow.py`): orchestrates the single `classify` Activity call with a 3-attempt retry policy - **Scripts**: - `worker.py`: runs the Temporal Worker - `start_workflow.py`: runs the two examples shown in [Expected output](#expected-output) ## Key patterns ### Forcing a structured verdict from the LLM The Activity forces Claude to call a single tool, so the response is always a well-formed `LLMVerdict` instead of free-form text to parse: ```python _SUBMIT_VERDICT_TOOL = { "name": "submit_verdict", "description": "Submit your content moderation classification.", "input_schema": LLMVerdict.model_json_schema(), } ``` ```python response = await client.messages.create( ... tools=[_SUBMIT_VERDICT_TOOL], tool_choice={"type": "tool", "name": "submit_verdict"}, ) ``` ### Overriding while preserving the original reasoning `apply_hard_rules` never discards the LLM's own verdict — a rule can only escalate a verdict to `block`, and when it does, the LLM's reasoning is embedded in the result so the override stays auditable: ```python def apply_hard_rules(signals: ContentSignals, llm_verdict: Verdict) -> Verdict: """Post-filter: override the LLM verdict if a hard rule matches. When a rule fires, the LLM's original reasoning is embedded in the returned verdict so the override is auditable. """ if llm_verdict.classification == "block": return llm_verdict hard = _hard_block(signals) if hard is None: return llm_verdict return Verdict( classification=hard.classification, confidence=hard.confidence, overridden_by_hard_rule=True, reasoning=( f"{hard.reasoning}\n\n" f"[LLM classified as '{llm_verdict.classification}' — " f"reasoning: {llm_verdict.reasoning}]" ), ) ``` ### Non-retryable Anthropic errors Permanent client errors (bad request, authentication, permission-denied, not-found, unprocessable-entity) are classified as non-retryable so the Workflow doesn't keep retrying a request that can never succeed: ```python except ( anthropic.BadRequestError, anthropic.AuthenticationError, anthropic.PermissionDeniedError, anthropic.NotFoundError, anthropic.UnprocessableEntityError, ) as exc: # These errors will never succeed on a retry: a retired model identifier, for # example, returns 404. Retrying them under the default policy would loop # forever instead of surfacing the problem. Everything else, such as rate # limits and 5xx responses, propagates so Temporal can retry it. raise ApplicationError( str(exc), type=exc.__class__.__name__, non_retryable=True, ) from exc ``` ## Extensions This pattern can be extended to: - Add more hard rules (regex, allow/deny lists, PII detectors) without touching the LLM prompt - Log every override to a compliance or audit sink for review - Route `review` verdicts to a human approval step — see [Human-in-the-loop AI agent](./human-in-the-loop-python.mdx) --- # Durable MCP weather server Source: https://docs.temporal.io/ai/cookbook/hello-world-durable-mcp-server > Build a durable MCP server in Python that runs weather tools reliably with Temporal Workflows. This example demonstrates how to build a durable MCP (Model Context Protocol) server using Temporal Workflows for Durable Execution. The server exposes weather tools that fetch alerts and forecasts from the National Weather Service API. MCP tools are "actions" that the MCP server can perform. Within a given MCP tool, there are often multiple steps (API calls, functions, etc.) that must happen in a certain order to complete an action. For example, the `get_forecast` tool performs the following steps: - Call the National Weather Service API to find which region corresponds to the given latitude and longitude coordinates - Call the National Weather Service API again to retrieve the forecast for that region - Format and return the response to the user In this one tool alone, we are taking several steps to complete a given action. We implement these steps in a Temporal Workflow, which provides durability. This means that whenever your MCP tool is called, it kicks off the Temporal Workflow, and every step (API call, function) is executed reliably and all the way to completion. We use [FastMCP](https://github.com/jlowin/fastmcp) to implement the MCP Server and create tools using the decorator `@mcp.tool`. > **📝 Note:** > External API calls are made within Temporal Activities. This ensures that network requests are retried appropriately and failures are handled. This recipe highlights the following key design decisions: - **Separation of concerns**: MCP tools act as thin wrappers that start Temporal Workflows. All business logic lives in Workflows, ensuring durability and reliability. - **Durable Execution**: By moving multi-step operations into Temporal Workflows, we guarantee that operations complete even in the face of failures, network issues, or process restarts. - **Activity-based external calls**: All external API calls (like NWS API requests) are made within Temporal Activities, which provides automatic retries and proper error handling. - **Retry policies**: Workflows use configurable retry policies to handle transient failures. Also see this foundational [recipe for basic tool calling](/ai/cookbook/tool-call-openai-python) using the same weather tools. ## Application components This example includes the following components: - The [MCP server](#create-the-mcp-server) (mcp_server.py) that exposes tools via FastMCP and starts Temporal Workflows - The [Workflows](#create-the-workflows) (weather_workflows.py) that orchestrate the multi-step weather operations - The [Activity](#create-the-activity) (weather_activities.py) for making external API calls to the National Weather Service - The [Worker](#create-the-worker) (worker.py) (that manages the Workflows and Activities) - [Config for Claude Desktop](#configure-claude-desktop) (claude_desktop_config.json) for connecting the MCP server to Claude Desktop ## Create the MCP server The MCP server is implemented using FastMCP and exposes tools via the `@mcp.tool` decorator. Each tool is a thin wrapper that starts a Temporal Workflow and waits for the result. This design ensures that all business logic lives in durable Workflows. *File: mcp_servers/weather.py* ```python from fastmcp import FastMCP from temporalio.client import Client from temporalio.envconfig import ClientConfig # Initialize FastMCP server mcp = FastMCP("weather") # Temporal client setup (do this once, then reuse) temporal_client = None async def get_temporal_client(): global temporal_client if not temporal_client: config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") temporal_client = await Client.connect(**config) return temporal_client @mcp.tool async def get_alerts(state: str) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow client = await get_temporal_client() handle = await client.start_workflow( "GetAlerts", state, id=f"alerts-{state.lower()}", task_queue="weather-task-queue", ) return await handle.result() @mcp.tool async def get_forecast(latitude: float, longitude: float) -> str: """Get weather forecast for a location. Args: latitude: Latitude of the location longitude: Longitude of the location """ # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow client = await get_temporal_client() handle = await client.start_workflow( workflow="GetForecast", args=[latitude, longitude], id=f"forecast-{latitude}-{longitude}", task_queue="weather-task-queue", ) return await handle.result() if __name__ == "__main__": # Initialize and run the server mcp.run(transport="stdio") ``` ## Create the Workflows The Workflows contain the business logic for fetching weather data. They orchestrate multiple steps, including API calls and data formatting. By implementing this logic in Workflows, we ensure that operations complete reliably even if there are failures or interruptions. ### GetAlerts Workflow The `GetAlerts` workflow fetches active weather alerts for a US state. *File: workflows/weather_workflows.py* ```python from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy retry_policy = RetryPolicy( maximum_attempts=0, # Infinite retries initial_interval=timedelta(seconds=2), maximum_interval=timedelta(minutes=1), backoff_coefficient=1.0, ) # Constants NWS_API_BASE = "https://api.weather.gov" USER_AGENT = "weather-app/1.0" # Import Activities and models, passing them through the sandbox with workflow.unsafe.imports_passed_through(): from activities.weather_activities import make_nws_request def format_alert(feature: dict) -> str: """Format an alert feature into a readable string.""" props = feature["properties"] return f""" Event: {props.get("event", "Unknown")} Area: {props.get("areaDesc", "Unknown")} Severity: {props.get("severity", "Unknown")} Description: {props.get("description", "No description available")} Instructions: {props.get("instruction", "No specific instructions provided")} """ @workflow.defn class GetAlerts: @workflow.run async def get_alerts(self, state: str) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ url = f"{NWS_API_BASE}/alerts/active/area/{state}" data = await workflow.execute_activity( make_nws_request, url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not data or "features" not in data: return "Unable to fetch alerts or no alerts found." alerts = [format_alert(feature) for feature in data["features"]] return "\n---\n".join(alerts) ``` ### GetForecast Workflow The `GetForecast` workflow demonstrates a multi-step operation: it first fetches the forecast grid endpoint for a location, then uses that information to fetch the detailed forecast. *File: workflows/weather_workflows.py* ```python @workflow.defn class GetForecast: @workflow.run async def get_forecast(self, latitude: float, longitude: float) -> str: """Get weather forecast for a location. Args: latitude: Latitude of the location longitude: Longitude of the location """ # First get the forecast grid endpoint points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}" points_data = await workflow.execute_activity( make_nws_request, points_url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not points_data: return "Unable to fetch forecast data for this location." # Get the forecast URL from the points response forecast_url = points_data["properties"]["forecast"] forecast_data = await workflow.execute_activity( make_nws_request, forecast_url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not forecast_data: return "Unable to fetch detailed forecast." # Format the periods into a readable forecast periods = forecast_data["properties"]["periods"] forecasts = [] for period in periods[:5]: # Only show next 5 periods forecast = f""" {period["name"]}: Temperature: {period["temperature"]}°{period["temperatureUnit"]} Wind: {period["windSpeed"]} {period["windDirection"]} Forecast: {period["detailedForecast"]} """ forecasts.append(forecast) return "\n---\n".join(forecasts) ``` ## Create the Activity We create an Activity for making HTTP requests to the National Weather Service API. All external API calls happen within Activities, which provides automatic retries and proper error handling through Temporal's retry mechanisms. *File: activities/weather_activities.py* ```python from typing import Any import httpx from temporalio import activity USER_AGENT = "weather-app/1.0" # External calls happen via Activities @activity.defn async def make_nws_request(url: str) -> dict[str, Any] | None: """Make a request to the NWS API with proper error handling.""" headers = {"User-Agent": USER_AGENT, "Accept": "application/geo+json"} async with httpx.AsyncClient() as client: response = await client.get(url, headers=headers, timeout=5.0) response.raise_for_status() return response.json() ``` ## Create the Worker The Worker is the process that executes Activities and Workflows. *File: worker.py* ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.envconfig import ClientConfig from temporalio.worker import Worker from activities.weather_activities import make_nws_request from workflows.weather_workflows import GetAlerts, GetForecast async def main(): # Connect to Temporal server config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") client = await Client.connect( **config, data_converter=pydantic_data_converter, ) # Register both Workflows and the Activity worker = Worker( client, task_queue="weather-task-queue", workflows=[GetAlerts, GetForecast], activities=[make_nws_request], ) print("Worker started. Listening for workflows...") await worker.run() # Start worker with both Workflows and Activities if __name__ == "__main__": asyncio.run(main()) ``` ## Configure Claude Desktop For this example, we are using Claude Desktop as the MCP Client. To use this MCP server with Claude Desktop, you need to configure it in your Claude Desktop configuration file. The config file tells Claude Desktop how to start the MCP server. *File: claude_desktop_config.json* ```json { "mcpServers": { "weather": { "command": "uv", "args": [ "--directory", "", "run", "mcp_servers/weather.py" ] } } } ``` Replace `` with the absolute path to the `hello_world_durable_mcp_server` directory. ## Configuration This recipe uses Temporal's environment configuration system to connect to Temporal. By default, it connects to a local Temporal server. To use Temporal Cloud: 1. Set the `TEMPORAL_PROFILE` environment variable to use the cloud profile: ```bash export TEMPORAL_PROFILE=cloud ``` 2. Configure the cloud profile using the Temporal CLI: ```bash temporal config set --profile cloud --prop address --value "" temporal config set --profile cloud --prop namespace --value "" temporal config set --profile cloud --prop api_key --value "" ``` For TLS certificate authentication instead of API key, refer to the [Temporal environment configuration documentation](/develop/environment-configuration) for details. ## Running the MCP server 1. Install dependencies: ```bash uv sync ``` 2. Start a Temporal server: ```bash # Using Temporal CLI temporal server start-dev ``` 3. Start the worker in one terminal: ```bash uv run worker.py ``` 4. Configure Claude Desktop by adding the configuration from `claude_desktop_config.json` to your Claude Desktop config file (typically located at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS). 5. Restart Claude Desktop to load the MCP server. Once configured, you should see the tool appear under the slider icon underneath the Claude Desktop chat input box. You can now ask Claude something like `What is the weather like in San Francisco, CA?`. Claude Desktop will understand that it needs to use the `get_forecast` tool in the Weather MCP server that you just configured. > **📝 Note:** > The National Weather Service API only supports US locations. Asking about weather in non-US locations (e.g., "What is the weather in London?") will result in a 404 error from the API. After tool execution, Claude Desktop will send the result over to the LLM (with other context) for human formatting, and then returns that result to the user. You can see these and other MCP-related actions in the `mcp_server.log`. --- # Hello world with LiteLLM Source: https://docs.temporal.io/ai/cookbook/hello-world-litellm-python > Integrate LiteLLM into a durable Temporal Workflow in Python to call and switch between LLM providers. [LiteLLM](https://github.com/BerriAI/litellm) is a library for calling LLMs from Python. It makes it easy to access, and switch between, many providers, including OpenAI, Anthropic, Google, and more. This recipe mirrors the [Basic Python recipe](./hello-world-openai-responses-python.mdx), but swaps the OpenAI SDK for LiteLLM. The Workflow still delegates LLM calls to an Activity, letting Temporal coordinate retries and durability, while LiteLLM forwards those calls to your configured provider. Key points: - A reusable Activity that wraps `litellm.acompletion` and keeps retries in Temporal. - The most common LiteLLM parameters are on `LiteLLMRequest` ensuring type checking and IDE completion. Others may be passed via the `extra_options` dictionary, which functions as `kwargs` for `litellm.acompletion`. - The Activity returns the full LiteLLM response for processing by the Workflow. ## Create the Activity `activities/models.py` ```python from dataclasses import dataclass, field from typing import Any, Dict, List, Optional, Type, Union @dataclass class LiteLLMRequest: """Lightweight container for the handful of LiteLLM options this sample surfaces.""" model: str messages: List[Dict[str, Any]] # Optional knobs: limited to the most common tweaks. temperature: Optional[float] = ( None # Controls response creativity without extra ceremony. ) max_tokens: Optional[int] = None # Caps response length/cost. timeout: Optional[Union[float, int]] = ( None # Lets callers bound slow provider responses. ) response_format: Optional[Union[dict, Type[Any]]] = ( None # Hook for JSON/object style responses. ) # Escape hatch for advanced parameters we do not model explicitly. extra_options: Dict[str, Any] = field(default_factory=dict) def to_acompletion_kwargs(self) -> Dict[str, Any]: """Convert this request to kwargs suitable for litellm.acompletion().""" kwargs = { "model": self.model, "messages": self.messages, } optional_values = { "temperature": self.temperature, "max_tokens": self.max_tokens, "timeout": self.timeout, "response_format": self.response_format, } for key, value in optional_values.items(): if value is not None: kwargs[key] = value if self.extra_options: kwargs.update(self.extra_options) return kwargs ``` `activities/litellm_completion.py` ```python from typing import Any, Dict import litellm from temporalio import activity from temporalio.exceptions import ApplicationError from activities.models import LiteLLMRequest @activity.defn(name="activities.litellm_completion.create") async def create(request: LiteLLMRequest) -> Dict[str, Any]: # Temporal best practice: disable LiteLLM retries and let Temporal handle them. kwargs = request.to_acompletion_kwargs() kwargs["num_retries"] = 0 try: response = await litellm.acompletion(**kwargs) except ( litellm.AuthenticationError, litellm.BadRequestError, litellm.InvalidRequestError, litellm.UnsupportedParamsError, litellm.JSONSchemaValidationError, litellm.ContentPolicyViolationError, litellm.NotFoundError, ) as exc: raise ApplicationError( str(exc), type=exc.__class__.__name__, non_retryable=True, ) from exc except litellm.APIError: raise return response ``` LiteLLM supports many providers. Configure credentials via environment variables (for example `OPENAI_API_KEY`) before running the Activity. For Google-hosted models (Vertex AI or Gemini), the sample relies on the `google-cloud-aiplatform` and `google-auth` dependencies included in `pyproject.toml`; set the usual Google application credentials (`GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT`, `VERTEXAI_LOCATION`, etc.) so LiteLLM can obtain an access token. ## Create the Workflow `workflows/hello_world_workflow.py` ```python from datetime import timedelta from temporalio import workflow from activities.models import LiteLLMRequest @workflow.defn class HelloWorld: @workflow.run async def run(self, input: str) -> str: messages = [ {"role": "system", "content": "You only respond in haikus."}, {"role": "user", "content": input}, ] response = await workflow.execute_activity( "activities.litellm_completion.create", LiteLLMRequest( # LiteLLM allows you to switch between models easily # model="gpt-4o-mini", model="gemini-2.5-flash-lite", messages=messages, ), start_to_close_timeout=timedelta(seconds=30), ) message = response["choices"][0]["message"]["content"] if isinstance(message, list): message = "".join( part.get("text", "") for part in message if isinstance(part, dict) ) return message ``` Temporal manages Activity retries, so LiteLLM's retry helper is disabled via `num_retries=0`. Use the `extra_options` escape hatch on `LiteLLMRequest` if you need to surface additional LiteLLM parameters without editing the sample. ## Create the Worker `worker.py` ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.worker import Worker from activities import litellm_completion from workflows.hello_world_workflow import HelloWorld async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="hello-world-python-task-queue", workflows=[ HelloWorld, ], activities=[ litellm_completion.create, ], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Create the Workflow Starter `start_workflow.py` ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from workflows.hello_world_workflow import HelloWorld async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) # Submit the Hello World workflow for execution result = await client.execute_workflow( HelloWorld.run, "Tell me about recursion in programming.", id="my-workflow-id", task_queue="hello-world-python-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Install dependencies ```bash uv sync ``` Set the appropriate environment variables before launching the worker (for example `export OPENAI_API_KEY=...` or export `GEMINI_API_KEY=...`) so LiteLLM can reach your chosen provider. Run the worker: ```bash uv run worker.py ``` Start the workflow: ```bash uv run start_workflow.py ``` --- # Hello world Source: https://docs.temporal.io/ai/cookbook/hello-world-openai-responses-python > Call an LLM from a durable Temporal Workflow in Python using the OpenAI API library. This is a simple example showing how to call an LLM from Temporal using the [OpenAI Python API library](https://github.com/openai/openai-python). Being an external API call, the LLM invocation happens in a Temporal Activity. This recipe highlights three key design decisions: - A generic Activity for invoking an LLM API. This Activity can be re-used with different arguments throughout your codebase. - Configuring the Temporal client with a `dataconverter` to allow serialization of Pydantic types. - Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. ## Create the Activity We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. In this implementation, we include only the `instructions` and `input` argument, but it could be extended to others. *File: activities/openai_responses.py* ```python from dataclasses import dataclass from openai import AsyncOpenAI from openai.types.responses import Response from temporalio import activity # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class OpenAIResponsesRequest: model: str instructions: str input: str @activity.defn async def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, timeout=15, ) return resp ``` ## Create the Workflow In this example, we take the user input and generate a response in haiku format, using the OpenAI Responses Activity. The Workflow returns `result.output_text` from the OpenAI `Response`. As per usual, the Activity retry configuration is set here in the Workflow. In this case, a retry policy is not specified so the default retry policy is used (exponential backoff with 1s initial interval, 2.0 backoff coefficient, max interval 100× initial, unlimited attempts, no non-retryable errors). The Activity module is imported inside `workflow.unsafe.imports_passed_through()`. Importing it pulls in the OpenAI client and, through it, `httpx`, which touches modules that the Workflow sandbox restricts at import time. Activity code runs outside the sandbox, so passing the module through is safe. *File: workflows/hello_world_workflow.py* ```python from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import openai_responses @workflow.defn class HelloWorld: @workflow.run async def run(self, input: str) -> str: system_instructions = "You only respond in haikus." result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=system_instructions, input=input, ), start_to_close_timeout=timedelta(seconds=30), ) return result.output_text ``` ## Create the Worker Create the process for executing Activities and Workflows. We configure the Temporal client with `pydantic_data_converter` so Temporal can serialize/deserialize output of the OpenAI SDK. *File: worker.py* ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.worker import Worker from activities import openai_responses from workflows.hello_world_workflow import HelloWorld async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="hello-world-python-task-queue", workflows=[ HelloWorld, ], activities=[ openai_responses.create, ], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Create the Workflow Starter The starter script submits the Workflow to Temporal for execution, then waits for the result and prints it out. It uses the `pydantic_data_converter` to match the Worker configuration. *File: start_workflow.py* ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from workflows.hello_world_workflow import HelloWorld async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) # Submit the Hello World workflow for execution result = await client.execute_workflow( HelloWorld.run, "Tell me about recursion in programming.", id="my-workflow-id-2", task_queue="hello-world-python-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Run the worker: ```bash uv run worker.py ``` Start execution: ```bash uv run start_workflow.py ``` --- # Retry policy from HTTP responses Source: https://docs.temporal.io/ai/cookbook/http-retry-enhancement-python > Extract retry information from HTTP response headers and pass it to Temporal's retry mechanisms in Python. This recipe extends the [Basic Example](./hello-world-openai-responses-python.mdx) to show how to extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms. HTTP response codes and headers on API calls have implications for retry behavior. For example, an HTTP `404 Not Found` generally represents an application-level error that should not be retried. By contrast, a `500 Internal Server Error` is typically transient, so should be retried. Servers can also set the `Retry-After` header to tell the client when to retry. This recipe introduces a utility function that processes the HTTP response and populates a Temporal `ApplicationError` to provide inputs to the retry mechanism. Temporal combines this information with other configuration, such as timeouts and exponential backoff, to implement the complete retry policy. ## Generate Temporal ApplicationErrors from HTTP responses We introduce a utility function that takes an `httpx.Response` object and returns a Temporal `ApplicationError` with two key fields populated: `non-retryable` and `next_retry_delay`. The `non-retryable` is determined by categorizing the HTTP status codes. The `X-Should-Retry` HTTP response header, when present, overrides the status code. **Example Retryable Status Codes:** - **408 Request Timeout** → Retry because server is unresponsive, which can have many causes - **409 Conflict** → Retry when resource is temporarily locked or in use - **429 Too Many Requests** → Retry after rate limit cooldown (respect `Retry-After` header when available) - **500 Internal Server Error** → Retry for temporary server issues - **502 Bad Gateway** → Retry when upstream server is temporarily unavailable - **503 Service Unavailable** → Retry when service is temporarily overloaded - **504 Gateway Timeout** → Retry when upstream server times out **Example Non-Retryable Status Codes:** - **400 Bad Request** → Do not retry - fix request format/parameters - **401 Unauthorized** → Do not retry - provide valid authentication - **403 Forbidden** → Do not retry - insufficient permissions - **404 Not Found** → Do not retry - resource does not exist - **422 Unprocessable Entity** → Do not retry - fix request validation errors - **Other 4xx Client Errors** → Do not retry - client-side issues need fixing - **2xx Success** → Do not expect to see this - call succeeded - **3xx Redirects** → Do not expect to see this - typically handled by httpx (with `follow_redirects=True`) If the error is retryable and if the `Retry-After` header is present, we parse it to set the retry delay. This implementation duplicates logic present in the [OpenAI Python API Library](https://github.com/openai/openai-python), where it is part of the code generated by [Stainless](https://www.stainless.com/). Duplicating the logic makes sense because it is not accessible via the public library interface and because it applies to HTTP APIs in general, not just the OpenAI API. *File: util/translate_http_errors.py* ```python import email.utils import time from datetime import timedelta from typing import Optional, Tuple from temporalio import workflow from temporalio.exceptions import ApplicationError with workflow.unsafe.imports_passed_through(): from httpx import Headers, Response # Adapted from the OpenAI Python client (https://github.com/openai/openai-python/blob/main/src/openai/_base_client.py) # which is generated by the Stainless SDK Generator. def _parse_retry_after_header(response_headers: Optional[Headers] = None) -> float | None: """Returns a float of the number of seconds (not milliseconds) to wait after retrying, or None if unspecified. About the Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#syntax """ if response_headers is None: return None # First, try the non-standard `retry-after-ms` header for milliseconds, # which is more precise than integer-seconds `retry-after` try: retry_ms_header = response_headers.get("retry-after-ms", None) return float(retry_ms_header) / 1000 except (TypeError, ValueError): pass # Next, try parsing `retry-after` header as seconds (allowing nonstandard floats). retry_header = response_headers.get("retry-after") try: # note: the spec indicates that this should only ever be an integer # but if someone sends a float there's no reason for us to not respect it return float(retry_header) except (TypeError, ValueError): pass # Last, try parsing `retry-after` as a date. retry_date_tuple = email.utils.parsedate_tz(retry_header) if retry_date_tuple is None: return None retry_date = email.utils.mktime_tz(retry_date_tuple) return float(retry_date - time.time()) def _should_retry(response: Response) -> Tuple[bool, str]: # Note: this is not a standard header should_retry_header = response.headers.get("x-should-retry") # If the server explicitly says whether or not to retry, obey. if should_retry_header == "true": return True, f"Server requested retry via x-should-retry=true header (HTTP {response.status_code})" if should_retry_header == "false": return False, f"Server prevented retry via x-should-retry=false header (HTTP {response.status_code})" # Retry on request timeouts. if response.status_code == 408: return True, f"HTTP request timeout ({response.status_code}), will retry with backoff" # Retry on lock timeouts. if response.status_code == 409: return True, f"HTTP conflict/lock timeout ({response.status_code}), will retry with backoff" # Retry on rate limits. if response.status_code == 429: return True, f"HTTP rate limit exceeded ({response.status_code}), will retry with backoff" # Retry internal errors. if response.status_code >= 500: return True, f"HTTP server error ({response.status_code}), will retry with backoff" return False, f"HTTP client error ({response.status_code}), not retrying - check your request" def http_response_to_application_error(response: Response) -> ApplicationError: """Transform HTTP response into Temporal ApplicationError for retry handling. This function implements generic HTTP retry logic based on status codes and headers. Args: response: The httpx.Response from a failed HTTP request Returns: ApplicationError: Always returns an ApplicationError configured for Temporal's retry system: - non_retryable: False for retryable errors, True for non-retryable - next_retry_delay: Server-provided delay hint (if valid) Note: Even when x-should-retry=true, this function returns an ApplicationError with non_retryable=False rather than raising an exception, for cleaner functional style. """ should_retry, retry_message = _should_retry(response) if should_retry: # Calculate the retry delay only when retrying retry_after = _parse_retry_after_header(response.headers) # Make sure that the retry delay is in a reasonable range if retry_after is not None and 0 < retry_after <= 60: retry_after = timedelta(seconds=retry_after) else: retry_after = None # Add delay info for rate limits if response.status_code == 429 and retry_after is not None: retry_message = f"HTTP rate limit exceeded (429) (server requested {retry_after.total_seconds():.1f}s delay), will retry with backoff" return ApplicationError( retry_message, non_retryable=False, next_retry_delay=retry_after, ) else: return ApplicationError( retry_message, non_retryable=True, next_retry_delay=None, ) ``` ## Raise the exception from the Activity When API calls fail, the OpenAI Client raises an `APIStatusError` exception which contains a `response` field, containing the underlying `httpx.Response` object. We use the `http_response_to_application_error` function defined above to translate this to a Temporal `ApplicationError`, which we re-throw to pass the retry information to Temporal. *File: activities/openai_responses.py* ```python from dataclasses import dataclass from openai import APIStatusError, AsyncOpenAI from openai.types.responses import Response from temporalio import activity from util.translate_http_errors import http_response_to_application_error # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class OpenAIResponsesRequest: model: str instructions: str input: str @activity.defn async def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) try: resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, timeout=15, ) return resp except APIStatusError as e: raise http_response_to_application_error(e.response) from e ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Run the worker: ```bash uv run worker.py ``` Start execution: ```bash uv run start_workflow.py ``` --- # Human-in-the-loop AI agent Source: https://docs.temporal.io/ai/cookbook/human-in-the-loop-python > Add human-in-the-loop approval to a durable AI agent using Temporal Signals in Python. This example demonstrates how to build an AI agent that requires human approval; we use Temporal Signals to bring that user input into the agent. ## Overview The Workflow implements the agent flow: 1. Uses an LLM to analyze a user request and propose an action. 2. If the proposed action is deemed risky, pauses and waits for human approval via Temporal Signal 3. Executes the action if auto-approved (if not risky) or human approved, or cancels if rejected/timed out Key features: - **Resource efficient waiting**: Can wait for approval for hours, days or indefinitely; while waiting, the agent consumes no compute resources. - **Signal-based approval**: External systems send approval decisions via Temporal Signals - **Durable timers**: Time limits placed on human-in-the-loop steps survive any execution disruptions. - **Complete audit trail**: All decisions are logged for compliance ## Prerequisites - Python 3.10+ - Temporal server running locally - OpenAI API key ## Setup 1. Install dependencies: ```bash uv sync ``` 2. Set your OpenAI API key: ```bash export OPENAI_API_KEY='your-api-key-here' ``` 3. Start Temporal Dev Server: ```bash temporal server start-dev ``` ## Running ### Start the Worker In one terminal: ```bash uv run worker.py ``` ### Start a Workflow In another terminal: ```bash uv run start_workflow.py "Delete all test data from the production database" ``` The Workflow will start, analyze the request, and pause for approval. Watch the Worker output for instructions. ### Send approval decision The Worker output will show the Workflow Id and request identifier. In another terminal, run the `send_approval` script to approve or reject: **To approve:** ```bash uv run send_approval.py approve "Looks good" ``` **To reject:** ```bash uv run send_approval.py reject "Too risky" ``` ### Testing timeout To test timeout behavior, don't send any approval signal. After 5 minutes (default), the Workflow will automatically complete with a timeout result. ## Architecture - **Models** (`models/models.py`): Data structures for workflow input, approval requests and decisions - **Activities**: - `openai_responses.py`: Generic LLM invocation activity - `execute_action.py`: Executes approved actions - The "execution" of approved actions in this sample logs messages. - In a realistic scenario, a set of tools will have been provided to the LLM and the result might be a recommended tool call. In this case, if approved, the agent would invoke the tool via an Activity. See the [agentic loop with tool calling](/ai/cookbook/agentic-loop-tool-call-openai-python) for guidance on how to use dynamic Activities, allowing the tools to be loosely coupled from the agent implementation. - `notify_approval_needed.py`: Notifies external systems of approval requests - In this sample the notification comes in the form of messages printed in the terminal running the worker. - In a realistic scenario, the notification activity may send emails, deliver messages to slack, etc. - **Workflow** (`workflows/human_in_the_loop_workflow.py`): Orchestrates the approval process - **Scripts**: - `worker.py`: Runs the Temporal worker - `start_workflow.py`: Starts workflow execution - `send_approval.py`: Helper script to send approval signals ## Key patterns We use a Temporal Signal to inject information from the human into the waiting Workflow. The Signal is delivered from some UI (in this case the `send_approval.py` script) that uses a Temporal client to deliver the data. ![](./human-in-the-loop-python-assets-temporal-signal-handling.png) Within the agent implementation there are three main elements to the solution. ### Local state within the Workflow implementation This state will be written to via the Signal handler and will be part of the condition that defines the wait point. ```python @workflow.defn class HumanInTheLoopWorkflow: def __init__(self): self.current_decision: Optional[ApprovalDecision] = None self.pending_request_id: Optional[str] = None ``` ### Signal handler The Workflow uses a Signal handler to receive approval decisions asynchronously: ```python @workflow.signal async def approval_decision(self, decision: ApprovalDecision): ... if decision.request_id == self.pending_request_id: self.current_decision = decision ... ``` ### Waiting with timeout The Workflow waits for approval with a configurable timeout: ```python await workflow.wait_condition( lambda: self.current_decision is not None, timeout=timedelta(seconds=timeout_seconds), ) ``` ## Extensions This pattern can be extended to support: - Multiple approvers with voting - Escalation workflows - Conditional approval based on action risk - Integration with Slack, email, or custom UIs - Query handlers to check approval status --- # Durable agent with tools using the OpenAI Agents SDK Source: https://docs.temporal.io/ai/cookbook/openai-agents-sdk-python > Build a durable AI agent with the OpenAI Agents SDK and Temporal that chooses tools to answer user questions. In this example, we show you how to build a durable agent using the [OpenAI Agents SDK Integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents). The AI agent we build will have access to [tools](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents#tool-calling) (Temporal Activities) to answer user questions. The agent can determine which tools to use based on the user's input and execute them as needed. This recipe highlights key implementation patterns: - **Agent-based architecture**: Uses the OpenAI Agents SDK to create an agent that can reason about which tools to use and handles LLM invocation for you. - **Tool integration**: Temporal Activities can be used as tools by the agent. The integration offers the **activity_as_tool** helper function, which: - Automatically generates OpenAI-compatible tool schemas from Activity function signatures - Wraps Activities as agent tools that can be provided directly to the Agent - Enables the agent to invoke Temporal Activities as tools, using Temporal's durable execution for tool calls - **Durable execution**: The agent's state and execution are managed by Temporal, providing reliability and observability - **Plugin configuration**: Uses the `OpenAIAgentsPlugin` to configure Temporal for OpenAI Agents SDK integration ## Create the Activity We create Activities that serve as tools for the agent. These Activities can perform tasks like getting weather information or performing calculations. *File: activities/tools.py* ```python import math from dataclasses import dataclass from temporalio import activity # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class Weather: city: str temperature_range: str conditions: str @activity.defn async def get_weather(city: str) -> Weather: """Get the weather for a given city.""" return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.") @activity.defn async def calculate_circle_area(radius: float) -> float: """Calculate the area of a circle given its radius.""" return math.pi * radius ** 2 ``` ## Create the Workflow The Workflow creates an agent with specific instructions and tools. The agent can then process user input and decide which tools to use to answer questions. Since LLM invocation is an external API call, this typically would happen in a Temporal Activity. However, because of the Temporal integration with the OpenAI Agents SDK, this is handled for us and we do not need to implement the Activity ourselves. *File: workflows/hello_world_workflow.py* ```python from datetime import timedelta from agents import Agent, Runner from temporalio import workflow from temporalio.contrib import openai_agents from activities.tools import calculate_circle_area, get_weather @workflow.defn class HelloWorldAgent: @workflow.run async def run(self, prompt: str) -> str: agent = Agent( name="Hello World Agent", instructions="You are a helpful assistant that determines what tool to use based on the user's question.", # Tools for the agent to use that are defined as activities tools=[ openai_agents.workflow.activity_as_tool( get_weather, start_to_close_timeout=timedelta(seconds=10) ), openai_agents.workflow.activity_as_tool( calculate_circle_area, start_to_close_timeout=timedelta(seconds=10) ) ] ) result = await Runner.run(agent, prompt) return result.final_output ``` ## Create the Worker Create the process for executing Activities and Workflows. We configure the Temporal client with the `OpenAIAgentsPlugin` to enable OpenAI Agents SDK integration. *File: worker.py* ```python import asyncio from datetime import timedelta from temporalio.client import Client from temporalio.contrib.openai_agents import ModelActivityParameters, OpenAIAgentsPlugin from temporalio.worker import Worker from activities.tools import calculate_circle_area, get_weather from workflows.hello_world_workflow import HelloWorldAgent async def worker_main(): # Use the plugin to configure Temporal for use with OpenAI Agents SDK client = await Client.connect( "localhost:7233", plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=30) ) ), ], ) worker = Worker( client, task_queue="hello-world-openai-agent-task-queue", workflows=[HelloWorldAgent], activities=[get_weather, calculate_circle_area], ) await worker.run() if __name__ == "__main__": asyncio.run(worker_main()) ``` ## Create the Workflow Starter The starter script submits the agent Workflow to Temporal for execution, then waits for the result and prints it out. It uses the `OpenAIAgentsPlugin` to match the Worker configuration. *File: start_workflow.py* ```python import asyncio from temporalio.client import Client from temporalio.common import WorkflowIDConflictPolicy from temporalio.contrib.openai_agents import OpenAIAgentsPlugin from workflows.hello_world_workflow import HelloWorldAgent async def main(): client = await Client.connect( "localhost:7233", # Use the plugin to configure Temporal for use with OpenAI Agents SDK plugins=[OpenAIAgentsPlugin()], ) # Start workflow print( 80 * "-" ) # Get user input user_input = input("Enter a question: ") # Submit the Hello World Agent workflow for execution result = await client.execute_workflow( HelloWorldAgent.run, user_input, id="my-workflow-id", task_queue="hello-world-openai-agent-task-queue", id_conflict_policy=WorkflowIDConflictPolicy.TERMINATE_EXISTING, ) print(f"Result: {result}") # End of workflow print( 80 * "-" ) print("Workflow completed") if __name__ == "__main__": asyncio.run(main()) ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Open a new terminal where you will run the agent worker. Set an OpenAI API key: ```bash export OPENAI_API_KEY=sk... ``` Run the worker: ```bash uv run worker.py ``` Start execution: ```bash uv run start_workflow.py ``` ## Example interactions Try asking the agent questions like: - "What's the weather in London?" - "Calculate the area of a circle with radius 5" - "What's the weather in Tokyo and calculate the area of a circle with radius 3" The agent will determine which tools to use and provide responses based on the available tools. Use the [OpenAI Traces dashboard](https://platform.openai.com/traces) to visualize and monitor your Workflows and tool calling. --- # Durable agent with MCP and Activity-backed tools using the Strands Agents SDK Source: https://docs.temporal.io/ai/cookbook/strands-agents-python > >- This recipe builds a durable AI agent using the [Strands Agents SDK Integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/strands). The `StrandsPlugin` makes every model call, tool call, and MCP interaction run as a durable Temporal Activity. The Workflow creates an agent and invokes it. The agent acts as an AWS assistant with two kinds of tools: - **An MCP tool**: the [AWS Documentation MCP server](https://github.com/awslabs/mcp/tree/main/src/aws-documentation-mcp-server), run locally with `uvx`, lets the agent search and read AWS documentation. - **A non-deterministic, Activity-backed tool**: `get_recent_aws_announcements` fetches the live AWS "What's New" RSS feed. Because it makes a network call, it is a Temporal Activity wrapped with `activity_as_tool`, which gives it Durable Execution, retries, and timeouts. It uses the Strands default Bedrock model. As of `strands-agents` 1.42.0 that model is Claude Sonnet 4.6, resolved through the `global.anthropic.claude-sonnet-4-6` inference profile. ## Prerequisites 1. **AWS Bedrock access**: Request access to Claude Sonnet 4.6 in the [Bedrock console](https://console.aws.amazon.com/bedrock/). To use a different model, pass a configured `BedrockModel` to `StrandsPlugin`. 2. **AWS credentials and region**: See [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/concepts/model-providers/amazon-bedrock/) for credential setup and model configuration. Strands falls back to `us-west-2` when `AWS_REGION` is unset, so set the region where you have model access. 3. **`uvx`**: Required to run the AWS Documentation MCP server (ships with [`uv`](https://docs.astral.sh/uv/)). 4. **A running Temporal dev server**: `temporal server start-dev`. ## Create the Activity-backed tool A non-deterministic tool (live HTTP call) is defined as a Temporal Activity. The blocking request is offloaded with `asyncio.to_thread`, and retries are left to Temporal. The docstring becomes the tool description the model sees. *File: activities/tools.py* ```python import asyncio import xml.etree.ElementTree as ET from temporalio import activity # AWS publishes a live "What's New" RSS feed of recent service launches and updates. WHATS_NEW_FEED = "https://aws.amazon.com/about-aws/whats-new/recent/feed/" @activity.defn async def get_recent_aws_announcements(limit: int = 5) -> list[dict]: """Fetch the most recent AWS 'What's New' announcements from the live RSS feed. Use this when the user asks what is new or recently launched in AWS. Returns a list of {title, link, published} for the latest service launches and updates. """ # Import requests lazily here (not at module top level) so the workflow can # import this activity without pulling a non-deterministic module into the # workflow sandbox. requests is blocking, so run it off the event loop; # Temporal handles retries, so no client-side retry configuration is needed. import requests response = await asyncio.to_thread(requests.get, WHATS_NEW_FEED, timeout=10) response.raise_for_status() root = ET.fromstring(response.content) return [ { "title": item.findtext("title", ""), "link": item.findtext("link", ""), "published": item.findtext("pubDate", ""), } for item in root.findall(".//item")[:limit] ] ``` ## Create the Workflow The Workflow creates a `TemporalAgent`, the replacement for the Strands `Agent` that is safe to use inside a Workflow, and gives it two tools: the AWS Documentation MCP server referenced by name through `TemporalMCPClient`, and the Activity wrapped with `activity_as_tool`. `invoke_async` drives the agentic loop, so there is no manual loop to maintain. *File: workflows/aws_assistant_workflow.py* ```python from datetime import timedelta from temporalio import workflow from temporalio.contrib.strands import TemporalAgent, TemporalMCPClient from temporalio.contrib.strands.workflow import activity_as_tool # activities.tools imports only stdlib at module level (requests is imported lazily # inside the activity), so it is safe to import directly into the workflow sandbox. from activities.tools import get_recent_aws_announcements SYSTEM_PROMPT = ( "You are an AWS expert assistant. Use the AWS documentation tools to answer " "questions about AWS services, and the announcements tool to report recent " "launches. Cite documentation links when they are relevant." ) @workflow.defn class AWSAssistantWorkflow: def __init__(self) -> None: # Reference the MCP server registered on the worker by name. The plugin runs # the server's list-tools / call-tool operations as Temporal Activities. # cache_tools avoids re-listing the server's tools on every model turn. aws_docs = TemporalMCPClient( server="aws-docs", cache_tools=True, start_to_close_timeout=timedelta(seconds=60), ) self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=120), system_prompt=SYSTEM_PROMPT, tools=[ # MCP tool: AWS documentation search/read, served over stdio. aws_docs, # Non-deterministic tool backed by a Temporal Activity (live HTTP call). activity_as_tool( get_recent_aws_announcements, start_to_close_timeout=timedelta(seconds=30), ), ], ) @workflow.run async def run(self, prompt: str) -> str: # TemporalAgent drives the agentic loop; every model call, tool call, and MCP # call runs as a durable Temporal Activity. result = await self.agent.invoke_async(prompt) return str(result) ``` ## Create the Worker The Worker registers the `StrandsPlugin` on the client. The plugin installs the Pydantic Data Converter, registers the model and MCP Activities, and uses the default `BedrockModel()` because no `models` are configured. MCP servers are registered by name through `mcp_clients`, each as a factory that launches the server (here, the AWS Documentation MCP server over stdio with `uvx`). *File: worker.py* ```python import asyncio from mcp import StdioServerParameters, stdio_client from strands.tools.mcp import MCPClient from temporalio.client import Client from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker from activities.tools import get_recent_aws_announcements from workflows.aws_assistant_workflow import AWSAssistantWorkflow TASK_QUEUE = "strands-aws-assistant-task-queue" def make_aws_docs_client() -> MCPClient: """Factory for the AWS Documentation MCP server, run locally via uvx.""" return MCPClient( lambda: stdio_client( StdioServerParameters( command="uvx", args=["awslabs.aws-documentation-mcp-server@latest"], ) ) ) async def main(): # The plugin registers the model invocation and MCP activities, installs the # Pydantic data converter, and (since no `models` are given) uses the default # BedrockModel(). MCP servers are registered by name via `mcp_clients`. plugin = StrandsPlugin(mcp_clients={"aws-docs": make_aws_docs_client}) client = await Client.connect("localhost:7233", plugins=[plugin]) worker = Worker( client, task_queue=TASK_QUEUE, workflows=[AWSAssistantWorkflow], activities=[get_recent_aws_announcements], ) print(f"Worker started, task queue: {TASK_QUEUE}") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Create the Workflow Starter The starter connects a client configured with the same plugin, so the Data Converters match, prompts for a question, and executes the Workflow. *File: start_workflow.py* ```python import asyncio from temporalio.client import Client from temporalio.common import WorkflowIDConflictPolicy from temporalio.contrib.strands import StrandsPlugin from workflows.aws_assistant_workflow import AWSAssistantWorkflow TASK_QUEUE = "strands-aws-assistant-task-queue" async def main(): # Match the worker's plugin so the client uses the same data converter. client = await Client.connect("localhost:7233", plugins=[StrandsPlugin()]) print(80 * "-") user_input = input("Ask the AWS assistant a question: ") result = await client.execute_workflow( AWSAssistantWorkflow.run, user_input, id="strands-aws-assistant", task_queue=TASK_QUEUE, id_conflict_policy=WorkflowIDConflictPolicy.TERMINATE_EXISTING, ) print(80 * "-") print(f"Result: {result}") print(80 * "-") if __name__ == "__main__": asyncio.run(main()) ``` ## Running Start the Temporal dev server: ```bash temporal server start-dev ``` In a new terminal, install dependencies: ```bash uv sync ``` Run the Worker, with AWS credentials and region configured as described in the prerequisites: ```bash uv run worker.py ``` In another terminal, start the Workflow: ```bash uv run start_workflow.py ``` ## Example interactions Try questions that exercise both tools: - "What did AWS launch recently, and how do I enable S3 bucket versioning?" - "Summarize the latest AWS announcements." - "How do I configure a Lambda function URL?" The agent decides which tools to use. Open the [Temporal UI](http://localhost:8233) to see the model invocation, the `get_recent_aws_announcements` Activity, and the `aws-docs` MCP list-tools and call-tool operations recorded as Activities in the Event History. ## Troubleshooting **Credentials not found**: See [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/concepts/model-providers/amazon-bedrock/). **Access denied or model not found**: Confirm you have access to Claude Sonnet 4.6 in the Bedrock console for the region you are using, and that `AWS_REGION` names that region. Strands defaults to `us-west-2` when the region is unset. **`uvx: command not found`**: Install [`uv`](https://docs.astral.sh/uv/); `uvx` runs the AWS Documentation MCP server. ## Learn more - [Temporal Strands Agents Plugin](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/strands) - [Strands' Amazon Bedrock guide](https://strandsagents.com/docs/user-guide/concepts/model-providers/amazon-bedrock/) - [Temporal Strands Agents Samples (samples-python)](https://github.com/temporalio/samples-python/tree/main/strands_plugin) - [Strands Agents Documentation](https://strandsagents.com/latest/documentation/) --- # Structured outputs with Temporal and OpenAI Source: https://docs.temporal.io/ai/cookbook/structured-output-openai-responses-python > Use Temporal and the OpenAI Responses API to reliably request output conforming to a specific data structure. The OpenAI Responses API provides the [Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs) allowing you to request responses conforming to a specific data structure. In this example, we use structured outputs in a business data cleaning scenario. Structured outputs are also commonly used for tool calling. OpenAI usually returns the correct type. However, this is not always the case due to the non-deterministic nature of LLMs. When OpenAI returns an incorrect type, Temporal automatically retries the LLM call Activity. ## Invoke model Activity We create a model-calling Activity that uses the `responses.parse` method of the OpenAI client. Key challenges are related to serialization: 1. In `InvokeModelRequest` the `response_format` field is a class reference. We provide custom Pydantic serialization and deserialization logic. 2. In `InvokeModelResponse` the `response_model` must be deserialized to the correct type. We serialize the type in one field and the model, represented as a dictionary, in another. *File: activities/invoke_model.py* ```python import importlib from typing import Any, Generic, List, Optional, TypeVar, cast from openai import AsyncOpenAI from pydantic import BaseModel from pydantic.functional_serializers import PlainSerializer from pydantic.functional_validators import BeforeValidator from temporalio import activity from typing_extensions import Annotated T = TypeVar("T", bound=BaseModel) def _coerce_class(v: Any) -> type[Any]: """Pydantic validator: convert string path to class during deserialization.""" if isinstance(v, str): mod_path, sep, qual = v.partition(":") if not sep: # support "package.module.Class" mod_path, _, qual = v.rpartition(".") module = importlib.import_module(mod_path) obj = module for attr in qual.split("."): obj = getattr(obj, attr) return cast(type[Any], obj) elif isinstance(v, type): return v else: raise ValueError(f"Cannot coerce {v} to class") def _dump_class(t: type[Any]) -> str: """Pydantic serializer: convert class to string path during serialization.""" return f"{t.__module__}:{t.__qualname__}" # Custom type that automatically handles class <-> string conversion in Pydantic serialization ClassReference = Annotated[ type[T], BeforeValidator(_coerce_class), PlainSerializer(_dump_class, return_type=str), ] class InvokeModelRequest(BaseModel, Generic[T]): model: str instructions: str input: str response_format: Optional[ClassReference[T]] = None tools: Optional[List[dict]] = None class InvokeModelResponse(BaseModel, Generic[T]): # response_format records the type of the response model response_format: Optional[ClassReference[T]] = None response_model: Any @property def response(self) -> T: """Reconstruct the original response type if response_format was provided.""" if self.response_format: model_cls = self.response_format return model_cls.model_validate(self.response_model) return self.response_model @activity.defn async def invoke_model(request: InvokeModelRequest[T]) -> InvokeModelResponse[T]: client = AsyncOpenAI(max_retries=0) kwargs: dict[str, Any] = { "model": request.model, "instructions": request.instructions, "input": request.input, } if request.response_format: kwargs["text_format"] = request.response_format if request.tools: kwargs["tools"] = request.tools # Use responses API consistently resp = await client.responses.parse(**kwargs) if request.response_format: # Convert structured response to dict for managed serialization. # This allows us to reconstruct the original response type while maintaining type safety. parsed_model = cast(BaseModel, resp.output_parsed) return InvokeModelResponse( response_model=parsed_model.model_dump(), response_format=request.response_format, ) else: return InvokeModelResponse( response_model=resp.output_text, response_format=None ) ``` ## Workflow We define the `Business` class as a Pydantic model. We use the Pydantic's `EmailStr` type for the email field. For the phone field, we use a custom validator to ensure the phone number is in E.164 format. The validators should check for obvious structural errors that LLMs will only get wrong sporadically. If the LLM produces invalid responses consistently, Activity retries will fail consistently. To mitigate the cost of such futile retries, we limit the number of retry attempts when using structured outputs. *File: workflows/clean_data_workflow.py* ```python import re from datetime import timedelta from typing import List, Optional from pydantic import BaseModel, EmailStr, Field, field_validator from pydantic_core import PydanticCustomError from temporalio import workflow from temporalio.common import RetryPolicy from activities import invoke_model from activities.invoke_model import InvokeModelRequest class Business(BaseModel): name: Optional[str] = Field( None, description="The business name", json_schema_extra={"example": "Acme Corporation"}, ) email: Optional[EmailStr] = Field( None, description="Primary business email address", json_schema_extra={"example": "info@acmecorp.com"}, ) phone: Optional[str] = Field( None, description="Primary business phone number in E.164 format", json_schema_extra={"example": "+12025550173"}, ) address: Optional[str] = Field( None, description="Business mailing address", json_schema_extra={ "example": "123 Business Park Dr, Suite 100, New York, NY 10001" }, ) website: Optional[str] = Field( None, description="Business website URL", json_schema_extra={"example": "https://www.acmecorp.com"}, ) industry: Optional[str] = Field( None, description="Business industry or sector", json_schema_extra={"example": "Technology"}, ) @field_validator("phone", mode="before") def validate_phone(cls, v): # Allow None values if v is None: return None if isinstance(v, str): v = v.strip() # Allow empty strings to be converted to None for optional fields if not v: return None # E.164 format: + followed by 1-9, then 9-15 more digits e164_pattern = r"^\+[1-9]\d{9,15}$" if not re.match(e164_pattern, v): raise PydanticCustomError( "phone_format", "Phone number must be in E.164 format (e.g., +12025550173)", {"invalid_phone": v}, ) return v @field_validator("name", mode="before") def validate_name(cls, v): # Allow None values if v is None: return None if isinstance(v, str): v = v.strip() # Convert empty strings to None (this is acceptable) if not v: return None return v class BusinessList(BaseModel): businesses: List[Business] @workflow.defn class CleanDataWorkflow: @workflow.run async def run(self, data: str) -> BusinessList: results = await workflow.execute_activity( invoke_model.invoke_model, InvokeModelRequest( model="gpt-4o", instructions="""Extract and clean business data with these specific rules: 1. BUSINESS NAME: Extract the main business name, normalize capitalization (Title Case for proper nouns) 2. EMAIL: - Extract only ONE primary email address - If multiple emails, choose the one marked as "primary" or the first valid one - Validate format (must have @ and valid domain with .) - Set to null if invalid (e.g., "bob@email", "NONE PROVIDED") 3. PHONE: - Convert to E.164 format (+1 prefix for US numbers, add if not provided) - Convert letters to numbers where appropriate (e.g., "1-800-FLOWERS" → "+18003569377") - Set to null if cannot be converted to valid E.164 format - Examples: "(555) 123-4567" → "+15551234567", "555 234 5678 ext 349i" → null (invalid), "5551234567" → "+15551234567" 4. ADDRESS: - Provide complete, standardized address - Set to null if vague/incomplete (e.g., "north end of main st", "unknown", "[PRIVATE]") 5. WEBSITE: - Standardize to https:// format - Remove "www." prefix, add https:// if missing - Set to null if broken/invalid (e.g., "broken-link.com/404", "down for maintenance") 6. INDUSTRY: - Use clear, professional industry categories - Normalize similar terms (e.g., "fix cars and trucks" → "Automotive Repair") Return null for any field that cannot be reliably extracted or validated.""", input=data, response_format=BusinessList, ), start_to_close_timeout=timedelta(seconds=300), retry_policy=RetryPolicy( maximum_attempts=3, ), summary="Clean data", ) return results.response ``` ## Running Start the Temporal Dev Server: ```bash temporal server start-dev ``` Run the worker: ```bash uv run worker.py ``` Start execution: ```bash uv run start_workflow.py ``` --- # Tool calling agent Source: https://docs.temporal.io/ai/cookbook/tool-call-openai-python > >- In this example, we demonstrate how function calling (also known as tool calling) works with the [OpenAI API](https://github.com/openai/openai-python) and Temporal. Tool calling allows the model to make decisions on which, if any, functions should be invoked. It also provides information to the LLM that will allow it to structure the response so that the agent can invoke the functions. Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools). The `tools` parameter is in `json` and includes a description of the function as well as descriptions of each of the arguments. > **⚠️ Caution:** > The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. Being external API calls, invoking the LLM and invoking the function are each done within a Temporal Activity. This example lays the foundation for the core agentic pattern where the LLM makes the decision on functions/tools to invoke, the agent calls the function/tool(s) and the response from such calls is sent back to the LLM for interpretation. ![](./tool-call-openai-python-assets-tool-calling-flow.png) This recipe highlights these key design decisions: - A generic Activity for invoking an LLM API; that is, instructions and other responses arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases. - We have intentionally not implemented the agentic loop so as to focus on how tool details are made available to the LLM and how functions are invoked. We do take the tool output and have the LLM interpret it in a manner consistent with the AI agent pattern. - Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. ## Create the Activity for LLM invocations We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools. `activities/openai_responses.py` ```python from dataclasses import dataclass from typing import Any from openai import AsyncOpenAI from openai.types.responses import Response from temporalio import activity # Temporal best practice: Create a data structure to hold the request parameters. @dataclass class OpenAIResponsesRequest: model: str instructions: str input: object tools: list[dict[str, Any]] @activity.defn async def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, tools=request.tools, timeout=30, ) return resp ``` ## Create the Activity for the tool invocation We create a wrapper for invoking the [National Weather Service API](https://www.weather.gov/documentation/services-web-api), specifically for the weather alerts endpoint. We follow the Temporal best practice of encapsulating all input parameters to the Activity in a data structure, even here where this is only one argument. The `WEATHER_ALERTS_TOOL_OAI` uses a function defined in `helpers/tool_helpers.py` that calls the aforementioned internal OpenAI function, generating a dictionary that becomes the argument passed into the OpenAI responses API. `activities/get_weather_alerts.py` ```python # weather_activities.py import json from typing import Any import httpx from pydantic import BaseModel, Field from temporalio import activity from helpers import tool_helpers # Constants NWS_API_BASE = "https://api.weather.gov" USER_AGENT = "weather-app/1.0" def _alerts_url(state: str) -> str: return f"{NWS_API_BASE}/alerts/active/area/{state}" # External calls happen via activities now async def _make_nws_request(url: str) -> dict[str, Any] | None: """Make a request to the NWS API with proper error handling.""" headers = { "User-Agent": USER_AGENT, "Accept": "application/geo+json" } async with httpx.AsyncClient() as client: response = await client.get(url, headers=headers, timeout=5.0) response.raise_for_status() return response.json() # Build the tool for the OpenAI Responses API. We use Pydantic to create a structure # that encapsulates the input parameters for both the weather alerts activity and the # tool definition that is passed to the OpenAI Responses API. class GetWeatherAlertsRequest(BaseModel): state: str = Field(description="Two-letter US state code (e.g. CA, NY)") WEATHER_ALERTS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_weather_alerts", "Get weather alerts for a US state.", GetWeatherAlertsRequest) @activity.defn async def get_weather_alerts(weather_alerts_request: GetWeatherAlertsRequest) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ data = await _make_nws_request(_alerts_url(weather_alerts_request.state)) return json.dumps(data) ``` ### Create the helper function The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API. > **⚠️ Caution:** > The API used to generate the tools json is an internal function from the [OpenAI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. `helpers/tool_helpers.py` ```python from openai.lib._pydantic import to_strict_json_schema # private API; may change # there currently is no public API to generate the tool definition from a Pydantic model # or a function signature. from pydantic import BaseModel def oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]): return { "type": "function", "name": name, "description": description, "parameters": to_strict_json_schema(model), "strict": True, } ``` ## Create the agent The agent is implemented as a Temporal Workflow that orchestrates - the initial LLM call with the initial user input and guidance to the LLM that they should respond in haiku when the user input doesn't lead to a tool call, - the invocation of the function, if the LLM has chosen one - and if a function has been called, the result is appended to the context that is then sent back to the LLM for interpretation (the LLM is instructed to format the tool response). `workflows/get_weather_workflow.py` ```python import json from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import get_weather_alerts, openai_responses @workflow.defn class ToolCallingWorkflow: @workflow.run async def run(self, input: str) -> str: input_list = [ {"role": "user", "content": input} ] # We take the user input and pass it to the LLM with the system instructions # and the tool to use, if applicable. system_instructions = "if no tools seem to be needed, respond in haikus." result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=system_instructions, input=input_list, tools=[get_weather_alerts.WEATHER_ALERTS_TOOL_OAI], ), start_to_close_timeout=timedelta(seconds=30), ) # For this simple example, we only have one item in the output list item = result.output[0] # if the result is a tool call, call the tool if item.type == "function_call": if item.name == "get_weather_alerts": # serialize the output, which is an OpenAI object input_list += [ i.model_dump() if hasattr(i, "model_dump") else i for i in result.output ] result = await workflow.execute_activity( get_weather_alerts.get_weather_alerts, get_weather_alerts.GetWeatherAlertsRequest(state=json.loads(item.arguments)["state"]), start_to_close_timeout=timedelta(seconds=30), ) # add the tool call result to the input list for context input_list.append({"type": "function_call_output", "call_id": item.call_id, "output": result}) result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions="return the tool call result in a readable format", input=input_list, tools=[] ), start_to_close_timeout=timedelta(seconds=30), ) result = result.output_text return result ``` ## Create the Worker The Worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for the LLM and tool invocations. *File: worker.py* ```python import asyncio from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from temporalio.worker import Worker from activities import get_weather_alerts, openai_responses from workflows.get_weather_workflow import ToolCallingWorkflow async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-calling-python-task-queue", workflows=[ ToolCallingWorkflow, ], activities=[ openai_responses.create, get_weather_alerts.get_weather_alerts, ], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## Initiate an interaction with the agent To interact with this simple AI agent, we create a Temporal client and execute a Workflow. `start_workflow.py` ```python import asyncio import sys from temporalio.client import Client from temporalio.contrib.pydantic import pydantic_data_converter from workflows.get_weather_workflow import ToolCallingWorkflow async def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Hello, how are you?" # Submit the Tool Calling workflow for execution result = await client.execute_workflow( ToolCallingWorkflow.run, query, id="my-workflow-id", task_queue="tool-calling-python-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Running ### Start the Temporal Dev Server ```bash temporal server start-dev ``` ### Install dependencies From this directory: ```bash uv sync ``` ### Run the worker First set the `OPENAI_API_KEY` environment variable and then: ```bash uv run worker.py ``` ### Initiate an interaction with the agent This user input should not result in any tool call ```bash uv run start_workflow.py "Tell me about recursion in programming." ``` This user input should invoke the tool and respond with current weather alerts for California. ```bash uv run start_workflow.py "Are there any weather alerts in California?" ``` --- # Best practices Source: https://docs.temporal.io/best-practices This section collects prescriptive, validated guidance for platform teams, architects, and developers establishing Temporal standards across an organization. Each page recommends a specific approach based on real-world deployments, rather than explaining how a feature works internally — for that, see [Encyclopedia](/temporal). Looking for a working code example instead of a principle? See [Guides](/guides) for pattern-by-pattern implementations with runnable code. ## Namespace, tenancy, and capacity Start here to decide how many Namespaces you need, how tenants share them, and what capacity model fits your traffic. These pages form one decision chain: draw your Namespace boundaries, decide how tenants share those boundaries, size your Actions-per-second capacity, and understand what it costs. - **[Namespace best practices](./managing-namespace.mdx)** — naming conventions, organizational patterns for splitting Namespaces, and production safeguards like deletion protection and Infrastructure as Code. - **[Multi-tenant application patterns](./multi-tenant-patterns.mdx)** — Task Queue and Namespace isolation patterns for multi-tenant applications, with worked capacity-planning examples. - **[Managing Actions per Second (APS) limits](./managing-aps-limits.mdx)** — why workloads hit APS limits, how to design Workflows that use Actions efficiently, and when to use Provisioned Capacity. - **[Cost optimization](./cost-optimization.mdx)** — common cost anti-patterns and strategies for reducing Actions and Storage costs without sacrificing observability. - **[Cost governance](./cost-governance.mdx)** — a design framework for cost attribution, budget forecasting, and anomaly detection using the Billing API and OpenMetrics. ## Security and access control Temporal Cloud secures the managed service; you're responsible for how your applications authenticate to it and who can administer your account. These two pages cover both halves. - **[Managing Temporal Cloud access control](./cloud-access-control.mdx)** — choosing between mTLS certificates and API keys, structuring Service Accounts, and rotating credentials without downtime. - **[Security controls for Temporal Cloud](./security-controls.mdx)** — identity and access management, network isolation, data encryption, and availability guidance for a Temporal Cloud account. ## Worker and Workflow reliability These pages cover the full lifecycle of running reliable Workflows in production: deploy and tune Workers correctly, alert on the metrics that catch failures early, handle errors correctly in Workflow and Activity code, and validate that all of it survives real failure conditions before you rely on it. - **[Worker deployment and performance](./worker.mdx)** — deployment, scaling, and tuning practices for Workers, illustrated with a reference application. - **[Alerting on Worker metrics](./worker-alerting.mdx)** — a recommended alert set with thresholds and triage links. - **[Error handling](./error-handling.mdx)** — categorizing failures, when to mark errors non-retryable, and implementing compensation with the Saga pattern. - **[Pre-production testing](./pre-production-testing.mdx)** — failure injection, load testing, and a game-day runbook for validating operational readiness. ## Organizational enablement Once your own standards are established, the next challenge is getting every team to follow them without funneling every question through the platform team. - **[Knowledge hub](./knowledge-hub.mdx)** — what belongs in an internal Temporal knowledge hub, how to measure its effectiveness, and how to keep it current. > **📝 Note:** > Scope > Most of this section covers Temporal Cloud operations. Namespace best practices, Worker deployment, and error handling > apply to self-hosted Temporal too; security controls, access control, cost optimization, cost governance, and APS > limits are Cloud-specific. For self-hosted security guidance, see [Security (self-hosted)](/self-hosted-guide/security). --- # Managing Temporal Cloud access control Source: https://docs.temporal.io/best-practices/cloud-access-control > Best practices for managing access control, permissions, and user management in Temporal Cloud. Temporal Cloud supports two secure authentication methods for Workers: - **mTLS Certificates** - **API Keys** (configured via the UI when creating a namespace) Both options help secure communication between workers and Temporal Cloud. Choosing the right method and managing it properly is key to maintaining security and minimizing downtime. Use this page to define your operating model for machine access to Temporal Cloud. For setup steps and product-specific mechanics, see [Manage API keys](/cloud/api-keys) and [Manage service accounts](/cloud/manage-access/service-accounts). Related guidance: - [Namespace best practices](/best-practices/managing-namespace) - [Multi-tenant application patterns](/best-practices/multi-tenant-patterns) The high-level end-to-end rotation process is: 1. **Generate new credentials**: Create new certificates or API keys in Temporal Cloud before the current ones expire 2. **Support dual credentials**: Update Temporal Cloud to support both old and new credentials 3. **Migrate Workers**: Transition Worker applications from old credentials to new credentials 4. **Validate connectivity**: Confirm all Workers can authenticate, and business processes operate normally with new credentials 5. **Remove old credentials**: Remove old certificates and API keys from your secrets provider after confirming successful migration This approach ensures near-zero-downtime rotation and prevents authentication failures that could impact running workflows. For specific guidance to rotate mTLS certificates and API keys, see: - [How to add, update, and remove certificates in a Temporal Cloud Namespace](/cloud/certificates#manage-certificates) - [Rotate an API key](/cloud/api-keys#rotate-an-api-key) - Per-SDK code for rotating a Worker's mTLS client certificate without a restart is documented on each SDK's Temporal Client page, in the "Connect to Temporal Cloud" section (for example, [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud)) — Go, Java, Python, .NET, Ruby, and TypeScript are all supported; PHP and Rust are not yet. The [temporal-worker-cert-rotation](https://github.com/temporal-sa/temporal-worker-cert-rotation) reference implementation walks through automating this with cert-manager on Kubernetes. For mutual TLS (mTLS) implementations, using Let's Encrypt is not recommended, as it is designed primarily for public-facing services and lacks support for internal certificate requirements. While we are not making a specific product recommendation, there are several valid options for managing certificates. Many organizations choose vendor solutions such as AWS Private CA, Sectigo, Microsoft Certification Authority, or DigiCert for their robust integration and lifecycle features. Alternatively, self-signed certificates are a valid and commonly used approach, even in production environments. If you choose to self-sign, tools like [OpenSSL](https://openssl-library.org/), [CFSSL](https://github.com/cloudflare/cfssl), or [step CLI](https://github.com/smallstep/cli) can help generate and manage certificates effectively. Select the option that aligns best with your infrastructure, security requirements, and operational needs. In the case that you are using multiple certificates signed by the same CA, and some of these certificates are for production environments, there are some workarounds you can employ. One convention is to give certificates a common name that matches the namespace. If you do this when using the same CA for dev and prod, then you can leverage Certificate Filters to prevent access to production environments. This is described in detail under the [authorization section](/cloud/certificates#control-authorization) of the documentation. ## Recommendations ### Establish clear guidelines on authentication methods Teams should standardize on either [mTLS certificates](/cloud/certificates) or [API keys](/cloud/api-keys) for the following operations: - Connect Temporal clients to Temporal Cloud (for example, Worker processes) - Automation (for example, Temporal Cloud [Operations API](/ops), [Terraform provider](/cloud/terraform-provider), [Temporal CLI](/cli/setup-cli)) By default, teams should use API keys with [service accounts](/cloud/manage-access/service-accounts) for both operations. API keys are generally easier to set up and rotate than mTLS certificates, and service accounts let you assign account-level and namespace-level roles. If your organization requires mutual authentication and stronger cryptographic guarantees, use [mTLS certificates](/cloud/certificates) to authenticate Temporal clients to Temporal Cloud and use API keys for automation, because the Temporal Cloud [Operations API](/ops) and [Terraform provider](/cloud/terraform-provider) only support API key authentication. Unlike API keys tied to users or service accounts, mTLS certificate authentication is not tied to Temporal Cloud RBAC identities. Namespace access is based on CA trust, with optional [Certificate Filters](/cloud/certificates#manage-certificate-filters) to narrow access by Common Name. ### Default operating model for service accounts and API keys For most organizations, use the following defaults: - Create one Service Account per service or worker deployment, not one shared Service Account for an entire team - Use account-level Service Accounts only when a service genuinely needs cross-Namespace or account-wide access - Prefer Namespace-scoped Service Accounts when a service should only access one Namespace - Grant Service Accounts namespace-level access only to the specific Namespaces they need This approach gives you cleaner ownership, easier rotation, and better auditability than sharing a single machine identity across multiple services. ### Use access boundaries that match your Namespace boundaries The way you partition Namespaces should usually match the way you partition machine identities. - If multiple services share a Namespace, you may still want one Service Account per service so that each deployment can rotate credentials independently. - If you split workloads into separate Namespaces for security, capacity, or team ownership reasons, those Namespaces should usually have separate Service Accounts and API keys as well. - If you use Namespace-per-tenant isolation, expect your credential model and RBAC model to become correspondingly more granular. For more on topology tradeoffs, see [Namespace best practices](/best-practices/managing-namespace) and [Multi-tenant application patterns](/best-practices/multi-tenant-patterns). ### Rotate credentials without downtime Use the following sequence when rotating credentials: 1. Create the replacement credential before the existing one expires. 2. For API keys, create the new valid key while the old key still works, then roll your Workers and clients to use the new key. 3. For client certificates, stage the new certificate before removing the old one when your deployment process supports that transition. 4. Validate connectivity and normal Workflow execution using the new credential. 5. Remove the old credential only after all clients and Workers have switched. ### Use Certificate Filters to restrict access when using shared CAs (for example, `dev` vs `prod`) Certificate Filters are an additional way of validating using the client certificate presented during client authentication. Give certificates a common name that matches the namespace. This is not a requirement. If you do this when using the same CA for dev and prod environments, then you can leverage Certificate Filters to prevent access to production. --- # Cost governance on Temporal Cloud Source: https://docs.temporal.io/best-practices/cost-governance > A design framework for attributing Temporal Cloud spend to teams, forecasting budget burn rate, and detecting usage anomalies using the Billing API, Billing Center, Usage Dashboards, and OpenMetrics. This guide lays out a cost-governance framework built entirely on existing Temporal Cloud tooling: the [Cloud Billing API](/cloud/billing-api), [Billing Center](/cloud/billing), [Usage Dashboards](/cloud/actions-usage), and the [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count). Enterprise platform teams consistently need to solve three problems: - **Cost attribution**: attributing spend to the team, environment, and workload that generated it. - **Budget forecasting and trend tracking**: knowing where a Namespace stands against its monthly budget before the invoice closes, not after. - **Anomaly detection on usage**: treating a spend spike as an early operational signal, not just a finance surprise. This guide helps you build cost visibility in from the start, rather than discover the gap after costs have already accumulated. ## Key tools and concepts Temporal provides tooling across Temporal Cloud, metrics, and the Cloud Operations API to address each of these use cases programmatically: | Tool | Grain | Best for | Who sees it | | :--- | :--- | :--- | :--- | | [Billing Center](/cloud/billing) | Monthly invoice | Summary invoices, credits, plan management | Account Owner, Finance Admin | | [Billing API](/cloud/billing-api) | Hourly, daily, monthly | Namespace and tag cost attribution, FinOps ingestion | Account Owner, Finance Admin | | [Usage Dashboards](/cloud/actions-usage) | Namespace, by Action category | At-a-glance usage in the Cloud UI | Account Owners, Finance Admins, and Global Admins at account level; anyone with Namespace access at Namespace level | | [Actions in Event History](/cloud/actions-usage#actions-in-workflows) | Per Workflow Execution | Estimating Actions for a specific execution | Account Owners, Global Admins, Namespace Admins, Developers, and Read-Only users | | [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count) | 1-minute, by Workflow and Action type | Near-real-time usage behavior, alerting, debugging | Service account with the Metrics Read-Only role | ### Actions Temporal Cloud bills primarily on [Actions](/cloud/actions), which are billable operations such as starting a Workflow, scheduling an Activity, recording a Heartbeat, sending a Signal, or receiving a Query or Update. Actions are grouped into categories (Workflow, Activity, Timer, Signal, Query, Update, Schedule, Nexus) plus a few billed features (Export, Fairness, Capacity). For background on how usage-based pricing works on Temporal Cloud, see [Improved cost transparency with usage-based billing](https://temporal.io/blog/improved-cost-transparency-with-usage-based-billing). > **📝 Note:** > Not every Action type appears in every surface. Billing and usage data has the most complete picture; Event History > and OpenMetrics are best used for estimation and trend analysis, not invoice reconciliation. ### Namespace tags The Billing API attributes every charge to a Namespace and enriches it with [tags](/cloud/namespaces#tag-a-namespace), user-defined key/value pairs such as team, environment, or cost center. The Billing API reads tags in real time, so a Namespace that shipped last month without a tag can be tagged today and have its historical cost re-attributed retroactively. This solves the most common early mistake: forgetting to tag a Namespace before it starts accruing spend. ```json { "$tmprl_project": ["claims-platform"], "team": ["map"], "env": ["prod"] } ``` ## Attribute cost by team and workload **Goal**: give every Temporal Cloud team, use case, and application (for example, Claim, Payment, Notification) its own line item, sourced directly from Temporal Cloud's billing data instead of an estimate. Use this for both cost attribution and budget tracking. ### Design artifacts - Namespace-per-tenant-per-environment as the attribution boundary. Namespace is the finest-grained unit the Billing API attributes cost to. - A required tagging convention enforced at Namespace creation time. At minimum: team, env, and workload or product. - A daily pull of the Billing API report, joined on tags, feeding the internal FinOps or budget system so each tenant sees its own line item. **Recommended tagging taxonomy** | Tag key | Example value | Purpose | | :--- | :--- | :--- | | team | claim-team, payment-team, notification-team | Chargeback owner | | env | prd, stg, dev | Separates production spend from lower environments | | workload | claims-processing, payment, notification | Sub-team or product-line attribution | | cost-center | cc-12345 | Direct feed into the finance general ledger, if required | ### Implementation path - Generate a [Billing API](/cloud/billing-api) report at daily granularity for the current and prior two billing months: create it with `CreateBillingReport`, poll `GetBillingReport`, then download the CSV. You can do this through the Cloud UI or the [Cloud Operations API](/ops). - Parse the FOCUS-aligned CSV. Group by `ResourceName` (Namespace name plus Temporal Cloud account ID) and `Tags`; sum `ContractedCost` by `ChargeDescription`. - Feed the grouped output into your internal FinOps or budget system as a recurring Workflow, or connect Temporal Cloud's native Datadog Cloud Cost Management or Vantage integration if either tool is already in your FinOps stack. ## Forecast budget burn rate **Goal**: answer "where are we trending against this Namespace's budget, and will we exceed it before month end?" with enough lead time to act. ### Two data sources, two jobs Budget tracking needs both a financially accurate source and a fast, directional source, because the Billing API's current-month data is provisional until the billing month closes: | Source | Accuracy | Latency | Role in burn-rate tracking | | :--- | :--- | :--- | :--- | | [Billing API](/cloud/billing-api) (daily grain) | Invoice-aligned | Usage up to current time minus 24 hours; final at month close | Month-to-date actual spend per Namespace, financial source of truth | | [Action metric](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count) | Directional usage estimate | Available within about 3 minutes | Early-warning proxy for spend trajectory between billing pulls | ### Burn-rate methodology - Maintain a budget table per Namespace (or per team, rolling up Namespaces) with a monthly allocation. - Each day, pull the Billing API daily report and compute month-to-date (MTD) actual spend per Namespace. - Project end-of-month (EOM) spend using a simple run-rate extrapolation: EOM projection = MTD spend ÷ days represented in the report × days in month. Derive the represented period from the report's charge dates so the roughly 24-hour Billing API lag doesn't depress the projection. - Compare the projection against the allocated budget and flag any Namespace trending to exceed it. - Between daily Billing API pulls, use the 24-hour Actions estimate from [OpenMetrics](/cloud/metrics/openmetrics) as a same-day directional check on whether a Namespace's usage trajectory has shifted: ``` # 24-hour Actions estimate from OpenMetrics (PromQL) sum( avg_over_time( temporal_cloud_v1_billable_action_count{temporal_namespace="$namespace"}[24h:1m] ) ) * 86400 ``` ### Alerting - Trigger a budget alert when a Namespace's EOM projection exceeds 90% (warning), 100% (target), or 110% (over budget) of its allocation. - Route alerts to the Namespace owner and the FinOps or platform DRI. > **❗ Important:** > `temporal_cloud_v1_billable_action_count` is a usage estimate, not your bill. It doesn't account for storage, support > fees, pricing-tier calculations, some features (TRUs, Fairness), or rounding. Use it to catch trend shifts early in > the month, and reconcile final numbers against the Billing API or Billing Center once the billing month closes. ## Detect usage anomalies early **Goal**: treat a sudden spend or Action spike as an early operational signal for a runaway Workflow or bad deploy. ### Why OpenMetrics is the right layer The `temporal_cloud_v1_billable_action_count` metric is broken down by both `action_type` and `temporal_workflow_type` at one-minute granularity. That combination answers the two questions that matter most during an incident: which Workflow is driving the spike, and what kind of Action it's doing more of. For example queries, see [Getting the most out of the Billable Action Count metric](https://temporal.io/blog/getting-the-most-out-of-the-billable-action-count-metric). For a real-world example of using this metric to validate metering, see [Dogfooding the Billable Actions metric](https://temporal.io/blog/dogfooding-the-billable-actions-metric-how-granular-observability-improved-our-metering-validation). ### Defining and finding anomalies Establish a baseline of Actions consumption. Your workload may have seasonality across hours, days, and weeks, which makes setting alerts harder. Some starting points: - Alert if the number of Actions for an hour exceeds the expected maximum for all hours in a day. - Alert if the number of Actions for a day exceeds the expected maximum for all days in a month. When an alert triggers, or during root cause analysis, check monitoring dashboards for more nuanced anomalies, including: - New business generating more Action usage. Determine whether the pattern is a spike or steady growth that's now exceeding typical thresholds — if it's growth, it's time to establish a new baseline. - Correlation with deploys. Feed deploy markers from your CI/CD pipeline into the same Datadog or Grafana dashboard as your Action-rate charts; a spike that lines up with a deploy timestamp is a regression. ## Guardrails to design around - `temporal_cloud_v1_billable_action_count` is a directional estimate. Don't use it for financial reconciliation — use the Billing API instead. - Current-month billing data is provisional. It finalizes only when the billing month closes, and available data lags current time by roughly 24 hours. - Only one billing report generates at a time per account; additional requests queue rather than fail. Use an idempotency key (`async_operation_id`) on retries and poll with exponential backoff. - High-cardinality metrics need filtering at scale. Plan Namespace and label filtering into your [OpenMetrics](/cloud/metrics/openmetrics) scrape configuration from day one, not after you hit the datapoint ceiling. None of the three use cases in this guide need new Temporal Cloud features. They need a deliberate design applied to tooling that already exists: a tagging convention enforced at Namespace creation, a scheduled pull of the Billing API, and an OpenMetrics-based alerting layer tuned to your account's scale. The organizations that get the most value from this treat it as a platform requirement before the first tenant onboards, not as a remediation project after the first unexplained invoice. The result is a Temporal Cloud bill that behaves like any other well-governed enterprise OpEx line: attributable, forecastable, and defensible. --- # Workflow cost optimization Source: https://docs.temporal.io/best-practices/cost-optimization > Strategies for optimizing costs associated with workloads running on Temporal Cloud while maintaining workflow reliability and observability. This guide provides strategies for optimizing costs associated with workloads running on Temporal Cloud while maintaining Workflow reliability and observability. ## Overview Temporal Cloud uses consumption-based pricing with two primary cost components: [Actions and Storage](/cloud/pricing#action). Optimization opportunities vary significantly based on your workload characteristics - Workflows with high signal volume face different cost drivers than long-running Workflows with large payloads. > **❗ Important:** > Build Workflows following best practices first, then optimize based on observed costs. > Premature optimization can compromise observability and create operational challenges. Every optimization involves tradeoffs. This guide helps you make informed decisions about where and how to optimize based on your specific requirements. Should you need additional guidance on Workflow design considerations, please reach out to a Temporal Solutions Architect. ## Common anti-patterns Avoid these patterns that either inflate costs unnecessarily or create problems through aggressive optimization: ### Premature Activity consolidation Combining Activities before understanding failure modes reduces observability and retry control. Activities should be split based on failure boundaries and retry requirements, not cost optimization alone. See [How many Activities should I use in my Temporal Workflow](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) for a decision framework. ### Inappropriate use of Local Activities Using Local Activities for all operations without understanding their failure semantics and limitations. Local Activities don't provide Worker-level isolation and have different retry behavior. See [Local Activities](/local-activity) for guidance. ### Missing Continue-As-New Long-running Workflows that don't implement Continue-As-New accumulate large Event Histories, increasing storage costs and impacting performance. Workflows running days or weeks or processing thousands of events require [Continue-As-New](/workflow-execution/continue-as-new). ### High volume of Activity retries Generally, the default values for [Activity retries](/encyclopedia/retry-policies) are quite good. However, excessive Activity retries often indicate underlying issues like timeouts that are too short or Activities that frequently fail. Detect Activity retry frequency and if high, consider increasing retry intervals or Activity timeouts before failures occur. See [Spooky Stories: Chilling Temporal Anti-Patterns](https://temporal.io/blog/spooky-stories-chilling-temporal-anti-patterns-part-2#2-hiding-behind-the-chainsaws) for guidance on retry defaults and patterns. ### Large payloads in Event History Passing multi-megabyte payloads through Workflows when external storage (S3, blob storage) is more appropriate. Use [compression](/troubleshooting/blob-size-limit-error#payload-size-limit) or the [claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern) for large data. ### Over-optimization at the expense of observability Aggressively optimizing costs without maintaining sufficient visibility for debugging and operational needs. Balance cost reduction with your team's observability requirements. For example, merging five separate Activities (validate input, call payment API, update database, send notification, generate receipt) into a single "processOrder" Activity reduces from 5 Actions to 1, but you lose per-step visibility in the Temporal UI. When the notification step fails, you can't see which of the five steps failed, you lose independent retry control (a notification failure retries the entire flow including the payment call), and you can't filter Workflows by failure stage. Similarly, removing [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) from a long-running data processing Activity saves Actions, but means you can't detect a stuck Worker until the full Activity timeout expires and you lose progress tracking (for example, "processed 500 of 1,000 records"). ### Excessive Activity Heartbeats Each Heartbeat counts as one Action. Only use Heartbeats for long-running Activities (10+ minutes) where you need to detect Worker failures and track progress. Short-running Activities that complete in seconds or minutes don't need Heartbeats. See [Activity Heartbeat documentation](/encyclopedia/detecting-activity-failures#which-activities-should-heartbeat) for guidance. ## Understanding cost drivers Temporal Cloud pricing consists of Actions, Storage, and Support. If you are new to Temporal Cloud, see the [pricing documentation](/cloud/pricing#action) to learn more and familiarize yourself with [what results in a billable Action](/cloud/actions) in Temporal Cloud. ### Cost distribution For most workloads, Actions represent the majority of total costs, with storage typically accounting for 10% or less of a monthly bill. Focus optimization efforts on what's driving costs with a specific workload: **High Actions costs generally indicate**: - [Many Activities per Workflow](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) - Frequent [Signals, Queries, or Updates](/encyclopedia/workflow-message-passing#choosing-messages) - Long-running Activities with [Heartbeats](/cloud/worker-health#manage-worker-heartbeating) - High [Activity retry rates](/demos/activity-retry-simulator) - Extensive Query usage **High Storage costs generally indicate**: - Large payloads in Workflow inputs, outputs, or Activity results - Long retention periods with high Workflow volume - Long-running Workflows without Continue-As-New - Workflows accumulating large Event Histories ### Optimization priority 1. **Actions optimization**: Usually provides the largest cost reduction opportunity 2. **Active Storage optimization**: Relevant for long-running Workflows or large payloads 3. **Retained Storage optimization**: Relevant for high volume combined with long retention periods ## Measuring Establish [baseline metrics](/cloud/metrics/reference) before optimizing and be sure to validate impact after implementation. Specifically: - Actions consumption (per Workflow, per day/month, by Namespace) - Storage consumption (Active and Retained) - Monthly costs (total, per Namespace, per Workflow Type) - Observability metrics (time to debug, incident detection) ## Actions optimization [Actions](/cloud/actions) encompass Workflow operations, Activity Executions, Signals, Queries, and other interactions with Temporal. Each represents a unit of consumption. ### Activity granularity Activity granularity is a fundamental architectural decision that impacts both costs and observability. More Activities provide better visibility and retry control but increase the Action count. Fewer Activities reduce costs but limit observability. For detailed discussion of this tradeoff, see [How many Activities should I use in my Temporal Workflow?](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) ### Child Workflows vs Activities [Child Workflows cost 2 Actions](/cloud/actions#workflow) compared to an Activity's 1 Action. See [Child Workflows documentation](/child-workflows) for detailed comparison of capabilities and use cases. ### Retry Policies Each Activity retry counts as one Action. Default [Retry Policies](/encyclopedia/retry-policies) can be aggressive, which is appropriate for most operations but costly for expensive external operations. For example, consider an Activity that calls a third-party payment API. With Temporal's default Retry Policy (1s initial interval, 2.0 backoff coefficient, unlimited maximum attempts), if that API goes down for 30 minutes, the Activity retries approximately 20 times before reaching the 100s maximum interval cap, then continues retrying every 100 seconds. Each retry counts as 1 Action. Across 1,000 concurrent Workflows hitting the same outage, that produces 20,000+ extra Actions from retries alone. For expensive external operations like this, consider: - Setting `MaximumAttempts` to cap total retries - Increasing `InitialInterval` (for example, to 10s) to reduce retry frequency - Adding error types to `NonRetryableErrorTypes` for errors that won't resolve on retry (such as 4xx HTTP status codes) - Using [next retry delay](/encyclopedia/retry-policies#per-error-next-retry-delay) to dynamically control retry timing based on failure types (for example, respecting rate-limit headers) - Implementing an [Activity pause pattern](/cli/command-reference/activity#pause) to wait for manual intervention rather than automatic retries Use the [Activity Retry Simulator](/demos/activity-retry-simulator) to visualize how different Retry Policy configurations affect retry behavior and Action consumption. Refer to this blog post on [Mastering Workflow retry logic for resilient applications](https://temporal.io/blog/failure-handling-in-practice) for additional guidance. ### Local Activities A [Local Activity](/local-activity#local-activity) is an Activity Execution that executes in the same process as the Workflow Execution that spawns it. Therefore, multiple Local Activities that run back-to-back only [count as a single billable action](/cloud/actions#activity), whereas each regular Activity counts as a billable action. However, there are tradeoffs to converting regular Activities to Local Activities. For example, if a specific Local Activity fails, *all* of them will be retried together. Review [the docs](/local-activity) or reach out to your account team to learn more. #### When to stick with Regular Activities Use Regular Activities instead of Local Activities if you require any of the following: - Activities may take more than 10 seconds to complete - Independent retry control for each Activity - Need to avoid re-running expensive Activities when unrelated Activities fail - Immediate Signal/Update handling during execution - Separate resource management (like rate limits) for each Activity ### Batching operations #### Search Attributes 1. [Search Attributes](/search-attribute#custom-search-attribute) provided at Workflow start do not count as billable Actions. If Search Attribute values are known before starting the Workflow, provide them at Workflow start to eliminate these costs entirely. 2. For Search Attributes that must be updated during Workflow Execution, each `UpsertSearchAttributes` call counts as 1 Action regardless of how many attributes are updated. Batch multiple related attribute updates into single operations to reduce Actions consumed. See the [Temporal Cloud Action Documentation](/cloud/actions#workflow) for details. #### Signal handling Where feasible, implement deduplication logic client-side or aggregate data into fewer Signals. Use `SignalWithStart` instead of separate `StartWorkflow` and `SignalWorkflow` calls when initiating Workflows with Signals. ## Storage optimization [Storage costs](/cloud/pricing#storage) are divided into Active Storage (open Workflows) and Retained Storage (closed Workflow History during retention period). Active Storage is significantly more expensive than Retained Storage. ### Active Storage Active Storage applies to open Workflows and their Event Histories. The following sections detail optimization opportunities for Active Storage. #### Continue-As-New For long-running Workflows with extended sleep/wait periods, calling Continue-As-New before sleeping closes the current execution (moving to cheaper Retained Storage) and starts fresh when work resumes, reducing Active Storage costs. See [Continue-As-New documentation](/workflow-execution/continue-as-new) to learn more. #### Compression Large payloads increase Active Storage costs. Implement a custom Data Converter with compression for moderately large payloads (100KB-1MB). See the [Data Converter documentation](/default-custom-data-converters#custom-data-converter) to learn more. #### Claim check pattern For very large payloads or binary data, store data externally (S3 or GCS) and pass references through Workflows. ### Retained Storage Retained Storage applies to closed Workflow History during the retention period. The following sections detail optimization opportunities for Retained Storage. #### Retention periods The default Namespace retention is 30 days (configurable between 1 and 90 days). Adjust based on operational and compliance requirements. **Considerations**: - Shorter retention reduces costs but limits historical analysis - Audit investigation patterns before shortening retention - Ensure compliance requirements are met See [Namespace retention documentation](/temporal-service/temporal-server#retention-period) for configuration details. #### Workflow export Temporal Cloud supports exporting Event Histories to external storage for compliance while maintaining shorter retention periods. Note that Workflow export costs one Action per export. See the [Workflow History export documentation](/cloud/export) for more details. Alternatively, if you are looking to do analysis on closed Workflow Executions, [review this blog post to learn how to gain insights from exported Event Histories](https://temporal.io/blog/get-insights-from-workflow-histories-export-on-temporal-cloud). ## Validation ### Validation approach 1. **Test in non-production**: Validate functional correctness before production deployment 2. **Monitor comprehensively**: Leverage the [Usage dashboard](/cloud/actions-usage#usage) in the Cloud UI to track the impact on Actions and Storage after optimizations are made 3. **Progressive rollout**: Deploy to a small percentage, validate, then expand. Review the [Worker Versioning documentation](/production-deployment/worker-deployments/worker-versioning) to learn about rolling out changes to Workflows 4. **Continuous review**: Re-evaluate optimization effectiveness quarterly as system evolves ### Success criteria - Cost reduced without increasing mean time to repair (MTTR) - Workflow success rates maintained or improved - Reduced observability does not increase mean time to detect (MTTD) incidents ### Tools - Temporal Cloud Usage dashboard for Actions and Storage metrics - Event History for per-Workflow billable Actions estimates - Export metrics to observability platforms such as Datadog or Grafana for custom monitoring ## When to get help Engage the Temporal team for Workflow audits when experiencing: - Complex Workflow patterns with unclear optimization paths - Compliance requirements limiting optimization options - Need for custom DataConverters or advanced patterns - Desire for expert validation of optimization strategies Contact your Temporal Account Representative or [Temporal support](http://support.temporal.io) to discuss optimization services. --- # Error handling Source: https://docs.temporal.io/best-practices/error-handling Temporal automatically retries failed Activities and recovers from infrastructure failures through [Durable Execution](/evaluate/why-temporal). But not all failures should be retried. This page covers how to categorize failures, when to mark errors as non-retryable, and how to handle failures that retries cannot resolve. For background on how Temporal represents and propagates failures, see [Application failures](/encyclopedia/application-failures). ## Categorize failures When an operation fails, the appropriate response depends on the nature of the failure. Failures fall into three categories based on whether retrying can resolve them. ### Transient failures A transient failure is a one-off event that resolves on its own without intervention. For example, a Worker happens to make a network request at the exact moment an administrator replaces a network cable. The cause is unlikely to affect future requests. Transient failures are resolved by retrying the operation shortly after the failure. Temporal's default [Retry Policy](/encyclopedia/retry-policies) handles transient failures automatically. ### Intermittent failures An intermittent failure is one that recurs but resolves over time. For example, a service that uses rate limiting will reject requests once the threshold is reached, but will accept requests again after the rate limiter resets. Intermittent failures require retries spaced out over a longer period. Configure your [Retry Policy](/encyclopedia/retry-policies) with an appropriate `backoffCoefficient` and `maximumInterval` to avoid overwhelming the failing service. ### Permanent failures A permanent failure is one that will recur indefinitely until the cause is fixed. For example, a request that fails due to an invalid email address will continue to fail no matter how many times the operation retries. The only resolution is to correct the email address. Permanent failures cannot be resolved through retries. They require different input data, a code fix, or some external intervention. Mark these errors as [non-retryable](#non-retryable-errors) to fail fast instead of consuming resources on retries that will not succeed. ## Mark permanent errors as non-retryable When your code detects a permanent failure, mark the error as non-retryable to prevent unnecessary retry attempts. For background on what Application Failures are and how the `non_retryable` flag works, see [Application Failure](/encyclopedia/application-failures#failure-representation). Use non-retryable errors for situations like: - **Invalid input data**: A malformed email address, a negative payment amount, or a missing required field. - **Business rule violations**: A customer outside the service area, an order exceeding credit limits, or an expired promotion code. - **Authorization failures**: The caller does not have permission to perform the operation. - **Data validation errors**: A referenced record does not exist, or data fails integrity checks. There are two ways to mark errors as non-retryable: **In the Activity (implementer decides):** Set the `non_retryable` flag when throwing an [Application Failure](/encyclopedia/application-failures#failure-representation). This enforces the constraint for all callers. Use this when the Activity implementer knows that the error can never be resolved through retries. **In the Retry Policy (caller decides):** Add the error type to the Retry Policy's list of [non-retryable error types](/encyclopedia/retry-policies#non-retryable-errors). This lets different Workflows make different decisions about the same Activity. Use this when the decision depends on the caller's business logic. ### Preserve retryability when wrapping errors When an Activity returns an error, the SDK checks the **outermost** error type to determine retryability. If you catch a non-retryable Application Failure and re-throw it wrapped in a generic language error, the `non_retryable` flag is lost and the Activity will be retried. To add context to an error while preserving its retry behavior, wrap it in another Application Failure with the same `non_retryable` flag. Do not wrap Application Failures in generic language errors. For a detailed explanation of how the SDK-to-server chain works, see [The outermost error type determines retryability](/encyclopedia/application-failures#outermost-error-type). ### Use non-retryable errors sparingly In most cases, let the Retry Policy handle retry limits through [timeouts](/encyclopedia/detecting-activity-failures) and maximum attempts. Reserve `non_retryable` for cases where retrying is guaranteed to be futile. For SDK-specific syntax and code examples, see the error handling guide for your language: - [Python](/develop/python/best-practices/error-handling) - [Go](/develop/go/best-practices/error-handling) - [.NET](/develop/dotnet/best-practices/error-handling) - [Ruby](/develop/ruby/best-practices/error-handling) ## Design Activities for idempotence Activities may execute more than once due to retries, so design them to be [idempotent](/activity-definition#idempotency): producing the same result whether executed once or multiple times. This is especially important because of an edge case in distributed systems. A Worker can execute an Activity, complete it, and then crash before reporting the result to the Temporal Service. The Activity is retried even though it completed, because the Service has no record of the completion. Use idempotency keys to prevent duplicate operations. Combine the Workflow Run ID and Activity ID for a value that is consistent across retries but unique across Workflow Executions. ## Implement compensation with the Saga pattern When a multi-step process fails partway through, previous steps may need to be undone. The [Saga pattern](/design-patterns/saga-pattern) coordinates a sequence of operations where each step has a compensating action that reverses its effects. If any step fails, the compensating actions for previously completed steps execute in reverse order. For SDK-specific implementations with working code examples, see: - [Python Saga pattern](/develop/python/best-practices/error-handling#implement-saga-pattern) --- # Knowledge Hub Source: https://docs.temporal.io/best-practices/knowledge-hub As organizations scale their Temporal adoption, the use cases become more complex, and it can be difficult to locate all the relevant information you need. Tribal knowledge gets siloed within teams, leading to inconsistent patterns for Workflow design, error handling, and testing. This fragmentation leads to an increased burden on support, as well as slower onboarding for new developers. You may also inadvertently introduce security vulnerabilities or compliance gaps. To prevent these issues, you can establish an internal Temporal knowledge hub to address common issues that arise when multiple teams adopt Temporal independently. This guide covers what belongs in a knowledge hub, how to communicate it, and how to keep it useful over time. To bootstrap your knowledge hub, use the [Temporal Platform Hub](https://go.temporal.io/platform-hub) template as a starting point. ## What belongs in your knowledge hub Although Temporal itself has [thorough documentation](/), not all of it applies to your organization or your teams' use cases. The knowledge hub distills the documentation into just the specific information your teams need. One way to organize the content is according to where developers are in their journey. The following sample outline shows what sections to include. ### Evaluate The Evaluation section helps developers understand what Temporal is and whether it fits their problem. Goals for this section are to increase knowledge hub traffic and decrease the support question rate. Include the following items in this section: - **Temporal overview** explains what Temporal is, why your organization chose it, and the business value metrics that justify adoption. - **Decision framework** provides qualifying questions, good and bad use cases, and alternative recommendations so developers can determine whether Temporal fits their problem. ### Build The Build section is aimed at developers who are just getting started with Temporal. The goal of this section is to give developers a single path from zero to a running Workflow. Include the following items in this section: - **Getting started** walks developers through a 30-minute quickstart covering environment setup, a starter template, and running a first Workflow locally and on Temporal Cloud. Use an existing Temporal [quickstart](/quickstarts), or build one from your own use case. - **Learning paths** provides self-paced courses from foundational to advanced topics, tailored by persona, and links to [Temporal's free training](https://learn.temporal.io/). ### Ship The Ship section provides the architecture standards and guardrails developers need to go from a local prototype to a production deployment. The goal for this section is to provide reference material developers need to reduce their time to production. Include the following items in this section: - **Architecture and standards** documents Namespace conventions, connectivity requirements, and Worker deployment standards that every team follows. - **Cost guidance** explains billable Actions, storage tiers, and cost-saving tips so developers build cost-efficient Workflows. - **Shared responsibility** defines an ownership matrix between Platform and Application teams across IAM, infrastructure, development, deployment, observability, and operations. - **Design patterns** curates Workflow patterns with descriptions and code sample links that developers can adopt directly. ### Operate The Operate section gives developers the tools to self-serve during incidents and find answers without escalating to the Platform team. The goal of this section is to decrease the number of questions to support. Include the following items in this section: - **Troubleshooting and escalation** covers observability tools, runbooks for common issues, escalation paths, SLAs, and example alert definitions. - **Support and FAQs** documents your support tier, ticket submission process, Temporal account contacts, expert-led session types, and frequently asked questions. ## Measuring success of your knowledge hub After you've created your knowledge hub, establish metrics to measure its effectiveness for your organization. The following table shows example indicators that organizations use to measure the impact of their knowledge hub, along with realistic before-and-after targets: | Metric | Before Knowledge Hub | Target | What it tells you | | :--- | :--- | :--- | :--- | | **Time to first Workflow** | Days to weeks (developers piece together scattered resources) | Under 30 minutes (developers follow a single getting started guide) | Measures onboarding friction. A short time to first Workflow signals that your getting started guide is effective. | | **Time to Workflow in production** | Weeks to months (blocked by unclear Namespace provisioning and deployment processes) | Under 2 weeks (developers follow documented self-service provisioning) | Measures the gap between development and delivering value. Long times point to missing documentation and automation opportunities. | | **Support question rate** | 20-30+ questions per week to the Platform team via Slack | Fewer than 5 per week | Measures self-service resolution. A declining trend shows that developers are finding answers in the knowledge hub instead of asking the Platform team. | | **Knowledge Hub traffic** | N/A (no centralized resource exists) | Steady or growing page views per month | Identifies which content developers rely on and where gaps remain. Declining traffic on a page may indicate it is outdated; high traffic with high bounce rates may indicate the page is not answering the question. | These metrics create a feedback loop: measure, identify gaps, improve content, and measure again. ## What doesn't belong in your knowledge hub A knowledge hub is not a mirror of [Temporal's official documentation](/). Avoid duplicating SDK API references, concept explanations, or release notes that Temporal already maintains. When that content changes, your copy becomes a source of confusion rather than clarity. Instead, link to the official docs and reserve your knowledge hub for organization-specific decisions, conventions, and operational procedures that Temporal's public documentation does not cover. ## How to maintain and communicate your knowledge hub Having a thorough, complete knowledge hub isn't useful if the information becomes stale, or if developers don't know it exists. ### Assign ownership Designate a Platform team or developer experience team as the owner. This team is responsible for initial content creation, ongoing maintenance, and reviewing contributions from application teams. ### Make it discoverable A knowledge hub that developers can not find is the same as not having one. - Register a short URL (for example, `go/temporal`) that redirects to the knowledge hub. - Pin the link in your Temporal-related communication channels (Slack, Microsoft Teams). - When answering questions in Slack, respond with a link to the relevant knowledge hub page instead of re-explaining inline. This builds the habit of checking the hub first. ### Review metrics regularly Track the metrics from [Measuring success of your knowledge hub](#measuring-success-of-your-knowledge-hub) on a regular cadence to identify what is working and where gaps remain. Capture every question that reaches the Platform team through Slack or tickets as a candidate for new content. Solicit contributions from application teams through a lightweight process such as a pull request template. ### Keep content current - **Review on a cadence**: Review each page at least quarterly. Assign a review owner and date to each page so staleness is visible. - **Tie updates to events**: Update the knowledge hub whenever your organization changes its Temporal architecture, updates its deployment tooling, or modifies its shared responsibility model. - **Prune aggressively**: Remove or archive content that no longer applies. Outdated documentation is worse than no documentation because developers follow it and get unexpected results. ## Get started Start with the [Temporal Platform Hub template](https://go.temporal.io/platform-hub) as your foundation. --- # Managing Actions per Second (APS) limits in Temporal Cloud Source: https://docs.temporal.io/best-practices/managing-aps-limits > Control how limits are assigned to a Namespace with Capacity Modes Every Namespace on Temporal Cloud has an Actions Per Second (APS) limit. An action is any operation that modifies Workflow state or interacts with the Temporal Service — starting or completing a Workflow, executing an Activity, sending a Signal. When a Namespace exceeds its APS limit, Temporal throttles requests. Depending on the business priority of the Workflow, that may be fine, or it may have significant impact. APS consumption isn't always intuitive: a single Workflow Execution generates multiple actions from the moment it starts, so use cases that fit comfortably within APS limits at small scale can exhaust them as they grow. This guide covers why workloads hit APS limits, how to design Workflows that use actions efficiently, and when to use [Provisioned Capacity](#provisioned-capacity-and-trus) — Temporal Resource Units (TRUs) that reserve additional capacity for spiky or unpredictable workloads. ## Understanding actions in Temporal ### What counts as an action? Actions are the fundamental operations that drive your Workflows forward. The following is a partial list — see [the full list in our documentation](/cloud/actions). - Workflows: Starting, completing, resetting. Also starting Child Workflows, as well as Schedules and Timers - Activities: Starting, retrying, Heartbeating - Signals, Updates, and Queries Actions that count toward an APS limit are, with a few exemptions, the same as actions that are billable. The key insight here is that nearly everything that happens in Temporal--state changes, decision points, interactions--is counted as an action. ### The action multiplier effect What this means is that when you start a single Workflow, you're not performing just one action as it relates to APS because a Workflow isn’t a single atomic operation, it’s a series of events that Temporal orchestrates. Each Activity at the start of the Workflow is an Action, so there can be a burst of Activities at the start of a Workflow. Additionally, there are often business reasons to start multiple Workflows at the same time. These can all contribute to the multiplier effect. ### The effect of rate limiting In Temporal Cloud, the effect of rate limiting is increased latency, not lost work. Workers [might take longer](/cloud/service-availability#throughput) to complete Workflows. ## Common reasons customers hit APS limits The following patterns most commonly push customers into APS constraints. ### Bursty traffic Most businesses don't operate at constant velocity—they have rhythms, cycles, and spikes. These patterns can create APS challenges because Temporal Cloud enforces limits at the per-second level. Common bursty patterns include: - Calendar-driven spikes: Month-end financial close processes, quarterly reporting Workflows, payroll that runs on the 1st and 15th, scheduled batch jobs that kick off at midnight. These create predictable but intense load concentrations. - Event-driven surges: Product launches, marketing campaigns, flash sales, breaking news, or seasonal events like Black Friday. - Recovery scenarios: When a downstream dependency fails and then recovers, you often get a thundering herd effect—hundreds or thousands of Workflows that were waiting all suddenly resume execution simultaneously, creating an artificial spike in APS consumption. - Geographic/business hours concentration: Global applications often see load follow the sun, with peak activity during business hours in each region. If your business concentrates in specific markets, you'll see daily peaks rather than even 24/7 distribution. - Retry Storms: when a large number of Workflows get stuck on an Activity, and that Activity is failing, if retry delay is very short, this can cause a spike in Actions. - Timer Storms: a large number of Workflows all set a Timer for the exact same time--triggering a spike as those Timers fire and then Activities run, causing a lot of actions all at the same time. These types of processes can result in your Namespace averaging 200 APS over a day, but spiking to 800 APS or more during your peak hour/day/event/etc. #### How to mitigate You can’t change the patterns of how customers interact with your systems, but there are some adjustments you can make to your Workflows to make traffic patterns more consistent, especially for use cases where immediate response isn’t necessary. These adjustments include: - Implement application-level queuing or rate limiting to smooth out predictable spikes. - For scheduled batch operations, stagger start times rather than triggering everything at once--implement jitter in your high-volume [Schedules](/schedule#spec). - Implement jitter when starting Workflows, such as with [Start Delay](/workflow-execution/timers-delays#delay-workflow-execution). - Accept rate limiting - [Provisioned Capacity](/cloud/capacity-modes#provisioned-capacity) ### Cascading Workflows and fan-out patterns Decomposing complex processes into parent and Child Workflows (or with Nexus) is a common and often appropriate pattern, but the APS costs multiply dramatically with depth and fan-out. Consider an order fulfillment Workflow that spawns Child Workflows for payment processing, inventory management, shipping, and customer notifications. Each Child Workflow goes through its full action lifecycle (start, tasks, activities, completion), and all of those actions count toward the APS limits on your Namespace. This pattern appears frequently in: - Batch processing: A parent workflow processes a file with 1,000 records, spawning a Child Workflow for each record. Batch processing is also often bursty whenever the batch begins. - Map-reduce patterns: Data processing Workflows that fan out to process partitions in parallel, then aggregate results. This challenge additionally compounds when you have multiple levels of nesting--parent Workflows that create children, which create their own children. #### How to mitigate - Evaluate whether Child Workflows are necessary--other options include Activities or Workflows in another Namespace (via Nexus) - When you do use Child Workflows, limit fan-out size--design a Child Workflow to process its work in batches rather than one Child per work item. [This sample application](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow) shows more detail. - Consider flattening deeply nested hierarchies into shallower structures. ### Human-in-the-loop processes at scale Workflows that incorporate human decision-making--approvals, reviews, manual data entry, quality checks--tend to be long-running and interaction-intensive, which creates sustained APS load. These Workflows can involve Queries from UIs to display current state and pending tasks. At small scale, this is manageable. But when you're running thousands of them at the same time--like a content moderation queue with pending reviews, or a loan approval system processing applications, or a support ticket system managing thousands of open cases--the cumulative APS load from all of those long-running Workflows adds up. #### How to mitigate - Avoid polling patterns where UIs constantly query Workflow state. Instead, push state changes to a database that UIs can read. ### Real-time SLAs and deadline management Businesses with strict service level agreements often implement active monitoring and escalation in their Workflows. This is generally accomplished by setting Timers every [x] minutes to determine if an SLA deadline is approaching, allowing the Workflow to trigger escalations or alerts. Each of these Timers/monitoring actions affect APS. When you have thousands of in-flight Workflows all actively monitoring their own SLAs, the background load becomes significant. You're consuming substantial APS capacity even when Workflows aren't doing their primary work. #### How to mitigate - Use longer monitoring intervals where possible. For example, check SLAs every 30 minutes rather than every 1 minute. - Where possible, consolidate Timers. Rather than 10 Timers that check 10 tasks, have 1 Timer and then check those 10 tasks. - Where possible, have an external system signal your Workflow rather than using short-lived Timers to poll. - For retries, use exponential backoff with reasonable initial intervals. ## Additional design patterns There are some design patterns that can lead to high APS that are consistent across many different types of business use cases. ### Many small Activities Consider two approaches to processing 1,000 records: - Approach A: Create a Workflow that spawns 1,000 separate activities, one per record. - Approach B: Create a Workflow that spawns 10 activities, each processing 100 records in a batch. Approach B will clearly result in less APS. This is a simple example, but this pattern shows up everywhere: processing individual transactions versus batches, sending individual notifications versus bulk operations, or making separate API calls versus batch endpoints. Each separate Activity adds Action overhead. #### How to mitigate - Consider if you can combine multiple external calls within a single Activity. - If processing a large amount of data, process it in chunks. - See [How Many Activities should I use in my Temporal Workflow?](https://temporal.io/blog/how-many-activities-should-i-use-in-my-temporal-workflow) for more information. ### Multiple use cases in one Namespace Often when starting with Temporal, the first use case is implemented in a single Namespace, generally one per logical environment. When the second Temporal use case is implemented, it runs in the same Namespace, the same for the third, fourth, etc. An APS limit is set per Namespace, so multiple use cases with multiple traffic patterns in the same Namespace can exhaust this limit quickly. #### How to mitigate Plan for a set of Namespaces (one per environment) per use case. This gives each use case its own APS envelope and reduces the blast radius when one workload spikes or misbehaves. If you are deciding whether to split by use case, service, or domain, see [Managing a Namespace](/best-practices/managing-namespace) for a topology decision framework. ## Provisioned capacity and TRUs The strategies above help you design Workflows that use actions efficiently. But sometimes you need more capacity than the on-demand model provides, especially for spiky or unpredictable workloads. Temporal Cloud offers two [Capacity Modes](/cloud/capacity-modes): - **On-Demand mode** (default): Your Namespace automatically scales based on your trailing 7-day usage. This works well for steady, predictable workloads. - **Provisioned mode**: You reserve capacity by adding Temporal Resource Units (TRUs), giving you guaranteed headroom for traffic spikes. See [Capacity Modes](/cloud/capacity-modes) for complete details on TRUs, available increments, and how to manage capacity via UI, CLI, or API. ### Choosing the right approach Use Provisioned capacity when the on-demand model can't respond quickly enough: | Scenario | Pattern | Recommendation | |----------|---------|----------------| | **Planned spikes** | Promotions, holiday traffic, product launches | Pre-provision TRUs before the event starts | | **Unplanned spikes** | Sudden traffic surges, viral events | React instantly via UI/CLI/API when you see throttling | | **Load testing** | Validating new services at scale | Provision TRUs for the test, deprovision after | | **Batch jobs** | Scheduled high-throughput jobs | Automate TRU scaling via API around job schedules | | **Migrations** | Onboarding a new workload faster than on-demand adjusts | Bridge with TRUs for approximately 7 days while the on-demand envelope catches up | > **📝 Note:** > When switching back to on-demand mode, your APS limit resets to the running average from the last 7 days. > If Temporal Support has set a custom limit for your Namespace, that limit is preserved across the switch. > Plan for this if your workload is sensitive to the transition. ### Cost optimization tips See [Capacity Modes Pricing](/cloud/pricing#capacity-modes-pricing) for billing details. To minimize costs: - Provision only when you need extra capacity - Deprovision promptly after spikes end - For predictable patterns, automate scaling to minimize time in provisioned mode ### Automation best practices Since you understand your workload patterns better than any auto-scaling system, consider building your own TRU automation: - **Use the [Cloud Ops API](/ops), [Terraform Provider](/cloud/terraform-provider), or [tcld CLI](/cloud/tcld)** to programmatically scale capacity based on your application's signals - **Set utilization thresholds**: For example, scale up when hitting 70-80% of your limit, scale down after sustained low usage - **Schedule capacity changes**: Use [Temporal Schedules](/schedule) or Workflows to increase TRUs before known events - **React to leading indicators**: If your application has upstream signals (incoming order queue depth, marketing campaign start), use those to trigger capacity changes proactively ## Knowing if you're hitting APS limits In addition to understanding the patterns that can affect APS limits on a Temporal Namespace, it's also important to know if you're approaching (or exceeding) these limits. Temporal Cloud provides several metrics that, if tracked, will tell you if you're being rate limited due to APS. See the documentation on [detecting resource exhaustion](/cloud/service-health#rps-aps-rate-limits) for an explanation of those metrics as well as a sample Grafana dashboard that shows how they could be viewed. ### Monitoring for TRU decisions If you're considering Provisioned capacity, set up monitoring to understand your usage patterns: - **Use [OpenMetrics](/cloud/metrics/openmetrics)**: For real-time visibility into APS consumption, integrate Temporal Cloud metrics with your observability stack - **Track APS usage vs. limits**: Monitor [`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) to detect throttling events - **Set alerts at 70-80% utilization**: This gives you time to provision TRUs before hitting limits - **Analyze historical patterns**: Understanding your traffic patterns helps you decide between reactive TRU provisioning and proactive automation ## Key takeaways The following table recaps the main reasons customers hit APS limits and how to address them: | Reason for Hitting APS Limits | How to Address It | |-------------------------------|-------------------| | Bursty Traffic | Implement application-level queuing or rate limiting to smooth spike, stagger start times for scheduled batch operations. | | Cascading Workflows
and Fan-Out Patterns | Evaluate if Child Workflows are necessary (consider activities or another Namespace), limit fan-out size by processing work in batches within a Child Workflow, consider flattening deeply nested hierarchies. | | Human-in-the-Loop
Processes at Scale | Design long-running Workflows to minimize sustained APS load from interaction (by avoiding polling where UIs constantly Query state and using Signals only for key human inputs). | | Many small activities | Consider if you can combine multiple external calls within a single Activity. If processing a large amount of data, process it in chunks. | | Multiple use cases
in one Namespace | Plan for a set of Namespaces (one per environment) per use case. | | Planned traffic spikes | Pre-provision TRUs before the event, then deprovision after. | | Unpredictable spikes
requiring instant response | Switch to Provisioned mode for self-service capacity scaling via UI, CLI, or API. | | Load testing at scale | Provision TRUs for the test duration, deprovision when complete. | | New workload onboarding | Bridge with TRUs while the on-demand envelope adjusts (approximately 7 days). | ## General guidance When designing Temporal Workflows with an eye toward APS limits, ask yourself the following questions: - How many actions will a single execution of this Workflow consume? - How many Workflows will typically be running at the same time? - What happens to APS consumption when the number of Actions * number of active Workflows scales to 100x current volume? - Are there natural opportunities to combine operations: combine activities, or process chunks of data together? - Am I polling when I could be using Signals? - Does this Workflow need to run continuously, or can it be event-driven? A few hours spent optimizing Workflow design can save you from capacity crunches, emergency limit increases, and potentially significant cost increases down the road. --- # Namespace best practices Source: https://docs.temporal.io/best-practices/managing-namespace > Best practices for organizing and managing Temporal Namespaces, including naming conventions, organizational patterns, and production safeguards. > **ℹ️ Info:** > Applies to both open source and Temporal Cloud > This page covers namespace best practices that apply to **both** open source Temporal and Temporal Cloud. > Platform-specific guidance is clearly labeled throughout. > > For reference documentation, see: > - [Namespace concepts](/namespaces) > - [Managing Namespaces (open source)](/self-hosted-guide/namespaces) > - [Namespaces (Temporal Cloud)](/cloud/namespaces) A [Namespace](/namespaces) is a unit of isolation within the Temporal Platform. It ensures that Workflow Executions, Task Queues, and resources are logically separated, preventing conflicts and enabling safe multi-tenant usage. Use this page to decide how many Namespaces you need and where to draw boundaries between environments, services, domains, and tenants. For Cloud-specific namespace mechanics such as creation, tagging, and gRPC endpoints, see [Namespaces (Temporal Cloud)](/cloud/namespaces). Related guidance: - [Managing Temporal Cloud access control](/best-practices/cloud-access-control) - [Multi-tenant application patterns](/best-practices/multi-tenant-patterns) - [Managing Actions per Second (APS) limits in Temporal Cloud](/best-practices/managing-aps-limits) ## Naming conventions ### Use lowercase and hyphens Use lowercase letters and hyphens (`-`) as separators in Namespace names. - **Temporal Cloud**: Namespace names are case-insensitive, so `MyNamespace` and `mynamespace` refer to the same Namespace. - **Open source**: Namespace names are case-sensitive, so `MyNamespace` and `mynamespace` are different Namespaces. To avoid confusion across environments, always use lowercase. **Example**: `payment-checkout-prd` ### Follow a consistent naming pattern Use a pattern like `--` to name Namespaces: | Component | Max Length | Examples | |-----------|------------|----------| | Use case | 10 chars | `payments`, `fulfill`, `orders` | | Domain | 10 chars | `checkout`, `notify`, `inventory` | | Environment | 3 chars | `dev`, `stg`, `prd` | **Examples**: `payments-checkout-dev`, `fulfill-notify-prd`, `orders-inventory-stg` **Why this pattern?** - Simple and easy to understand - Clearly separates environments - Groups related services under domains - Allows platform teams to implement chargeback to application teams - Namespace-level limits are isolated between different services and environments > **💡 Tip:** > Temporal Cloud > Cloud Namespace names are limited to [39 characters](/cloud/namespaces#temporal-cloud-namespace-name). > If you need to include region, use short codes (for example, `aps1`, `use1`). ## Organizational patterns ### Choose your Namespace boundary intentionally Start with the smallest number of Namespaces that gives you clear ownership and safe isolation. In Temporal Cloud, a Namespace boundary affects: - [APS limits](/cloud/limits#actions-per-second) and rate limiting - access control and credential scope - blast radius for misconfigured or overloaded Workers - observability boundaries for dashboards and alerts - operational overhead for provisioning, tagging, and lifecycle management Use the following decision table as a starting point: | If you need... | Prefer... | Why | |---|---|---| | Basic environment isolation for a single application or use case | Namespace per use case and environment | This is the simplest pattern and works well for most initial deployments | | Separate operational ownership for services within the same use case | Namespace per use case, service, and environment | This isolates credentials, limits, and operational changes per service | | Stronger boundaries across teams, domains, or business capabilities | Namespace per use case, domain, and environment | This reduces blast radius and lets teams own their own Namespace contracts | | Tenant-specific credentials, rate limits, or compliance boundaries | Namespace per tenant | Use this only for a small number of high-value tenants because of the operational overhead | As a default, start with one Namespace per use case and environment. Split later when APS pressure, security requirements, ownership boundaries, or troubleshooting needs justify the extra operational cost. ### Pattern 1: Namespace per use case and environment For simple configurations without multiple services or team boundaries. **Naming convention**: `-` **Example**: `payments-prd`, `orders-dev` Choose this pattern when: - one team owns the use case - environments need clean separation - workload volume and criticality do not yet require further isolation ### Pattern 2: Namespace per use case, service, and environment When multiple services that are part of the same use case communicate externally to Temporal via API (HTTP/gRPC). **Naming convention**: `--` **Example**: `payments-gateway-prd`, `payments-processor-prd` Choose this pattern when: - services need separate credentials or access policies - one service can exhaust APS or operational limits independently of the others - teams want separate ownership of deployment, alerting, or on-call boundaries ### Pattern 3: Namespace per use case, domain, and environment When multiple services need to communicate with each other, use [Temporal Nexus](/nexus) to connect Workflows across Namespace boundaries. This provides better security, fault isolation, and modularity than sharing a Namespace. **Naming convention**: `--` **Example**: `payments-checkout-prd`, `payments-refunds-prd` Choose this pattern when: - multiple teams or domains need independent release cadence and ownership - failures in one domain should not affect the others - you want a stronger permission boundary between capabilities - you plan to expose cross-Namespace contracts through Nexus For systems without Nexus, services can communicate via [Signals](/sending-messages#sending-signals) or [Child Workflows](/child-workflows) within the same Namespace. > **📝 Note:** > Workflow ID uniqueness > When multiple teams share a Namespace, prefix each Workflow ID with a service-specific string to ensure uniqueness. > Task Queue names must also be unique within the Namespace. ### Pattern 4: Namespace per tenant Use a separate [Namespace](/namespaces) per tenant only when each tenant needs a true isolation boundary. This is usually appropriate only for a small number of high-value tenants that require: - dedicated credentials and access control - tenant-specific rate limits or capacity decisions - strict compliance or data-isolation boundaries - independent debugging, alerting, and operational ownership For most SaaS use cases, a shared Namespace with per-tenant [Task Queues](/task-queue) is simpler and more scalable. See [Multi-tenant application patterns](/best-practices/multi-tenant-patterns) for those designs. ### What should cause you to split a Namespace later? Revisit your topology when one or more of the following becomes true: - one workload is consuming enough APS that it regularly threatens others in the same Namespace - one team needs tighter access controls or dedicated credentials - production troubleshooting requires clearer dashboards, alerts, or ownership boundaries - one application or domain is business-critical enough that its blast radius must be reduced - a tenant or regulated workload needs stronger separation than Task Queue isolation can provide Splitting a Namespace increases safety, but it also adds overhead for provisioning, tagging, credentials, and cross-Namespace coordination. Use [Nexus](/nexus) where possible instead of sharing Temporal primitives across team or domain boundaries. ## Production safeguards ### Use an Authorizer (open source only) Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to set restrictions on who can create, update, or deprecate Namespaces. If an Authorizer is not set, Temporal uses the `nopAuthority` authorizer that unconditionally allows all API calls. On Temporal Cloud, [role-based access controls](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) provide namespace-level authorization without custom configuration. ### Enable deletion protection (Temporal Cloud only) [Enable deletion protection](/cloud/namespaces#delete-protection) for production Namespaces to prevent accidental deletion. ### Enable High Availability (Temporal Cloud only) For business-critical use cases with strict uptime requirements, enable [High Availability features](/cloud/high-availability) for a [99.99% contractual SLA](/cloud/high-availability#high-availability-features). ### Use Infrastructure as Code (Temporal Cloud only) Use the [Temporal Cloud Terraform provider](/cloud/terraform-provider) to manage Namespaces. If Terraform isn't suitable, scripting against the [Cloud Ops API](/ops) or [tcld](/cloud/tcld) is a good alternative. This provides: - Documentation of each Namespace's purpose and owners - Prevention of infrastructure drift - Version-controlled configuration changes Use `prevent_destroy = true` in your Terraform configuration to prevent accidental Namespace deletion via Terraform. This is separate from [Temporal Cloud deletion protection](/cloud/namespaces#delete-protection), which prevents deletion through any interface. **Reference**: [Example Terraform configuration](https://github.com/kawofong/temporal-terraform) ## Tagging (Temporal Cloud only) [Tags](/cloud/namespaces#tag-a-namespace) are key-value metadata pairs that help organize, track, and manage Namespaces. Tags complement your naming convention by adding metadata that doesn't fit in the Namespace name. While the name captures use case, domain, and environment, tags can capture additional dimensions like team ownership, data sensitivity, or business criticality. ### Recommended tag categories | Tag Key | Purpose | Examples | |---------|---------|----------| | `environment` | Deployment stage | `dev`, `staging`, `production` | | `team` | Owning team | `platform`, `payments`, `identity` | | `division` | Business unit | `engineering`, `finance`, `ops` | | `criticality` | Business importance | `high`, `medium`, `low` | | `data-sensitivity` | Data classification | `pii`, `pci`, `public` | | `latency-sensitivity` | Performance tier | `realtime`, `batch`, `async` | For tag structure, limits, and management instructions, see [How to tag a Namespace](/cloud/namespaces#tag-a-namespace). ## SDK client configuration Set Namespaces in your SDK Client to isolate your Workflow Executions. If you do not set a Namespace, all Workflow Executions started using the Client will be associated with the `default` Namespace. You must register a Namespace before setting it in your Client. For configuration details, see: - [Namespace concepts](/namespaces) - [Namespaces (Temporal Cloud)](/cloud/namespaces#access-namespaces) --- # Multi-tenant application patterns Source: https://docs.temporal.io/best-practices/multi-tenant-patterns > Learn how to build multi-tenant applications using Temporal with task queue isolation patterns, worker design, and best practices. Many SaaS providers and large enterprise platform teams use a single Temporal [Namespace](/namespaces) with [per-tenant Task Queues](#1-task-queues-per-tenant-recommended) or [Task Queue Fairness](#2-single-task-queue-with-fairness) to power their multi-tenant applications. These approaches maximize resource efficiency while maintaining logical separation between tenants. This guide covers architectural patterns, design considerations, and practical examples for building multi-tenant applications with Temporal. For related guidance on where to draw Namespace boundaries and how to scope credentials and permissions, see [Namespace best practices](/best-practices/managing-namespace) and [Managing Temporal Cloud access control](/best-practices/cloud-access-control). ## Architectural principles When designing a multi-tenant Temporal application, follow these principles: - **Define your tenant model** - Determine what constitutes a tenant in your business (customers, pricing tiers, teams, etc.) - **Prefer simplicity** - Start with the simplest pattern that meets your needs - **Understand Temporal limits** - Design within the constraints of your Temporal deployment - **Test at scale** - Performance testing must drive your capacity decisions - **Plan for growth** - Consider how you'll onboard new tenants and scale workers ## Architectural patterns There are four main patterns for multi-tenant applications in Temporal, listed from most to least recommended: ### 1. Task queues per tenant (Recommended) **Use different [Task Queues](/task-queue) for each tenant's [Workflows](/workflows) and [Activities](/activities).** This is the recommended pattern for most use cases. Each tenant gets dedicated Task Queue(s), with [Workers](/workers) polling multiple tenant Task Queues in a single process. **Pros:** - Strong isolation between tenants - Efficient resource utilization - Flexible worker scaling - Easy to add new tenants - Can handle thousands of tenants per [Namespace](/namespaces) **Cons:** - Requires worker configuration management - Potential for uneven resource distribution - Need to prevent "noisy neighbor" issues at the worker level **Related:** - [Task Queue Isolation Pattern Details](#task-queue-isolation-pattern) ### 2. Single Task Queue with Fairness **Use a single [Task Queue](/task-queue) with [Fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control how much of the Task Queue's capacity each tenant receives. You can also set per-fairness-key rate limits (requests per second) to cap individual tenant throughput, preventing any single tenant from consuming too much capacity. **Pros:** - Priority and Fairness keys and weights can be adjusted without redeployment - Onboarding new tenants doesn't require spinning up additional [Workers](/workers) - Simpler Worker topology than per-tenant Task Queues **Cons:** - Fairness is probabilistic and may be harder to debug than strict isolation - The fairness weight applies at schedule time, not at dispatch time, so it only affects newly-scheduled Tasks - When using [Worker Versioning](/worker-versioning), Fairness isn't guaranteed between versions > **💡 Tip:** > This pattern works well when you have many tenants with different service tiers and want to manage them without the operational overhead of per-tenant Task Queues or Workers. **Related:** - [Task Queue Fairness Reference](/develop/task-queue-priority-fairness#task-queue-fairness) ### 3. Shared Workflow Task Queues, separate Activity Task Queues **Share [Workflow Task Queues](/task-queue) but use different [Activity Task Queues](/task-queue) per tenant.** Use this pattern when [Workflows](/workflows) are lightweight but [Activities](/activities) have heavy resource requirements or external dependencies that need isolation. **Pros:** - Easier worker management than full isolation - Activity-level tenant isolation - Good for compute-intensive Activities **Cons:** - Less isolation than pattern #1 - Workflow visibility is shared - More complex to reason about ### 4. Namespace per tenant **Use a separate [Namespace](/namespaces) for each tenant.** Only practical for a smaller number of high-value tenants due to operational overhead. Most teams find this manageable for fewer than 50 tenants, though organizations with strong automation may scale higher. This pattern is not a good fit if you expect a very large number of tenants (10,000+). **Pros:** - Complete isolation between tenants — no noisy neighbor problem - Each Namespace has its own [rate limits](/cloud/limits) that can be provisioned on demand per customer - Each Namespace can be deployed across [multiple regions](/cloud/service-availability) globally - Per-Namespace observability is available by default - Maximum security boundary **Cons:** - Higher operational overhead - Credential and connectivity management per [Namespace](/namespaces) - Requires a new [Worker](/workers) pool deployment for each customer (minimum 2 per Namespace for high availability) - Not cost-effective at scale This pattern is usually chosen when tenant boundaries also need to be credential boundaries. For example, each tenant may need its own service accounts, API keys, dashboards, or rate limits. If that is your primary driver, review [Managing Temporal Cloud access control](/best-practices/cloud-access-control) together with [Namespace best practices](/best-practices/managing-namespace) before committing to Namespace-per-tenant isolation. **Related:** - [Namespace Isolation in Temporal Cloud](/evaluate/development-production-features/multi-tenancy#namespace-isolation) ### Pattern comparison | | Task Queues per tenant | Fairness-based | Shared Workflow / Separate Activity TQs | Namespace per tenant | |---|---|---|---|---| | **Isolation** | Task Queue level | Probabilistic (weighted) | Activity-level only | Complete | | **Noisy neighbor protection** | Strong | Weight-based throttling | Activity-level | Full — separate rate limits | | **Worker management** | Moderate — config per tenant | Simple — single Task Queue | Moderate | High — Worker pool per tenant | | **Onboarding new tenants** | Config update and restart | Set fairness and priority values (no new Workers) | Config update and Worker restart | New Namespace and Worker pool | | **Observability** | Per-Task Queue metrics | Per-Task Queue metrics | Mixed | Per-Namespace | | **Rate limiting** | Shared across Task Queue | Per-key rate limits | Shared across Namespace | Independent per Namespace | | **Scale ceiling** | Thousands of tenants | Thousands of tenants | Thousands of tenants | 10,000 (Namespace limit) | | **Best for** | Most multi-tenant apps | Tiered SaaS with many tenants | Heavy Activity workloads | High-value, compliance-sensitive tenants | ## Task Queue isolation pattern This section details the recommended pattern for most multi-tenant applications. ### Worker design When a [Worker](/workers) starts up: 1. **Load tenant configuration** - Retrieve the list of tenants this Worker should handle (from config file, API, or database) 2. **Create [Task Queues](/task-queue)** - For each tenant, generate a unique Task Queue name (for example, `customer-{tenant-id}`) 3. **Register [Workflows](/workflows) and [Activities](/activities)** - Register your Workflow and Activity implementations once, passing the tenant-specific Task Queue name 4. **Poll multiple Task Queues** - A single Worker process polls all assigned tenant Task Queues ```go // Example: Go worker polling multiple tenant Task Queues for _, tenant := range assignedTenants { taskQueue := fmt.Sprintf("customer-%s", tenant.ID) worker := worker.New(client, taskQueue, worker.Options{}) worker.RegisterWorkflow(YourWorkflow) worker.RegisterActivity(YourActivity) } ``` ### Routing requests to Task Queues Your application needs to route [Workflow](/workflows) starts and other operations to the correct tenant [Task Queue](/task-queue): ```go // Example: Starting a Workflow for a specific tenant taskQueue := fmt.Sprintf("customer-%s", tenantID) workflowOptions := client.StartWorkflowOptions{ ID: workflowID, TaskQueue: taskQueue, } ``` Consider creating an API or service that: - Maps tenant IDs to Task Queue names - Tracks which [Workers](/workers) are handling which tenants - Allows both your application and Workers to read the mappings of: 1. Tenant IDs to Task Queues 1. Workers to tenants ### Capacity planning Key questions to answer through performance testing: **[Namespace](/namespaces) capacity:** - How many concurrent [Task Queue](/task-queue) pollers can your Namespace support? - What are your [Actions Per Second (APS)](/cloud/limits#actions-per-second) limits? - What are your [Operations Per Second (OPS)](/references/operation-list) limits? **[Worker](/workers) capacity:** - How many tenants can a single Worker process handle? - What are the CPU and memory requirements per tenant? - How many concurrent [Workflow](/workflows) executions per tenant? - How many concurrent [Activity](/activities) executions per tenant? **SDK configuration to tune:** - `MaxConcurrentWorkflowTaskExecutionSize` - `MaxConcurrentActivityExecutionSize` - `MaxConcurrentWorkflowTaskPollers` - `MaxConcurrentActivityTaskPollers` - Worker replicas (in Kubernetes deployments) ### Provisioning new tenants Automate tenant onboarding with a Temporal [Workflow](/workflows): 1. Create a tenant onboarding Workflow that: - Validates tenant information - Provisions infrastructure - Deploys/updates [Worker](/workers) configuration - Triggers Worker restarts or scaling - Verifies the tenant is operational 2. Store tenant-to-Worker mappings in a database or configuration service 3. Update Worker deployments to pick up new tenant assignments ## Practical example **Scenario:** A SaaS company has 1,000 customers and expects to grow to 5,000 customers over 3 years. They have 2 [Workflows](/workflows) and ~25 [Activities](/activities) per Workflow. All customers are on the same tier (no segmentation yet). ### Assumptions | Item | Value | |------|-------| | Current customers | 1,000 | | Workflow Task Queues per customer | 1 | | Activity Task Queues per customer | 1 | | Max Task Queue pollers per Namespace | 20,000 (per [Cloud limits](/cloud/limits)) | | SDK concurrent Workflow task pollers | 5 | | SDK concurrent Activity task pollers | 5 | | Max concurrent Workflow executions | 200 | | Max concurrent Activity executions | 200 | ### Capacity calculations **[Task Queue](/task-queue) poller limits:** - Each [Worker](/workers) uses 10 pollers per tenant (5 Workflow + 5 Activity) - Maximum Workers in [Namespace](/namespaces): 20,000 pollers ÷ 10 = **2,000 Workers** **Worker capacity:** - Each Worker can theoretically handle 200 [Workflows](/workflows) and 200 [Activities](/activities) concurrently - Conservative estimate: **250 tenants per Worker** (accounting for overhead) - For 1,000 customers: **4 Workers minimum** (plus replicas for HA) - For 5,000 customers: **20 Workers minimum** (plus replicas for HA) **Namespace capacity:** - At 250 tenants per Worker, need 2 Workers per group of tenants (for HA) - Maximum tenants in Namespace: (2,000 Workers ÷ 2) × 250 = **250,000 tenants** > **📝 Note:** > These are theoretical calculations based on SDK defaults. **Always perform load testing** to determine actual capacity for your specific workload. Monitor CPU, memory, and Temporal metrics during testing. > > While testing, also pay attention to your [metrics capacity and cardinality](/cloud/metrics/openmetrics/api-reference#managing-high-cardinality). ### Worker assignment strategies **Option 1: Static configuration** - Each [Worker](/workers) reads a config file listing assigned tenant IDs - Simple to implement - Requires deployment to add tenants **Option 2: Dynamic API** - Workers call an API on startup to get assigned tenants - Workers identified by static ID (1 to N) - API returns tenant list based on Worker ID - More flexible, no deployment needed for new tenants ## Best practices ### How tenant isolation affects Namespace and access design Your multi-tenant architecture also determines how much isolation you get from Namespaces and access controls: - **Shared Namespace with per-tenant Task Queues**: Best for scale and operational simplicity, but tenant isolation is mostly enforced by your application and worker routing logic rather than by Temporal credentials. - **Separate Namespaces for domains or services**: Useful when teams need separate credentials, dashboards, APS envelopes, or on-call boundaries. - **Namespace per tenant**: Strongest isolation, but highest provisioning and credential-management overhead. If tenants, teams, or regulated workloads need different credentials or RBAC boundaries, decide that together with your Namespace topology. See [Namespace best practices](/best-practices/managing-namespace) and [Managing Temporal Cloud access control](/best-practices/cloud-access-control). ### Monitoring Track these [metrics](/references/sdk-metrics) per tenant: - [Workflow completion](/cloud/metrics/openmetrics/metrics-reference#workflow-completion-metrics) rates - [Activity execution](/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics) rates - [Task Queue backlog](/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics) - [Worker resource utilization](/references/sdk-metrics#worker_task_slots_used) - [Workflow failure rates](/encyclopedia/detecting-workflow-failures) ### Handling noisy neighbors Even with [Task Queue](/task-queue) isolation, monitor for tenants that: - Generate excessive load - Have high failure rates - Cause [Worker](/workers) resource exhaustion Strategies: - Implement per-tenant rate limiting in your application - Implement fairness keys and apply per-key rate limits - Move problematic tenants to dedicated Workers - Use [Workflow](/workflows)/[Activity](/activities) timeouts aggressively ### Tenant lifecycle Plan for: - **Onboarding** - Automated provisioning [Workflow](/workflows) - **Scaling** - When to add new [Workers](/workers) for growing tenants - **Offboarding** - Graceful tenant removal and data cleanup - **Rebalancing** - Redistributing tenants across Workers ### Search Attributes Use [Search Attributes](/search-attribute) to enable tenant-scoped queries: ```go // Add tenant ID as a Search Attribute searchAttributes := map[string]interface{}{ "TenantId": tenantID, } ``` This allows filtering [Workflows](/workflows) by tenant in the UI and SDK: ```sql TenantId = 'customer-123' AND ExecutionStatus = 'Running' ``` ## Related resources **Related:** - [Multi-tenancy Overview](/evaluate/development-production-features/multi-tenancy) - [Temporal Cloud Limits](/cloud/limits) - [Visibility and Search Attributes](/visibility) --- # Pre-production testing Source: https://docs.temporal.io/best-practices/pre-production-testing > Experience-driven testing practices for teams running Temporal applications, covering failure injection, load testing, and operational validation. This guide collects practical, experience-driven testing practices for teams running Temporal applications. The goal is not just to verify that things fail and recover, but to build confidence that *recovery*, *correctness*, *consistency*, and *operability* hold under real-world conditions. The scenarios below assume familiarity with Temporal concepts such as [Namespaces](/namespaces), [Workers](/workers), [Task Queues](/task-queue), [History shards](/temporal-service/temporal-server#history-shard), [Timers](/workflow-execution/timers-delays), and [Workflow replay](/workflow-execution#replay). Start with [Understanding Temporal](/evaluate/understanding-temporal#durable-execution) if you need background. Before starting any load testing in Temporal Cloud, we recommend connecting with your Temporal Account team and our Developer Success Engineering team. ## Guiding principles Before diving into specific experiments, keep these principles in mind: - **Failure is normal**: Temporal is designed to survive failure and issues, but *your application logic* must be too. - **Partial failure is often harder to deal with than total failure**: Systems that are "mostly working" expose the most flaws. - **Recovery paths deserve as much testing as steady state**: Analyze recovering application behavior as much as you analyze failing behavior. - **Build observability before you break things**: Ensure metrics, logs, and visibility tools are in place before injecting failures. - **Testing is a continual process**: Testing is never finished. Testing is a practice. ## Worker testing **Relevant best practices**: [Worker deployment and performance](/best-practices/worker), appropriate timeouts, managing Worker shutdown, idempotency - [Worker shutdown](/encyclopedia/workers/worker-shutdown) ### Kill all Workers, then restart them **What to test** Abruptly terminate all Workers processing a Task Queue, then restart them. **Why it matters** - Validates at-least-once execution semantics. - Ensures Activities are idempotent and Workflows replay cleanly. - Validates Task timeouts and retries and that Workers can finish business processes. **How to run this** Depending on execution environment: - **Kubernetes**: Set pod count to zero: ```bash kubectl scale deployment --replicas=0 -n kubectl scale deployment --replicas=3 -n ``` - **Azure App Service**: ```bash az webapp restart --name --resource-group ``` **Things to watch** - Duplicate/improper Activity results - Workflow failures - Workflow backlog growth and drain time ### Frequent Worker restart **What to test** Periodically restart a fixed or random percentage (for example, 20-30%) of your Worker fleet every few minutes. **Why it matters** - Mimics failure modes where Workers restart due to high CPU utilization and out-of-memory errors from compute-intensive logic in Activities. - Ensures Temporal invalidates specific Sticky Task Queues and reschedules the task to the associated non-Sticky Task Queue. **How to run this** - **Kubernetes**: Build a script using `kubectl` to randomly delete pods in a loop. - **Chaos Mesh**: [Simulate pod faults](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/). - **App Services**: Scale down and up again. **Things to watch** - Replay latency - Drop in Workflow and Activity completion - Duplicate/improper Activity results - Workflow failures - Workflow backlog growth and drain time ## Load testing ### Pre-load test setup: expectations for success 1. Have SDK metrics accessible (not just the Cloud metrics) 2. Understand and predict what you should see from these metrics: - Rate limiting ([`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) and [`temporal_cloud_v1_service_request_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_throttled_count)) - Workflow failures (`temporal_cloud_v1_workflow_failed_count`) - Workflow execution time (`workflow_endtoend_latency`) - High Cloud latency (`temporal_cloud_v1_service_latency_p95`) - [Worker metrics](/develop/worker-performance) (`workflow_task_schedule_to_start_latency` and `activity_schedule_to_start_latency`) 3. Determine throughput requirements ahead of time. Work with your account team to match that to the Namespace capacity to avoid rate limiting. Capacity increases are done via Temporal support and can be requested for a load test (short-term). 4. Automate how you run the load test so you can start and stop it at will. How will you clear Workflow Executions that are just temporary? 5. What does "success" look like for this test? Be specific with metrics and numbers stated in business terms. ### Validate downstream load capacity **Relevant best practices**: Idempotent Activities, bounded retries, appropriate timeouts and retry policies, understand behavior when limits are reached **What to test** - Schedule a large number of Actions and Requests by starting many Workflows - Increase the number until you start overloading downstream systems **Why it matters** Validates behavior of Temporal application and application dependencies under high load. **How to run this** Start Workflows at a rate to surpass throughput limits. Example: [temporal-ratelimit-tester-go](https://github.com/joshmsmith/temporal-ratelimit-tester-go) **Things to watch** - Downstream service error rates (HTTP 5xx, database errors) - Increased downstream service latency and saturation metrics - Activity failure rates, specifically classifying between retryable and non-retryable errors - Activity retry and backoff behavior against the overloaded system - Workflow backlog growth and drain time - Correctness and consistency of data (ensuring Activity idempotency holds under duress) - Worker CPU/memory utilization ### Validate rate limiting behavior **Relevant best practices**: [Manage Namespace capacity limits](/best-practices/managing-aps-limits), understand behavior when limits are reached **What to test** - Schedule a large number of Actions and Requests by starting many Workflows - Increase the number until you get rate limited and trigger [`temporal_cloud_v1_total_action_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_total_action_throttled_count) or [`temporal_cloud_v1_service_request_throttled_count`](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_throttled_count) **Why it matters** Validates behavior of Cloud service under high load: "In Temporal Cloud, the effect of rate limiting is increased latency, not lost work. Workers might take longer to complete Workflows." **How to run this** 1. (Optional) Decrease a test Namespace's rate limits to make it easier to hit limits 2. Calculate current APS at current throughput (in production) 3. Calculate Workflow throughput needed to surpass limits 4. Start Workflows at a rate to surpass throughput limits using [temporal-ratelimit-tester-go](https://github.com/joshmsmith/temporal-ratelimit-tester-go) **Things to watch** - Worker behavior when rate limited - Client behavior when rate limited - Temporal request and long_request failure rates - Workflow success rates - Workflow latency rates ## Failover and availability **Relevant best practices**: Use [High Availability features](/cloud/high-availability) for critical workloads. - [High Availability monitoring](/cloud/high-availability/monitoring) ### Test region failover **What to test** Trigger a [High Availability](/cloud/high-availability) failover event for a Namespace. **Why it matters** - Real outages are messy and rarely isolated. - Ensures your operational playbooks and automation are resilient. - Validates Worker and Namespace failover behavior. **How to run this** Execute a manual failover per the [manual failovers documentation](/cloud/high-availability/failovers/manage#trigger-failover). **Things to watch** - Namespace availability - Client and Worker connectivity to failover region - Workflow Task reassignments - Human-in-the-loop recovery steps ## Dependency and downstream testing ### Break the things your Workflows call **What to test** Intentionally break or degrade downstream dependencies used by Activities: - Make databases read-only or unavailable - Inject high latency or error rates into external APIs - Throttle or pause message queues and event streams **Why it matters** - Temporal guarantees Workflow durability, not dependency availability. - Validates that Activities are retryable, idempotent, and correctly timeout-bounded. - Ensures Workflows make forward progress instead of livelocking on broken dependencies. **Things to watch** - Activity retry and backoff behavior - Heartbeat effectiveness for long-running Activities - Database connection exhaustion and retry storms - API timeouts vs Activity timeouts - Whether failures propagate as Signals, compensations, or Workflow-level errors **Anti-patterns this reveals** - Non-idempotent Activities - Infinite retries without circuit breaking - Using Workflow logic to "wait out" broken dependencies ## Deployment and code-level testing ### Deploy a Workflow change with versioning **Relevant best practices**: Implement a versioning strategy. - [Workflow Versioning Strategies - Developer Corner](https://community.temporal.io/t/workflow-versioning-strategies/6911) - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) - [Replay Testing](/evaluate/development-production-features/testing-suite) **What to test** - Deploy Workflow code that would introduce non-deterministic errors (NDEs) but use a versioning strategy to deploy successfully - Validate Workflow success and clear the backlog of tasks **Why it matters** - Unplanned NDEs can be a painful surprise - Tests versioning strategy and patching discipline to build production confidence **Things to watch** - Workflow Task failure reasons - Effectiveness of versioning and patching patterns ### Deploy a version that causes NDEs, then recover **Relevant best practices**: Implement a versioning strategy. - [Workflow Versioning Strategies - Developer Corner](https://community.temporal.io/t/workflow-versioning-strategies/6911) - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) - [Replay Testing](/evaluate/development-production-features/testing-suite) **What to test** - Deploy Workflow code that introduces non-deterministic errors (NDEs) - Attempt rollback to a known-good version, or apply versioning strategies to apply the new changes successfully - Clear or recover the backlog of tasks **Why it matters** - Unplanned NDEs can be a painful surprise - Tests versioning strategy, patching discipline, and recovery tooling **Things to watch** - Workflow Task failure reasons - Backlog growth and drain time - Effectiveness of versioning and patching patterns ## Network-level testing The scenarios below are most relevant if your infrastructure introduces network boundaries (such as firewalls, VPNs, or network policies) between Workers and the Temporal service, or if you need to verify application behavior during prolonged disconnections. **Relevant best practices**: Idempotent Activities, bounded retries, appropriate timeouts - [Activity timeouts](https://temporal.io/blog/activity-timeouts) - [Idempotency and durable execution](https://temporal.io/blog/idempotency-and-durable-execution) ### Remove network connectivity to a Namespace **What to test** Temporarily block all network access between Workers and the Temporal service for a Namespace. **Why it matters** - Validates Worker retry behavior, Sticky Task Queue behavior, Worker recovery performance, backoff policies, and Workflow replay determinism under prolonged disconnection. - Ensures no assumptions are made about "always-on" connectivity. **Temporal failure modes exercised** - Workflow Task timeouts vs retries - Activity retry semantics - Replay correctness after long gaps **How to run this** - **Kubernetes**: Apply a NetworkPolicy that denies egress from Worker pods to the Temporal APIs. - **[ToxiProxy](https://github.com/Shopify/toxiproxy)**: Proves your application doesn't have single points of failure. - **Chaos Mesh / Litmus**: NetworkChaos with full packet drop. - **Local testing**: Block ports with iptables or firewall rules. **Things to watch** - Workflow failures (replay, timeout) - Workflow Task retries - Activity failures, classifications (retryable vs non-retryable) - Worker CPU usage during reconnect storms ## Observability checklist Before (and during) testing, ensure visibility into: - Workflow Task and Activity failure rates - Throughput limits and usage - Workflow and Activity end-to-end latencies - Task latency and backlog depth - Event History size and event counts - Worker CPU, memory, and restart counts - gRPC error codes - Retry behavior ## Game day runbook Use this checklist when running tests during a scheduled game day or real incident simulation. ### Before you start - Make sure people know you're testing and what scenarios you're trying - Let the teams that support the APIs you're calling know you're testing - Reach out to the Temporal Cloud Support and Account teams to coordinate - Dashboards for SDK and Cloud metrics - Task latency, backlog depth, Workflow failures, Activity failures - Alerts muted or routed appropriately - Known-good deployment artifact available - Rollback and scale controls verified ### During testing - Introduce *one variable at a time* - Record start/stop times of each experiment - Capture screenshots or logs of unexpected behavior - Track backlog growth and drain rate ### Recovery validation - Workflows resume without manual intervention - No permanent Workflow Task failures (unless intentional) - Activity retries behave as expected - Backlogs drain in predictable time ### After action review - Identify unclear alerts or missing metrics/alerts - Update retry, timeout, or versioning policies - Document surprises and operational debt ## Summary Pre-production testing with Temporal is about more than proving durability - it's about proving *operability under stress*. You want to go through the exercise and know what to do before you go to production and have to do it for real. If your system survives: - Connectivity issues - Repeated failovers - Greater than expected load - Mass Worker churn ...then you can have confidence it's ready for many kinds of production chaos. --- # Security controls for Temporal Cloud Source: https://docs.temporal.io/best-practices/security-controls > Best practices for implementing and managing security controls in Temporal Cloud environments. Temporal Cloud provides the capabilities of self-hosted Temporal as a managed service; it does not manage your applications or workers. Applications and services written using Temporal SDKs still run in your compute environment, and you have full control over how you secure your applications and services. These best practices ensure your Temporal Cloud environment adheres to the security guidelines recommended by our team. You can also learn more about our security practices, compliance posture, and subscribe for vulnerability (CVE) updates at https://trust.temporal.io/. If you have questions, contact our security team at security@temporal.io. > **💡 Tip:** > > **Stay Updated on Temporal Security Advisories:** > Subscribe to Temporal’s security updates on the [Temporal Trust Portal](https://trust.temporal.io/) so you are aware of any patches or CVEs. While Temporal Cloud server-side updates are handled by the vendor, your Temporal SDKs (in application code) should be kept up-to-date. > ## Identity and access management Strong identity management in Temporal Cloud is crucial for ensuring secure access for your Temporal account. It’s critical that only authorized users and services can access your Temporal Cloud account and that each has the minimum necessary permissions needed for their role. ### Enable [SAML single sign-on](/cloud/manage-access/saml) (SSO) for user access Integrate Temporal Cloud with your organization's identity provider via SAML 2.0 for centralized authentication. SSO allows you to enforce your corporate login policies (such as MFA and password complexity). When you configure SAML with Temporal Cloud, you can disable social logins (that is, Microsoft, Google) by opening a support ticket. ### Use least-privilege roles for Temporal Cloud users Temporal Cloud provides [preconfigured account-level roles](/cloud/manage-access/users) (Account Owner, Finance Admin, Global Admin, Developer, Read-Only) and Namespace-level permissions. Assign users the lowest level of access they need. For example, give developers access only to the Namespaces they work on, and use read-only roles for auditors or reviewers. Regularly review user roles and remove or downgrade accounts that are no longer needed ### Use SCIM or automated user provisioning When applicable, use [SCIM](/cloud/manage-access/scim) or the Temporal Cloud user management API to automate adding and removing user accounts. This ensures timely removal of access when people change roles or leave the organization. ### Use Service Accounts for automation For non-human access (CI/CD pipelines, backend services), use [Temporal Cloud Service Accounts](/cloud/manage-access/service-accounts) instead of shared user logins. Service Accounts are machine identities that can be granted specific permissions without ties to an individual. Create separate Service Accounts with unique API keys for different applications or microservices, and apply least privilege to each (for example, a service account that only has access to one Namespace). ## Secure application authentication and API access Clients interact with the Temporal Service to initiate and manage Workflows, while Workers execute the business logic defined in Workflows and Activities in your own environment. A crucial aspect of strengthening your usage involves securing these interactions. Temporal Cloud offers two authentication methods for your applications: mutual TLS certificates and API keys. ### Use mutual TLS (mTLS) for comprehensive security Temporal Cloud secures its gRPC endpoint per Namespace via mutual TLS. This means you provide a Certificate Authority (CA) certificate for your Namespace, and all your Temporal clients/workers must present client certificates signed by that CA. We recommend you enable mTLS for strong identity assurance of clients; it ensures only systems holding a valid certificate (issued by your trusted CA) can connect. Generate a private key and CA certificate (or use your enterprise CA) and upload the CA to Temporal Cloud. Do not share these certificates and associated keys beyond the authorized services. ### Proactively manage and rotate certificates Track the expiration dates of your client and [Certificate Authority certificates](/cloud/certificates). Temporal Cloud trusts the uploaded CA; if it expires, all client authorizations will fail. Establish and automate a certificate rotation schedule (for example, rotate client certificates quarterly and CA certificates annually, well before expiry). Temporal supports uploading a new CA certificate alongside the old one to allow seamless rollover. Always test new certificates in a staging environment if possible. ### Handle API keys with strict care Temporal Cloud API keys are an alternative to mTLS for authentication of SDKs, CLI, and automation. If you opt for API keys, handle them with strict care by enacting the following practices: - Keep them secret: store in a secrets manager, never in code or Git. - Rotate at least every 90 days: Temporal lets you create a new key, swap it in, then delete the old one. - One key per service/person: no sharing or reuse. - Monitor usage & revoke on anomalies: feed Temporal audit logs to SIEM. - Optional: Admins can disable all user API keys if your policy is “mTLS only.” ## Network configuration and isolation Although Temporal Cloud is a SaaS offering, you retain control over its networking configurations, allowing for tailored security measures. By minimizing public internet exposure and segmenting Temporal workflows into suitable network zones, you can significantly bolster security and reduce potential vulnerabilities. This approach ensures that your workflows are isolated and protected within your defined network boundaries, even while using a cloud-based service. ### Use private connectivity Temporal Cloud supports private connectivity options such as [AWS PrivateLink](/cloud/connectivity/aws-connectivity) and [Google Cloud Private Service Connect](/cloud/connectivity/gcp-connectivity). If your infrastructure is in AWS or GCP, configure a PrivateLink/PSC endpoint for Temporal Cloud. This allows your workers and applications to reach Temporal Cloud over a private network path, avoiding traversal of the public internet. Private connectivity reduces the surface for man-in-the-middle attacks and can meet stringent network security policies. ### Separate environments by Namespace Use [Temporal Namespaces](/best-practices/managing-namespace#naming-conventions) to isolate workflows for different environments or teams (for example, development, staging, production). Each Namespace is logically segregated and cannot interact with others by default, providing a security boundary. Ensure that your production Namespace uses stricter network controls (for example, only accessible from the prod network) and that credentials for it are separate from non-prod Namespaces. This limits the impact of any compromise in a lower environment, and as workflow data is only visible to users with access to that Namespace, separating environments by Namespace also enforces data-visibility boundaries. ## Data protection and encryption Temporal's data encryption capabilities ensure the security and confidentiality of your Workflows and provide protection without compromising performance. Protecting the data that you send to and store in Temporal Cloud is a joint responsibility. Temporal Cloud already encrypts all data at rest on the server side, but you can add additional layers of encryption and control. ### Enable client-side encryption for Workflow data Temporal provides an optional [data conversion framework](/dataconversion) (Data Converter) and payload codec interface; customers must implement, deploy, and operate their own custom codec and manage encryption keys. In practice, this means you can encrypt any sensitive data before it is sent to Temporal Cloud and only decrypt it on the Client/Worker side. Because encryption keys stay under your control, you are responsible for key generation, secure storage, rotation, and versioning. Implementing this involves developing a custom codec plugin in your Temporal SDK and optionally (if you need to inspect decrypted payloads in the Web UI or CLI) deploying a dedicated codec server. ### Encode Workflow failure details with a Failure Converter Temporal’s default behavior copies error messages and call stacks as plain text, and this text is directly accessible in the Message field of Workflow Executions. If your failure messages and stack traces contain sensitive information, it is recommended that you configure the [Failure Converter](/failure-converter) to encrypt the error information. This would encrypt the `message` and `stack_trace` fields in the payloads. ### Use Namespace data retention policies Temporal Cloud Namespace has a [Retention Period](/temporal-service/temporal-server#retention-period) setting for workflow histories (1 to 90 days). Set an appropriate retention period to balance operational needs with security. Shorter retention means completed workflow data (history, payloads) is purged sooner, reducing the amount of sensitive data stored in the cloud at any time. Document your retention choices to align with your company’s data retention policies and regulatory requirements. For retention periods over 90 days, these can be exported to your own GCS or S3 buckets. ## Availability and disaster recovery Temporal Cloud’s platform is engineered for fault-tolerance out of the box, but you determine which Namespaces merit the very highest availability guarantees. Use the table below to decide when to turn on different High Availability models and how to operationalize them. | Namespace scope | Use Case | Uptime SLA | Recovery Time Objective (RTO) | Recovery Point Objective (RPO) | |-----------------|----------|------------|--------------------------------|--------------------------------| | **Single-Region** | **If your application is built for one region and does not have stringent high-availability or disaster recovery requirements.** | 99.9% | ≤ 8 hours | ≤ 8 hours | | **Same-Region Replication** | **If you want higher availability but your application is designed for a single region or if cross region latency doesn’t meet SLAs for application** | 99.99% | ≤ 20 minutes | Near-zero (≈ seconds) | | **Multi-Region Replication** | **If a disruption of your workflow will cause loss of revenue, poor end-user experience, or issues with regulatory compliance.** | 99.99% | ≤ 20 minutes | Near-zero (≈ seconds) | | **Multi-Cloud Replication** | **If you need the highest level of disaster tolerance, protecting against outages of an entire cloud provider (for example, AWS or GCP)** | 99.99% | ≤ 20 minutes | Near-zero (≈ seconds) | ### Identify availability-sensitive Namespaces Run a business-impact analysis to flag workflows where a regional outage would cause significant customer, revenue, or safety impact. Identify Namespaces that are availability-sensitive where a regional outage may have outsized business impacts such as revenue loss, poor customer experience, or inability to meet legal obligations. ### Enable High Availability for business-critical use cases For many organizations, ensuring High Availability (HA) is required because of strict uptime requirements, compliance, and regulatory needs. For these critical use cases, enable High Availability features for specific namespaces for a [99.99% contractual SLA](/cloud/high-availability#high-availability-features). When choosing between [same-region, multi-region, and multi-cloud replication](/cloud/high-availability), it is recommended to use multi-region/multi-cloud replication to distribute your dependencies across regions. Using physically separated regions improves the fault tolerance of your application. By default, Temporal Cloud provides a [99.9% contractual SLA guarantee](/cloud/high-availability) against service errors for all namespaces. Note: [enabling HA features for namespaces will 2x the consumption cost.](/cloud/pricing#high-availability-features) --- # Worker deployment and performance Source: https://docs.temporal.io/best-practices/worker > Best practices for deploying and optimizing Temporal Workers for performance and reliability. This document outlines best practices for deploying and optimizing Workers to ensure high performance, reliability, and scalability. It covers deployment strategies, scaling techniques, tuning recommendations, and monitoring approaches to help you get the most out of your Temporal Workers. We also provide a reference application, the Order Management System (OMS), that demonstrates the deployment best practices in action. You can find the OMS codebase on [GitHub](https://github.com/temporalio/reference-app-orders-go/tree/main/docs). ## Quick checklist Designing a comprehensive Worker deployment strategy to optimize production performance involves many considerations. We provide a quick checklist to help you get started. Before deploying Workers to production, ensure you address the following. Follow the links to the relevant sections for more details. - **[Configure each Worker appropriately](#actively-tune-worker-options-instead-of-relying-on-defaults)**: Actively tune Worker options based on your code, language runtime limits, and system resource constraints. Don't rely on defaults, which are designed for ease in development and testing, but not optimal for production environments. - **[Deploy enough Workers](#interpret-metrics-as-a-whole)**: Monitor performance metrics and scale Workers to meet your workload requirements. - **[Separate Task Queues logically](#separate-task-queues-logically)**: Size and split work across Task types (Activities and Workflows) and Task Queues based on workload characteristics. - **[Version Workers for safe deployments](#use-worker-versioning-to-safely-deploy-new-workflow-code)**: Ensure you can deploy new Workflow code without breaking running Executions. - **Run benchmarks**: Test your configuration under realistic load to confirm limits and settings are appropriate for your environment. ## Deployment and lifecycle management Well-designed Worker deployment ensures resilience, observability, and maintainability. A Worker should be treated as a long-running service that can be deployed, upgraded, and scaled in a controlled way. ### Package and configure Workers for flexibility Workers should be artifacts produced by a CI/CD pipeline. Inject all required parameters for connecting to Temporal Cloud or a self-hosted Temporal Service at runtime via environment variables, configuration files, or command-line parameters. This allows for more granularity, easier testability, easier upgrades, scalability, and isolation of Workers. In the order management reference app, Workers are packaged as Docker images with configuration provided via environment variables and mounted configuration files. The following Dockerfile uses a multi-stage build to create a minimal, production-ready Worker image: [Dockerfile](https://github.com/temporalio/reference-app-orders-go/blob/main/Dockerfile) ```Dockerfile FROM golang:1.25.4 AS oms-builder WORKDIR /usr/src/oms COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go mod download COPY app ./app COPY cmd ./cmd RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ CGO_ENABLED=0 go build -v -o /usr/local/bin/oms ./cmd/oms FROM busybox AS oms-worker ``` This Dockerfile uses a multi-stage build pattern with two stages: 1. `oms-builder` stage: compiles the Worker binary. 1. Copies dependency files and downloads dependencies using BuildKit cache mounts to speed up subsequent builds. 2. Copies the application code and builds a statically linked binary that doesn't require external libraries at runtime. 2. `oms-worker` stage: creates a minimal final image. 1. Copies only the compiled binary from the `oms-builder` stage. 2. Sets the entrypoint to run the Worker process. The entrypoint `oms worker` starts the Worker process, which reads configuration from environment variables at runtime. For example, the [Billing Worker deployment in Kubernetes](https://github.com/temporalio/reference-app-orders-go/blob/main/deployments/k8s/billing-worker-deployment.yaml) uses environment variables to configure the Worker: [deployments/k8s/billing-worker-deployment.yaml](https://github.com/temporalio/reference-app-orders-go/blob/main/deployments/k8s/billing-worker-deployment.yaml) ```yaml # ... spec: containers: - args: - -k - supersecretkey - -s - billing env: - name: FRAUD_API_URL value: http://billing-api:8084 - name: TEMPORAL_ADDRESS value: temporal-frontend.temporal:7233 image: ghcr.io/temporalio/reference-app-orders-go-worker:latest name: billing-worker imagePullPolicy: Always enableServiceLinks: false ``` ### Separate Task Queues logically Use separate Task Queues for distinct workloads. This isolation allows you to control rate limiting, prioritize certain workloads, and prevent one workload from starving another. For each Task Queue, ensure you configure at least two Workers to poll the Task Queue. In the order management reference app, each microservice has its own Task Queue. For example, the Billing Worker polls the `billing` Task Queue, while the Order Worker polls the `order` Task Queue. This separation allows each service to scale independently based on its workload. ![Diagram showing separate Task Queues for different Workers](/diagrams/worker-best-practice-oms-architecture.png) The following code snippet shows how the Billing Worker is set up to poll its Task Queue. The default value for `TaskQueue` is a constant defined in the `api.go` configuration file and is set to `billing`. Since Task Queues are created dynamically when first used, a mismatch between the Client and Worker Task Queue names does not result in an error. Instead, it creates two different Task Queues, and the Worker never receives Tasks from the Temporal Service because it's polling the wrong queue. Define the Task Queue name as a constant that both the Client and Worker reference to avoid this issue. [app/billing/worker.go](https://github.com/temporalio/reference-app-orders-go/blob/main/app/billing/worker.go) ```go // ... // RunWorker runs a Workflow and Activity worker for the Billing system. func RunWorker(ctx context.Context, config config.AppConfig, client client.Client) error { w := worker.New(client, TaskQueue, worker.Options{ MaxConcurrentWorkflowTaskPollers: 8, MaxConcurrentActivityTaskPollers: 8, }) w.RegisterWorkflow(Charge) w.RegisterActivity(&Activities{FraudCheckURL: config.FraudURL}) return w.Run(temporalutil.WorkerInterruptFromContext(ctx)) } ``` ### Use Worker Versioning to safely deploy new Workflow code Use Worker Versioning to deploy new Workflow code without breaking running Executions. Worker Versioning lets you map each Workflow Execution to a specific Worker Deployment Version identified by a build ID, which guarantees that pinned Workflows always run on the same Worker version where they started. For most teams, Worker Versioning should be the default approach for evolving Workflow code in production. It gives you safer rollouts, clearer rollback behavior, and a cleaner operational model than maintaining multiple code paths inside a Workflow definition. To learn more about versioning Workflows, see the [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) guide or take our [Worker Versioning course](https://learn.temporal.io/courses/worker_versioning/). > **💡 Tip:** > > If your deployment environment cannot yet support Worker Versioning, you can use [Patching](/patching) to introduce > changes to your Workflow code without breaking running Executions. Treat patching as a fallback for environments that > cannot adopt versioned worker deployments yet, not as the default recommendation. > ### Manage Event History growth If a Worker goes offline and another Worker picks up the same Workflow Execution, the new Worker must replay the existing Event History to resume the Workflow Execution. If the Event History is too large or has too many Events, replay affects the performance of the new Worker and may even cause timeout errors well before the hard limit of 51,200 Events is reached. We recommend not exceeding a few thousand Events in a single Workflow Execution. The best way to handle Event History growth is to use the [Continue-As-New](/workflow-execution/continue-as-new) mechanism to continue under a new Workflow Execution with a new Event History, repeating this process as you approach the limits again. All Temporal SDKs provide functions to suggest when to use [Continue-As-New](/workflow-execution/continue-as-new). For example, the Python SDK has the [`is_continue_as_new_suggested()`](https://python.temporal.io/temporalio.workflow.Info.html#is_continue_as_new_suggested) function that returns a `bool` indicating whether to use Continue-As-New. In addition to the number of Events, monitor the size of the Event History. Input parameters and output values of both Workflows and Activities are stored in the Event History. Storing large amounts of data can lead to performance problems, so the Temporal Cluster limits both the size of individual payloads and the total Event History size. A Workflow Execution may be terminated if any single payload exceeds 2 MB or if the entire Event History exceeds 50 MB. To avoid hitting these limits, avoid passing large amounts of data into and out of Workflows and Activities. A common way to reduce payload and Event History size is the [Claim Check](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern) pattern, widely used with messaging systems such as Apache Kafka. Instead of passing large data into your function, store that data external to Temporal in a database or file system. Pass an identifier for the data, such as a primary key or path, into the function and use an Activity to retrieve it as needed. If your Activity produces large output, use a similar approach: write the data to an external system and return an identifier that can be used to retrieve it later. ## Scaling, monitoring, and tuning Scaling and tuning are critical to Worker performance and cost efficiency. The goal is to balance concurrency, throughput, and resource utilization while maintaining low Task latency. ### Actively tune Worker options instead of relying on defaults Default Worker settings are designed to work across a wide range of use cases primarily for ease in development and testing. In production environments, your Workflow complexity, Activity duration, payload sizes, and infrastructure constraints all influence optimal Worker configuration. Actively tuning your Workers ensures they perform well under your specific workload conditions. To get started, focus on these key Worker options: - **Task slots**: Limits how many Tasks execute concurrently. Set based on your Worker's CPU, memory, and the resource demands of your code. You can choose different types of Slot Suppliers or implement a custom Slot Supplier to control how Task slots are assigned to different Task types. Refer to [Slot Suppliers](/develop/worker-performance#slot-suppliers) for more details. - **Sticky cache size**: Controls the size of the sticky cache for Workflow Executions. Larger caches reduce replay overhead but consume more memory. Refer to [Workflow Cache Tuning](/develop/worker-performance/workflow-cache) for more details. - **Poller counts**: Controls the number of pollers for Tasks. We recommend you use the Poller Autoscaling feature to automatically adjust the number of pollers based on your workload. Refer to [Configuring Poller Options](/develop/worker-performance/configuration#configuring-poller-options) for more details. Use the metrics listed in [Interpret metrics as a whole](#interpret-metrics-as-a-whole) to guide your tuning decisions. ### Interpret metrics as a whole No single metric tells the full story. The following are some of the most useful Worker-related metrics to monitor. We recommend having all metrics listed below on your Worker monitoring dashboard. When you observe anomalies, correlate across multiple metrics to identify root causes. - Worker CPU and memory utilization - `workflow_task_schedule_to_start_latency` and `activity_task_schedule_to_start_latency` - `worker_task_slots_available` - `temporal_long_request_failure`, `temporal_request_failure`, `temporal_long_request_latency`, and `temporal_request_latency` For example, Schedule-to-Start latency measures how long a Task waits in the queue before a Worker starts it. High latency means your Workers or pollers can’t keep up with incoming Tasks, but the root cause depends on your resource metrics: - High Schedule-to-Start latency and high CPU/memory: Workers are saturated. Scale up your Workers or add more Workers. It's also possible your Workers are blocked on Activities. Refer to [Troubleshooting - Depletion of Activity Task Slots](../troubleshooting/performance-bottlenecks.mdx#depletion-of-temporal_worker_task_slots_available-for-activityworker) for guidance. - High Schedule-to-Start latency and low CPU/memory: Workers are underutilized. Increase the number of pollers, executor slots, or both. If this is accompanied by high `temporal_long_request_latency` or `temporal_long_request_failure`, your Workers are struggling to reach the Temporal Service. Refer to [Troubleshooting - Long Request Latency](../troubleshooting/performance-bottlenecks.mdx#high-temporal_long_request_failure) for guidance. - Low Schedule-to-Start latency and low CPU/memory: Depending on your workload, this could be normal. If you are consistently seeing low memory usage and low CPU usage, you may be over-provisioning your Workers and can consider scaling down. Refer to [Intro to Worker Tuning](https://temporal.io/blog/an-introduction-to-worker-tuning) for more details and examples on how to interpret Worker metrics. ### Optimize Worker cache Workers keep a cache of Workflow Executions to improve performance by reducing replay overhead. However, larger caches consume more memory. The `temporal_sticky_cache_size` tracks the size of the cache. If you observe high memory usage for your Workers and high `temporal_sticky_cache_size`, you can be reasonably sure the cache is contributing to memory pressure. Having a high `temporal_sticky_cache_size` by itself isn't necessarily an issue, but if your Workers are memory-bound, consider reducing the cache size to allow more concurrent executions. We recommend you experiment with different cache sizes in a staging environment to find the optimal setting for your Workflows. Refer to [Troubleshooting - Caching](../troubleshooting/performance-bottlenecks.mdx#caching) for more details on how to interpret the different cache-related metrics. ### Manage scale-down safely Before shutting down a Worker, verify that it does not have too many active Tasks. This is especially relevant if your Workers are handling long-running, expensive Activities. If `worker_task_slots_available` is at or near zero, the Worker is running active Tasks. Shutting it down could trigger expensive retries or timeouts for long-running Activities. Use [Graceful Shutdowns](/encyclopedia/workers/worker-shutdown#graceful-shutdown) to allow the Worker to complete its current Tasks before shutting down. All SDKs provide a way to configure Graceful Shutdowns. For example, the Go SDK has the [`WorkerStopTimeout` option](https://pkg.go.dev/go.temporal.io/sdk@v1.38.0/internal#WorkerOptions) that lets you configure how long the Worker has to complete its current Tasks before shutting down. --- # Alerting on Worker metrics Source: https://docs.temporal.io/best-practices/worker-alerting > A recommended alert set for Temporal Workers, with tag filters, thresholds, and links to triage guidance Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window. Nobody catches those for you. This page recommends a set of alerts that covers them. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). ## Before you start None of this works until your Workers are emitting metrics and something is scraping them: - **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). - **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). ### Resolve the metric names for your setup The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). The names you query depend on your SDK and metrics reporter: - Every metric carries a `temporal_` prefix. - **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`. - **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. - **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why. Tag coverage varies too. `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely. Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript. Confirm the exact names and tags in your own metrics endpoint before writing queries. ## Start with these five If you are setting up Worker alerts for the first time, start here. These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing. 1. [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected). Your Workers have stopped polling. 1. [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error). Workflow code no longer matches recorded history. 1. [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). Executions are being terminated and losing work. 1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up. 1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled. Add the rest once these are tuned and quiet. ## Choose your thresholds Every threshold on this page is a starting point, not a service level objective. A high-throughput Task Queue needs different values than a bursty batch workload. There are two knobs per alert: - **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here. - **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection. Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero. There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page. ### Tune the `for` duration to your deploys Several of these conditions appear briefly whenever you deploy. A rolling restart drops pollers to zero on each pod as it cycles, produces NOT_FOUND on respond operations for Tasks that were in flight, and can produce short-lived non-determinism errors while two Worker versions overlap. A Temporal Service upgrade does the same for INTERNAL. All of those clear on their own once the rollout finishes, so the fix is a `for` duration longer than a deploy takes, not a quieter threshold. Time your own rollout and set it from that. Two conditions need a different fix, because no `for` duration makes them correct: - **An idle Task Queue** legitimately reports zero Task completions and an empty sticky cache. Pair both alerts with a demand signal instead, as [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) and [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) describe. - **Autoscaling to zero** legitimately reports no pollers and no free slots. Exclude scaled-down Workers from those alerts, or scope them to Task Queues you keep warm. ## Recommended alert set The condition column gives the tag filters that tell each alert apart. Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from. ### Request failures These fire on gRPC responses coming back from the Temporal Service to your Worker or Client. For triage, see [Request failures](/troubleshooting/request-failures). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | | [NOT_FOUND on respond operations](/troubleshooting/request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical | | [NOT_FOUND on Activity heartbeat](/troubleshooting/request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning | | [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical | | [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical | | [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning | | [UNIMPLEMENTED from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical | | [INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical | | [Request latency high on user-facing operations](/troubleshooting/request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical | ### Worker capacity These fire when your Workers stop keeping up with the Task Queue. For triage, see [Worker capacity](/troubleshooting/worker-capacity). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | | [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | | [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | | [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | | [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical | | [Workflow Task schedule-to-start latency severe](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | | [Activity schedule-to-start latency severe](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | | [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | > **📝 Note:** > > `worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. > It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance/metrics#slot-availability-metrics). > If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short. > ### Execution failures These fire when your Workflow or Activity code fails on the Worker. For triage, see [Execution failures](/troubleshooting/execution-failures). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | | [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | | [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | | [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | | [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | | [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `failure_reason=ActivityError`, `activity_type` | Rate above 10/s | 2m | Warning | | [Activity Payloads too large](/troubleshooting/execution-failures#activity-payloads-too-large) | `activity_execution_failed` | `failure_reason=PayloadsTooLarge`, `activity_type` | Any occurrence | 1m | Critical | | [Unregistered Activity invocation](/troubleshooting/execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | | [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | ## Route alerts by severity The severity in each table is a suggested routing default, not a property of the metric. - **Page on Critical.** These mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects. - **Send Warning to a channel someone reads during the day.** These mean something is degrading but Executions are still moving. Then adjust, because the defaults won't fit every deployment: - **Some Critical rows are early warnings, not confirmed stoppage.** High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. Downgrade the latency alerts if that describes you. - **Some Warning rows can still lose you data.** A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start. If that Activity isn't idempotent, treat it as Critical. One last thing worth knowing before you wire up routing: these alerts are chained together. Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. Expect them to fire in clusters, and check the triage pages to find which one is the cause rather than responding to all four. When several fire together, the triage pages identify which is the root cause and which are symptoms. --- # Child Workflows Source: https://docs.temporal.io/child-workflows > A Child Workflow Execution in the Temporal platform is initiated from another Workflow within the same Namespace. A Child Workflow Execution is a [Workflow Execution](/workflow-execution) that is spawned from within another Workflow in the same Namespace. - [Go SDK Child Workflow feature guide](/develop/go/workflows/child-workflows) - [Java SDK Child Workflow feature guide](/develop/java/workflows/child-workflows) - [PHP SDK Child Workflow feature guide](/develop/php/workflows/child-workflows) - [Python SDK Child Workflow feature guide](/develop/python/workflows/child-workflows) - [TypeScript SDK Child Workflow feature guide](/develop/typescript/workflows/child-workflows) - [.NET SDK Child Workflow feature guide](/develop/dotnet/workflows/child-workflows) - [Ruby SDK Child Workflow feature guide](/develop/ruby/workflows/child-workflows) - [Rust SDK Child Workflow feature guide](/develop/rust/workflows/child-workflows) A Workflow Execution can be both a Parent and a Child Workflow Execution because any Workflow can spawn another Workflow. ![Parent and Child Workflow Execution entity relationship](/diagrams/parent-child-workflow-execution-relationship.svg) A Parent Workflow Execution must await on the Child Workflow Execution to spawn. The Parent can optionally await on the result of the Child Workflow Execution. Consider the Child's [Parent Close Policy](/parent-close-policy) if the Parent does not await on the result of the Child, which includes any use of Continue-As-New by the Parent. > **📝 Note:** > > Child Workflows do not carry over when the Parent uses [Continue-As-New](/workflow-execution/continue-as-new). > This means that if a Parent Workflow Execution uses Continue-As-New, any ongoing Child Workflow Executions will not be retained in the new continued instance of the Parent. > When a Parent Workflow Execution reaches a Closed status, the Temporal Service propagates Cancellation Requests or Terminations to Child Workflow Executions depending on the Child's [Parent Close Policy](/parent-close-policy). If a Child Workflow Execution uses Continue-As-New, from the Parent Workflow Execution's perspective the entire chain of Runs is treated as a single execution. ![Parent and Child Workflow Execution entity relationship with Continue As New](/diagrams/parent-child-workflow-execution-with-continue-as-new.svg) ## When to use Child Workflows There is no reason to use Child Workflows just for code organization. You can use object oriented structure and other code organization techniques to deal with complexities. It is typically recommended to start from a single Workflow Definition if your problem has bounded size in terms of the number of Activity Executions and processed Signals. It is simpler than multiple asynchronously communicating Workflows. However, there are several valid reasons for using Child Workflows. ### Create a separate service Because a Child Workflow Execution can be processed by a completely separate set of [Workers](/workers#worker) than the Parent Workflow Execution, it can act as an entirely separate service. However, this also means that a Parent Workflow Execution and a Child Workflow Execution do not share any local state. As all Workflow Executions, they can communicate only via asynchronous [Signals](/sending-messages#sending-signals). ### Partition problems into smaller chunks An individual Workflow Execution has an [Event History](/workflow-execution/event#event-history) size limit, which imposes a couple of considerations for using Child Workflows. On one hand, because Child Workflow Executions have their own Event Histories, they are often used to partition large workloads into smaller chunks. For example, a single Workflow Execution does not have enough space in its Event History to spawn 100,000 [Activity Executions](/activity-execution). But a Parent Workflow Execution can spawn 1,000 Child Workflow Executions that each spawn 1,000 Activity Executions to achieve a total of 1,000,000 Activity Executions. However, because a Parent Workflow Execution Event History contains [Events](/workflow-execution/event#event) that correspond to the status of the Child Workflow Execution, a single Parent should not spawn more than 1,000 Child Workflow Executions. In general, however, Child Workflow Executions result in more overall Events recorded in Event Histories than Activities. Because each entry in an Event History is a _cost_ in terms of compute resources, this could become a factor in very large workloads. Therefore, we recommend starting with a single Workflow implementation that uses Activities until there is a clear need for Child Workflows. ### Represent a single resource As all Workflow Executions, a Child Workflow Execution can create a one to one mapping with a resource. It can be used to manage the resource using its ID to guarantee uniqueness. For example, a Workflow that manages host upgrades could spawn a Child Workflow Execution per host (hostname being a Workflow ID) and use them to ensure that all operations on the host are serialized. ### Periodic logic execution A Child Workflow can be used to execute some periodic logic without overwhelming the Parent Workflow Event History. In this scenario, the Parent Workflow starts a Child Workflow which executes periodic logic calling [Continue-As-New](/workflow-execution/continue-as-new) as many times as needed, then completes. From the Parent point of view, it is just a single Child Workflow invocation. ### Child Workflow versus an Activity Child Workflow Executions and Activity Executions are both started from Workflows, so you might feel confused about when to use which. Here are some important differences: - A Child Workflow has access to all Workflow APIs but is subject to the same [deterministic constraints](/workflow-definition#deterministic-constraints) as other Workflows. An Activity has the inverse pros and cons—no access to Workflow APIs but no Workflow constraints. - A Child Workflow Execution can continue on if its Parent is canceled with a [Parent Close Policy](/parent-close-policy) of `ABANDON`. An Activity Execution is _always_ canceled when its Workflow Execution is canceled. (It can react to a cancellation Signal for cleanup.) The decision is roughly analogous to spawning a child process in a terminal to do work versus doing work in the same process. - Temporal tracks all state changes within a Child Workflow Execution in Event History. Only the input, output, and retry attempts of an Activity Execution is tracked. A Workflow models composite operations that consist of multiple Activities or other Child Workflows. An Activity usually models a single operation on the external world. Our advice: **When in doubt, use an Activity.** --- # Temporal CLI Source: https://docs.temporal.io/cli The Temporal CLI (`temporal`) provides direct access to a Temporal Service via the terminal. Use it to manage, monitor, and debug Temporal applications, plus run an embedded development service when you need fast local feedback. In addition, we provide and maintain an official extension for the Temporal CLI you can use to interact with Temporal Cloud. Use the extension to manage your Temporal Cloud control plane resources, including Namespaces, Users, Service Accounts, API keys, and perform other operational and administrative tasks. ## Install and configure the CLI Install the CLI and cloud extension, run a local development server, and configure your environment in [Install and configure the CLI](/cli/setup-cli). ## Use with Temporal Cloud Connect to Temporal Cloud with [environment configuration](/develop/environment-configuration) and use the Cloud extension to manage your Temporal Cloud control plane resources in [Use with Temporal Cloud](/cli/cloud). ## Start a development server The CLI includes a local Temporal development service for fast feedback while building or testing your application. ```bash temporal server start-dev ``` ## CLI basics Get started with the basics of the CLI in [CLI basics](/cli/common-operations). ## Command reference Refer to the [command reference](/cli/command-reference) for the complete list of commands. --- # Use with Temporal Cloud Source: https://docs.temporal.io/cli/cloud > Use the Temporal CLI with Temporal Cloud to run Workflows, manage Namespaces, and administer your account. The Temporal CLI works with Temporal Cloud. The same commands you use for local or self-hosted Temporal services, such as `temporal workflow start` and `temporal workflow list`, work with Temporal Cloud as allowed by your role once you provide an address and credentials. For administrative tasks, [install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension). The extension adds `temporal cloud` commands for managing your Temporal Cloud account, including Namespaces, users, API keys, and Nexus endpoints. > **Public Preview** Access to Temporal Cloud is governed by role-based access control (RBAC). Your ability to perform actions, including running CLI commands against in Temporal Cloud is determined by the roles and permissions you have been assigned. Refer to the [Access control](/cloud/manage-access) page for more details. ## Connect to Temporal Cloud To connect the CLI to Temporal Cloud, provide the Temporal service address, Namespace name, and credentials. Temporal Cloud supports three credential types: - OAuth tokens obtained through the `temporal cloud login` command (requires the [Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension)) - API keys - mTLS certificates ### Interactive login The `temporal cloud login` command opens a browser to authenticate with Temporal Cloud using OAuth. Provide a profile name to store credentials in. If no profile is specified, credentials are stored in the `default` profile. ```bash temporal cloud login --profile prod ``` Complete the interactive login process in your browser. After login, your OAuth token is stored in the specified configuration profile. To confirm your login, run the `temporal cloud whoami` command: ```bash temporal cloud whoami --profile prod ``` Run commands against Temporal Cloud by specifying the profile, address, and Namespace: ```bash temporal workflow list --profile prod \ --address
\ --namespace ``` ### Non-interactive login For AI agents, CI pipelines, scripts, and other non-interactive environments, use API keys or mTLS certificates. Store credentials in a [configuration profile](/develop/environment-configuration#cli-integration) or set them as [environment variables](/cli/setup-cli#environment-variables) to avoid passing them on every command. To pass credentials inline: ```bash # Using an API key temporal workflow list \ --address ..tmprl.cloud:7233 \ --namespace . \ --api-key # Using mTLS certificates temporal workflow list \ --address ..tmprl.cloud:7233 \ --namespace . \ --tls-cert-path /path/to/client.pem \ --tls-key-path /path/to/client.key ``` ### Log out To log out, run the `temporal cloud logout` command. ```bash temporal cloud logout --profile prod ``` This will remove the OAuth token from the specified configuration profile. If you provided API keys or mTLS certificates, they will remain in the profile. ## Cloud administration The Temporal Cloud extension adds `temporal cloud` commands for managing your Temporal Cloud control plane resources in your Temporal Cloud account, including Namespaces, Users, Service Accounts, API keys, and others. Any of the authentication methods above grants access to these commands. The extension enables you to do the following through the CLI: - Create, configure, and delete Namespaces. - Create and manage API keys for programmatic access. - Invite users, assign roles, and manage user groups. - Create and configure Nexus endpoints. - View account information and manage connectivity rules. For installation instructions, see [Install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension). For the full list of commands, see the [`cloud` command reference](/cli/command-reference/cloud). ## Next steps - [CLI basics](/cli/common-operations) for common CLI commands. - [Environment configuration](/develop/environment-configuration) for managing connection profiles across environments. - [Cloud command reference](/cli/command-reference/cloud) for all `temporal cloud` commands. --- # Temporal CLI command reference Source: https://docs.temporal.io/cli/command-reference > Complete command reference for the Temporal CLI, including the cloud extension. This section includes the complete command reference for the `temporal` CLI, including the cloud extension. - [activity](/cli/command-reference/activity) - [batch](/cli/command-reference/batch) - [cloud](/cli/command-reference/cloud) - [config](/cli/command-reference/config) - [env](/cli/command-reference/env) - [operator](/cli/command-reference/operator) - [schedule](/cli/command-reference/schedule) - [server](/cli/command-reference/server) - [task-queue](/cli/command-reference/task-queue) - [worker](/cli/command-reference/worker) - [workflow](/cli/command-reference/workflow) --- # Temporal CLI activity command reference Source: https://docs.temporal.io/cli/command-reference/activity > Learn how to use Temporal Activity commands to perform operations on Activity Executions. This page provides a reference for the `temporal` CLI `activity` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## cancel Request cancellation of a Standalone Activity. ``` temporal activity cancel \ --activity-id YourActivityId ``` Requesting cancellation transitions the Activity's run state to CancelRequested. If the Activity is heartbeating, a cancellation error will be raised when the next heartbeat response is received; if the Activity allows this error to propagate, the Activity transitions to canceled status. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--reason` | No | **string** Reason for cancellation. | | `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. | ## complete Complete an Activity, marking it as successfully finished. Specify the Activity ID and include a JSON result for the returned value: ``` temporal activity complete \ --activity-id YourActivityId \ --workflow-id YourWorkflowId \ --result '{"YourResultKey": "YourResultVal"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. This may be the ID of an Activity invoked by a Workflow, or of a Standalone Activity. | | `--result` | Yes | **string** Result `JSON` to return. | | `--run-id`, `-r` | No | **string** Run ID. For workflow Activities (when --workflow-id is provided), this is the Workflow Run ID. For Standalone Activities, this is the Activity Run ID. | | `--workflow-id`, `-w` | No | **string** Workflow ID. Required for workflow Activities. Omit for Standalone Activities. | ## count Return a count of Standalone Activities. Use `--query` to filter the activities to be counted. ``` temporal activity count \ --query 'ActivityType="YourActivity"' ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and queries. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--query`, `-q` | No | **string** Query to filter Activity Executions to count. | ## describe Display information about a Standalone Activity. ``` temporal activity describe \ --activity-id YourActivityId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--raw` | No | **bool** Print properties without changing their format. | | `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. | ## execute Start a new Standalone Activity and block until it completes. The result is output to stdout. ``` temporal activity execute \ --activity-id YourActivityId \ --type YourActivity \ --task-queue YourTaskQueue \ --start-to-close-timeout 30s \ --input '{"some-key": "some-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. | | `--headers` | No | **string[]** Temporal activity headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times. | | `--heartbeat-timeout` | No | **duration** Maximum time between successful Worker heartbeats. On expiry the current activity attempt fails. | | `--id-conflict-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID is currently running. Accepted values: Fail, UseExisting. | | `--id-reuse-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID exists and has completed. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--priority-key` | No | **int** Priority key (1-5, lower = higher priority). Default is 3 when not specified. | | `--retry-backoff-coefficient` | No | **float** Coefficient for calculating the next retry interval. Must be 1 or larger. | | `--retry-initial-interval` | No | **duration** Interval of the first retry. If "retry-backoff-coefficient" is 1.0, it is used for all retries. | | `--retry-maximum-attempts` | No | **int** Maximum number of attempts. Setting to 1 disables retries. Setting to 0 means unlimited attempts. | | `--retry-maximum-interval` | No | **duration** Maximum interval between retries. | | `--schedule-to-close-timeout` | No | **duration** Maximum time for the Activity Execution, including all retries. Either this or "start-to-close-timeout" is required. | | `--schedule-to-start-timeout` | No | **duration** Maximum time an Activity task can stay in a task queue before a Worker picks it up. On expiry it results in a non-retryable failure and no further attempts are scheduled. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. Can be passed multiple times. See https://docs.temporal.io/visibility. | | `--start-to-close-timeout` | No | **duration** Maximum time for a single Activity attempt. On expiry a new attempt may be scheduled if permitted by the retry policy and schedule-to-close timeout. Either this or "schedule-to-close-timeout" is required. | | `--static-details` | No | **string** Static Activity details for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ | | `--static-summary` | No | **string** Static Activity summary for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Activity task queue. | | `--type` | Yes | **string** Activity Type name. | ## fail Fail an Activity, marking it as having encountered an error: ``` temporal activity fail \ --activity-id YourActivityId \ --workflow-id YourWorkflowId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. This may be the ID of an Activity invoked by a Workflow, or of a Standalone Activity. | | `--detail` | No | **string** Failure detail (JSON). Attached as the failure details payload. | | `--reason` | No | **string** Failure reason. Attached as the failure message. | | `--run-id`, `-r` | No | **string** Run ID. For workflow Activities (when --workflow-id is provided), this is the Workflow Run ID. For Standalone Activities, this is the Activity Run ID. | | `--workflow-id`, `-w` | No | **string** Workflow ID. Required for workflow Activities. Omit for Standalone Activities. | ## list List Standalone Activities. Use `--query` to filter results. ``` temporal activity list \ --query 'ActivityType="YourActivity"' ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and queries. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--limit` | No | **int** Maximum number of Activity Executions to display. | | `--page-size` | No | **int** Maximum number of Activity Executions to fetch at a time from the server. | | `--query`, `-q` | No | **string** Query to filter the Activity Executions to list. | ## pause Pause an Activity. Not supported for Standalone Activities. If the Activity is not currently running (e.g. because it previously failed), it will not be run again until it is unpaused. However, if the Activity is currently running, it will run until the next time it fails, completes, or times out, at which point the pause will kick in. Pause does not stop or extend the Activity's Schedule-To-Close Timeout. A paused Activity can still time out. Use `temporal activity update-options` to extend timeout settings before a long pause. If the Activity is on its last retry attempt and fails, the failure will be returned to the caller, just as if the Activity had not been paused. Specify the Activity and Workflow IDs: ``` temporal activity pause \ --activity-id YourActivityId \ --workflow-id YourWorkflowId ``` To later unpause the activity, see [unpause](#unpause). You may also want to [reset](#reset) the activity to unpause it while also starting it from the beginning. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | No | **string** The Activity ID to pause. Required. | | `--identity` | No | **string** The identity of the user or client submitting this request. | | `--reason` | No | **string** Reason for pausing the Activity. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## reset Reset an activity. Not supported for Standalone Activities. This restarts the activity as if it were first being scheduled: the attempt count returns to one, its per-attempt timeouts are re-armed, and its [heartbeat details](#reset-heartbeats) are cleared. If the activity may be executing (i.e. it has not yet timed out), the reset will take effect the next time it fails, heartbeats, or times out. If is waiting for a retry (i.e. has failed or timed out), the reset will apply immediately. If the activity is already paused, it will be unpaused by default. You can specify `keep_paused` to prevent this. If the activity is paused and the `keep_paused` flag is not provided, it will be unpaused. If the activity is paused and `keep_paused` flag is provided - it will stay paused. Either `--activity-id` (with `--workflow-id`) or `--query` must be specified. ### Resetting activities that heartbeat Activities that heartbeat will receive a [Canceled failure](/references/failures#cancelled-failure) the next time they heartbeat after a reset. If, in your Activity, you need to do any cleanup when an Activity is reset, handle this error and then re-throw it when you've cleaned up. If the `reset_heartbeats` flag is set, the heartbeat details will also be cleared. Specify the Activity and Workflow IDs: ``` temporal activity reset \ --activity-id YourActivityId \ --workflow-id YourWorkflowId --keep-paused --reset-heartbeats ``` Activities can be reset in bulk with a visibility query list filter: ``` temporal activity reset \ --query 'WorkflowType="YourWorkflow"' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | No | **string** The Activity ID to reset. Mutually exclusive with `--query`. Requires `--workflow-id` to be specified. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--jitter` | No | **duration** The activity will reset at random a time within the specified duration. Can only be used with --query. | | `--keep-paused` | No | **bool** If the activity was paused, it will stay paused. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. Note: Using --query for batch activity operations is an experimental feature and may change in the future. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--reset-heartbeats` | No | **bool** Reset the Activity's heartbeats. | | `--restore-original-options` | No | **bool** Restore the original options of the activity. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## result Wait for a Standalone Activity to complete and output the result. ``` temporal activity result \ --activity-id YourActivityId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. | ## start Start a new Standalone Activity. Outputs the Activity ID and Run ID. ``` temporal activity start \ --activity-id YourActivityId \ --type YourActivity \ --task-queue YourTaskQueue \ --start-to-close-timeout 5m \ --input '{"some-key": "some-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. | | `--headers` | No | **string[]** Temporal activity headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times. | | `--heartbeat-timeout` | No | **duration** Maximum time between successful Worker heartbeats. On expiry the current activity attempt fails. | | `--id-conflict-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID is currently running. Accepted values: Fail, UseExisting. | | `--id-reuse-policy` | No | **string-enum** Policy for handling activity start when an Activity with the same ID exists and has completed. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--priority-key` | No | **int** Priority key (1-5, lower = higher priority). Default is 3 when not specified. | | `--retry-backoff-coefficient` | No | **float** Coefficient for calculating the next retry interval. Must be 1 or larger. | | `--retry-initial-interval` | No | **duration** Interval of the first retry. If "retry-backoff-coefficient" is 1.0, it is used for all retries. | | `--retry-maximum-attempts` | No | **int** Maximum number of attempts. Setting to 1 disables retries. Setting to 0 means unlimited attempts. | | `--retry-maximum-interval` | No | **duration** Maximum interval between retries. | | `--schedule-to-close-timeout` | No | **duration** Maximum time for the Activity Execution, including all retries. Either this or "start-to-close-timeout" is required. | | `--schedule-to-start-timeout` | No | **duration** Maximum time an Activity task can stay in a task queue before a Worker picks it up. On expiry it results in a non-retryable failure and no further attempts are scheduled. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. Can be passed multiple times. See https://docs.temporal.io/visibility. | | `--start-to-close-timeout` | No | **duration** Maximum time for a single Activity attempt. On expiry a new attempt may be scheduled if permitted by the retry policy and schedule-to-close timeout. Either this or "schedule-to-close-timeout" is required. | | `--static-details` | No | **string** Static Activity details for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ | | `--static-summary` | No | **string** Static Activity summary for human consumption in UIs. Uses standard Markdown formatting excluding images, HTML, and script tags. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Activity task queue. | | `--type` | Yes | **string** Activity Type name. | ## terminate Terminate a Standalone Activity. ``` temporal activity terminate \ --activity-id YourActivityId \ --reason YourReason ``` Activity code cannot see or respond to terminations. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | Yes | **string** Activity ID. | | `--reason` | No | **string** Reason for termination. Defaults to a message with the current user's name. | | `--run-id`, `-r` | No | **string** Activity Run ID. If not set, targets the latest run. | ## unpause Re-schedule a previously-paused Activity for execution. Not supported for Standalone Activities. If the Activity is not running and is past its retry timeout, it will be scheduled immediately. Otherwise, it will be scheduled after its retry timeout expires. Use `--reset-attempts` to reset the number of previous run attempts to zero. For example, if an Activity is near the maximum number of attempts N specified in its retry policy, `--reset-attempts` will allow the Activity to be retried another N times after unpausing. Use `--reset-heartbeat` to reset the Activity's heartbeats. Either `--activity-id` (with `--workflow-id`) or `--query` must be specified. Specify the Activity and Workflow IDs: ``` temporal activity unpause \ --activity-id YourActivityId \ --workflow-id YourWorkflowId --reset-attempts --reset-heartbeats ``` Activities can be unpaused in bulk via a visibility Query list filter: ``` temporal activity unpause \ --query 'TemporalPauseInfo IS NOT NULL' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | No | **string** The Activity ID to unpause. Mutually exclusive with `--query`. Requires `--workflow-id` to be specified. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--jitter` | No | **duration** The activity will start at random a time within the specified duration. Can only be used with --query. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. Note: Using --query for batch activity operations is an experimental feature and may change in the future. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--reset-attempts` | No | **bool** Reset the activity attempts. | | `--reset-heartbeats` | No | **bool** Reset the Activity's heartbeats. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## update-options Update the options of a running Activity that were passed into it from a Workflow. Updates are incremental, only changing the specified options. Not supported for Standalone Activities. For example: ``` temporal activity update-options \ --activity-id YourActivityId \ --workflow-id YourWorkflowId \ --task-queue NewTaskQueueName \ --schedule-to-close-timeout DURATION \ --schedule-to-start-timeout DURATION \ --start-to-close-timeout DURATION \ --heartbeat-timeout DURATION \ --retry-initial-interval DURATION \ --retry-maximum-interval DURATION \ --retry-backoff-coefficient NewBackoffCoefficient \ --retry-maximum-attempts NewMaximumAttempts ``` You may follow this command with `temporal activity reset`, and the new values will apply after the reset. Either `--activity-id` or `--query` must be specified. Activity options can be updated in bulk with a visibility query list filter: ``` temporal activity update-options \ --query 'WorkflowType="YourWorkflow"' \ --task-queue NewTaskQueueName ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--activity-id`, `-a` | No | **string** The Activity ID to update options. Mutually exclusive with `--query`. Requires `--workflow-id` to be specified. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--heartbeat-timeout` | No | **duration** Maximum permitted time between successful worker heartbeats. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. Note: Using --query for batch activity operations is an experimental feature and may change in the future. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--restore-original-options` | No | **bool** Restore the original options of the activity. | | `--retry-backoff-coefficient` | No | **float** Coefficient used to calculate the next retry interval. The next retry interval is previous interval multiplied by the backoff coefficient. Must be 1 or larger. | | `--retry-initial-interval` | No | **duration** Interval of the first retry. If retryBackoffCoefficient is 1.0 then it is used for all retries. | | `--retry-maximum-attempts` | No | **int** Maximum number of attempts. When exceeded the retries stop even if not expired yet. Setting this value to 1 disables retries. Setting this value to 0 means unlimited attempts(up to the timeouts). | | `--retry-maximum-interval` | No | **duration** Maximum interval between retries. Exponential backoff leads to interval increase. This value is the cap of the increase. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--schedule-to-close-timeout` | No | **duration** Indicates how long the caller is willing to wait for an activity completion. Limits how long retries will be attempted. | | `--schedule-to-start-timeout` | No | **duration** Limits time an activity task can stay in a task queue before a worker picks it up. This timeout is always non retryable, as all a retry would achieve is to put it back into the same queue. Defaults to the schedule-to-close timeout or workflow execution timeout if not specified. | | `--start-to-close-timeout` | No | **duration** Maximum time an activity is allowed to execute after being picked up by a worker. This timeout is always retryable. | | `--task-queue` | No | **string** Name of the task queue for the Activity. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI batch command reference Source: https://docs.temporal.io/cli/command-reference/batch > Use Temporal CLI to manage multiple Workflow Executions with Batch Jobs that can Cancel, Signal, or Terminate Workflows. Filter and monitor Batch Jobs effectively. This page provides a reference for the `temporal` CLI `batch` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## describe Show the progress of an ongoing batch job. Pass a valid job ID to display its information: ``` temporal batch describe \ --job-id YourJobId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--job-id` | Yes | **string** Batch job ID. | ## list Return a list of batch jobs on the Service or within a single Namespace. For example, list the batch jobs for "YourNamespace": ``` temporal batch list \ --namespace YourNamespace ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--limit` | No | **int** Maximum number of batch jobs to display. | ## terminate Terminate a batch job with the provided job ID. You must provide a reason for the termination. The Service stores this explanation as metadata for the termination event for later reference: ``` temporal batch terminate \ --job-id YourJobId \ --reason YourTerminationReason ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--job-id` | Yes | **string** Job ID to terminate. | | `--reason` | Yes | **string** Reason for terminating the batch job. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI cloud command reference Source: https://docs.temporal.io/cli/command-reference/cloud > Command reference for the temporal cloud extension. > **Public Preview** This section includes the command reference for the `temporal cloud` CLI extension. - [account](/cli/command-reference/cloud/account) - [apikey](/cli/command-reference/cloud/apikey) - [async-operation](/cli/command-reference/cloud/async-operation) - [connectivity](/cli/command-reference/cloud/connectivity) - [custom-role](/cli/command-reference/cloud/custom-role) - [login](/cli/command-reference/cloud/login) - [logout](/cli/command-reference/cloud/logout) - [namespace](/cli/command-reference/cloud/namespace) - [nexus](/cli/command-reference/cloud/nexus) - [project](/cli/command-reference/cloud/project) - [region](/cli/command-reference/cloud/region) - [service-account](/cli/command-reference/cloud/service-account) - [user](/cli/command-reference/cloud/user) - [user-group](/cli/command-reference/cloud/user-group) - [whoami](/cli/command-reference/cloud/whoami) --- # Temporal CLI cloud account command reference Source: https://docs.temporal.io/cli/command-reference/cloud/account > Account Management Commands > **Public Preview** Manage the Temporal Cloud account. This page provides a reference for the `temporal cloud account` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## audit-log Commands for working with account audit logs. ### audit-log list Returns a paginated list of audit logs for the account, optionally filtered by time range. Example: temporal cloud account audit-log get --page-size 50 temporal cloud account audit-log get --start-time 2024-01-01T00:00:00Z --end-time 2024-02-01T00:00:00Z Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--end-time` | No | **timestamp** Filter for logs before this UTC time (RFC3339 format, e.g. 2024-02-01T00:00:00Z). Defaults to current time. | | `--page-size` | No | **int** Number of logs to retrieve per page. Cannot exceed 1000. Defaults to 100. | | `--page-token` | No | **string** Page token from a previous response to retrieve the next page. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--start-time` | No | **timestamp** Filter for logs at or after this UTC time (RFC3339 format, e.g. 2024-01-01T00:00:00Z). Defaults to 30 days ago. | ### audit-log sink Commands for working with account audit log sinks. #### audit-log sink delete Delete an audit log sink for the account. This action is irreversible. Example: temporal cloud account audit-log sink delete --name my-sink Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the audit log sink to delete. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink disable Disable an audit log sink for the account. Example: temporal cloud account audit-log sink disable --name my-sink Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the audit log sink to disable. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink enable Enable an audit log sink for the account. Example: temporal cloud account audit-log sink enable --name my-sink Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the audit log sink to enable. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink get Returns the details of an audit log sink for the account. Example: temporal cloud account audit-log sink get --name my-sink Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--name` | Yes | **string** The name of the audit log sink to get. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink kinesis Commands for managing Kinesis-based audit log sinks. ##### audit-log sink kinesis create Create an account audit log sink that streams audit events to Amazon Kinesis. Temporal Cloud assumes the specified IAM role to write events to the Kinesis stream identified by the destination URI. Example: temporal cloud account audit-log sink kinesis create \ --name my-sink \ --role-name MyRole \ --destination-uri arn:aws:kinesis:us-east-1:123456789012:stream/MyStream \ --region us-east-1 Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--destination-uri` | Yes | **string** ARN of the Kinesis stream to deliver audit log events to. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** Name of the audit log sink. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). | | `--role-name` | Yes | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ##### audit-log sink kinesis update Update an existing Kinesis audit log sink. Only the flags you provide are changed; omitted string flags retain their current values. Example: temporal cloud account audit-log sink kinesis update \ --name my-sink \ --role-name NewRole Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--destination-uri` | No | **string** ARN of the Kinesis stream to deliver audit log events to. If omitted, the current value is kept. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** Name of the audit log sink to update. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | No | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). If omitted, the current value is kept. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--role-name` | No | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. If omitted, the current value is kept. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ##### audit-log sink kinesis validate Validate an audit log sink configuration against Amazon Kinesis without creating it. Use this to verify that the IAM role and Kinesis stream are correctly configured before creating or updating the sink. Example: temporal cloud account audit-log sink kinesis validate \ --name my-sink \ --role-name MyRole \ --destination-uri arn:aws:kinesis:us-east-1:123456789012:stream/MyStream \ --region us-east-1 Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--destination-uri` | Yes | **string** ARN of the Kinesis stream to deliver audit log events to. | | `--region` | Yes | **string** AWS region where the Kinesis stream is located (e.g. us-east-1). | | `--role-name` | Yes | **string** Name of the IAM role that Temporal Cloud assumes to write to the Kinesis stream. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink list Returns a paginated list of audit log sinks for the account. Example: temporal cloud account audit-log sink list temporal cloud account audit-log sink list --page-size 50 Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of sinks to retrieve per page. Cannot exceed 1000. Defaults to 100. | | `--page-token` | No | **string** Page token from a previous response to retrieve the next page. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### audit-log sink pubsub Commands for managing PubSub audit log sinks. ##### audit-log sink pubsub create Creates a new PubSub audit log sink for the account using Google Cloud Pub/Sub. Example: ``` temporal cloud account audit-log sink pubsub create \ --name my-sink \ --service-account-email my-sa@my-project.iam.gserviceaccount.com \ --topic-name my-topic ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the audit log sink. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | Yes | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. | | `--topic-name` | Yes | **string** The destination PubSub topic name where audit logs will be sent. | ##### audit-log sink pubsub update Updates an existing PubSub audit log sink for the account. Example: ``` temporal cloud account audit-log sink pubsub update \ --name my-sink \ --service-account-email new-sa@new-project.iam.gserviceaccount.com \ --topic-name new-topic ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the audit log sink to update. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | No | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. | | `--topic-name` | No | **string** The destination PubSub topic name where audit logs will be sent. | ##### audit-log sink pubsub validate Validates a PubSub audit log sink specification without creating or modifying any resources. Example: ``` temporal cloud account audit-log sink pubsub validate \ --name my-sink \ --service-account-email my-sa@my-project.iam.gserviceaccount.com \ --topic-name my-topic ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | Yes | **string** The email of the GCP service account that Temporal Cloud impersonates for writing records to the customer's PubSub topic (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. | | `--topic-name` | Yes | **string** The destination PubSub topic name where audit logs will be sent. | ## metrics Commands for managing the Temporal Cloud account metrics configuration. ### metrics cert-ca Commands for managing the CA certificates used to authenticate clients accessing the Temporal Cloud account metrics endpoint. #### metrics cert-ca create Add a CA certificate to the list of accepted client CA certificates for the Temporal Cloud account metrics endpoint. Example: temporal cloud account metrics cert-ca create --ca-certificate-file /path/to/cert.pem temporal cloud account metrics cert-ca create --ca-certificate \ Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. | | `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### metrics cert-ca delete Remove a CA certificate from the list of accepted client CA certificates for the Temporal Cloud account metrics endpoint. Example: temporal cloud account metrics cert-ca delete --ca-certificate-file /path/to/cert.pem temporal cloud account metrics cert-ca delete --ca-certificate \ Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. | | `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### metrics cert-ca list List the CA certificates accepted for authenticating clients accessing the Temporal Cloud account metrics endpoint. Example: temporal cloud account metrics cert-ca list Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud apikey command reference Source: https://docs.temporal.io/cli/command-reference/cloud/apikey > API Key Management Commands > **Public Preview** Commands for creating, listing, and managing Temporal Cloud API keys. API keys authenticate requests to the Temporal Cloud API. This page provides a reference for the `temporal cloud apikey` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## create-for-me Create a new API key owned by the currently authenticated user. The token is printed once on creation and cannot be retrieved again. Example: ``` temporal cloud apikey create-for-me --display-name "My Key" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional description for the API key. | | `--display-name` | Yes | **string** A human-readable display name for the API key. | | `--expiry-duration` | No | **duration** Expiry duration relative to now (e.g. 30d, 24h, 90m). Supports days (d), hours (h), minutes (m), and seconds (s). Mutually exclusive with --expiry-time. | | `--expiry-time` | No | **timestamp** Expiry time for the API key in RFC3339 format (e.g. 2025-12-31T00:00:00Z). Mutually exclusive with --expiry-duration. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create-for-service-account Create a new API key owned by the specified service account. The token is printed once on creation and cannot be retrieved again. Example: ``` temporal cloud apikey create-for-service-account --service-account-id my-sa-id --display-name "My Key" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional description for the API key. | | `--display-name` | Yes | **string** A human-readable display name for the API key. | | `--expiry-duration` | No | **duration** Expiry duration relative to now (e.g. 30d, 24h, 90m). Supports days (d), hours (h), minutes (m), and seconds (s). Mutually exclusive with --expiry-time. | | `--expiry-time` | No | **timestamp** Expiry time for the API key in RFC3339 format (e.g. 2025-12-31T00:00:00Z). Mutually exclusive with --expiry-duration. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account to create the API key for. | ## delete Delete a Temporal Cloud API key. This action is irreversible. Example: ``` temporal cloud apikey delete --key-id my-key-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key-id` | Yes | **string** The ID of the API key to delete. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## disable Disable a Temporal Cloud API key. Disabled keys cannot be used for authentication. Example: ``` temporal cloud apikey disable --key-id my-key-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key-id` | Yes | **string** The ID of the API key to disable. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## edit Open an API key configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Example: ``` temporal cloud apikey edit --key-id my-key-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key-id` | Yes | **string** The ID of the API key to edit. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## enable Enable a previously disabled Temporal Cloud API key. Example: ``` temporal cloud apikey enable --key-id my-key-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key-id` | Yes | **string** The ID of the API key to enable. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## get Retrieve the configuration and status of a Temporal Cloud API key. Example: ``` temporal cloud apikey get --key-id my-key-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--key-id` | Yes | **string** The ID of the API key to retrieve. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List API keys. Optionally filter by user ID, user email, or service account ID. At most one filter may be specified. Example: ``` temporal cloud apikey list temporal cloud apikey list --user-id my-user-id temporal cloud apikey list --service-account-id my-sa-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of API keys to return per page. | | `--page-token` | No | **string** Token for retrieving the next page of results. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | No | **string** Filter API keys by service account ID. Mutually exclusive with --user-id and --user-email. | | `--user-email` | No | **string** Filter API keys by user email. Mutually exclusive with --user-id and --service-account-id. | | `--user-id` | No | **string** Filter API keys by user ID. Mutually exclusive with --user-email and --service-account-id. | ## update Update an API key's display name, description, or disabled status. Only flags that are explicitly provided are changed. Example: ``` temporal cloud apikey update --key-id my-key-id --display-name "New Name" temporal cloud apikey update --key-id my-key-id --disabled=true ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** New description for the API key. | | `--disabled` | No | **bool** Set to true to disable the API key, or false to enable it. | | `--display-name` | No | **string** New display name for the API key. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key-id` | Yes | **string** The ID of the API key to update. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud async-operation command reference Source: https://docs.temporal.io/cli/command-reference/cloud/async-operation > Async Operation Commands > **Public Preview** Commands for working with Temporal Cloud async operations. This page provides a reference for the `temporal cloud async-operation` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## await Wait for a Temporal Cloud async operation to reach a terminal state. Polls the operation status until it completes, fails, or is cancelled. Example: ``` temporal cloud async-operation await --async-operation-id my-op-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async-operation-id` | Yes | **string** The ID of the async operation to wait for. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). Default is 1s. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## get Retrieve the status and details of a Temporal Cloud async operation. Example: ``` temporal cloud async-operation get --async-operation-id my-op-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async-operation-id` | Yes | **string** The ID of the async operation to retrieve. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud connectivity command reference Source: https://docs.temporal.io/cli/command-reference/cloud/connectivity > Connectivity Management Commands > **Public Preview** Commands for managing connectivity rules for Temporal Cloud. This page provides a reference for the `temporal cloud connectivity` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## delete Delete a connectivity rule by its ID. Example: ``` temporal cloud connectivity delete --id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--id` | Yes | **string** The ID of the connectivity rule. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## get Get details of a specific connectivity rule by its ID. Example: ``` temporal cloud connectivity get --id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--id` | Yes | **string** The ID of the connectivity rule. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List connectivity rules, optionally filtered by namespace. Example: ``` temporal cloud connectivity list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | No | **string** Filter connectivity rules by namespace (e.g., 'my-namespace.my-account'). | | `--page-size` | No | **int** Number of connectivity rules to return per page. | | `--page-token` | No | **string** Page token for pagination. | | `--project-id` | No | **string** Filter connectivity rules by project ID. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## private Commands for managing private connectivity rules. ### private create Create a new private connectivity rule for AWS, GCP, or Azure. For AWS, provide --connection-id (VPC endpoint ID) and --region. For GCP, provide --connection-id (PSC connection ID), --gcp-project-id, and --region. For Azure, provide --azure-pe-resource-id (ARM resource ID) and --region. Examples: ``` temporal cloud connectivity private create --connection-id vpce-12345 --region aws-us-west-2 temporal cloud connectivity private create \ --azure-pe-resource-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/privateEndpoints/{name} \ --region azure-eastus ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--azure-pe-resource-id` | No | **string** The ARM resource ID of the Azure Private Endpoint (only for Azure private connectivity). Example: `/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/privateEndpoints/{name}`. | | `--connection-id` | No | **string** The connection ID for private connectivity (AWS VPC endpoint ID or GCP PSC connection ID). | | `--gcp-project-id` | No | **string** The GCP project ID (only for GCP private connectivity). | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | No | **string** The ID of the project to create the connectivity rule in. | | `--region` | Yes | **string** The region for private connectivity. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## public Commands for managing public connectivity rules. ### public create Create a new public internet connectivity rule. Example: ``` temporal cloud connectivity public create ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--enable-stable-ips` | No | **bool** Connect the namespace via a predictable set of IPs on the public internet. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | No | **string** The ID of the project to create the connectivity rule in. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud custom-role command reference Source: https://docs.temporal.io/cli/command-reference/cloud/custom-role > Custom Role Management Commands > **Public Preview** Commands for managing Temporal Cloud custom roles. Custom roles enable fine-grained authorization by binding sets of permissions (resource + actions) to a named role that can be assigned to users, user groups, and service accounts. This page provides a reference for the `temporal cloud custom-role` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## apply Apply a custom role configuration to Temporal Cloud. Creates a new role if no role with the given name exists, or updates the existing one to match the specification. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example: ``` temporal cloud custom-role apply --spec @custom-role.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## create Create a new Temporal Cloud custom role from a JSON specification. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example with inline JSON: ``` temporal cloud custom-role create --spec '{"name":"reader","description":"...","permissions":[...]}' ``` Example with file path: ``` temporal cloud custom-role create --spec @custom-role.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | ## delete Delete a Temporal Cloud custom role. This action is irreversible. Example: ``` temporal cloud custom-role delete --role-id my-role-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--role-id` | Yes | **string** The ID of the custom role. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## edit Open a custom role configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Example: ``` temporal cloud custom-role edit --role-id my-role-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--role-id` | Yes | **string** The ID of the custom role. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## get Retrieve the configuration and status of a Temporal Cloud custom role. Example: ``` temporal cloud custom-role get --role-id my-role-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--role-id` | Yes | **string** The ID of the custom role. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List all Temporal Cloud custom roles accessible with the current authentication credentials. Example: ``` temporal cloud custom-role list ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of custom roles to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## update Update an existing Temporal Cloud custom role from a JSON specification. Replaces the role's spec with the provided one. Example: ``` temporal cloud custom-role update --role-id my-role-id --spec @custom-role.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--role-id` | Yes | **string** The ID of the custom role. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** Custom role specification in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud login command reference Source: https://docs.temporal.io/cli/command-reference/cloud/login > Login > **Public Preview** Authenticate with Temporal Cloud using browser-based OAuth login. This command opens your default browser to complete authentication. Once logged in, your credentials are stored locally for subsequent commands. Example: ``` temporal cloud login ``` For headless environments, use --disable-pop-up and follow the printed URL. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--audience` | No | **string** OAuth audience parameter for token generation. | | `--client-id` | No | **string** OAuth client identifier for authentication. | | `--domain` | No | **string** Authentication domain for the OAuth provider. | | `--redirect-url` | No | **string** Redirect URL for OAuth authentication flow. | | `--reset` | No | **bool** Clear stored login credentials and configuration, then re-authenticate. Use this if you need to switch accounts or fix authentication issues. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud logout command reference Source: https://docs.temporal.io/cli/command-reference/cloud/logout > Logout > **Public Preview** Log out from Temporal Cloud by clearing stored authentication tokens and credentials from the local configuration. Example: ``` temporal cloud logout ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--domain` | No | **string** Authentication domain for the OAuth provider. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud namespace command reference Source: https://docs.temporal.io/cli/command-reference/cloud/namespace > Namespace Management Commands > **Public Preview** Commands for creating, updating, and managing Temporal Cloud namespaces. Namespaces provide isolation for workflows and activities. Each namespace has its own configuration including retention period, region, and access controls. This page provides a reference for the `temporal cloud namespace` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## api-key Commands for managing API key authentication configuration of Temporal Cloud namespaces. ### api-key disable Disable API key authentication for a Temporal Cloud namespace. Example: ``` temporal cloud namespace api-key disable --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### api-key enable Enable API key authentication for a Temporal Cloud namespace. Example: ``` temporal cloud namespace api-key enable --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### api-key get Retrieve the current API key authentication configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace api-key get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## apply Apply a namespace configuration to Temporal Cloud. Creates a new namespace if it doesn't exist, or updates an existing one to match the specification. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example with inline JSON: ``` temporal cloud namespace apply --spec '{"name": "namespace-name", "region": "us-west-2", "retention_days": 7}' ``` Example with file path: ``` temporal cloud namespace apply --spec @namespace-spec.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the namespace already matches the specification. Without this flag, the command errors when no changes are needed. | | `--project-id` | No | **string** The ID of the project to create the namespace in. If omitted, the namespace is created in the account's default project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** Namespace configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## capacity Commands for managing the capacity of Temporal Cloud namespaces. Capacity controls whether a namespace runs in on-demand mode or provisioned mode (with a fixed TRU allocation). ### capacity get Retrieve capacity information for a Temporal Cloud namespace, including the current mode (on-demand or provisioned), mode options, and recent usage stats. Example: ``` temporal cloud namespace capacity get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### capacity update Update the capacity of a Temporal Cloud namespace. Choose either on-demand mode or provisioned mode (with a fixed TRU allocation). Example (switch to on-demand): ``` temporal cloud namespace capacity update --namespace my-namespace.my-account --capacity-mode on_demand ``` Example (switch to provisioned with 4 TRUs): ``` temporal cloud namespace capacity update --namespace my-namespace.my-account --capacity-mode provisioned --capacity-value 4 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--capacity-mode` | Yes | **string-enum** Capacity mode for the namespace. Must be either 'on_demand' or 'provisioned'. Accepted values: on_demand, provisioned. | | `--capacity-value` | No | **float** The provisioned capacity in Temporal Resource Units (TRUs). Required and must be greater than 0 when --capacity-mode is 'provisioned'. Ignored when --capacity-mode is 'on_demand'. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## codec Commands for managing the codec server configuration of Temporal Cloud namespaces. The codec server is used to encode and decode payloads for workflows and activities. ### codec delete Delete the codec server configuration from a Temporal Cloud namespace. Example: ``` temporal cloud namespace codec delete --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### codec get Retrieve the current codec server configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace codec get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### codec set Set the codec server configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace codec set --namespace my-namespace.my-account --endpoint https://my-codec.example.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-error-message-default-link` | No | **string** A link to display alongside the custom error message for remote codec server errors. | | `--custom-error-message-default-message` | No | **string** A custom message to display for remote codec server errors. | | `--endpoint` | Yes | **string** The codec server endpoint URL. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--include-cross-origin-credentials` | No | **bool** Whether to include cross-origin credentials in requests to the codec server. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--pass-access-token` | No | **bool** Whether to pass the user access token to the codec server endpoint. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## connectivity Commands for attaching and detaching connectivity rules on a Temporal Cloud namespace. Use 'cloud connectivity' to manage the rules themselves. ### connectivity attach Attach an existing connectivity rule to a Temporal Cloud namespace. Example: ``` temporal cloud namespace connectivity attach \ --namespace my-namespace.my-account \ --connectivity-rule-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--connectivity-rule-id` | Yes | **string[]** The ID of a connectivity rule to attach. Repeat to attach multiple. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### connectivity detach Detach a connectivity rule from a Temporal Cloud namespace. Example: ``` temporal cloud namespace connectivity detach \ --namespace my-namespace.my-account \ --connectivity-rule-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--connectivity-rule-id` | Yes | **string[]** The ID of a connectivity rule to detach. Repeat to detach multiple. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### connectivity list List all connectivity rules currently attached to a Temporal Cloud namespace. Example: ``` temporal cloud namespace connectivity list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create Create a new Temporal Cloud namespace with the specified configuration. Options are passed as individual flags. To create or update a namespace using a full JSON specification, use 'namespace apply' instead. Example: ``` temporal cloud namespace create --name my-namespace --region aws-us-east-1 --retention-days 30 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--api-key-auth-enabled` | No | **bool** Enable API key authentication for the namespace. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. | | `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. | | `--certificate-filter` | No | **string[]** Certificate filter as a JSON object (e.g. `'{"commonName":"foo"}'`). Repeat to add multiple. | | `--certificate-filter-file` | No | **string** Path to a JSON file containing a certificate filter object. | | `--codec-endpoint` | No | **string** HTTPS codec server endpoint URL. | | `--codec-include-cross-origin-credentials` | No | **bool** Include cross-origin credentials in codec server requests. | | `--codec-pass-access-token` | No | **bool** Pass the user access token to the codec server endpoint. | | `--connection-rule-id` | No | **string[]** Private connectivity rule ID. Repeat to specify multiple. | | `--description` | No | **string** The description is a human-readable description of the namespace. Must be at most 255 printable ASCII characters plus whitespace. Optional, default is empty. | | `--enable-delete-protection` | No | **bool** Prevent accidental deletion of this namespace. | | `--enable-task-queue-fairness` | No | **bool** Enable task queue fairness for the namespace. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--mtls-auth-enabled` | No | **bool** Enable mTLS authentication for the namespace. | | `--name`, `-n` | Yes | **string** The name for the new namespace (becomes part of the namespace ID). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | No | **string** The ID of the project to create the namespace in. If omitted, the namespace is created in the account's default project. | | `--region` | Yes | **string[]** Cloud region where the namespace will be hosted. Repeat to specify multiple regions for High Availability (e.g. --region aws-us-east-1 --region aws-us-west-2). | | `--retention-days` | No | **int** Number of days to retain closed workflow history. If not specified, the server default applies. | | `--search-attribute` | No | **string[]** Custom search attribute as 'name=Type' (e.g. --search-attribute myAttr=Keyword). Valid types: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. Repeat to add multiple. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## delete Delete a Temporal Cloud namespace and all associated data. This action is irreversible and will permanently remove all workflows, activities, and history within the namespace. Example: ``` temporal cloud namespace delete --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the namespace does not exist. Without this flag, the command errors if the namespace is not found. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## description Commands for viewing and updating the description of a Temporal Cloud namespace. The description is a human-readable description of the namespace. Must be at most 255 printable ASCII characters plus whitespace. Optional, default is empty. ### description get Retrieve the current description for a Temporal Cloud namespace. The description is a human-readable description of the namespace. Example: ``` temporal cloud namespace description get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### description set Set the description for a Temporal Cloud namespace without changing other namespace settings. The description is a human-readable description of the namespace. Must be at most 255 printable ASCII characters plus whitespace. Pass an empty string to clear the description. Example: ``` temporal cloud namespace description set --namespace my-namespace.my-account --value "Updated namespace description" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--value` | Yes | **string** New description for the namespace. Must be at most 255 printable ASCII characters plus whitespace. Pass an empty string to clear it. | ## edit Open a namespace configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Example: ``` temporal cloud namespace edit --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if no changes were made in the editor. Without this flag, the command errors when the configuration is unchanged. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## export Commands for managing workflow history export sinks for Temporal Cloud namespaces. Export sinks define destinations (S3, GCS, or Azure Blob) to which workflow history is exported. ### export azure-blob Commands for managing Azure Blob workflow history export sinks for Temporal Cloud namespaces. #### export azure-blob create Create a new Azure Blob workflow history export sink for a Temporal Cloud namespace. The sink is created in the enabled state. Example: ``` temporal cloud namespace export azure-blob create --namespace my-namespace.my-account --sink-name my-sink \ --tenant-id 11111111-1111-1111-1111-111111111111 \ --subscription-id 22222222-2222-2222-2222-222222222222 \ --resource-group my-resource-group --storage-account my-storage-account \ --container-name my-container --region eastus ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--container-name` | Yes | **string** The name of the destination Azure Blob container where Temporal will send data. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The region where the Azure storage account is located. | | `--resource-group` | Yes | **string** The Azure resource group that contains the storage account. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | | `--storage-account` | Yes | **string** The name of the destination Azure storage account where Temporal will send data. | | `--subscription-id` | Yes | **string** The Azure subscription ID that contains the storage account. | | `--tenant-id` | Yes | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. | #### export azure-blob update Update the configuration of an existing Azure Blob workflow history export sink. Only the flags you provide are changed; omitted flags keep their current values. The enabled/disabled state and region are also preserved. Example (rotate storage account only): ``` temporal cloud namespace export azure-blob update --namespace my-namespace.my-account --sink-name my-sink \ --storage-account my-new-storage-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--container-name` | No | **string** The name of the destination Azure Blob container where Temporal will send data. If omitted, the current value is kept. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-group` | No | **string** The Azure resource group that contains the storage account. If omitted, the current value is kept. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | | `--storage-account` | No | **string** The name of the destination Azure storage account where Temporal will send data. If omitted, the current value is kept. | | `--subscription-id` | No | **string** The Azure subscription ID that contains the storage account. If omitted, the current value is kept. | | `--tenant-id` | No | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. If omitted, the current value is kept. | #### export azure-blob validate Validate an Azure Blob workflow history export sink configuration without creating or updating it. A successful response means the configuration is valid. Example: ``` temporal cloud namespace export azure-blob validate --namespace my-namespace.my-account --sink-name my-sink \ --tenant-id 11111111-1111-1111-1111-111111111111 \ --subscription-id 22222222-2222-2222-2222-222222222222 \ --resource-group my-resource-group --storage-account my-storage-account \ --container-name my-container --region eastus ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--container-name` | Yes | **string** The name of the destination Azure Blob container where Temporal will send data. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--region` | Yes | **string** The region where the Azure storage account is located. | | `--resource-group` | Yes | **string** The Azure resource group that contains the storage account. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | | `--storage-account` | Yes | **string** The name of the destination Azure storage account where Temporal will send data. | | `--subscription-id` | Yes | **string** The Azure subscription ID that contains the storage account. | | `--tenant-id` | Yes | **string** The Azure tenant ID where the storage account exists and where Temporal's app registration is consented/granted access. | ### export delete Delete a workflow history export sink from a Temporal Cloud namespace. Example: ``` temporal cloud namespace export delete --namespace my-namespace.my-account --sink-name my-sink ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | ### export disable Disable a workflow history export sink for a Temporal Cloud namespace. The sink configuration is preserved and can be re-enabled later. Example: ``` temporal cloud namespace export disable --namespace my-namespace.my-account --sink-name my-sink ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | ### export enable Enable a previously disabled workflow history export sink for a Temporal Cloud namespace. Example: ``` temporal cloud namespace export enable --namespace my-namespace.my-account --sink-name my-sink ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | ### export gcs Commands for managing GCS workflow history export sinks for Temporal Cloud namespaces. #### export gcs create Create a new GCS workflow history export sink for a Temporal Cloud namespace. The sink is created in the enabled state. Example: ``` temporal cloud namespace export gcs create --namespace my-namespace.my-account --sink-name my-sink \ --service-account-email my-service-account@my-project.iam.gserviceaccount.com \ --bucket-name my-bucket --region us-central1 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--bucket-name` | Yes | **string** The name of the destination GCS bucket. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The GCS bucket region. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | Yes | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. | | `--sink-name` | Yes | **string** The name of the export sink. | #### export gcs update Update the configuration of an existing GCS workflow history export sink. Only the flags you provide are changed; omitted flags keep their current values. The enabled/disabled state and region are also preserved. Example (rotate service account only): ``` temporal cloud namespace export gcs update --namespace my-namespace.my-account --sink-name my-sink \ --service-account-email my-new-service-account@my-project.iam.gserviceaccount.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--bucket-name` | No | **string** The name of the destination GCS bucket. If omitted, the current value is kept. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | No | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. If omitted, the current value is kept. | | `--sink-name` | Yes | **string** The name of the export sink. | #### export gcs validate Validate a GCS workflow history export sink configuration without creating or updating it. A successful response means the configuration is valid. Example: ``` temporal cloud namespace export gcs validate --namespace my-namespace.my-account --sink-name my-sink \ --service-account-email my-service-account@my-project.iam.gserviceaccount.com \ --bucket-name my-bucket --region us-central1 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--bucket-name` | Yes | **string** The name of the destination GCS bucket. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--region` | Yes | **string** The GCS bucket region. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-email` | Yes | **string** The email of the customer service account that Temporal Cloud impersonates for writing to GCS (e.g. my-sa@my-project.iam.gserviceaccount.com). The service account ID and GCP project ID are parsed from this email. | | `--sink-name` | Yes | **string** The name of the export sink. | ### export get Retrieve the configuration and status of a workflow history export sink for a Temporal Cloud namespace. Example: ``` temporal cloud namespace export get --namespace my-namespace.my-account --sink-name my-sink ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | ### export list List all workflow history export sinks configured for a Temporal Cloud namespace. Example: ``` temporal cloud namespace export list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### export s3 Commands for managing S3 workflow history export sinks for Temporal Cloud namespaces. #### export s3 create Create a new S3 workflow history export sink for a Temporal Cloud namespace. The sink is created in the enabled state. Example: ``` temporal cloud namespace export s3 create --namespace my-namespace.my-account --sink-name my-sink \ --role-arn arn:aws:iam::123456789012:role/my-role --bucket-name my-bucket \ --region us-east-1 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--bucket-name` | Yes | **string** The name of the destination S3 bucket. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. Optional. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The AWS region where the S3 bucket is located. | | `--role-arn` | Yes | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | #### export s3 update Update the configuration of an existing S3 workflow history export sink. Only the flags you provide are changed; omitted flags keep their current values. The enabled/disabled state and region are also preserved. Example (rotate IAM role only): ``` temporal cloud namespace export s3 update --namespace my-namespace.my-account --sink-name my-sink \ --role-arn arn:aws:iam::123456789012:role/my-new-role ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--bucket-name` | No | **string** The name of the destination S3 bucket. If omitted, the current value is kept. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. If omitted, the current value is kept. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--role-arn` | No | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. If omitted, the current value is kept. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | #### export s3 validate Validate an S3 workflow history export sink configuration without creating or updating it. A successful response means the configuration is valid. Example: ``` temporal cloud namespace export s3 validate --namespace my-namespace.my-account --sink-name my-sink \ --role-arn arn:aws:iam::123456789012:role/my-role --bucket-name my-bucket \ --region us-east-1 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--bucket-name` | Yes | **string** The name of the destination S3 bucket. | | `--kms-arn` | No | **string** The AWS KMS key ARN for server-side encryption of exported data. Optional. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--region` | Yes | **string** The AWS region where the S3 bucket is located. | | `--role-arn` | Yes | **string** The IAM role ARN that Temporal Cloud assumes for writing to S3 (e.g. arn:aws:iam::123456789012:role/my-role). The role name and AWS account ID are parsed from this ARN. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--sink-name` | Yes | **string** The name of the export sink. | ## fairness Commands for managing task queue fairness configuration of Temporal Cloud namespaces. ### fairness get Retrieve the current task queue fairness configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace fairness get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### fairness set Set the task queue fairness configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace fairness set --namespace my-namespace.my-account --enable-task-queue-fairness=true ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--enable-task-queue-fairness` | Yes | **bool** Enable or disable task queue fairness for the namespace. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## get Retrieve the configuration and status of a Temporal Cloud namespace. Returns details including region, retention period, endpoints, and certificate information. Example: ``` temporal cloud namespace get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | No | **bool** Output only the namespace specification in JSON format, omitting metadata and status information. | ## ha Commands for managing High Availability (HA) settings of Temporal Cloud namespaces. HA settings control active region, managed failover, and replica regions. ### ha failover Trigger a failover for a Temporal Cloud namespace to a different region. The target region must already be a replica region of the namespace. Example: ``` temporal cloud namespace ha failover --namespace my-namespace.my-account --region aws-us-west-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The target region to failover to (e.g., aws-us-west-2). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### ha get Retrieve the current High Availability configuration for a Temporal Cloud namespace. Shows the active region, whether managed failover is enabled, and whether passive poller forwarding is enabled. Example: ``` temporal cloud namespace ha get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### ha region Commands for managing replica regions of Temporal Cloud namespaces. #### ha region add Add a replica region to a Temporal Cloud namespace. The region will be added as a passive replica and can later be used for failover. Example: ``` temporal cloud namespace ha region add --namespace my-namespace.my-account --region aws-us-west-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The region ID to add as a replica (e.g., aws-us-west-2). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### ha region delete Remove a replica region from a Temporal Cloud namespace. Note that a 7-day cooldown period applies before the same region can be re-added. Example: ``` temporal cloud namespace ha region delete --namespace my-namespace.my-account --region aws-us-west-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--region` | Yes | **string** The region ID to remove (e.g., aws-us-west-2). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### ha region list List all regions and their states for a Temporal Cloud namespace. Example: ``` temporal cloud namespace ha region list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### ha update Update the High Availability configuration for a Temporal Cloud namespace. Use --auto-failover to enable or disable Temporal-managed automatic failover. Use --passive-poller-forwarding to enable or disable passive poller forwarding. Example: ``` temporal cloud namespace ha update \ --namespace my-namespace.my-account \ --auto-failover enabled \ --passive-poller-forwarding disabled ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--auto-failover` | No | **string-enum** Enable or disable Temporal-managed automatic failover for the namespace. Accepted values: enabled, disabled. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--passive-poller-forwarding` | No | **string-enum** Enable or disable passive poller forwarding for the namespace. Accepted values: enabled, disabled. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## lifecycle Commands for managing lifecycle settings of Temporal Cloud namespaces. Lifecycle settings control the behavior and protection of namespaces, including delete protection to prevent accidental deletion. ### lifecycle get Retrieve the current lifecycle configuration for a Temporal Cloud namespace. Lifecycle settings include delete protection status. Example: ``` temporal cloud namespace lifecycle get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### lifecycle set Set the lifecycle configuration for a Temporal Cloud namespace. Lifecycle settings include delete protection to prevent accidental deletion. Example: ``` temporal cloud namespace lifecycle set --namespace my-namespace.my-account --enable-delete-protection true ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--enable-delete-protection` | Yes | **bool** Enable or disable delete protection for the namespace. When enabled, the namespace cannot be deleted until this flag is set to false. | | `--idempotent` | No | **bool** Succeed silently if the lifecycle configuration is already set to the specified value. Without this flag, the command errors when no change is needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--resource-version` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## list List all Temporal Cloud namespaces accessible with the current authentication credentials. Example: ``` temporal cloud namespace list ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--name` | No | **string** Filter namespaces by the name as defined in the specification of the namespace. | | `--page-size` | No | **int** Number of namespaces to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | No | **string** Filter namespaces by project ID. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## mtls Commands for managing mTLS authentication configuration of Temporal Cloud namespaces. ### mtls cert-ca Commands for managing the client CA certificates of Temporal Cloud namespaces. #### mtls cert-ca create Add client CA certificates to a Temporal Cloud namespace from a PEM file or base64 encoded string. These certificates are used to verify client connections and enable mTLS authentication. Specify either --ca-certificate-file or --ca-certificate, but not both. Example with file: ``` temporal cloud namespace mtls cert-ca create --namespace my-namespace.my-account --ca-certificate-file ca-cert.pem ``` Example with base64 encoded data: ``` temporal cloud namespace mtls cert-ca create --namespace my-namespace.my-account --ca-certificate ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. | | `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### mtls cert-ca delete Delete client CA certificates from a Temporal Cloud namespace. This operation requires confirmation and will remove the specified certificates from the namespace configuration. Specify either --ca-certificate-file or --ca-certificate, but not both. Example with file: ``` temporal cloud namespace mtls cert-ca delete --namespace my-namespace.my-account --ca-certificate-file ca-cert.pem ``` Example with base64 encoded data: ``` temporal cloud namespace mtls cert-ca delete --namespace my-namespace.my-account --ca-certificate ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--ca-certificate` | No | **string** Base64-encoded CA certificate for mTLS authentication. Mutually exclusive with --ca-certificate-file. | | `--ca-certificate-file` | No | **string** Path to a CA certificate PEM file for mTLS authentication. Mutually exclusive with --ca-certificate. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### mtls cert-ca list Retrieve the list of client CA certificates configured for a Temporal Cloud namespace. These certificates are used for client authentication. Example: ``` temporal cloud namespace mtls cert-ca list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### mtls cert-filter Commands for managing certificate filters for Temporal Cloud namespaces. Certificate filters restrict mTLS connections to client certificates with specific distinguished name properties. #### mtls cert-filter create Add new certificate filters to a Temporal Cloud namespace. Certificate filters restrict mTLS connections to client certificates whose distinguished name properties match at least one of the filters. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--common-name` | No | **string** The common name (CN) field from the certificate's distinguished name. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--organization` | No | **string** The organization (O) field from the certificate's distinguished name. | | `--organizational-unit` | No | **string** The organizational unit (OU) field from the certificate's distinguished name. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--subject-alternative-name` | No | **string** The subject alternative name (SAN) from the certificate. | #### mtls cert-filter delete Delete certificate filters from a Temporal Cloud namespace. Filters are matched by exact field equality. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--common-name` | No | **string** The common name (CN) field from the certificate's distinguished name. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--organization` | No | **string** The organization (O) field from the certificate's distinguished name. | | `--organizational-unit` | No | **string** The organizational unit (OU) field from the certificate's distinguished name. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--subject-alternative-name` | No | **string** The subject alternative name (SAN) from the certificate. | #### mtls cert-filter list List all certificate filters configured for a Temporal Cloud namespace. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### mtls disable Disable mTLS authentication for a Temporal Cloud namespace. Example: ``` temporal cloud namespace mtls disable --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### mtls enable Enable mTLS authentication for a Temporal Cloud namespace. Example: ``` temporal cloud namespace mtls enable --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### mtls get Retrieve the current mTLS authentication configuration for a Temporal Cloud namespace. Example: ``` temporal cloud namespace mtls get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## retention Commands for managing data retention settings of Temporal Cloud namespaces. Retention determines how long closed workflow history data are stored before being automatically deleted. ### retention get Retrieve the current data retention period for a Temporal Cloud namespace. The retention period defines how long closed workflow history data are stored. Example: ``` temporal cloud namespace retention get --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### retention set Set the data retention period for a Temporal Cloud namespace. The retention period defines how long closed workflow history data are stored. Example: ``` temporal cloud namespace retention set --namespace my-namespace.my-account --retention-days 14 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--retention-days` | Yes | **int** New retention period in days for closed workflow history data. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## search-attribute Commands for managing custom search attributes for Temporal Cloud namespaces. Search attributes enable filtering and searching workflows by custom fields. ### search-attribute create Create a new custom search attribute for a Temporal Cloud namespace. Example: ``` temporal cloud namespace search-attribute create --namespace my-namespace.my-account --name MyField --type Keyword ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the search attribute to create. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--type` | Yes | **string** The type of the search attribute. Valid values: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. | ### search-attribute list List all custom search attributes configured for a Temporal Cloud namespace. Example: ``` temporal cloud namespace search-attribute list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### search-attribute rename Rename an existing custom search attribute for a Temporal Cloud namespace. This operation preserves all existing data associated with the search attribute. Example: ``` temporal cloud namespace search-attribute rename --namespace my-namespace.my-account --existing-name OldField --new-name NewField ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--existing-name` | Yes | **string** The current name of the search attribute to rename. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--new-name` | Yes | **string** The new name for the search attribute. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## service-account Commands for inspecting the service accounts that have access to a Temporal Cloud namespace. ### service-account list List the service accounts that have access to a Temporal Cloud namespace, including both directly-assigned and inherited access. Example: ``` temporal cloud namespace service-account list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## tag Commands for managing tags of Temporal Cloud namespaces. Tags are key-value pairs used for organization and categorization of namespaces. ### tag create Create a new tag for a Temporal Cloud namespace. Fails if a tag with the specified key already exists. Example: ``` temporal cloud namespace tag create --namespace my-namespace.my-account --key environment --value production ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key` | Yes | **string** The tag key. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--value` | Yes | **string** The tag value. | ### tag delete Delete a tag from a Temporal Cloud namespace by its key. Example: ``` temporal cloud namespace tag delete --namespace my-namespace.my-account --key environment ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key` | Yes | **string** The tag key to delete. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### tag list List all tags configured for a Temporal Cloud namespace. Example: ``` temporal cloud namespace tag list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### tag update Update the value of an existing tag for a Temporal Cloud namespace. Fails if the specified tag key does not exist. Example: ``` temporal cloud namespace tag update --namespace my-namespace.my-account --key environment --value staging ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--key` | Yes | **string** The tag key to update. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--value` | Yes | **string** The new value for the tag. | ## user Commands for inspecting the users that have access to a Temporal Cloud namespace. ### user list List the users that have access to a Temporal Cloud namespace, including both directly-assigned and inherited access. Example: ``` temporal cloud namespace user list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--page-size` | No | **int** Number of users to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## user-group Commands for inspecting the user groups that have access to a Temporal Cloud namespace. ### user-group list List the user groups that have access to a Temporal Cloud namespace, including both directly-assigned and inherited access. Example: ``` temporal cloud namespace user-group list --namespace my-namespace.my-account ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--namespace`, `-n` | Yes | **string** The fully qualified namespace name in the format 'namespace.account' (e.g., 'my-namespace.my-account'). | | `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud nexus command reference Source: https://docs.temporal.io/cli/command-reference/cloud/nexus > Nexus Operations Management Commands > **Public Preview** Commands for managing Nexus Operations in Temporal Cloud. This page provides a reference for the `temporal cloud nexus` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## endpoint Commands for managing Nexus Endpoints in Temporal Cloud. ### endpoint allowed-namespace Commands for managing allowed namespaces for Nexus Endpoints. #### endpoint allowed-namespace add Add namespaces that are allowed to call this Nexus Endpoint. Namespaces that are already allowed are silently ignored. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the Nexus Endpoint. | | `--namespace` | Yes | **string[]** A namespace to allow. Can be specified multiple times. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### endpoint allowed-namespace list List all namespaces that are allowed to call this Nexus Endpoint. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--name` | Yes | **string** The name of the Nexus Endpoint. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### endpoint allowed-namespace remove Remove namespaces from the list of allowed callers of this Nexus Endpoint. Namespaces that are not currently allowed are silently ignored. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the Nexus Endpoint. | | `--namespace` | Yes | **string[]** A namespace to remove. Can be specified multiple times. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | #### endpoint allowed-namespace set Set the full list of namespaces that are allowed to call this Nexus Endpoint, replacing any previously allowed namespaces. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the Nexus Endpoint. | | `--namespace` | Yes | **string[]** A namespace to allow. Can be specified multiple times. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### endpoint create Create a new Nexus Endpoint on the Cloud Account. An endpoint name is used in workflow code to invoke Nexus operations. The endpoint target is a worker and `--target-namespace` and `--target-task-queue` must both be provided. This will fail if an endpoint with the same name is already registered. Example: ``` temporal cloud nexus endpoint create --name my-endpoint --target-namespace my-ns.my-account --target-task-queue my-tq ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--allow-namespace` | No | **string[]** A namespace that is allowed to call this endpoint. Can be specified multiple times. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional endpoint description in markdown format. | | `--description-file` | No | **string** Path to a file containing an endpoint description in markdown format. Mutually exclusive with --description. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the Nexus Endpoint to create. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | No | **string** The ID of the project to create the Nexus Endpoint in. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--target-namespace` | Yes | **string** The namespace in which a handler worker will be polling for Nexus tasks. | | `--target-task-queue` | Yes | **string** The task queue on which a handler worker will be polling for Nexus tasks. | ### endpoint delete Delete a Nexus Endpoint on the Cloud Account. Specify either `--name` or `--id` (exactly one is required). Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--id` | No | **string** The ID of the Nexus Endpoint to delete. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | No | **string** The name of the Nexus Endpoint to delete. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### endpoint get Get a Nexus Endpoint configuration from the Cloud Account. Specify either `--name` or `--id` (exactly one is required). Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--id` | No | **string** The ID of the Nexus Endpoint to retrieve. | | `--name` | No | **string** The name of the Nexus Endpoint to retrieve. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### endpoint list List Nexus Endpoint configurations on the Cloud Account. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of endpoints to return per page. If no page size is provided, it will default to 100. A maximum of 1000 endpoints can be fetched at a time. | | `--page-token` | No | **string** Token for retrieving the next page of results. Initial value is empty string. | | `--project-id` | No | **string** Filter Nexus Endpoints by project ID. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### endpoint update Update an existing Nexus Endpoint on the Cloud Account. An endpoint name is used in workflow code to invoke Nexus operations. Specify either `--name` or `--id` to identify the endpoint (exactly one is required). The endpoint is patched leaving any existing fields for which flags are not provided as they were. Example: ``` temporal cloud nexus endpoint update --name my-endpoint --target-namespace new-ns.my-account --target-task-queue new-tq ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional endpoint description in markdown format. | | `--description-file` | No | **string** Path to a file containing an endpoint description in markdown format. Mutually exclusive with --description. | | `--id` | No | **string** The ID of the Nexus Endpoint to update. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | No | **string** The name of the Nexus Endpoint to update. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--target-namespace` | No | **string** The namespace in which a handler worker will be polling for Nexus tasks. | | `--target-task-queue` | No | **string** The task queue on which a handler worker will be polling for Nexus tasks. | | `--unset-description` | No | **bool** Unset the endpoint description. Cannot be used with --description or --description-file. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud project command reference Source: https://docs.temporal.io/cli/command-reference/cloud/project > Project Management Commands > **Public Preview** Commands for managing Temporal Cloud projects. Projects provide an account-level grouping for Temporal Cloud resources. This page provides a reference for the `temporal cloud project` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## apply Apply a project configuration to Temporal Cloud. An existing project with the specification's display_name is updated if found; if no matching project exists, a new project is created. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example: ``` temporal cloud project apply --spec '{"display_name": "Engineering", "description": "Engineering workloads"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** Project configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## create Create a new Temporal Cloud project. Example: ``` temporal cloud project create --display-name "Engineering" --description "Engineering workloads" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** A description of the project. | | `--display-name` | Yes | **string** The display name of the project. | | `--enable-delete-protection` | No | **bool** Prevent the project from being deleted while enabled. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## delete Delete a Temporal Cloud project. Example: ``` temporal cloud project delete --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## edit Open an existing project specification in your default editor and apply the edited configuration. Example: ``` temporal cloud project edit --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## get Retrieve the configuration and status of a Temporal Cloud project. Example: ``` temporal cloud project get --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--project-id` | Yes | **string** The ID of the project. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List Temporal Cloud projects in the current account. Example: ``` temporal cloud project list --page-size 50 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Maximum number of projects to return. | | `--page-token` | No | **string** Token for retrieving the next page of results. | | `--project-id` | No | **string[]** Filter by project ID. Can be specified multiple times. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## service-account Commands for inspecting the service accounts that have access to a Temporal Cloud project. ### service-account list List the service accounts that have access to a Temporal Cloud project, including both directly-assigned and inherited access. Example: ``` temporal cloud project service-account list --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | Yes | **string** The ID of the project. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## update Update an existing Temporal Cloud project. Only explicitly provided flags are changed. Example: ``` temporal cloud project update --project-id my-project-id --display-name "Platform" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** A description of the project. | | `--display-name` | No | **string** The display name of the project. | | `--enable-delete-protection` | No | **bool** Prevent the project from being deleted while enabled. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## user Commands for inspecting the users that have access to a Temporal Cloud project. ### user list List the users that have access to a Temporal Cloud project, including both directly-assigned and inherited access. Example: ``` temporal cloud project user list --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of users to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | Yes | **string** The ID of the project. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## user-group Commands for inspecting the user groups that have access to a Temporal Cloud project. ### user-group list List the user groups that have access to a Temporal Cloud project, including both directly-assigned and inherited access. Example: ``` temporal cloud project user-group list --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | Yes | **string** The ID of the project. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud region command reference Source: https://docs.temporal.io/cli/command-reference/cloud/region > Region Commands > **Public Preview** Commands for listing Temporal Cloud regions. This page provides a reference for the `temporal cloud region` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## get Get details for a specific Temporal Cloud region. Example: ``` temporal cloud region get --region aws-us-east-1 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--region`, `-r` | Yes | **string** The ID of the region to get. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List all available Temporal Cloud regions. Example: ``` temporal cloud region list ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud service-account command reference Source: https://docs.temporal.io/cli/command-reference/cloud/service-account > Service Account Management Commands > **Public Preview** Commands for managing Temporal Cloud service accounts. This page provides a reference for the `temporal cloud service-account` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## create Create a new Temporal Cloud service account with account-level access. Optionally assign an account role, namespace-level permissions, and project-level roles. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Example: ``` temporal cloud service-account create --name my-sa --account-role developer \ --namespace-access my-namespace.my-account=write \ --project-access my-project-id=write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. | | `--description` | No | **string** An optional description for the service account. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create-namespace-scoped Create a new Temporal Cloud service account scoped to a single namespace. Example: ``` temporal cloud service-account create-namespace-scoped --name my-sa \ --namespace my-namespace.my-account --namespace-permission write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional description for the service account. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. | | `--namespace` | Yes | **string** The namespace to scope the service account to. | | `--namespace-permission` | Yes | **string** The permission to grant on the namespace. Valid values: admin, write, read. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create-project-scoped Create a new Temporal Cloud service account scoped to a single project. Optionally assign namespace-level permissions within the project. Project roles: admin, write, read, list, contribute, member. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Example: ``` temporal cloud service-account create-project-scoped --name my-sa \ --project-id my-project-id --project-role write \ --namespace-access my-namespace.my-account=read ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--description` | No | **string** An optional description for the service account. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | Yes | **string** The name of the service account. Must be unique across all active service accounts. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project to scope the service account to. | | `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## delete Delete a Temporal Cloud service account. This action is irreversible. Example: ``` temporal cloud service-account delete --service-account-id my-sa-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account to delete. | ## edit Open a service account configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Example: ``` temporal cloud service-account edit --service-account-id my-sa-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account to edit. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## get Retrieve the configuration and status of a Temporal Cloud service account. Example: ``` temporal cloud service-account get --service-account-id my-sa-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account to retrieve. | ## list List all Temporal Cloud service accounts accessible with the current authentication credentials. Example: ``` temporal cloud service-account list temporal cloud service-account list --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--page-size` | No | **int** Number of service accounts to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | No | **string** The ID of the project to list project-scoped service accounts for. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## remove-project-access Remove a Temporal Cloud service account's direct project-level access. Example: ``` temporal cloud service-account remove-project-access --service-account-id my-sa-id \ --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account. | ## set-custom-roles Set the custom roles assigned to a Temporal Cloud service account. Replaces the service account's current custom role list. Pass no --custom-role flags to remove all custom roles. Not valid for namespace-scoped service accounts. Example: ``` temporal cloud service-account set-custom-roles --service-account-id my-sa-id \ --custom-role role-id-1 --custom-role role-id-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account. | ## set-project-access Set project-level access for a Temporal Cloud service account. Project roles: admin, write, read, list, contribute, member. Example: ``` temporal cloud service-account set-project-access --service-account-id my-sa-id \ --project-id my-project-id --project-role write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account. | ## update Update a Temporal Cloud service account. Only flags that are explicitly provided are changed. For account-scoped service accounts, use --account-role, --namespace-access, and/or --project-access. For namespace-scoped service accounts, use --namespace-permission. For project-scoped service accounts, use --project-role and/or --namespace-access. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Use 'namespace=' (empty permission) to remove access to a namespace. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Use 'project-id=' (empty role) to remove access to a project. Project-scoped service account roles: admin, write, read, list, contribute, member. Example: ``` temporal cloud service-account update --service-account-id my-sa-id --name new-name temporal cloud service-account update --service-account-id my-sa-id --account-role admin temporal cloud service-account update --service-account-id my-sa-id --project-access my-project-id=write temporal cloud service-account update --service-account-id my-sa-id --project-role write temporal cloud service-account update --service-account-id my-sa-id --namespace-permission write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. Only valid for account-scoped service accounts. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. | | `--description` | No | **string** New description for the service account. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--name` | No | **string** New name for the service account. | | `--namespace-access` | No | **string[]** Namespace access to set, in the format 'namespace=permission'. Use 'namespace=' to remove access. Permission must be one of: admin, write, read. Can be repeated. Only valid for account-scoped and project-scoped service accounts. | | `--namespace-permission` | No | **string** The permission to grant on the scoped namespace. Valid values: admin, write, read. Only valid for namespace-scoped service accounts. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to set, in the format 'project-id=role'. Use 'project-id=' to remove access. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. Only valid for account-scoped service accounts. | | `--project-role` | No | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. Only valid for project-scoped service accounts. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--service-account-id` | Yes | **string** The ID of the service account to update. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud user command reference Source: https://docs.temporal.io/cli/command-reference/cloud/user > User Management Commands > **Public Preview** Commands for managing Temporal Cloud users. This page provides a reference for the `temporal cloud user` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## apply Apply a user configuration to Temporal Cloud. Creates a new user invitation if the email does not exist, or updates the existing user to match the specification. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example with inline JSON: ``` temporal cloud user apply --spec '{"email": "alice@example.com", "access": {"account_access": {"role": "developer"}}}' ``` Example with file path: ``` temporal cloud user apply --spec @user-spec.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** User configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## delete Delete a Temporal Cloud user. This action is irreversible. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user delete --user-id my-user-id temporal cloud user delete --user-email alice@example.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## edit Open a user configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user edit --user-id my-user-id temporal cloud user edit --user-email alice@example.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## get Retrieve the configuration and status of a Temporal Cloud user. Example: ``` temporal cloud user get --user-id my-user-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## invite Invite a user to Temporal Cloud by email. Optionally assign an account-level role, namespace-level access permissions, and project-level access roles. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Example: ``` temporal cloud user invite --email alice@example.com --account-role developer \ --namespace-access my-namespace.my-account=write \ --project-access my-project-id=write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. | | `--email` | Yes | **string** The email address of the user to invite. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List all Temporal Cloud users accessible with the current authentication credentials. Example: ``` temporal cloud user list ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--email` | No | **string** Filter users by email address. | | `--namespace` | No | **string** Filter users by the namespace they have access to. | | `--page-size` | No | **int** Number of users to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | No | **string** List users with access to the project ID. Cannot be combined with --email or --namespace. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## remove-project-access Remove a Temporal Cloud user's direct project-level access. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user remove-project-access --user-id my-user-id \ --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## set-account-role Set the account-level role for a Temporal Cloud user. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user set-account-role --user-id my-user-id --account-role developer temporal cloud user set-account-role --user-email alice@example.com --account-role admin ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | Yes | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## set-custom-roles Set the custom roles assigned to a Temporal Cloud user. Replaces the user's current custom role list. Pass no --custom-role flags to remove all custom roles. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user set-custom-roles --user-email alice@example.com \ --custom-role role-id-1 --custom-role role-id-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## set-namespace-permissions Add, update, or remove namespace-level permissions for a Temporal Cloud user. Changes are applied additively: namespaces not listed are left unchanged. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. To remove access to a namespace, pass an empty permission: 'namespace='. Specify the user with either --user-id or --user-email (not both). Example: ``` # Grant write access to my-namespace and read access to other-namespace: temporal cloud user set-namespace-permissions --user-id my-user-id \ --namespace-access my-namespace.my-account=write \ --namespace-access other-namespace.my-account=read # Remove access to a namespace: temporal cloud user set-namespace-permissions --user-id my-user-id \ --namespace-access my-namespace.my-account= ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | Yes | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## set-project-access Set project-level access for a Temporal Cloud user. Project roles: admin, write, read, list, contribute, member. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user set-project-access --user-id my-user-id \ --project-id my-project-id --project-role write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud user-group command reference Source: https://docs.temporal.io/cli/command-reference/cloud/user-group > User Group Management Commands > **Public Preview** Commands for managing Temporal Cloud user groups. This page provides a reference for the `temporal cloud user-group` commands. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## apply Apply a user group configuration to Temporal Cloud. Creates a new user group if no group with the given display name exists, or updates the existing one to match the specification. The specification can be provided as inline JSON or loaded from a file by prefixing the path with '@'. Example with inline JSON: ``` temporal cloud user-group apply --spec '{"display_name": "Engineering", "cloud_group": {}, "access": {"account_access": {"role": "developer"}}}' ``` Example with file path: ``` temporal cloud user-group apply --spec @user-group-spec.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--spec` | Yes | **string** User group configuration in JSON format. Provide inline JSON directly, or use '@path/to/file.json' to load from a file. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## create-cloud-group Create a new Temporal Cloud-managed user group. Members can be managed using the add-member and remove-member commands. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Example: ``` temporal cloud user-group create-cloud-group --display-name "Engineering" \ --account-role developer \ --namespace-access my-namespace.my-account=write \ --project-access my-project-id=write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. | | `--display-name` | Yes | **string** The display name of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create-google-group Create a new user group backed by a Google Group. Members are managed via the Google Group itself. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Example: ``` temporal cloud user-group create-google-group --display-name "Platform" \ --google-group-email platform@example.com \ --account-role developer \ --project-access my-project-id=write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. | | `--display-name` | Yes | **string** The display name of the user group. | | `--google-group-email` | Yes | **string** The email address of the Google Group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## create-scim-group Create a new user group backed by a SCIM identity provider group. Members are managed via the upstream identity provider. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. Project access format: 'project-id=role' where role is one of: admin, write, read, list, contribute, member. Example: ``` temporal cloud user-group create-scim-group --display-name "Security" \ --scim-idp-id idp-group-id-123 \ --account-role read \ --project-access my-project-id=read ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. | | `--display-name` | Yes | **string** The display name of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | No | **string[]** Namespace access to grant, in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access to grant, in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. | | `--scim-idp-id` | Yes | **string** The identity provider ID for the SCIM group. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## delete Delete a Temporal Cloud user group. This action is irreversible. Example: ``` temporal cloud user-group delete --group-id my-group-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## edit Open a user group configuration in your default editor for interactive modification. After saving and closing the editor, the changes are applied to Temporal Cloud. The editor is determined by the EDITOR environment variable, falling back to 'vi' if not set. Example: ``` temporal cloud user-group edit --group-id my-group-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--verbose-diff` | No | **bool** Show detailed differences between the current and desired namespace configurations when changes are detected. | ## get Retrieve the configuration and status of a Temporal Cloud user group. Example: ``` temporal cloud user-group get --group-id my-group-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--group-id` | Yes | **string** The ID of the user group. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## list List all Temporal Cloud user groups accessible with the current authentication credentials. Example: ``` temporal cloud user-group list ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--display-name` | No | **string** Filter user groups by display name. | | `--google-group-email-address` | No | **string** Filter user groups by Google group email address. | | `--namespace` | No | **string** Filter user groups by the namespace they have access to. | | `--page-size` | No | **int** Number of user groups to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--project-id` | No | **string** List user groups with access to the project ID. Cannot be combined with --namespace, --display-name, --google-group-email-address, or --scim-group-idp-id. | | `--scim-group-idp-id` | No | **string** Filter user groups by SCIM group IDP ID. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## members Commands for managing members of Temporal Cloud user groups. ### members add Add a user to a Temporal Cloud user group. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user-group members add --group-id my-group-id --user-id my-user-id temporal cloud user-group members add --group-id my-group-id --user-email alice@example.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ### members list List all members of a Temporal Cloud user group. Example: ``` temporal cloud user-group members list --group-id my-group-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--group-id` | Yes | **string** The ID of the user group. | | `--page-size` | No | **int** Number of members to return per page. Use for paginated results. | | `--page-token` | No | **string** Token for retrieving the next page of results in a paginated list. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ### members remove Remove a user from a Temporal Cloud user group. Specify the user with either --user-id or --user-email (not both). Example: ``` temporal cloud user-group members remove --group-id my-group-id --user-id my-user-id temporal cloud user-group members remove --group-id my-group-id --user-email alice@example.com ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | | `--user-email` | No | **string** The email address of the user. Mutually exclusive with --user-id. | | `--user-id` | No | **string** The ID of the user. Mutually exclusive with --user-email. | ## remove-project-access Remove a Temporal Cloud user group's direct project-level access. Example: ``` temporal cloud user-group remove-project-access --group-id my-group-id \ --project-id my-project-id ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## set-account-role Set the account-level role for a Temporal Cloud user group. Account roles: owner, admin, developer, finance-admin, read, metrics-read. Example: ``` temporal cloud user-group set-account-role --group-id my-group-id --account-role developer ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | Yes | **string** The account-level role to assign. Valid values: owner, admin, developer, finance-admin, read, metrics-read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## set-custom-roles Set the custom roles assigned to a Temporal Cloud user group. Replaces the group's current custom role list. Pass no --custom-role flags to remove all custom roles. Example: ``` temporal cloud user-group set-custom-roles --group-id my-group-id \ --custom-role role-id-1 --custom-role role-id-2 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## set-namespace-permissions Add, update, or remove namespace-level permissions for a Temporal Cloud user group. Changes are applied additively: namespaces not listed are left unchanged. Namespace access format: 'namespace=permission' where permission is one of: admin, write, read. To remove access to a namespace, pass an empty permission: 'namespace='. Example: ``` temporal cloud user-group set-namespace-permissions --group-id my-group-id \ --namespace-access my-namespace.my-account=write \ --namespace-access other-namespace.my-account=read ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | Yes | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## set-project-access Set project-level access for a Temporal Cloud user group. Project roles: admin, write, read, list, contribute, member. Example: ``` temporal cloud user-group set-project-access --group-id my-group-id \ --project-id my-project-id --project-role write ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-id` | Yes | **string** The ID of the project. | | `--project-role` | Yes | **string** The project-level role to assign. Valid values: admin, write, read, list, contribute, member. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## update Update an existing Temporal Cloud user group's access settings. Provide at least one of --account-role, --namespace-access, --project-access, or --custom-role. Example: ``` temporal cloud user-group update --group-id my-group-id --account-role developer temporal cloud user-group update --group-id my-group-id \ --namespace-access my-namespace.my-account=write temporal cloud user-group update --group-id my-group-id \ --project-access my-project-id=write temporal cloud user-group update --group-id my-group-id --account-role admin \ --namespace-access my-namespace.my-account=write \ --namespace-access other-namespace.my-account=read \ --project-access my-project-id=read ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--account-role` | No | **string** The account role to assign to the group. Role must be one of: admin, developer, finance-admin, read. | | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--async` | No | **bool** Return immediately after initiating the operation instead of waiting for completion. Use the returned operation ID to check status later. | | `--async-operation-id` | No | **string** Custom identifier for tracking this async operation. If not provided, a unique ID is generated automatically. | | `--custom-role` | No | **string[]** Custom role ID to assign. Repeat to assign multiple. When provided, replaces the existing custom role list. | | `--group-id` | Yes | **string** The ID of the user group. | | `--idempotent` | No | **bool** Succeed silently if the resource already exists or matches the specification. Without this flag, the command errors when no changes are needed. | | `--namespace-access` | No | **string[]** Namespace access change in the format 'namespace=permission'. Permission must be one of: admin, write, read. Can be repeated. Use an empty permission (e.g. 'testns=') to remove access to a namespace. Changes are additive: namespaces not listed are left unchanged. | | `--poll-interval` | No | **duration** Time to wait between status checks when waiting for operation completion. Cannot be greater than 10 minutes. Supports minutes (m) and seconds (s). | | `--project-access` | No | **string[]** Project access change in the format 'project-id=role'. Role must be one of: admin, write, read, list, contribute, member. Can be repeated. Use an empty role (e.g. 'project-id=') to remove access to a project. Changes are additive: projects not listed are left unchanged. | | `--resource-version`, `-v` | No | **string** Resource version for optimistic concurrency control. If not provided, the current version is fetched automatically. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI cloud whoami command reference Source: https://docs.temporal.io/cli/command-reference/cloud/whoami > Who Am I > **Public Preview** Display information about the currently authenticated identity. Shows whether you are authenticated as a user or service account, along with the associated API key if one is in use. Example: ``` temporal cloud whoami ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--api-key` | No | **string** API key for authenticating with Temporal Cloud. Can be used instead of interactive login for automation and CI/CD pipelines. | | | `--auto-confirm` | No | **bool** Automatically confirm prompts and actions that require user confirmation. Useful for scripting and automation. | | | `--config-dir` | No | **string** Directory path where CLI configuration files are stored, including authentication tokens and settings. | | | `--disable-pop-up` | No | **bool** Prevent the CLI from opening a browser window during authentication. Useful for headless environments or when using alternative auth methods. | | | `--server` | No | **string** Override the Temporal Cloud API server address. Used for connecting to non-production environments. | `saas-api.tmprl.cloud:443` | --- # Temporal CLI config command reference Source: https://docs.temporal.io/cli/command-reference/config > Temporal CLI 'config' commands allow the getting, setting, deleting, and listing of configuration properties for connecting to Temporal. This page provides a reference for the `temporal` CLI `config` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## delete Remove a property within a profile. ``` temporal config delete \ --prop tls.client_cert_path ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--prop`, `-p` | Yes | **string** Specific property to delete. If unset, deletes entire profile. | ## delete-profile Remove a full profile entirely. The `--profile` must be set explicitly. ``` temporal config delete-profile \ --profile my-profile ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ## get Display specific properties or the entire profile. ``` temporal config get \ --prop address ``` or ``` temporal config get ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--prop`, `-p` | No | **string** Specific property to get. | ## list List profile names in the config file. ``` temporal config list ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ## set Assign a value to a property and store it in the config file: ``` temporal config set \ --prop address \ --value us-west-2.aws.api.temporal.io:7233 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--prop`, `-p` | Yes | **string** Property name. | | `--value`, `-v` | Yes | **string** Property value. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI env command reference Source: https://docs.temporal.io/cli/command-reference/env > Temporal CLI 'env' commands allow the configuration, setting, deleting, and listing of environmental properties, making it easy to manage Temporal Server instances. This page provides a reference for the `temporal` CLI `env` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## delete Remove a presets environment entirely _or_ remove a key-value pair within an environment. If you don't specify an environment (with `--env` or by setting the `TEMPORAL_ENV` variable), this command updates the "default" environment: ``` temporal env delete \ --env YourEnvironment ``` or ``` temporal env delete \ --env prod \ --key tls-key-path ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--key`, `-k` | No | **string** Property name. | ## get List the properties for a given environment: ``` temporal env get \ --env YourEnvironment ``` Print a single property: ``` temporal env get \ --env YourEnvironment \ --key YourPropertyKey ``` If you don't specify an environment (with `--env` or by setting the `TEMPORAL_ENV` variable), this command lists properties of the "default" environment. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--key`, `-k` | No | **string** Property name. | ## list List the environments you have set up on your local computer. Environments are stored in "$HOME/.config/temporalio/temporal.yaml". Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ## set Assign a value to a property key and store it to an environment: ``` temporal env set \ --env environment \ --key property \ --value value ``` If you don't specify an environment (with `--env` or by setting the `TEMPORAL_ENV` variable), this command sets properties in the "default" environment. Storing keys with CLI option names lets the CLI automatically set those options for you. This reduces effort and helps avoid typos when issuing commands. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--key`, `-k` | No | **string** Property name (required). | | `--value`, `-v` | No | **string** Property value (required). | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI operator command reference Source: https://docs.temporal.io/cli/command-reference/operator > Operator commands in Temporal allow actions on Namespaces, Search Attributes, Clusters and Nexus Endpoints using specific subcommands. Execute with "temporal operator [command] [subcommand] [options]". This page provides a reference for the `temporal` CLI `operator` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## cluster Perform operator actions on Temporal Services (also known as Clusters). ``` temporal operator cluster [subcommand] [options] ``` For example to check Service/Cluster health: ``` temporal operator cluster health ``` ### describe View information about a Temporal Cluster (Service), including Cluster Name, persistence store, and visibility store. Add `--detail` for additional info: ``` temporal operator cluster describe [--detail] ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--detail` | No | **bool** Show history shard count and Cluster/Service version information. | ### health View information about the health of a Temporal Service: ``` temporal operator cluster health ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### list Print a list of remote Temporal Clusters (Services) registered to the local Service. Report details include the Cluster's name, ID, address, History Shard count, Failover version, and availability: ``` temporal operator cluster list [--limit max-count] ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--limit` | No | **int** Maximum number of Clusters to display. | ### remove Remove a registered remote Temporal Cluster (Service) from the local Service. ``` temporal operator cluster remove \ --name YourClusterName ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name` | Yes | **string** Cluster/Service name. | ### system Show Temporal Server information for Temporal Clusters (Service): Server version, scheduling support, and more. This information helps diagnose problems with the Temporal Server. The command defaults to the local Service. Otherwise, use the `--frontend-address` option to specify a Cluster (Service) endpoint: ``` temporal operator cluster system \ --frontend-address "YourRemoteEndpoint:YourRemotePort" ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### upsert Add, remove, or update a registered ("remote") Temporal Cluster (Service). ``` temporal operator cluster upsert [options] ``` For example: ``` temporal operator cluster upsert \ --frontend-address "YourRemoteEndpoint:YourRemotePort" --enable-connection false ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--enable-connection` | No | **bool** Set the connection to "enabled". | | `--enable-replication` | No | **bool** Set the replication to "enabled". | | `--frontend-address` | Yes | **string** Remote endpoint. | ## namespace Manage Temporal Cluster (Service) Namespaces: ``` temporal operator namespace [command] [command options] ``` For example: ``` temporal operator namespace create \ --namespace YourNewNamespaceName ``` ### create Create a new Namespace on the Temporal Service: ``` temporal operator namespace create \ --namespace YourNewNamespaceName \ [options] ``` Create a Namespace with multi-region data replication: ``` temporal operator namespace create \ --global \ --namespace YourNewNamespaceName ``` Configure settings like retention and Visibility Archival State as needed. For example, the Visibility Archive can be set on a separate URI: ``` temporal operator namespace create \ --retention 5d \ --visibility-archival-state enabled \ --visibility-uri YourURI \ --namespace YourNewNamespaceName ``` Note: URI values for archival states can't be changed once enabled. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--active-cluster` | No | **string** Active Cluster (Service) name. | | `--cluster` | No | **string[]** Cluster (Service) names for Namespace creation. Can be passed multiple times. | | `--data` | No | **string[]** Namespace data as `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. | | `--description` | No | **string** Namespace description. | | `--email` | No | **string** Owner email. | | `--global` | No | **bool** Enable multi-region data replication. | | `--history-archival-state` | No | **string-enum** History archival state. Accepted values: disabled, enabled. | | `--history-uri` | No | **string** Archive history to this `URI`. Once enabled, can't be changed. | | `--retention` | No | **duration** Time to preserve closed Workflows before deletion. | | `--visibility-archival-state` | No | **string-enum** Visibility archival state. Accepted values: disabled, enabled. | | `--visibility-uri` | No | **string** Archive visibility data to this `URI`. Once enabled, can't be changed. | ### delete Removes a Namespace from the Service. ``` temporal operator namespace delete [options] ``` For example: ``` temporal operator namespace delete \ --namespace YourNamespaceName ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--yes`, `-y` | No | **bool** Request confirmation before deletion. | ### describe Provide long-form information about a Namespace identified by its ID or name: ``` temporal operator namespace describe \ --namespace-id YourNamespaceId ``` or ``` temporal operator namespace describe \ --namespace YourNamespaceName ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--namespace-id` | No | **string** Namespace ID. | ### list Display a detailed listing for all Namespaces on the Service: ``` temporal operator namespace list ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### update Update a Namespace using properties you specify. ``` temporal operator namespace update [options] ``` Assign a Namespace's active Cluster (Service): ``` temporal operator namespace update \ --namespace YourNamespaceName \ --active-cluster NewActiveCluster ``` Promote a Namespace for multi-region data replication: ``` temporal operator namespace update \ --namespace YourNamespaceName \ --promote-global ``` You may update archives that were previously enabled or disabled. Note: URI values for archival states can't be changed once enabled. ``` temporal operator namespace update \ --namespace YourNamespaceName \ --history-archival-state enabled \ --visibility-archival-state disabled ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--active-cluster` | No | **string** Active Cluster (Service) name. | | `--cluster` | No | **string[]** Cluster (Service) names. | | `--data` | No | **string[]** Namespace data as `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. | | `--description` | No | **string** Namespace description. | | `--email` | No | **string** Owner email. | | `--history-archival-state` | No | **string-enum** History archival state. Accepted values: disabled, enabled. | | `--history-uri` | No | **string** Archive history to this `URI`. Once enabled, can't be changed. | | `--promote-global` | No | **bool** Enable multi-region data replication. | | `--replication-state` | No | **string-enum** Replication state. Accepted values: normal, handover. | | `--retention` | No | **duration** Length of time a closed Workflow is preserved before deletion. | | `--visibility-archival-state` | No | **string-enum** Visibility archival state. Accepted values: disabled, enabled. | | `--visibility-uri` | No | **string** Archive visibility data to this `URI`. Once enabled, can't be changed. | ## nexus These commands manage Nexus resources. Nexus commands follow this syntax: ``` temporal operator nexus [command] [subcommand] [options] ``` ### endpoint These commands manage Nexus Endpoints. Nexus Endpoint commands follow this syntax: ``` temporal operator nexus endpoint [command] [options] ``` #### create Create a Nexus Endpoint on the Server. A Nexus Endpoint name is used in Workflow code to invoke Nexus Operations. The endpoint target may either be a Worker, in which case `--target-namespace` and `--target-task-queue` must both be provided, or an external URL, in which case `--target-url` must be provided. This command will fail if an Endpoint with the same name is already registered. ``` temporal operator nexus endpoint create \ --name your-endpoint \ --target-namespace your-namespace \ --target-task-queue your-task-queue \ --description-file DESCRIPTION.md ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--description` | No | **string** Nexus Endpoint description. You may use Markdown formatting in the Nexus Endpoint description. | | `--description-file` | No | **string** Path to the Nexus Endpoint description file. The contents of the description file may use Markdown formatting. | | `--name` | Yes | **string** Endpoint name. | | `--target-namespace` | No | **string** Namespace where a handler Worker polls for Nexus tasks. | | `--target-task-queue` | No | **string** Task Queue that a handler Worker polls for Nexus tasks. | | `--target-url` | No | **string** An external Nexus Endpoint that receives forwarded Nexus requests. May be used as an alternative to `--target-namespace` and `--target-task-queue`. _(Experimental)_ | #### delete Delete a Nexus Endpoint from the Server. ``` temporal operator nexus endpoint delete --name your-endpoint ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name` | Yes | **string** Endpoint name. | #### get Get a Nexus Endpoint by name from the Server. ``` temporal operator nexus endpoint get --name your-endpoint ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name` | Yes | **string** Endpoint name. | #### list List all Nexus Endpoints on the Server. ``` temporal operator nexus endpoint list ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. #### update Update an existing Nexus Endpoint on the Server. A Nexus Endpoint name is used in Workflow code to invoke Nexus Operations. The Endpoint target may either be a Worker, in which case `--target-namespace` and `--target-task-queue` must both be provided, or an external URL, in which case `--target-url` must be provided. The Endpoint is patched; existing fields for which flags are not provided are left as they were. Update only the target task queue: ``` temporal operator nexus endpoint update \ --name your-endpoint \ --target-task-queue your-other-queue ``` Update only the description: ``` temporal operator nexus endpoint update \ --name your-endpoint \ --description-file DESCRIPTION.md ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--description` | No | **string** Nexus Endpoint description. You may use Markdown formatting in the Nexus Endpoint description. | | `--description-file` | No | **string** Path to the Nexus Endpoint description file. The contents of the description file may use Markdown formatting. | | `--name` | Yes | **string** Endpoint name. | | `--target-namespace` | No | **string** Namespace where a handler Worker polls for Nexus tasks. | | `--target-task-queue` | No | **string** Task Queue that a handler Worker polls for Nexus tasks. | | `--target-url` | No | **string** An external Nexus Endpoint that receives forwarded Nexus requests. May be used as an alternative to `--target-namespace` and `--target-task-queue`. _(Experimental)_ | | `--unset-description` | No | **bool** Unset the description. | ## search-attribute Create, list, or remove Search Attributes fields stored in a Workflow Execution's metadata: ``` temporal operator search-attribute create \ --name YourAttributeName \ --type Keyword ``` Supported types include: Text, Keyword, Int, Double, Bool, Datetime, and KeywordList. If you wish to delete a Search Attribute, please contact support at https://support.temporal.io. ### create Add one or more custom Search Attributes: ``` temporal operator search-attribute create \ --name YourAttributeName \ --type Keyword ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name` | Yes | **string[]** Search Attribute name. | | `--type` | Yes | **string-enum[]** Search Attribute type. Accepted values: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. | ### list Display a list of active Search Attributes that can be assigned or used with Workflow Queries. You can manage this list and add attributes as needed: ``` temporal operator search-attribute list ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### remove Remove custom Search Attributes from the options that can be assigned or used with Workflow Queries. ``` temporal operator search-attribute remove \ --name YourAttributeName ``` Remove attributes without confirmation: ``` temporal operator search-attribute remove \ --name YourAttributeName \ --yes ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name` | Yes | **string[]** Search Attribute name. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm removal. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI schedule command reference Source: https://docs.temporal.io/cli/command-reference/schedule > Temporal's Schedule commands allow users to create, update, and manage Workflow Executions seamlessly for automation, supporting commands for creation, backfill, deletion, and more. This page provides a reference for the `temporal` CLI `schedule` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## backfill Batch-execute actions that would have run during a specified time interval. Use this command to fill in Workflow runs from when a Schedule was paused, before a Schedule was created, from the future, or to re-process a previously executed interval. Backfills require a Schedule ID and the time period covered by the request. It's best to use the `BufferAll` or `AllowAll` policies to avoid conflicts and ensure no Workflow Executions are skipped. For example: ``` temporal schedule backfill \ --schedule-id "YourScheduleId" \ --start-time "2022-05-01T00:00:00Z" \ --end-time "2022-05-31T23:59:59Z" \ --overlap-policy BufferAll ``` The policies include: * **AllowAll**: Allow unlimited concurrent Workflow Executions. This significantly speeds up the backfilling process on systems that support concurrency. You must ensure running Workflow Executions do not interfere with each other. * **BufferAll**: Buffer all incoming Workflow Executions while waiting for the running Workflow Execution to complete. * **Skip**: If a previous Workflow Execution is still running, discard new Workflow Executions. * **BufferOne**: Same as 'Skip' but buffer a single Workflow Execution to be run after the previous Execution completes. Discard other Workflow Executions. * **CancelOther**: Cancel the running Workflow Execution and replace it with the incoming new Workflow Execution. * **TerminateOther**: Terminate the running Workflow Execution and replace it with the incoming new Workflow Execution. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--end-time` | Yes | **timestamp** Backfill end time. | | `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | | `--start-time` | Yes | **timestamp** Backfill start time. | ## create Create a new Schedule on the Temporal Service. A Schedule automatically starts new Workflow Executions at the times you specify. For example: ``` temporal schedule create \ --schedule-id "YourScheduleId" \ --calendar '{"dayOfWeek":"Fri","hour":"3","minute":"30"}' \ --workflow-id YourBaseWorkflowIdName \ --task-queue YourTaskQueue \ --type YourWorkflowType ``` Schedules support any combination of `--calendar`, `--interval`, and `--cron`: * Shorthand `--interval` strings. For example: 45m (every 45 minutes) or 6h/5h (every 6 hours, at the top of the 5th hour). * JSON `--calendar`, as in the preceding example. * Unix-style `--cron` strings and robfig declarations (@daily/@weekly/@every X/etc). For example, every Friday at 12:30 PM: `30 12 * * Fri`. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--calendar` | No | **string[]** Calendar specification in JSON. For example: `{"dayOfWeek":"Fri","hour":"17","minute":"5"}`. | | `--catchup-window` | No | **duration** Maximum catch-up time for when the Service is unavailable. | | `--cron` | No | **string[]** Calendar specification in cron string format. For example: `"30 12 * * Fri"`. | | `--end-time` | No | **timestamp** Schedule end time. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--interval` | No | **string[]** Interval duration. For example, 90m, or 60m/15m to include phase offset. | | `--jitter` | No | **duration** Max difference in time from the specification. Vary the start time randomly within this amount. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--notes` | No | **string** Initial notes field value. | | `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. | | `--pause-on-failure` | No | **bool** Pause schedule after Workflow failures. | | `--paused` | No | **bool** Pause the Schedule immediately on creation. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--remaining-actions` | No | **int** Total allowed actions. Default is zero (unlimited). | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | | `--schedule-memo` | No | **string[]** Set schedule memo using `KEY="VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--schedule-search-attribute` | No | **string[]** Set schedule Search Attributes using `KEY="VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-time` | No | **timestamp** Schedule start time. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--time-zone` | No | **string** Interpret calendar specs with the `TZ` time zone. For a list of time zones, see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. | | `--type` | Yes | **string** Workflow Type name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## delete Deletes a Schedule on the front end Service: ``` temporal schedule delete \ --schedule-id YourScheduleId ``` Removing a Schedule won't affect the Workflow Executions it started that are still running. To cancel or terminate these Workflow Executions, use `temporal workflow delete` with the `TemporalScheduledById` Search Attribute instead. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | ## describe Show a Schedule configuration, including information about past, current, and future Workflow runs: ``` temporal schedule describe \ --schedule-id YourScheduleId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | ## list Lists the Schedules hosted by a Namespace: ``` temporal schedule list \ --namespace YourNamespace ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--long`, `-l` | No | **bool** Show detailed information. | | `--query`, `-q` | No | **string** Filter results using given List Filter. | | `--really-long` | No | **bool** Show extensive information in non-table form. | ## list-matching-times Note: This is an experimental feature and may change in the future. List the times a Schedule's spec would match within a given time range. The time range may be in the past or future. Use this command to preview when a Schedule will take actions without actually running them. For example: ``` temporal schedule list-matching-times \ --schedule-id "YourScheduleId" \ --start-time "2024-01-01T00:00:00Z" \ --end-time "2024-01-31T23:59:59Z" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--end-time` | Yes | **timestamp** End of time range to list matching times. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | | `--start-time` | Yes | **timestamp** Start of time range to list matching times. | ## toggle Pause or unpause a Schedule by passing a flag with your desired state: ``` temporal schedule toggle \ --schedule-id "YourScheduleId" \ --pause \ --reason "YourReason" ``` and ``` temporal schedule toggle --schedule-id "YourScheduleId" \ --unpause \ --reason "YourReason" ``` The `--reason` text updates the Schedule's `notes` field for operations communication. It defaults to "(no reason provided)" if omitted. This field is also visible on the Service Web UI. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--pause` | No | **bool** Pause the Schedule. | | `--reason` | No | **string** Reason for pausing or unpausing the Schedule. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | | `--unpause` | No | **bool** Unpause the Schedule. | ## trigger Trigger a Schedule to run immediately: ``` temporal schedule trigger \ --schedule-id "YourScheduleId" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | ## update Update an existing Schedule with new configuration details, including time specifications, action, and policies: ``` temporal schedule update \ --schedule-id "YourScheduleId" \ --workflow-type "NewWorkflowType" ``` This command performs a full replacement of the Schedule configuration. Any options not provided will be reset to their default values. You must re-specify all options, not just the ones you want to change. To view the current configuration of a Schedule, use `temporal schedule describe` before updating. Schedule memo and search attributes cannot be updated with this command. They are set only during Schedule creation and are not affected by updates. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--calendar` | No | **string[]** Calendar specification in JSON. For example: `{"dayOfWeek":"Fri","hour":"17","minute":"5"}`. | | `--catchup-window` | No | **duration** Maximum catch-up time for when the Service is unavailable. | | `--cron` | No | **string[]** Calendar specification in cron string format. For example: `"30 12 * * Fri"`. | | `--end-time` | No | **timestamp** Schedule end time. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--interval` | No | **string[]** Interval duration. For example, 90m, or 60m/15m to include phase offset. | | `--jitter` | No | **duration** Max difference in time from the specification. Vary the start time randomly within this amount. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--notes` | No | **string** Initial notes field value. | | `--overlap-policy` | No | **string-enum** Policy for handling overlapping Workflow Executions. Accepted values: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. | | `--pause-on-failure` | No | **bool** Pause schedule after Workflow failures. | | `--paused` | No | **bool** Pause the Schedule immediately on creation. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--remaining-actions` | No | **int** Total allowed actions. Default is zero (unlimited). | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--schedule-id`, `-s` | Yes | **string** Schedule ID. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-time` | No | **timestamp** Schedule start time. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--time-zone` | No | **string** Interpret calendar specs with the `TZ` time zone. For a list of time zones, see: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. | | `--type` | Yes | **string** Workflow Type name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI server command reference Source: https://docs.temporal.io/cli/command-reference/server > Manage your Temporal Server easily with CLI commands. Start a local server using `temporal server start-dev` and access the Web UI at http://localhost:8233. Customize with multiple options. This page provides a reference for the `temporal` CLI `server` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## start-dev Run a development Temporal Server on your local system. ``` +------------------------------------------------------------------------+ | WARNING: The development server is not intended for production use. | | It skips certain HTTP security checks to make local use simpler. | | | | For production use, see: | | https://docs.temporal.io/production-deployment | +------------------------------------------------------------------------+ ``` View the Web UI for the default configuration at: http://localhost:8233 ``` temporal server start-dev ``` Add persistence for Workflow Executions across runs: ``` temporal server start-dev \ --db-filename path-to-your-local-persistent-store ``` Set the port from the front-end gRPC Service (7233 default): ``` temporal server start-dev \ --port 7000 ``` Use a custom port for the Web UI. The default is the gRPC port (7233 default) plus 1000 (8233): ``` temporal server start-dev \ --ui-port 3000 ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--db-filename`, `-f` | No | **string** Path to file for persistent Temporal state store. By default, Workflow Executions are lost when the server process dies. | | `--dynamic-config-value` | No | **string[]** Dynamic configuration value using `KEY=VALUE` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey="YourString"` Can be passed multiple times. | | `--headless` | No | **bool** Disable the Web UI. | | `--http-port` | No | **int** Port for the HTTP API service. Defaults to a random free port. | | `--ip` | No | **string** IP address bound to the front-end Service. | | `--log-config` | No | **bool** Print the server config to stderr. | | `--metrics-port` | No | **int** Port for the '/metrics' HTTP endpoint. Defaults to a random free port. | | `--namespace`, `-n` | No | **string[]** Namespaces to be created at launch. The "default" Namespace is always created automatically. | | `--port`, `-p` | No | **int** Port for the front-end gRPC Service. | | `--search-attribute` | No | **string[]** Search attributes to register using `KEY=VALUE` pairs. Keys must be identifiers, and values must be the search attribute type, which is one of the following: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. | | `--sqlite-pragma` | No | **string[]** SQLite pragma statements in "PRAGMA=VALUE" format. | | `--ui-asset-path` | No | **string** UI custom assets path. | | `--ui-codec-endpoint` | No | **string** UI remote codec HTTP endpoint. | | `--ui-disable-news-fetch` | No | **bool** Disable the Web UI newsfeed. When set, the UI will not request the newsfeed and the button to open the newsfeed panel is hidden. | | `--ui-ip` | No | **string** IP address bound to the Web UI. Defaults to same as '--ip' value. | | `--ui-port` | No | **int** Port for the Web UI. Defaults to '--port' value + 1000. | | `--ui-public-path` | No | **string** The public base path for the Web UI. Defaults to `/`. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI task-queue command reference Source: https://docs.temporal.io/cli/command-reference/task-queue > Temporal Task Queue commands facilitate operations like describing poller info, displaying partitions, fetching compatible Build IDs, and determining Build ID reachability for effective Workflow and Activity management. This page provides a reference for the `temporal` CLI `task-queue` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## config Manage Task Queue configuration: ``` temporal task-queue config [command] [options] ``` Available commands: - `get`: Retrieve the current configuration for a task queue - `set`: Update the configuration for a task queue ### get Retrieve the current configuration for a Task Queue: ``` temporal task-queue config get \ --task-queue YourTaskQueue \ --task-queue-type activity ``` This command returns the current configuration including: - Queue rate limit: The overall rate limit of the task queue. This setting overrides the worker rate limit if set. Unless modified, this is the system-defined rate limit. - Fairness key rate limit defaults: Default rate limits for fairness keys. If set, each individual fairness key will be limited to this rate, scaled by the weight of the fairness key. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--task-queue`, `-t` | Yes | **string** Task Queue name. | | `--task-queue-type` | Yes | **string-enum** Task Queue type. Accepted values: workflow, activity, nexus. Accepted values: workflow, activity, nexus. | ### set Update configuration settings for a Task Queue. ``` temporal task-queue config set \ --task-queue YourTaskQueue \ --task-queue-type activity \ --namespace YourNamespace \ --queue-rps-limit \ --queue-rps-limit-reason \ --fairness-key-rps-limit-default \ --fairness-key-rps-limit-reason \ --fairness-key-weight HighPriority=2.0 \ --fairness-key-weight LowPriority=0.5 ``` This command supports updating: - Queue rate limits: Controls the overall rate limit of the task queue. This setting overrides the worker rate limit if set. Unless modified, this is the system-defined rate limit. - Fairness key rate limit defaults: Sets default rate limits for fairness keys. If set, each individual fairness key will be limited to this rate, scaled by the weight of the fairness key. - Fairness key weight overrides: Set custom weights for specific fairness keys. Weights control the relative share of capacity each key receives. To unset a rate limit, pass in 'default', for example: --queue-rps-limit default To unset a specific fairness weight, use --fairness-key-weight \=default To unset all fairness weights, use --fairness-key-weight-clear-all Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--fairness-key-rps-limit-default` | No | **float\|default** Fairness key rate limit default in requests per second. Accepts a float; or 'default' to unset. | | `--fairness-key-rps-limit-reason` | No | **string** Reason for fairness key rate limit update. | | `--fairness-key-weight` | No | **string[]** Set or unset fairness key weight overrides in format key=weight or key=default. Use key=weight to set a positive weight value; use key=default to unset. Can be specified multiple times. Example: --fairness-key-weight HighPriority=2.0 --fairness-key-weight LowPriority=default. | | `--fairness-key-weight-clear-all` | No | **bool** Unset all fairness key weight overrides. Cannot be used with --fairness-key-weight. | | `--queue-rps-limit` | No | **float\|default** Queue rate limit in requests per second. Accepts a float; or 'default' to unset. | | `--queue-rps-limit-reason` | No | **string** Reason for queue rate limit update. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | | `--task-queue-type` | Yes | **string-enum** Task Queue type. Accepted values: workflow, activity, nexus. Accepted values: workflow, activity, nexus. | ## describe Display a list of active Workers that have recently polled a Task Queue. The Temporal Server records each poll request time. A `LastAccessTime` over one minute may indicate the Worker is at capacity or has shut down. Temporal Workers are removed if 5 minutes have passed since the last poll request. ``` temporal task-queue describe \ --task-queue YourTaskQueue ``` This command provides poller information for a given Task Queue. Workflow and Activity polling use separate Task Queues: ``` temporal task-queue describe \ --task-queue YourTaskQueue \ --task-queue-type "activity" ``` This command provides the following task queue statistics: - `ApproximateBacklogCount`: The approximate number of tasks backlogged in this task queue. May count expired tasks but eventually converges to the right value. - `ApproximateBacklogAge`: Approximate age of the oldest task in the backlog, based on its creation time, measured in seconds. - `TasksAddRate`: Approximate rate at which tasks are being added to the task queue, measured in tasks per second, averaged over the last 30 seconds. Includes tasks dispatched immediately without going to the backlog (sync-matched tasks), as well as tasks added to the backlog. (See note below.) - `TasksDispatchRate`: Approximate rate at which tasks are being dispatched from the task queue, measured in tasks per second, averaged over the last 30 seconds. Includes tasks dispatched immediately without going to the backlog (sync-matched tasks), as well as tasks added to the backlog. (See note below.) - `BacklogIncreaseRate`: Approximate rate at which the backlog size is increasing (if positive) or decreasing (if negative), measured in tasks per second, averaged over the last 30 seconds. This is roughly equivalent to: `TasksAddRate` - `TasksDispatchRate`. NOTE: The `TasksAddRate` and `TasksDispatchRate` metrics may differ from the actual rate of add/dispatch, because tasks may be dispatched eagerly to an available worker, or may apply only to specific workers (they are "sticky"). Such tasks are not counted by these metrics. Despite the inaccuracy of these two metrics, the derived metric of `BacklogIncreaseRate` is accurate for backlogs older than a few seconds. Safely retire Workers assigned a Build ID by checking reachability across all task types. Use the flag `--report-reachability`: ``` temporal task-queue describe \ --task-queue YourTaskQueue \ --select-build-id "YourBuildId" \ --report-reachability ``` Task reachability information is returned for the requested versions and all task types, which can be used to safely retire Workers with old code versions, provided that they were assigned a Build ID. Note that task reachability status is deprecated in favor of Drainage Status (ie. of a Drained or Draining Worker Deployment Version) and will be removed in a future release. Also, determining task reachability incurs a non-trivial computing cost. Task reachability states are reported per build ID. The state may be one of the following: - `Reachable`: using the current versioning rules, the Build ID may be used by new Workflow Executions or Activities OR there are currently open Workflow or backlogged Activity tasks assigned to the queue. - `ClosedWorkflowsOnly`: the Build ID does not have open Workflow Executions and can't be reached by new Workflow Executions. It MAY have closed Workflow Executions within the Namespace retention period. - `Unreachable`: this Build ID is not used for new Workflow Executions and isn't used by any existing Workflow Execution within the retention period. Task reachability is eventually consistent. You may experience a delay until reachability converges to the most accurate value. This is designed to act in the most conservative way until convergence. For example, `Reachable` is more conservative than `ClosedWorkflowsOnly`. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--disable-stats` | No | **bool** Disable task queue statistics. | | `--legacy-mode` | No | **bool** Enable a legacy mode for servers that do not support rules-based worker versioning. This mode only provides pollers info. | | `--partitions-legacy` | No | **int** Query partitions 1 through `N`. Experimental/Temporary feature. Legacy mode only. | | `--report-config` | No | **bool** Include task queue configuration in the response. When enabled, the command will return the current rate limit configuration for the task queue. | | `--report-reachability` | No | **bool** Display task reachability information. | | `--select-all-active` | No | **bool** Include all active versions. A version is active if it had new tasks or polls recently. | | `--select-build-id` | No | **string[]** Filter the Task Queue based on Build ID. | | `--select-unversioned` | No | **bool** Include the unversioned queue. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | | `--task-queue-type` | No | **string-enum[]** Task Queue type. If not specified, all types are reported. Accepted values: workflow, activity, nexus. | | `--task-queue-type-legacy` | No | **string-enum** Task Queue type (legacy mode only). Accepted values: workflow, activity. | ## get-build-id-reachability ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Show if a given Build ID can be used for new, existing, or closed Workflows in Namespaces that support Worker versioning: ``` temporal task-queue get-build-id-reachability \ --task-queue YourTaskQueue \ --build-id "YourBuildId" ``` You can specify the `--build-id` and `--task-queue` flags multiple times. If `--task-queue` is omitted, the command checks Build ID reachability against all Task Queues. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | No | **string[]** One or more Build ID strings. Can be passed multiple times. | | `--reachability-type` | No | **string-enum** Reachability filter. `open`: reachable by one or more open workflows. `closed`: reachable by one or more closed workflows. `existing`: reachable by either. New Workflow Executions reachable by a Build ID are always reported. Accepted values: open, closed, existing. | | `--task-queue`, `-t` | No | **string[]** Search only the specified task queue(s). Can be passed multiple times. | ## get-build-ids ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Fetch sets of compatible Build IDs for specified Task Queues and display their information: ``` temporal task-queue get-build-ids \ --task-queue YourTaskQueue ``` This command is limited to Namespaces that support Worker versioning. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--max-sets` | No | **int** Max return count. Use 1 for default major version. Use 0 for all sets. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ## list-partition Display a Task Queue's partition list with assigned matching nodes: ``` temporal task-queue list-partition \ --task-queue YourTaskQueue ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ## update-build-ids ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Add or change a Task Queue's compatible Build IDs for Namespaces using Worker versioning: ``` temporal task-queue update-build-ids [subcommands] [options] \ --task-queue YourTaskQueue ``` ### add-new-compatible Add a compatible Build ID to a Task Queue's existing version set. Provide an existing Build ID and a new Build ID: ``` temporal task-queue update-build-ids add-new-compatible \ --task-queue YourTaskQueue \ --existing-compatible-build-id "YourExistingBuildId" \ --build-id "YourNewBuildId" ``` The new ID is stored in the set containing the existing ID and becomes the new default for that set. This command is limited to Namespaces that support Worker versioning. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID to be added. | | `--existing-compatible-build-id` | Yes | **string** Pre-existing Build ID in this Task Queue. | | `--set-as-default` | No | **bool** Set the expanded Build ID set as the Task Queue default. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ### add-new-default ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Create a new Task Queue Build ID set, add a Build ID to it, and make it the overall Task Queue default. The new set will be incompatible with previous sets and versions. ``` temporal task-queue update-build-ids add-new-default \ --task-queue YourTaskQueue \ --build-id "YourNewBuildId" ``` ``` +------------------------------------------------------------------------+ | NOTICE: This command is limited to Namespaces that support Worker | | versioning. Worker versioning is experimental. Versioning commands are | | subject to change. | +------------------------------------------------------------------------+ ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID to be added. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ### promote-id-in-set ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Establish an existing Build ID as the default in its Task Queue set. New tasks compatible with this set will now be dispatched to this ID: ``` temporal task-queue update-build-ids promote-id-in-set \ --task-queue YourTaskQueue \ --build-id "YourBuildId" ``` ``` +------------------------------------------------------------------------+ | NOTICE: This command is limited to Namespaces that support Worker | | versioning. Worker versioning is experimental. Versioning commands are | | subject to change. | +------------------------------------------------------------------------+ ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID to set as default. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ### promote-set ``` +-----------------------------------------------------------------------------+ | CAUTION: This command is deprecated and will be removed in a later release. | +-----------------------------------------------------------------------------+ ``` Promote a Build ID set to be the default on a Task Queue. Identify the set by providing a Build ID within it. If the set is already the default, this command has no effect: ``` temporal task-queue update-build-ids promote-set \ --task-queue YourTaskQueue \ --build-id "YourBuildId" ``` ``` +------------------------------------------------------------------------+ | NOTICE: This command is limited to Namespaces that support Worker | | versioning. Worker versioning is experimental. Versioning commands are | | subject to change. | +------------------------------------------------------------------------+ ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID within the promoted set. | | `--task-queue`, `-t` | Yes | **string** Task Queue name. | ## versioning ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Provides commands to add, list, remove, or replace Worker Build ID assignment and redirect rules associated with Task Queues: ``` temporal task-queue versioning [subcommands] [options] \ --task-queue YourTaskQueue ``` Task Queues support the following versioning rules and policies: - Assignment Rules: manage how new executions are assigned to run on specific Worker Build IDs. Each Task Queue stores a list of ordered Assignment Rules, which are evaluated from first to last. Assignment Rules also allow for gradual rollout of new Build IDs by setting ramp percentage. - Redirect Rules: automatically assign work for a source Build ID to a target Build ID. You may add at most one redirect rule for each source Build ID. Redirect rules require that a target Build ID is fully compatible with the source Build ID. ### add-redirect-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Add a new redirect rule for a given Task Queue. You may add at most one redirect rule for each distinct source build ID: ``` temporal task-queue versioning add-redirect-rule \ --task-queue YourTaskQueue \ --source-build-id "YourSourceBuildID" \ --target-build-id "YourTargetBuildID" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--source-build-id` | Yes | **string** Source build ID. | | `--target-build-id` | Yes | **string** Target build ID. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### commit-build-id ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Complete a Build ID's rollout and clean up unnecessary rules that might have been created during a gradual rollout: ``` temporal task-queue versioning commit-build-id \ --task-queue YourTaskQueue --build-id "YourBuildId" ``` This command automatically applies the following atomic changes: - Adds an unconditional assignment rule for the target Build ID at the end of the list. - Removes all previously added assignment rules to the given target Build ID. - Removes any unconditional assignment rules for other Build IDs. Rejects requests when there have been no recent pollers for this Build ID. This prevents committing invalid Build IDs. Use the `--force` option to override this validation. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Target build ID. | | `--force` | No | **bool** Bypass recent-poller validation. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### delete-assignment-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Deletes a rule identified by its index in the Task Queue's list of assignment rules. ``` temporal task-queue versioning delete-assignment-rule \ --task-queue YourTaskQueue \ --rule-index YourIntegerRuleIndex ``` By default, the Task Queue must retain one unconditional rule, such as "no hint filter" or "percentage". Otherwise, the delete operation is rejected. Use the `--force` option to override this validation. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--force` | No | **bool** Bypass one-unconditional-rule validation. | | `--rule-index`, `-i` | Yes | **int** Position of the assignment rule to be replaced. Requests for invalid indices will fail. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### delete-redirect-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Deletes the routing rule for the given source Build ID. ``` temporal task-queue versioning delete-redirect-rule \ --task-queue YourTaskQueue \ --source-build-id "YourBuildId" ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--source-build-id` | Yes | **string** Source Build ID. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### get-rules ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Retrieve all the Worker Build ID assignments and redirect rules associated with a Task Queue: ``` temporal task-queue versioning get-rules \ --task-queue YourTaskQueue ``` Task Queues support the following versioning rules: - Assignment Rules: manage how new executions are assigned to run on specific Worker Build IDs. Each Task Queue stores a list of ordered Assignment Rules, which are evaluated from first to last. Assignment Rules also allow for gradual rollout of new Build IDs by setting ramp percentage. - Redirect Rules: automatically assign work for a source Build ID to a target Build ID. You may add at most one redirect rule for each source Build ID. Redirect rules require that a target Build ID is fully compatible with the source Build ID. Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### insert-assignment-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Inserts a new assignment rule for this Task Queue. Rules are evaluated in order, starting from index 0. The first applicable rule is applied, and the rest ignored: ``` temporal task-queue versioning insert-assignment-rule \ --task-queue YourTaskQueue \ --build-id "YourBuildId" ``` If you do not specify a `--rule-index`, this command inserts at index 0. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Target Build ID. | | `--percentage` | No | **int** Traffic percent to send to target Build ID. | | `--rule-index`, `-i` | No | **int** Insertion position. Ranges from 0 (insert at start) to count (append). Any number greater than the count is treated as "append". | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### replace-assignment-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Change an assignment rule for this Task Queue. By default, this enforces one unconditional rule (no hint filter or percentage). Otherwise, the operation will be rejected. Set `force` to true to bypass this validation. ``` temporal task-queue versioning replace-assignment-rule \ --task-queue YourTaskQueue \ --rule-index AnIntegerIndex \ --build-id "YourBuildId" ``` To assign multiple assignment rules to a single Build ID, use 'insert-assignment-rule'. To update the percent: ``` temporal task-queue versioning replace-assignment-rule \ --task-queue YourTaskQueue \ --rule-index AnIntegerIndex \ --build-id "YourBuildId" \ --percentage AnIntegerPercent ``` Percent may vary between 0 and 100 (default). Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Target Build ID. | | `--force` | No | **bool** Bypass the validation that one unconditional rule remains. | | `--percentage` | No | **int** Divert percent of traffic to target Build ID. | | `--rule-index`, `-i` | Yes | **int** Position of the assignment rule to be replaced. Requests for invalid indices will fail. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ### replace-redirect-rule ``` +-------------------------------------------------------------+ | CAUTION: This API has been deprecated by Worker Deployment. | +-------------------------------------------------------------+ ``` Updates a Build ID's redirect rule on a Task Queue by replacing its target Build ID: ``` temporal task-queue versioning replace-redirect-rule \ --task-queue YourTaskQueue \ --source-build-id YourSourceBuildId \ --target-build-id YourNewTargetBuildId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--source-build-id` | Yes | **string** Source Build ID. | | `--target-build-id` | Yes | **string** Target Build ID. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI worker command reference Source: https://docs.temporal.io/cli/command-reference/worker > Learn how to read or modify state associated with a Worker, such as Worker Deployments. This page provides a reference for the `temporal` CLI `worker` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## deployment Deployment commands perform operations on Worker Deployments: ``` temporal worker deployment [command] [options] ``` For example: ``` temporal worker deployment list ``` Lists the Deployments in the client's namespace. Arguments can be Worker Deployment Versions associated with a Deployment, specified using the Deployment name and Build ID. For example: ``` temporal worker deployment set-current-version \ --deployment-name YourDeploymentName --build-id YourBuildID ``` Sets the current Deployment Version for a given Deployment. ### create Create a new Worker Deployment: ``` temporal worker deployment create [options] ``` Worker Deployments are lazily created the first time a Worker polls the Temporal Server and specifies a VersionOverride. However, if you need to pre-define a compute configuration (for instance to set up a serverless Worker), you need to call `temporal worker deployment create-version` and pass in the name of the Worker Deployment. The `temporal worker deployment create` command allows you to pre-define a Worker Deployment so that calls to `temporal worker deployment create-version` will succeed. If a Worker Deployment with the supplied name already exists, this command will return an error. Note: This is an experimental feature and may change in the future. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name`, `-d` | Yes | **string** Name for a Worker Deployment. | ### create-version Create a new Worker Deployment Version: ``` temporal worker deployment create-version [options] ``` Configure a Worker Deployment Version's compute configuration as needed. For example, pass compute provider information for an AWS Lambda function that spawns a Worker in the Worker Deployment: ``` temporal worker deployment create-version \ --namespace YourNamespaceName \ --deployment-name YourDeploymentName \ --build-id YourBuildID \ --aws-lambda-function-arn LambdaFunctionARN \ --aws-lambda-assume-role-arn LambdaAssumeRoleARN \ --aws-lambda-assume-role-external-id LambdaAssumeRoleExternalID ``` Or pass compute provider information for an AWS Bedrock Agentcore Runtime that spawns a Worker in the Worker Deployment: ``` temporal worker deployment create-version \ --namespace YourNamespaceName \ --deployment-name YourDeploymentName \ --build-id YourBuildID \ --aws-agentcore-endpoint-arn AgentcoreRuntimeEndpointARN \ --aws-agentcore-assume-role-arn AgentcoreAssumeRoleARN \ --aws-agentcore-assume-role-external-id AgentcoreAssumeRoleExternalID ``` Or pass compute provider information for a GCP Cloud Run worker pool that spawns a Worker in the Worker Deployment: ``` temporal worker deployment create-version \ --namespace YourNamespaceName \ --deployment-name YourDeploymentName \ --build-id YourBuildID \ --gcp-cloud-run-project YourGCPProject \ --gcp-cloud-run-region us-central1 \ --gcp-cloud-run-worker-pool YourWorkerPool \ --gcp-cloud-run-service-account customer-sa@proj.iam.gserviceaccount.com \ --gcp-cloud-run-min-instances 1 \ --gcp-cloud-run-max-instances 3 \ --gcp-cloud-run-initial-instances 1 \ --gcp-cloud-run-utilization-target 0.75 \ --gcp-cloud-run-scale-down-stabilization-duration 5m ``` If a Worker Deployment Version with the supplied BuildID already exists, this command will return an error. Returns an error if all compute configuration fields are empty. Note: This is an experimental feature and may change in the future. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--aws-agentcore-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Agentcore Runtime that spawns a new Worker in this Worker Deployment Version. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. | | `--aws-agentcore-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-agentcore-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. | | `--aws-agentcore-endpoint-arn` | No | **string** AWS Bedrock Agentcore Runtime endpoint ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. The endpoint ARN encodes the runtime, endpoint name, and region. | | `--aws-agentcore-skip-role-and-external-id` | No | **bool** When --aws-agentcore-endpoint-arn is specified, --aws-agentcore-assume-role-arn and --aws-agentcore-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. | | `--aws-lambda-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Lambda function that spawns a new Worker in this Worker Deployment Version. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. | | `--aws-lambda-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-lambda-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. | | `--aws-lambda-function-arn` | No | **string** Qualified (contains version suffix) or unqualified AWS Lambda function ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. | | `--aws-lambda-skip-role-and-external-id` | No | **bool** When --aws-lambda-function-arn is specified, --aws-lambda-assume-role-arn and --aws-lambda-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. | | `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--gcp-cloud-run-initial-instances` | No | **int** Number of Cloud Run worker pool instances the scaler starts with. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together, and this value must be between the min and max (inclusive). Defaults to 0 when unset. Only valid with --gcp-cloud-run-worker-pool. | | `--gcp-cloud-run-max-instances` | No | **int** Maximum number of Cloud Run worker pool instances the scaler may scale up to. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Defaults to 30 when unset. Only valid with --gcp-cloud-run-worker-pool. | | `--gcp-cloud-run-min-instances` | No | **int** Minimum number of Cloud Run worker pool instances the scaler will maintain. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Defaults to 0 when unset. Only valid with --gcp-cloud-run-worker-pool. | | `--gcp-cloud-run-project` | No | **string** GCP project ID hosting the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-region` | No | **string** Region of the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-scale-down-stabilization-duration` | No | **duration** Duration the scaler waits after it last saw unmet task demand before it may scale the Cloud Run worker pool down. Raise this to keep the pool from scaling down before long-running or bursty activities finish. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. A value of 0s disables the wait. Defaults to 90s when unset. Only valid with --gcp-cloud-run-worker-pool. | | `--gcp-cloud-run-service-account` | No | **string** Customer GCP service account the Temporal server impersonates to manage the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-utilization-target` | No | **float** Target average worker utilization the scaler aims for, as a fraction in the range (0, 1]. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Lower values keep more spare capacity per worker. Defaults to 0.8 when unset. Only valid with --gcp-cloud-run-worker-pool. | | `--gcp-cloud-run-worker-pool` | No | **string** GCP Cloud Run worker pool name to scale when there are no active pollers for task queue targets in the Worker Deployment. | ### delete Remove a Worker Deployment given its Deployment Name. A Deployment can only be deleted if it has no Version in it. ``` temporal worker deployment delete [options] ``` For example, setting the user identity that removed the deployment: ``` temporal worker deployment delete \ --name YourDeploymentName \ --identity YourIdentity ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name`, `-d` | Yes | **string** Name for a Worker Deployment. | ### delete-version Remove a Worker Deployment Version given its fully-qualified identifier. This is rarely needed during normal operation since unused Versions are eventually garbage collected. The client can delete a Version only when all of the following conditions are met: - It is not the Current or Ramping Version for this Deployment. - It has no active pollers, i.e., none of the task queues in the Version have pollers. - It is not draining. This requirement can be ignored with the option `--skip-drainage`. ``` temporal worker deployment delete-version [options] ``` For example, skipping the drainage restriction: ``` temporal worker deployment delete-version \ --deployment-name YourDeploymentName --build-id YourBuildID \ --skip-drainage ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--skip-drainage` | No | **bool** Ignore the deletion requirement of not draining. | ### describe Describe properties of a Worker Deployment, such as the versions associated with it, routing information of new or existing tasks executed by this deployment, or its creation time. ``` temporal worker deployment describe [options] ``` For example, to describe a deployment `YourDeploymentName` in the default namespace: ``` temporal worker deployment describe \ --name YourDeploymentName ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--name`, `-d` | Yes | **string** Name for a Worker Deployment. | ### describe-version Describe properties of a Worker Deployment Version, such as the task queues polled by workers in this Deployment Version, or drainage information required to safely decommission workers, or user-provided metadata, or its creation/modification time. ``` temporal worker deployment describe-version [options] ``` For example, to describe a deployment version in a deployment `YourDeploymentName`, with Build ID `YourBuildID`, and in the default namespace: ``` temporal worker deployment describe-version \ --deployment-name YourDeploymentName --build-id YourBuildID ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--report-task-queue-stats` | No | **bool** Report stats for task queues that are present in this version. | ### list List existing Worker Deployments in the client's namespace. ``` temporal worker deployment list [options] ``` For example, listing Deployments in YourDeploymentNamespace: ``` temporal worker deployment list \ --namespace YourDeploymentNamespace ``` Use [global flags](#global-flags) to customize the connection to the Temporal Service for this command. ### manager-identity Manager Identity commands change the `ManagerIdentity` of a Worker Deployment: ``` temporal worker deployment manager-identity [command] [options] ``` When present, `ManagerIdentity` is the identity of the user that has the exclusive right to make changes to this Worker Deployment. Empty by default. When set, users whose identity does not match the `ManagerIdentity` will not be able to change the Worker Deployment. This is especially useful in environments where multiple users (such as CLI users and automated controllers) may interact with the same Worker Deployment. `ManagerIdentity` allows different users to communicate with one another about who is expected to make changes to the Worker Deployment. The current Manager Identity is returned with `describe`: ``` temporal worker deployment describe \ --deployment-name YourDeploymentName ``` #### set Set the `ManagerIdentity` of a Worker Deployment given its Deployment Name. When present, `ManagerIdentity` is the identity of the user that has the exclusive right to make changes to this Worker Deployment. Empty by default. When set, users whose identity does not match the `ManagerIdentity` will not be able to change the Worker Deployment. This is especially useful in environments where multiple users (such as CLI users and automated controllers) may interact with the same Worker Deployment. `ManagerIdentity` allows different users to communicate with one another about who is expected to make changes to the Worker Deployment. ``` temporal worker deployment manager-identity set [options] ``` For example: ``` temporal worker deployment manager-identity set \ --deployment-name DeploymentName \ --self \ --identity YourUserIdentity # optional, populated by CLI if not provided ``` Sets the Manager Identity of the Deployment to the identity of the user making this request. If you don't specifically pass an identity field, the CLI will generate your identity for you. For example: ``` temporal worker deployment manager-identity set \ --deployment-name DeploymentName \ --manager-identity NewManagerIdentity ``` Sets the Manager Identity of the Deployment to any string. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--deployment-name` | No | **string** Name for a Worker Deployment. Required. | | `--manager-identity` | No | **string** New Manager Identity. Required unless --self is specified. | | `--self` | No | **bool** Set Manager Identity to the identity of the user submitting this request. Required unless --manager-identity is specified. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm set Manager Identity. | #### unset Unset the `ManagerIdentity` of a Worker Deployment given its Deployment Name. When present, `ManagerIdentity` is the identity of the user that has the exclusive right to make changes to this Worker Deployment. Empty by default. When set, users whose identity does not match the `ManagerIdentity` will not be able to change the Worker Deployment. This is especially useful in environments where multiple users (such as CLI users and automated controllers) may interact with the same Worker Deployment. `ManagerIdentity` allows different users to communicate with one another about who is expected to make changes to the Worker Deployment. ``` temporal worker deployment manager-identity unset [options] ``` For example: ``` temporal worker deployment manager-identity unset \ --deployment-name YourDeploymentName ``` Clears the Manager Identity field for a given Deployment. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--deployment-name` | No | **string** Name for a Worker Deployment. Required. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm unset Manager Identity. | ### set-current-version Set the Current Version for a Deployment. When a Version is current, Workers of that Deployment Version will receive tasks from new Workflows, and from existing AutoUpgrade Workflows that are running on this Deployment. If not all the expected Task Queues are being polled by Workers in the new Version the request will fail. To override this protection use `--ignore-missing-task-queues`. Note that this would ignore task queues in a deployment that are not yet discovered, leading to inconsistent task queue configuration. ``` temporal worker deployment set-current-version [options] ``` For example, to set the Current Version of a deployment `YourDeploymentName`, with a version with Build ID `YourBuildID`, and in the default namespace: ``` temporal worker deployment set-current-version \ --deployment-name YourDeploymentName --build-id YourBuildID ``` The target of set-current-version can also be unversioned workers: ``` temporal worker deployment set-current-version \ --deployment-name YourDeploymentName --unversioned ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--allow-no-pollers` | No | **bool** Override protection and set version as current even if it has no pollers. | | `--build-id` | No | **string** Build ID of the Worker Deployment Version. Required unless --unversioned is specified. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--ignore-missing-task-queues` | No | **bool** Override protection to accidentally remove task queues. | | `--unversioned` | No | **bool** Set unversioned workers as the target version. Cannot be used with --build-id. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm set Current Version. | ### set-ramping-version Set the Ramping Version and Percentage for a Deployment. The Ramping Version can be set using deployment name and build ID, or set to unversioned workers using the --unversioned flag. The Ramping Percentage is a float with values in the range [0, 100]. A value of 100 does not make the Ramping Version Current, use `set-current-version` instead. To remove a Ramping Version use the flag `--delete`. If not all the expected Task Queues are being polled by Workers in the new Ramping Version the request will fail. To override this protection use `--ignore-missing-task-queues`. Note that this would ignore task queues in a deployment that are not yet discovered, leading to inconsistent task queue configuration. ``` temporal worker deployment set-ramping-version [options] ``` For example, to set the Ramping Version of a deployment `YourDeploymentName`, with a version with Build ID `YourBuildID`, with 10 percent of tasks redirected to this version, and using the default namespace: ``` temporal worker deployment set-ramping-version \ --deployment-name YourDeploymentName --build-id YourBuildID \ --percentage 10.0 ``` And to remove that ramping: ``` temporal worker deployment set-ramping-version \ --deployment-name YourDeploymentName --build-id YourBuildID \ --delete ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--allow-no-pollers` | No | **bool** Override protection and set version as ramping even if it has no pollers. | | `--build-id` | No | **string** Build ID of the Worker Deployment Version. Required unless --unversioned is specified. | | `--delete` | No | **bool** Delete the Ramping Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--ignore-missing-task-queues` | No | **bool** Override protection to accidentally remove task queues. | | `--percentage` | No | **float** Percentage of tasks redirected to the Ramping Version. Valid range [0,100]. | | `--unversioned` | No | **bool** Set unversioned workers as the target version. Cannot be used with --build-id. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm set Ramping Version. | ### update-version-compute-config Update compute configuration associated with a Worker Deployment Version. For example, to update the AWS Lambda function ARN associated with an existing Worker Deployment Version: ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --aws-lambda-function-arn UpdatedLambdaFunctionARN ``` To update the AWS IAM role ARN that is assumed by the serverless worker manager associated with an existing Worker Deployment Version: ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --aws-lambda-assume-role-arn UpdatedRoleARN ``` To update the AWS Bedrock Agentcore Runtime endpoint associated with an existing Worker Deployment Version: ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --aws-agentcore-endpoint-arn UpdatedAgentcoreRuntimeEndpointARN \ --aws-agentcore-assume-role-arn UpdatedRoleARN \ --aws-agentcore-assume-role-external-id UpdatedExternalID ``` To update the GCP Cloud Run worker pool associated with an existing Worker Deployment Version: ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --gcp-cloud-run-project YourGCPProject \ --gcp-cloud-run-region us-central1 \ --gcp-cloud-run-worker-pool UpdatedWorkerPool \ --gcp-cloud-run-service-account customer-sa@proj.iam.gserviceaccount.com \ --gcp-cloud-run-min-instances 1 \ --gcp-cloud-run-max-instances 3 \ --gcp-cloud-run-initial-instances 1 \ --gcp-cloud-run-utilization-target 0.75 \ --gcp-cloud-run-scale-down-stabilization-duration 5m ``` To update only the scaling settings on an existing GCP Cloud Run Worker Deployment Version, supply the five scaler flags without the provider fields (all five must be set together): ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --gcp-cloud-run-min-instances 1 \ --gcp-cloud-run-max-instances 3 \ --gcp-cloud-run-initial-instances 1 \ --gcp-cloud-run-utilization-target 0.75 \ --gcp-cloud-run-scale-down-stabilization-duration 5m ``` Provider fields are only required when changing the compute provider. Switching the provider resets the scaling settings for the new provider. If --remove is specified, the compute configuration for the Worker Deployment Version will be removed: ``` temporal worker deployment update-version-compute-config \ --deployment-name YourDeploymentName --build-id YourBuildID \ --remove ``` If a Worker Deployment Version with the supplied BuildID does not exist, this command will return an error. Note: This is an experimental feature and may change in the future. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--aws-agentcore-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Agentcore Runtime that spawns a new Worker in this Worker Deployment Version. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. | | `--aws-agentcore-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-agentcore-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-agentcore-endpoint-arn is specified, and must be omitted when --aws-agentcore-skip-role-and-external-id is passed. | | `--aws-agentcore-endpoint-arn` | No | **string** AWS Bedrock Agentcore Runtime endpoint ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. The endpoint ARN encodes the runtime, endpoint name, and region. | | `--aws-agentcore-skip-role-and-external-id` | No | **bool** When --aws-agentcore-endpoint-arn is specified, --aws-agentcore-assume-role-arn and --aws-agentcore-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. | | `--aws-lambda-assume-role-arn` | No | **string** AWS IAM role ARN that the Temporal server will assume when invoking the Lambda function that spawns a new Worker in this Worker Deployment Version. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. | | `--aws-lambda-assume-role-external-id` | No | **string** Temporal server will enforce that the AWS IAM trust policy associated with the AWS IAM role specified in --aws-lambda-assume-role-arn has an aws:ExternalId condition that matches the supplied value. Required when --aws-lambda-function-arn is specified, and must be omitted when --aws-lambda-skip-role-and-external-id is passed. | | `--aws-lambda-function-arn` | No | **string** Qualified (contains version suffix) or unqualified AWS Lambda function ARN to invoke when there are no active pollers for task queue targets in the Worker Deployment. | | `--aws-lambda-skip-role-and-external-id` | No | **bool** When --aws-lambda-function-arn is specified, --aws-lambda-assume-role-arn and --aws-lambda-assume-role-external-id are required unless this flag is passed, in which case both must be omitted. | | `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--gcp-cloud-run-initial-instances` | No | **int** Number of Cloud Run worker pool instances the scaler starts with. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together, and this value must be between the min and max (inclusive). If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. | | `--gcp-cloud-run-max-instances` | No | **int** Maximum number of Cloud Run worker pool instances the scaler may scale up to. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. | | `--gcp-cloud-run-min-instances` | No | **int** Minimum number of Cloud Run worker pool instances the scaler will maintain. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. | | `--gcp-cloud-run-project` | No | **string** GCP project ID hosting the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-region` | No | **string** Region of the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-scale-down-stabilization-duration` | No | **duration** Duration the scaler waits after it last saw unmet task demand before it may scale the Cloud Run worker pool down. Raise this to keep the pool from scaling down before long-running or bursty activities finish. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. A value of 0s disables the wait. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. | | `--gcp-cloud-run-service-account` | No | **string** Customer GCP service account the Temporal server impersonates to manage the Cloud Run worker pool. Required when --gcp-cloud-run-worker-pool is specified. | | `--gcp-cloud-run-utilization-target` | No | **float** Target average worker utilization the scaler aims for, as a fraction in the range (0, 1]. Optional, but --gcp-cloud-run-min-instances, --gcp-cloud-run-max-instances, --gcp-cloud-run-initial-instances, --gcp-cloud-run-utilization-target, and --gcp-cloud-run-scale-down-stabilization-duration must all be set together. Lower values keep more spare capacity per worker. If omitted, the version's existing scaling settings are left unchanged. Only applies to a GCP Cloud Run worker pool. | | `--gcp-cloud-run-worker-pool` | No | **string** GCP Cloud Run worker pool name to scale when there are no active pollers for task queue targets in the Worker Deployment. | | `--remove` | No | **bool** Removes any compute configuration associated with this Worker Deployment Version. | ### update-version-metadata Update metadata associated with a Worker Deployment Version. For example: ``` temporal worker deployment update-version-metadata \ --deployment-name YourDeploymentName --build-id YourBuildID \ --metadata bar=1 \ --metadata foo=true ``` The current metadata is also returned with `describe-version`: ``` temporal worker deployment describe-version \ --deployment-name YourDeploymentName --build-id YourBuildID \ ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--build-id` | Yes | **string** Build ID of the Worker Deployment Version. | | `--deployment-name` | Yes | **string** Name of the Worker Deployment. | | `--metadata` | No | **string[]** Set deployment metadata using `KEY="VALUE"` pairs. Keys must be identifiers, and values must be JSON values. For example: `YourKey={"your": "value"}` Can be passed multiple times. | | `--remove-entries` | No | **string[]** Keys of entries to be deleted from metadata. Can be passed multiple times. | ## describe Look up information of a specific worker. ``` temporal worker describe --namespace YourNamespace --worker-instance-key YourKey ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--worker-instance-key` | Yes | **string** Worker instance key to describe. | ## list Get a list of workers to the specified namespace. ``` temporal worker list --namespace YourNamespace --query 'TaskQueue="YourTaskQueue"' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--limit` | No | **int** Maximum number of workers to display. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # Temporal CLI workflow command reference Source: https://docs.temporal.io/cli/command-reference/workflow > Temporal Workflow commands enable operations on Workflow Executions, such as cancel, count, delete, describe, execute, list, update-options, query, reset, reset-batch, show, signal, stack, start, terminate, trace, and update, enhancing efficiency and control. This page provides a reference for the `temporal` CLI `workflow` command. The flags applicable to each subcommand are presented in a table within the heading for the subcommand. Refer to [Global Flags](#global-flags) for flags that you can use with every subcommand. ## cancel Canceling a running Workflow Execution records a `WorkflowExecutionCancelRequested` event in the Event History. The Service schedules a new Command Task, and the Workflow Execution performs any cleanup work supported by its implementation. Use the Workflow ID to cancel an Execution: ``` temporal workflow cancel \ --workflow-id YourWorkflowId ``` A visibility Query lets you send bulk cancellations to Workflow Executions matching the results: ``` temporal workflow cancel \ --query YourQuery ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## count Show a count of Workflow Executions, regardless of execution state (running, terminated, etc). Use `--query` to select a subset of Workflow Executions: ``` temporal workflow count \ --query YourQuery ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. | ## delete Delete a Workflow Execution and its Event History: ``` temporal workflow delete \ --workflow-id YourWorkflowId ``` The removal executes asynchronously. If the Execution is Running, the Service terminates it before deletion. WARNING: Deleting Workflow Executions in a global Namespace removes them from all replicas. Requests sent to a passive cluster are forwarded to the active cluster by default; to target the passive cluster directly, specify `--grpc-meta xdc-redirection=false`. Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## describe Display information about a specific Workflow Execution: ``` temporal workflow describe \ --workflow-id YourWorkflowId ``` Show the Workflow Execution's auto-reset points: ``` temporal workflow describe \ --workflow-id YourWorkflowId \ --reset-points true ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--raw` | No | **bool** Print properties without changing their format. | | `--reset-points` | No | **bool** Show auto-reset points only. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## execute Establish a new Workflow Execution and direct its progress to stdout. The command blocks and returns when the Workflow Execution completes. If your Workflow requires input, pass valid JSON: ``` temporal workflow execute --workflow-id YourWorkflowId \ --type YourWorkflow \ --task-queue YourTaskQueue \ --input '{"some-key": "some-value"}' ``` Use `--event-details` to relay updates to the command-line output in JSON format. When using JSON output (`--output json`), this includes the entire "history" JSON key for the run. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--cron` | No | **string** Cron schedule for the Workflow. | | `--detailed` | No | **bool** Display events as sections instead of table. Does not apply to JSON output. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fail-existing` | No | **bool** Fail if the Workflow already exists. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. | | `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--type` | Yes | **string** Workflow Type name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## execute-update-with-start Send a message to a Workflow Execution to invoke an Update handler, and wait for the update to complete. If the Workflow Execution is not running, then a new workflow execution is started and the update is sent. Experimental. ``` temporal workflow execute-update-with-start \ --update-name YourUpdate \ --update-input '{"update-key": "update-value"}' \ --workflow-id YourWorkflowId \ --type YourWorkflowType \ --task-queue YourTaskQueue \ --id-conflict-policy Fail \ --input '{"wf-key": "wf-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--cron` | No | **string** Cron schedule for the Workflow. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fail-existing` | No | **bool** Fail if the Workflow already exists. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. | | `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--type` | Yes | **string** Workflow Type name. | | `--update-first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. | | `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. | | `--update-input` | No | **string[]** Update input value. Use JSON content or set --update-input-meta to override. Can't be combined with --update-input-file. Can be passed multiple times to pass multiple arguments. | | `--update-input-base64` | No | **bool** Assume update inputs are base64-encoded and attempt to decode them. | | `--update-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --update-input-meta to override. Can't be combined with --update-input. Can be passed multiple times to pass multiple arguments. | | `--update-input-meta` | No | **string[]** Input update payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. | | `--update-name` | Yes | **string** Update name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## fix-history-json Reserialize an Event History JSON file: ``` temporal workflow fix-history-json \ --source /path/to/original.json \ --target /path/to/reserialized.json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--source`, `-s` | Yes | **string** Path to the original file. | | `--target`, `-t` | No | **string** Path to the results file. When omitted, output is sent to stdout. | ## list List Workflow Executions. The optional `--query` limits the output to Workflows matching a Query: ``` temporal workflow list \ --query YourQuery ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. View a list of archived Workflow Executions: ``` temporal workflow list \ --archived ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--archived` | No | **bool** Limit output to archived Workflow Executions. _(Experimental)_ | | `--limit` | No | **int** Maximum number of Workflow Executions to display. | | `--page-size` | No | **int** Maximum number of Workflow Executions to fetch at a time from the server. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. | ## metadata Issue a Query for and display user-set metadata like summary and details for a specific Workflow Execution: ``` temporal workflow metadata \ --workflow-id YourWorkflowId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--reject-condition` | No | **string-enum** Optional flag for rejecting Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## pause Pause a Workflow Execution. Note: This is an experimental feature and may change in the future. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--reason` | No | **string** Reason for pausing the Workflow Execution. Defaults to message with the current user's name. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## query Send a Query to a Workflow Execution by Workflow ID to retrieve its state. This synchronous operation exposes the internal state of a running Workflow Execution, which constantly changes. You can query both running and completed Workflow Executions: ``` temporal workflow query \ --workflow-id YourWorkflowId --type YourQueryType --input '{"YourInputKey": "YourInputValue"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--name` | Yes | **string** Query Type/Name. | | `--reject-condition` | No | **string-enum** Optional flag for rejecting Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## reset Reset a Workflow Execution so it can resume from a point in its Event History without losing its progress up to that point: ``` temporal workflow reset \ --workflow-id YourWorkflowId \ --event-id YourLastEvent ``` Start from where the Workflow Execution last continued as new: ``` temporal workflow reset \ --workflow-id YourWorkflowId \ --type LastContinuedAsNew ``` For batch resets, limit your resets to FirstWorkflowTask, LastWorkflowTask, or BuildId. Do not use Workflow IDs, run IDs, or event IDs with this command. Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. ### with-workflow-update-options Run Workflow Update Options atomically after the Workflow is reset. Workflows selected by the reset command are forwarded onto the subcommand. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--versioning-override-behavior` | Yes | **string-enum** Override the versioning behavior of a Workflow. Accepted values: pinned, auto_upgrade. | | `--versioning-override-build-id` | No | **string** When overriding to a `pinned` behavior, specifies the Build ID of the version to target. | | `--versioning-override-deployment-name` | No | **string** When overriding to a `pinned` behavior, specifies the Deployment Name of the version to target. | ## result Wait for and print the result of a Workflow Execution: ``` temporal workflow result \ --workflow-id YourWorkflowId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## show Show a Workflow Execution's Event History. When using JSON output (`--output json`), you may pass the results to an SDK to perform a replay: ``` temporal workflow show \ --workflow-id YourWorkflowId --output json ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--detailed` | No | **bool** Display events as detailed sections instead of table. Does not apply to JSON output. | | `--follow`, `-f` | No | **bool** Follow the Workflow Execution progress in real time. Does not apply to JSON output. | | `--reverse` | No | **bool** Fetch Event History newest-event-first. Cannot be combined with --follow. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## signal Send an asynchronous notification (Signal) to a running Workflow Execution by its Workflow ID. The Signal is written to the History. When you include `--input`, that data is available for the Workflow Execution to consume: ``` temporal workflow signal \ --workflow-id YourWorkflowId \ --name YourSignal \ --input '{"YourInputKey": "YourInputValue"}' ``` Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--name` | Yes | **string** Signal name. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## signal-with-start Send an asynchronous notification (Signal) to a Workflow Execution. If the Workflow Execution is not running or is not found, it starts the workflow then sends the signal. ``` temporal workflow signal-with-start \ --signal-name YourSignal \ --signal-input '{"some-key": "some-value"}' \ --workflow-id YourWorkflowId \ --type YourWorkflowType \ --task-queue YourTaskQueue \ --input '{"some-key": "some-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--cron` | No | **string** Cron schedule for the Workflow. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fail-existing` | No | **bool** Fail if the Workflow already exists. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. | | `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--signal-input` | No | **string[]** Signal input value. Use JSON content or set --signal-input-meta to override. Can't be combined with --signal-input-file. Can be passed multiple times to pass multiple arguments. | | `--signal-input-base64` | No | **bool** Assume signal inputs are base64-encoded and attempt to decode them. | | `--signal-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --signal-input-meta to override. Can't be combined with --signal-input. Can be passed multiple times to pass multiple arguments. | | `--signal-input-meta` | No | **string[]** Input signal payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. | | `--signal-name` | Yes | **string** Signal name. | | `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--type` | Yes | **string** Workflow Type name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## stack Perform a Query on a Workflow Execution using a `__stack_trace`-type Query. Display a stack trace of the threads and routines currently in use by the Workflow for troubleshooting: ``` temporal workflow stack \ --workflow-id YourWorkflowId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--reject-condition` | No | **string-enum** Optional flag to reject Queries based on Workflow state. Accepted values: not_open, not_completed_cleanly. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## start Start a new Workflow Execution. Returns the Workflow- and Run-IDs: ``` temporal workflow start \ --workflow-id YourWorkflowId \ --type YourWorkflow \ --task-queue YourTaskQueue \ --input '{"some-key": "some-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--cron` | No | **string** Cron schedule for the Workflow. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fail-existing` | No | **bool** Fail if the Workflow already exists. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. | | `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--type` | Yes | **string** Workflow Type name. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## start-update-with-start Send a message to a Workflow Execution to invoke an Update handler, and wait for the update to be accepted or rejected. If the Workflow Execution is not running, then a new workflow execution is started and the update is sent. Experimental. ``` temporal workflow start-update-with-start \ --update-name YourUpdate \ --update-input '{"update-key": "update-value"}' \ --update-wait-for-stage accepted \ --workflow-id YourWorkflowId \ --type YourWorkflowType \ --task-queue YourTaskQueue \ --id-conflict-policy Fail \ --input '{"wf-key": "wf-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--cron` | No | **string** Cron schedule for the Workflow. | | `--execution-timeout` | No | **duration** Fail a WorkflowExecution if it lasts longer than `DURATION`. This time-out includes retries and ContinueAsNew tasks. | | `--fail-existing` | No | **bool** Fail if the Workflow already exists. | | `--fairness-key` | No | **string** Fairness key (max 64 bytes) for proportional task dispatch. Tasks with same key share capacity based on their weight. | | `--fairness-weight` | No | **float** Weight [0.001-1000] for this fairness key. Keys are dispatched proportionally to their weights. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--id-conflict-policy` | No | **string-enum** Determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. Accepted values: Fail, UseExisting, TerminateExisting. | | `--id-reuse-policy` | No | **string-enum** Re-use policy for the Workflow ID in new Workflow Executions. Accepted values: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--memo` | No | **string[]** Memo using 'KEY="VALUE"' pairs. Use JSON values. | | `--priority-key` | No | **int** Priority key (1-5, lower numbers = higher priority). Tasks in a queue should be processed in close-to-priority-order. Default is 3 when not specified. | | `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. | | `--run-timeout` | No | **duration** Fail a Workflow Run if it lasts longer than `DURATION`. | | `--search-attribute` | No | **string[]** Search Attribute in `KEY=VALUE` format. Keys must be identifiers, and values must be JSON values. For example: `'YourKey={"your": "value"}'`. Can be passed multiple times. | | `--start-delay` | No | **duration** Delay before starting the Workflow Execution. Can't be used with cron schedules. If the Workflow receives a signal or update prior to this time, the Workflow Execution starts immediately. | | `--static-details` | No | **string** Static Workflow details for human consumption in UIs. Uses Temporal Markdown formatting, may be multiple lines. _(Experimental)_ | | `--static-summary` | No | **string** Static Workflow summary for human consumption in UIs. Uses Temporal Markdown formatting, should be a single line. _(Experimental)_ | | `--task-queue`, `-t` | Yes | **string** Workflow Task queue. | | `--task-timeout` | No | **duration** Fail a Workflow Task if it lasts longer than `DURATION`. This is the Start-to-close timeout for a Workflow Task. | | `--type` | Yes | **string** Workflow Type name. | | `--update-first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. | | `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. | | `--update-input` | No | **string[]** Update input value. Use JSON content or set --update-input-meta to override. Can't be combined with --update-input-file. Can be passed multiple times to pass multiple arguments. | | `--update-input-base64` | No | **bool** Assume update inputs are base64-encoded and attempt to decode them. | | `--update-input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --update-input-meta to override. Can't be combined with --update-input. Can be passed multiple times to pass multiple arguments. | | `--update-input-meta` | No | **string[]** Input update payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. | | `--update-name` | Yes | **string** Update name. | | `--update-wait-for-stage` | Yes | **string-enum** Update stage to wait for. The only option is `accepted`, but this option is required. This is to allow a future version of the CLI to choose a default value. Accepted values: accepted. | | `--workflow-id`, `-w` | No | **string** Workflow ID. If not supplied, the Service generates a unique ID. | ## terminate Terminate a Workflow Execution: ``` temporal workflow terminate \ --reason YourReasonForTermination \ --workflow-id YourWorkflowId ``` The reason is optional and defaults to the current user's name. The reason is stored in the Event History as part of the `WorkflowExecutionTerminated` event. This becomes the closing Event in the Workflow Execution's history. Executions may be terminated in bulk via a visibility Query list filter: ``` temporal workflow terminate \ --query YourQuery \ --reason YourReasonForTermination ``` Workflow code cannot see or respond to terminations. To perform clean-up work in your Workflow code, use `temporal workflow cancel` instead. Visit https://docs.temporal.io/visibility to read more about Search Attributes and Query creation. See `temporal batch --help` for a quick reference. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. | | `--reason` | No | **string** Reason for termination. Defaults to message with the current user's name. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Can only be set with --workflow-id. Do not use with --query. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm termination. Can only be used with --query. | ## trace Display the progress of a Workflow Execution and its child workflows with a real-time trace. This view helps you understand how Workflows are proceeding: ``` temporal workflow trace \ --workflow-id YourWorkflowId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--concurrency` | No | **int** Number of Workflow Histories to fetch at a time. | | `--depth` | No | **int** Set depth for your Child Workflow fetches. Pass -1 to fetch child workflows at any depth. | | `--fold` | No | **string[]** Fold away Child Workflows with the specified statuses. Case-insensitive. Ignored if --no-fold supplied. Available values: running, completed, failed, canceled, terminated, timedout, continueasnew. Can be passed multiple times. | | `--no-fold` | No | **bool** Disable folding. Fetch and display Child Workflows within the set depth. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## unpause Unpause a previously paused Workflow Execution. Note: This is an experimental feature and may change in the future. Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--reason` | No | **string** Reason for unpausing the Workflow Execution. Defaults to message with the current user's name. | | `--run-id`, `-r` | No | **string** Run ID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## update An Update is a synchronous call to a Workflow Execution that can change its state, control its flow, and return a result. ### describe Given a Workflow Execution and an Update ID, return information about its current status, including a result if it has finished. ``` temporal workflow update describe \ --workflow-id YourWorkflowId \ --update-id YourUpdateId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--run-id`, `-r` | No | **string** Run ID. If unset, updates the currently-running Workflow Execution. | | `--update-id` | Yes | **string** Update ID. Must be unique per Workflow Execution. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ### execute Send a message to a Workflow Execution to invoke an Update handler, and wait for the update to complete or fail. You can also use this to wait for an existing update to complete, by submitting an existing update ID. ``` temporal workflow update execute \ --workflow-id YourWorkflowId \ --name YourUpdate \ --input '{"some-key": "some-value"}' ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--name` | Yes | **string** Handler method name. | | `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. | | `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ### result Given a Workflow Execution and an Update ID, wait for the Update to complete or fail and print the result. ``` temporal workflow update result \ --workflow-id YourWorkflowId \ --update-id YourUpdateId ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--run-id`, `-r` | No | **string** Run ID. If unset, updates the currently-running Workflow Execution. | | `--update-id` | Yes | **string** Update ID. Must be unique per Workflow Execution. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ### start Send a message to a Workflow Execution to invoke an Update handler, and wait for the update to be accepted or rejected. You can subsequently wait for the update to complete by using `temporal workflow update execute`. ``` temporal workflow update start \ --workflow-id YourWorkflowId \ --name YourUpdate \ --input '{"some-key": "some-value"}' --wait-for-stage accepted ``` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--first-execution-run-id` | No | **string** Parent Run ID. The update is sent to the last Workflow Execution in the chain started with this Run ID. | | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--input`, `-i` | No | **string[]** Input value. Use JSON content or set --input-meta to override. Can't be combined with --input-file. Can be passed multiple times to pass multiple arguments. | | `--input-base64` | No | **bool** Assume inputs are base64-encoded and attempt to decode them. | | `--input-file` | No | **string[]** A path or paths for input file(s). Use JSON content or set --input-meta to override. Can't be combined with --input. Can be passed multiple times to pass multiple arguments. | | `--input-meta` | No | **string[]** Input payload metadata as a `KEY=VALUE` pair. When the KEY is "encoding", this overrides the default ("json/plain"). Can be passed multiple times. Repeated metadata keys are applied to the corresponding inputs in the provided order. | | `--name` | Yes | **string** Handler method name. | | `--run-id`, `-r` | No | **string** Run ID. If unset, looks for an Update against the currently-running Workflow Execution. | | `--update-id` | No | **string** Update ID. If unset, defaults to a UUID. | | `--wait-for-stage` | Yes | **string-enum** Update stage to wait for. The only option is `accepted`, but this option is required. This is to allow a future version of the CLI to choose a default value. Accepted values: accepted. | | `--workflow-id`, `-w` | Yes | **string** Workflow ID. | ## update-options Modify properties of Workflow Executions: ``` temporal workflow update-options [options] ``` It can override the Worker Deployment configuration of a Workflow Execution, which controls Worker Versioning. For example, to force Workers in the current Deployment execute the next Workflow Task change behavior to `auto_upgrade`: ``` temporal workflow update-options \ --workflow-id YourWorkflowId \ --versioning-override-behavior auto_upgrade ``` or to pin the workflow execution to a Worker Deployment, set behavior to `pinned`: ``` temporal workflow update-options \ --workflow-id YourWorkflowId \ --versioning-override-behavior pinned \ --versioning-override-deployment-name YourDeploymentName \ --versioning-override-build-id YourDeploymentBuildId ``` To remove any previous overrides, set the behavior to `unspecified`: ``` temporal workflow update-options \ --workflow-id YourWorkflowId \ --versioning-override-behavior unspecified ``` To see the current override use `temporal workflow describe` Use the following options to change the behavior of this command. You can also use any of the [global flags](#global-flags) that apply to all subcommands. | Flag | Required | Description | |------|----------|-------------| | `--headers` | No | **string[]** Temporal workflow headers in 'KEY=VALUE' format. Keys must be identifiers, and values must be JSON values. May be passed multiple times to set multiple Temporal headers. Note: These are workflow headers, not gRPC headers. | | `--query`, `-q` | No | **string** Content for an SQL-like `QUERY` List Filter. You must set either --workflow-id or --query. | | `--reason` | No | **string** Reason for batch operation. Only use with --query. Defaults to user name. | | `--rps` | No | **float** Limit batch's requests per second. Only allowed if query is present. | | `--run-id`, `-r` | No | **string** Run ID. Only use with --workflow-id. Cannot use with --query. | | `--versioning-override-behavior` | Yes | **string-enum** Override the versioning behavior of a Workflow. Accepted values: unspecified, pinned, auto_upgrade. | | `--versioning-override-build-id` | No | **string** When overriding to a `pinned` behavior, specifies the Build ID of the version to target. | | `--versioning-override-deployment-name` | No | **string** When overriding to a `pinned` behavior, specifies the Deployment Name of the version to target. | | `--workflow-id`, `-w` | No | **string** Workflow ID. You must set either --workflow-id or --query. | | `--yes`, `-y` | No | **bool** Don't prompt to confirm signaling. Only allowed when --query is present. | ## Global Flags The following options can be used with any command. | Flag | Required | Description | Default | |------|----------|-------------|--------| | `--address` | No | **string** Temporal Service gRPC endpoint. | `localhost:7233` | | `--api-key` | No | **string** API key for request. | | | `--client-authority` | No | **string** Temporal gRPC client :authority pseudoheader. | | | `--client-connect-timeout` | No | **duration** Client connection timeout. | | | `--codec-auth` | No | **string** Authorization header for Codec Server requests. | | | `--codec-endpoint` | No | **string** Remote Codec Server endpoint. | | | `--codec-header` | No | **string[]** HTTP headers for codec server (KEY=VALUE, repeatable). | | | `--color` | No | **string-enum** Output coloring. Accepted values: always, never, auto. | `auto` | | `--command-timeout` | No | **duration** Command execution timeout. | | | `--config-file` | No | **string** TOML config file path. | | | `--disable-config-env` | No | **bool** Disable loading config from environment variables. | | | `--disable-config-file` | No | **bool** Disable loading config from file. | | | `--env` | No | **string** Active environment name (`ENV`). | `default` | | `--env-file` | No | **string** Path to environment settings file. | | | `--grpc-meta` | No | **string[]** HTTP headers for requests (KEY=VALUE, repeatable). | | | `--identity` | No | **string** Identity of the client submitting requests. | | | `--log-format` | No | **string-enum** Log format. Accepted values: text, json. | `text` | | `--log-level` | No | **string-enum** Log level. Default is "never" for most commands and "warn" for "server start-dev". Accepted values: debug, info, warn, error, never. | `never` | | `--namespace`, `-n` | No | **string** Temporal Service Namespace. | `default` | | `--no-json-shorthand-payloads` | No | **bool** Raw payload output, even if the JSON option was used. | | | `--output`, `-o` | No | **string-enum** Non-logging data output format. Accepted values: text, json, jsonl, none. | `text` | | `--profile` | No | **string** Profile to use for config file. | | | `--time-format` | No | **string-enum** Time format. Accepted values: relative, iso, raw. | `relative` | | `--tls` | No | **bool** Enable base TLS encryption. Auto-enabled when api-key or TLS options are set. | | | `--tls-ca-data` | No | **string** Inline server CA certificate data. | | | `--tls-ca-path` | No | **string** Path to server CA certificate. | | | `--tls-cert-data` | No | **string** Inline x509 certificate data. | | | `--tls-cert-path` | No | **string** Path to x509 certificate. | | | `--tls-disable-host-verification` | No | **bool** Disable TLS host-name verification. | | | `--tls-key-data` | No | **string** Inline x509 private key data. | | | `--tls-key-path` | No | **string** Path to x509 private key. | | | `--tls-server-name` | No | **string** Override target TLS server name. | | --- # CLI basics Source: https://docs.temporal.io/cli/common-operations This page walks through the operations you're most likely to reach for when using the Temporal CLI. For the full set of commands, see the [command reference](/cli#command-reference). ## Run a development server The Temporal CLI ships with a complete Temporal binary. Start a local Temporal Service for development with a single command. The Temporal Service is available on `localhost:7233` and the Web UI at `http://localhost:8233`. To keep Workflow data between restarts, specify a database file. See [Install and configure](/cli/setup-cli#run-a-local-development-server) for more development server options. ```bash temporal server start-dev ``` ```bash temporal server start-dev --db-filename temporal.db ``` ## Start a Workflow Start a Workflow Execution with `temporal workflow start`. To start a Workflow and wait for the result, use `temporal workflow execute` instead. ```bash temporal workflow start \\ --task-queue my-task-queue \\ --type MyWorkflow \\ --workflow-id my-workflow-id \\ --input '"my-input-value"' ``` ```bash temporal workflow execute \\ --task-queue my-task-queue \\ --type MyWorkflow \\ --workflow-id my-workflow-id \\ --input '"my-input-value"' ``` ## Check Workflow status List running Workflows. Get details about a specific Workflow Execution. View the Event History for a Workflow Execution. ```bash temporal workflow list ``` ```bash temporal workflow describe --workflow-id my-workflow-id ``` ```bash temporal workflow show --workflow-id my-workflow-id ``` ## Send Signals and Queries Send a [Signal](/sending-messages#sending-signals) to a running Workflow. [Query](/sending-messages#sending-queries) a running Workflow for its current state. ```bash temporal workflow signal \\ --workflow-id my-workflow-id \\ --name my-signal \\ --input '"signal-value"' ``` ```bash temporal workflow query \\ --workflow-id my-workflow-id \\ --name my-query ``` ## Cancel or Terminate a Workflow Cancel a Workflow Execution. Cancellation allows cleanup logic to run before the Workflow completes. Terminate a Workflow Execution. Termination stops the Workflow immediately with no cleanup. ```bash temporal workflow cancel --workflow-id my-workflow-id ``` ```bash temporal workflow terminate --workflow-id my-workflow-id ``` ## Log in to Temporal Cloud Log in to Temporal Cloud with `temporal cloud login`. After login, you can run commands against Temporal Cloud by providing the address and Namespace. See [Use with Temporal Cloud](/cli/cloud) for all authentication options. ```bash temporal cloud login ``` ```bash temporal workflow list \\ --address ..tmprl.cloud:7233 \\ --namespace . ``` ## Work with Schedules Create a [Schedule](/schedule) that starts a Workflow on an interval. List all Schedules. Pause and unpause a Schedule. ```bash temporal schedule create \\ --schedule-id my-schedule \\ --interval 1h \\ --task-queue my-task-queue \\ --type MyScheduledWorkflow ``` ```bash temporal schedule list ``` ```bash temporal schedule toggle \\ --schedule-id my-schedule \\ --pause --reason "maintenance" ``` ```bash temporal schedule toggle \\ --schedule-id my-schedule \\ --unpause --reason "maintenance complete" ``` ## Manage Namespaces List available [Namespaces](/namespaces). Create a new Namespace. To manage Namespaces on Temporal Cloud, use the Temporal Cloud extension and [`temporal cloud namespace`](/cli/command-reference/cloud/namespace) commands. ```bash temporal operator namespace list ``` ```bash temporal operator namespace create --namespace my-namespace ``` ## Format and filter output Use `--output json` to get machine-readable output from any command. Combine with [jq](https://jqlang.github.io/jq/) for filtering. ```bash temporal workflow list --output json ``` ```bash temporal workflow list --output json | jq '.[].type.name' ``` ## Next steps - [Use with Temporal Cloud](/cli/cloud) to connect the CLI to Temporal Cloud. - [Command reference](/cli#command-reference) for the full set of commands and options. --- # Install and configure the CLI Source: https://docs.temporal.io/cli/setup-cli > Install the Temporal CLI, run a local development server, and configure your environment. The Temporal CLI is a command-line tool for interacting with the Temporal Service. It helps you manage, monitor, and debug Temporal applications. ## Install the CLI The CLI is available for macOS, Linux, and Windows, or as a Docker image. **macOS** Install with Homebrew: ```bash brew install temporal ``` Or download from the CDN: - [Darwin amd64](https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64) - [Darwin arm64](https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64) extract the archive and add the `temporal` binary to your `PATH`. **Linux** Install with Homebrew (if available): ```bash brew install temporal ``` Or download from the CDN: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) extract the archive and add the `temporal` binary to your `PATH`. **Windows** Download from the CDN: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) extract the archive and add the `temporal.exe` binary to your `PATH`. **Docker** Temporal CLI container image is available on [DockerHub](https://hub.docker.com/r/temporalio/temporal) and can be run directly: ```shell docker run --rm temporalio/temporal --help ``` > **📝 Note:** > > When running the Temporal CLI inside Docker, for the development server to be accessible from the host system, the > server needs to be configured to listen on external IP and the ports need to be forwarded: > > ```shell > docker run --rm -p 7233:7233 -p 8233:8233 temporalio/temporal server start-dev --ip 0.0.0.0 > # UI is now accessible from host at http://localhost:8233/ > ``` > ## Install the Temporal Cloud extension If you are using Temporal Cloud, install the Temporal Cloud extension for the Temporal CLI. You can install the extension using the following command: > **💡 Tip:** > Support, stability, and dependency info > > The Temporal Cloud extension is in [Public Preview](/evaluate/development-production-features/release-stages#public-preview). > APIs and configuration may change before the stable release. > ```bash brew install temporalio/brew/temporal-cloud ``` ## Run a local development server The CLI includes a local Temporal development service for fast feedback while building your application. Start the server: ```bash temporal server start-dev ``` This command automatically starts the Web UI, creates the `default` [Namespace](/namespaces), and uses an in-memory SQLite database. The Temporal Server will be available on `localhost:7233` and the Temporal Web UI will be available at [`http://localhost:8233`](http://localhost:8233/). Persist state locally by specifying a database file: ```shell temporal server start-dev --db-filename temporal.db ``` ### Development server configuration #### Namespace registration Namespaces are pre-registered at startup for immediate use. Customize pre-registered Namespaces with the following command: ```shell temporal server start-dev --namespace foo --namespace bar ``` Register Namespaces with `namespace create`: ```shell temporal operator namespace create --namespace foo ``` #### Enable or turn off the Temporal Web UI By default, the Temporal Web UI is enabled when running the development server using the Temporal CLI. To turn off the UI, use the `--headless` modifier: ```shell temporal server start-dev --headless ``` #### Dynamic configuration Advanced Temporal CLI configuration requires a dynamic configuration file. To set values on the command line, use `--dynamic-config-value KEY=JSON_VALUE`. For example, enable the Search Attribute cache: ```bash temporal server start-dev --dynamic-config-value system.forceSearchAttributesCacheRefreshOnRead=false ``` This setting makes created Search Attributes immediately available. ## Configure the CLI ### Environment variables You can configure the CLI with environment variables instead of passing flags on every command. Setting an environment variable is not the same as storing a preset with [`temporal env`](/cli/command-reference/env): an environment variable configures the CLI process, while `temporal env` writes named key-value presets to a file. For every variable the CLI reads, its equivalent flag, and its TOML configuration key, refer to [Environment configuration](/references/client-environment-configuration). ### Create and modify configuration files The Temporal CLI lets you create and modify TOML configuration files to store your environment variables and other settings. Refer to [Environment Configuration](../develop/environment-configuration#cli-integration) for more information. ### Configure proxy support The Temporal CLI provides support for users who are operating behind a proxy. This feature ensures seamless communication even in network-restricted environments. #### Setting up proxy support If you are behind a proxy, you'll need to instruct the Temporal CLI to route its requests via that proxy. You can achieve this by setting the `HTTPS_PROXY` environment variable. ```command export HTTPS_PROXY=: ``` Replace `` with the proxy's hostname or IP address, and `` with the proxy's port number. Once set, you can run the Temporal CLI commands as you normally would. > **📝 Note:** > > Temporal CLI uses the gRPC library which natively supports HTTP CONNECT proxies. The gRPC library checks for the > `HTTPS_PROXY` (and its case-insensitive variants) environment variable to determine if it should route requests through > a proxy. > In addition to `HTTPS_PROXY`, gRPC also respects the `NO_PROXY` environment variable. This can be useful if there are specific addresses or domains you wish to exclude from proxying. For more information, see [Proxy](https://github.com/grpc/grpc-go/blob/master/Documentation/proxy.md) in the gRPC documentation. ## Enable auto-completion Enable auto-completion using the following commands. ### zsh auto-completion 1. Add the following line to your `~/.zshrc` startup script: ```sh eval "$(temporal completion zsh)" ``` 2. Re-launch your shell or run: ```sh source ~/.zshrc ``` ### Bash auto-completion 1. Install [bash-completion](https://github.com/scop/bash-completion#installation) and add the software to your `~/.bashrc`. 2. Add the following line to your `~/.bashrc` startup script: ```sh eval "$(temporal completion bash)" ``` 3. Re-launch your shell or run: ```sh source ~/.bashrc ``` > **📝 Note:** > > If auto-completion fails with the error: `bash: _get_comp_words_by_ref: command not found`, you did not successfully > install [bash-completion](https://github.com/scop/bash-completion#installation). This package must be loaded into your > shell for `temporal` auto-completion to work. > ### Fish auto-completion 1. Create the Fish custom completions directory if it does not already exist: ```fish mkdir -p ~/.config/fish/completions ``` 2. Configure the completions to load when needed. Note: the filename must be `temporal.fish` or the completions will not be found: ```fish echo 'eval "$(temporal completion fish)"' >~/.config/fish/completions/temporal.fish ``` 3. Re-launch your shell or run: ```fish source ~/.config/fish/completions/temporal.fish ``` ## Getting CLI help From the command line: ``` temporal --help ``` For example: - `temporal --help` - `temporal workflow --help` - `temporal workflow delete --help` For a full list of commands, see the [Temporal CLI command reference](/cli#command-reference). --- # Temporal Cloud guide Source: https://docs.temporal.io/cloud > Find guides for onboarding, security, pricing, monitoring, access management, Nexus, and the Cloud Ops API in one place. Welcome to the Temporal Cloud guide. In this guide you will find information about Temporal Cloud, onboarding, features, and how to use them. To create a Temporal Cloud account, sign up [here](https://temporal.io/get-cloud). **[Get started with Temporal Cloud.](/cloud/get-started)** ## Become familiar with Temporal Cloud - [Overview of Temporal Cloud](/cloud/overview) - [Security model](/cloud/security) - [Service availability](/cloud/service-availability) (availability, region support, throughput, latency, and limits) - [Account, Namespace, and application level configurations](/cloud/limits) - [Service Level Agreement (SLA)](/cloud/sla) - [Pricing](/cloud/pricing) - [Support](/cloud/support) ## Feature guides - [Get started with Temporal Cloud](/cloud/get-started) - [Manage certificates](/cloud/certificates) - [Manage API keys](/cloud/api-keys) - [Manage Namespaces](/cloud/namespaces) - [Manage users](/cloud/manage-access/users) - [Manage user groups](/cloud/manage-access/user-groups) - [Manage billing](/cloud/billing) - [Manage Service Accounts](/cloud/manage-access/service-accounts) - [API key feature guide](/cloud/api-keys) - [Monitor Temporal Cloud](/cloud/monitor) - [Set up metrics](/cloud/metrics) - [Monitor Worker health](/cloud/worker-health) - [Monitor service health](/cloud/service-health) - [Receive notifications](/cloud/notifications) - [Temporal Nexus](/cloud/nexus) - [SAML authentication feature guide](/cloud/manage-access/saml) - [Cloud Ops API](/ops) - [Audit logging feature guide](/cloud/audit-logs) - [Temporal CLI Cloud extension](/cli/cloud) - [`tcld` (Temporal Cloud command-line interface) reference](/cloud/tcld) --- # Temporal Cloud Actions Source: https://docs.temporal.io/cloud/actions Temporal Cloud Actions are the primary unit of consumption-based pricing for Temporal Cloud. They track billable operations within the Temporal Cloud Service, such as starting Workflows, recording a Heartbeat, or sending Signals. Actions can be largely placed in the following categories: - [Workflow](#workflow) - [Activity](#activity) - [Timer](#timer) - [Signal](#signal) - [Query](#query) - [Update](#update) - [Schedule](#schedule) - [Nexus](#nexus) Some additional Temporal Cloud features are billed as Actions: - [Export](#export) - [Fairness](#fairness) - [Capacity](#capacity) Actions that occur during [Workflow Replay](/workflow-execution#replay) do not count towards billed Actions. Replay happens only on the Worker side to reconstruct Workflow state from Event History and does not generate new server-side operations. Billable Actions are visible in the [OpenMetrics endpoint](/cloud/metrics/openmetrics), [Event History](/cloud/actions-usage#actions-in-workflows) and [Usage](https://cloud.temporal.io/usage) Dashboards, and the [Billing API](/cloud/billing-api). Action types and categories can help estimate usage, identify specific Action types that are driving usage, optimize workflows by tracking usage spikes, and troubleshoot errors. For example: - You can see if a new Workflow type on a Namespace is driving a significant usage increase - You can identify the aggregate usage of Activity Heartbeats on a Namespace The following is the mapping of Action Categories to Action types. The Action Types that are visible on various endpoints can differ based on what is visible to a system. Action Categories and Action Types are not available in the billing API. Usage and Billing data will have the most complete Actions data while Event history and metrics will help with detailed _estimates_. For example, [History Event Types](/workflow-execution/event) are provided for transparency, but do not always have a simple 1:1 relationship with Action Types. The Categories, Action types, and available endpoints are listed in the following sections: ## Workflow - **Workflow started** - Occurs via client start, [Continue-As-New](/workflow-execution/continue-as-new), [Child Workflow](/child-workflows) start. - If a Workflow start fails, an Action is not recorded. - De-duplicated Workflow starts that share a Workflow ID do _not_ count as an Action. - **Workflow reset** - Occurs when a [Workflow](/workflows) is reset. - Actions that occur before a [Reset](/workflow-execution/event#reset) are counted on the original workflow. These events are not counted on the newly created workflow. - **Search Attribute upsert requested** - Occurs for each invocation of `UpsertSearchAttributes` command. - Multiple Search Attributes updated in a single `UpsertSearchAttributes` command count as one Action. - Search Attributes specified during Workflow start are _excluded_ from Action counts. - The `TemporalChangeVersion` Search Attribute, used for Workflow versioning, is also exempt from Action counting. - **Workflow Execution Options updated.** - Occurs for every [Workflow-Execution-Options-Updated](/references/events#workflowexecutionoptionsupdated) event. - This includes attaching a Workflow completion callback or modifying a Workflow versioning override. - **Start Child Workflow Execution initiated.** - Occurs when the parent Workflow durably records the intent to start a Child Workflow. - Results in two Actions, one for durably storing the intent to start a Child Workflow and one for the attempt to start it. - **Workflow started via Multi-Operation.** - Occurs when a Workflow is started as part of an ExecuteMultiOperation request — a single atomic RPC that bundles a Workflow start with one or more Workflow Updates. The start counts as one Action. - Each bundled Update that is accepted or rejected counts as an additional Action. - If the Workflow was already running and the start is de-duplicated, no Action is charged for the start (though an `update_workflow_options_via_start` Action may still apply). | Usage Name | Metric Name | History Event Type | | ----------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Start Workflow | `start_workflow` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` | | Start Child Workflow | `start_child_workflow` | `EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED` | | Continue As New Workflow | `continue_as_new_workflow` | `EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW` Note: The UI shows a billable action only for the `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` event of the newly started Workflow, not for this history event. | | Start Workflow (Multi-Operation) | `start_workflow_multi_operation` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` | | Update Workflow Options | `update_workflow_options` | `EVENT_TYPE_WORKFLOW_EXECUTION_OPTIONS_UPDATED` | | Update Workflow Options (Via Start) | `update_workflow_options_via_start` | `EVENT_TYPE_WORKFLOW_EXECUTION_OPTIONS_UPDATED` | | Upsert Workflow Search Attributes | `upsert_workflow_search_attributes` | `EVENT_TYPE_UPSERT_WORKFLOW_SEARCH_ATTRIBUTES` | | Reset Workflow | `reset_workflow` | `EVENT_TYPE_WORKFLOW_TASK_FAILED` Note: When `cause == WORKFLOW_TASK_FAILED_CAUSE_RESET_WORKFLOW` | | Record Workflow Heartbeat | `record_workflow_heartbeat` | N/A | ## Activity - **Activity started or retried**. Occurs each time an Activity is started or retried. - **Standalone Activity started**. Occurs when a [Standalone Activity](/standalone-activity) is started. - De-duplicated Standalone Activity starts that return an already-running Activity (sharing an Activity Id) do _not_ count as an Action, unless the start request tries to attach a callback to the running Activity — for example, within a Nexus handler using the `USE_EXISTING` conflict policy. - **Local Activity started**. All [Local Activities](/local-activity) associated with one Workflow Task count as a single Action. Temporal Cloud counts all [RecordMarkers](/references/commands#recordmarker) from each Workflow Task as one action, and not _N_ actions. Note: - Each Workflow Task Heartbeat counts as an additional Action. - Local Activities retried following a Workflow Task Heartbeat count as one Action (capped at 100 Actions). - **Activity Heartbeat recorded**. A Heartbeat call from Activity code counts as an Action only if it reaches the [Temporal Server](/temporal-service/temporal-server). Temporal SDKs throttle [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). The default throttle is 80% of the [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout). Heartbeats don't apply to Local Activities. | Usage Name | Metric Name | History Event Type | | ------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schedule Activity | `schedule_activity` | `EVENT_TYPE_ACTIVITY_TASK_SCHEDULED` | | Retry Activity | `retry_activity` | `EVENT_TYPE_ACTIVITY_TASK_STARTED` Note: Intermediate retry attempts do not produce separate history events. Total attempt count is recorded in the `ActivityTaskStartedEventAttributes.attempt` field on this event. The user should subtract 1, because the first attempt is not a retry. If the Workflow is terminated while Activity retry is in progress, this `STARTED` event is not written, so this data may not always be available. | | Retry Local Activity | `retry_activity_local` | `EVENT_TYPE_WORKFLOW_TASK_COMPLETED ` Note: the retry count is recorded in `WorkflowTaskCompletedEventAttributes.metering_metadata.nonfirst_local_activity_execution_attempts` field on this event. | | Record Activity Heartbeat | `record_activity_heartbeat` | N/A | | Record Activity Heartbeat by ID | `record_activity_heartbeat_by_id` | N/A | | Record Activity Markers | `record_activity_markers` | `EVENT_TYPE_MARKER_RECORDED` | | Start Standalone Activity | `start_standalone_activity` | N/A | | Retry Standalone Activity | `retry_standalone_activity` | N/A | | Record Standalone Activity Heartbeat | `record_standalone_activity_heartbeat` | N/A | | Record Standalone Activity Heartbeat By ID | `record_standalone_activity_heartbeat_by_id` | N/A | ## Timer - **Timer started**. Includes implicit Timers that are started by a Temporal SDK when timeouts are set, such as `AwaitWithTimeout` in Go or `condition` in TypeScript. | Usage Name | Metric Name | History Event Type | | ----------- | ------------- | -------------------------- | | Start Timer | `start_timer` | `EVENT_TYPE_TIMER_STARTED` | ## Signal - **Signal sent**. An Action occurs for every [Signal](/sending-messages#sending-signals), whether sent from a Client or from a Workflow. Also, one total action occurs for any [Signal-With-Start](/sending-messages#signal-with-start), regardless of whether the Workflow starts. | Usage Name | Metric Name | History Event Type | | -------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Signal Workflow | `signal_workflow` | `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` | | Signal External Workflow | `signal_external_workflow` | `EVENT_TYPE_SIGNAL_EXTERNAL_WORKFLOW_EXECUTION_INITIATED` (signaler event history) `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` (signaled Event history) Note: The UI shows a billable action only in the signaled Event history, not in the signaler Event history. | | Signal Workflow With Start | `signal_workflow_with_start` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` followed by `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` if the workflow didn’t exist, or only `EVENT_TYPE_WORKFLOW_EXECUTION_SIGNALED` if the Workflow existed. | ## Query - **Query received by Worker**. An Action occurs for every [Query](/sending-messages#sending-queries), including viewing the call stack in the Temporal Cloud UI, which results in a Query behind the scenes. `__temporal_workflow_metadata` is a built-in query used to retrieve workflow metadata at runtime and is excluded. | Usage Name | Metric Name | History Event Type | | -------------- | ---------------- | ------------------ | | Query Workflow | `query_workflow` | N/A | ## Update - **Update received by Worker**. - Occurs for every successful [Update](/sending-messages#sending-updates) and every [rejected](/handling-messages#update-validators) Update. - This includes [Update-With-Start](/sending-messages#update-with-start), and is in addition to the start Action in the case when the Workflow starts as well. - De-duplicated Updates that share an Update ID do _not_ count as an Action. | Name | Action ID | History Event Type | | ---------------------- | ------------------------ | --------------------------------------------------------------------------------------------- | | Accept Workflow Update | `accept_workflow_update` | `EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_ACCEPTED` | | Reject Workflow Update | `reject_workflow_update` | `EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_REJECTED` Note: this event is never written to history. | ## Schedule [Schedules](/schedule) allows you to schedule a Workflow to start at a particular time. Each execution of a Schedule accrues three actions: - **Schedule Start**. This accounts for two actions. - **Workflow started**. This is a single action to start the target Workflow. It includes initial Search Attributes as part of the start request. | Usage Name | Metric Name | History Event Type | | ----------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Create Schedule | `create_schedule` | N/A | | Update Schedule | `update_schedule` | N/A | | Patch Schedule | `patch_schedule` | N/A | | Delete Schedule | `delete_schedule` | N/A | | Schedule Workflow | `schedule_workflow` | `EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` Note: `event.WorkflowExecutionStartedEventAttributes.indexedFields.TemporalScheduledById` tells if a Workflow was started by a Schedule. | ## Temporal Nexus - For [Nexus Operation scheduled](/references/events#nexusoperationscheduled), the caller Workflow starting a Nexus Operation results in one Action on the caller Namespace. - For [Nexus Operation canceled](/references/events#nexusoperationcanceled), the caller Workflow canceling a Nexus Operation results in one Action on the caller Namespace. - The underlying Temporal primitives (such as Workflows, Activities, and Signals) created by a Nexus Operation handler (directly or indirectly) result in the normal Actions for those primitives billed to the handler’s Namespace. This includes retries for underlying Temporal primitives like Activities but _not_ for handling the Nexus Operation itself or a retry of the Nexus Operation itself. | Usage Name | Metric Name | History Event Type | | ------------------------------------ | -------------------------------------- | --------------------------------------------- | | Schedule Nexus Operation | `schedule_nexus_operation` | `EVENT_TYPE_NEXUS_OPERATION_SCHEDULED` | | Request Nexus Operation Cancellation | `request_nexus_operation_cancellation` | `EVENT_TYPE_NEXUS_OPERATION_CANCEL_REQUESTED` | ## Export [Workflow History Export](/cloud/export) enables you to export closed Workflow Histories to a cloud storage sink of your choice. - **Workflow exported**. Each Workflow exported accrues a single action. - Excluded from APS calculations. | Usage Name | Metric Name | History Event Type | | ----------------------- | ----------- | ------------------ | | Export Workflow History | N/A | N/A | ## Fairness For each hour a Namespace has the [Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) feature enabled, an additional `0.1` Action is charged per Action in the Namespace. - Excluded from APS calculations. | Usage Name | Metric Name | History Event Type | | --------------- | ----------- | ------------------ | | Enable Fairness | N/A | N/A | ## Capacity - For Namespace Capacity Temporal Resource Units (TRUs), Actions are generated up to the included hourly allocation for TRUs in any hour where TRUs are set and actual usage falls beneath the included hourly Action allocation. - Excluded from APS calculations. | Usage Name | Metric Name | History Event Type | | --------------------------- | ----------- | ------------------ | | Enable Provisioned Capacity | N/A | N/A | Actions usage can be tracked in multiple areas depending on the needed granularity. Refer to the [Billing and Usage](/cloud/billing-and-usage) documentation for more information. --- # Usage dashboards Source: https://docs.temporal.io/cloud/actions-usage ## Usage Actions usage is tracked across an account in the [usage dashboard](https://cloud.temporal.io/usage) and is visible to Account Owners, Finance Admin and Global Admin. For individual Namespaces, usage can be seen in the [Namespace summary](https://cloud.temporal.io/namespaces/) for a specific Namespace. ![Temporal Cloud Usage dashboard](/img/cloud/billing/usage-dashboard.png) ## Actions in Workflows When viewing a Event history, events that represent a Billable Action are annotated with the number consumed by the event in the **Billable Actions** Column. These Actions are summarized at the top of the workflow. ![Temporal Cloud Usage dashboard showing aggregated Billable Actions](/img/cloud/billing/aggregate-billable-actions.png) ![Temporal Cloud Usage dashboard showing individual Billable Actions associated with events](/img/cloud/billing/individual-billable-actions.png) This Billable Action estimate is useful for projecting the cost of Workflows. For example, if you ran a test Workflow that generated 20 Billable Actions and projected that it would be run 100 times a day for a month, you could anticipate that Workflow to generate 20 Actions x 100 runs/day x 30 days = 60,000 Billable Actions per month. You can also use the Billable Action estimate to help optimize Workflows by better understanding your cost drivers. > **💡 Tip:** > Excluded Billable Actions > > The Billable Action estimate is an experimental feature and only measures Billable Actions that exist within Workflow > event histories. Some billable concepts are not included in these calculations such as: > > - Query > - Activity Heartbeats > - Rejected Update Workflow Executions > - Export > - Schedule > - Replicated Actions that occur in a > [Namespace replication](../../cloud/high-availability/index.mdx#high-availability-features) > > Additionally, Workflows with the `TemporalNamespaceDivision` Search Attribute set may not have accurate Billable Action > Estimates. The estimated Billable Actions should only be treated as an estimate. If billable events exist outside of > event history, the Actions count could be higher. > [Reach out](https://pages.temporal.io/contact-us) to our team for more information or to help size your number of Actions. --- # Manage API keys Source: https://docs.temporal.io/cloud/api-keys Temporal Cloud API keys offer industry-standard identity-based authentication for Temporal users and [Service Accounts](/cloud/manage-access/service-accounts). This document introduces Temporal Cloud's API key features: - [API key overview](#overview) - [API key best practices](#best-practices) - [Global Administrator and Account Owner API key management](#manage-api-keys) - [User API key management](#user-api-keys) - [Manage API keys for Service Accounts](#serviceaccount-api-keys) - [API keys for Namespace authentication](#namespace-authentication) - [Use API keys to authenticate](#using-apikeys) - [Troubleshoot your API key use](#troubleshooting) - [API keys: Frequently Asked Questions](#faqs) ## API key overview Each Temporal Cloud API key is a unique identity linked to role-based access control (RBAC) settings to ensure secure and appropriate access. The authentication process follows this pathway: ![API key (authentication) → Identity (user or Service Account) → RBAC (authorization)](/img/cloud/apikeys/apikeyrbac.png) ## API key best practices - **Keep it secret; keep it safe**: Treat your API key like a password. Do not expose it in client-side code, public repositories, or other easily accessible locations. - **Rotate keys regularly**: Change your API keys periodically to reduce risks from potential leaks.[Email notifications](/cloud/notifications#admin-notifications) are sent at 30, 20, and 10 days before an API key expires. - **Design your code for key updates**: Use key management practices that retrieve your API keys without hard-coding them into your apps. This lets you restart your Workers to refresh your rotated keys without recompiling your code. - **Monitor API key usage**: Check usage metrics and logs regularly. Revoke the key immediately if you detect any unexpected or unauthorized activity. - **Use a Key Management System (KMS)**: Employ a Key Management System to minimize the risk of key leaks. For guidance on which identities should own API keys, when to use Namespace-scoped Service Accounts, and how to align API keys with your Namespace topology, see [Managing Temporal Cloud access control](/best-practices/cloud-access-control). ### API key use cases API keys are used for the following scenarios: - _**Cloud operations automation**_: API keys work with Temporal Cloud operational tools, including the [Temporal Cloud CLI extension](/cli/cloud), [`tcld`](/cloud/tcld), [Cloud Ops APIs](/ops), and [the Terraform provider](/cloud/terraform-provider). Use them to manage your Temporal Cloud account, Namespaces, certificates, and user identities. - _**Namespace authentication**_: API keys serve as an authentication mechanism for executing and managing Workflows via the SDK and Temporal CLI, offering an alternative to mTLS-based authentication. ### API key supported tooling Use API keys to authenticate with: - [The Temporal CLI](/cli), including the [Temporal Cloud extension](/cli/cloud) - [Temporal SDKs](/develop) - [`tcld`](/cloud/tcld/index.mdx) - [The Cloud Operations API](/cloud/operation-api.mdx) - [Temporalʼs Terraform provider](/cloud/terraform-provider) ### API key permissions API keys support both users and Service Accounts. Here are the differences in their permissions: - Any user can create, delete, and update their _own_ API key using the Cloud UI or the CLI. - Only Global Administrators and Account Owners can create, delete, and update access to API keys for all types of Service Accounts. - Namespace Admins can create, delete, and update access to API keys for the Namespace-scoped Service Accounts they administer. ### API key prerequisites Check these setup details before using API keys: - The Global Administrator or Account Owner may need to [enable API keys access](#manage-api-keys) for your Temporal Account. - Have access to the [Temporal Cloud UI](https://cloud.temporal.io/), the [Temporal Cloud CLI extension](/cli/cloud), or [`tcld`](/cloud/tcld/) to create an API key. ## Global administrator and account owner API key management Global Administrators and Account Owners can monitor, manage, disable, and delete API keys for any user or Service Account within their account. To manage your account’s API keys: 1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. 1. Go to [Settings → API Keys](https://cloud.temporal.io/settings/api-keys) Administrators can disable the creation of new API keys using the **Disable Create API Keys** button on the **API Keys** Settings page. Existing API keys can still be used to authenticate into Temporal Cloud normally until they are either disabled, deleted, or expired. Temporal Cloud automatically sends email notifications to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. For the full recipient list, see [Notifications](/cloud/notifications#admin-notifications). To disable or delete an individual API key use the vertical ellipsis menu in the API key table row. To find an API key, you can filter by API key state and identity type (Global Administrators and Account Owners only). > **⚠️ Caution:** > DISABLED API KEYS > > Deleting or disabling a key removes its ability to authenticate into Temporal Cloud. If you delete or disable an API key > being used by Workers to run a Workflow, those Workers will be unable to connect to Temporal until a new API key secret > is created and configured. > ## User API key management Manage your personal API keys with the Temporal Cloud UI or the CLI. These sections show you how to generate, manage, and remove API keys for a user. ### Generate an API key Create API keys using one of the following methods: > **⚠️ Caution:** > > - Once generated, copy and securely save the API key. It will be displayed only once for security purposes. > #### Generate API keys with the Temporal Cloud UI [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI and navigate to your [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). Then select **Create API key** and provide the following information: - **API key name**: A short, identifiable name for the key - **API key description**: A longer description of the key's use - **Expiration date**: The end date for the API key Finish by selecting **Generate API Key**. #### Generate API keys with the CLI To generate an API key, log into your account and issue the following command: **Temporal CLI** ```command temporal cloud login temporal cloud apikey create-for-me \ --display-name \ --description "" \ --expiry-duration ``` **tcld** ```command tcld login tcld apikey create \ --name \ --description "" \ --duration ``` Duration specifies the time until the API key expires, for example: "30d", "4d12h", etc. ### Enable or Disable an API key You can enable or disable API keys. When disabled, an API key cannot authenticate with Temporal Cloud. #### Manage API key state with the Temporal Cloud UI Follow these steps: 1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. 1. Go to your [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). 1. Select the vertical ellipsis menu in the API key table row. 1. Choose **Enable** or **Disable**. #### Manage API key state with the CLI To manage an API key, log into your account and use one of the following commands to enable or disable it: **Temporal CLI** ```command temporal cloud login temporal cloud apikey disable --key-id temporal cloud apikey enable --key-id ``` **tcld** ```command tcld login tcld apikey disable --id tcld apikey enable --id ``` ### Delete an API key Deleting an API key stops it from authenticating with Temporal Cloud. > **⚠️ Caution:** > > Deleting an API key used by Workers to run a Workflow will cause it to fail unless you rotate the key with a new one. > This can affect long-running Workflows that outlast the API key's lifetime. > #### Delete API keys with the Temporal Cloud UI Follow these steps to remove API keys: 1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. 1. Navigate to your [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). 1. Select the vertical ellipsis menu in the API key table row. 1. Choose **Delete**. #### Delete API keys with the CLI To delete an API key, log into your account and issue the following: **Temporal CLI** ```command temporal cloud login temporal cloud apikey delete --key-id ``` **tcld** ```command tcld login tcld apikey delete --id ``` ### Rotate an API key Temporal API keys automatically expire based on the specified expiration time. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. Follow these steps to rotate API keys: 1. Create a new key. You may reuse key names if that helps. 1. Ensure that both the original key and new key function properly before moving to the next step. 1. Switch clients to load the new key and start using it. 1. Delete the old key after it is no longer in use. For a broader machine-identity rotation strategy across API keys and Service Accounts, see [Managing Temporal Cloud access control](/best-practices/cloud-access-control). ## Manage API keys for Service Accounts Global Administrators and Account Owners can manage and generate API keys for _all_ Service Accounts in their account. Namespace Admins can manage and generate API keys for the Namespace-scoped Service Accounts they administer. This is different for non-admin users, who manage and generate their own API keys. ### Generate an API Key for a Service Account Create API keys for Service Accounts using one of the following methods: > **⚠️ Caution:** > > - Once generated, copy and securely save the API key. It will be displayed only once for security purposes. > #### Generate API Keys with the Temporal Cloud UI [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. Global Administrators or Account Owners can go to [Settings → API Keys](https://cloud.temporal.io/settings/api-keys). Namespace Admins can go to [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). Select **Create API Key**, then choose **Service Account** from the "Create an API key for" dropdown. In the "Mapped to identity" input box, select a Service Account and provide the following information: - **API key name**: A short, identifiable name for the key - **API key description**: A longer description of the key's use - **Expiration date**: The end date for the API key Finish by selecting **Generate API Key**. #### Generate API keys with the CLI Create an API key for a Service Account by passing the Service Account ID: **Temporal CLI** ```command temporal cloud apikey create-for-service-account \ --display-name \ --description "" \ --expiry-duration \ --service-account-id ``` **tcld** ```command tcld apikey create \ --name \ --description "" \ --duration \ --service-account-id ``` ### Enable or Disable an API key Global Administrators and Account Owners can manage API key access for any user in their account using the Temporal Cloud UI or the CLI. #### Manage keys with Temporal Cloud UI Follow these steps: 1. [Log in](https://cloud.temporal.io/) to the Temporal Cloud UI. 1. Global Administrators or Account Owners can go to [Settings → API Keys](https://cloud.temporal.io/settings/api-keys). Namespace Admins can go to [Profile Page → API Keys](https://cloud.temporal.io/profile/api-keys). 1. Find the API key. Use the vertical ellipsis menu in the table row and select the Disable/Enable option to perform the action. There may be a delay after changing the status. Once successful, the updated API key status will be shown in the row. #### Manage keys with the CLI Use the disable or enable command to change the state of an API key: **Temporal CLI** ```command temporal cloud login temporal cloud apikey disable --key-id temporal cloud apikey enable --key-id ``` **tcld** ```command tcld login tcld apikey disable --id tcld apikey enable --id ``` This command is the same for users and Service Accounts. ### Delete an API key for a Service Account Global Administrators and Account Owners can delete API keys for any user or Service Account in their account using the Temporal Cloud UI or the CLI. Deleting a key removes its ability to authenticate with Temporal Cloud. If you delete an API key used by a Worker to run a Workflow, that Worker will fail to connect to Temporal server unless you rotate the API key with a new one. #### Delete a Service Account API key with Temporal Cloud UI Follow these steps: 1. Go to [Settings → API Keys](https://cloud.temporal.io/settings/api-keys). 1. Locate the API key. Use the vertical ellipsis menu in the table row and select the Delete option. There may be a delay after deleting the API key. 1. Once successful, the updated API key status will be reflected in the row. #### Delete a Service Account API key with the CLI Use the delete command to remove an API key. The process is the same for a user or Service Account. **Temporal CLI** ```command temporal cloud login temporal cloud apikey delete --key-id ``` **tcld** ```command tcld login tcld apikey delete --id ``` ### Rotate a Service Account API key Temporal API keys automatically expire based on the specified expiration time. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. Follow these steps to rotate API keys: 1. Create a new key. You may reuse key names if that helps. 1. Ensure that both the original key and new key function properly before moving to the next step. 1. Switch clients to load the new key and start using it. 1. Delete the old key after it is no longer in use. > **💡 Tip:** > > Service Accounts can rotate their own API keys irrespective of their configured permissions. To use this feature, have > your Service Account create a new API key using the [Cloud Ops APIs](/ops) or the CLI before the current > one expires. Service Accounts cannot delete their own API keys without the requisite permissions, which helps keep > Workflow access secure. > ## API keys for Namespace authentication Create a Namespace with API key authentication as an alternative to mTLS-based authentication by selecting "Allow API key authentication" during setup. Use the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This is the recommended endpoint for all Namespaces. For Namespaces with [High Availability](/cloud/high-availability), the Namespace endpoint automatically directs traffic to the active region, so Workers and Clients don't need to change endpoints during a failover. See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. ## Use API keys to authenticate Authenticate with Temporal Cloud using API keys with the following clients: - [Temporal CLI](/cli), including the [Temporal Cloud extension](/cli/cloud) - [SDKs](/develop) - [Temporal Cloud CLI `tcld`](/cloud/tcld/index.mdx) - [The Cloud Operations API](/cloud/operation-api.mdx) - [Temporal’s Terraform Provider](/cloud/terraform-provider) ### Temporal CLI To use your API key with the Temporal CLI, either pass it with the `--api-key` flag or set an environment variable in your shell (recommended). The CLI automatically picks up the `TEMPORAL_API_KEY` environment variable from your shell. In addition to the API key, the following client options are required: - `--address`: Provide `..tmprl.cloud:7233` using your Namespace's info. - This can be copied from the Namespace UI's "Connect" box. - You can set the address using an environment variable. - `--namespace`: Provide `.` using your Namespace's info. - This can be copied from the top of the Namespace UI. - This can be set using an environment variable. For example, to connect to Temporal Cloud from the CLI using an environment variable for the API key: ```bash export TEMPORAL_API_KEY= temporal workflow list \ --address ..tmprl.cloud:7233 \ --namespace . ``` > **💡 Tip:** > ENVIRONMENT VARIABLES > > Do not confuse environment variables, set with your shell, with temporal env options. > ### SDKs To use your API key with a Temporal SDK, see the instructions in each SDK section. [How to connect to Temporal Cloud using an API Key with the Go SDK](/develop/go/client/temporal-client#connect-to-temporal-cloud) [How to connect to Temporal Cloud using an API Key with the Java SDK](/develop/java/client/temporal-client#connect-to-temporal-cloud) [How to connect to Temporal Cloud using an API Key with the Python SDK](/develop/python/client/temporal-client#connect-to-temporal-cloud) [How to connect to Temporal Cloud using an API Key with the TypeScript SDK](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) [How to connect to Temporal Cloud using an API Key with the .NET SDK](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) ### Temporal Cloud CLI To use an API key with the `temporal cloud` extension or `tcld`, choose one of these methods: - Use the `--api-key` flag. - Set the `TEMPORAL_API_KEY` environment variable in your shell. > **💡 Tip:** > ENVIRONMENT VARIABLES > > Do not confuse environment variables, set with your shell, with temporal env options. > ### Cloud Ops API To use an API key with the [Cloud Ops API](/ops), securely pass the API key in your API client. For a complete example, see [Cloud Samples in Go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go). ### Terraform provider To use an API key with the [Temporal Terraform Provider](/cloud/terraform-provider), pass the API key as a provider argument. ## Troubleshoot your API key use **Invalid API key errors**: Check that you copied the key correctly and that it hasn't been revoked or expired. ## API keys: Frequently asked questions **Q: Can I issue and use multiple API keys for the same account?** A: Yes, you can generate multiple API keys for different services or team members. **Q: How many API keys can be issued at once?** A: Up to 10 non-expired keys per user and 20 non-expired keys per Service Account. **Q: Do API keys expire?** A: Yes, API keys expire based on the specified expiration date. Temporal recommends rotating API keys periodically. [Email notifications](/cloud/notifications#admin-notifications) will be sent from Temporal Cloud to Global Administrators, Account Owners, and the key owner at **30, 20, and 10 days before** an API key expires. **Q: What's the maximum allowed expiration for an API key?** A: The maximum expiration time for an API key is 2 years. **Q: What happens if I misplace or lose my API bearer token/secret key?** A: The full key is displayed only once upon creation for security reasons. If you lose it, generate a new one. **Q: What is the `Generate API Key` button on the Namespace page?** A: The `Generate API Key` button on a Namespace page generates an API key with `Admin` permissions for the given Namespace and the maximum expiration time, which is 2 years. For additional details, refer to [Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped). --- # Audit Logs Source: https://docs.temporal.io/cloud/audit-logs > Audit Logs in Temporal Cloud provide forensic information, integrating with a data streaming service for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. Audit Logs is a feature of [Temporal Cloud](/cloud/overview) that provides forensic access information for a variety of operations in the Temporal Cloud Control Plane. Audit Logs answers "who, when, and what" questions about Temporal Cloud resources. These answers can help you evaluate the security of your organization, and they can provide information that you need to satisfy audit and compliance requirements. You need the Account Owner or Global Administrator role to view Audit Logs via UI, use the API, or to configure an Audit Log Integration with [AWS Kinesis](/cloud/audit-logs-aws) or [GCP Pub/Sub](/cloud/audit-logs-gcp). > **ℹ️ Info:** > > Audit Logs do NOT capture data plane events, like Workflow Start, Workflow Terminate, Schedule Create, etc. > Instead, explore the [Export](/cloud/export) feature, which does let you send closed Workflow Histories to external storage. > ## Which events are supported by Audit Logs? The `operation` field can contain the following event values: - Account - `ChangeAccountPlanType`: Change Account Plan Type - `CreateAccountAuditLogSink`: Create Account Audit Log Sink - `DeleteAccountAuditLogSink`: Delete Account Audit Log Sink - `OffboardAccount`: Offboard Account - `SelfServiceOffboardAccount`: Self-Service Offboard Account - `UpdateAccount`: Update Account - `UpdateAccountAuditLogSink`: Update Account Audit Log Sink - API keys - `CreateAPIKey`: Create API Key - `CreateApiKey`: Create API Key - `CreateServiceAccountAPIKey`: Create Service Account API Key - `DeleteAPIKey`: Delete API Key - `DeleteApiKey`: Delete API Key - `UpdateAPIKey`: Update API Key - `UpdateApiKey`: Update API Key - Billing - `CreateBillingReport`: Create Billing Report - Connectivity rules - `CreateConnectivityRule`: Create Connectivity Rule - `DeleteConnectivityRule`: Delete Connectivity Rule - Custom roles - `CreateCustomRole`: Create Custom Role - `DeleteCustomRole`: Delete Custom Role - `UpdateCustomRole`: Update Custom Role - Namespace - `CreateNamespace`: Create Namespace - `DeleteNamespace`: Delete Namespace - `FailoverNamespaces`: Failover Namespaces - `RenameCustomSearchAttribute`: Rename Custom Search Attribute - `UpdateNamespace`: Update Namespace - `UpdateNamespaceTags`: Update Namespace Tags - `UpdateSearchAttributes`: Update Search Attributes - Namespace export - `CreateNamespaceExportSink`: Create Namespace Export Sink - `DeleteNamespaceExportSink`: Delete Namespace Export Sink - `UpdateNamespaceExportSink`: Update Namespace Export Sink - `ValidateNamespaceExportSink`: Validate Namespace Export Sink - Nexus endpoints - `CreateNexusEndpoint`: Create Nexus Endpoint - `DeleteNexusEndpoint`: Delete Nexus Endpoint - `UpdateNexusEndpoint`: Update Nexus Endpoint - Projects - `CreateProject`: Create Project - `DeleteProject`: Delete Project - `SetServiceAccountProjectAccess`: Set Service Account Project Access - `SetUserGroupProjectAccess`: Set User Group Project Access - `SetUserProjectAccess`: Set User Project Access - `UpdateProject`: Update Project - Security - `SecretFound`: Detect Exposed API Key - Service accounts - `CreateServiceAccount`: Create Service Account - `DeleteServiceAccount`: Delete Service Account - `SetServiceAccountNamespaceAccess`: Set Service Account Namespace Access - `UpdateServiceAccount`: Update Service Account - Users - `CreateJITUser`: Create JIT User - `CreateUser`: Create User - `DeleteUser`: Delete User - `InviteUsers`: Invite Users - `SetUserNamespaceAccess`: Set User Namespace Access - `UpdateIdentityNamespacePermissions`: Update Identity Namespace Permissions - `UpdateUser`: Update User - `UpdateUserNamespacePermissions`: Update User Namespace Permissions - `UserAcceptInvitation`: User Accept Invitation - `UserLogin`: User Login - `UserSignup`: User Signup - User groups - `AddUserGroupMember`: Add User Group Member - `CreateUserGroup`: Create User Group - `DeleteUserGroup`: Delete User Group - `RemoveUserGroupMember`: Remove User Group Member - `SetUserGroupNamespaceAccess`: Set User Group Namespace Access - `UpdateUserGroup`: Update User Group ### Audit log format > **ℹ️ Info:** > DEPRECATION NOTICE > > The `request_id` field is deprecated and is planned for removal on or after November 1 2026. Use `async_operation_id` instead. > Audit Logs use the following JSON format: ```json { "operation": // Operation that was performed "principal": // Information about who initiated the operation "raw_details": // Details about the request "x_forwarded_for": // The IP address(es) making the call "emit_time": // Time the operation was recorded "log_id": // Unique ID of the log entry "async_operation_id": // Request or async operation identifier for correlation "request_id": // DEPRECATED, use async_operation_id "status": // Status, such as OK or ERROR "version": // Version of the log entry } ``` #### How to interpret the status For supported Control Plane operations, the `status` field describes the outcome of the Temporal Cloud API call: - `OK`: The API call returned successfully. - `ERROR`: The API call returned an error. This includes authorization failures, such as when the authenticated principal does not have permission to perform the operation. Audit logs do not capture authentication failures. For an asynchronous operation, `OK` means that Temporal Cloud accepted the API call. The operation can fail after the audit log event is emitted, and the event is not updated with the final outcome. Use [`GetAsyncOperation`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/operations/GET/cloud/operations/%7BasyncOperationId%7D) to check the operation state and failure reason. The `async_operation_id` field can contain a request identifier or an async operation identifier. Its presence does not indicate that the API call started an asynchronous operation. > **📝 Note:** > > The [`X-Forwarded-For`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) format is a comma-separated list of IP addresses which should be evaluated from the last to the first, until meeting the first untrusted IP address of the list. This allows for instance to consider proxies in the path. > > Temporal provides the caller IP address in that format to allow customers to identify a caller IP address even if one (or more proxies) are in the network path to reach Temporal Cloud. > ### Example of an audit log ```json [ { "operation": "UserLogin", "status": "OK", "version": 2, "logId": "edb3aa3e-78c4-48fc-9c7e-2078c6989775", "xForwardedFor": "10.1.2.3", "asyncOperationId": "", "emitTime": { "$typeName": "google.protobuf.Timestamp", "seconds": 1759436617, "nanos": 48000000 }, "principal": { "type": "user", "id": "", "name": "user@email.com", "apiKeyId": "" } }, { "operation": "UserLogin", "status": "OK", "version": 2, "logId": "5fe6a81e-8d3c-4f4d-88a5-52db864c9ea5", "xForwardedFor": "10.1.2.3", "asyncOperationId": "", "emitTime": { "seconds": 1759178573, "nanos": 671000000 }, "principal": { "type": "user", "id": "", "name": "user@email.com", "apiKeyId": "" } } ] ``` ## How to configure an audit log integration Audit Logs can be configured in AWS Kinesis or GCP Pub/Sub. - [AWS Kinesis Instructions](/cloud/audit-logs-aws) - [GCP Pub/Sub Instructions](/cloud/audit-logs-gcp) ## How to troubleshoot an audit log sink The Audit Logs page of the Temporal Cloud UI provides the current status of an Audit Log Integration. - If an error is detected, a summary of the error appears below the page title. - If the Audit Log Integration is functioning normally, an **On** badge appears next to the page heading. After an Admin Operation is performed, users can see Audit Log messages flow through the stream. Upon successful configuration of the Audit Log sink and set up of a stream, you will receive events within the hour of setup. Temporal is able to retain Audit Log information for up to 30 days. To retrieve logs up to the past 30 days, you will need to file a request. If you experience an issue with an Audit Log sink, we can provide the missing audit information. Open a support ticket to request assistance. ## How to delete an audit log sink To delete an Audit Log sink, follow these steps: 1. In the Temporal Cloud UI, select **Settings**. 1. On the **Settings** page, select **Audit Logs**. 1. In the **Audit Logs Integration** card, select **Edit**. 1. At the bottom of the **Audit Logs Integration** page, choose **Delete**. After you confirm the deletion, the Audit Log Sink is removed from your account and logs stop flowing to your stream. ## View an audit log An Audit Log can be viewed in the Temporal Cloud UI. 1. In the Temporal Cloud UI, select **Settings**. 1. On the **Settings** page, select **Audit Logs**. Up to 1,000 events can be downloaded from the Audit Log UI to a local file. ## Access an audit log via API An Audit Log can be accessed using the [Temporal Cloud Ops API](/ops). Use the API to access an Audit Log if you wish to make dashboards for viewing an Audit Log outside of Temporal Cloud. If your goal is to export an Audit Log, it is better to use an Audit Log sink and capture each entry as it is generated. Audit Logs are accessible for the past 30 days using the API. The API allows: - StartTimeInclusive: Filter for UTC time >= (defaults to 30 days ago) - optional - EndTimeExclusive: Filter for UTC time < (defaults to current time) - optional - PageSize: Cannot exceed 1000. Defaults to 100. - optional - PageToken: The page token if this is continuing from another response - optional --- # Audit Logs - AWS Kinesis Source: https://docs.temporal.io/cloud/audit-logs-aws > Audit Logs in Temporal Cloud provides forensic information, integrating with AWS Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. ## Configure Audit Logs using AWS Kinesis To set up Audit Logs, you must have an Amazon Web Services (AWS) account and set up Kinesis Data Streams. 1. If you don't have an AWS account, follow the instructions from AWS in [Create and activate an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). 2. To set up Kinesis Data Streams, open the [AWS Management Console](https://aws.amazon.com/console/), search for Kinesis, and start the setup process. You can use [this AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) to create an IAM role with access to a Kinesis stream you have in your account. Be aware that Kinesis has a rate limit of 1,000 messages per second and quotas for both the number of records written and the size of the records. For more information, see [Why is my Kinesis data stream throttling?](https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-data-stream-throttling/) ### Create an Audit Log sink 1. In the Temporal Cloud UI, select **Settings**. 1. On the **Settings** page, select **Audit Logs**. 1. In the **Audit Log Integration** card, select **Setup**. 1. On the **Audit Log Integration** page, choose your **Access method** (either **Auto** or **Manual**). - **Auto:** Configure the AWS CloudFormation stack in your AWS account from the Cloud UI. - **Manual:** Use a generated AWS CloudFormation template to set up Kinesis manually. 1. In **Kinesis ARN**, paste the Kinesis ARN from your AWS account. 1. In **Role name**, provide a name for a new IAM Role. 1. In **Select an AWS region**, select the appropriate region for your Kinesis stream. If you chose the **Auto** access method, continue with the following steps: 1. Select **Save and launch stack**. 1. In **Stack name** in the AWS CloudFormation console, specify a name for the stack. 1. In the lower-right corner of the page, select **Create stack**. If you chose the **Manual** access method, continue with the following steps: 1. Select **Save and download template**. 1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/). 1. Select **Create Stack**. 1. On the **Create stack** page, select **Template is ready** and **Update a template file**. 1. Select **Choose file** and specify the template you generated in step 1. 1. Select **Next** on this page and on the next two pages. 1. On the **Review** page, select **Create stack**. To ensure that Audit Logs can flow into the Kinesis stream, you can use the **Verify** button to confirm it is set up correctly. This validates that Temporal can successfully write to your stream. If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the kinesis stream. ## Consume an Audit Log **How to consume an Audit Log** After you create an Audit Log sink, wait for the logs to flow into the Kinesis stream. You will see the first logs within 10 minutes after you configure the sink. > **📝 Note:** > > You must configure and implement your own consumer of the Kinesis stream. > For an example, see [Example of consuming an Audit Log](#example-of-consuming-an-audit-log). > ### Example of consuming an Audit Log The following Go code is an example of consuming Audit Logs from a Kinesis stream and delivering them to an S3 bucket. ```go func main() { fmt.Println("print audit log from S3") cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithSharedConfigProfile("your_profile"), ) if err != nil { fmt.Println(err) } s3Client := s3.NewFromConfig(cfg) response, err := s3Client.GetObject( context.Background(), &s3.GetObjectInput{ Bucket: aws.String("your_bucket_name"), Key: aws.String("your_s3_file_path")}) if err != nil { fmt.Println(err) } defer response.Body.Close() content, err := io.ReadAll(response.Body) fmt.Println(string(content)) } ``` The preceding code also prints the logs in the terminal. The following is a sample result. ```json { "emit_time": "2023-11-14T07:56:55Z", "level": "LOG_LEVEL_INFO", "caller_ip_address": "10.1.2.3, 10.4.5.6", "user_email": "user1@example.com", "operation": "DeleteUser", "details": { "target_users": ["d7dca96f-adcc-417d-aafc-e8f5d2ba9fe1"], "search_attribute_update": {} }, "status": "OK", "category": "LOG_CATEGORY_ADMIN", "log_id": "0mc69c0323b871293ce231dd1c7fb639", "request_id": "445297d3-43a7-4793-8a04-1b1dd1999640", "principal": { "id": "988cb80b-d6be-4bb5-9c87-d09f93f58ed3", "type": "user", "name": "user1@example.com" } } ``` --- # Audit Logs - GCP Pub/Sub Source: https://docs.temporal.io/cloud/audit-logs-gcp > Audit Logs in Temporal Cloud provides forensic information, integrating with GCP Pub/Sub for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. ## Manual Setup Prerequisites > **📝 Note:** > > These steps are only required for manual setup. > If you use Terraform for your deployment, you don't need to complete these prerequisites. > Before configuring the manual Audit Log sink, complete the following steps in Google Cloud: 1. Create a Pub/Sub topic and make a note of its topic name, such as `test-auditlog`. 1. Set up a service account in the same project in Google Cloud and follow the instructions in the Temporal Cloud UI to configure the permissions for that service account. ## Create an Audit Log sink 1. In the Temporal Cloud UI, select **Settings**. 1. On the **Settings** page, select **Audit Logs**. 1. In the **Audit Logs Integration** card, select **Setup**. 1. On the **Audit Log Integration** page, select **Pub/Sub**. 1. In the **service account email** field, enter the email of the service account you created in the prerequisites. 1. In the **Topic name** field, enter the topic name of the Pub/Sub topic you created in the prerequisites. 1. There are two ways to configure the service account to write to the Pub/Sub sink: select **Manual** to configure the account manually, or **Deploy with Terraform** to use Terraform. If you use Terraform, you don't need to complete the prerequisite steps above. 1. Follow the instructions in the Temporal Cloud UI for the method you chose. 1. To ensure that audit logs can reach your Pub/Sub topic, you can use the **Verify** button to confirm it is set up correctly. This validates that Temporal can successfully write to your topic. If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the Pub/Sub topic. 1. Click **Create** to configure the audit log. Audit Logs will begin to show up in Pub/Sub within 10 minutes ![Temporal Cloud UI Setup for Audit Logs with GCP Pub/Sub](/img/cloud/gcp/audit-logging-pub-sub-gcp.png) > **ℹ️ Info:** > MORE INFORMATION > > For more details, refer to [Audit Logs with Temporal Cloud](/cloud/audit-logs). > --- # Billing Center Source: https://docs.temporal.io/cloud/billing > As an Account Owner, you can access and manage billing details anytime, with invoices available for download on the Billing page. Typical billing cycles begin on the first of the month (UTC). ## Current balance Your current balance card shows the balance for your current billing cycle and the date it was last updated. This balance adjusts with use and appears on the first line of your Invoices table. > **📝 Note:** > Billing Cycles > > Billing cycles normally begin on the first of the month (UTC). > The minimum plan fee for your first month is prorated based on your sign-up date. > ## Recent bill The "Recent Bill" card displays the previous bill amount. ![Recent bill card showing a balance of $0.00](/img/cloud/billing/billing-card.png) - If you pay your invoices through Stripe, you'll see a **Pay Now** button. It takes you to the Stripe portal to complete your payment - If your account is set up for auto-payment, you don’t need to manually pay bills. However, you can choose to make manual payments whenever you wish ## Invoices To review your invoices, follow these steps: 1. Click **Billing** on your left-side vertical navigation. 2. Under the **Invoices** section, select and download the invoice(s) you want to review. The Invoices table shows the following information: - Date (UTC): The date range covered by the invoice - Type: The type of invoice, such as credit purchase or cloud usage - Status: The current status of the invoice, such as paid or pending - Credit Granted: The total credits added to your account - Credit Purchase Amount: The amount paid for purchasing credits - Credit Usage: The credits used during the billing cycle - Subtotal: The total amount of the invoice before any adjustments - Balance Due: The amount to pay after applying credits ![Billing page showing Invoices tab](/img/cloud/billing/billing-invoices.png) You may download your Invoices prior to this calendar month by clicking the download icon by the date. > **📝 Note:** > Current Month Invoice > > During the current billing period, your invoice will not be finalized and the download option will not be available. > ## Credits The following information appears under the credits table: - Effective At (UTC): The date when the credit grant became effective - Type: Indicates whether the transaction was a deduction, expiry, or grant - Amount: The credit amount that was granted, deducted, or expired - Credits Remaining: The remaining credit available in the account ![Billing page showing Credits tab](/img/cloud/billing/billing-credits.png) ## Cost by Namespace > **💡 Tip:** > Temporal Cloud Billing API in Public Preview > > The [Temporal Cloud Billing API](/cloud/billing-api) allows you to access billing information on a Namespace basis to an hourly granularity, enriched with Tags and Projects. The Billing API will replace the Cost by Namespace UI. > Account Owners and Finance Admins can access a cost column on the Usage page. This allows you to monitor your cost on a per Namespace basis. If your organization separates work by Namespace—for development, production, or different products—you can view costs for each. ![Billing page showing Usage](/img/cloud/billing/billing-usage.png) > **📝 Note:** > Cost Breakdown Limitations > > Namespace cost details are not available for "last 90 days" or "last 120 days". > > Cost breakdowns distribute the total usage cost to namespaces proportionally based on their metered usage. The proration > reflects your effective price, factoring in included Actions/Storage and tiered pricing rates in your Temporal plan. > ## Plans Account Owners and Finance Admins can access their Temporal Plan information on the plans page. For customers on a standard agreement you will be able to: - View current plan information, pricing details and entitlements - View other available plans, pricing details and entitlements - View Pay-as-You-Go pricing rates applicable to your plan - Upgrade and Downgrade between plans available on a standard agreement ![Billing page showing Plans tab](/img/cloud/billing/billing-plans.png) Requests to upgrade your plan are processed immediately and you will be billed on a pro-rated basis for that billing period. Your monthly entitlements will reflect the full volume of included Actions and Storage of the upgrade plan for that billing month. After an upgrade, a downgrade cannot be processed until the following billing period. Requests to downgrade will be processed immediately. Billing and entitlements will be backdated to the beginning of the billing period. ## Account Cancellation The way you created your Temporal account determines how you can cancel your subscription and remove the account. - **For accounts managed by our sales team**.
Please submit a support ticket so we can help you. - **For accounts created through our self-signup portal**.
Account owners can delete their accounts on the Temporal Cloud Billing page, under the **Plan** tab. If you're no longer using Temporal Cloud, use the Delete Account button to begin the process. - Permanently deleted accounts will immediately cease billing and be scheduled for full deletion within 72 hours. - Account Data and Active Storage will be permanently deleted. Retained Storage will be deleted in accordance with its configured retention period. ![Billing page showing the Plan tab. The contents on the tab include "Manage Payment Method" and "Delete Account" buttons. The "Delete Account" button is placed below text asking "No longer using Temporal Cloud?"](/img/cloud/billing/billing-cancel.png) --- # Billing and usage management Source: https://docs.temporal.io/cloud/billing-and-usage > As an Account Owner, you can access and manage billing details anytime, with invoices available for download on the Billing page. Typical billing cycles begin on the first of the month (UTC). Temporal Cloud provides billing and costs information for your account. Use this information to assess your spending patterns, inspect your credit ledger, check your invoice histories, update payment details, and manage your current plan as needed. For more information on current Temporal Cloud pricing for Actions, storage, and services/support, please visit our [Pricing page](/cloud/pricing). Usage on Temporal is measured in Actions and Storage. This can help understand your bills, forecast usage, optimize Workflows, and troubleshoot errors. You can view your Action usage in multiple ways. The following tools are available for measuring Usage and Billing: - **[Billing Center](/cloud/billing):** Allows you to see summary invoices and credits, manage plans, and delete your accounts. - Viewable by Account Owners and Finance Admin - **[Billing API](/cloud/billing-api):** Allows you to access billing information on a Namespace basis down to an hourly granularity, enriched with Tags and Projects. The Billing API provides a FOCUS-guided data format that can be ingested into your cloud cost management platform or analytics tooling. - Viewable by Account Owners and Finance Admin - **[Usage Dashboards](/cloud/actions-usage):** Aggregate Actions on a Namespace level and includes Action categories that groups similar types of Actions as seen in [Actions](/cloud/actions). Available in the Cloud UI in the usage dashboard and Namespace overview pages. - Viewable by Account Owners, Finance Admin, Global Admin on an account level. Namespace level usage is visible on the Namespace pages to those with access. - **[Actions in Event History](/cloud/actions-usage#actions-in-workflows):** Highlights Actions in a given Event History via the Temporal Cloud UI. Note that some Actions are not measured in Workflow histories. - Viewable by Account Owners, Global Admin and Namespace Admin, Developers and Read-only - **[Actions Metrics](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_billable_action_count):** A high cardinality billable action metric that include labels for Category, Action Type, Workflow Type and Namespace down to minute granularity. - Viewable by creating a service account with the "Metrics Read-Only" role. See the [OpenMetrics](/cloud/metrics/openmetrics#api-key-authentication) page for more information. --- # Cloud Billing API Source: https://docs.temporal.io/cloud/billing-api > Generate Namespace-level cost reports in FOCUS-friendly CSV, with hourly, daily, and monthly granularity for FinOps tools. The Temporal Cloud Billing API provides Namespace-level cost attribution through on-demand billing reports. Reports are delivered in CSV format and can be accessed via API or downloaded directly for use in FinOps tooling and cost management platforms. This API is part of the [Cloud Operations API](/ops). The Billing API allows you to: - Generate billing reports for specified invoice months - Retrieve report status and metadata - Download CSV reports that can be fed into internal analytics tooling or cloud cost management platforms The Billing Report contains: - Accurate Namespace-level cost attribution - Hourly, daily, and monthly granularities - A [FOCUS](https://focus.finops.org/)-friendly data format For complete request and response schemas, refer to the Schema below. Billing report generation is **asynchronous**. You initiate report creation, then poll for completion. ## Report data limitations Reports can be generated with hourly, daily and monthly granularities, each of which have their own data ranges. - Hourly: Current billing month and previous billing month - Daily: Current billing month and previous two billing months - Monthly: Current billing month and previous eleven billing months Data from the current billing month is subject to change and only finalized once the billing month is closed. ## Allowed date ranges Date ranges must use billing-month boundaries (MM/YYYY). Requests may include the current billing month. The data in finalized reports includes usage up to `current_time` \- 24 hours (rounded down to the granularity level). ## Rate limits and concurrency Rate limits apply to API usage. ### Per-account concurrency Within a single account: - Only one billing report per account is generated at a time - Additional requests are accepted but queued ### Report Generation Latency Report generation time varies and is not guaranteed. Factors that affect it include the size of the requested date range and overall platform load. ## Best practices Provide an idempotency key (`async_operation_id`) when retrying requests. Poll `GetBillingReport` using exponential backoff. Download reports immediately after generation (URLs expire). Avoid frequent generation of large overlapping ranges in the current billing period. ## Billing report schema Billing reports are delivered in CSV format. Each row represents a charge record. | Column Name | Description | Example | | ----- | ----- | ----- | | BillingAccountID | Temporal Cloud account ID | a2dd6 | | BillingAccountName | Temporal Cloud account name | temporal | | BillingCurrency | The currency an account is billed in | USD (cents) | | BillingPeriodEnd | The exclusive end bound of a billing period | 2024-02-01T00:00:00Z | | BillingPeriodStart | The inclusive start bound of a billing period | 2024-01-01T00:00:00Z | | ChargeCategory | The highest level classification of a charge based on how it is billed | Usage | | ChargeDescription | A self contained summary of the charge’s purpose | Actions \- Tier 1 | | ChargeFrequency | Indicates how often a charge will occur | Usage-Based | | ChargePeriodEnd | Time period end from when this charge took place, correlates to data granularity | 2025-10-01T01:00:00.000Z | | ChargePeriodStart | Time period start from when this charge took place, correlates to data granularity | 2025-10-01T00:00:00.000Z | | ContractedCost | Cost calculated by multiplying ContractedUnitPrice and PricingQuantity | 100.00 | | ContractedUnitPrice | The agreed-upon unit price for a single pricing unit of the associated SKU. Inclusive of negotiated discounts | 10.00 | | InvoiceID | The ID of the invoice for this billing period | in\_XXXXXXXXXXXXXXXXXXXX | | InvoiceIssuer | The entity responsible for issuing payable invoices | stripe | | PricingQuantity | The volume of a given SKU used or purchased | 10.00 | | PricingUnit | The measurement unit used for PricingQuantity | 1 Million Actions | | Provider | The provider of purchased resources or services | Temporal Technologies | | Publisher | The publisher of purchased resources or services | Temporal Technologies | | ResourceID | Namespace name \+ Temporal Cloud account ID | production.a2dd6 | | ResourceName | Namespace name \+ Temporal Cloud account ID | production.a2dd6 | | ResourceType | The type of resource the charge applies to | Namespace | | ServiceCategory | The highest level classification of a service based on the core function of the service | Temporal Cloud | | ServiceName | An offering that can be purchased from a provider | Temporal Cloud | | ServiceSubcategory | A secondary classification of the service category for a service based on its core function | Actions | | SKUID | A unique identifier that represents a specific SKU | essentials-actions | | SKUMeter | The functionality being metered or measured by a particular SKU in a charge | Actions | | Tags | Provider and customer defined tags associated with resources | `{"$tmprl_project":["project-id"],"namespace-tag-key":["namespace-tag-value"]}` | ## Generate a report To generate a report, follow these steps: 1. Create a billing report using `CreateBillingReport`. The response includes a `billing_report_id` and `async_operation_id`. 1. Poll `GetBillingReport` using the `billing_report_id` 1. When the report state becomes `BILLING_REPORT_STATE_GENERATED`, retrieve the download URL 1. Download the report before the URL expires ### Key identifiers | Identifier | Purpose | | ----- | ----- | | `billing_report_id` | Identifies the billing report and is used to retrieve metadata and download URLs | | `async_operation_id` | Identifies the background operation responsible for generating the report | The async operation follows the standard Cloud Operations async model (see [Async Operations](/ops#per-identity-rate-limits)). --- # Capacity modes Source: https://docs.temporal.io/cloud/capacity-modes > Control how limits are assigned to a Namespace with Capacity Modes Each Namespace in Temporal has a rate limit, which is measured in [Actions](/cloud/pricing#action) per second. Temporal offers two different modes for adjusting capacity: On-Demand Capacity or Provisioned Capacity. With On-Demand Capacity, Namespace capacity is increased automatically along with usage. With Provisioned Capacity, you can control your capacity limits by requesting Temporal Resource Units (TRUs). ## Namespace Capacity Namespaces in Temporal can be set to either an **On-Demand** or **Provisioned Capacity** Mode. These modes govern how limits are assigned to a Namespace. Actions Per Second (APS) is the primary limit for Namespaces and is based on the operating billable Actions that occur each second. Some Actions can result in multiple back-end operations, so limits are also set on Requests Per Second (RPS) and Operations Per Second (OPS) to maintain reliability. See [Service-level RPS limits](/references/dynamic-configuration#service-level-rps-limits) for more about RPS. See the [operations list](/references/operation-list) for the list of operations. See the [Actions page](/cloud/actions) for the list of actions. > **💡 Tip:** > Measuring throughput with APS, RPS, and OPS > > APS, RPS, and OPS are all measures of throughput that apply to different aspects of Temporal. > > APS, or Actions Per Second, is specific to Temporal Cloud. > It measures the rate at which Actions, like starting or signaling a Workflow, can be performed in a specific Namespace. > Temporal Cloud uses APS to protect the system from sudden major spikes in load. > > RPS, or Requests Per Second, is used in the Temporal Service, both in self-hosted Temporal and Temporal Cloud. > It measures and controls the rate of gRPC requests to the Service. > This is a lower-level measure that manages rates at the service level. > > OPS, or Operations per Second, is used by Temporal Cloud. > An operation is anything a user does directly, or that Temporal does on behalf of the user in the background, that results in load on Temporal Server. > This is a lower-level measure that manages rates across Temporal cloud services. > > In summary, APS is a higher-level measure to limit and mitigate Action spikes in Temporal Cloud. > RPS and OPS are lower-level measures to control and balance request rates at the service level. > ### What happens when my Actions rate exceeds my limit? When your Action rate exceeds your quota, Temporal Cloud throttles Actions. Throttling limits the rate at which Actions are performed to prevent the Namespace from exceeding its APS limit. **How throttling works:** - Low-priority operations are throttled first; higher-priority operations (like starting or signaling Workflows) continue when possible. - Rate limiting is not instantaneous, so usage may briefly exceed your limit before throttling takes effect. - When throttled, the server returns `ResourceExhausted` errors that SDK clients automatically retry. - If throttling persists beyond the SDK's retry limit, client calls can fail. **To avoid data loss during throttling:** - Log any failed client calls (with payloads) so you can retry or backfill later. - Set up [limit metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) to alert when approaching your limits. See [Throttling behavior](/cloud/limits#throttling-behavior) for more details. Your rate limits can be adjusted automatically over time or provisioned manually with Capacity Modes. We recommend tracking your Actions Rate and Limits using Temporal metrics to assess your use cases specific needs. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) to track usage trends. For Namespaces using Provisioned Capacity, on-demand envelope metrics show what your limits would be if operating in on-demand mode. Use these to evaluate whether on-demand capacity would meet your needs before switching modes. See [On-demand envelope limits](/cloud/service-health#on-demand-envelope-limits) for details. > **📝 Note:** > Actions that don't count against APS > Actions that are external to the core Temporal service do not contribute to your APS. These Calls include: > * [Export](/cloud/export) > * Capacity Related Actions ## On-demand capacity Using On-Demand Capacity, your rate limit grows automatically along with your usage. Each Namespace has an Actions per second (APS), Requests per second (RPS), and Operations per second (OPS) limit that scales automatically with usage. Your APS limit never falls below its [default limit](/cloud/limits#actions-per-second). If Temporal Support has manually set your Namespace's limit, that value becomes your floor in place of the default, and it persists across capacity mode changes. Scaling automatically adjusts based on the lesser of 4 * APS Average or 2 * APS P90 over the past 7 days. To see your Namespace's current APS, RPS, and OPS limits, track the `temporal_cloud_v1_action_limit`, `temporal_cloud_v1_service_request_limit`, and `temporal_cloud_v1_operations_limit` metrics. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits). If you experience usage spikes, you may hit a throughput limit. In that case, consider switching to [Provisioned Capacity](#provisioned-capacity). You can also optimize your workload to remain under the On-Demand limits. See [Best Practices for Managing APS Limits](/best-practices/managing-aps-limits) for more information. ### What kind of throughput can I get on Temporal Cloud with On-Demand Capacity? Each Namespace has a rate limit, which is measured in Actions per second (APS). Your APS limit automatically adjusts based on a formula that compares your average usage over the last 7 days and your usage at the 90th percentile, or P90. Your throughput limit will never fall below the [default limit](/cloud/limits#actions-per-second) for your Namespace. Under On-Demand capacity you are only charged for the Actions you use. To see your current limit, view it in the Temporal Cloud UI under the Namespace overview, retrieve it with the CLI by running [`temporal cloud namespace capacity get`](/cli/command-reference/cloud/namespace#capacity-get) or [`tcld namespace capacity get`](/cloud/tcld/namespace/#capacity), or track the `temporal_cloud_v1_action_limit` metric described in [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits). For example, if your average APS over the last 7 days was 200 and your P90 was 500, your limit would be the greater of: * The [default limit](/cloud/limits#actions-per-second) * The lesser of: * 4 * 200 APS Mean = 800 APS * 2 * 500 APS P90 = 1000 APS Because 800 APS exceeds the default limit, your limit in this example would be 800 APS. ![Usage graph showing increasing APS usage for one month, with occasional spikes, and a rising APS limit](/img/cloud/provisioned-capacity/usage_graph.png) ## Provisioned Capacity Provisioned Capacity provides an alternative to On-Demand Capacity by allowing you to control the limits on your Namespace based on your specific need. | | Actions Per Second | Requests Per Second | Operations Per Second| |---------------|--------------------|---------------------|----------------------| | TRU | 500 | 1500 | 4000 | Customers can set 2, 3, 4, 6, 8, 10, 12 TRUs, subject to availability. TRUs can be adjusted hourly. See [Capacity Mode Pricing](/cloud/pricing#capacity-modes-pricing) for pricing implications. ### What kind of throughput can I get with Temporal Cloud with Provisioned Capacity? With Provisioned Capacity, you can set your rate limits by selecting the number of Temporal Resource Units (TRUs) on your Namespace. Each TRU supports up to 500 APS and can be provisioned in groups of 2, 3, 4, 6, 8, 10, or 12 TRUs if there is capacity available in a region. When TRUs are requested we aim to provision the additional capacity within two minutes. > **💡 Tip:** > Large TRU requests > > For Requests in excess of 4 TRUs in regions outside of the US, we recommend submitting a support ticket to ensure capacity availability. > ### Provisioned Capacity Availability The amount of capacity available within a region may vary. Temporal will check available capacity at the time of your request and aims to provision requested capacity within two minutes. If you need capacity beyond what is self-serviceable or available in a region, please [file a support ticket](/cloud/support#ticketing) indicating the limit, region, and timeframe that the capacity is needed. ### When should I use Provisioned Capacity? Provisioned Capacity works well when you’re aware of specific increases in load on your Namespace. For example: * Planned events * Unplanned events/usage spikes * Known but sudden system spikes * Load testing * Migrating workloads Depending on your usage patterns and your system monitoring, you can use Provisioned Capacity to quickly remedy rate limiting without contacting support. You can also automate changes in capacity if you have a known event or a recurring usage pattern that produces predictable usage spikes. ### Monitoring provisioned capacity utilization Sustained usage well below your provisioned limit can mean you are paying for capacity you are not using, since each TRU beyond the first carries a minimum hourly charge (see [Capacity Mode Pricing](/cloud/pricing#capacity-modes-pricing)). For the metrics to watch and how to alert on utilization, see [Provisioned capacity utilization](/cloud/service-health#provisioned-capacity-utilization). ## Set capacity modes Capacity Modes and TRUs can be set via the Temporal Cloud UI, CLI, or API. Capacity modes can be set and adjusted by Global Admin and Namespace Admin. When you switch to Provisioned Capacity, your limit is set by the number of TRUs you select. When you switch back to On-Demand, your limit is recalculated from the trailing 7-day usage formula and never falls below your floor. If Temporal Support has set a custom limit for your Namespace, that limit is your floor, so switching to Provisioned and back to On-Demand preserves it. ### Setting Capacity Modes from the UI You can set Capacity Modes for an individual Namespace by navigating to the Namespace page in the Temporal Cloud UI (`https://cloud.temporal.io/namespaces/`). To view your current capacity configuration and change your capacity mode, navigate to the capacity tile and click *Manage Capacity*. ![Manage Capacity button in the Temporal UI](/img/cloud/provisioned-capacity/manage_capacity_button.png) Under *Manage Capacity* you will be able to select between *On-Demand* and *Provisioned Capacity* modes. The *On-Demand* section will display your available On-Demand capacity. The *Provisioned* section will display the limit available with selected TRUs and the Included Actions required per hour. [See details on Provisioned Capacity Pricing](/cloud/pricing#capacity-modes-pricing). To switch to Provisioned capacity: 1. Select the *Provisioned* radio button. 1. Specify the requested number of TRUs using the slider. 1. Check the dialog acknowledging potential pricing implications. 1. Click *Confirm*. In addition to the Capacity Mode selections, a summary of APS usage over the last seven days is included to help you estimate your current usage. For more detailed information, we recommend setting up metrics that track your APS and Limits. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) to track usage trends. ![Manage Capacity panel in the Temporal UI](/img/cloud/provisioned-capacity/manage_capacity_panel.png) ### Set capacity modes from the CLI **Temporal CLI** ```command temporal cloud namespace capacity update \ --namespace \ --capacity-mode provisioned \ --capacity-value ``` Use this command to specify the Namespace name and configure the capacity settings: * `--capacity-mode provisioned` sets a fixed capacity allocation for the Namespace. * `--capacity-value` sets the provisioned throughput value in TRUs (Temporal Resource Units). To switch to on-demand capacity, omit `--capacity-value`: ```command temporal cloud namespace capacity update \ --namespace \ --capacity-mode on_demand ``` Optional flags: * `--async-operation-id` specifies an identifier for tracking the asynchronous operation. If not specified, the CLI generates one automatically. * `--resource-version` specifies the resource version (etag) to update from. If not set, the CLI uses the latest version. **tcld** ```command tcld namespace capacity update \ --namespace \ --capacity-mode \ --capacity-value \ [--request-id --resource-version ] ``` Use this command to specify the Namespace name and configure the capacity settings: * `--capacity-mode` sets the billing mode for the Namespace. Use `on_demand` for automatic scaling or `provisioned` for a fixed capacity allocation. * `--capacity-value` sets the throughput value in TRUs (Temporal Resource Units). Optional flags: * `--request-id` specifies a request identifier for the asynchronous operation. If not specified, the server assigns one automatically. * `--resource-version` specifies the resource version (etag) to update from. If not set, the CLI uses the latest version. If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before capacity update. ### Setting Capacity Modes from the API Call the `UpdateNamespace` API after Namespace creation and define the desired capacity state as part of the capacity spec. --- # Authenticate with mTLS certificates Source: https://docs.temporal.io/cloud/certificates [Temporal Cloud](https://temporal.io/cloud) supports both mTLS and [API key](/cloud/api-keys) authentication for namespace access. When using mTLS authentication, each [Worker Process](/workers#worker-process) uses a CA certificate and private key to connect to Temporal Cloud. Temporal Cloud does not require an exchange of secrets; only the certificates produced by private keys are used for verification. > **⚠️ Caution:** > Don't let your certificates expire > > An expired root CA certificate invalidates all downstream certificates. > > An expired end-entity certificate prevents a [Temporal Client](/encyclopedia/temporal-client) from > connecting to a Namespace or starting a Workflow Execution. If the client is on a Worker, any current Workflow > Executions that are processed by that Worker either run indefinitely without making progress until the Worker resumes or > fail because of timeouts. > > Temporal Cloud sends [courtesy emails](/cloud/notifications#admin-notifications) prior to certificate expiry. > > > To update certificates, see > [How to add, update, and remove certificates in a Temporal Cloud Namespace](#manage-certificates). > All certificates used by Temporal Cloud must meet the following requirements. ## Requirements for CA certificates in Temporal Cloud Certificates provided to Temporal for your [Namespaces](/namespaces) _must_ meet the following requirements. **When a client presents an end-entity certificate, and the whole certificate chain is constructed, each certificate in the chain (from end-entity to the root) must have a unique Distinguished Name.** > **⚠️ Caution:** > > Distinguished Names are _not_ case sensitive; that is, uppercase letters (such as ABC) and lowercase letters (such as > abc) are equivalent. > ### CA certificates A CA certificate is a type of X.509v3 certificate used for secure communication and authentication. In Temporal Cloud, CA certificates are required for configuring mTLS. CA certificates _must_ meet the following criteria: - The certificates must be X.509v3. - Each certificate in the bundle must be either a root certificate or issued by another certificate in the bundle. - Each certificate in the bundle must include `CA: true`. - A certificate cannot be a well-known CA (such as DigiCert or Let's Encrypt) _unless_ the user also specifies certificate filters. - The signing algorithm must be either RSA or ECDSA and must include SHA-256 or stronger message authentication. SHA-1 and MD5 cannot be used. - The certificates cannot be generated with a passphrase. > **ℹ️ Info:** > > A certificate bundle can contain up to 16 CA certificates. A certificate bundle can have a maximum payload size of 32 KB > before base64 encoding. > ### End-entity certificates An end-entity certificate is a type of X.509v3 certificate used by clients to authenticate themselves. Temporal Cloud lets you limit access to specific end-entity certificates by using [certificate filters](#manage-certificate-filters). An end-entity (leaf) certificate _must_ meet the following criteria: - The certificate must be X.509v3. - Basic constraints must include `CA: false`. - The key usage must include Digital Signature. - The signing algorithm must be either RSA or ECDSA and must include SHA-256 or stronger message authentication. SHA-1 and MD5 cannot be used. - The certificate needs the `extendedKeyUsage` set to `clientAuth` or it needs to be omitted. ## How to issue root CA and end-entity certificates Temporal Cloud authenticates a client connection by validating the client certificate against one or more CA certificates that are configured for the specified Namespace. Choose one of the following options to generate and manage the certificates: ### Option 1: You already have certificate management infrastructure If you have existing certificate management infrastructure that supports issuing CA and end-entity certificates, export the CA and generate an end-entity certificates using your existing tools. Ensure that the CA certificate is long-lived and that the end-entity certificate expires before the CA certificate. Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui) and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity certificate. ### Option 2: You don't have certificate management infrastructure If you don't have an existing certificate management infrastructure, we recommend using API keys for authentication instead. API keys are generally easier to manage than mTLS certs if you're not using certificate management infrastructure otherwise. If you still want to use mTLS, issue the CA and end-entity certificates using [tcld](#use-tcld-to-generate-certificates) or open source tools like OpenSSL or [step CLI](#use-step-cli-to-generate-certificates). #### Use tcld to generate certificates You can generate CA and end-entity certificates by using [tcld](/cloud/tcld). Although Temporal Cloud supports long-lived CA certificates, a CA certificate generated by [tcld](/cloud/tcld) has a maximum duration of 1 year (`-d 1y`). You must set an end-entity certificate to expire before its root CA certificate, so specify its duration appropriately. To create a new CA certificate, use `tcld gen ca`. ```sh mkdir temporal-certs cd temporal-certs tcld gen ca --org temporal -d 1y --ca-cert ca.pem --ca-key ca.key ``` The contents of the generated `ca.pem` should be pasted into the "Authentication" section of your Namespace settings page. To create a new end-entity certificate, use `tcld gen leaf`. ```sh tcld gen leaf --org temporal -d 364d --ca-cert ca.pem --ca-key ca.key --cert client.pem --key client.key ``` You can now use the generated CA certificate (`ca.pem`) with Temporal Cloud and configure your client with these certs (`client.pem`, `client.key`). Upload the contents of the `ca.pem` file to the **Authentication** section of your **Namespace** settings. Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui) and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity certificate. #### Use step CLI to generate certificates Temporal Cloud requires client certificates for authentication and secure communication. [The step CLI](https://github.com/smallstep/cli) is a popular and easy-to-use tool for issuing certificates. Before you begin, ensure you have installed smallstep/cli by following the instructions in the [installation guide](https://github.com/smallstep/cli#installation). A Certificate Authority (CA) is a trusted entity that issues digital certificates. These certificates certify the ownership of a public key by the named subject of the certificate. End-entity certificates are issued and signed by a CA, and they are used by clients to authenticate themselves to Temporal Cloud. Create a self-signed CA certificate and use it to issue an end-entity certificate for your Temporal Cloud namespace. ##### 1. Create a certificate authority (CA) Create a new Certificate Authority (CA) using step CLI: ```command step certificate create "CertAuth" CertAuth.crt CertAuth.key --profile root-ca --no-password --insecure ``` This command creates a self-signed CA certificate named `CertAuth.crt` and private key `CertAuth.key`. This CA certificate will be used to sign and issue end-entity certificates. ##### 2. Set the Namespace Name Set the Namespace Name as the common name for the end-entity certificate: **macOS** For Linux or macOS: ```command export NAMESPACE_NAME=your-namespace ``` **Windows** For Windows: ```command set NAMESPACE_NAME=your-namespace ``` Replace `your-namespace` with the name of your Temporal Cloud namespace. ##### 3. Create and sign an end-entity certificate Create and sign an end-entity certificate with a common name equal to the Namespace Name: ```command step certificate create ${NAMESPACE_NAME} ${NAMESPACE_NAME}.crt ${NAMESPACE_NAME}.key --ca CertAuth.crt --ca-key CertAuth.key --no-password --insecure --not-after 8760h ``` This command creates an end-entity certificate (`your-namespace.crt`) and private key (`your-namespace.key`) that is signed by your CA (`CertAuth`). ##### 4. Convert to PKCS8 format for Java SDK (optional) If you are using the Temporal Java SDK, you will need to convert the PKCS1 file format to PKCS8 file format. Export the end-entity's private key to a PKCS8 file: ```command openssl pkcs8 -topk8 -inform PEM -outform PEM -in ${NAMESPACE_NAME}.key -out ${NAMESPACE_NAME}.pkcs8.key -nocrypt ``` ##### 5. Use the Certificates with Temporal Cloud You can now use the generated client certificate (`your-namespace.crt`) and the CA certificate (`CertAuth.crt`) with Temporal Cloud. Upload the contents of the `CertAuth.crt` file to the **CA Certificates** section of your **Namespace** settings. Follow the instructions to [upload the CA certificate](/cloud/certificates#update-certificates-using-temporal-cloud-ui) and [configure your client](/cloud/certificates#configure-clients-to-use-client-certificates) with the end-entity certificate. ## How to control authorization for Temporal Cloud Namespaces We recommend that an end-entity certificate be scoped to a specific Namespace to enforce the principle of least privilege. Temporal Cloud requires full CA chains, so you can achieve authorization in two ways. ### Option 1: Issue a separate root certificate for each Namespace Each certificate must belong to a chain up to the root CA certificate. Temporal uses the root CA certificate as the trusted authority for access to your Namespaces. 1. Ensure that your certificates meet the [certificate requirements](#certificate-requirements). 1. [Add client CA certificates to a Cloud Namespace](/cli/command-reference/cloud/namespace#mtls-cert-ca-create). ### Option 2: Use the same root certificate for all Namespaces but create a separate certificate filter for each Namespace [How to manage certificate filters in Temporal Cloud](#manage-certificate-filters) ## How to receive notifications about certificate expiration To keep your Namespace secure and online, you must update the CA certificate for the Namespace _before_ the certificate expires. To help you remember to do so, Temporal Cloud sends [email notifications](/cloud/notifications#admin-notifications). ## How to handle a compromised end-entity certificate > **⚠️ Warning:** > > Temporal does not support or check certificate revocation lists (CRLs). Customers are expected to keep their > certificates up to date. > The recommended approach to avoiding compromised certificates is to have short-lived end-entity certificates. A short-lived compromised certificate can be left to expire on its own. Seek guidance from your infosec team to determine an appropriate value of "short-lived" for your business. If you suspect or confirm that an end-entity certificate has been compromised, and leaving it to expire is not an option, take immediate action to secure your Temporal Cloud Namespace and prevent unauthorized access. If you're using certificate filters, you can set the filters to block a compromised certificate. Follow the instructions in [How to manage certificate filters in Temporal Cloud](#manage-certificate-filters). If you need to replace a compromised certificate manually, follow these steps: ### 1. Generate a new CA certificate Follow the instructions in [How to issue CA and end-entity certificates](#issue-certificates). All end-entity certificates that can be reached by the previous CA must be regenerated. Ensure the new CA certificate meets [the certificate requirements](#certificate-requirements). ### 2. Deploy the new CA certificate to the Namespace Follow the instructions for issuing a [new CA certificate to a Namespace](#option-1-issue-a-separate-root-certificate-for-each-namespace). Deploy the new CA certificate alongside the existing one, so you don’t lose connectivity with old end-entity certificates before the new ones are generated and deployed. ### 3. Regenerate end-entity certificates with the new CA certificate [Configure all clients](/cloud/certificates#configure-clients-to-use-client-certificates) (Temporal CLI, SDKs, or Workers) to use the new certificate and private key alongside the compromised key. Update the client configuration as described in [Configure clients to use client certificates](#configure-clients-to-use-client-certificates). Test the new certificate to confirm clients can connect to the Namespace without issues. ### 4. Remove the compromised CA certificate Follow the instructions in [How to add, update, and remove certificates in a Temporal Cloud Namespace](#manage-certificates) to remove the compromised CA certificate from the Namespace. ### 5. Monitor and audit After implementing the changes, monitor your Namespace for unauthorized access attempts or unusual activity in audit logs. Review your certificate management practices to identify how the compromise occurred. Consider implementing stricter controls, such as: - Limiting end-entity certificates to specific Namespaces - Rotating certificates regularly as a preventive measure - [Audit logs on a regular schedule](/cloud/audit-logs) ## How to add, update, and remove certificates in a Temporal Cloud Namespace > **📝 Note:** > > To manage certificates for a Namespace, a user must have [Namespace Admin](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) > permission for that Namespace. > To manage certificates for Temporal Cloud Namespaces, use the **Namespaces** page in Temporal Cloud UI or the [`temporal cloud namespace mtls cert-ca` commands](/cli/command-reference/cloud/namespace#mtls-cert-ca). Don't let your certificates expire! Add reminders to your calendar to issue new CA certificates well before the expiration dates of the existing ones. Temporal Cloud begins sending notifications 15 days before expiration. For details, see the previous section ([How to receive notifications about certificate expiration](#expiration-notifications)). When updating CA certificates, it's important to follow a rollover process (sometimes referred to as "certificate rotation"). Doing so enables your Namespace to serve both CA certificates for a period of time until traffic to your old CA certificate ceases. This prevents any service disruption during the rollover process. ### Update certificates using Temporal Cloud UI Updating certificates using the following strategy allows for a zero-downtime rotation of certificates. 1. On the left side of the window, select **Namespaces**. 2. Select the name of the Namespace to update. 3. In the top-right portion of the page for the Namespace, select **Edit**. 4. On the **Edit** page, select the **Authentication** card to expand it. 5. In the certificates box, scroll to the end of the existing certificate (that is, past `-----END CERTIFICATE-----`). 6. On the following new line, paste the entire PEM block of the new certificate. 7. Select **Save**. 8. Wait until all Workers are using the new certificate. 9. Return to the **Edit** page of the Namespace and select the **Authentication** card. 10. In the certificates box, delete the old certificate, leaving the new one in place. 11. Select **Save**. ### Update certificates using the Temporal CLI Updating certificates using the following strategy allows for a zero-downtime rotation of certificates. 1. Add the new CA certificate alongside the existing certificate: ```command temporal cloud namespace mtls cert-ca create \ --namespace . \ --ca-certificate-file ``` 1. Monitor traffic to the old certificate until it ceases. 1. Remove the old CA certificate: ```command temporal cloud namespace mtls cert-ca delete \ --namespace . \ --ca-certificate-file ``` ### Update certificates using tcld Updating certificates using the following strategy allows for a zero-downtime rotation of certificates. 1. Create a single file that contains both your old and new CA certificate PEM blocks. Just concatenate the PEM blocks on adjacent lines. ``` -----BEGIN CERTIFICATE----- ... old CA cert ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... new CA cert ... -----END CERTIFICATE----- ``` 1. Run the `tcld namespace accepted-client-ca set` command with the CA certificate bundle file. ```bash tcld namespace accepted-client-ca set --ca-certificate-file ``` 1. Monitor traffic to your old certificate until it ceases. 1. Create another file that contains only the new CA certificate. 1. Run the `tcld namespace accepted-client-ca set` command again with the updated CA certificate bundle file. ## How to manage certificate filters in Temporal Cloud To limit access to specific [end-entity certificates](#end-entity-certificates), create certificate filters. Each filter contains values for one or more of the following fields: - commonName (CN) - organization (O) - organizationalUnit (OU) - subjectAlternativeName (SAN) Corresponding fields in the client certificate must match every specified value in the filter. The values for the fields are case-insensitive. If no wildcard is used, each specified value must match its field exactly. To match a substring, place a single `*` wildcard at the beginning or end (but not both) of a value. You cannot use a `*` wildcard by itself. You can create a maximum of 25 certificate filters in a Namespace. If you provide a well-known CA certificate, you cannot clear a certificate filter. A well-known CA certificate is one that is typically included in the certificate store of an operating system. **Examples** In the following example, only the CN field of the certificate's subject is checked, and it must be exactly `code.example.com`. The other fields are not checked. ```json AuthorizedClientCertificate { CN : "code.example.com" } ``` In the following example, the CN field must be `stage.example.com` and the O field must be `Example Code Inc.` ```json AuthorizedClientCertificate { CN : "stage.example.com" O : "Example Code Inc." } ``` When using a `*` wildcard, the following values are valid: - `*.example.com` matches `code.example.com` and `text.example.com`. - `Example Code*` matches `Example Code` and `Example Code Inc`. The following values are not valid: - `.example.*` - `code.*.com` - `*` ### Manage certificate filters using Temporal Cloud UI To add or remove a certificate filter, follow these steps: 1. On the left side of the window, click **Namespaces**. 1. On the **Namespaces** page, click the name of the Namespace to manage. 1. On the right side of the page for the selected Namespace, click **Edit**. 1. On the **Edit** page, click the **Authentication** card. - To add a certificate filter, click **Add a Certificate Filter** and enter values in one or more fields. - To remove a certificate filter, click the **×** in the upper-right corner of the filter details. 1. To cancel your changes, click **Back to Namespace**. To save your changes, click **Save**. ### Manage certificate filters using the Temporal CLI Use the following commands to add, remove, or list certificate filters: - [`temporal cloud namespace mtls cert-filter create`](/cli/command-reference/cloud/namespace#mtls-cert-filter-create) - [`temporal cloud namespace mtls cert-filter delete`](/cli/command-reference/cloud/namespace#mtls-cert-filter-delete) - [`temporal cloud namespace mtls cert-filter list`](/cli/command-reference/cloud/namespace#mtls-cert-filter-list) ### Manage certificate filters using tcld To set or clear certificate filters, use the following [tcld](/cloud/tcld) commands: - [tcld namespace certificate-filters import](/cloud/tcld/namespace/#import) - [tcld namespace certificate-filters clear](/cloud/tcld/namespace/#clear) To view the current certificate filters, use the [tcld namespace certificate-filters export](/cloud/tcld/namespace/#export) command. ## Configure clients to use client certificates - [Go SDK](/develop/go/client/temporal-client#connect-to-temporal-cloud) - [Java SDK](/develop/java/client/temporal-client#connect-to-temporal-cloud) - [PHP SDK](/develop/php/client/temporal-client#connect-to-a-dev-cluster) - [Python SDK](/develop/python/client/temporal-client#connect-to-temporal-cloud) - [TypeScript SDK](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) - [.NET SDK](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) ### Configure Temporal CLI To connect to a Temporal Namespace using the Temporal CLI and certificate authentication, specify your credentials and the TLS server name: ```sh temporal \ --tls-ca-path \ --tls-cert-path \ --tls-key-path \ --tls-server-name ``` For more information on Temporal CLI environment variables, see [Environment variables](/cli/setup-cli#environment-variables). --- # Connectivity Source: https://docs.temporal.io/cloud/connectivity > Network connectivity details for using Temporal Cloud ## Private network connectivity for namespaces Temporal Cloud supports private connectivity to Namespaces via AWS PrivateLink or GCP Private Service Connect, in addition to the default public internet endpoints. Namespace access is always securely authenticated via [API keys](/cloud/api-keys#overview) or [mTLS](/cloud/certificates), regardless of how you choose to connect. For information about IP address stability and allowlisting, see [IP addresses](/cloud/connectivity/ip-addresses). If your security team requires allowlisting specific IP addresses for firewall rules, Temporal Cloud also offers a [Stable IPs configuration option](/cloud/connectivity/ip-addresses) that provides non-changing IP addresses for your Namespace endpoint. ### Required steps Setting up private connectivity is a three-step process — and it's important to understand that **private connectivity** (the network path) and **Connectivity Rules** (Temporal's enforcement layer) are related but separate concepts: 1. **Set up the private connection** from your VPC to the region where your Temporal Namespace is located. 1. **Update your private DNS and/or client configuration** to actually use the private connection. Activating private connectivity does not change your Namespace Endpoint or Regional Endpoint automatically — clients keep resolving the public addresses until you do this step. 1. **(GCP PSC: required. AWS PrivateLink: optional.) Create a Connectivity Rule** for the private connection and attach it to the target Namespace(s). This blocks all access to the Namespace that does not arrive over a configured connection. You can mix private and public rules to also allow internet connectivity. For steps 1 and 2, follow the guide for your Namespace's cloud provider: - [AWS PrivateLink](/cloud/connectivity/aws-connectivity) creation and private DNS setup - [Google Cloud Private Service Connect](/cloud/connectivity/gcp-connectivity) creation and private DNS setup > **⚠️ Caution:** > Finish client setup (complete step 2) > > After creating a private connection, you must set up private DNS or update the configuration of all clients you want to use the private connection. > > We recommend using private DNS. > > Without this step, your clients may connect to the Namespace over the internet if they were previously using public connectivity, or they will not be able to connect at all. > > If that's not an option for you, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity). > For step 3, keep reading for details on [Connectivity Rules](/cloud/connectivity#connectivity-rules). ## Connectivity rules > **ℹ️ Info:** > Web UI Connectivity > > The Temporal Cloud Web UI is not currently subject to connectivity rule enforcement. > Even if a namespace is configured with private connectivity rules, the Web UI for that namespace remains accessible over the public internet. > ### Definition Connectivity Rules are Temporal Cloud's mechanism for restricting the network paths that can reach a Namespace. They are enforced by Temporal Cloud — they do not create or modify the underlying network connection. By default, a Namespace has zero Connectivity Rules and is reachable over (1) the public internet and (2) any private connections you've already configured to the region containing the Namespace. Namespace access is always securely authenticated via [API keys](/cloud/api-keys#overview) or [mTLS](/cloud/certificates), regardless of Connectivity Rules. When you attach one or more Connectivity Rules to a Namespace, Temporal Cloud immediately blocks any traffic that does not match a rule on that Namespace. A Namespace can have multiple Connectivity Rules, and you can mix public and private rules. Each Connectivity Rule specifies either generic public (internet) access or a specific private connection. #### When you need a Connectivity Rule | Provider | Connectivity Rule for private access | Why | | -------- | ------------------------------------ | --- | | AWS PrivateLink | **Optional.** Add one only if you want to enforce private-only access (block internet traffic to that Namespace). | AWS PrivateLink connections become usable as soon as the VPC endpoint is `Available`. Adding a Connectivity Rule restricts access; it does not establish it. | | GCP Private Service Connect | **Required.** The PSC endpoint stays in `Pending` until a matching Connectivity Rule is created. | The Connectivity Rule is what tells Temporal Cloud to accept the PSC connection. | A public Connectivity Rule can optionally enable [Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses), so Namespaces attached to it resolve their Namespace endpoint to a published, fixed set of IP addresses you can allowlist in your firewall. Only one public Connectivity Rule is allowed per account (see [Permissions and limits](#permissions-and-limits) below). > **⚠️ Caution:** > > Connectivity Rules cannot be updated in place. If you already have a public Connectivity Rule and want to turn on Stable IPs, you must delete the existing public rule, create a new one with Stable IPs enabled, and re-attach it to every Namespace that needs it. Attempting to create a second public rule alongside the existing one returns an error. > An AWS PrivateLink (PL) private Connectivity Rule requires: - `connection-id`: The **VPC endpoint identifier** of the PL connection — the `vpce-…` value from your AWS account, *not* the endpoint service or DNS name (ex: `vpce-00939a7ed9EXAMPLE`). - `region`: The region of the PL connection, prefixed with `aws-` (ex: `aws-us-east-1`). Must be the same region as the Namespace. Refer to the [Temporal Cloud region list](/cloud/regions) for supported regions. A GCP Private Service Connect (PSC) private Connectivity Rule requires: - `connection-id`: The **PSC connection identifier** of the endpoint (ex: `1234567890123456789`). Find it on the endpoint's detail page in the Google Cloud console. - `region`: The region of the PSC connection, prefixed with `gcp-` (ex: `gcp-us-east1`). Must be the same region as the Namespace. Refer to the [Temporal Cloud region list](/cloud/regions) for supported regions. - `gcp-project-id`: The identifier of the GCP project where you created the PSC connection (ex: `my-example-project-123`). Connectivity Rules can be created and managed with the [Temporal Cloud CLI extension](/cli/cloud), [tcld](/cloud/tcld/), [Terraform](https://github.com/temporalio/terraform-provider-temporalcloud/), or the [Cloud Ops API](/ops). There is no Temporal Cloud Web UI option for creating or managing Connectivity Rules at this time. > **💡 Tip:** > Connectivity Rules give Temporal visibility into your private connections > > Without a Connectivity Rule, Temporal Cloud has no record that your PrivateLink or PSC endpoint exists. If you open a support ticket about a private-connectivity issue, having a Connectivity Rule attached to the affected Namespace lets us correlate the connection on our side and is the fastest path to debugging. > ### Permissions and limits Only [Account Admins and Account Owners](/cloud/manage-access/roles-and-permissions#account-level-roles) can create and manage connectivity rules. Connectivity rules are visible to Account Developers, Account Admins, and Account Owners. By default each namespace is limited to 5 private connectivity rules, and each account is limited to 50 private connectivity rules. You can [contact support](/cloud/support#support-ticket) to request a higher limit. There is only one public rule allowed per account, because it's generic and can be reused for all namespaces that you want to be available on the internet. Trying to create more than one public rule will throw an error. ## Creating a connectivity rule ### Temporal Cloud CLI **Temporal CLI** Create a private connectivity rule (AWS): ```bash temporal cloud connectivity private create --connection-id "vpce-abcde" --region "aws-us-east-1" ``` Create a private connectivity rule (GCP): ```bash temporal cloud connectivity private create --connection-id "1234567890" --region "gcp-us-central1" --gcp-project-id "my-project-123" ``` Create a public connectivity rule. You only need to do this once for your account: ```bash temporal cloud connectivity public create ``` **tcld** Create a private connectivity rule (AWS): ```bash tcld connectivity-rule create --connectivity-type private --connection-id "vpce-abcde" --region "aws-us-east-1" ``` Create a private connectivity rule (GCP): ```bash tcld connectivity-rule create --connectivity-type private --connection-id "1234567890" --region "gcp-us-central1" --gcp-project-id "my-project-123" ``` Create a public connectivity rule. You only need to do this once for your account: ```bash tcld connectivity-rule create --connectivity-type public ``` The `cr` alias works the same way: ```bash tcld cr create --connectivity-type private --connection-id "vpce-abcde" --region "aws-us-east-1" ``` ```bash tcld cr create --connectivity-type public ``` To enable [Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses) on the public rule, use the Cloud Ops API or Terraform. See [How to enable Stable IPs](/cloud/connectivity/ip-addresses#how-to-enable-stable-ips). ### Terraform [Examples in the Terraform repo](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/examples/resources/temporalcloud_connectivity_rule/resource.tf) ## Attach connectivity rules to a namespace Be careful! When any connectivity rules are set on a namespace, that namespace is ONLY accessible via the connections defined in those rules. If you remove a connectivity rule that your workers are using, your traffic will be interrupted. If you already have workers using a namespace, adding both a public rule and any private rules simultaneously can help you avoid unintended loss of access. You can then ensure all workers are using private connections, and then remove the public rule. ### Temporal Cloud CLI **Temporal CLI** Attach connectivity rules to a namespace: ```bash temporal cloud namespace connectivity attach --namespace "my-namespace.abc123" \ --connectivity-rule-id "rule-id-1" --connectivity-rule-id "rule-id-2" ``` Rules attach and detach individually. To detach `rule-c` while leaving `rule-a` and `rule-b` in place, detach only `rule-c`: ```bash temporal cloud namespace connectivity detach --namespace "my-namespace.abc123" \ --connectivity-rule-id "rule-c" ``` Detaching every rule makes the namespace public. List the attached rules first, then detach them: ```bash temporal cloud namespace connectivity list --namespace "my-namespace.abc123" ``` **tcld** Set the connectivity rules on a namespace: ```bash tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --connectivity-rule-ids "rule-id-1" --connectivity-rule-ids "rule-id-2" ``` Or using aliases: ```bash tcld n scrs -n "my-namespace.abc123" --ids "rule-id-1" --ids "rule-id-2" ``` Connectivity rules are attached as a set, so if rules `rule-a`, `rule-b`, and `rule-c` were attached to a namespace and you wanted to detach `rule-c` only, you'd make one call attaching both `rule-a` and `rule-b`: ```bash tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --ids rule-a --ids rule-b ``` Remove all connectivity rules (this will make the namespace public): ```bash tcld namespace set-connectivity-rules --namespace "my-namespace.abc123" --remove-all ``` ### Terraform [Example in the Terraform repo](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main/examples/resources/temporalcloud_namespace/resource.tf#L113-L128) ## View the connectivity rules for a namespace You have two ways to view the connectivity rules attached to a particular namespace. ### Get namespace Connectivity rules are included in the namespace details returned by the `namespace get` command. **Temporal CLI** ```bash temporal cloud namespace get -n "my-namespace.abc123" ``` **tcld** ```bash tcld namespace get -n "my-namespace.abc123" ``` ### List connectivity rules by namespace To see only the connectivity rules for a specific namespace, without other namespace details, list the rules for that namespace. **Temporal CLI** ```bash temporal cloud namespace connectivity list -n "my-namespace.abc123" ``` **tcld** ```bash tcld connectivity-rule list -n "my-namespace.abc123" ``` ## Update DNS or clients to use private connectivity We strongly recommend using private DNS instead of updating client server and TLS settings: - [How to set up private DNS in AWS](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink) - [How to set up private DNS in GCP](/cloud/connectivity/gcp-connectivity#configuring-private-dns-for-gcp-private-service-connect) If you are unable to configure private DNS, you must update two settings in your Temporal clients: 1. Set the endpoint server address to the PrivateLink or Private Service Connect endpoint (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233` or `:7233`). 2. Set TLS configuration to override the TLS server name (the Namespace Endpoint, for example, `my-namespace.my-account.tmprl.cloud`). The TLS server name override depends on your authentication method: | Authentication | TLS server name to use | | -------------- | ---------------------- | | mTLS (single-region Namespace) | The Namespace Endpoint, for example `my-namespace.my-account.tmprl.cloud` | | API key (single-region Namespace) | The regional API endpoint, for example `us-east-1.aws.api.temporal.io` or `us-central1.gcp.api.temporal.io` | | Multi-region Namespace (mTLS or API key) | The active region endpoint, for example `us-east-1.aws.api.temporal.io` | If you authenticate with an API key over PrivateLink/PSC and use the wrong server name, the TLS handshake will fail with errors such as `connection reset by peer` even though `nc` reports the port as open. Updating these settings depends on the client you're using. #### temporal CLI ```bash TEMPORAL_ADDRESS=vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 TEMPORAL_NAMESPACE=my-namespace.my-account TEMPORAL_TLS_CERT= TEMPORAL_TLS_KEY= TEMPORAL_TLS_SERVER_NAME=my-namespace.my-account.tmprl.cloud temporal workflow count -n $TEMPORAL_NAMESPACE ``` #### grpcurl ```bash grpcurl \ -servername my-namespace.my-account.tmprl.cloud \ -cert path/to/cert.pem \ -key path/to/cert.key \ vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \ temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo ``` #### Temporal SDKs **Go** ```go c, err := client.Dial(client.Options{ HostPort: "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233", Namespace: "namespace-name.accId", ConnectionOptions: client.ConnectionOptions{ TLS: &tls.Config{ Certificates: []tls.Certificate{cert}, ServerName: "my-namespace.my-account.tmprl.cloud", }, }, }) ``` **Java** ```java WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs( WorkflowServiceStubsOptions.newBuilder() .setSslContext(sslContext) .setTarget("vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233") .setChannelInitializer( c -> c.overrideAuthority("my-namespace.my-account.tmprl.cloud")) .build()); ``` **TypeScript** ```ts const connection = await NativeConnection.connect({ address: "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233", tls: { serverNameOverride: "my-namespace.my-account.tmprl.cloud", //serverRootCACertificate, // See docs for other TLS options clientCertPair: { crt: fs.readFileSync(clientCertPath), key: fs.readFileSync(clientKeyPath), }, }, }); ``` **Python** ```python client_config["tls"] = TLSConfig( client_cert=bytes(crt, "utf-8"), client_private_key=bytes(key, "utf-8"), domain="my-namespace.my-account.tmprl.cloud", ) client = await Client.connect("vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233") ``` **.NET** ```dotnet // Create client var client = await TemporalClient.ConnectAsync( new(ctx.ParseResult.GetValueForOption(targetHostOption)!) { Namespace = ctx.ParseResult.GetValueForOption (namespaceOption)!, // Set TLS options with client certs. Note, more options could // be added here for server CA (i.e. "ServerRootCACert") or SNI // override (i.e. "Domain") for self-hosted environments with // self-signed certificates. Tls = new() { ClientCert = await File.ReadAllBytesAsync(ctx.ParseResult.GetValueForOption(clientCertOption) !.FullName), ClientPrivateKey = await File.ReadAllBytesAsync(ctx.ParseResult.GetValueFor0ption(clientKey0ption)!.FullName), Domain = "my-namespace.my-account.tmprl.cloud", }, }); // dotnet run --target-host "vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233" ``` To check whether your client has network connectivity to the private endpoint in question, run: ```bash nc -zv vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233 ``` ## Control plane connectivity Using the Temporal Cloud [web UI](/web-ui), [Terraform provider](/cloud/terraform-provider), [`tcld` CLI](/cloud/tcld), or [Cloud Ops APIs](/ops) requires network access to the Temporal Cloud Control Plane. ### Control plane hostnames Different hostnames are used for different parts of the service. - `saas-api.tmprl.cloud` (required for Terraform, tcld, and Cloud Ops APIs) - `web.onboarding.tmprl.cloud` (required for Web UI) - `web.saas-api.tmprl.cloud` (required for Web UI) ### AWS PrivateLink connectivity to Temporal Cloud Control Plane Temporal Cloud supports [AWS PrivateLink](https://aws.amazon.com/privatelink/) connections to the Control Plane, which allows access from applications running in VPCs that cannot egress to the public internet. Temporal Cloud does **not** support restricting an account so that private connectivity is the sole connectivity method to the Control Plane; the Control Plane is always accessible via public internet. Control Plane access is always securely authenticated via [API keys](/cloud/api-keys#overview) or JWT tokens, regardless of how you choose to connect. To set up a PrivateLink connection to the Temporal Cloud Control Plane, follow [these instructions](/cloud/connectivity/aws-connectivity), but use the Control Plane endpoint information below: | Hostname | Region | Control Plane PrivateLink Service Name | | ---------------------- | ----------- | --------------------------------------------------------- | | `saas-api.tmprl.cloud` | `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0c57a5930b6f6be0e` | The Control Plane PrivateLink endpoint includes a [private DNS name](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html), which lets your clients use the PrivateLink connection without having to set up private DNS or having to override client configuration. To use the DNS name, make sure your VPC has the `Enable DNS hostnames` and `Enable DNS support` options enabled. If you cannot use the DNS name, you can also manually [set up private DNS](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink) or [override the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity). > **⚠️ Caution:** > Finish client setup to programmatically access Temporal Cloud Control Plane over PrivateLink > > For Temporal clients to access the Control Plane over AWS PrivateLink, you must either use the provided DNS name, set up private DNS, or update the client configuration, as described above. > > Without this step, your clients may connect to the Temporal Cloud Control Plane over the internet, or they may not connect at all. #### Extend the Control Plane PrivateLink endpoint to other AWS regions with VPC Peering The Temporal Cloud Control Plane PrivateLink Service is only exposed in `us-west-2`. To reach the Control Plane privately from another AWS region, set up a us-west-2 VPC with a VPC Endpoint to the Control Plane, then [VPC-Peer](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) that VPC to your other-region VPCs. To set this up: 1. Create a VPC in `us-west-2`. 2. Create the [VPC Endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html) to the Control Plane in that us-west-2 VPC, using the service name from the table above. Follow the [standard PrivateLink setup steps](/cloud/connectivity/aws-connectivity), but use the Control Plane service name. 3. Peer the us-west-2 VPC to each VPC in another region that needs Control Plane access. [VPC Peering shares VPC Endpoints](https://docs.aws.amazon.com/vpc/latest/peering/peering-configurations-partial-access.html) across peered VPCs, which gives each peered VPC a private path to the Control Plane. Peering is per-VPC, so you must peer the us-west-2 VPC to every VPC that needs Control Plane access. There is no limit on how many regions you can extend to using this approach. --- # AWS PrivateLink connectivity Source: https://docs.temporal.io/cloud/connectivity/aws-connectivity > Connect to Temporal Cloud using AWS PrivateLink [AWS PrivateLink](https://aws.amazon.com/privatelink/) allows you to open a path to Temporal without opening a public egress. It establishes a private connection between your Amazon Virtual Private Cloud (VPC) and Temporal Cloud. This one-way connection means Temporal cannot establish a connection back to your service. This is useful if normally you block traffic egress as part of your security protocols. If you use a private environment that does not allow external connectivity, you will remain isolated. After creating the PrivateLink endpoint, configure your clients to use it through either [private DNS](#configuring-private-dns-for-aws-privatelink) or [direct VPCE targeting](#direct-vpce). Direct VPCE targeting is simplest for single-region Namespaces, but also works for High Availability Namespaces with [more careful setup](#direct-vpce). ## Requirements * Your AWS PrivateLink (PL) endpoint must be in the same region as your Temporal Cloud Namespace or one of its [High Availability](/cloud/high-availability) replicas. See [cross-region PrivateLink connectivity](#cross-region-privatelink) to access the Namespace from a different region. * Your Private DNS must be configured to direct Worker / Client traffic to your VPC Endpoint, as described below. * If the Worker / Client does not use the Namespace Endpoint as the connection string in its code, it may need to set the `server_name` config to the Namespace Endpoint string, as described below. ### Cross-region PrivateLink Connectivity Temporal Cloud does **not** support [cross-region connectivity for AWS PrivateLink](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cross-region-connectivity-for-aws-privatelink/) out of the box. However, if you need to reach Temporal Cloud privately from a different region than your Namespace, you can route traffic to your VPC Endpoint in the Namespace's region using [AWS's native cross-region networking features](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/centralized-access-to-vpc-private-endpoints.html#cross-region-endpoint-access). When using High Availability on Temporal Cloud, it's best practice to have two VPC Endpoints, one in each of the Namespace's regions, to ensure at least one VPC Endpoint is accessible during a regional outage. ## Creating an AWS PrivateLink connection Set up PrivateLink connectivity with Temporal Cloud with these steps: 1. Open the AWS console with the region you want to use to establish the PrivateLink. 2. Search for "VPC" in _Services_ and select the option. ![AWS console showing services, features, resources](/img/cloud/privatelink/aws-console.png) 3. Select _Virtual private cloud_ > _Endpoints_ from the left menu bar. 4. Click the _Create endpoint_ button to the right of the _Actions_ pulldown menu. 5. Under _Type_ category, select _Endpoint services that use NLBs and GWLBs_. This option lets you find services shared with you by service name. 6. Under _Service settings_, fill in the _Service name_ with the PrivateLink Service Name for the region you’re trying to connect from: > **💡 Tip:** > > PrivateLink endpoint services are regional. > Individual Namespaces do not use separate services. > | Region | PrivateLink Service Name | DNS Record Override | | --- | --- | --- | | ap-northeast-1 | com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a | | ap-northeast-2 | com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308 | | ap-south-1 | com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662 | | ap-south-2 | com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1 | | ap-southeast-1 | com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd | | ap-southeast-2 | com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08 | | ca-central-1 | com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d | | eu-central-1 | com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3 | | eu-west-1 | com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739 | | eu-west-2 | com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695 | | sa-east-1 | com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a | | us-east-1 | com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f | | us-east-2 | com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4 | | us-west-2 | com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94 | 7. Confirm your service by clicking on the _Verify service_ button. AWS should respond "Service name verified." ![The service name field is filled out and the Verify service button is shown](/img/cloud/privatelink/service-settings.png) 8. Select the VPC and subnets to peer with the Temporal Cloud service endpoint. 9. Select the security group that will control traffic sources for this VPC endpoint. The security group must accept TCP ingress traffic to port 7233 for gRPC communication with Temporal Cloud. 10. Click the _Create endpoint_ button at the bottom of the screen. If successful, AWS reports "Successfully created VPC endpoint." and lists the new endpoint. The new endpoint appears in the Endpoints list, along with its ID. ![The created endpoint appears in the Endpoints list](/img/cloud/privatelink/endpoint-created.png) 11. Click on the VPC endpoint ID in the Endpoints list to check its status. Wait for the status to be “Available”. This can take up to 10 minutes. 12. Once the status is "Available", the AWS PrivateLink is ready for use. ![Highlighted DNS names section shows your hostname](/img/cloud/privatelink/details.png) The next step is to [configure private DNS](#configuring-private-dns-for-aws-privatelink) so your clients can use the PrivateLink connection. For single-region Namespaces that don't need per-Namespace DNS records, you can use [direct VPCE targeting](#direct-vpce) instead. ## Configuring Private DNS for AWS PrivateLink ### Why configure private DNS? When you connect to Temporal Cloud through AWS PrivateLink you normally must: 1. **Point your SDKs/Workers at the PrivateLink DNS name** for the VPC Endpoint (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com`), **and** 2. **Override the Server Name Indicator (SNI)** so that the TLS handshake still presents the public Temporal Cloud hostname (for example, `my-namespace.my-account.tmprl.cloud`). By creating a Route 53 **private hosted zone (PHZ)** that maps the public Temporal Cloud hostname (or region hostname) to your VPC Endpoint, you can: - Keep using the standard Temporal Cloud hostnames in code and configuration. - Eliminate the need to set a custom SNI override. - Make future Endpoint rotations transparent—only the PHZ record changes. This approach is **optional**; Temporal Cloud works without it. It simply streamlines configuration and operations. If you cannot use private DNS, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity). ### Prerequisites | Requirement | Notes | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | AWS VPC with DNS resolution and DNS hostnames enabled | _VPC console → Edit DNS settings → enable both checkboxes._ | | Interface VPC Endpoint for Temporal Cloud | Subnets must be associated with the VPC and Security Group must allow TCP ingress traffic to port 7233 from the appropriate hosts. | | Route 53 available in your AWS account | You need permission to create Private Hosted Zones and records. | | Namespace details | Needed to choose the correct override domain pattern below. | ### Choose the override domain and endpoint | Endpoint type | PHZ domain format | Example | Use when | | ------------------ | ---------------------------------- | ------------------------------------ | -------- | | Namespace endpoint | `.tmprl.cloud` | `payments.abcde.tmprl.cloud` | **Single-region Namespaces.** Simplest pattern — one record per Namespace. For [High Availability](/cloud/high-availability/ha-connectivity) Namespaces, overriding the Namespace Endpoint is nuanced — see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity). | | Regional endpoint | `..api.temporal.io` | `ap-northeast-2.aws.api.temporal.io` | You want to pin a client to a specific Temporal Cloud region. | > **⚠️ Caution:** > HA Namespaces need a more nuanced PHZ setup > > For Namespaces with [High Availability](/cloud/high-availability/ha-connectivity), the PHZ pattern to use depends on how you want Workers to reach the active region. Overriding the Namespace Endpoint directly is read out of the PHZ before public DNS, so the regional CNAME that Temporal Cloud rewrites on failover isn't followed — which is usually not what you want, but can be the right choice in some topologies (for example, multi-cloud HA with one region per cloud, where Workers on each cloud should always reach their local region). Because the trade-offs depend on your setup, see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity) before choosing a pattern. > The step-by-step below walks through the **Namespace endpoint** pattern, which is the simpler single-region case. For HA, follow the [HA Connectivity guide](/cloud/high-availability/ha-connectivity) instead, which uses the same Route 53 mechanics but on the regional records. ### Step-by-step instructions > **⚠️ Warning:** > Order matters > > A Route 53 private hosted zone with no records causes DNS resolution to fail (NXDOMAIN) inside any associated VPC. If you create an empty PHZ for `.tmprl.cloud` and associate it with a VPC where Workers are running, **all Worker traffic to Temporal Cloud in that VPC stops** until you add the CNAME record. Follow the steps below in order to avoid this. > #### 1. Collect your PrivateLink endpoint DNS name ```bash aws ec2 describe-vpc-endpoints \ --vpc-endpoint-ids $VPC_ENDPOINT_ID \ --query "VpcEndpoints[0].DnsEntries[0].DnsName" \ --output text # Example output: # vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com ``` Save the **`vpce-*.amazonaws.com`** value — you will target it in the CNAME record. #### 2. Create a Route 53 Private Hosted Zone (do not yet attach Worker VPCs) a. Open _Route 53 → Hosted zones → Create hosted zone_. b. Enter the domain chosen from the table above, for example, `payments.abcde.tmprl.cloud`. c. Type: _Private hosted zone for Temporal Cloud_. d. Leave VPC associations empty for now (you'll add them in step 4). e. Create the hosted zone. #### 3. Add a CNAME record Inside the new PHZ: | Field | Value | | --------------- | ------------------------------------------------------------------------------------- | | **Record name** | the Namespace Endpoint (for example, `payments.abcde.tmprl.cloud`). | | **Record type** | `CNAME` | | **Value** | Your VPC Endpoint DNS name (`vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com`) | | **TTL** | 60s is typical; 15s for Namespaces with High Availability (to minimize recovery time after failover). | #### 4. Associate the PHZ with your Worker VPCs and verify Now that the record exists, associate the PHZ with every VPC that contains Temporal Workers or SDK clients (Route 53 → your zone → _Edit settings_ → _Add VPC_). > **💡 Tip:** > Test with a non-production VPC first > > We strongly recommend that you test with a non-production VPC first. Attach the PHZ to a non-production VPC, validate end-to-end resolution and connectivity from a host in that VPC, and only then attach production Worker VPCs. This catches misconfigured records before they affect production traffic. > Verify DNS resolution from inside one of the associated VPCs: ```bash dig payments.abcde.tmprl.cloud ``` If the record resolves to the VPC Endpoint, you are ready to use Temporal Cloud without SNI overrides. ### Updating your workers/clients With private DNS in place, configure your SDKs exactly as the public-internet examples show (filling in your own namespace): ```go clientOptions := client.Options{ HostPort: "payments.abcde.tmprl.cloud:7233", Namespace: "payments", // No TLS SNI override needed } ``` The DNS resolver inside your VPC returns the private endpoint, while TLS still validates the original hostname—simplifying both code and certificate management. ## Configure private DNS for Namespaces with High Availability For Namespaces with [High Availability features](/cloud/high-availability), you need to override DNS for `region.tmprl.cloud` so each region resolves to the local VPC Endpoint, and you need to ensure Workers can reach whichever region is active. Failover is transparent to clients only when this is set up correctly. The complete guidance — including single-cloud (AWS-only) HA, multi-cloud HA (AWS PrivateLink + GCP Private Service Connect), and a recommended failover-testing plan — lives on a single page: [Connectivity for High Availability](/cloud/high-availability/ha-connectivity). ## Direct VPCE targeting without per-Namespace DNS You can avoid creating DNS records for each Namespace by pointing Workers directly at the VPC Endpoint and overriding the TLS Server Name Indicator (SNI): 1. Create the PrivateLink VPC Endpoint (one per region — all Namespaces in that region share it). 2. Configure each Worker with: - **Endpoint**: the DNS name of the VPC Endpoint in the region where the Worker runs (for example, `vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233`) - **Server name** (SNI override): the Namespace Endpoint value (for example, `my-namespace.my-account.tmprl.cloud`) With this approach, new Namespaces do not require new DNS records. Workers set their **Endpoint** to the VPC Endpoint DNS name and their **Server name** to the Namespace Endpoint, so the client SDK accepts the TLS handshake from Temporal Cloud. **Single-region Namespace** ```mermaid --- title: Direct VPCE targeting (single region) --- %%{init: {'themeVariables':{'fontFamily':'Inter, ui-sans-serif, system-ui, sans-serif'},'flowchart':{'nodeSpacing':18,'rankSpacing':45,'curve':'basis','subGraphTitleMargin':{'top':6,'bottom':12}}}}%% flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; subgraph SREG["Your VPC"] SW["Worker(s)
Server name:
my-namespace.my-account.tmprl.cloud"]:::worker SVPCE["VPC Endpoint
vpce-…"]:::endpoint SW --> SVPCE end SNS["Namespace
(Temporal Cloud)"]:::ns SVPCE -->|PrivateLink| SNS class SREG region ``` > **📝 Note:** > Using direct VPCE targeting with High Availability Namespaces > > Direct VPCE targeting also works for Namespaces with [High Availability features](/cloud/high-availability), but it takes more careful setup. Because each VPC Endpoint is pinned to its region and does not follow Temporal Cloud's active-region CNAME on failover, you cannot rely on DNS to move Workers between regions. This is the same trade-off as targeting a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) instead of the Namespace Endpoint. > **Namespace with High Availability** To use the Direct VPCE approach with a Namespace that has High Availability: - Point each region's Workers at the VPC Endpoint **local to that region** — the Endpoint value differs per region. - Set the **Server name** (SNI override) to the Namespace Endpoint value in every region — it is the same everywhere. - **(Recommended)** To stay available during a failover, run Workers in every region the Namespace can be active in. Workers connected to the passive region's VPC Endpoint stay productive: Temporal Cloud forwards their tasks to the active region (you can [configure this forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior)). On failover, no DNS change is needed, because Workers are already connected in both regions and the surviving region takes over. ```mermaid --- title: Direct VPCE targeting (High Availability) --- %%{init: {'themeVariables':{'fontFamily':'Inter, ui-sans-serif, system-ui, sans-serif'},'flowchart':{'nodeSpacing':55,'rankSpacing':70,'curve':'basis','subGraphTitleMargin':{'top':6,'bottom':12}}}}%% flowchart TD classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; subgraph HSEC["us-west-2 (passive)"] HWB["Worker(s)
Server name
namespace.
acct.
tmprl.cloud"]:::worker HVPB["VPC
Endpoint
vpce-…
us-west-2"]:::endpoint HNSB["Replica
(Passive,
forwards
to active)"]:::ns HWB --> HVPB HVPB --> HNSB end subgraph HPRIM["us-east-1 (active)"] HWA["Worker(s)
Server name
namespace.
acct.
tmprl.cloud"]:::worker HVPA["VPC
Endpoint
vpce-…
us-east-1"]:::endpoint HNSA["Namespace
(Active)"]:::ns HWA --> HVPA HVPA --> HNSA end class HPRIM,HSEC region ``` ## Adding PrivateLink from additional AWS accounts A common pattern is to have separate AWS accounts for different lines of business, environments (staging, production), or compliance scopes (PCI vs non-PCI), each with its own VPC and Workers connecting to the same Temporal Cloud account. You can create as many AWS PrivateLink VPC endpoints as you need to the same Temporal Cloud regional service — there is nothing to register, approve, or open a ticket for on the Temporal side. For each additional AWS account or VPC: 1. In that account, create the AWS PrivateLink VPC endpoint targeting the regional service name from the [regions table](#available-aws-regions-privatelink-endpoints-and-dns-record-overrides) — same as in the [creation steps](#creating-an-aws-privatelink-connection) above. 2. Configure DNS in that VPC. You have two options: - Create a Route 53 Private Hosted Zone in that account scoped to the appropriate VPC(s), following the [private DNS steps](#configuring-private-dns-for-aws-privatelink) above. Each VPC's PHZ should point at the VPC Endpoint local to that VPC. - Or, use [direct VPCE targeting](#direct-vpce). For High Availability Namespaces, follow the [additional setup](#direct-vpce) so each region's Workers target their local VPC Endpoint. 3. **Optional:** if you want to enforce private-only access for a Namespace, add a Connectivity Rule for each VPC endpoint and attach all of them (plus a public rule, if needed) to the Namespace. See [Connectivity Rules](/cloud/connectivity#connectivity-rules). There is no upper limit on the number of VPC endpoints you can connect from your side to a regional PrivateLink service. The default per-account limit on private Connectivity Rules is 50 — [contact support](/cloud/support#support-ticket) if you need to raise it. ## Available AWS regions, PrivateLink endpoints, and DNS record overrides The following table lists the available Temporal regions, PrivateLink endpoints, and regional endpoints used for DNS record overrides: ### Asia Pacific - Tokyo (`ap-northeast-1`) - **Cloud API Code**: `aws-ap-northeast-1` - **Regional Endpoint**: `ap-northeast-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Seoul (`ap-northeast-2`) - **Cloud API Code**: `aws-ap-northeast-2` - **Regional Endpoint**: `ap-northeast-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Mumbai (`ap-south-1`) - **Cloud API Code**: `aws-ap-south-1` - **Regional Endpoint**: `ap-south-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Hyderabad (`ap-south-2`) - **Cloud API Code**: `aws-ap-south-2` - **Regional Endpoint**: `ap-south-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Singapore (`ap-southeast-1`) - **Cloud API Code**: `aws-ap-southeast-1` - **Regional Endpoint**: `ap-southeast-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Sydney (`ap-southeast-2`) - **Cloud API Code**: `aws-ap-southeast-2` - **Regional Endpoint**: `ap-southeast-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Europe - Frankfurt (`eu-central-1`) - **Cloud API Code**: `aws-eu-central-1` - **Regional Endpoint**: `eu-central-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-west-1` - `aws-eu-west-2` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### Europe - Ireland (`eu-west-1`) - **Cloud API Code**: `aws-eu-west-1` - **Regional Endpoint**: `eu-west-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-central-1` - `aws-eu-west-2` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### Europe - London (`eu-west-2`) - **Cloud API Code**: `aws-eu-west-2` - **Regional Endpoint**: `eu-west-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-central-1` - `aws-eu-west-1` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### North America - Central Canada (`ca-central-1`) - **Cloud API Code**: `aws-ca-central-1` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` - **Regional Endpoint**: `ca-central-1.aws.api.temporal.io:7233` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Northern Virginia (`us-east-1`) - **Cloud API Code**: `aws-us-east-1` - **Regional Endpoint**: `us-east-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-2` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Ohio (`us-east-2`) - **Cloud API Code**: `aws-us-east-2` - **Regional Endpoint**: `us-east-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Oregon (`us-west-2`) - **Cloud API Code**: `aws-us-west-2` - **Regional Endpoint**: `us-west-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### South America - São Paulo (`sa-east-1`) - **Cloud API Code**: `aws-sa-east-1` - **Regional Endpoint**: `sa-east-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - None --- # Google Private Service Connect connectivity Source: https://docs.temporal.io/cloud/connectivity/gcp-connectivity > Connect to Temporal Cloud using Google Private Services Connect [Google Cloud Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) allows you to open a path to Temporal without opening a public egress. It establishes a private connection between your Google Virtual Private Cloud (VPC) and Temporal Cloud. This one-way connection means Temporal cannot establish a connection back to your service. This is useful if normally you block traffic egress as part of your security protocols. If you use a private environment that does not allow external connectivity, you will remain isolated. > **⚠️ Warning:** > Namespaces with High Availability features and GCP Private Service Connect > > Automatic failover via Temporal Cloud DNS is not currently supported with GCP Private Service Connect. > If you use GCP Private Service Connect, you must manually update your workers to point to the active region's Private Service Connect endpoint when a failover occurs. > ## Requirements * Your GCP Private Service Connect endpoint must be in the same region as your Temporal Cloud namespace. If using [replication for High Availability](/cloud/high-availability), the PSC connection must be in the same region as one of the replicas. * Your Private DNS must be configured to direct Worker / Client traffic to your PSC endpoint, as described below. * If the Worker / Client is not using the Namespace Endpoint, it may need to set the `server_name` config to the Namespace Endpoint string, as described below. ## Creating a Private Service Connect connection Set up Private Service Connect with Temporal Cloud with these steps: 1. Open the Google Cloud console 2. Navigate to **Network Services**, then **Private Service Connect**. If you haven't used **Network Services** recently, you might have to find it by clicking on **View All Products** at the bottom of the left sidebar. ![GCP console showing Network Services, and the View All Products button](/img/cloud/gcp/gcp-console.png) 3. Go to the **Endpoints** section. Click on **Connect endpoint**. ![GCP console showing the endpoints, and the Connect endpoint button](/img/cloud/gcp/connect-endpoint-button.png) 4. Under **Target**, select **Published service**, this will change the contents of the form to allow you to fill the rest as described below ![GCP console showing the endpoints, and the Connect endpoint button](/img/cloud/gcp/connect-endpoint.png) - For **Target service**, fill in the **Service name** with the Private Service Connect Service Name for the region you’re trying to connect to: > **💡 Tip:** > > GCP Private Service Connect services are regional. > Individual Namespaces do not use separate services. > | Region | Private Service Connect Service Name | | --- | --- | | asia-south1 | projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw | | europe-west3 | projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh | | us-central1 | projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng | | us-east4 | projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy | | us-west1 | projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x | - For **Endpoint name**, enter a unique identifier to use for this endpoint. It could be for instance `temporal-api` or `temporal-api-` if you want a different endpoint per namespace. - For **Network** and **Subnetwork**, choose the network and subnetwork where you want to publish your endpoint. - For **IP address**, click the dropdown and select **Create IP address** to create an internal IP from your subnet dedicated to the endpoint. Select this IP. - Check **Enable global access** if you intend to connect the endpoint to virtual machines outside of the selected region. We recommend regional connectivity instead of global access, as it can be better in terms of latency for your workers. _**Note:** this requires the network routing mode to be set to **GLOBAL**._ 5. Click the **Add endpoint** button at the bottom of the screen. The endpoint will appear with status **Pending**. This is expected — the next step is what flips it to **Accepted**. 6. [Create a Temporal Cloud Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) using the Connection ID of the newly created endpoint and the corresponding GCP project. Use the **Connection ID** from the endpoint's detail page in the Google Cloud console (a numeric string such as `1234567890123456789`). 7. Once the status changes from "Pending" to "Accepted", the GCP Private Service Connect endpoint is ready for use. > **⚠️ Warning:** > PSC stays "Pending" until you create a Connectivity Rule > > For GCP Private Service Connect, the Connectivity Rule is what tells Temporal Cloud to accept your PSC connection. Until you [create a Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) for the connection, the endpoint will remain in **Pending**. There is no separate producer-side approval step — creating the Connectivity Rule is the approval. > > If your endpoint is stuck Pending, the most common causes are: > > - No Connectivity Rule exists for the connection ID. (Most common.) > - The Connectivity Rule was created with the wrong `connection-id`, `region`, or `gcp-project-id`. > - The endpoint is in a region that is not a [supported Temporal Cloud region](/cloud/regions). > - Take note of the **IP address** assigned to your endpoint — you will use it to connect to Temporal Cloud. > **⚠️ Caution:** > You still need to set up private DNS or override client configuration for your clients to actually use the new Private Service Connect connection to connect to Temporal Cloud. > > See [configuring private DNS for GCP Private Service Connect](#configuring-private-dns-for-gcp-private-service-connect) ## Configuring Private DNS for GCP Private Service Connect ### Why configure private DNS? When you connect to Temporal Cloud through GCP Private Service Connect you normally must: 1. **Point your SDKs/Workers at the Private Service Connect endpoint IP address** _and_ 2. **Override the Server Name Indicator (SNI)** so that the TLS handshake still presents the public Temporal Cloud hostname (for example, `my-namespace.my-account.tmprl.cloud`). By creating a **private Cloud DNS zone (PZ)** that maps the public Temporal Cloud hostname (or the region hostname) directly to the PSC endpoint IP address, you can: - Keep using the standard Temporal Cloud hostnames in code and configuration. - Eliminate the need to set a custom SNI override. - Make future endpoint rotations transparent—only the DNS record changes. This approach is **optional**; Temporal Cloud works without it. It simply streamlines configuration and operations. If you cannot use private DNS, refer to [our guide for updating the server and TLS settings on your clients](/cloud/connectivity#update-dns-or-clients-to-use-private-connectivity). ### Prerequisites | Requirement | Notes | | ----------------------------------------------------- | --------------------------------------------------------------------------------- | | Google Cloud VPC Network with DNS enabled | PSC endpoints and the DNS zone must live in (or be attached to) the same network. | | Private Service Connect endpoint for Temporal Cloud | Create an endpoint and reserve an internal IP in the namespace region | | Cloud DNS API enabled and roles/dns.admin permissions | Needed to create private zones and records. | | Namespace details | Determines which hostname pattern you override (table below). | ### Choose the override domain and endpoint | Temporal Cloud setup | Use this PHZ domain | Example | | ------------------------------------------ | ---------------------------------- | ---------------------------------------------- | | Single-region namespace with mTLS auth | `.tmprl.cloud` | `payments.abcde.tmprl.cloud` ↔️ `X.X.X.X` | | Single-region namespace with API-key auth | `.api.temporal.io` | `us-central1.gcp.api.temporal.io` ↔️ `X.X.X.X` | | Multi-region namespace | `.api.temporal.io` | `us-central1.gcp.api.temporal.io` ↔️ `X.X.X.X` | ### Step-by-step instructions #### 1. Collect your PSC endpoint IP address ```shell # List the forwarding rule you created for the endpoint gcloud compute forwarding-rules list \ --filter="NAME:" \ --format="value(IP_ADDRESS)" # Example output: 10.1.2.3 ``` Save the internal IP -- you will point the A record at it. #### 2. Create a Cloud DNS private zone 1. Open _Network Services → Cloud DNS → Create zone_. 2. Select zone type **Private**. 3. Enter a **Zone name** (for example, `temporal-cloud`). 4. Enter a **DNS name** based on the table above (for example, `payments.abcde.tmprl.cloud` or `us-east-1.aws.api.temporal.io`). 5. Select **Add networks** and choose the Project and Network that contains your PSC endpoint. 6. Click **Create**. #### 3. Add an A record Inside the new zone, add a _standard A record_: | Field | Value | | -------------------- | -------------------------------------------------------------- | | DNS name | the namespace endpoint (for example, `payments.abcde.tmprl.cloud`) | | Resource record type | A | | TTL | 60s is typical, but you can adjust as needed. | | IPv4 Address | the internal IP address of your PSC endpoint (for example, `10.1.2.3`) | #### 4. Verify DNS resolution from inside the Network ```shell dig payments.abcde.tmprl.cloud ``` If the hostname resolves to the PSC endpoint IP address from a VM in the bound network, the override is working. ### Updating your workers/clients With private DNS in place, configure your SDKs exactly as the public-internet examples show (filling in your own namespace): ```go clientOptions := client.Options{ HostPort: "payments.abcde.tmprl.cloud:7233", Namespace: "payments", // No TLS SNI override needed } ``` The DNS resolver inside your network returns the private endpoint IP address, while TLS still validates the original hostname—simplifying both code and certificate management. ## Available GCP regions, PSC endpoints, and DNS record overrides The following table lists the available Temporal regions, PrivateLink endpoints, and regional endpoints used for DNS record overrides: ### North America - Iowa (`us-central1`) - **Cloud API Code**: `gcp-us-central1` - **Regional Endpoint**: `us-central1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-west1` - `gcp-us-east4` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### North America - Oregon (`us-west1`) - **Cloud API Code**: `gcp-us-west1` - **Regional Endpoint**: `us-west1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-central1` - `gcp-us-east4` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### North America - Northern Virginia (`us-east4`) - **Cloud API Code**: `gcp-us-east4` - **Regional Endpoint**: `us-east4.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-central1` - `gcp-us-west1` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### Europe - Frankfurt (`europe-west3`) - **Cloud API Code**: `gcp-europe-west3` - **Regional Endpoint**: `europe-west3.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - `aws-eu-central-1` - `aws-eu-west-1` - `aws-eu-west-2` ### Asia Pacific - Mumbai (`asia-south1`) - **Cloud API Code**: `gcp-asia-south1` - **Regional Endpoint**: `asia-south1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` ### Asia Pacific - Jakarta (`asia-southeast2`) - **Cloud API Code**: `gcp-asia-southeast2` - **Regional Endpoint**: `gcp-asia-southeast2.region.tmprl.cloud` - **Private Service Connect Service Attachment URI**: `projects/prod-bsbyrfwqqq885qkcr3s43y524/regions/asia-southeast2/serviceAttachments/pl-c3ayi` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - None --- # Temporal Cloud IP addresses Source: https://docs.temporal.io/cloud/connectivity/ip-addresses > Temporal Cloud assigns dynamic IPs by default; enable Stable IPs for a fixed, published range you can allowlist in firewalls. By default, Temporal Cloud resources use dynamic IP addresses that may change at any time. These IP addresses may be any IPs within the IP ranges published by the relevant cloud provider. If you need to limit outbound access from your client network, we recommend using [AWS PrivateLink or GCP Private Services Connect](/cloud/connectivity#private-network-connectivity-for-namespaces). Alternatively, you can enable [Stable IPs](#stable-ip-addresses) for your Namespace. > **⚠️ Warning:** > Do not rely on default dynamic public IP addresses > > By default, Temporal Cloud IPs are not static and may change without notice. If you need fixed, public IP addresses for your Namespace without using [private connectivity](/cloud/connectivity#private-network-connectivity-for-namespaces), enable [Stable IPs](#stable-ip-addresses) for Namespace Endpoints. > ## When Namespace Endpoint DNS resolution is predictable In general, do not take dependencies on what Temporal Cloud endpoints resolve to over DNS. The resolved values are subject to change without notice, and Workers, Temporal Clients, and firewalls that target the specific IPs provided at a specific point in time will eventually fail. Temporal Cloud guarantees the DNS resolution behavior of the Namespace Endpoint in two cases: 1. **Stable IPs enabled.** The Namespace Endpoint resolves to one of the fixed IP ranges for the Namespace's active region, so it is safe to allowlist these IP ranges in a firewall. See [Stable IP addresses](#stable-ip-addresses). 2. **High Availability features with Private Connectivity.** When High Availability and Private Connectivity Rules are both enabled on a Namespace, the Namespace Endpoint is guaranteed to resolve via CNAME to `-.region.tmprl.cloud` for the active region. See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity) for recommendations to configure private DNS. Note: Do not attach a public connectivity rule with Stable IPs to the Namespace, as that supersedes this behavior. For all other connectivity patterns, treat the resolved IP as temporary and use one of the two approaches above, or allowlist the entire cloud provider IP range. ## Stable IP addresses Stable IPs is an optional Namespace setting that provides a fixed, published set of IP addresses for your Namespace endpoint. When enabled, Workers and Temporal Clients connecting to your Namespace through its Namespace endpoint will resolve to a predictable set of IP addresses that you can allowlist in your firewall. ### When to use Stable IPs Use Stable IPs when your security requirements mandate IP-based allowlisting for egress traffic, and you cannot use AWS PrivateLink or GCP Private Services Connect. **Recommended options for secure connectivity** (in order): 1. **AWS PrivateLink or GCP Private Services Connect** - Keep traffic within your cloud provider's private network 2. **Stable IPs** - Use published, stable IP ranges for firewall allowlisting 3. **Cloud provider IP ranges** - Allowlist all AWS or GCP IP ranges (overly permissive) Stable IPs provide enterprise-grade compliance for organizations that require IP-based security controls but cannot use private network connectivity options. ### How stable IPs work When Stable IPs are enabled on a Namespace: - The Namespace endpoint (`..tmprl.cloud`) resolves to IP addresses from a published list of Stable IPs. - The list is grouped by cloud region. The Namespace endpoint will resolve to an IP from the group for the Namespace's active region. - Workers and Temporal Clients connecting via the Namespace endpoint will connect to these stable IP addresses. - Regional endpoints continue to work but resolve to dynamic IP addresses. - For replicated Namespaces, the Stable IP corresponds to the Namespace's active region. During failover to a different region, DNS updates to point to a Stable IP in the new active region. > **📝 Note:** > > Stable IPs apply only to Namespace traffic, sometimes called the "data plane." The Temporal Cloud management plane, observability endpoints, and Web UI do not have stable public IP addresses. > > **⚠️ Warning:** > Stable IPs supersedes HA + Private Connectivity DNS > > If you attach a public Connectivity Rule with Stable IPs to a Namespace that is also configured for [High Availability with Private Connectivity](/cloud/high-availability/ha-connectivity), the Namespace Endpoint resolves to a public Stable IP instead of to `-.region.tmprl.cloud`. Stable IPs DNS behavior supersedes the regional DNS behavior that HA + Private Connectivity relies on, so the Namespace Endpoint's DNS resolution will not work in the way the Private Hosted Zone needs. Do not attach a Stable IPs public Connectivity Rule to a Namespace where you want HA + Private Connectivity to keep routing traffic over PrivateLink / PSC. > ### How to enable Stable IPs Stable IPs is a setting on a public [Connectivity Rule](/cloud/connectivity#connectivity-rules). You enable it by creating (or updating) a public Connectivity Rule with Stable IPs enabled, then attaching that rule to the Namespaces that should resolve to Stable IPs. > **📝 Note:** > Create Stable IPs rules with the Temporal CLI > > Use the [Temporal CLI](/cli/cloud) to create a public Connectivity Rule with Stable IPs enabled: > > ```command > temporal cloud connectivity public create --enable-stable-ips > ``` > > `tcld` does not support creating this rule. You can also use the [Cloud Ops API](/ops) directly (HTTP or gRPC at > `saas-api.tmprl.cloud`) or the generated client in the [`temporalio/cloud-api`](https://github.com/temporalio/cloud-api) > repository. > > After the rule exists, attach it to a Namespace with [`temporal cloud namespace connectivity attach`](/cli/command-reference/cloud/namespace#connectivity-attach). There is no Temporal Cloud UI option at this time. > To enable Stable IPs: 1. Create a public Connectivity Rule with Stable IPs enabled: `temporal cloud connectivity public create --enable-stable-ips`. Only one public Connectivity Rule exists per account, so if you already have one without Stable IPs enabled, delete and recreate it. See [How to enable Stable IPs with curl](#how-to-enable-stable-ips-with-curl) for a Cloud Ops API alternative. 2. Attach the rule to your Namespace: `temporal cloud namespace connectivity attach --namespace . --connectivity-rule-id `. 3. Retrieve the list of Stable IPs from the [public JSON file](/cloud/connectivity/ip-addresses#how-to-view-stable-ip-ranges). 4. Configure your firewall to allowlist the Stable IP ranges for your Namespace's region. 5. Ensure your Workers and Temporal Clients connect using the Namespace endpoint, not regional endpoints. You can enable Stable IPs on new Namespaces at creation time or on existing Namespaces. > **📝 Note:** > DNS propagation > > When enabling Stable IPs on an existing Namespace, Temporal updates DNS records to point the Namespace endpoint to Stable IPs. DNS propagation typically takes 5 to 15 minutes for AWS, but can take up to 60 minutes or longer if you have custom TTLs configured. > #### How to enable Stable IPs with curl If you cannot use the Temporal CLI or Terraform, you can access the Cloud Ops API directly. The following procedure creates a public Connectivity Rule with Stable IPs enabled and attaches it to a Namespace. All requests require an authorization header: - `Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY` Base URL: `https://saas-api.tmprl.cloud` Set up environment variables: ```bash export TEMPORAL_CLOUD_OPS_API_KEY='' export NS='.' # The Cloud-side Namespace identifier, e.g. "myns.a1b2c3" H_AUTH="Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" ``` `NS` is the Cloud-side Namespace identifier, not the SDK gRPC hostname (`myns.a1b2c3.tmprl.cloud`). Strip the trailing `.tmprl.cloud`. **Step 1. Create a public Connectivity Rule with Stable IPs enabled.** Only one public Connectivity Rule exists per account. If your account already has one, update or recreate that rule with Stable IPs enabled rather than creating a second. Creating the rule returns both the new rule's ID and an async operation that tracks activation: ```bash curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/connectivity-rules" \ -H "$H_AUTH" -H "Content-Type: application/json" \ -d '{ "spec": { "publicRule": { "enableStableIps": true } } }' | tee /tmp/cr.json CR_ID=$(jq -r '.connectivityRuleId' /tmp/cr.json) OP_ID=$(jq -r '.asyncOperation.id // .asyncOperation.operationId' /tmp/cr.json) ``` Field names follow proto3 JSON camelCase: `enable_stable_ips` becomes `enableStableIps`. Wait for the create operation to reach a terminal state before continuing: ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/operations/$OP_ID" \ -H "$H_AUTH" | jq '.asyncOperation.state' ``` Re-run until `state` is `FULFILLED` (the rule is now `ACTIVE`). Any `*_FAILED` state means you should stop and inspect the operation. Alternatively, list every Connectivity Rule on the account and read the new rule's state directly: ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/connectivity-rules" \ -H "$H_AUTH" | \ jq '.connectivityRules[] | {id, state, spec}' ``` The rule is ready when its `state` is `RESOURCE_STATE_ACTIVE`. This also shows you whether the account already has a public rule, which matters because only one is allowed. **Step 2. Attach the rule to the Namespace.** `UpdateNamespace` replaces the entire spec and requires the latest `resource_version`, so read the current Namespace first. Any field you leave out of `spec` is cleared, so build the update body from the response instead of writing one by hand. ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "$H_AUTH" | tee /tmp/ns.json | \ jq '{resource_version: .namespace.resourceVersion, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}' RV=$(jq -r '.namespace.resourceVersion' /tmp/ns.json) ``` Then post the whole `NamespaceSpec` back, appending `$CR_ID` to `connectivityRuleIds`. A Namespace can have at most one public Connectivity Rule attached, so if `connectivityRuleIds` already references a public rule without Stable IPs, replace that entry instead of appending to the list. ```bash jq --arg id "$CR_ID" --arg rv "$RV" --arg ns "$NS" ' { namespace: $ns, resourceVersion: $rv, spec: (.namespace.spec | .connectivityRuleIds = ((.connectivityRuleIds // []) + [$id])) }' /tmp/ns.json > /tmp/update.json curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "$H_AUTH" -H "Content-Type: application/json" \ -d @/tmp/update.json | tee /tmp/update-resp.json ``` Poll the returned `asyncOperation` the same way as in Step 1. Alternatively, describe the Namespace and confirm the rule is attached: ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "$H_AUTH" | \ jq '{state: .namespace.state, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}' ``` The update is complete when `state` is `RESOURCE_STATE_ACTIVE` and `connectivityRuleIds` contains your `$CR_ID`. #### How to enable Stable IPs with gRPC If you prefer to call the Cloud Ops API directly over gRPC (for example, with `grpcurl` or a generated client), the same flow uses the [CloudService](https://github.com/temporalio/api-cloud) gRPC API. Proto definitions live in [`temporal/api/cloud/cloudservice/v1/service.proto`](https://github.com/temporalio/api-cloud/blob/main/temporal/api/cloud/cloudservice/v1/service.proto) and [`temporal/api/cloud/connectivityrule/v1/message.proto`](https://github.com/temporalio/api-cloud/blob/main/temporal/api/cloud/connectivityrule/v1/message.proto). - Endpoint: `saas-api.tmprl.cloud:443` (TLS required) - Fully qualified service: `temporal.api.cloud.cloudservice.v1.CloudService` - Required metadata on every call: - `authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY` The metadata key is `authorization`, lowercase, per the gRPC convention. Authentication uses Cloud API keys. `saas-api.tmprl.cloud` does not accept mTLS client certificates. The examples below use `grpcurl`. They assume you have a local checkout of [`temporalio/api-cloud`](https://github.com/temporalio/api-cloud) at `./api-cloud` and a `proto/` directory containing the standard `google/api/annotations.proto` and associated files. Set up environment variables: ```bash export TEMPORAL_CLOUD_OPS_API_KEY='' export NS='.' # The Cloud-side Namespace identifier, e.g. "myns.a1b2c3" GRPC_HEADERS=(-H "authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY") PROTO_FLAGS=(-import-path ./api-cloud -import-path ./proto -proto temporal/api/cloud/cloudservice/v1/service.proto) ``` `NS` is the Cloud-side Namespace identifier, not the SDK gRPC hostname (`myns.a1b2c3.tmprl.cloud`). Strip the trailing `.tmprl.cloud`. **Step 1. Create a public Connectivity Rule with Stable IPs enabled.** Only one public Connectivity Rule exists per account. If your account already has one, update or recreate that rule with Stable IPs enabled rather than creating a second. Creating the rule returns both the new rule's ID and an async operation that tracks activation: ```bash grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d '{ "spec": { "public_rule": { "enable_stable_ips": true } } }' \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/CreateConnectivityRule | tee /tmp/cr.json CR_ID=$(jq -r '.connectivityRuleId' /tmp/cr.json) OP_ID=$(jq -r '.asyncOperation.id' /tmp/cr.json) ``` Field names in gRPC JSON requests can use either snake_case (proto field names) or camelCase — `grpcurl` accepts both. Responses come back in camelCase. Wait for the create operation to reach a terminal state before continuing: ```bash grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d "{\"async_operation_id\": \"$OP_ID\"}" \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/GetAsyncOperation | jq '.asyncOperation.state' ``` Re-run until `state` is `FULFILLED` (the rule is now `ACTIVE`). Any `*_FAILED` state means you should stop and inspect the operation. Alternatively, list every Connectivity Rule on the account and read the new rule's state directly: ```bash grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d '{}' \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/GetConnectivityRules | \ jq '.connectivityRules[] | {id, state, spec}' ``` The rule is ready when its `state` is `RESOURCE_STATE_ACTIVE`. This also shows you whether the account already has a public rule, which matters because only one is allowed. **Step 2. Attach the rule to the Namespace.** `UpdateNamespace` replaces the entire spec and requires the latest `resource_version`, so read the current Namespace first. Any field you leave out of `spec` is cleared, so build the update body from the `GetNamespace` response instead of writing one by hand. ```bash grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d "{\"namespace\": \"$NS\"}" \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/GetNamespace | tee /tmp/ns.json RV=$(jq -r '.namespace.resourceVersion' /tmp/ns.json) ``` Then post the whole `NamespaceSpec` back, appending `$CR_ID` to `connectivity_rule_ids`. A Namespace can have at most one public Connectivity Rule attached, so if `connectivity_rule_ids` already references a public rule without Stable IPs, replace that entry instead of appending to the list. ```bash jq --arg id "$CR_ID" --arg rv "$RV" --arg ns "$NS" ' { namespace: $ns, resource_version: $rv, spec: (.namespace.spec | .connectivity_rule_ids = ((.connectivityRuleIds // []) + [$id])) }' /tmp/ns.json > /tmp/update.json grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d @ \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/UpdateNamespace < /tmp/update.json | tee /tmp/update-resp.json ``` Poll the returned `asyncOperation` the same way as in Step 1. Alternatively, describe the Namespace and confirm the rule is attached: ```bash grpcurl "${PROTO_FLAGS[@]}" "${GRPC_HEADERS[@]}" \ -d "{\"namespace\": \"$NS\"}" \ saas-api.tmprl.cloud:443 \ temporal.api.cloud.cloudservice.v1.CloudService/GetNamespace | \ jq '{state: .namespace.state, connectivity_rule_ids: .namespace.spec.connectivityRuleIds}' ``` The update is complete when `state` is `RESOURCE_STATE_ACTIVE` and `connectivityRuleIds` contains your `$CR_ID`. ### How to view Stable IP ranges The full list of Stable IP ranges is published as a public JSON file at: **[https://docs.temporal.io/json/stable-ip-ranges-prod.json](/json/stable-ip-ranges-prod.json)** Click the link to inspect the list in your browser, or copy the URL into your firewall tooling, a `curl`/`wget` script, or a Terraform `http` data source: ``` https://docs.temporal.io/json/stable-ip-ranges-prod.json ``` The file is publicly accessible without authentication and returns a JSON response grouped by cloud provider and region: ```json { "all_stable_ip_ranges": [ "34.195.80.228/32", "34.195.80.10/30", "44.235.214.171/32" ], "clouds": { "aws": { "regions": { "us-east-1": { "stable_ip_ranges": [ "34.195.80.228/32", "34.195.80.10/30" ] }, "us-west-2": { "stable_ip_ranges": [ "44.235.214.171/32" ] } } }, "gcp": { "regions": { ... } } } } ``` > **📝 Note:** > IP format > > IP addresses are provided in IPv4 or IPv6 format with CIDR notation. At launch, Temporal uses predominantly `/32` ranges (single IP addresses). Each region typically contains approximately 6 IP addresses. > ### How to connect using Stable IPs To connect to a Namespace with Stable IPs enabled: 1. **Use the Namespace endpoint**: Configure Workers and Temporal Clients to use the Namespace endpoint format: `..tmprl.cloud:7233` 2. **Do not use Regional endpoints**: Regional endpoints (`..api.temporal.io`) will route to the Namespace but resolve to dynamic IP addresses. 3. **Do not use PrivateLink**: PrivateLink endpoints do not resolve to Stable IPs. To use Stable IPs, connect over public endpoints. Stable IPs work with both mTLS certificate-based authentication and API key authentication. If your Workers run on the same cloud provider as your Namespace (for example, both on AWS), traffic stays on the cloud provider's backbone network and does not traverse the public internet, even when using public Stable IPs. AWS, GCP, and Azure all guarantee that traffic between locations on their networks never leaves their backbone. ### How to migrate to Stable IPs When you enable Stable IPs on an existing Namespace: **Workers using Namespace endpoints**: - Existing connections continue uninterrupted - DNS updates to point the Namespace endpoint to Stable IPs - Existing connections remain connected to their current dynamic IP - New connections automatically use Stable IPs after DNS propagates - No Worker restart required **Workers using Regional endpoints**: - Existing connections continue uninterrupted - To use Stable IPs, update Worker configuration to use the Namespace endpoint and restart Workers **Workers using PrivateLink**: - Existing connections continue uninterrupted through PrivateLink - To use Stable IPs, update Worker configuration to use the Namespace endpoint and restart Workers - Note: You cannot use both PrivateLink and Stable IPs simultaneously **Connectivity Rules**: Stable IPs is enabled when the `enableStableIps` config is `true` on a public [Connectivity Rule](/cloud/connectivity#connectivity-rules). Every Namespace that should resolve to Stable IPs must have that public Connectivity Rule attached. Namespaces without the rule attached continue to resolve to dynamic IPs. #### How to migrate from AWS PrivateLink to Stable IPs If your Namespace is currently reached over AWS PrivateLink and protected by a private Connectivity Rule, the safe migration path is to add the public path while the private path is still working, verify, and then remove the private path. This applies whether your Namespace is single-region or replicated. Two things do not change in this migration: - **Network path for same-region traffic.** When your Worker and the Namespace are both in AWS in the same region, traffic to a Stable IP stays inside the AWS network instead of routing over the public internet. Moving off PrivateLink does not change that. - **TLS and authentication.** You are still connecting to the Namespace endpoint (`..tmprl.cloud`), so the certificate and SNI are the same and only the IP it resolves to changes. mTLS and API key authentication continue to work unchanged. **Prerequisites:** - Note your existing private (PrivateLink) Connectivity Rule ID — you'll keep it attached during the overlap window. - Check whether your account already has a public Connectivity Rule. Only one is allowed per account; if one exists without Stable IPs enabled, you'll need to delete and recreate it (see [Permissions and limits](/cloud/connectivity#permissions-and-limits)). - Confirm that the VPCs and security groups that host your Workers and Temporal Clients allow egress to the Stable IP ranges on port `443`. If your egress is currently locked down to the PrivateLink VPC endpoint, you must expand it first. **Steps:** 1. **Allowlist Stable IPs in your firewall first.** Fetch the IP ranges for your Namespace's region from [https://docs.temporal.io/json/stable-ip-ranges-prod.json](/json/stable-ip-ranges-prod.json) and add them to your egress allowlist before changing anything else. If you skip this step, traffic will be lost. 2. **Create the account's public Connectivity Rule with Stable IPs enabled.** Use the Temporal CLI: ```command temporal cloud connectivity public create --enable-stable-ips ``` If you already have a public Connectivity Rule without Stable IPs, delete and recreate it — there is no in-place update flag, and creating a second public rule returns an error. Wait for the rule to reach `ACTIVE` before continuing. 3. **Attach the public rule to the Namespace.** Keep the existing private rule attached during the overlap window, then add the public rule: ```command temporal cloud namespace connectivity attach \ --namespace "." \ --connectivity-rule-id "" ``` Workers continue to use PrivateLink at this point, because your private DNS still resolves the Namespace endpoint to the VPC endpoint. The public path is now *allowed* on Temporal's side but not yet *used*. Keep the private rule attached until step 4 is complete. If you detach the private rule while Workers are still resolving the Namespace endpoint to the PrivateLink VPC endpoint, Temporal Cloud blocks those connections at the edge. 4. **Switch client DNS resolution from PrivateLink to public.** This is the actual cutover. Remove the private DNS override — Route 53 private hosted zone, `/etc/hosts`, or VPC DNS resolver rules — so that `..tmprl.cloud` resolves through public DNS to a Stable IP. Restart Workers to drop existing PrivateLink connections and force re-resolution. A single connection uses one network path or the other. Workers connect over either PrivateLink or Stable IPs based on DNS resolution at connect time, and there is no in-place switch without dropping the connection. That is why this step includes a Worker restart. 5. **Verify traffic is flowing over Stable IPs.** On a Worker host, run `dig ..tmprl.cloud` and confirm the answer is a Stable IP from the [published JSON list](/json/stable-ip-ranges-prod.json), not your VPC endpoint IP. Confirm Workers are polling and there are no `RESOURCE_EXHAUSTED` or connection errors in their logs. 6. **Detach the private Connectivity Rule.** Once you are confident all traffic is on the public path, remove the private rule: ```command temporal cloud namespace connectivity detach \ --namespace "." \ --connectivity-rule-id "" ``` 7. **(Optional) Tear down the PrivateLink VPC endpoint** in AWS to stop incurring PrivateLink hourly and data-processing charges. Do not do this until step 6 has been stable for at least a few hours. ### Changes to Stable IP ranges Once a region's Stable IP ranges are published, Temporal avoids adding new ranges to that region's list, and Temporal has no plans to ever remove a published range. If an unforeseen event requires Temporal to add or remove a region's published ranges, Temporal will give at least three months, and preferably six months, advance notice through targeted communication to accounts that have Stable IPs enabled. Changes to IP Ranges are publicly announced in the [Temporal Cloud changelog](https://temporal.io/change-log). Temporal never redirects Namespace Endpoints with Stable IPs to IP addresses that are not on the published list. ### Pricing Temporal Cloud does not charge additional fees for using Stable IPs. ### How to disable Stable IPs You can disable Stable IPs on a Namespace by detaching the public Connectivity Rule from that Namespace. Temporal performs a DNS update to point the Namespace endpoint back to dynamic IP addresses. Existing Worker connections are not interrupted. New Worker connections will no longer resolve to Stable IPs after DNS propagates. --- # Workflow History Export Source: https://docs.temporal.io/cloud/export > Export closed Workflow Histories from Temporal Cloud to S3 or GCS hourly, in protobuf format, for compliance and analytics. Workflow History Export allows users to export closed Workflow Histories from Temporal Cloud to cloud object storage (AWS S3 or GCP GCS), enabling: - Compliance and audit trails of complete Event History data in [proto format](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) - Analytics on Event History when ingested to the data platform of your choice Workflow History Export in Temporal Cloud provides similar functionality as [Archival](/self-hosted-guide/archival) in a Self-Hosted Temporal Server. Archival is not supported in Temporal Cloud. Exports run hourly, beginning 10 minutes after the hour. Allow up to 24 hours for a closed Workflow to appear in the exported file. Delivery is guaranteed at least once. ## What's in the exported data Each exported file contains one or more complete Workflow Execution histories serialized as protocol buffers using the [`WorkflowExecutions`](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) proto. Each history is an ordered sequence of events that records everything that happened during a Workflow Execution: - **Workflow configuration** - Input data, timeouts, Task Queue, retry policies, search attributes, and memo - **Activity lifecycle** - Each Activity scheduled, started, completed, or failed/timed out, including inputs and results - **Timers** - Timer starts and fires - **Signals and Updates** - External Signals received and Update requests handled - **Child Workflows** - Child Workflow starts and completions - **Workflow result** - How the Workflow ended (completed, failed, timed out, terminated, canceled, or continued-as-new) Search attributes in the export use your **user-defined names** (for example, `customerId`), not internal column names. The export format is **protobuf binary**. You must deserialize using the [proto schema](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) before the data is human-readable. The following is a simplified JSON representation of what one Workflow Execution looks like after deserialization. This example shows a Workflow that started, ran one Activity, and completed: ```json { "items": [ { "history": { "events": [ { "eventId": "1", "eventTime": "2025-02-24T18:00:00Z", "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED", "workflowExecutionStartedEventAttributes": { "workflowType": { "name": "OrderWorkflow" }, "taskQueue": { "name": "order-processing" }, "input": { "payloads": ["...serialized input..."] }, "workflowExecutionTimeout": "3600s", "searchAttributes": { "indexedFields": { "customerId": { "data": "\"customer-42\"" }, "orderType": { "data": "\"standard\"" } } } } }, { "eventId": "2", "eventTime": "2025-02-24T18:00:00Z", "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED" }, { "eventId": "3", "eventTime": "2025-02-24T18:00:01Z", "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED", "activityTaskScheduledEventAttributes": { "activityType": { "name": "ChargeCustomer" }, "taskQueue": { "name": "order-processing" }, "input": { "payloads": ["...serialized input..."] }, "scheduleToCloseTimeout": "300s", "startToCloseTimeout": "60s" } }, { "eventId": "4", "eventTime": "2025-02-24T18:00:02Z", "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED" }, { "eventId": "5", "eventTime": "2025-02-24T18:00:03Z", "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED", "activityTaskCompletedEventAttributes": { "result": { "payloads": ["...serialized result..."] } } }, { "eventId": "6", "eventTime": "2025-02-24T18:00:03Z", "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED" }, { "eventId": "7", "eventTime": "2025-02-24T18:00:03Z", "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_COMPLETED", "workflowExecutionCompletedEventAttributes": { "result": { "payloads": ["...serialized result..."] } } } ] } } ] } ``` The outer `items` array can contain multiple Workflow Executions per file. Only the key fields are shown above. Actual events include additional fields like `version`, `taskId`, and `workerVersion`. ## Prerequisites To use Workflow History Export, you must have: 1. A cloud account in the cloud provider where your Namespace is hosted. 2. An object storage bucket available to receive the exported History. ## Configure Workflow History Export ### AWS [AWS S3 Export Configuration](/cloud/export/aws-export-s3) ### GCP [GCP GCS Export Configuration](/cloud/export/gcp-export-gcs) ## Verify export setup From the Export configuration page, select **Verify**. This validates that Temporal can successfully write a test file to your object storage. If everything is configured correctly, you will see a `Success` status indicating Temporal has written to the object store. ## Monitor export progress After Export has been configured, you can check that it's still working in several ways: 1. **Object Storage**: - File Delivery: After the initial hour of setting up, inspect your object storage. You should see the exported Event History files. - Directory Structure: Your exported files will adhere to the following naming convention and path: ```bash //[bucket-name]/temporal-workflow-history/export/[Namespace]/[Year]/[Month]/[Day]/[Hour]/[Minute]/ ``` The exported file name will include a randomly generated ID. The time recorded in the directory structure is the time the export uploads to object storage, not the Workflow completion time. 2. **Temporal Cloud Web UI**: - Export UI: - Last Successful Export: This displays the timestamp of the most recent successful export. - Last Status Check: This reflects the timestamp of the latest internal Workflow healthcheck. - Usage Dashboard: - Actions from the Export Job are included in the [Usage Dashboard](/cloud/actions-usage). 3. **Email**: - Emails are sent to `Namespace Administrator`, `Account Owner`, and `Global Administrator` roles when a Workflow History Export job fails due to a user related error (such as Object Store permissions issue). > **📝 Note:** > > An export configuration that fails for 7 consecutive days is automatically disabled. > ## Working with exported files Use the proto schema defined [here](https://github.com/temporalio/api/blob/main/temporal/api/export/v1/message.proto) to deserialize exported files. ### Using exported files in analytics It can be useful to convert protos to another format to perform analytics on the data. To convert protos to parquet, follow [the example Python Workflow](https://github.com/temporalio/samples-python/tree/main/cloud_export_to_parquet). Note that this example Workflow: * Transforms the nested proto structure into a flat, tabular format. * Each row in the table represents a single history event from a Workflow. To preserve their relationship post-conversion, the `workflowID` and `runID` is included in every row. * If you have enabled the codec server, the payload field is encrypted. This field may contain characters that are not recognized when loaded into a database so the payload field is excluded in this example. ## Export and High Availability Namespaces ### Export Region Persistence When Export is configured for a [High Availability](/cloud/high-availability) Namespace, the export is tied to the specific region where it was initially set up. The export configuration does not automatically failover with the Namespace. - If Export is configured in Region A, it will continue to export from Region A's storage even after a Namespace failover to Region B - Exports always read from and write to the same region where they were originally configured - The export process is independent of Namespace failover events - Export does not fail over automatically because we prioritize data completeness and consistency over real-time availability for exports. HA data replication has inherent latency, which could result in incomplete or inconsistent exports during a failover. ### Failover Scenarios **Namespace Failover with Healthy Primary Region**: When a Namespace fails over to a secondary region but the primary region remains healthy (including its blob storage), the export job continues to operate from the primary region. It does not automatically switch to export data from the secondary region. **Primary Region Outage**: If the primary region (where Export was configured) experiences a complete outage including S3/GCS storage: Exports will be unavailable until the primary region recovers. Once the primary region recovers, export will resume and include any Workflow histories that occurred during the outage. There may be delays in export processing, but the complete dataset will eventually be available. It does not automatically switch to export data from the secondary region. --- # Exporting Workflow Event History to AWS S3 Source: https://docs.temporal.io/cloud/export/aws-export-s3 > Export Workflow History to AWS S3 ## Prerequisites Before configuring the Export Sink, ensure you have the following: - An AWS S3 bucket. - The S3 bucket must reside in the same region as your Namespace. - (Optional) An IAM role that has write permission to the above S3 bucket. - You can follow the automation in the UI to create the IAM role. Please pre-create the role if setting up Export via Terraform or the CLI. - (Optional) A KMS ARN associated with the S3 bucket. ## Configure Workflow History export There are multiple ways to configure export: through the [Temporal Cloud UI](#using-temporal-cloud-ui), the [CLI](#using-the-cli), or [`terraform`](#using-terraform). ### Using Temporal Cloud UI You can use the Temporal Cloud UI to configure the Workflow History Export. The Temporal Cloud UI provides two ways for configuring Workflow History Export: - [Automated setup](#automated-setup) (recommended): The Cloud UI launches the AWS CloudFormation Console to create a stack with write permission to the S3 bucket. - [Manual setup](#manual-setup): The Cloud UI provides a CloudFormation template for users to manually configure a CloudFormation stack. > **ℹ️ Info:** > Why does Temporal Cloud provision multiple internal IAM roles to trust for Export? > > Temporal Cloud creates multiple intermediary IAM roles for export operations for security purposes. > The system randomly selects from these roles when writing to your storage sink, which provides several benefits: > > - **Security isolation**: If one IAM role is compromised or needs to be decommissioned, other IAM roles remain available > - **Load distribution**: Avoids relying on a single IAM role, reducing security risk > - **Warm standby**: Keeps multiple IAM roles active to avoid potential throttling when switching between IAM roles > - **Reliability**: Provides resilience against cloud provider account-level issues that could affect a single IAM role > > This approach prioritizes security and availability, ensuring robust export operations even if individual IAM roles encounter issues. The following steps guide you through setting up Workflow History Export using the Temporal Cloud UI. ![](/img/cloud/gcp/export-sink-ui.png) > **💡 Tip:** > > Don't forget to click **Create** at the end of your setup to confirm your export. > #### Automated setup You can use the automated setup to create a CloudFormation stack with write permission to your S3 bucket. Make sure to verify the export setup before you save the configuration. 1. Open the Temporal Cloud UI and navigate to the Namespace you want to configure. 2. Select **Configure** from the **Export** card. 3. Provide the following information to configure the export sink and then select **Create and launch stack**: - Name: A name for the export sink. - AWS S3 Bucket Name: The name of the configured AWS S3 bucket to send Closed Workflow Histories to. - AWS Account ID: The AWS account ID. - Role Name: The name of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket. - KMS ARN: (optional) The ARN of the AWS KMS key to use for encryption of the exported Event History. 4. You will be taken to the CloudFormation Console to create the stack with pre-populated information. - Review the information and then select **Create stack**. #### Manual setup You can manually configure a CloudFormation stack using the provided template. 1. Open the Temporal Cloud UI and navigate to the Namespace you want to configure. 2. Select **Configure** from the **Export** card. 3. Select **Manual** from **Access method**. - Enter the Template URL into your web browser to download your copy of the CloudFormation template. - Configure the CloudFormation template for your export sink. - Follow the steps in the [AWS documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-console-create-stack-template.html) by uploading the template to the CloudFormation console. ### Using the CLI **Temporal CLI** Run the `temporal cloud namespace export s3 create` command and provide the following information: - `--namespace`: The Namespace to configure export for. - `--sink-name`: The name of the export sink. - `--role-arn`: The ARN of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket. - `--bucket-name`: The name of the AWS S3 bucket. - `--region`: The AWS region the S3 bucket is in. For example: ```command temporal cloud namespace export s3 create \ --namespace "your-namespace.your-account" \ --sink-name "your-sink-name" \ --role-arn "arn:aws:iam::123456789012:role/test-sink" \ --bucket-name "your-aws-s3-bucket-name" \ --region "us-east-1" ``` Retrieve the status of this command by running the `temporal cloud namespace export get` command. For example: ```command temporal cloud namespace export get \ --namespace "your-namespace.your-account" \ --sink-name "your-sink-name" ``` **tcld** Run the `tcld namespace export s3 create` command and provide the following information: - `--namespace`: The Namespace to configure export for. - `--sink-name`: The name of the export sink. - `--role-arn`: The ARN of the AWS IAM role to use for the CloudFormation stack that has write permission to the S3 bucket. - `--s3-bucket-name`: The name of the AWS S3 bucket. For example: ```command tcld namespace export s3 create --namespace "your-namespace.your-account" --sink-name "your-sink-name" --role-arn "arn:aws:iam::123456789012:role/test-sink" --s3-bucket-name "your-aws-s3-bucket-name" ``` Retrieve the status of this command by running the `tcld namespace export s3 get` command. For example: ```command tcld namespace export s3 get --namespace "your-namespace.your-account" --sink-name "your-sink-name" ``` The following is an example of the output: ```json { "name": "your-sink-name", "resourceVersion": "a6442895-1c07-4da4-aaca-58d57d338345", "state": "Active", "spec": { "name": "your-sink-name", "enabled": true, "destinationType": "S3", "s3Sink": { "roleName": "your-export-test", "bucketName": "your-export-test", "region": "us-east-1", "kmsArn": "", "awsAccountId": "123456789012" } }, "health": "Ok", "errorMessage": "", "latestDataExportTime": "0001-01-01T00:00:00Z", "lastHealthCheckTime": "2023-08-14T21:30:02Z" } ``` ### Using `terraform` See the [terraform export support](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace_export_sink) for setup instructions. ### Next Steps - [Verify export setup](/cloud/export#verify) - [Monitor export progress](/cloud/export#monitor) - [Work with exported files](/cloud/export#working-with-exported-files) --- # Exporting Workflow Event History to GCS Source: https://docs.temporal.io/cloud/export/gcp-export-gcs > Export Workflow History to GCS ## Prerequisites Before configuring the Export sink, complete the following steps in Google Cloud. 1. Create a GCS bucket and take note of its bucket name, for example, "test-export" - Enable customer-managed encryption keys (CMEK) if you need additional security for your GCS bucket. - Currently, only single region buckets are supported (choose "Region" option when creating the bucket in GCS, not "Multi-region" or "Same-region") - The region of the bucket must be the same as the region of your Temporal Cloud Namespace. 2. Record the GCP Project ID that owns the bucket. 3. Create a service account in the same project that grants Temporal permission to write to your GCS bucket. 4. Follow the instructions in the Temporal Cloud UI. There are two ways to set up this service account: - Manual Setup: - Input the service account ID, GCP project ID and GCS bucket name. - Follow the instructions, manually set up a new service account. - Automated Setup: - Use the [Terraform template](https://github.com/temporalio/terraform-modules/tree/main/modules/export-sa) to create the service account. ## Configure Workflow History export There are multiple ways to configure export: through the [Temporal Cloud UI](#using-temporal-cloud-ui), the [CLI](#using-the-cli), or [`terraform`](#using-terraform). > **📝 Note:** > Why does Temporal Cloud provision multiple service accounts for Export? > > Temporal Cloud creates multiple intermediary service accounts for export operations primarily for security purposes. The system randomly selects from these accounts when writing to your storage sink, which provides several benefits: > > - **Security isolation**: If one service account is compromised or needs to be decommissioned, other accounts remain available > - **Load distribution**: Prevents exclusively using a single account, reducing security risk > - **Warm standby**: Keeps multiple accounts active to avoid potential throttling when switching between accounts > - **Reliability**: Provides resilience against cloud provider account-level issues that could affect a single service account > > This approach prioritizes security and availability, ensuring robust export operations even if individual service accounts encounter issues. ### Using Temporal Cloud UI The following steps guide you through setting up Workflow History Export using the Temporal Cloud UI. ![](/img/cloud/gcp/export-sink-ui-gcp.png) 1. In the Cloud UI, navigate to the Namespaces section. Confirm that the Export feature is visible and properly displayed. 2. Configure the Export sink for a Namespace: 1. Choose GCS as the sink type. 2. Provide the following information: 1. Name 2. Service account ID 3. GCP Project ID 4. GCS bucket name 3. After inputting the necessary values, click on **Verify**. You should be able to write to the sink successfully. If not, please fix any errors or reach out to support for help. - If you just created the GCS bucket and granted permission for your service account, it may take some time for the permission to propagate. You may need to wait up to 5 minutes before clicking the **Verify** button to verify the connection. 4. Clicking **Create** will complete the Export sink setup. 5. The page will auto-refresh and you should see the status “Enabled” on the Export screen. You are now ready to export Workflow histories. 6. You can toggle the enable button if you want to stop export and resume in the future. **Note**: when you re-enable the feature, it will start from the current point in time, and not from the time when you disabled export. 7. You can also delete export by clicking **Delete**. > **💡 Tip:** > > Don't forget to click Create at the end of your setup to confirm your export. > ### Using the CLI **Temporal CLI** 1. [Install the Temporal Cloud extension](/cli/setup-cli#install-the-temporal-cloud-extension) for the Temporal CLI. 2. Run the `temporal cloud namespace export gcs create` command and provide the following information: - `--namespace`: The Namespace to configure export for. - `--sink-name`: The name of the export sink. - `--service-account-email`: The service account that has access to the sink. - `--bucket-name`: The name of the GCP GCS bucket. - `--region`: The region the GCS bucket is in. For example: ```bash temporal cloud namespace export gcs create \ --namespace test.ns \ --sink-name test-sink \ --service-account-email test-sink@test-export-sink.iam.gserviceaccount.com \ --bucket-name test-export-validation \ --region us-central1 ``` 3. Check the status of this command by either viewing the Namespace Export status in the Temporal Cloud UI or by retrieving the sink and looking for the state of "Active": ```bash temporal cloud namespace export get --namespace test.ns --sink-name test-sink ``` **tcld** To access export-related commands in tcld, please follow these steps: 1. [Download the latest version of tcld](/cloud/tcld#install-tcld). 2. Make sure your tcld version is v0.35.0 or above. 3. Run the command: `tcld n export gcs`: ```bash NAME: tcld namespace export gcs - Manage GCS export sink USAGE: tcld namespace export gcs command [command options] [arguments...] COMMANDS: create, c Create export sink update, u Update export sink validate, v Validate export sink get, g Get export sink delete, d Delete export sink list, l List export sinks help, h Shows a list of commands or help for one command OPTIONS: --help, -h show help ``` 4. Run the `tcld n export gcs create` command and provide the following information: - `--namespace`: The Namespace to configure export for. - `--sink-name`: The name of the export sink. - `--service-account-email`: The service account that has access to the sink. - `--gcs-bucket`: The name of the GCP GCS bucket. For example: ```bash tcld n export gcs create -n test.ns --sink-name test-sink --service-account-email test-sink@test-export-sink.iam.gserviceaccount.com --gcs-bucket test-export-validation ``` 5. Check the status of this command by either viewing the Namespace Export status in the Temporal Cloud UI or using the following command and looking for the state of "Active": ```bash tcld n export gcs g -n test.ns --sink-name test-sink { "name": "test.ns", "resourceVersion": "b954de0c-c6ae-4dcc-90bd-3918b52c3f28", "state": "Active", "spec": { "name": "test-sink", "enabled": true, "destinationType": "Gcs", "s3Sink": null, "gcsSink": { "saId": "test-sink", "bucketName": "test-export-validation", "gcpProjectId": "test-export-sink", } }, "health": "Ok", "errorMessage": "", "latestDataExportTime": "0001-01-01T00:00:00Z", "lastHealthCheckTime": "2024-01-23T06:40:02Z" } ``` ### Using `terraform` See the [Terraform export support](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace_export_sink) for setup instructions. ### Next steps - [Verify export setup](/cloud/export#verify) - [Monitor export progress](/cloud/export#monitor) - [Work with exported files](/cloud/export#working-with-exported-files) --- # Get started with Temporal Cloud Source: https://docs.temporal.io/cloud/get-started > Sign up, create a Namespace, choose API key or mTLS authentication, and connect your Clients and Workers to run a Workflow. Getting started with Temporal Cloud involves a few key steps: 1. [Sign up for Temporal Cloud](#sign-up-for-temporal-cloud) 1. [Create a Namespace](#create-a-namespace) 1. [Set up your Clients and Workers](#set-up-your-clients-and-workers) 1. [Run your first Workflow](#run-your-first-workflow) 1. [Invite your team](#invite-your-team) ## Sign up for Temporal Cloud To create a Temporal Cloud account, you can: - Sign up [directly](https://temporal.io/get-cloud); or - Subscribe at the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) or [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go). Signing up through a cloud marketplace is similar to signing up directly on the Temporal Cloud site, but billing goes through your AWS/GCP account. For information about Temporal Cloud Pricing, see our [Pricing Page](/cloud/pricing). ## Create a Namespace See [Managing Namespaces](/cloud/namespaces#create-a-namespace) to create your first namespace. Temporal Cloud supports either [API key](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication for each namespace. If you're not sure which to use, we recommend using [API keys](/cloud/api-keys) because they're easier to manage and rotate for most teams. If your organization already has private key infrastructure (PKI) and is familiar with cert management, then [mTLS](/cloud/certificates) is an excellent choice. ## Set up your Clients and Workers See our guides for connecting each SDK to your Temporal Cloud Namespace: - [Connect to Temporal Cloud in .NET](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in Go](/develop/go/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in Java](/develop/java/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in PHP](/develop/php/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in Python](/develop/python/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in Ruby](/develop/ruby/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in Rust](/develop/rust/client/temporal-client#connect-to-temporal-cloud) - [Connect to Temporal Cloud in TypeScript](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) ## Run your first Workflow See our guides for starting a workflow using each SDK: - [Start a workflow in .NET](/develop/dotnet/client/temporal-client#start-workflow) - [Start a workflow in Go](/develop/go/client/temporal-client#start-workflow-execution) - [Start a workflow in Java](/develop/java/client/temporal-client#start-workflow-execution) - [Start a workflow in PHP](/develop/php/client/temporal-client#start-workflow-execution) - [Start a workflow in Python](/develop/python/client/temporal-client#start-workflow-execution) - [Start a workflow in Ruby](/develop/ruby/client/temporal-client#start-workflow) - [Start a workflow in Rust](/develop/rust/client/temporal-client#start-workflow-execution) - [Start a workflow in TypeScript](/develop/typescript/client/temporal-client#start-workflow-execution) ## Invite your team See [Managing users](/cloud/manage-access/users) to add other users and assign them roles. You can also use [Service Accounts](/cloud/manage-access/service-accounts) to represent machine identities. Since you created the account when you signed up, your email address is the first [Account Owner](/cloud/manage-access/roles-and-permissions#account-level-roles) for your account. --- # User management Source: https://docs.temporal.io/cloud/get-started/users-invite > Learn how to manage user invitations for Temporal Cloud **Web UI** To invite users using the Temporal Cloud UI: 1. In Temporal Web UI, select **Settings** in the left portion of the window. 1. On the **Settings** page, select **Create Users** in the upper-right portion of the window. 1. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses. 1. In **Account-Level Role**, select a [Role](/cloud/manage-access/roles-and-permissions#account-level-roles). The Role applies to all users whose email addresses appear in **Email Addresses**. 1. If the account has any Namespaces, they are listed under **Grant access to Namespaces**. To add a permission, select the checkbox next to a Namespace, and then select a [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions). Repeat as needed. 1. When all permissions are assigned, select **Send Invite**. **Temporal CLI** Use the [`temporal cloud user invite`](/cli/command-reference/cloud/user#invite) command. Specify the user's email, an account-level role, and optionally one or more Namespace permissions. Available account roles: `owner` | `admin` | `developer` | `finance-admin` | `read` | `metrics-read`. Available Namespace permissions: `admin` | `write` | `read`. ```command temporal cloud user invite \ --email \ --account-role \ --namespace-access = ``` Repeat `--namespace-access` to grant permissions on more than one Namespace. `--email` takes a single address, so invite one user per command: ```command temporal cloud user invite \ --email user1@example.com \ --account-role developer \ --namespace-access ns1.my-account=admin \ --namespace-access ns2.my-account=write ``` **tcld** Use the [`tcld user invite`](/cloud/tcld/user/#invite) command. Specify the user's email, an account-level role, and optionally one or more Namespace permissions. Available account roles: `admin` | `developer` | `read`. Available Namespace permissions: `Admin` | `Write` | `Read`. ```command tcld user invite \ --user-email \ --account-role \ --namespace-permission = ``` You can invite multiple users and assign multiple Namespace permissions in a single command: ```command tcld user invite \ --user-email user1@example.com \ --user-email user2@example.com \ --account-role developer \ --namespace-permission ns1=Admin \ --namespace-permission ns2=Write ``` ### Frequently asked questions #### Can multiple Temporal Cloud accounts share the same email domain? Yes. Multiple Temporal Cloud accounts can coexist with users from the same email domain. Each account has its own independent SAML configuration, tied to its unique Account Id. We recommend configuring [SAML](/cloud/manage-access/saml) for each account independently. For the smoother login experience, you can configure SAML for each account separately and use IdP-initiated login: you click the relevant app tile in your identity provider's portal to access the Temporal Cloud account associated with your email address directly. #### Can the same email be used across different Temporal Cloud accounts? No. Each email address can only be associated with a single Temporal Cloud account. If you need access to multiple accounts, you’ll need a separate invite for each one using a different email address. #### Can I use Google or Microsoft SSO after signing up with email and password? If you originally signed up for Temporal Cloud using an email and password, you won’t be able to log in using Google or Microsoft single sign-on. If you prefer SSO, ask your Account Owner to delete your current user and send you a new invitation. During re-invitation, be sure to sign up using your preferred authentication method. Use the [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users) endpoint to invite a user. ``` POST /cloud/users ``` The request body includes a `spec` with the following fields: - `spec.email` — The email address of the user to invite. - `spec.access.account_access.role` — The account-level role to assign. - `spec.access.namespace_accesses` — A map of Namespace names to permissions. Available roles: `ROLE_ADMIN` | `ROLE_DEVELOPER` | `ROLE_READ` | `ROLE_OWNER` | `ROLE_FINANCE_ADMIN`. Available Namespace permissions: `PERMISSION_ADMIN` | `PERMISSION_WRITE` | `PERMISSION_READ`. The new users receive an email with a link to accept the invitation and complete their setup. The new user must use this link to sign up to be added to your account unless the account has a SAML configuration. If your account has a SAML configuration, the new user can sign in using their existing SAML credentials and be included in the account automatically. > **⚠️ Caution:** > > The new user must use the same authentication method they originally signed up with to sign in to Temporal Cloud. If > they used single sign-on (SSO), they must use the same SSO provider to sign in to Temporal Cloud. If they used email and > password authentication, they must use the same email and password to sign in to Temporal Cloud, and cannot use SSO, > even if the underlying email address is the same. > --- # High Availability Source: https://docs.temporal.io/cloud/high-availability > Temporal Cloud's Namespace with High Availability features offers automatic failover, synchronized data, and replication for workloads requiring disaster-tolerant deployment and 99.99% uptime. Temporal keeps your Workflows running even when a Worker crashes. But what happens when a whole data center crashes? Or a region? In the cloud, outages are commonplace. An outage can bring down a whole data center, cluster, region, or cloud provider. To be durable in the cloud, Workflows and applications must handle these outages smoothly, just like Temporal handles a Worker crash. Temporal Cloud's High Availability features add extra reliability to Temporal Cloud Namespaces by handling cloud outages. Using asynchronous replication between multiple regions or cloud providers, combined with automatic outage detection and failover, High Availability keeps your Workflows running even during a cloud region outage. This extra availability comes with an enhanced [SLA](/cloud/sla) of 99.99%, _including_ cloud provider outages. > **💡 Tip:** > White paper > > For an in-depth guide covering everything from why you need High Availability to setting it up in production and advanced options, read the [High Availability White Paper](https://temporal.io/pages/high-availability-white-paper). > ## Built-in reliability Even without High Availability features, Temporal Cloud provides robust reliability and a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)) guarantee against service errors. Each standard Temporal Namespace uses replication across three [Availability Zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones) (AZs) to ensure high availability. An Availability Zone is akin to an isolated data center managed by a cloud hyperscaler, with independent power, networking, and cooling infrastructure. Replication across AZs makes sure that any changes to Workflow state or History are saved in all three AZs _before_ the Temporal Service acknowledges a change back to the Client. As a result, your standard Temporal Namespace stays operational even if one of its three AZs becomes unavailable. This provides the basis of the 99.9% service level agreement for Temporal Cloud Namespaces. However some critical use cases--such as customer-facing applications--require even better availability. That is where Temporal Cloud's High Availability features come in. ## High Availability features High Availability features extend Temporal Cloud's replication across regions and cloud providers, so your Namespace keeps running even when a whole region or cloud provider goes down: | **Deployment** | **Description** | | --------------------------------------- | ---------------------------------------------------------- | | **Multi‑region Replication** | Namespace is replicated across two cloud regions | | **Multi‑cloud Replication** | Namespace is replicated across different cloud providers | ### Key features - **Real-time replication** — Temporal replicates your Namespace across distant regions or cloud providers with no performance impact to your Workers or Workflows. - **Automatic failover with 20-minute RTO** — Temporal manages failover with a 20-minute [RTO](/cloud/rpo-rto). You can also [trigger failover](/cloud/high-availability/failovers) manually at any time, for example for testing. - **Transparent DNS routing** — On failover, DNS reroutes your [Namespace Endpoint](/cloud/namespaces#access-namespaces) to the active region. [Requests that reach the replica are forwarded to the active region automatically](#request-forwarding). - **Sub-1-minute RPO** — In a failover during an outage, the [Recovery Point Objective](/cloud/rpo-rto) is under one minute. - **Real-time lag monitoring** — Monitor your Namespace's replication lag in real time to understand your current RPO. - **Conflict resolution** — If the two regions are not fully in sync at the time of failover, Temporal's conflict resolution process reconciles discrepancies and ensures data integrity. > **ℹ️ Info:** > Region availability > > You can usually choose your replica region, but the replica must be on the same continent as the primary region. > This means that a few Temporal Cloud regions do not yet support Multi-region Replication or Multi-cloud Replication. > See [Regions](/cloud/regions) for a full list of supported replica regions. > > You can't enable both Multi-region Replication and Multi-cloud Replication on the same Namespace at the same time. > ### Multi-cloud Replication Multi-cloud Replication spreads a Namespace across entirely different cloud providers, keeping your Namespace running even during a cloud-wide outage. If a provider outage, service disruption, or network issue occurs, traffic automatically shifts to the replica. Replicated data is encrypted and transmitted across the public internet between cloud providers. This internet connectivity also allows Workers in one cloud to reach the replica in a different cloud during failover. If you use [private connectivity](/cloud/high-availability/ha-connectivity), additional architecture work may be required to ensure your Workers can reach the replica region. > **ℹ️ Info:** > > When you adopt Temporal's High Availability features, don't forget to consider the reliability of your own workers, infrastructure, and dependencies. > Issues like network outages, hardware failures, or misconfigurations in your own systems can affect your application performance. > > For the highest level of reliability, distribute your dependencies across regions, and use our Multi-region or Multi-cloud replication features. > Using physically separated regions improves the fault tolerance of your application. > ## Request forwarding A Namespace with High Availability features replicates across two regions, with one replica active and one passive at any moment. The active replica accepts reads and writes; the passive replica receives replicated state asynchronously and stands ready for failover. When a request reaches the passive replica — for example, through the passive region's Regional Endpoint — Temporal Cloud forwards the request transparently to the active replica and the response back to the Worker. This allows Workers and Clients to connect to the passive region during healthy times, and when an outage hits, start processing Workflows immediately after a failover. Forwarding adds a cross-region hop, so requests that travel through the passive replica complete with higher average latency than requests that reach the active replica directly. You can turn off forwarding for Worker poll requests, which keeps the passive region's Workers from processing tasks. Client requests such as Start Workflow, Signal, Query, Cancel, and Terminate are always forwarded, so a Client that reaches the passive replica keeps working either way. To route Workers to the passive region's replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica). To stop forwarding Worker polls that reach the passive replica, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior). To see which Client requests keep forwarding either way, see [Client requests are forwarded regardless of this setting](/cloud/high-availability/enable#client-requests-still-forwarded). To run Worker fleets in both regions that rely on this forwarding, see [Active/Active](/cloud/high-availability/architecture-patterns#active-active). To keep passive-region Workers on standby until failover by disabling this forwarding, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot). ## Service levels and recovery objectives Namespaces using High Availability have a 99.99% [uptime SLA](/cloud/sla) with sub-1-minute [RPO](/cloud/rpo-rto) and 20-minute [RTO](/cloud/rpo-rto). For detailed information: - [Service Level Agreement (SLA)](/cloud/sla) - [Recovery Point Objective (RPO) and Recovery Time Objective (RTO)](/cloud/rpo-rto) ## Failover High Availability Namespaces can automatically or manually [fail over](/cloud/high-availability/failovers) to the replica if the primary is unavailable or unhealthy. The Namespace fails over automatically, but your Workers and the rest of your architecture need their own plan — see [Worker deployment patterns for Active-Passive and Active-Active HA/DR](/cloud/high-availability/architecture-patterns). ## Target workloads High Availability Namespaces are a great solution for Workloads where an outage would cause: - Revenue loss - Poor customer experience - Problems stemming from policy/legal requirements that demand high availability These are often major concerns for financial services, e-commerce, gaming, global SaaS platforms, bookings & reservations, delivery & shipping, and order management. ## Same-region Replication In selected regions, you can add a replica to a Namespace in the same region. Temporal operates a "cell architecture" and will replicate the Namespace across multiple cells in that region. This feature is currently in [Public Preview](/evaluate/development-production-features/release-stages) in selected regions. Failovers between cells are always managed automatically by Temporal. Unlike Multi-region and Multi-cloud Replication, you cannot disable automatic failovers and you cannot trigger a manual failover for a Same-region Replication Namespace. See [Failovers](/cloud/high-availability/failovers) for details. ## Related considerations A few other Temporal features interact with High Availability in ways worth understanding before you enable it. ### Serverless Workers You can use [Serverless Workers](/serverless-workers) with a Namespace that has Multi-region or Multi-cloud Replication, but without manual intervention, the workloads that failed over to the new region will keep invoking Workers in the old region. If the compute provider's region is experiencing degraded performance, this may impact your workloads even though your Temporal Cloud Namespace has failed over successfully. This is because each compute provider configuration (for example, a Lambda ARN) is scoped to a single region. The [Worker Controller Instance](/serverless-workers#worker-controller-instance) has no mechanism to detect a failover or redirect invocations to a Worker in the new active region. To keep processing Workflows after a failover, you must manually reconfigure the Worker Deployment Version's compute provider to point at a Worker in the new region. See [Constraints - Serverless Workers](/serverless-workers#constraints). ### External Storage If your Workflows use [External Storage](/external-storage) to offload large payloads, durability of the external store is a separate concern from Namespace replication. - For S3, see [Durable External Storage](/external-storage#durable-external-storage) to see how CRR + MRAP can be used. - For external storage providers other than what is listed above, we do not yet provide guidance. Consult the providers documentation for more information. --- # Architecture patterns for Active-Passive and Active-Active HA/DR Source: https://docs.temporal.io/cloud/high-availability/architecture-patterns > Choose a Worker deployment pattern for high availability and disaster recovery (HA/DR) on Temporal Cloud — Active/Passive, Active/Hot-Passive, and Active/Active — and plan multi-region failover for Workers, Clients, Codec Servers, and databases to meet your RTO and RPO. When a cloud outage strikes, a Temporal Cloud Namespace with [High Availability](/cloud/high-availability) fails over to another region automatically, providing high availability and disaster recovery for your most important Workflows. But Workers, Workflow starters, Codec Servers, databases, network load balancers, and other external systems that touch your Workflows each need their own failover story. In a real outage, your [recovery time](/cloud/rpo-rto) depends on your **Worker deployment pattern**: where Worker fleets run, how they failover, and which region (or regions) processes Workflows at each step. This page describes the Active/Passive, Active/Hot-Passive, and Active/Active patterns for deploying Workers and the other systems in your cloud architecture to achieve your business goals for high availability and disaster recovery (HA/DR). ## Pieces of a highly available cloud architecture To keep your Workflows running during a cloud outage, these components need to failover to a healthy region: - **Temporal Cloud Namespace** - achieve this simply by enabling [High Availability](/cloud/high-availability) on the Namespace. - **Workers** (the focus of this page) — the compute resources that execute Workflows and Activities. - **Workflow starters and Clients** — the applications that start and Signal Workflows. - **Codec Servers** — a critical dependency for Workers, the Web UI, and the CLI. - **Proxies between Workers and Temporal Cloud** — any forward proxy or mTLS terminator in the connection path between Workers / Starters / Clients → Namespace. - **Datastores** — databases, queues, and any other systems that Activities read and write. ```mermaid flowchart LR classDef app stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef env stroke-width:1.5px; subgraph WFENV["Application environment"] WFW["Workers"]:::app WFCL["Workflow starters / Clients"]:::app WFCODEC["Codec Server"]:::app WFDB[("DB / queue")]:::app WFPROXY["Proxy"]:::endpoint end WFNS["Namespace
(Temporal Cloud)"]:::ns WFW <--> WFCODEC WFW <--> WFDB WFW --> WFPROXY WFCL --> WFPROXY WFPROXY --> WFNS class WFENV env ``` ## Highly available Worker patterns A Worker deployment pattern pairs with Namespace High Availability to achieve a disaster recovery (DR) and business continuity plan for your full Temporal architecture. This page covers three main patterns: **Active/Passive**, **Active/Hot-Passive**, and **Active/Active**. They trade off **recovery time** after an outage, **cost during normal operation**, and **operational complexity**. They are defined by where the Workers run and where Workflows process: - **[Active/Passive](#active-cold)** — Workflows process in one region at a time, the "active" region. The other region is "passively" waiting, without any Workers. On a failover, the passive region becomes active, and new Workers are launched (from a "cold" start) to process Workflows. - **[Active/Hot-Passive](#active-hot)** — Workflows process in one region at a time, the "active" region. However, Workers run in **both regions** simultaneously: processing Workflows in the "active" region, and on "hot" standby in the passive region. This achieves a faster failover and lower recovery time. - **[Active/Active](#active-active)** — Workflows process in both/multiple regions at the same time, and Workers run in all regions at all times. ```mermaid --- title: Active/Passive --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef empty stroke-width:1px,stroke-dasharray:4 3; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph ICPRIM["Primary"] subgraph ICWP["Worker Pool"] ICW1["Worker"]:::worker ICW2["Worker"]:::worker ICW3["Worker"]:::worker end ICNS["Namespace"]:::ns ICWP <-->|Workflows| ICNS end subgraph ICSEC["Secondary"] ICR["Replica"]:::ns subgraph ICWP2["Worker Pool"] ICE["      Empty      "]:::empty end ICR ~~~ ICWP2 end ICNS --> ICR class ICPRIM,ICSEC region class ICWP,ICWP2 pool ``` ```mermaid --- title: Active/Hot-Passive --- flowchart LR classDef worker stroke-width:1px; classDef standby stroke-width:1px; classDef ns stroke-width:1px; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph IHPRIM["Primary"] subgraph IHWP["Worker Pool"] IHW1["Worker
Active"]:::worker IHW2["Worker
Active"]:::worker IHW3["Worker
Active"]:::worker end IHNS["Namespace"]:::ns IHWP <-->|Workflows| IHNS end subgraph IHSEC["Secondary"] IHR["Replica"]:::ns subgraph IHWP2["Worker Pool"] IHS1["Worker
Standby"]:::standby IHS2["Worker
Standby"]:::standby IHS3["Worker
Standby"]:::standby end IHR <-.-> IHWP2 end IHNS --> IHR class IHPRIM,IHSEC region class IHWP,IHWP2 pool ``` ```mermaid --- title: Active/Active --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph IAPRIM["Primary"] subgraph IAWP["Worker Pool"] IAW1["Worker
Active"]:::worker IAW2["Worker
Active"]:::worker end IANS["Namespace"]:::ns IAWP <-->|Workflows| IANS end subgraph IASEC["Secondary"] IAR["Replica"]:::ns subgraph IAWP2["Worker Pool"] IAS1["Worker
Active"]:::worker IAS2["Worker
Active"]:::worker end IAR <-->|Workflows| IAWP2 end IANS --> IAR class IAPRIM,IASEC region class IAWP,IAWP2 pool ``` > **ℹ️ Info:** > > **Namespaces always have a single active region, but can support an Active/Active Worker deployment pattern.** > > A Temporal Cloud Namespace with High Availability has exactly one active region at a time. The other region holds a replica that passively receives replicated state. > > However, since **Workers don't need to run in the same region as the active Namespace replica**, Temporal Cloud Namespaces can still fit into an Active/Active HA/DR strategy, as described below. > These patterns work across two cloud regions, which could be in the same cloud provider ("multi-region") or different cloud providers ("multi-cloud"): - **Primary region** — the region where the Namespace is active during normal operation, also called the "preferred region." - **Secondary region** — the region the Namespace fails over to. It can be any [Temporal Cloud region](/cloud/regions) that supports replication from the primary region. Multi-region and multi-cloud architectures use the same Worker deployment patterns. ### Compare patterns at a glance | Pattern | Where Workers run | Best for and benefits | Major tradeoffs | | --- | --- | --- | --- | | **[Active/Passive](#active-cold)** | One region at a time | Easy initial deployment; acts like a single region with no special setup | Failing over Workers is your responsibility; highest recovery time of the three | | **[Active/Hot-Passive](#active-hot)** | Both regions; secondary on warm standby | Low RTO with all task processing in the active region; fast Worker failover with no cold start | More configuration and the cost of a full standby fleet | | **[Active/Active](#active-active)** | All regions, all processing Workflows | Low RTO with Workers active in every region; fast failover that uses fleet capacity instead of a standby fleet | Cross-region requests add Workflow latency; external systems need a cross-region consistency story | ## Active/Passive Workers run in only one region at a time. The secondary region stays empty until a failover, when you bring up a fresh Worker fleet there from a cold start. ```mermaid --- title: Normal operation --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef empty stroke-width:1px,stroke-dasharray:4 3; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph CPRIM["Primary"] subgraph CWP["Worker Pool"] CW1["Worker"]:::worker CW2["Worker"]:::worker CW3["Worker"]:::worker end CNS["Namespace"]:::ns CWP <-->|Workflows| CNS end subgraph CSEC["Secondary"] CR["Replica"]:::ns subgraph CWP2["Worker Pool"] CE["      Empty      "]:::empty end CR ~~~ CWP2 end CNS --> CR class CPRIM,CSEC region class CWP,CWP2 pool ``` ```mermaid --- title: After failover --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef down stroke-width:1px,stroke-dasharray:3 3; classDef region stroke-width:1.5px; classDef regiondown stroke-width:1.5px; classDef pool stroke-width:1px; subgraph FPRIM["Primary (outage)"] subgraph FPP["Worker Pool"] FPW1["Unavailable"]:::down end FPN["Namespace"]:::down FPP ~~~ FPN end subgraph FSEC["Secondary"] FSN["Namespace
(Active)"]:::ns subgraph FSP["Worker Pool"] FSW1["Worker
Cold start"]:::worker FSW2["Worker
Cold start"]:::worker FSW3["Worker
Cold start"]:::worker end FSN <-->|Workflows| FSP end FPN -->|"Failover"| FSN class FPRIM regiondown class FSEC region class FPP,FSP pool ``` Here's how each component behaves during normal operation and after a failover: | Component | Normal operation | On failover | | --- | --- | --- | | **Workers** | Run only in the primary region, processing all Workflows. No Workers run in the secondary region. | Brought up from nothing in the secondary region — a "cold" start. No Workflows progress until they're running. | | **Namespace** | Active replica in the primary region; passive replica in the secondary region, continuously receiving replicated Workflow state. | Temporal Cloud promotes the secondary region's replica to active automatically. To trigger or test a failover yourself, see [Failovers](/cloud/high-availability/failovers). | | **Workflow starters and Clients** | Run with the Workers in the primary region. | Brought up in the secondary region along with the Workers. | | **Codec Servers and proxies** | Run alongside the active Workers in the primary region. | Scaled up in the secondary region as part of the failover. | | **Databases and queues** | Replicate to the secondary region, if your Workflows depend on that data. | Promote the secondary region's copy to active, if needed, so the new Workers can read and write it. | Setup is minimal: turn on Replication for your Namespace (see [Enable and manage High Availability](/cloud/high-availability/enable)) and enable replication on any databases or queues your Workflows use. At that point you're technically already running Active/Passive: the secondary region holds a ready replica, and failing over is a matter of bringing your Workers up there. Recovery time is dominated by two things: how quickly you detect the outage, since Workflows make no progress until you respond (see [Detect a failover or an outage](/cloud/high-availability/monitoring#detect-failover-or-outage)), and how long the secondary-region Worker fleet takes to cold start — container or VM startup, image pulls, and application warm-up. The [Active/Hot-Passive](#active-hot) pattern removes both by keeping Workers already running and warm in the secondary region. ### Benefits Active/Passive offers the simplest operational model of the three patterns: - **Easy to reason about.** - Only one region is active at a time, so traffic routing and interactions with systems (such as databases and queues) are simpler to understand, and the pattern pairs naturally with other active / passive systems. Active/Active, by contrast, requires deciding how Workers reach an active database: either a local active database in each region, or a single active / passive database that some Workers must reach cross-region. - **Simple to operate.** - During normal operation it resembles a single-region deployment. - **Lowest overall architecture cost.** - The size of the Worker fleet is simply the capacity needed to operate in one region. There are no standby Workers during steady state. ### Tradeoffs That simplicity comes at the cost of recovery time: - Highest overall recovery time of the three patterns, due to cold starting the Worker fleet after failover. - Depends on tested automation to bring up the secondary-region fleet quickly. ### Recommendations and important constraints Keep these in mind when setting up Active/Passive: - **Failing over the Workers is the operator's responsibility.** The Namespace fails over automatically, but bringing up the Workers in the secondary region is up to you. Plan for these sub-considerations: - **How do you detect an outage and decide to fail over?** Define the failover conditions and the signals (alerts, health checks) that trigger them. Because Workflows make no progress until you detect the outage and respond, detection is on the critical path of your recovery time. To monitor for an outage and a failover, see [Detect a failover or an outage](/cloud/high-availability/monitoring#detect-failover-or-outage). - **How do you scale up the Workers?** Bring up the secondary-region fleet, ideally with tested automation, and scale down the primary region's fleet so Workers run in only one region at a time. - **Do you need to enforce single-region task processing?** This pattern relies on the operator to keep Workers in one region. To have Temporal enforce that Workflow and Activity Tasks are processed only in the active region, use the [Active/Hot-Passive](#active-hot) pattern. ```mermaid flowchart LR FS1["Detect
outage"] FS3["Failover
DBs / queues"] FS4["Scale down
Primary region
Workers"] FS5["Scale up
Secondary region
Workers"] FS6["Confirm
Workflows run
normally"] FS1 --> FS3 --> FS4 --> FS5 --> FS6 ``` - **Use the Namespace Endpoint.** - Connect Workers through the [Namespace Endpoint](/cloud/namespaces#access-namespaces), which always connects to the Namespace in its active region and automatically fails over to the new region. - **Rationale:** If a Temporal Cloud incident requires the Namespace to fail over while the rest of the primary region is healthy, the Workers in the primary region can still connect through the Namespace Endpoint and process Workflows. If the Workers use the Regional Endpoint for the primary region, they will not reliably connect to the Namespace during a Temporal Cloud incident in the primary region. ```mermaid flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; NEW["Worker"]:::worker NEEP["Namespace Endpoint"]:::endpoint NEW --> NEEP subgraph NEPRIM["Primary"] NEPNS["Namespace"]:::ns end subgraph NESEC["Secondary"] NESNS["Namespace"]:::ns end NEEP -->|"normal operation"| NEPNS NEEP -.->|"after failover"| NESNS class NEPRIM,NESEC region ``` - **Set up cross-region private connectivity.** - If you use private connectivity, give the primary region's Workers a network route to the VPC Endpoint in the other region, so they can reach the active replica after a Namespace-only failover. If you can't provide that cross-region route, use the [Active/Hot-Passive](#active-hot) pattern instead, where each region's Workers connect to their local replica. - For the full setup of Regional Endpoints, VPC Endpoints, and cross-region routing, see [Connectivity for High Availability](/cloud/high-availability/ha-connectivity). ```mermaid flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; subgraph CRCPRIM["Primary Region"] CRCPW["Worker"]:::worker CRCPEP["VPC Endpoint"]:::endpoint CRCPNS["Namespace"]:::ns end subgraph CRCSEC["Secondary Region"] CRCSEP["VPC Endpoint"]:::endpoint CRCSNS["Replica"]:::ns end CRCPW -->|"normal operation"| CRCPEP CRCPEP --> CRCPNS CRCSEP --> CRCSNS CRCPW -.->|"after a Namespace failover"| CRCSEP class CRCPRIM,CRCSEC region ``` - **Route Workers to the active region's Codec Server.** Two common approaches: - Put DNS or a load balancer in front of the Codec Server address, and update it on failover to point at the new region's instance. - Pass each Worker the Codec Server address for its own region as configuration, so a Worker always uses the service local to it. This is common in Kubernetes or with service discovery. - **Route Workers to the active region's proxy.** Two common approaches: - Put DNS or a load balancer in front of the proxy address, and update it on failover to point at the new region's instance. - Pass each Worker the proxy address for its own region as configuration, so a Worker always uses the service local to it. This is common in Kubernetes or with service discovery. ## Active/Hot-Passive A full Worker fleet runs in both regions, but only the primary region's fleet processes Workflows. The secondary region's fleet stays warm on standby, ready to take over the moment a failover promotes it, with no cold start. ```mermaid --- title: Normal operation --- flowchart LR classDef worker stroke-width:1px; classDef standby stroke-width:1px; classDef ns stroke-width:1px; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph HPRIM["Primary"] subgraph HWP["Worker Pool"] HW1["Worker
Active"]:::worker HW2["Worker
Active"]:::worker HW3["Worker
Active"]:::worker end HNS["Namespace"]:::ns HWP <-->|Workflows| HNS end subgraph HSEC["Secondary"] HR["Replica"]:::ns subgraph HWP2["Worker Pool"] HS1["Worker
Standby"]:::standby HS2["Worker
Standby"]:::standby HS3["Worker
Standby"]:::standby end HR <-.-> HWP2 end HNS --> HR class HPRIM,HSEC region class HWP,HWP2 pool ``` ```mermaid --- title: After failover --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef down stroke-width:1px,stroke-dasharray:3 3; classDef region stroke-width:1.5px; classDef regiondown stroke-width:1.5px; classDef pool stroke-width:1px; subgraph HFPRIM["Primary (outage)"] subgraph HFPP["Worker Pool"] HFPW1["Unavailable"]:::down end HFPN["Namespace"]:::down HFPP ~~~ HFPN end subgraph HFSEC["Secondary"] HFSN["Namespace
(Active)"]:::ns subgraph HFSP["Worker Pool"] HFSW1["Worker
Active"]:::worker HFSW2["Worker
Active"]:::worker HFSW3["Worker
Active"]:::worker end HFSN <-->|Workflows| HFSP end HFPN -->|"Failover"| HFSN class HFPRIM regiondown class HFSEC region class HFPP,HFSP pool ``` Here's how each component behaves during normal operation and after a failover: | Component | Normal operation | On failover | | --- | --- | --- | | **Workers** | Run in both regions. The primary region's Workers are active and process all Workflows; the secondary region's Workers stay connected and warm on standby, doing no work. Forwarding is disabled for Worker polls, so the standby fleet adds no cross-region overhead. | The secondary region's standby Workers — already connected and warm — begin processing immediately. No cold start and no DNS wait. | | **Namespace** | Active replica in the primary region; passive replica in the secondary region, continuously receiving replicated Workflow state. | The Namespace and Workers fail over together, automatically: Temporal Cloud promotes the secondary replica to active. | | **Workflow starters and Clients** | Run in both regions alongside the Workers. Disabling forwarding does not apply to them: their requests, such as Start Workflow, Signal, and Query, are still forwarded from the secondary region to the active region and succeed. | No changes needed — already running in both regions. | | **Codec Servers and proxies** | Run in both regions continuously, not just after a failover. | No changes needed — already running in the secondary region. | | **Databases and queues** | Workers in each region typically read and write only their local, active copy. | Promote the secondary region's copy to active, if needed, so the now-active Workers can read and write it. | Because a full Worker fleet is already running and warm in the secondary region, there's nothing to start or scale up before processing resumes. Along with Active/Active, this gives Active/Hot-Passive the lowest recovery time of the three patterns; because the standby fleet is already sized for full load, failover needs no scale-up. ### Benefits Active/Hot-Passive trades steady-state cost for a faster, more predictable failover: - **Easy to reason about.** - Only one region is active at a time, so traffic routing and interactions with systems (such as databases and queues) are simpler to understand, and the pattern pairs naturally with other active / passive systems. Active/Active, by contrast, requires deciding how Workers reach an active database: either a local active database in each region, or a single active / passive database that some Workers must reach cross-region. - **Lowest recovery time, tied with Active/Active.** - The secondary-region Workers are already connected and warm, so failover involves no cold start. Because the standby fleet is already sized for full load, it also needs no scale-up. - **Low latency during normal operation.** - Workflow and Activity Tasks are processed only in the active region, with no cross-region forwarding of Worker polls. Client requests that originate in the secondary region are still forwarded, and pay the cross-region hop. ### Tradeoffs That speed comes at a steady-state cost: - Highest overall architecture cost: a full standby Worker fleet runs in the secondary region at all times, even during steady state. ### Recommendations and important constraints Keep this in mind when setting up Active/Hot-Passive: - **Use Regional or VPC Endpoints and disable forwarding.** - Connect each Worker fleet through its region's [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) (or VPC Endpoint) and [disable forwarding](/cloud/high-availability/enable#change-forwarding-behavior) for Worker polls. Using the Namespace Endpoint by mistake routes the standby Workers to the active region and defeats the pattern. - **Disabling forwarding applies to Worker polls only.** Client requests, such as Start Workflow, Signal, Query, Cancel, and Terminate, are always forwarded to the active region. A Workflow starter in the secondary region keeps working, and the Workflows it starts are processed by the active region's Workers. For the full list of APIs that keep forwarding, see [Client requests are forwarded regardless of this setting](/cloud/high-availability/enable#client-requests-still-forwarded). ```mermaid flowchart LR classDef worker stroke-width:1px; classDef standby stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; subgraph HEPRIM["Primary"] HEPW["Workers
Active"]:::worker HEPEP["Regional / VPC Endpoint"]:::endpoint HEPNS["Namespace"]:::ns HEPW --> HEPEP HEPEP --> HEPNS end subgraph HESEC["Secondary"] HESW["Workers
Standby"]:::standby HESEP["Regional / VPC Endpoint"]:::endpoint HESNS["Replica"]:::ns HESW --> HESEP HESEP --> HESNS end HEPNS -. replicates .-> HESNS class HEPRIM,HESEC region ``` ## Active/Active Worker fleets run in as many regions as you want, all processing Workflows against the Namespace's single active replica through the Namespace Endpoint. If one region goes down, the others keep processing without interruption. ```mermaid --- title: Normal operation --- flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef region stroke-width:1.5px; classDef pool stroke-width:1px; subgraph APRIM["Primary"] subgraph AWP["Worker Pool"] AW1["Worker
Active"]:::worker AW2["Worker
Active"]:::worker end ANS["Namespace"]:::ns AWP <-->|Workflows| ANS end subgraph ASEC["Secondary"] AR["Replica"]:::ns subgraph AWP2["Worker Pool"] AS1["Worker
Active"]:::worker AS2["Worker
Active"]:::worker end AR <-->|Workflows| AWP2 end ANS --> AR class APRIM,ASEC region class AWP,AWP2 pool ``` ```mermaid --- title: After failover --- flowchart LR classDef worker stroke-width:1px; classDef workerhollow stroke-width:1px,stroke-dasharray:4 3; classDef ns stroke-width:1px; classDef down stroke-width:1px,stroke-dasharray:3 3; classDef region stroke-width:1.5px; classDef regiondown stroke-width:1.5px; classDef pool stroke-width:1px; subgraph AFPRIM["Primary (outage)"] subgraph AFPP["Worker Pool"] AFPW1["Unavailable"]:::down end AFPN["Namespace"]:::down AFPP ~~~ AFPN end subgraph AFSEC["Secondary"] AFSN["Namespace
(Active)"]:::ns subgraph AFSP["Worker Pool"] AFSW1["Worker
Active"]:::worker AFSW2["Worker
Active"]:::worker AFSW3["Worker
Scaled up
(as needed)"]:::workerhollow end AFSN <-->|Workflows| AFSP end AFPN -->|"Failover"| AFSN class AFPRIM regiondown class AFSEC region class AFPP,AFSP pool ``` Here's how each component behaves during normal operation and after a failover: | Component | Normal operation | On failover | | --- | --- | --- | | **Workers** | Run in as many regions as you want — fleets don't have to match the Namespace's regions. Every fleet connects through the Namespace Endpoint. | Fleets in unaffected regions keep processing with no cold-start gap. Scale them up if needed to carry the full load. | | **Namespace** | One active replica; every other region holds a passive replica that receives replicated state. The Namespace Endpoint always redirects via DNS to whichever region currently holds the active Namespace. | Temporal Cloud promotes a passive replica in another region to active. Every fleet follows automatically — no reconfiguration, nothing to bring up. | | **Workflow starters and Clients** | Run wherever convenient and connect through the Namespace Endpoint, like the Workers. | Automatically follow the Namespace Endpoint to the new active region, like the Workers. | | **Codec Servers and proxies** | Run in every region where Workers run. | Already running in every surviving region — no action needed. | | **Databases and queues** | Accessed from every Worker region, so you need a cross-region consistency story. | Promote the active region's copy, if needed, so the Workers there can read and write it. | Because surviving regions keep processing without a cold start, recovery time depends mainly on how quickly you can scale them up to absorb the extra load, not on starting new Workers. ### Benefits Active/Active spreads capacity across regions instead of parking it in a standby fleet: - **Hands-off Worker failover.** - With the Namespace Endpoint, Workers in every region follow the Namespace to the new active region automatically — there's no Worker failover step to run. - **Lowest recovery time, no standby fleet.** - Surviving regions keep processing, so there's no cold start, giving the same low recovery time as Active/Hot-Passive while spreading capacity across regions instead of parking it in a dedicated standby fleet. Surviving regions may need to scale up to absorb the failed region's load. - **Resilient to losing a region.** - Like spreading across Availability Zones, losing one region's fleet leaves the others running. ### Tradeoffs Running Workers active in multiple regions introduces new considerations: - Workers outside the active region reach it across regions (directly or via forwarding), which adds latency that can matter for latency-sensitive Workflows. - External systems are harder: Workers are active in multiple regions at once, so any databases and queues they touch need a cross-region consistency story. ### Recommendations and important constraints Keep these in mind when setting up Active/Active: - **Default to the Namespace Endpoint.** - All fleets, in any region, connect through the single Namespace Endpoint. It always routes to the active region and follows failovers automatically, so every fleet keeps reaching the active Namespace with no reconfiguration — it "just works," and Workers in all regions fail over automatically. One endpoint everywhere also keeps configuration and management simple. - **Use a Regional Endpoint only when you need the lowest recovery time.** - Connecting each fleet to its region's [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) (or VPC Endpoint) removes the DNS step from the connection path, which can shave time off failover for the lowest possible RTO. The tradeoffs: more setup, and a real risk of misconfiguration (such as routing a fleet to the wrong region). Reach for it only when you absolutely need low recovery time. With Regional Endpoints, keep forwarding enabled so passive-region polls still reach the active replica. ```mermaid flowchart LR classDef worker stroke-width:1px; classDef ns stroke-width:1px; classDef endpoint stroke-width:1px; classDef region stroke-width:1.5px; subgraph REPR1["Region 1"] REPW1["Workers"]:::worker REPE1["Regional Endpoint"]:::endpoint REPN1["Namespace
(Active)"]:::ns REPW1 --> REPE1 REPE1 --> REPN1 end subgraph REPR2["Region 2"] REPW2["Workers"]:::worker REPE2["Regional Endpoint"]:::endpoint REPN2["Replica"]:::ns REPW2 --> REPE2 REPE2 --> REPN2 end REPN2 -.->|"forwards polls"| REPN1 class REPR1,REPR2 region ``` ## Clients, Codec Servers, and databases The Worker deployment pattern sets the approach; the supporting pieces follow it. - **Workflow starters and Clients.** Deploy these with the same regional pattern as the Workers, since a starter or Client often shares the same in-region dependencies (databases, queues, upstream services) and should fail over alongside them. Point Clients at the Namespace Endpoint so they follow the active region automatically with no configuration change on failover, and use a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) only when a Client must be pinned to a region. - **Codec Servers and proxies.** Anything in the connection path between Workers and Temporal Cloud must be reachable from every region where Workers connect. In Active/Passive, scale them up in the secondary region as part of a failover; in the Active/Hot-Passive and Active/Active patterns, run them in both regions at all times. - **Databases and queues.** These remain the application's responsibility, and the right approach depends on the Worker deployment pattern: a single-region-active datastore pairs naturally with the Active/Passive and Active/Hot-Passive patterns, while running Workers active in both regions (as in Active/Active) raises consistency questions that must be designed for. Detailed guidance is out of scope for this page. ## Frequently asked questions ### What is the difference between active-passive and active-active patterns? Both **Active/Passive** and **Active/Hot-Passive** keep Workflows processing in one region at a time, with the other region standing by for failover. **Active/Active** runs Workers in every region and processes Workflows in all of them at once. See [Worker deployment patterns](#ha-worker-patterns) for the full comparison. ### How do I fail over Workers to another region? A Namespace with High Availability fails over automatically, but bringing up or activating Workers in the secondary region is your responsibility. The exact steps depend on your pattern; see [Active/Passive](#active-cold), [Active/Hot-Passive](#active-hot), and [Active/Active](#active-active). ### Which pattern has the lowest recovery time (RTO)? **Active/Hot-Passive** and **Active/Active** both achieve the lowest recovery time, because neither requires a cold start after failover. In Active/Hot-Passive, a warm standby Worker fleet in the secondary region begins processing the moment it becomes active. In Active/Active, Workers are already processing in every region, so the surviving regions keep running with no gap. By contrast, **Active/Passive** must cold start a Worker fleet in the secondary region, giving it the highest recovery time. See [Active/Hot-Passive](#active-hot), [Active/Active](#active-active), and [RPO and RTO](/cloud/rpo-rto). ### Do I have to run Workers in both regions for high availability? No. **Active/Passive** runs Workers in one region at a time and is the simplest starting point for disaster recovery. Running Workers in both regions — **Active/Hot-Passive** or **Active/Active** — lowers recovery time at higher cost. ### Does Temporal Cloud support Active/Active for HA/DR? Yes, as a Worker deployment pattern — not as a database-level active/active. A Temporal Cloud Namespace with High Availability always has exactly one active region and one passive replica underneath, no matter which [Worker deployment pattern](#ha-worker-patterns) you choose. But because Workers in any region reach the active replica through the Namespace Endpoint, you can run Worker fleets in every region and process Workflows in all of them at once. See [Active/Active](#active-active). ### What special patterns are needed for multi-cloud HA/DR? None specific to multi-cloud. Multi-region and multi-cloud HA/DR use the same [Worker deployment patterns](#ha-worker-patterns) — the secondary region can be any [Temporal Cloud region](/cloud/regions) that supports replication from the primary, whether in the same cloud provider or a different one. The same considerations apply: route Workers through the right [endpoints and private connectivity](/cloud/high-availability/ha-connectivity), and give any databases and queues a cross-region — here, cross-cloud — consistency story. If you use the Active/Passive pattern, it is highly recommended that you ensure Workers in the active cloud can reach the Namespace if it fails over to the passive cloud due to a Temporal-specific outage, so they can keep processing Tasks across the cross-cloud path. ### What special considerations are there for private networking? The patterns are the same. But if you use the Active/Passive pattern, it is highly recommended that you provide a network path for Workers in the active region to reach the VPC Endpoint in the passive region, so they can keep processing Tasks if the Namespace fails over due to a Temporal-specific outage. See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity). --- # Enable and manage High Availability Source: https://docs.temporal.io/cloud/high-availability/enable > Add a replica to a Namespace to enable High Availability, then manage forwarding behavior and automatic failover settings. You can enable [High Availability](/cloud/high-availability) features for a new or existing Namespace by adding a replica. When you add a replica, Temporal Cloud begins asynchronously replicating ongoing and existing Workflow Executions. Adding a replica fails the Namespace over automatically; to plan how your Workers fail over with it, see [Worker deployment patterns for high availability and disaster recovery](/cloud/high-availability/architecture-patterns). The replica region must be on the same continent as the primary region. Because of that, not all replication options are available in all Temporal Cloud regions. See the [Service regions](/cloud/regions) page for the supported replica regions for each active region. Using private network connectivity with a HA namespace requires extra setup. See [Connectivity for HA](/cloud/high-availability/ha-connectivity). There are charges associated with Replication and enabling High Availability features. For pricing details, visit Temporal Cloud's [Pricing](/cloud/pricing) page. ## Create a Namespace with High Availability features To create a new Namespace with High Availability features, you can use the Temporal Cloud UI or the command line utility. **Web UI** 1. Visit Temporal Cloud in your Web browser. 1. During Namespace creation, specify the primary [region](/cloud/regions) for the Namespace. 1. Select "Add a replica". 1. Choose the [region](/cloud/regions) for the replica. The web interface will present an estimated time for replication to complete. This time is based on your selection and the size and scale of the Workflows in your Namespace. **Temporal CLI** At the command line, enter: ``` temporal cloud namespace create \ --name \ --region \ --region ``` Specify the [region codes](/cloud/regions) as arguments to the two `--region` flags. **tcld** At the command line, enter: ``` tcld namespace create \ --namespace . \ --region \ --region ``` Specify the [region codes](/cloud/regions) as arguments to the two `--region` flags. If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace create`. Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use. ## Add High Availability to an existing Namespace A replica can be added after a namespace has already been created. **Web UI** 1. Visit Temporal Cloud Namespaces in your Web browser. 1. Navigate to the Namespace details page. 1. Select the “Add a replica” button. 1. Choose the [region](/cloud/regions) for the replica. The web interface will present an estimated time for replication to complete. This time is based on your selection and the size and scale of the Workflows in your Namespace. Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use. **Temporal CLI** At the command line, enter: ``` temporal cloud namespace ha region add \ --namespace . \ --region ``` Specify the region ID (for example, `aws-us-east-1`) of the region where you want to create the replica as an argument to the `--region` flag. See [Regions](/cloud/regions) for available region names. Temporal Cloud sends an email alert once your Namespace is ready for use. **tcld** At the command line, enter: ``` tcld namespace add-region \ --namespace . \ --region ``` Specify the region name (for example, `us-east-1`) of the region where you want to create the replica as an argument to the `--region` flag. See [Regions](/cloud/regions) for available region names. If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace add-region`. Temporal Cloud sends an email alert once your Namespace is ready for use. ## Change a replica location Temporal Cloud can't change replica locations directly. To change a replica's location, you need to remove the replica and add a new one. > **⚠️ Caution:** > > We discourage changing the location of your replica for deployed applications, except under exceptional circumstances. > If you remove your replica, you lose the availability guarantees of the Namespace, and it can take time to add another > replica. > > If you remove a replica from a region, you must wait seven days before you can re-enable High Availability (HA) in that > same location. During this period, you may add a replica to a different region, provided you have not had one > there within the last seven days. > Follow these steps to change the replica location: 1. [Remove your replica](#disable). This disables High Availability for your Namespace. 2. [Add a new replica](#upgrade) to your Namespace. You will receive an email alert once your Namespace is ready for use. ## Change the forwarding behavior Requests that reach the passive replica can be [forwarded](/cloud/high-availability/#request-forwarding) to the active region, and responses sent back to the Worker or Client. The `disablePassivePollerForwarding` Namespace setting controls this behavior for Worker poll traffic only. With `disablePassivePollerForwarding` enabled, Worker polls that reach a passive replica are not forwarded, and these Workers do not execute Workflows or Activities. Workers connected to such a passive replica receive a `NamespaceNotActive` error on poll requests. These Workers stay connected and will start executing Workflows and Activities if the replica becomes active. Same-region replicas are not affected by this setting. To deploy Worker fleets in both regions that stay on standby in the passive region until failover, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot). Set `disablePassivePollerForwarding` through the [Cloud Ops API](/ops), the [cloud-api SDK](https://github.com/temporalio/cloud-sdk-go), or the [`temporal cloud` CLI extension](/cli/cloud), using one of the recipes in this section. > **ℹ️ Info:** > > To see which endpoints route to which replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica). > ### Client requests are forwarded regardless of this setting `disablePassivePollerForwarding` stops Worker polls. It does not stop Client requests. The following APIs are forwarded from the passive replica to the active region whether or not the setting is enabled, with responses returned to the Client: | API group | Forwarded operations | | --- | --- | | Workflow | Start Workflow, Signal-with-Start, Signal, Cancel, Terminate, Delete, Query | | [Standalone Activity](/standalone-activity) | Start, Cancel, Terminate, Delete, Pause, Unpause, Reset, Update Activity Options | | [Standalone Nexus Operation](/standalone-nexus-operation) | Start, Cancel, Terminate, Delete | For the current list, see [`selectedAPIsForwardingRedirectionPolicyWhitelistedAPIs`](https://github.com/temporalio/temporal/blob/main/common/rpc/interceptor/dc_redirection_policy.go#L59-L71) in the Temporal Server source. A Client that reaches the passive replica, for example through the passive region's Regional Endpoint, keeps working while the setting is enabled: it can start Workflows, send Signals, and run Queries successfully. Workers in the active region process the resulting Workflow and Activity Tasks, because the passive region's Workers receive none. Plan for this if you disable forwarding to keep a region's traffic out of the active region. The setting stops that region's Workers from processing tasks. It does not stop Workflow starters, Signal senders, or other Clients in that region from reaching the Namespace. ### Set the forwarding behavior with the `temporal cloud` CLI Use the [`temporal cloud namespace ha update`](/cli/command-reference/cloud/namespace#ha-update) command: ```bash temporal cloud namespace ha update \ --namespace . \ --passive-poller-forwarding disabled ``` Set the flag to `enabled` to re-enable forwarding. ### Set the forwarding behavior with the Cloud Ops API This recipe uses `curl` against the Cloud Ops API. Because `UpdateNamespace` replaces the entire Namespace spec, it fetches the current spec, merges in the new value, and posts it back with the current `resourceVersion`. The `jq` filter preserves any other High Availability fields you have set (such as `disableManagedFailover`). Set the Cloud Ops API key and Namespace ID. The Namespace ID is in `.` format (the full identifier shown in the Cloud Web UI): ```bash export TEMPORAL_CLOUD_OPS_API_KEY='' export NS='.' ``` Fetch the current spec: ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" > /tmp/ns.json ``` Build the update payload. Set the value to `true` to disable forwarding, or `false` to restore the default: ```bash jq --arg rv "$(jq -r '.namespace.resourceVersion' /tmp/ns.json)" '{ spec: (.namespace.spec | .highAvailability = ((.highAvailability // {}) + {disablePassivePollerForwarding: true})), resourceVersion: $rv }' /tmp/ns.json > /tmp/ns-update.json ``` Post the update. The response contains an `asyncOperation` ID; the change is complete when `GET /cloud/operations/` reports a terminal state. ```bash curl -sS -X POST "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" \ -H "Content-Type: application/json" \ -d @/tmp/ns-update.json ``` Verify the current value: ```bash curl -sS "https://saas-api.tmprl.cloud/cloud/namespaces/$NS" \ -H "Authorization: Bearer $TEMPORAL_CLOUD_OPS_API_KEY" \ | jq '.namespace.spec.highAvailability.disablePassivePollerForwarding' ``` A result of `null` means the field has never been set, which is equivalent to `false` — proto3 JSON omits default-`false` values from responses. ## Enable or disable automatic failovers When a Temporal Cloud Namespace has a replica in a different region or cloud, Temporal Cloud automatically fails over the Namespace to its replica in the event of an outage. _This is the recommended and default option._ If you prefer to disable automatic failovers and handle your own failovers, follow these instructions: > **⚠️ Warning:** > Disabling automatic failovers voids Temporal's RTO > > With automatic failovers disabled, Temporal Cloud cannot fail your Namespace over to its replica during an outage. You take responsibility for detecting outages and [triggering a failover](/cloud/high-availability/failovers/manage#trigger-failover) yourself. Temporal's [20-minute RTO](/cloud/rpo-rto) does not apply while this setting is disabled. > **Web UI** 1. Navigate to the Namespace detail page in Temporal Cloud. 1. Choose the "Disable Temporal-initiated failovers" option. **Temporal CLI** To disable automatic failovers, run the following command in your terminal: ``` temporal cloud namespace ha update \ --namespace . \ --auto-failover disabled ``` **tcld** To disable automatic failovers, run the following command in your terminal: ``` tcld namespace update-high-availability \ --namespace . \ --disable-auto-failover=true ``` If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace update-high-availability`. To restore the default behavior, unselect the option in the Web UI, pass `--auto-failover enabled` with the Temporal CLI, or pass `--disable-auto-failover=false` with `tcld`. > **📝 Note:** > Automatic failovers are always enabled for Same-region Replication > > This setting applies only to Multi-region and Multi-cloud Replication. You cannot disable automatic failovers for a [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace, because same-region failovers between cells are always managed by Temporal. > ## Disable High Availability (remove a replica) To disable High Availability features on a Namespace, remove the replica from that Namespace. Removing a replica disables all High Availability features: - Discontinues replication of the Workflows in the Namespace. - Disables the Namespace's ability to trigger a failover to a different region or cloud. - Ends High Availability charges. > **⚠️ Caution:** > > After removing a Namespace's replica, you cannot add a new replica to that same region for seven days. > During that time, you can still add a replica to any other region. > Follow these steps to remove a replica from a Namespace: **Web UI** 1. Navigate to the Namespace details page in Temporal Cloud 1. Select the option to "Remove Replica" on the "Region" card. **Temporal CLI** Run the following command to remove the replica: ``` temporal cloud namespace ha region delete \ --namespace . \ --region ``` See [Regions](/cloud/regions) for available region names. **tcld** Run the following command to remove the replica: ``` tcld namespace delete-region \ --api-key \ --namespace . \ --region ``` See [Regions](/cloud/regions) for available region names. --- # Failovers Source: https://docs.temporal.io/cloud/high-availability/failovers > How automatic and manual failovers work with Temporal Cloud High Availability. When a Namespace with [High Availability](/cloud/high-availability) is disrupted by an outage, Temporal Cloud can fail over the Namespace from the primary to the replica. This lets in-flight Workflow Executions continue, new Workflow Executions start, and closed Workflow Executions be inspected, all with minimal interruptions or data loss. Returning control from the replica to the primary is called a failback. After an automatic failover, Temporal automatically fails back to the original region once it is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). See [Failbacks](/cloud/high-availability/failovers/manage#failbacks) for details. ## Automatic failover Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage and complete a failover in minutes from when the outage began, according to the stated [Recovery Time Objective (RTO)](/cloud/rpo-rto). After an automatic failover, the Namespace will have a replica in its original region. Once the original region is healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks), moving the Namespace back to its original region. ![On failover, the replica becomes active and the Namespace endpoint directs access to it.](/img/cloud/high-availability/failover.png) To opt out of automatic failovers and their RTO, you can [disable automatic failovers](/cloud/high-availability/enable#automatic-failovers). ### Conditions that trigger an automatic failover While the failover operation itself usually completes in seconds, the bulk of the Recovery Time in an outage is spent detecting the disruption and deciding to trigger a failover. See [The failover process](#failover-process) for a detailed breakdown. Temporal Cloud runs automated Workflows that detect outages and trigger failovers. These Workflows continuously monitor the health of Temporal Cloud in every region and every cell. If any of the monitored conditions are failing for too long, Temporal Cloud automatically triggers a failover on any Namespaces with High Availability that have a healthy replica. Temporal's on-call engineers may also trigger a failover at their discretion, for example, if they see early signs of a regional outage. > **ℹ️ Info:** > > The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive > list, and it may change over time. > - Whether Temporal Cloud's services in the cell are reachable from the Control Plane. - The average latency of inbound RPC calls (excluding long-polling APIs) to Temporal services in the cell. - The percentage of inbound RPC calls that returned errors related to server health. - The average latency of calls from Temporal Cloud's services in the cell to its persistence layer. - The percentage of calls to the persistence layer that returned errors related to persistence health. ## Manual failover You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (that is, a failover triggered by a user) is warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover process with your Clients and Workers, or to move a Namespace to a different region. - **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the outage with your own monitoring and trigger a failover yourself. - **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover. > **📝 Note:** > Same-region Replication > > Manual failovers apply only to Multi-region and Multi-cloud Replication. A > [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between > cells and cannot be failed over manually or have its automatic failovers disabled. > ## The failover process The failover process is the same whether it is triggered automatically by Temporal or manually by a user. 1. **During normal operation**, the primary asynchronously replicates data to the replica, keeping them in sync. 2. **A failover is triggered.** For automatic failovers, the majority of time is spent on outage detection. Temporal's automated health checks must confirm the disruption before initiating a failover. For the overall timing target, see the [Recovery Time Objective (RTO)](/cloud/rpo-rto). 3. **The Namespace becomes active in the replica's region.** 1. Temporal Cloud first attempts a _graceful failover_: it pauses traffic, drains in-flight replication, and switches to the replica with no data conflicts. 2. If the graceful attempt does not complete within 10 seconds, Temporal Cloud falls back to a _forced failover_, which immediately activates the replica. In a forced failover, any events not yet replicated undergo [conflict resolution](#conflict-resolution) once the original region comes back. 3. This hybrid strategy balances consistency and availability. During the switch, Workflow operations are briefly paused, and Temporal Cloud returns a retryable "Service unavailable" error to SDKs. 4. **The Namespace Endpoint redirects via DNS to the active region.** This change can take a few minutes to fully propagate to all Clients and Workers. If your application has an extremely demanding Recovery Time, you can eliminate this stage by connecting through a [Regional Endpoint](/cloud/high-availability/ha-connectivity#regional-endpoint) instead of the Namespace Endpoint. 5. **Failback.** If the failover was triggered by Temporal, Temporal automatically triggers a failback to the original region once the region is healthy. If the failover was triggered by a user, the Namespace continues as-is until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. ## Post-failover events After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, the Namespace stays in the replica region until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. ## Split-brain scenario At any time, only the primary or the replica should be active. However, if a network partition separates the two regions, the regions cannot communicate with each other. If you promote the replica to active during a network partition, both regions will be active simultaneously, accepting writes independently. This is known as a split-brain scenario. When the network partition resolves and the regions can communicate again, Temporal's [conflict resolution](#conflict-resolution) process reconciles the divergent histories and determines which region remains active. ## Conflict resolution Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to replication lag, particularly during failovers. In the event of a non-graceful failover, replication lag causes a temporary setback in Workflow progress. At the moment of non-graceful failover: - Operations that had already replicated remain durable in the replica. - Operations that had not yet replicated (that is, that are still in the replication backlog) are reconciled when the region recovers, according to the conflict resolution process. > **⚠️ Caution:** > Conflict resolution requires a recoverable region > > Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never > recovers, Workflow API calls that fall within the [RPO](/cloud/rpo-rto) — under one minute — may be permanently lost. > Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history > of Temporal Cloud. > In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a retry policy's [maximum attempts](/encyclopedia/retry-policies#maximum-attempts) is set to 0. High Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be re-dispatched in a newly active Namespace, leading to repeated executions. The same durability boundary applies to Workflow starts, Signals, and Updates: a `StartWorkflowExecution`, `SignalWorkflowExecution`, `SignalWithStartWorkflowExecution`, or `UpdateWorkflowExecution` call that returns success is durably committed in the active region, and replicated asynchronously to the replica. ### How Workflow Id uniqueness is preserved after a forced failover The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid#workflow-id) — at most one Open Workflow Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. This guarantee limits how many Executions are _Open_ at the same time; reuse of a Workflow Id after an Execution Closes is governed separately by the [Workflow Id Reuse Policy](/workflow-execution/workflowid-runid#workflow-id-reuse-policy), and a start request that collides with an already-Open Execution is governed by the [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy). Because the guarantee constrains only concurrency, and not how many [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow Id twice concurrently. 1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event History to the replica. 2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently append events under the same Workflow Id. When the regions reconnect, their Event Histories have diverged for that Workflow Id. 3. **One Execution stays Open.** Temporal Cloud does not interleave the divergent histories. Events from the previously active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History into a new branch. Its conflict resolution process then keeps a single Workflow Execution Open. The competing Execution in the previously active region becomes a [zombie Workflow Execution](/temporal-service/multi-cluster-replication#zombie-workflows) — an Execution that region can no longer mutate on its own — and is terminated there once replication informs it of the competing Workflow Id. The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs. --- # Manage failovers Source: https://docs.temporal.io/cloud/high-availability/failovers/manage > Trigger, configure, and test failovers for Temporal Cloud High Availability Namespaces. ## Trigger a failover You can trigger a failover manually using the Temporal Cloud Web UI, the CLI, or the Cloud Ops API. Manual failovers apply only to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually. > **⚠️ Warning:** > Check your replication lag > > Always check the replication lag before initiating a failover. A forced failover when there is a > significant replication lag has a higher likelihood of rolling back Workflow progress. > **Web UI** 1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI. 1. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu. 1. Confirm your action. After confirmation, Temporal initiates the failover. **Temporal CLI** To manually trigger a failover, run the following command in your terminal: ``` temporal cloud namespace ha failover \ --namespace . \ --region ``` The `` must be the ID of a region (example: `aws-us-east-1`) where the Namespace has a replica that is ready to be failed over to (replica state is `Activated`). **tcld** To manually trigger a failover, run the following command in your terminal: ``` tcld namespace failover \ --namespace . \ --region ``` The `` must be the name of a region (example: `us-east-1`) where the Namespace has a replica that is ready to be failed over to (replica state is `Activated`). If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace failover`. **Cloud Ops API** You can trigger a failover programmatically using the [Cloud Ops API](/ops). The API is available via both HTTP and gRPC. **Using HTTP** Send a POST request to the [`FailoverNamespaceRegion`](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/{namespace}/failover-region) endpoint: ``` POST https://saas-api.tmprl.cloud/cloud/namespaces//failover-region ``` Request body: ```json { "region": "", "asyncOperationId": "" } ``` - `region` (required): The [region code](/cloud/regions) of the region to failover to. Must be a region where the Namespace has a replica in `Activated` replica state, indicating the replica is ready to be failed over to. Example: `aws-us-east-1` - `asyncOperationId` (optional): A user-defined ID for tracking the async operation. If not set, the server will assign one. **Using gRPC** Use the [`FailoverNamespaceRegion`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService.FailoverNamespaceRegion) RPC with a [`FailoverNamespaceRegionRequest`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.FailoverNamespaceRegionRequest): ```protobuf message FailoverNamespaceRegionRequest { // The namespace to failover. string namespace = 1; // The id of the region to failover to. // Must be a region that the namespace is currently available in. string region = 2; // The id to use for this async operation - optional. string async_operation_id = 3; } ``` Both methods return a [`FailoverNamespaceRegionResponse`](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.FailoverNamespaceRegionResponse) containing an async operation that you can use to track the failover status. > **ℹ️ Info:** > Terraform not supported > > The [Temporal Cloud Terraform provider](https://registry.terraform.io/providers/temporalio/temporalcloud/latest) does > not support triggering failovers. You must use the Web UI, the CLI, or the Cloud Ops API. > Once the failover async operation returns successfully, the Namespace will be failed over. Temporal manages retries for the failover Workflow. In the rare event that an internal error prevents the failover from completing, the Temporal on-call team is automatically paged to intervene and force the failover to completion. ## Return to the primary with failbacks Failback behavior depends on whether the failover was automatic or manually triggered. ### After an automatic failover After an automatic failover, Temporal Cloud automatically fails back to the original region once the region is healthy. No action is required from you. Follow [Temporal's status page](https://status.temporal.io) for updates on the original region's health. If you prefer to manage failback yourself, you have two options: - **Opt out of automatic failback (manage failback manually):** After the automatic failover has completed, [disable automatic failovers](/cloud/high-availability/enable#automatic-failovers) on the Namespace to prevent Temporal from automatically failing back. When you're ready to return to the original region, [trigger a failover](#trigger-failover) to that region and then re-enable automatic failovers. - **Stay on the new region permanently ("fail forward"):** After the automatic failover has completed, [trigger a failover](#trigger-failover) to the region that is already active. This tells Temporal that you want to treat the new region as your primary for as long as it's healthy. Automatic failovers remain enabled, so Temporal will still protect you if the new region has an outage. ### After a user-triggered failover If you triggered a failover yourself during an outage (instead of relying on an automatic failover), Temporal will _not_ automatically fail back for you. You must [trigger a failover](#trigger-failover) back to the original region when it is healthy. Monitor [Temporal's status page](https://status.temporal.io) for updates on region health. Automatic failback is only available when the most recent failover was automatic. ### How to check whether your Namespace will be automatically failed back If you are not sure whether your Namespace will be automatically failed back, check the list of failovers in the Temporal Cloud Web UI on your Namespace's detail page: - If the most recent failover was **automatic**, then Temporal will fail the Namespace back when the original region is healthy. - If the most recent failover was **user-triggered**, then the Namespace will _not_ be automatically failed back. You must trigger the failback yourself. ## Workers and failovers Enabling High Availability for Namespaces does not require specific Worker configuration. When a Namespace fails over to the replica, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. Temporal Cloud forwards their requests from the passive replica to the active region and the responses back, so Workers keep running through a failover. To choose where your Worker fleets run across regions, see [Deployment patterns for High Availability](/cloud/high-availability/architecture-patterns). To route Workers to the passive region's replica, see [How requests reach the replica](/cloud/high-availability/ha-connectivity#how-requests-reach-the-replica). To stop forwarding Worker polls to the active region, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior). To disable automatic failovers, see [Enable or disable automatic failovers](/cloud/high-availability/enable#automatic-failovers). When a Namespace fails over to a replica in a different region, Workers will be communicating cross-region. - If your application cannot tolerate this latency, deploy a second set of Workers in the replica's region or opt for a replica in the same region. - In the case of a complete regional outage, Workers in the original region may fail alongside the original Namespace. To keep Workflows moving during this level of outage, deploy a second set of Workers to the secondary region. Temporal Cloud enforces a maximum connection lifetime of 5 minutes, which gives your Workers an opportunity to re-resolve the DNS. ## Test failovers Temporal recommends regular failover testing for mission-critical applications in production. By testing in non-emergency conditions, you verify that your application continues to function even when parts of the infrastructure fail. Because failover testing relies on manually triggering a failover, it applies to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually for testing. > **💡 Tip:** > > If this is your first time performing a failover test, run it with a test-specific Namespace and application. Practice > runs help ensure the process runs smoothly during real incidents in production. > Failover testing (also known as "trigger testing") can: - **Validate replicated deployments:** In multi-region setups, failover testing ensures your application can run from another region when the primary region experiences outages. - **Assess replication lag:** In multi-region deployments, monitoring [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) between regions is important. Check the lag before initiating a failover to avoid rolling back Workflow progress. - **Assess recovery time:** Manual testing helps you measure actual recovery time and check if it meets your expected [Recovery Time Objective (RTO)](/cloud/rpo-rto). - **Identify potential issues:** Failover testing uncovers problems not visible during normal operation, including issues like [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how external dependencies behave during a failover event. - **Operational readiness:** Regular testing familiarizes your team with the failover process, improving their ability to handle real incidents. --- # Connectivity for High Availability Source: https://docs.temporal.io/cloud/high-availability/ha-connectivity > How to choose endpoints and configure private network connectivity for Namespaces with Temporal Cloud High Availability features. A Namespace with High Availability features spans two regions, and the endpoint your Workers and Clients connect through determines how they behave before, during, and after a failover. This page covers: - How to choose between the Namespace Endpoint and a Regional Endpoint for a Namespace with High Availability features. - How to configure PrivateLink so that failover remains transparent to Workers on private networks. ## How to choose an endpoint for a Namespace with High Availability features Temporal Cloud exposes two kinds of gRPC endpoints for a Namespace. See [How to access a Namespace](/cloud/namespaces#access-namespaces) for the general definitions; this section focuses on how each behaves with replication and failover. ### Namespace Endpoint (recommended) Format: `..tmprl.cloud:7233` The Namespace Endpoint always connects to whichever region is currently active. Under the hood, it is a CNAME that points at the active region's Regional Endpoint. When Temporal Cloud fails the Namespace over, it updates the CNAME to point at the new active region. The DNS TTL is 15 seconds, so Clients converge within about 30 seconds with no configuration change on your side. Use the Namespace Endpoint unless you have a specific reason to pin traffic to a region. ### Regional Endpoint Format: `-.region.tmprl.cloud:7233` (for example, `aws-us-west-2.region.tmprl.cloud` or `gcp-us-central1.region.tmprl.cloud`). See [regions](/cloud/regions) for the full list. A Regional Endpoint is shared across every Namespace that is active or replicated in that region. Unlike the Namespace Endpoint, a Regional Endpoint stays pinned to the region in its name — if that region holds the passive replica of your Namespace, the Regional Endpoint connects to the passive replica. Use a Regional Endpoint only when you need explicit control over which replica a Client or Worker reaches. Trade-offs to consider: - **Faster recovery.** A Worker connecting through a Regional Endpoint skips the DNS step that Clients on the Namespace Endpoint wait for during a failover. This removes the ~30-second DNS convergence window from the recovery path, which is useful for Workloads that must minimize [Recovery Time](/cloud/rpo-rto) at all costs. - **You are responsible for regional coverage.** A Worker using the Regional Endpoint of a region cannot reach the Namespace if that region is in an outage. To stay available through a failover, you must run Workers that use the **replica** region's Regional Endpoint — Workers pointed only at the outage region's Regional Endpoint will not reconnect automatically. When authenticating with mTLS, set the Client's `server_name` / `serverNameOverride` config equal to the Namespace Endpoint. This overrides the SNI that the Client will expect during the TLS Handshake with Temporal Cloud. The Regional Endpoint forwards the request to your Namespace, so the Client must expect the Namespace's certificate during the TLS handshake. For example, in Typescript, the Client's config would be set like this: ```typescript await Connection.connect({ address: 'aws-us-east-1.region.tmprl.cloud:7233', tls: { serverNameOverride: 'my-namespace.my-account.tmprl.cloud', clientCertPair: { crt: clientCert, key: clientKey }, }, ... }); ``` ### How endpoints route on failover Consider a Namespace replicated across `us-east-1` (initially active) and `us-west-2` (initially the replica), with a failover that swaps the two. | Client connects via | Before failover | After failover | | ------------------------------------------- | --------------- | ----------------------- | | Namespace Endpoint | `us-east-1` | `us-west-2` (automatic) | | Regional Endpoint `aws-us-east-1.region...` | `us-east-1` | `us-east-1` | | Regional Endpoint `aws-us-west-2.region...` | `us-west-2` | `us-west-2` | The Namespace Endpoint moves with the active region via an updated CNAME — no Client changes required. The Regional Endpoints do not change their targets on failover: each continues to route to the replica that lives in its region. ## How requests reach the replica A request can reach the passive replica in three ways: - **Through the passive region's Regional Endpoint.** A [Regional Endpoint](#regional-endpoint) is pinned to its region, so the Regional Endpoint of the region that currently holds the passive replica connects to the passive replica. - **Through a PrivateLink or Private Service Connect endpoint in the passive region.** A VPC Endpoint or PSC endpoint in the passive region routes to the passive replica. - **Through the Namespace Endpoint during a failover.** When a Namespace fails over, two things happen in parallel: 1. The replica becomes active, and the former active becomes a replica. 2. The Namespace Endpoint changes to point at the replica's region, and the Worker re-resolves the Namespace Endpoint to connect to the new active region via DNS. If #1 completes before #2, a Worker that was connected to the former active before the failover will stay connected to it even after it becomes the replica. The Worker will change to point at the new active when the DNS changes propagate and the Client re-resolves DNS (typically 30 seconds, though up to 5 minutes, bounded by Temporal Cloud's maximum connection lifetime). By default, Temporal Cloud transparently forwards any request that reaches the passive replica to the active region, and the response back. You can turn forwarding off for Worker polls, but Client requests such as Start Workflow, Signal, and Query are always forwarded. To learn what forwarding does, see [Request forwarding](/cloud/high-availability/#request-forwarding). To stop forwarding Worker polls on a Namespace, see [Change the forwarding behavior](/cloud/high-availability/enable#change-forwarding-behavior). To run Worker fleets in both regions that rely on this forwarding, see [Active/Active](/cloud/high-availability/architecture-patterns#active-active). To keep passive-region Workers on standby until failover by disabling this forwarding, see [Active/Hot-Passive](/cloud/high-availability/architecture-patterns#active-hot). ## How to use PrivateLink with High Availability features > **💡 Tip:** > > Proper networking configuration is required for failover to be transparent to Clients and Workers when using PrivateLink. > This section describes how to configure routing for Namespaces with High Availability features on AWS PrivateLink. > These instructions assume you already have the private connections in place. If not, follow the [AWS PrivateLink](/cloud/connectivity/aws-connectivity) or [GCP Private Service Connect](/cloud/connectivity/gcp-connectivity) creation guides first. ## How HA + private connectivity works A Namespace with High Availability features has two replicas — a primary and a secondary, in different regions or different cloud providers. At any moment, one is **active** and one is **passive**. On failover, Temporal Cloud changes the active replica. Temporal Cloud expresses the active replica through DNS: - The Namespace DNS record (`..tmprl.cloud`) is a CNAME. - It points to the active region's regional record (`-.region.tmprl.cloud`). - On failover, Temporal Cloud rewrites the CNAME target. Namespace DNS records have a 15-second TTL. Clients should converge to the new region within roughly 30 seconds (about twice the TTL) once their resolver cache expires. > **📝 Note:** > Deterministic DNS behavior is unique to HA Namespace Endpoints > > This is the **only** place in Temporal Cloud where you can depend on Temporal-managed DNS to behave in a specific, deterministic way — a Namespace Endpoint on an HA Namespace CNAMEing to its active region's regional record. Everywhere else, take a dependency on Temporal's published endpoints (the hostnames themselves), not on what they resolve to. The underlying IP addresses, CNAME chains, and resolution behavior of non-HA endpoints can change at any time without notice. > For private connectivity, your job is to make sure that: - Override the Regional Endpoint's DNS zone to resolve to a VPC Endpoint. - Ensure network connectivity between the two regions. > **⚠️ Warning:** > Do not override the Namespace Endpoint in your private hosted zone > > For HA Namespaces, the PHZ must override only the regional records (`-.region.tmprl.cloud`) — never the Namespace Endpoint itself (`..tmprl.cloud`). > > If the PHZ holds a record for the Namespace Endpoint, the resolver answers from the PHZ before consulting public DNS, so Temporal Cloud's active-region CNAME is never followed. On failover, Workers keep resolving to the old (now passive) region's VPC Endpoint and never reach the new active region. > > This matters most when **enabling HA on a Namespace that previously used the [single-region PHZ pattern](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink)**, where the Namespace Endpoint itself was the overridden name. See [How to enable HA on a Namespace using Private Connectivity](#how-to-enable-ha-on-a-namespace-using-private-connectivity) below for the migration steps. > > **⚠️ Warning:** > Do not attach a Stable IPs public Connectivity Rule > > If you attach a public [Connectivity Rule with Stable IPs](/cloud/connectivity/ip-addresses#stable-ip-addresses) to a Namespace, the Namespace Endpoint resolves to a public Stable IP instead of to `-.region.tmprl.cloud`. Stable IPs DNS behavior supersedes the regional DNS behavior described here, so the Namespace Endpoint's DNS resolution will not work in the way the Private Hosted Zone needs. To keep HA + Private Connectivity working, do not attach a Stable IPs public Connectivity Rule to that Namespace. > ## How to enable HA on a Namespace using Private Connectivity: changing private DNS overrides from single-region to multi-region If you are turning on [High Availability features](/cloud/high-availability/enable) on a Namespace that already uses AWS PrivateLink or GCP Private Service Connect, the existing private DNS setup almost certainly needs to change before failover will work. The common single-region private DNS pattern (described in the [AWS PrivateLink guide](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink) and the [GCP PSC guide](/cloud/connectivity/gcp-connectivity)) overrides the **Namespace Endpoint** directly. That pattern short-circuits Temporal Cloud's regional CNAME and prevents failover from working — see the warning above. Follow these steps in order to update your private DNS overrides without interrupting traffic: 1. **Inventory the existing PHZ records.** List the records in your private hosted zone for `tmprl.cloud`. Note any CNAME (or A record) for `..tmprl.cloud` — that is the single-region override you'll be removing. 2. **Add regional records for both the source and target HA regions.** Before removing anything, create: - `aws-.region.tmprl.cloud` → source-region VPC Endpoint - `aws-.region.tmprl.cloud` → target-region VPC Endpoint (Or the GCP Cloud DNS equivalents — see [GCP PSC](#single-cloud-ha-on-gcp-private-service-connect) below.) These records are additive and do not yet affect resolution of `..tmprl.cloud`, because the Namespace-endpoint override still short-circuits the chain. 3. **Confirm both VPC Endpoints are reachable from your Worker VPCs.** From a Worker host, `dig` the new regional names and confirm they resolve to the right VPC Endpoint. Also verify the network path actually works in both regions (security groups, route tables, cross-region connectivity). 4. **Enable HA on the Namespace.** Follow [Enable High Availability features](/cloud/high-availability/enable). Temporal Cloud creates the replica and starts replicating. 5. **Remove the Namespace-endpoint PHZ record.** Delete the `..tmprl.cloud` record from the PHZ. With it gone, Workers resolve the name through public DNS → regional CNAME → PHZ regional override → VPC Endpoint, which is the correct HA chain. **Do not skip this step.** If the Namespace-endpoint override remains, failover does not work. 6. **Test failover end-to-end.** Use [forced failover](/cloud/high-availability/failovers) in a staging environment to confirm Workers converge to the new active region within the expected window (about 30 seconds after the public CNAME update, given the 15-second TTL). > **⚠️ Caution:** > Order matters > > Adding the regional records first (step 2) and removing the Namespace-endpoint record last (step 5) means Workers always have a working DNS resolution. Reversing the order leaves a window where Workers cannot resolve the Namespace Endpoint at all. > ## How to migrate to another Temporal Cloud Region when using Private Connectivity To move a Namespace to a new Temporal Cloud Region while keeping Private Connectivity in place, the recommended pattern is to use a **separate private hosted zone in each region**, each overriding the Namespace Endpoint to point at that region's own VPC Endpoint. Because a PHZ is scoped only to the VPCs it is associated with, Workers in each region resolve the Namespace Endpoint to their local VPC Endpoint and traffic stays in-region throughout the move. This is not the only way to handle a region change with Private Connectivity, but it is the most commonly used. Steps: 1. **Keep the existing Namespace Endpoint PHZ override in the original region.** Do not change anything in the original region's private DNS setup yet. 2. **Create a new VPC Endpoint in the new region.** Follow the [AWS PrivateLink](/cloud/connectivity/aws-connectivity) (or [GCP PSC](/cloud/connectivity/gcp-connectivity)) creation steps in the new region's VPC. 3. **In the new region, create a PHZ that overrides the Namespace Endpoint to point at the new VPC Endpoint.** Use the [single-region PHZ pattern](/cloud/connectivity/aws-connectivity#configuring-private-dns-for-aws-privatelink), scoped only to the new region's Worker VPCs. With one PHZ per region, traffic in each region routes through that region's own VPC Endpoint. 4. **Add a replica in the new region.** Follow [Enable High Availability features](/cloud/high-availability/enable). The new region comes up as a passive replica. 5. **Start Workers in the new region.** They begin processing tasks immediately, even though they are connecting to the passive replica, because Temporal Cloud forwards Workflow and Activity tasks across regions transparently. This is what keeps the region change zero-downtime. 6. **Failover to the new region.** Trigger a [forced failover](/cloud/high-availability/failovers) to make the new region active. Workers in the old region keep running and keep using the old VPC Endpoint — they now connect to the passive replica that lives in the old region, and Temporal Cloud forwards their tasks to the new active region. 7. **Drain and remove Workers and the VPC Endpoint in the old region.** Once you are confident the new region is handling all new work and you no longer need old-region Workers, stop them and tear down the old region's VPC Endpoint and PHZ. 8. **Remove the replica in the old region.** See [Migrate between regions](/cloud/migrate/migrate-within-cloud) for the replica-removal step. The Namespace is now single-region in the new location, and the HA pricing surcharge no longer applies. > **📝 Note:** > Per-region PHZ vs. shared regional-record PHZ > > This pattern is **different** from the long-term HA setup described in [Single-cloud HA on AWS PrivateLink](#single-cloud-ha-on-aws-privatelink), which uses one shared PHZ holding regional records (`aws-.region.tmprl.cloud`) and relies on DNS-based failover to switch Workers between regions. The region-change pattern instead uses one PHZ per region, each overriding the Namespace Endpoint itself, and relies on Temporal Cloud's cross-region task forwarding rather than DNS to keep Workers productive across the cutover. > ## Single-cloud HA on AWS PrivateLink ### How Namespace DNS records work with PrivateLink When using PrivateLink, you connect to Temporal Cloud through a VPC Endpoint, which uses addresses local to your network. Temporal treats each `region.tmprl.cloud` zone as a separate zone, so you override resolution per region — this routes traffic to your VPC Endpoint internally for the regions you're using. A Namespace's active region is reflected in the target of the Namespace Endpoint's CNAME record. For example, if the active region of a Namespace is AWS us-east-1, the DNS configuration would look like this: | Record name | Record type | Value | | ----------------------------------- | ----------- | ---------------------------------- | | ha-namespace.account-id.tmprl.cloud | CNAME | aws-us-east-1.region.tmprl.cloud | After a failover, the CNAME record is updated to point to the failover region, for example: | Record name | Record type | Value | | ----------------------------------- | ----------- | ---------------------------------- | | ha-namespace.account-id.tmprl.cloud | CNAME | aws-us-west-2.region.tmprl.cloud | The Temporal domain did not change, but the CNAME updated from us-east-1 to us-west-2. ![Customer side solution example](/img/cloud/high-availability/private-link.png) ### How to set up the DNS override In AWS, use a Route 53 private hosted zone for `region.tmprl.cloud` to override resolution per region: | Record name | Record type | Value (your VPC Endpoint DNS) | | ------------------------------------ | ----------- | ------------------------------------------------------------ | | `aws-us-west-2.region.tmprl.cloud` | CNAME | `vpce-...-us-west-2.vpce.amazonaws.com` | | `aws-us-east-1.region.tmprl.cloud` | CNAME | `vpce-...-us-east-1.vpce.amazonaws.com` | Link the private zone to every VPC where Workers run. When your Workers connect to the Namespace, they first resolve `..tmprl.cloud`, which CNAMEs to `aws-.region.tmprl.cloud`, which then resolves to your local VPC Endpoint. You also need to decide how Workers reach whichever region becomes active. Either: - Run Workers in **both** regions continuously (recommended), or - Establish cross-region connectivity (Transit Gateway, VPC Peering) so Workers in one region can reach the VPC Endpoint in the other. ## Single-cloud HA on GCP Private Service Connect For GCP-only HA, the same model applies, but use a Cloud DNS private zone for `region.tmprl.cloud` and point each `gcp-.region.tmprl.cloud` record at the local PSC endpoint IP address. | Record name | Record type | Value (your PSC endpoint IP) | | ---------------------------------------- | ----------- | ----------------------------------- | | `gcp-us-central1.region.tmprl.cloud` | A | `10.x.x.x` (PSC endpoint IP) | | `gcp-us-east1.region.tmprl.cloud` | A | `10.x.x.x` (PSC endpoint IP) | A Connectivity Rule is required for each PSC connection — see [GCP PSC setup](/cloud/connectivity/gcp-connectivity) and [Connectivity Rules](/cloud/connectivity#connectivity-rules). ## Multi-cloud HA (AWS PrivateLink + GCP Private Service Connect) If your replicas span clouds — for example, AWS `us-east-1` (active) and GCP `us-east4` (passive) — your Workers need a way to reach the active replica regardless of which cloud it's in. The Temporal-managed CNAME rewrites still work the same way; the harder problems are on the client side. Plan for these three things: 1. **DNS overrides for both clouds.** Your private DNS for `region.tmprl.cloud` needs entries for both the AWS region (CNAME → AWS VPCE) and the GCP region (A → PSC IP). This typically means a Route 53 private hosted zone in your AWS Worker VPCs *and* a Cloud DNS private zone in your GCP Worker network — both for the same `region.tmprl.cloud` parent — each with the records relevant to the cloud the Workers run in. 2. **Worker reachability across clouds.** Your AWS-resident Workers must be able to reach the GCP PSC endpoint when GCP is active, and vice versa. Options include: - Run Workers in both clouds (preferred — simplest, lowest latency, matches the failover model). - Establish cross-cloud connectivity (for example, AWS Transit Gateway + GCP Cloud Interconnect, or a third-party transit) so Workers in one cloud can resolve and reach the other cloud's private endpoint. 3. **Connectivity Rules in both regions.** GCP PSC requires a Connectivity Rule. AWS PrivateLink does not, but if you want to enforce private-only access, add one for the AWS side as well so the Namespace is private-only in both regions. > **⚠️ Caution:** > Alpine/musl + GCP PSC: missing AAAA records can break Workers > > GCP Private Service Connect endpoints return only A (IPv4) records — there is no AAAA (IPv6) record. Most Linux distributions handle a missing AAAA gracefully, but **Alpine Linux's musl resolver returns a SERVFAIL** when AAAA is missing, which can cause Temporal SDK clients to fail name resolution after a failover from AWS to GCP. > > If you run Workers on Alpine and use multi-cloud HA, either: > > - Switch the Worker base image to a glibc-based distribution (Debian, Ubuntu, distroless), or > - Configure your application/runtime to disable AAAA lookups (for example, set `GODEBUG=netdns=go+v4` for Go, or prefer IPv4 in the Java/Node/Python runtimes you use). > ### Available regions, PrivateLink endpoints, and DNS record overrides > **⚠️ Caution:** > > The `sa-east-1` region is not yet available for use with Multi-region Namespaces. Currently, it is the only region on the continent. > The following tables list the available Temporal regions and the DNS record overrides used for HA + private connectivity: ### AWS regions and PrivateLink endpoints | Region | PrivateLink Service Name | DNS Record Override | | --- | --- | --- | | ap-northeast-1 | com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a | aws-ap-northeast-1.region.tmprl.cloud | | ap-northeast-2 | com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308 | aws-ap-northeast-2.region.tmprl.cloud | | ap-south-1 | com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662 | aws-ap-south-1.region.tmprl.cloud | | ap-south-2 | com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1 | aws-ap-south-2.region.tmprl.cloud | | ap-southeast-1 | com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd | aws-ap-southeast-1.region.tmprl.cloud | | ap-southeast-2 | com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08 | aws-ap-southeast-2.region.tmprl.cloud | | ca-central-1 | com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d | aws-ca-central-1.region.tmprl.cloud | | eu-central-1 | com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3 | aws-eu-central-1.region.tmprl.cloud | | eu-west-1 | com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739 | aws-eu-west-1.region.tmprl.cloud | | eu-west-2 | com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695 | aws-eu-west-2.region.tmprl.cloud | | sa-east-1 | com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a | aws-sa-east-1.region.tmprl.cloud | | us-east-1 | com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f | aws-us-east-1.region.tmprl.cloud | | us-east-2 | com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4 | aws-us-east-2.region.tmprl.cloud | | us-west-2 | com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94 | aws-us-west-2.region.tmprl.cloud | ### GCP regions and Private Service Connect endpoints | Region | Private Service Connect Service Name | | --- | --- | | asia-south1 | projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw | | europe-west3 | projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh | | us-central1 | projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng | | us-east4 | projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy | | us-west1 | projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x | When using a Namespace with High Availability features, the Namespace's DNS record `..tmprl.cloud` points to a regional DNS record in the format `-.region.tmprl.cloud`, where `-` is the currently active region for your Namespace. During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. Namespace DNS records are configured with a 15-second TTL. Any DNS cache should re-resolve the record within this time. As a rule of thumb, receiving an updated DNS record takes about twice (2x) the TTL — clients should converge to the newly targeted region within, at most, a 30-second delay, assuming their resolver and language runtime honor the TTL. --- # Monitoring High Availability Source: https://docs.temporal.io/cloud/high-availability/monitoring > How to track the health and performance of your High Availability Namespaces. Temporal Cloud offers several ways for you to track the health and performance of your [High Availability](/cloud/high-availability) namespaces. ## Detect a failover or an outage With some [Worker deployment patterns](/cloud/high-availability/architecture-patterns) — most notably [Active/Passive](/cloud/high-availability/architecture-patterns#active-cold) — detecting an outage is your responsibility, and your Workflows make no progress until you detect it and bring up Workers in the new active region. Fast, reliable detection therefore directly determines your recovery time, so it is worth monitoring for both of the following. ### Detect a failover The clearest way to detect that a failover has happened is to watch whether your Namespace's active region changed. When Temporal Cloud promotes the replica in the secondary region to active, the active region reported for the Namespace changes — a reliable, unambiguous signal that a failover occurred. To track failovers as they happen, look for the `FailoverNamespace` operation described in [Failover audit log](#failover-audit-log). ### Detect an outage A failover is not the only signal worth watching. You may want to detect a regional outage directly, before or independently of a Namespace failover, so you can begin your own response. Watch for: - **A spike in replication lag** between the primary and the replica. See [Monitoring replication](#monitoring-replication). - **A drop in Workflow throughput**, such as a sudden decline in the rate of Workflows started, completed, or Tasks processed. - **A spike in errors across your overall stack**, not just Temporal — for example, application errors, failed Activities, or connection failures. - **A drop in throughput across your overall stack**, such as fewer requests reaching your services or fewer Activities executing. - **Errors or failovers in other cloud systems you depend on**, such as databases, queues, or other regional services, which often signal a broader regional outage. ## Replication status You can monitor your replica status with the Temporal Cloud UI. If the replica is unhealthy, Temporal Cloud disables the "Trigger a failover" option to prevent failing over to an unhealthy replica. An unhealthy replica might be due to: - **Data synchronization issues:** The replica fails to remain in sync with the primary due to network or performance problems. - **Replication lag:** The replica falls behind the primary, causing it to be out of sync. - **Network issues:** Loss of communication between the replica and the primary causes problems. - **Failed health checks:** If the replica fails health checks, it's marked as unhealthy. These issues prevent the replica from being used during a failover, ensuring system stability and consistency. ## Monitoring replication Temporal Cloud's High Availability features use asynchronous replication between the primary and the replica. Workflow updates in the primary, along with associated History Events, are transmitted to the replica. Replication lag refers to the transmission delay of Workflow updates and history events from the primary to the replica. > **💡 Tip:** > > Temporal Cloud strives to maintain a P95 replication lag of less than 1 minute. In this context, > P95 means 95% of updates are processed faster than this limit. > A forced failover, when there is significant replication lag, increases the likelihood of rolling back Workflow progress. Always check the replication lag metrics before initiating a failover. Temporal Cloud emits replication lag [metrics](/cloud/metrics/openmetrics/metrics-reference#replication-metrics) as pre-computed percentiles (p50, p95, p99) that are labeled with `temporal_namespace`. When a Namespace is using a replica, you may notice that the Action count in `temporal_cloud_v1_total_action_count` is 2x what it was before adding a replica. This happens because Actions are replicated; they occur on both the primary and the replica. ## Failover audit log When Temporal triggers failovers, the [audit log](/cloud/audit-logs) will update with details. Look for `"operation": "FailoverNamespace"` in the logs. --- # System limits - Temporal Cloud Source: https://docs.temporal.io/cloud/limits > Learn about Temporal Cloud limits, including accounts, namespaces, throughput, retention, task pollers, batch jobs, gRPC, search attributes, and more. Temporal Cloud enforces a variety of limits to keep the service reliable, including rate limits (how often something may occur in a unit of time), resource limits (how many of a given resource may exist at any one time), and configuration limits (minimum or maximum values for a setting) Every limit applies at a specific scope (level of the application): - At the Temporal Cloud [Account level](#account-level) - At the [Namespace level](#namespace-level) - At the [Nexus endpoint level](#nexus-endpoint-level) - Within the [programming model](#programming-model-level) itself ## Account level The following limits apply at the Temporal Cloud Account level (per account). ### Users - Scope: Account - Default limit: 300 Users - How to increase: [Contact support](/cloud/support#support-ticket) ### Projects - Scope: Account - Default limit: 5 Projects - How to increase: [Contact support](/cloud/support#support-ticket) ### Namespaces - Scope: Account - Default limit: 10 Namespaces - How to increase: - Automatically increased as you start creating Namespaces - [Contact support](/cloud/support#support-ticket) for large-scale needs ## Project level > **Pre-release** The following limits apply at the Project level for any new Project created. The Default Project uses the Account-level resource limits rather than the per-Project defaults shown above. For example, the Default Project inherits the Account's Namespace quota and Account-level limits for Service Accounts, Nexus Endpoints, and Connectivity Rules. ### Namespaces and Project-scoped Service accounts - Scope: Project - Default: 10 - How to increase: [Contact support](/cloud/support#support-ticket) ### Nexus Endpoints - Scope: Project - Default: 5 - How to increase: [Contact support](/cloud/support#support-ticket) ### Connectivity rules - Scope: Project. Each non-default Project supports up to one Public Connectivity Rule. - Default: 6 - How to increase: [Contact support](/cloud/support#support-ticket) ## Namespace level The following limits apply at the Namespace level. ### Actions per second - Scope: Namespace - Default limit: 500 actions per second (APS). With On-Demand Capacity, the default limit is the floor your APS limit automatically scales above based on the last 7 days of usage; your limit never falls below the default limit. With Provisioned Capacity, your limit is set by the Temporal Resource Units you provision. - See your current limit: View it in the Temporal Cloud UI under the Namespace overview, retrieve it with the [Temporal CLI](/cli/command-reference/cloud/namespace#capacity) (`temporal cloud namespace capacity get`), or track the `temporal_cloud_v1_action_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits). - How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/cloud/support#support-ticket). - What happens when you exceed the limit: See [Throttling behavior](#throttling-behavior) below. See the [Actions page](/cloud/actions) for the list of actions. ### Requests per second - Scope: Namespace - Limit: Your requests per second (RPS) limit is dynamic and depends on your [capacity mode](/cloud/capacity-modes). With On-Demand Capacity it automatically increases (and decreases) based on the last 7 days of RPS usage and never falls below the On-Demand floor; with Provisioned Capacity it is set by the Temporal Resource Units you provision. - See your current limit: Track the `temporal_cloud_v1_service_request_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits). - How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/cloud/support#support-ticket). See the [glossary](/glossary#requests-per-second-rps) for more about RPS. ### Operations per second - Scope: Namespace - Limit: Your operations per second (OPS) limit is dynamic and depends on your [capacity mode](/cloud/capacity-modes). With On-Demand Capacity it automatically increases (and decreases) based on the last 7 days of OPS usage and never falls below the On-Demand floor; with Provisioned Capacity it is set by the Temporal Resource Units you provision. - See your current limit: Track the `temporal_cloud_v1_operations_limit` metric. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits). - How to increase: See [Capacity Modes](/cloud/capacity-modes) or [contact support](/cloud/support#support-ticket). See the [operations list](/references/operation-list) for the list of operations. ### Throttling behavior When you exceed your APS, RPS, or OPS limits, Temporal Cloud throttles requests. Here's what happens: 1. **Priority-based throttling**: Low-priority operations are throttled first. Higher-priority operations like `StartWorkflowExecution`, `SignalWorkflowExecution`, and `UpdateWorkflowExecution` continue to go through when possible. Temporal Cloud uses similar [throttling priorities as the open source server](https://github.com/temporalio/temporal/blob/main/service/frontend/configs/quotas.go#L66). 2. **Throttling latency**: Rate limiting is not instantaneous, so usage may briefly exceed your limit before throttling takes effect. 3. **ResourceExhausted errors**: When throttled, the server returns a `ResourceExhausted` gRPC error. SDK clients automatically retry these based on the default gRPC retry policy. 4. **Potential failure**: If throttling persists beyond the SDK's retry limit, client calls fail. This means work _can_ be lost if you don't handle these failures. **Best practices for handling throttling:** - Log any failed `StartWorkflowExecution`, `SignalWorkflowExecution`, or `UpdateWorkflowExecution` calls on the client side, including the payload, so you can retry or backfill later. - Set up [Cloud metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) to alert when throttling occurs and when you approach your limits. - Consider [Provisioned Capacity](/cloud/capacity-modes#provisioned-capacity) if you have predictable spikes or need guaranteed throughput. ### Schedules rate limit - Scope: Namespace - Default limit: 10 schedule requests per second (RPS) - How to increase: [Contact support](/cloud/support#support-ticket) To avoid throttling, don't schedule all your Workflow Executions to start at the same time (such as daily, weekly, or monthly). Every Temporal SDK supports jittering, which adds small random delays to Schedule specifications, helping to reduce load at any specific moment. Set the `jitter` value to the largest delay you will permit before your Workflow Execution must begin. This approach uniformly distributes the scheduled Workflow Execution launches through that period and reduces your Schedule Workflow Execution RPS load. ### Per-primitive Id reuse rate limits - Scope: Namespace, primitive type, and Id - Primitives covered: Workflows, Standalone Activities, Schedules, and Nexus Operations - Default limit: 1 new Execution per second with a burst allowance - Not configurable When executing [Workflows](/workflows), [Standalone Activities](/standalone-activity), and [Nexus Operations](/standalone-nexus-operation), and when creating [Schedules](/schedule), the system enforces a per-primitive limit on successive Executions with the same Id. Every operation that creates an Execution counts toward the limit: - `StartWorkflowExecution` - `SignalWithStartWorkflowExecution` - `ExecuteMultiOperation` (Update) - `ResetWorkflowExecution` - [Continue-As-New](/workflow-execution/continue-as-new), whether requested by the Workflow or triggered by a [Retry Policy](/encyclopedia/retry-policies) or [Cron Job](/cron-job) - `CreateSchedule` - `StartActivityExecution` - `StartNexusOperationExecution` This limit applies only to requests that will create new Executions, and so this limit applies after the deduplication check. A duplicate start that returns `WorkflowExecutionAlreadyStarted` has no effect on the limit. Exceeding this limit returns a `ResourceExhausted` error. ### Visibility API Rate Limit - Scope: Namespace - Default limit: 30 Visibility API calls per second - Not configurable The Visibility API rate limit applies to every read API that lists, searches, or counts across executions. These are: - **Workflow search and count:** `ListWorkflowExecutions`, `ListOpenWorkflowExecutions`, `ListClosedWorkflowExecutions`, `ScanWorkflowExecutions`, `CountWorkflowExecutions` - **Schedules:** `ListSchedules`, `CountSchedules` - **Batch operations:** `ListBatchOperations` - **Task Queue and worker reachability:** `GetWorkerTaskReachability`, and `DescribeTaskQueue` (subject to this limit since Server v1.24) - **Worker deployments:** `ListDeployments`, `GetDeploymentReachability`, `ListWorkerDeployments` Any API that returns a set of executions or a count falls under this limit. Single-entity lookups by ID, such as `DescribeWorkflowExecution` and `DescribeSchedule`, do not. Those read the primary store and count against the general namespace rate limit. ### Nexus Rate Limit Nexus requests (such as starting a Nexus Operation or sending a Nexus completion callback) are counted as part of the overall Namespace RPS limit. If too many Nexus requests are sent at once, they may be throttled, along with other requests to the Namespace. Throttling limits the rate at which Nexus requests are processed, ensuring the RPS limit isn't exceeded. You can request this limit be manually raised by [opening a support ticket](/cloud/support#support-ticket). > **📝 Note:** > > For the target Namespace of a Nexus Endpoint, even though there are no Action results for handling a Nexus Operation itself, the Nexus requests on a target Namespace do count towards the overall RPS limit for the Namespace as a whole. > ### Certificates Temporal Cloud limits each Namespace to a total of 32 KB or 16 certificates, whichever is reached first. ### Concurrent Task pollers Temporal Cloud limits each Namespace to 20,000 Activity pollers and 20,000 Workflow Task pollers concurrently. Each SDK offers a way to configure Workers for per-Worker maximum Activity and Workflow Task pollers. Those values do not affect the global Namespace limit. ### Default Retention Period The [Retention Period](/temporal-service/temporal-server#retention-period) is set per Namespace. Temporal Cloud sets the default Retention Period to 30 days. This is configurable in the Temporal Web UI. [Navigate to your list of Namespaces](https://cloud.temporal.io/namespaces), choose the Namespace you want to update, and select edit: ![Choose your Namespace and select Edit](/img/cloud/cloud-guide/edit-namespace-option.png) ![Update the Retention Period](/img/cloud/cloud-guide/edit-retention-period.png) You can set the Retention Period between 1 and 90 days. ### Batch jobs A Namespace can have just one [Batch job](/cli/command-reference/batch) running at a time. Each batch job operates on a maximum of 50 Workflow Executions per second. ### Number of Custom Search Attributes There is a limit to the number of custom Search Attributes per attribute type per Namespace: | Search Attribute type | Limit | | --------------------- | ----- | | Bool | 20 | | Datetime | 20 | | Double | 20 | | Int | 20 | | Keyword | 40 | | KeywordList | 5 | | Text | 5 | ### Custom Search Attribute names When creating custom Search Attributes in Temporal Cloud, the attribute names must adhere to the following constraints: - Maximum characters: 64 - Allowed characters: `[a-zA-Z0-9.,:-_\/@ ]`. For more information on custom Search Attributes see [Custom Search Attributes limits](/search-attribute#custom-search-attribute). ### Custom Roles limits Each account can create up to 25 [Custom Roles](/cloud/manage-access/custom-roles). The UI can display up to 100 Custom Roles. A single principal (such as a user) can be assigned up to 10 Custom Roles. A Custom Role can contain up 20 permissions. Each permission is a pair of - A list of resource IDs of a single type (for example, specific Namespace IDs) - A list of actions that apply to those resources (for example, cloud.namespace.get, cloud.namespace.update) For example, - 1 permission: Namespaces [ns-prod, ns-staging] with actions [cloud.namespace.get, cloud.namespace.update] - 2 permissions: Namespace [ns-prod] with action [cloud.namespace.get]; Namespace [ns-staging] with action [cloud.namespace.update] defined separately. There is no limit on the number of resource IDs or actions within a single permission pair. Applying a permission to all resources of a type counts as one permission regardless of how many resources exist in your account. ## Nexus Endpoint level ### Nexus Endpoints limits By default, each account is provisioned with 100 Nexus Endpoints. You can request further increases beyond the initial 100 Endpoint limit by [opening a support ticket](/cloud/support#support-ticket). ### Nexus caller Namespace limits By default, a single Nexus Endpoint can have a maximum of 1,000 caller Namespaces in its [Access Policy](/nexus/security#runtime-access-controls), the allowlist of Namespaces permitted to use the Endpoint. You can request further increases beyond the initial 1,000 caller Namespace limit by [opening a support ticket](/cloud/support#support-ticket). ## Programming model level The following limits apply at the programming model level. See also: [Self-hosted Temporal Service defaults](/self-hosted-guide/defaults). ### Identifier length limit Identifiers, such as Workflow Id, Workflow Type, and Task Queue names, are limited to a maximum length of 1,000 bytes. Note that Unicode characters may use multiple bytes. ### Per message gRPC limit Each gRPC message received has a limit of 4 MB. This limit applies to all gRPC endpoints across the Temporal Platform. ### Event History transaction size limit An Event History transaction encompasses a set of operations such as initiating a new Workflow, scheduling an Activity, processing a Signal, or starting a Child Workflow. These operations create Events that are then logged in the Event History. The transaction size limit restricts the total size of Events that can be accommodated within a single transaction. The size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB. This limit is non-configurable for Temporal Cloud. ### Transaction Payload size limit Blob size limit for Payloads, including Workflow context and each Workflow and Activity argument and return value: - The max payload for a single request is 2 MB. - The max size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB. This limit is non-configurable for Temporal Cloud. The [BlobSizeLimitError guide](/troubleshooting/blob-size-limit-error) provides solutions for handling large payloads. > **📝 Note:** > > The 2 MB limit is the maximum size for a single request, not a sustained-throughput target. > Sustained, high-volume use of large payloads can degrade Namespace performance, and Temporal reserves the right to rate limit such traffic to protect the service. > For workloads that regularly produce large payloads, offload them to [External Storage](/external-storage) and keep inline payloads small. > ### Per Workflow Execution concurrency limits If a Workflow Execution has 2,000 incomplete Activities, Signals, Child Workflows, or external Workflow Cancellation requests, additional [Commands](/workflow-execution#command) of that type will fail to be applied to that Workflow Execution: - `ScheduleActivityTask` - `SignalExternalWorkflowExecution` - `StartChildWorkflowExecution` - `RequestCancelExternalWorkflowExecution` For optimal performance, limit concurrent operations to 500 or fewer. This reduces Workflow's Event History size and decreases the loading time in the Web UI. ### Per Workflow Execution Signal limit A single Workflow Execution may receive up to 10,000 Signals. After that limit is reached, no more Signals will be processed for that Workflow Execution. ### Per Workflow Execution Update limits A single Workflow Execution can have a maximum of 10 in-flight Updates and 2000 total Updates in History. ### Workflow Execution Event History limits As a precautionary measure, a Workflow Execution's Event History is limited to 51,200 Events or 50 MB. It warns you after 10,240 Events or 10 MB. This limit applies to all Temporal Workflow Executions, whether on Temporal Cloud or other deployments. This limit is non-configurable for Temporal Cloud. Read more about [Temporal Workflow Execution limits](/workflow-execution/limits) on the [Temporal Workflow](/workflows) documentation page. ### Per Workflow Callback limits A single Workflow Execution can have a maximum of 2000 total Callbacks. These limits may be exceeded when [multiple Nexus callers attach to the same handler Workflow](/nexus/operations#attaching-multiple-nexus-callers). See the Nexus Encyclopedia entry for [additional details](/workflow-execution/limits#workflow-execution-callback-limits). ### Per Workflow Nexus Operation limits A single Workflow Execution can have a maximum of 30 in-flight Nexus Operations. See the Nexus Encyclopedia entry for [additional details](/workflow-execution/limits#workflow-execution-nexus-operation-limits). ### Nexus Operation request timeout Less than 10 seconds is the maximum duration for a Nexus handler to process a single Nexus start or cancel request. The timeout is measured from the calling History Service and the request must go through matching, so the available time for a handler to respond is often much less than 10 seconds. Handlers should observe the context deadline and ensure they don't exceed it. This includes fully processing a synchronous Nexus operation and starting an asynchronous Nexus operation, for example one that starts a Workflow. If a Nexus handler doesn’t process a start or cancel request within 10 seconds, it will receive a context deadline exceeded error, and the caller will retry, with an exponential backoff, for the ScheduleToClose duration for the overall Nexus Operation. This has a default and maximum as defined below in [Nexus Operation duration limits](/cloud/limits#nexus-operation-duration-limits). ### Nexus Operation duration limits Each Nexus Operation has a maximum ScheduleToClose duration of 60 days. This is most applicable to asynchronous Nexus Operations completed with an asynchronous callback using a separate Nexus request from the handler back to the caller Namespace. For enhanced security, you may sign completion callbacks with a single-use token in the future, and the 60 day maximum allows you to rotate the asymmetric encryption keys used for completion callback request signing. While the caller of a Nexus Operation can configure the ScheduleToClose duration to be shorter than 60 days, the maximum duration can not extend beyond 60 days and capped by the server to 60 days. ### Timer duration limit Timers have a maximum duration of 100 years in Temporal Cloud. ## Worker Versioning level ### Max Worker deployments limits The maximum number of Worker deployments that the server allows to be registered in a single Namespace. Defaults to 100. ### Max versions in deployment limits The maximum number of versions that the server allows to be registered in a single Worker deployments at a given time. Note that unused versions will be deleted by the system automatically when this limit is reached. Defaults to 100. ### Max Task Queues In Deployment Version limits The maximum number of Task Queues that the server allows to be registered in a single Worker Deployment Version. Defaults to 100. --- # Account access Source: https://docs.temporal.io/cloud/manage-access > Manage access to your Temporal Cloud account Access to Temporal Cloud is governed by role-based access control (RBAC). Within an account, each access principal, such as user, user group or service account, has one account-level role and optionally, one or more Namespace-level permissions. Each principal can only perform actions that are allowed by their assigned roles and permissions. Temporal Cloud supports Security Assertion Markup Language (SAML) and System for Cross-domain Identity Management (SCIM) for integration with your organization's identity provider (IdP). SAML enables single sign-on (SSO) by allowing your identity provider to authenticate users into Temporal Cloud. SCIM automatically creates, updates, and removes users and groups in Temporal Cloud based on changes in your identity provider. ## Temporal Cloud accounts Accounts are the top-level container for access control. Each account has at least one user assigned the Account Owner role, which has full administrative permissions across the account, including users, billing and usage. An account is **not** an access principal itself. When you sign up for Temporal Cloud without joining an existing account, you are automatically assigned the Account Owner role for a new account. You can then invite other users to join the account and assign them roles. If your organization has an IdP, we recommend using [a SAML integration](/cloud/manage-access/saml) for enterprise identity management. > **ℹ️ Info:** > > Multiple accounts can coexist on the same email domain. Each account can have its own independent SAML configuration, > tied to its unique Account ID. > > However, each email address can only be associated with a single Temporal Cloud account. If you need access to multiple > accounts, you’ll need a separate invite for each one using a different email address. > ## Access principals Temporal Cloud offers the following principals for access control: - [**Users**](/cloud/manage-access/users) - Manage individual user accounts and permissions - [**User Groups**](/cloud/manage-access/user-groups) - Organize users into groups for simplified access management - [**Service Accounts**](/cloud/manage-access/service-accounts) - Configure service accounts for automated access - [**Custom Roles**](/cloud/manage-access/custom-roles) - Define custom permissions for specific use cases - [**SAML**](/cloud/manage-access/saml) - Configure SAML-based single sign-on integration - [**SCIM**](/cloud/manage-access/scim) - Use your IDP to manage Temporal Cloud users and access via SCIM integration ## Troubleshoot account access issues ### Recover your account after losing access to your authenticator app Accounts registered with email and password require multi-factor authentication (MFA) with an authenticator app. If you lose access to your authenticator app, you can still log in by clicking **Try another method** on the MFA screen. From there, you can either: - Enter your recovery code (provided when you first set up MFA) - Receive a verification code through email Once you're logged in, you can reset your authenticator app by navigating to **My Profile** > **Password and Authentication** and then clicking **Authenticator App** > **Remove method**. ### Reset your password If you're currently logged in and would like to change your password, click your profile icon at the top right of the Temporal Cloud UI, navigate to **My Profile** > **Password and Authentication**, and then click **Reset Password**. If you're not currently logged in, navigate to the login page of the Temporal Cloud UI, enter your email address, click **Continue**, and then select **Forgot password**. In both cases, you will receive an email with instructions on how to reset your password. ### Sign in after email domain changes If your organization changed its email domain (for example, from `@oldcompany.com` to `@newcompany.com`), you may be unable to sign in to Temporal Cloud with your existing account. **Why this happens:** When you sign in using "Continue with Google" or "Continue with Microsoft", Temporal Cloud identifies your account by your email address. If your email address changes, Temporal Cloud sees this as a different identity and cannot match it to your existing account. **How to resolve this:** [Create a support ticket](/cloud/support#support-ticket) with the following information: - Your previous email address (the one originally used to access Temporal Cloud) - Your new email address - Your Temporal Cloud Account Id (if known) Temporal Support can update your account to use your new email address. > **💡 Tip:** > Use SAML for enterprise identity management > > If your organization frequently changes email domains or wants centralized control over user authentication, consider > using [SAML authentication](/cloud/manage-access/saml). With SAML, your identity provider (IdP) manages user identities, and email > domain changes can be handled within your IdP without affecting Temporal Cloud access. > --- # Manage custom roles Source: https://docs.temporal.io/cloud/manage-access/custom-roles > **Pre-release** With Custom Roles, you can define granular permissions in Temporal Cloud, giving your team precise control over who can perform specific actions within your account. ## What are Custom Roles? Custom Roles are user-defined collections of permissions that grant access to specific Temporal Cloud resources ([Account](/cloud/manage-access#temporal-cloud-accounts), [Namespace](/namespaces), [Nexus Endpoint](/nexus/endpoints), or [Connectivity Rule](/cloud/connectivity#connectivity-rules)). They allow you to define custom permission sets that are more granular than the predefined roles, and assign them to any principal (user, group, service account). ## Why use Custom Roles? Use Custom Roles when you need more granular access control than the [predefined roles](/cloud/manage-access/users#account-level-roles) provide. Common use cases include: - **Least-privilege access**: Grant users only the permissions they need to perform their job functions. - **Delegated administration**: Allow teams to manage specific Temporal Cloud resources without granting full account administration privileges. - **Service account security**: Create narrowly scoped permissions for automation and integrations, reducing risk if credentials are compromised. ## Defining Custom Roles When defining a Custom Role, you select the permission actions to include and the resources each action applies to. Every principal must still have a predefined account role (such as Developer or Read-Only). The Custom Role is applied alongside the predefined role, and the effective access is the union of both. Custom Roles cannot narrow or remove permissions granted by the predefined role. For example, you might want users with the Account Developer role to view all Namespaces for troubleshooting, without allowing them to modify Namespace configuration. To do this, create a Custom Role named `NamespaceGlobalReadOnly` that grants: - `cloud.namespace.list`, scoped to the Cloud Account - `cloud.namespace.get`, scoped to all Namespace resources ## Delegating Custom Roles Custom role administration defaults to the Account Owner, but can be delegated. Account Owners are the primary principals that can create, list, update, delete, or assign a Custom Role. Account Owners can delegate Custom Role administration to other roles such as Global Admin, but this comes with risk. If the Account Owner creates a Custom Role that includes the ability to create, update, or assign Custom Roles, and assigns that role to any other principal, such as Global Admin or Developer, users with that new role can create or modify roles with other operations and assign them to themselves or others. > **⚠️ Warning:** > Use caution when delegating Custom Role operations > > Delegating Custom Role administration to another user will give that user the ability to create or modify roles. This > leads to privilege escalation risk where a user would have the ability to assign themselves or others to a role with any > operations, even ones they're not approved for. > The following operations present privilege escalation risk when delegated: - `cloud.customrole.create` - `cloud.customrole.update` - `cloud.customrole.assign` Receiving `cloud.customrole.assign` alone does not grant the ability to update a principal's access on its own. Assigning a Custom Role to a user, group, or service account also requires the relevant principal-update permission such as [update a user's access](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users/{userId}). Developer and Read-only predefined roles do not include those permissions. Global Admin can perform full Custom Role delegation after receiving `cloud.customrole.assign` because it has the required principal-update permissions by default. ## Available permissions Most of the operations listed in the Cloud Ops API references ([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction), [GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/)) can be assigned to a Custom Role. For the operations that are supported by Custom Roles, see [the Custom Role Permissions table](/cloud/manage-access/permissions-reference#custom-role-permissions-reference). ## Create Custom Roles **Cloud UI** To create a Custom Role from the Web UI, select Settings in the left sidebar, and then click the Custom Roles tab on the Settings page. On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page. Click the three dots menu to view details about an existing Custom Role, or to edit or delete that role. Click the Create Custom Role button to create a new role. On the Create Custom Role page, give the Custom Role a name, and optionally a description. ![Creating a new custom role](/img/cloud/custom-roles/custom-roles-cloud-ui.png) In the Permissions section, you'll assign the appropriate resources and its permissions. The following resources are available for Custom Roles: - Account: Permissions scoped to the current account, listed by type. - Namespace: Permissions scoped to the selected Namespace, listed by type. You can only assign permissions to Namespaces to which you have access. - Nexus Endpoint: Permissions scoped to the selected Nexus Endpoint, listed by type. You must have at least one Nexus Endpoint enabled before you can assign Nexus Endpoint permissions to a Custom Role. - Connectivity Rules: Permissions scoped to the selected Connectivity Rules, listed by type. Account-level permissions apply to the current account. For each of the other resources, you'll need to first select the resource that you want to grant permissions for. Once you select a resource, the list of available permissions will appear, and you can turn them on or off as desired. At the bottom of the Account tab are permissions that relate to Custom Roles. Here, you can define whether members of the Custom Role you're creating will be able to create, update, or delete Custom Roles. When you're done assigning permissions to resources, click Create Custom Role at the bottom of the page. **Cloud Operations API** Use the [`CreateCustomRoleRequest` API call](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CreateCustomRoleRequest) to create a Custom Role. The structure of the `CreateCustomRoleRequest` is defined by a JSON spec, as follows: ``` { "name": "my-role-name" "description": "Optional description", "permissions": [ { "resources": { "resource_type": "namespace", "allow_all": true }, "actions": ["cloud.namespace.get"] } ] } ``` This example creates a Custom Role with the defined `spec`. The example shown here creates a role named `my-role-name` and grants the ability to get all Namespaces. The request returns a `RoleID`. The following constraints apply to the request: - **Name constraints:** - A name is required, with a maximum of 64 characters - The following characters are allowed: letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (\_) - Names are not unique per account, meaning two different roles can share the same name - **Description constraints:** - A description is optional, with a maximum of 256 characters - **Permissions constraints:** - At least one permission is required - A Custom Role can contain up to 20 permissions. See [Custom Roles limits](/cloud/limits#custom-role-limits). **Permission Structure** Each permission has two parts: actions and resources. - **Resource Types & Scoping Behavior** | **Resource Type** | **Scoping Behavior** | | ------------------ | ----------------------------------------------------------------------------------------- | | account | Account-level; no specific resource ID needed | | Namespaces | Scoped to specific Namespace IDs, or `allow_all: true` for all Namespaces | | Nexus endpoints | Scoped to specific Nexus endpoint IDs, or `allow_all: true` for all Nexus endpoints | | connectivity rules | Scoped to specific connectivity rule IDs, or `allow_all: true` for all connectivity rules | Resource IDs must exist in your account. Specifying an unknown resource ID will return an error. Either `allow_all: true` or `resource_ids` must be set, not both. For a list of available actions, see [the list of Custom Role permissions](/cloud/manage-access/permissions-reference#custom-role-permissions-reference). - Account-scoped actions (resource_type: "accounts") - Namespace-scoped actions (resource_type: "namespaces") - Nexus endpoint-scoped actions (resource_type: "nexus_endpoints") - Connectivity rule-scoped actions (resource_type: "connectivity_rules") The following example allows read access to specific Namespaces. The action is `cloud.namespace.get`, the resource type is `namespaces`, and the list of resource IDs is specified. ```jsx { "actions": ["cloud.namespace.get"], "resources": { "resource_type": "namespaces", "resource_ids": ["my-namespace.account-id"] } } ``` The following example assigns multiple permissions to one Role. The permissions are assigned in an array, with the action `cloud.namespace.get` granted for all Namespaces, and `cloud.user.list` granted for all accounts. ```jsx { "spec": { "name": "ns-reader-user-lister", "description": "Can read namespaces and list users", "permissions": [ { "actions": ["cloud.namespace.get"], "resources": { "resource_type": "namespaces", "allow_all": true } }, { "actions": ["cloud.user.list"], "resources": { "resource_type": "accounts", "allow_all": true } } ] } } ``` For more information about the `Custom Role` and `CustomRoleSpec` definitions, see the [HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles) and/or [GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/) API reference doc. **Terraform** For more detailed examples on how to manage Custom Roles via Terraform, see the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role) for usage guidance. **Cloud CLI** The a Custom Role is defined by a JSON spec, as follows: ``` { "name": "my-role-name" "description": "Optional description", "permissions": [ { "resources": { "resource_type": "namespace", "allow_all": true }, "actions": ["cloud.namespace.get"] } ] } ``` This example creates a Custom Role with the defined `spec`. The example shown here creates a role named `my-role-name` and grants the ability to get all Namespaces. Save the spec to a file (such as `role.json`) so it can be passed via `@file`. The following constraints apply to the request: - **Name constraints:** - A name is required, with a maximum of 64 characters - The following characters are allowed: letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (\_) - Names are not unique per account, meaning two different roles can share the same name - **Description constraints:** - A description is optional, with a maximum of 256 characters - **Permissions constraints:** - At least one permission is required - You can assign up to 25 permissions per role (account default, may vary) **Permission Structure** Each permission has two parts: actions and resources. - **Resource Types & Scoping Behavior** | **Resource Type** | **Scoping Behavior** | | ------------------ | ----------------------------------------------------------------------------------------- | | account | Account-level; no specific resource ID needed | | Namespaces | Scoped to specific Namespace IDs, or `allow_all: true` for all Namespaces | | Nexus endpoints | Scoped to specific Nexus endpoint IDs, or `allow_all: true` for all Nexus endpoints | | connectivity rules | Scoped to specific connectivity rule IDs, or `allow_all: true` for all connectivity rules | Resource IDs must exist in your account. Specifying an unknown resource ID will return an error. Either `allow_all: true` or `resource_ids` must be set, not both. For a list of available actions, see [the list of Custom Role permissions](/cloud/manage-access/permissions-reference#custom-role-permissions-reference). - Account-scoped actions (resource_type: "accounts") - Namespace-scoped actions (resource_type: "namespaces") - Nexus endpoint-scoped actions (resource_type: "nexus_endpoints") - Connectivity rule-scoped actions (resource_type: "connectivity_rules") To create a custom role: ``` temporal cloud custom-role create --spec @role.json ``` ## Assigning Custom Roles to users Once you have created a Custom Role, it is available on the Identities page to assign to a user or group, the same as the pre-defined Temporal permissions. See [How to update an account-level role in Temporal Cloud](/cloud/manage-access/users#update-roles) for more information. ## Modifying a Custom Role **Cloud UI** To modify a Custom Role from the Web UI, select Settings in the left sidebar, and then click the Custom Roles tab on the Settings page. On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page. Click the three dots menu of the Custom Role you want to modify and select Edit. The Edit Custom Role page has the same options as the Create Custom Role page. You can change the Custom Role's name or description, or you can modify any of the current permissions assigned to that Role. When finished, click Update Custom Role. **Cloud Operations API** Use the UpdateCustomRole API ([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/POST/cloud/custom-roles/{roleId}), [GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/#updatecustomrole))to modify the Custom Role. **Terraform** To update a Custom Role, modify the `temporalcloud_custom_role` resource in your Terraform configuration and run `terraform apply`. Updating a Custom Role replaces the full permissions list. Include all permissions you want to retain. Any permission omitted from the update will be removed. For more details see the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role). **Cloud CLI** To update a role spec by ID, use `update`: ``` temporal cloud custom-role update --role-id my-role-id --spec @role.json ``` You can override the resource version explicitly with `--resource-version`; otherwise the latest version is fetched and used. ## Delete a Custom Role **Cloud UI** To delete a Custom Role from the Web UI, select Settings in the left sidebar, and then click the Custom Roles tab on the Settings page. On the Custom Roles tab, you'll see a list of the roles that have already been defined for your account, 50 to a page. Click the three dots menu of the Custom Role you want to delete and select Delete. A pop-up notification will let you know that the Custom Role has been deleted. **Cloud Operations API** Use the DeleteCustomRole API ([HTTP](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/custom-roles/DELETE/cloud/custom-roles/{roleId}), [GRPC](https://saas-api.tmprl.cloud/docs/grpcapi/#deletecustomrole))to delete the Custom Role. **Terraform** To delete a Custom Role, remove the `temporalcloud_custom_role` resource block from your Terraform configuration and run `terraform apply`. You can also target a specific resource: Deleting a Custom Role permanently removes it and automatically revokes it from all principals it was assigned to. This action cannot be undone. For more details see the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/custom_role). **Cloud CLI** To delete a Custom Role, use `delete`: ``` temporal cloud custom-role delete --role-id my-role-id ``` ## Custom Roles limits For more information about the limits of Custom Roles, such as the maximum number of custom roles per Account, see [Custom Roles limits](/cloud/limits#custom-role-limits). --- # Permissions reference Source: https://docs.temporal.io/cloud/manage-access/permissions-reference > Reference for permissions in Temporal Cloud Temporal Cloud access controls are organized across three scopes: - Account-level role permissions - Project-level permissions - Namespace-level permissions Within each scope, permissions apply to publicly documented [Temporal Cloud Ops API](/ops) endpoints and to additional non-Cloud Ops capabilities, such as Temporal Cloud UI and internal automation behaviors. ## Account-level access Account-level access is granted to users and service accounts by assigning them an account-level role. Temporal Cloud supports the following account-level roles: - Account Owner - Global Admin - Developer - Finance Admin - Read-Only ### Cloud Ops API permissions This table provides API-level details for permissions granted through account-level roles. These permissions are configured per user. | Permission | Read-only | Developer | Finance Admin | Global Admin | Account Owner | | --------------------------- | :-------: | :-------: | :-----------: | :----------: | :-----------: | | [AddUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/members) | | | | ✅ | ✅ | | [CreateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks) | | | | ✅ | ✅ | | [CreateApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/POST/cloud/api-keys) | ✅\* | ✅\* | ✅\* | ✅\* | ✅\* | | [CreateBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/POST/cloud/billing-reports) | | | ✅ | | ✅ | | [CreateConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules) | | | | ✅ | ✅ | | [CreateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces) | | ✅ | | ✅ | ✅ | | [CreateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints) | | ✅ | | ✅ | ✅ | | [CreateProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects) | | ✅ | | ✅ | ✅ | | [CreateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/manage-access/service-accounts) | ✅† | ✅† | ✅† | ✅† | ✅† | | [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users) | | | | ✅ | ✅ | | [CreateUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups) | | | | ✅ | ✅ | | [DeleteAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/DELETE/cloud/audit-log-sinks/%7Bname%7D) | | | | ✅ | ✅ | | [DeleteApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/DELETE/cloud/api-keys/%7BkeyId%7D) | ✅\* | ✅\* | ✅\* | ✅\* | ✅\* | | [DeleteConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/DELETE/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | | | | ✅ | ✅ | | [DeleteNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/DELETE/cloud/nexus/endpoints/%7BendpointId%7D) | | ✅ | | ✅ | ✅ | | [DeleteServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/DELETE/cloud/manage-access/service-accounts/%7BserviceAccountId%7D) | ✅† | ✅† | ✅† | ✅† | ✅† | | [DeleteUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/DELETE/cloud/users/%7BuserId%7D) | | | | ✅ | ✅ | | [DeleteUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/DELETE/cloud/user-groups/%7BgroupId%7D) | | | | ✅ | ✅ | | [GetAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/account) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks/%7Bname%7D) | | | | ✅ | ✅ | | [GetAccountAuditLogSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-log-sinks) | | | | ✅ | ✅ | | [GetApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/GET/cloud/api-keys/%7BkeyId%7D) | ✅\* | ✅\* | ✅\* | ✅\* | ✅\* | | [GetApiKeys](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/GET/cloud/api-keys) | ✅\* | ✅\* | ✅\* | ✅\* | ✅\* | | [GetAsyncOperation](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/operations/GET/cloud/operations/%7BasyncOperationId%7D) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetAuditLogs](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/audit-logs) | | | | ✅ | ✅ | | [GetBillingReport](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/billing/GET/cloud/billing-reports/%7BbillingReportId%7D) | | | ✅ | | ✅ | | [GetConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | | ✅ | | ✅ | ✅ | | [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules) | | ✅ | | ✅ | ✅ | | [GetCurrentIdentity](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/current-identity) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints/%7BendpointId%7D) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetProjects](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions/%7Bregion%7D) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetRegions](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/regions/GET/cloud/regions) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/manage-access/service-accounts/%7BserviceAccountId%7D) | ✅† | ✅† | ✅† | ✅† | ✅† | | [GetServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/manage-access/service-accounts) | ✅† | ✅† | ✅† | ✅† | ✅† | | [GetUsage](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/GET/cloud/usage) | | | ✅ | ✅ | ✅ | | [GetUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users/%7BuserId%7D) | ✅‡ | ✅‡ | ✅‡ | ✅‡ | ✅‡ | | [GetUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetUserGroupMembers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups/%7BgroupId%7D/members) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetUserGroups](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/GET/cloud/user-groups) | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetUsers](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/users) | ✅‡ | ✅‡ | ✅‡ | ✅‡ | ✅‡ | | [RemoveUserGroupMember](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D/remove-member) | | | | ✅ | ✅ | | [UpdateAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/account) | | | | ✅ | ✅ | | [UpdateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sinks/%7Bspec.name%7D) | | | | ✅ | ✅ | | [UpdateApiKey](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/api-keys/POST/cloud/api-keys/%7BkeyId%7D) | ✅\* | ✅\* | ✅\* | ✅\* | ✅\* | | [UpdateNamespaceTags](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/update-tags) | | | | ✅ | ✅ | | [UpdateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints/%7BendpointId%7D) | | ✅ | | ✅ | ✅ | | [UpdateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/manage-access/service-accounts/%7BserviceAccountId%7D) | ✅† | ✅† | ✅† | ✅† | ✅† | | [UpdateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/users/%7BuserId%7D) | | | | ✅ | ✅ | | [UpdateUserGroup](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/user-groups/%7BgroupId%7D) | | | | ✅ | ✅ | | [ValidateAccountAuditLogSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/account/POST/cloud/audit-log-sink-validate) | | | | ✅ | ✅ | - \* See [API Key Authorization Behavior](#api-key-authorization-behavior) - † See [Service Account Authorization Behavior](#service-account-authorization-behavior) - ‡ See [User authorization behavior](#user-authorization-behavior) ## Namespace-level permissions Namespace-level permissions are granted to users and service accounts by assigning them a Namespace-level permission. Temporal Cloud supports the following Namespace-level permissions: - Namespace Admin - Write - Read Users with the Global Admin and Account Owner roles automatically have Namespace Admin permissions on all Namespaces in the account. ### Cloud Ops API permissions This table provides API-level details for permissions granted through Namespace-level permissions. These permissions are configured per Namespace per user. | Permission | Read | Write | Namespace Admin | | -------------------------------- | :--: | :---: | :-------------: | | [AddNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/add-region) | | | ✅ | | [CreateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks) | | | ✅ | | [DeleteNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/DELETE/cloud/namespaces/%7Bnamespace%7D) | | | ✅ | | [DeleteNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/DELETE/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D) | | | ✅ | | [DeleteNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/DELETE/cloud/namespaces/%7Bnamespace%7D/regions/%7Bregion%7D) | | | ✅ | | [FailoverNamespaceRegion](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/high-availability/POST/cloud/namespaces/%7Bnamespace%7D/failover-region) | | | ✅ | | [GetNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D) | ✅ | ✅ | ✅ | | [GetNamespaceCapacityInfo](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces/%7Bnamespace%7D/capacity-info) | ✅ | ✅ | ✅ | | [GetNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bname%7D) | ✅ | ✅ | ✅ | | [GetNamespaceExportSinks](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/GET/cloud/namespaces/%7Bnamespace%7D/export-sinks) | ✅ | ✅ | ✅ | | [GetUserNamespaceAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/GET/cloud/namespaces/%7Bnamespace%7D/user-assignments) | | | ✅ | | [RenameCustomSearchAttribute](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D/rename-custom-search-attribute) | | | ✅ | | [SetServiceAccountNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/namespaces/%7Bnamespace%7D/service-accounts/%7BserviceAccountId%7D/access) | | | ✅ | | [SetUserGroupNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/groups/POST/cloud/namespaces/%7Bnamespace%7D/user-groups/%7BgroupId%7D/access) | | | ✅ | | [SetUserNamespaceAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users/POST/cloud/namespaces/%7Bnamespace%7D/users/%7BuserId%7D/access) | | | ✅ | | [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/%7Bnamespace%7D) | | | ✅ | | [UpdateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sinks/%7Bspec.name%7D) | | | ✅ | | [ValidateNamespaceExportSink](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/export/POST/cloud/namespaces/%7Bnamespace%7D/export-sink-validate) | | | ✅ | ### Workflow-level permissions This table provides API-level details for Workflow-level Data Plane permissions granted through Namespace-level permissions. These permissions are configured per Namespace per user. | Permission | Read | Write | Namespace Admin | | ------------------------------------- | :--: | :---: | :-------------: | | CountActivityExecutions | ✅ | ✅ | ✅ | | CountSchedules | ✅ | ✅ | ✅ | | CountWorkflowExecutions | ✅ | ✅ | ✅ | | CreateSchedule | | ✅ | ✅ | | CreateWorkflowRule | | ✅ | ✅ | | DeleteActivityExecution | | ✅ | ✅ | | DeleteSchedule | | ✅ | ✅ | | DeleteWorkerDeployment | | ✅ | ✅ | | DeleteWorkerDeploymentVersion | | ✅ | ✅ | | DeleteWorkflowExecution | | ✅ | ✅ | | DeleteWorkflowRule | | ✅ | ✅ | | DescribeActivityExecution | ✅ | ✅ | ✅ | | DescribeBatchOperation | ✅ | ✅ | ✅ | | DescribeNamespace | ✅ | ✅ | ✅ | | DescribeSchedule | ✅ | ✅ | ✅ | | DescribeTaskQueue | ✅ | ✅ | ✅ | | DescribeWorker | ✅ | ✅ | ✅ | | DescribeWorkerDeployment | ✅ | ✅ | ✅ | | DescribeWorkerDeploymentVersion | ✅ | ✅ | ✅ | | DescribeWorkflowExecution | ✅ | ✅ | ✅ | | DescribeWorkflowRule | ✅ | ✅ | ✅ | | ExecuteMultiOperation | | ✅ | ✅ | | FetchWorkerConfig | ✅ | ✅ | ✅ | | GetSearchAttributes | ✅ | ✅ | ✅ | | GetWorkerBuildIdCompatibility | ✅ | ✅ | ✅ | | GetWorkerTaskReachability | ✅ | ✅ | ✅ | | GetWorkerVersioningRules | ✅ | ✅ | ✅ | | GetWorkflowExecutionHistory | ✅ | ✅ | ✅ | | GetWorkflowExecutionHistoryReverse | ✅ | ✅ | ✅ | | ListActivityExecutions | ✅ | ✅ | ✅ | | ListBatchOperations | ✅ | ✅ | ✅ | | ListClosedWorkflowExecutions | ✅ | ✅ | ✅ | | ListOpenWorkflowExecutions | ✅ | ✅ | ✅ | | ListScheduleMatchingTimes | ✅ | ✅ | ✅ | | ListSchedules | ✅ | ✅ | ✅ | | ListTaskQueuePartitions | ✅ | ✅ | ✅ | | ListWorkerDeployments | ✅ | ✅ | ✅ | | ListWorkers | ✅ | ✅ | ✅ | | ListWorkflowExecutions | ✅ | ✅ | ✅ | | ListWorkflowRules | ✅ | ✅ | ✅ | | PatchSchedule | | ✅ | ✅ | | PauseActivity | | ✅ | ✅ | | PauseWorkflowExecution | | ✅ | ✅ | | PollActivityExecution | | ✅ | ✅ | | PollActivityTaskQueue | | ✅ | ✅ | | PollNexusTaskQueue | | ✅ | ✅ | | PollWorkflowExecutionUpdate | | ✅ | ✅ | | PollWorkflowTaskQueue | | ✅ | ✅ | | QueryWorkflow | ✅ | ✅ | ✅ | | RecordActivityTaskHeartbeat | | ✅ | ✅ | | RecordActivityTaskHeartbeatById | | ✅ | ✅ | | RecordWorkerHeartbeat | | ✅ | ✅ | | RequestCancelActivityExecution | | ✅ | ✅ | | RequestCancelWorkflowExecution | | ✅ | ✅ | | ResetActivity | | ✅ | ✅ | | ResetStickyTaskQueue | | ✅ | ✅ | | ResetWorkflowExecution | | ✅ | ✅ | | RespondActivityTaskCanceled | | ✅ | ✅ | | RespondActivityTaskCanceledById | | ✅ | ✅ | | RespondActivityTaskCompleted | | ✅ | ✅ | | RespondActivityTaskCompletedById | | ✅ | ✅ | | RespondActivityTaskFailed | | ✅ | ✅ | | RespondActivityTaskFailedById | | ✅ | ✅ | | RespondNexusTaskCompleted | | ✅ | ✅ | | RespondNexusTaskFailed | | ✅ | ✅ | | RespondQueryTaskCompleted | | ✅ | ✅ | | RespondWorkflowTaskCompleted | | ✅ | ✅ | | RespondWorkflowTaskFailed | | ✅ | ✅ | | SetWorkerDeploymentCurrentVersion | | ✅ | ✅ | | SetWorkerDeploymentManager | | ✅ | ✅ | | SetWorkerDeploymentRampingVersion | | ✅ | ✅ | | ShutdownWorker | | ✅ | ✅ | | SignalWithStartWorkflowExecution | | ✅ | ✅ | | SignalWorkflowExecution | | ✅ | ✅ | | StartActivityExecution | | ✅ | ✅ | | StartBatchOperation | | ✅ | ✅ | | StartWorkflowExecution | | ✅ | ✅ | | StopBatchOperation | | ✅ | ✅ | | TerminateActivityExecution | | ✅ | ✅ | | TerminateWorkflowExecution | | ✅ | ✅ | | TriggerWorkflowRule | | ✅ | ✅ | | UnpauseActivity | | ✅ | ✅ | | UnpauseWorkflowExecution | | ✅ | ✅ | | UpdateActivityOptions | | ✅ | ✅ | | UpdateSchedule | | ✅ | ✅ | | UpdateTaskQueueConfig | | ✅ | ✅ | | UpdateWorkerBuildIdCompatibility | | ✅ | ✅ | | UpdateWorkerConfig | | ✅ | ✅ | | UpdateWorkerDeploymentVersionMetadata | | ✅ | ✅ | | UpdateWorkerVersioningRules | | ✅ | ✅ | | UpdateWorkflowExecution | | ✅ | ✅ | | UpdateWorkflowExecutionOptions | | ✅ | ✅ | ## Project-level permissions [Project-level permissions](/cloud/manage-access/permissions-reference#project-level-permissions) are granted to users, groups, and service accounts by assigning them a Project-level role. Temporal Cloud supports the following Project-level roles: - Project Admin - Project Write - Project Read - Project Contribute - Project List - Project Member Users with the Global Admin and Account Owner roles automatically have Project Admin on all Projects in the account. Project Admin, Project Write, and Project Read also grant Namespace Admin, Write, and Read respectively on every Namespace in the Project. Project Contribute, Project List, and Project Member grant no Namespace-level access. This is why Project Read and Project List grant the same Cloud Ops API permissions, as do Project Write and Project Contribute: they differ in the Namespace-level and [Workflow-level permissions](#workflow-level-permissions) they carry, not in the Cloud Ops API endpoints they can call. ### Cloud Ops API permissions This table provides API-level details for permissions granted through Project-level roles. These permissions are configured per Project per user. These permissions control who can manage Nexus Endpoints in the Project. They do not restrict which Namespaces can call an Endpoint at runtime, which continues to be controlled by the Endpoint [allowlist](/nexus/security#runtime-access-controls). See [Use Nexus across Projects](/cloud/projects#use-nexus-across-projects). | Permission | Member | List | Read | Contribute | Write | Project Admin | | ----------------------------------- | :----: | :--: | :--: | :--------: | :---: | :-----------: | | [CreateConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules) | | | | ✅ | ✅ | ✅ | | [CreateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces) | | | | ✅ | ✅ | ✅ | | [CreateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints) | | | | ✅ | ✅ | ✅ | | [CreateServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/POST/cloud/service-accounts) | | | | | | ✅† | | [DeleteConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/DELETE/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | | | | ✅ | ✅ | ✅ | | [DeleteNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/DELETE/cloud/nexus/endpoints/%7BendpointId%7D) | | | | ✅ | ✅ | ✅ | | [DeleteProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/DELETE/cloud/projects/%7BprojectId%7D) | | | | | | ✅ | | [GetConnectivityRule](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules/%7BconnectivityRuleId%7D) | | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetConnectivityRules](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/GET/cloud/connectivity-rules) | | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNamespaces](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/GET/cloud/namespaces) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints/%7BendpointId%7D) | | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetNexusEndpoints](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/GET/cloud/nexus/endpoints) | | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | [GetProjectScopedServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/service-accounts) | | | | | | ✅ | | [GetServiceAccount](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/service-accounts/%7BserviceAccountId%7D) | | ✅† | ✅† | ✅† | ✅† | ✅† | | [GetServiceAccountProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/service-account-assignments) | | | | | | ✅ | | [GetServiceAccounts](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/service-accounts/GET/cloud/service-accounts) | | ✅† | ✅† | ✅† | ✅† | ✅† | | [GetUserGroupProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/user-group-assignments) | | | | | | ✅ | | [GetUserProjectAssignments](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/GET/cloud/projects/%7BprojectId%7D/user-assignments) | | | | | | ✅ | | [SetServiceAccountProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/service-accounts/%7BserviceAccountId%7D/access) | | | | | | ✅ | | [SetUserGroupProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/user-groups/%7BgroupId%7D/access) | | | | | | ✅ | | [SetUserProjectAccess](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D/users/%7BuserId%7D/access) | | | | | | ✅ | | [UpdateNexusEndpoint](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints/%7BendpointId%7D) | | | | ✅ | ✅ | ✅ | | [UpdateProject](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/%7BprojectId%7D) | | | | | | ✅ | - † See [Service Account Authorization Behavior](#service-account-authorization-behavior) Project Developer is a compatibility role that users with the Account Developer role inherit automatically. It cannot be assigned directly. It grants everything Project Contribute grants except CreateConnectivityRule and DeleteConnectivityRule, and carries no Namespace-level access. For details, see [Project roles](/cloud/projects#project-roles). ## Custom Role permissions reference The following tables list the permission action strings available when defining [Custom Roles](/cloud/manage-access/custom-roles). Use these strings in the `actions` field of a Custom Role permission grant. Not all Cloud Ops API operations can be assigned to a Custom Role. For details on which operations are excluded and why, see [Available permissions](/cloud/manage-access/custom-roles#available-permissions). ## Additional authorization behaviors Some APIs are granted to all account-level roles but enforce additional authorization rules at runtime. The action group grants access to call the API, but the scope of what the caller can interact with depends on their role. ### API key authorization behavior All roles can create and manage their **own** API keys. An API key inherits the permissions of its owner — it cannot grant access beyond what the owning user or service account already has. | Behavior | Read-only | Developer | Finance Admin | Global Admin | Account Owner | | --------------------------------------------------- | :-------: | :-------: | :-----------: | :----------: | :-----------: | | Create, view, update, and delete own API keys | ✅ | ✅ | ✅ | ✅ | ✅ | | View, update, and delete any API key in the account | | | | ✅ | ✅ | **Affected APIs:** CreateApiKey, GetApiKey, GetApiKeys, UpdateApiKey, DeleteApiKey ### Service account authorization behavior All roles can list service accounts within their account. However, the ability to create, update, and delete service accounts depends on the scope of the service account and the caller's role. | Behavior | Read-only | Developer | Finance Admin | Global Admin | Account Owner | | ------------------------------------------------ | :-------: | :-------: | :-----------: | :----------: | :-----------: | | List all service accounts in the account | ✅ | ✅ | ✅ | ✅ | ✅ | | Manage unscoped (account-level) service accounts | | | | ✅ | ✅ | | Manage Namespace-scoped service accounts | § | § | § | ✅ | ✅ | | Manage Project-scoped service accounts | † | † | † | ✅ | ✅ | § Requires Namespace Admin permission on the target Namespace. Any role can manage Namespace-scoped service accounts if they hold Namespace Admin on that Namespace. † Requires Project Admin permission on the target Project. Any role can manage Project-scoped service accounts if they hold Project Admin on that Project. **Affected APIs:** CreateServiceAccount, GetServiceAccount, GetServiceAccounts, UpdateServiceAccount, DeleteServiceAccount ### User authorization behavior Every account-level role can call GetUser and GetUsers. The response includes the user's ID, email, and state. Another user's access (account role, Namespace permissions, Project access, and Custom Roles) and invitation are returned only to Account Owner and Global Admin. Namespace Admin does not change this. GetUser of the caller always includes that caller's own access. To list who has access to a Namespace, use GetUserNamespaceAssignments. **Affected APIs:** GetUser, GetUsers --- # Roles and permissions Source: https://docs.temporal.io/cloud/manage-access/roles-and-permissions > Temporal Cloud RBAC gives every access principal an account-level role, with Namespace-level permissions and optional Custom Roles for granular access. Temporal Cloud uses role-based access control (RBAC) to manage access to resources. Access is governed both on the account-level and within a Namespace. At the account-level, each access principal is assigned one [account-level role](#account-level-roles). At the Namespace-level, each access principal can be assigned one [Namespace-level permissions](#namespace-level-permissions). Some account-level roles, such as Account Owner and Global Admin, automatically have Namespace Admin permissions on all Namespaces in the account. Each access principal, including users, groups, and Service Accounts, must have a predefined account-level role. For more granular access control, you can also use [Custom Roles](/cloud/manage-access/custom-roles) to define specific permissions for specific Temporal Cloud resources and assign them to any principal. ## Account-level roles Account-level roles are assigned to access principals at the account level. They control access to account resources, such as: - Users and Service Accounts - Billing and usage - Namespaces. This includes creating and managing Namespaces only, not access to resources within a Namespace, which is controlled by [Namespace-level permissions](#namespace-level-permissions). - Nexus Endpoints The following table provides a summary of the account-level roles and their primary purpose. Refer to the [Permissions reference](/cloud/manage-access/permissions-reference#account-level-access) for API-level details. | Role | Primary purpose | Can create Namespaces | Automatic Namespace Admin | Billing and usage access | | ------------- | ------------------------------------------- | --------------------- | --------------------------------------- | --------------------------------- | | Account Owner | Owns and governs the account | Yes | All Namespaces (cannot be revoked) | Full billing, payments, and usage | | Global Admin | Administers account configuration and users | Yes | All Namespaces (cannot be revoked) | Usage only | | Developer | Creates and manages Namespaces they own | Yes | Namespaces they create (can be revoked) | None | | Finance Admin | Manages billing and payment information | No | None | Full billing, payments, and usage | | Read-Only | Views account configuration and resources | No | None | None | Account-level roles don't govern day-to-day operations within a Namespace. Access to resources inside a Namespace, such as Workflows and Workflow Executions, is controlled by [Namespace-level permissions](#namespace-level-permissions). Account Owner and Global Admin roles automatically have Namespace Admin permissions on all Namespaces in the account, and these permissions cannot be revoked without removing the role. Developers can create Namespaces, and have Namespace Admin permissions for each Namespace they create. This permission can be revoked. Developer roles also don't have automatic access to Namespaces that they didn't create. ### Best practice for assigning the Account Owner role The Account Owner role holds the highest level of access in the system. This role configures account-level parameters and manages Temporal billing and payment information. It allows users to perform all actions within the Temporal Cloud account. We strongly recommend the following precautions when assigning the Account Owner role to users: - Assign the role to at least two users in your organization. Otherwise, limit the number of users with this role. - Associate a person’s direct email address to the Account Owner, rather than a shared or generic address, so Temporal Support can contact the right person in urgent situations. This latter rule is useful for anyone on your team who may need to be contacted urgently, regardless of their Account role. ## Namespace-level permissions Namespace-level permissions govern access to resources within a Namespace, such as the following: - Workflows - Workflow Executions - Task Queues - Activity Executions - Search Attributes - History - Events Namespace-level permissions are assigned to access principals within a Namespace. Each permission has a set of actions that grant access to specific resources within the Namespace. The following table provides a summary of the Namespace-level permissions and their primary purpose. Refer to the [Permissions reference](/cloud/manage-access/permissions-reference#namespace-level-permissions) for API-level details. | Permission level | Intended use | Human access | Worker runtime access | Namespace administration | | ---------------- | --------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Read | Observe Namespace activity | View Workflows, Workflow Executions, Schedules, Task Queues, and metadata | None | None | | Write | Operate Workflows and run Workers | Start, signal, cancel, terminate, and reset Workflows; manage Schedules and batch operations | Poll Task Queues and complete Workflow and Activity Tasks | None | | Namespace Admin | Administer the Namespace | All Read and Write capabilities | All Read and Write capabilities | Update Namespace settings, manage Search Attributes, Export Sinks, replication, and Namespace user access | You can grant Namespace Admin, Write, or Read-Only permissions to principals with the account-level roles of Developer, Finance Admin, or Read-Only. Account Owners and Global Admins already have Namespace Admin permissions on all Namespaces in the account and do not need to be manually assigned Namespace-level permissions. --- # SAML authentication Source: https://docs.temporal.io/cloud/manage-access/saml To authenticate the users of your Temporal Cloud account, you can connect an identity provider (IdP) to your account by using Security Assertion Markup Language (SAML) 2.0. > **ℹ️ Info:** > > SAML is a paid feature. See the [pricing page](/cloud/pricing) for details. > ## Integrate SAML with your Temporal Cloud account 1. Locate your [Temporal Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id). Your Account Id can be viewed and copied from the Temporal Cloud user profile dropdown menu in the top right corner. Alternatively, find your [Namespace Id](/cloud/namespaces#temporal-cloud-namespace-id). The Account Id is the five or six characters following the period (.), such as `f45a2`. You will need the Account Id to construct your callback URL and your entity identifier. 1. Configure SAML with your IdP: - [Microsoft Entra ID example](#configure-saml-with-azure-ad) - [Okta example](#configure-saml-with-okta) - [Generic SAML 2.0 configuration](#configure-saml-with-other-identity-providers) 1. [Share your connection information with us and test your connection.](#finish-saml-configuration) ## How to configure SAML with Microsoft Entra ID If you want to use the general Microsoft login mechanism, you don't need to set up SAML with Entra ID. Just select **Continue with Microsoft** on the Temporal Cloud sign-in page. To use Entra ID as your SAML IdP, create a Microsoft Entra ID Enterprise application. 1. Sign in to the [Microsoft Entra ID](https://portal.azure.com/). 1. On the home page, under **Manage Microsoft Entra ID**, select **View**. 1. On the **Overview** page near the top, select **Add > Enterprise application**. 1. On the **Browse Microsoft Entra ID Gallery** page near the top, select **Create your own application**. 1. In the **Create your own application** pane, provide a name for your application (such as `temporal-cloud`) and select **Integrate any other application you don't find in the gallery**. 1. Select **Save**. 1. In the **Getting Started** section, select **2. Set up single sign on**. 1. On the **Single sign-on** page, select **SAML**. 1. In the **Basic SAML Configuration** section of the **SAML-based Sign-on** page, select **Edit**. 1. In **Identifier (Entity ID)**, enter the following entity identifier, including your Account Id where indicated: ```bash urn:auth0:prod-tmprl:ACCOUNT_ID-saml ``` A correctly formed entity identifier looks like this: ```bash urn:auth0:prod-tmprl:f45a2-saml ``` 1. In **Reply URL (Assertion Consumer Service URL)**, enter the following callback URL, including your Account Id where indicated: ```bash https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml ``` A correctly formed callback URL looks like this: ```bash https://login.tmprl.cloud/login/callback?connection=f45a2-saml ``` 1. In **Sign on URL**, enter the following login url, including your Account Id where indicated: ```bash https://cloud.temporal.io/login/saml?connection=ACCOUNT_ID-saml ``` A correctly formed login URL looks like this: ```bash https://cloud.temporal.io/login/saml?connection=f45a2-saml ``` 1. You can leave the other fields blank. Near the top of the pane, select **Save**. 1. In the **Attributes & Claims** section, select **Edit**. Configure the following settings. Under **Required claim**: - Set **Unique User Identifier (NameID)** to `user.userprincipalname` - Set the **NameID format** to `emailAddress` These are the default settings for Microsoft Entra ID. Then under **Additional claims**, ensure **Email** and **Name** are present. 1. Collect information that you need to send to us: - In the **SAML Certificates** section of the **SAML-based Sign-on** page, select the download link for **Certificate (Base64)**. - In the **Set up _APPLICATION_NAME_** section of the **SAML-based Sign-on** page, copy the value of **Login URL**. To finish setting up Microsoft Entra ID as your SAML IdP, see [Finish SAML configuration](#finish-saml-configuration). ## How to configure SAML with Okta To use Okta as your SAML IdP, configure a new Okta application integration. 1. Sign in to the [Okta Admin Console](https://www.okta.com/login/). 1. In the left navigation pane, select **Applications > Applications**. 1. On the **Applications** page, select **Create App Integration**. 1. In the **Create a new app integration** dialog, select **SAML 2.0** and then select **Next**. 1. On the **Create SAML Integration** page in the **General Settings** section, provide a name for your application (such as `temporal-cloud`) and then select **Next**. 1. In the **Configure SAML** section in **Single sign on URL**, enter the following callback URL, including your Account Id where indicated: ```bash https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml ``` A correctly formed callback URL looks like this: ```bash https://login.tmprl.cloud/login/callback?connection=f45a2-saml ``` 1. In **Audience URI (SP Entity ID)**, enter the following entity identifier, including your Account Id where indicated: ```bash urn:auth0:prod-tmprl:ACCOUNT_ID-saml ``` A correctly formed entity identifier looks like this: ```bash urn:auth0:prod-tmprl:f45a2-saml ``` 1. We require the user's full email address when connecting to Temporal. - In **Name ID format**, select `EmailAddress`. - In **Attribute Statements**, set **email** and **name**. 1. Select **Next**. 1. In the **Feedback** section, select **Finish**. 1. On the **Applications** page, select the name of the application integration you just created. 1. On the application integration page, select the **Sign On** tab. 1. Under **SAML Setup**, select **View SAML setup instructions**. 1. Collect information that you need to send to us: - Copy the IdP settings. - Download the active certificate. To finish setting up Okta as your SAML IdP, see [Finish SAML configuration](#finish-saml-configuration). ## How to configure SAML with another identity provider Temporal Cloud works with identity providers that meet the SAML 2.0 requirements described below. Provider names in this documentation are configuration examples, not an exhaustive compatibility list. An identity provider does not need to be listed to work with Temporal Cloud. Configure your IdP with the following values, replacing `ACCOUNT_ID` with your Temporal Cloud Account Id: - **Service provider entity identifier:** ```text urn:auth0:prod-tmprl:ACCOUNT_ID-saml ``` - **Assertion Consumer Service URL:** ```text https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml ``` Configure the SAML connection to: - Use SAML 2.0 with HTTP POST binding. - Use an RSA X.509 signing certificate and RSA-SHA256 signing. - Provide a stable subject `NameID` that uniquely identifies each user. - Include the user's email address and name as attributes. The following example shows how to apply the same configuration to an additional identity provider. It does not represent the complete set of compatible providers. ### PingFederate configuration example Create a SAML 2.0 Browser SSO service provider connection in PingFederate using the Temporal Cloud entity identifier and Assertion Consumer Service URL shown above. Configure the connection with an RSA signing certificate, a stable subject `NameID`, and the required `email` and `name` attributes. For IdP-initiated SSO, publish the connection-specific `/idp/startSSO.ping` URL in your organization's application portal. For detailed instructions, see: - [Configure IdP Browser SSO](https://docs.pingidentity.com/pingfederate/13.1/administrators_reference_guide/help_spconnectionconfigtasklet_spbrowserssostate.html) - [PingFederate IdP endpoints](https://docs.pingidentity.com/pingfederate/13.1/developers_reference_guide/pf_idp_endpoints.html) After configuring your IdP, continue to [Finish SAML configuration](#finish-saml-configuration). ## How to finish your SAML configuration After you configure SAML with your IdP, we can finish the configuration on our side. [Create a support ticket](/cloud/support#support-ticket) that includes the following information: - The sign-in URL from your application - The X.509 SAML sign-in certificate in PEM format - One or more IdP domains to map to the SAML connection Generally, the provided IdP domain is the same as the domain for your email address. You can provide multiple IdP domains. When you receive confirmation from us that we have finished configuration, log in to Temporal Cloud. This time, though, enter your email address in **Enterprise identity** and select **Continue**. Do not select **Continue with Google** or **Continue with Microsoft**. You will be redirected to the authentication page of your IdP. --- # SCIM user management Source: https://docs.temporal.io/cloud/manage-access/scim > Link your IdP with your Temporal Cloud account to securely automate user and group management. [SCIM](https://scim.cloud/) lets you integrate your identity provider (IdP) with Temporal Cloud to automate user provisioning and access. Once SCIM is configured, changes in your IdP are automatically reflected in Temporal Cloud, including: - User creation / onboarding - User deletion / offboarding - User membership in groups You can map SCIM groups to Temporal Cloud [roles and permissions](/cloud/manage-access/users#account-level-roles-and-namespace-level-permissions), so users automatically get the Temporal Cloud access they need based on the groups they belong to. > **ℹ️ Info:** > > SCIM is a paid feature. See the [pricing page](/cloud/pricing) for details. > ## Supported IdP Vendors Supported upstream IdP vendors include: * [Okta](#configure-scim-with-okta) * Microsoft Entra ID (Azure AD) * Google Workspace * OneLogin * CyberArk * JumpCloud * PingFederate * Any SCIM 2.0-compliant provider ## Preparing for SCIM Before starting your work with SCIM, you'll need to complete this checklist: 1. Configure [SAML](/cloud/manage-access/saml) SSO. 1. Identify your organization's **IdP administrator**, who is responsible for configuring and managing your SCIM integration. Specify their contact details when you reach out to support in the next stage of this process. After completing these steps, you're ready to submit your [support ticket](/cloud/support#support-ticket) to enable SCIM. > **💡 Tip:** > Adding and removing users > > When SCIM is enabled for user management, you can still add and remove users outside of SCIM using the Temporal Cloud interface, until you disable user lifecycle management. > You can always change a user's or group's Account Role from the Temporal Cloud interface. > ## Onboarding with SCIM and Okta 1. Temporal Support enables the SCIM integration on your account. Enabling integration automatically emails a configuration link to your Okta administrator. This authorizes them to set up the integration. 1. Your Okta administrator opens the supplied link. The link leads to step-by-step instructions for configuring the integration. 1. Once configured in Okta, Temporal Cloud will begin to receive SCIM messages and automatically onboard and offboard the users and groups configured in Okta. Some points to note: - User and group change events are applied within 10 minutes of them being made in Okta. - User lifecycle management with SCIM also allows user roles to be derived from group membership. - Once a group has been synced in Temporal Cloud, you can use `tcld` to assign roles to the group. For instructions, see the [User Group Management](https://github.com/temporalio/tcld?tab=readme-ov-file#user-group-management) page. --- # Manage service accounts Source: https://docs.temporal.io/cloud/manage-access/service-accounts > Service Accounts are machine identities that authenticate to Temporal Cloud with API Keys, managed via the Cloud UI or tcld. Temporal Cloud provides Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) with the option to create machine identities named Service Accounts. Service Accounts are a type of identity in Temporal Cloud. Temporal Cloud supports User identities as a representation of a human user who uses Temporal Cloud. Service Accounts afford Temporal Cloud Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) the ability to create an identity for machine authentication, an identity not associated with a human user. With the addition of Service Accounts, Temporal Cloud now supports 2 identity types: - Users (tied to a human, identified by email address or ID) - Service Accounts (not tied to a human, email address optional, identified by name or ID) Service Accounts use API Keys as the authentication mechanism to connect to Temporal Cloud. You should use Service Accounts to represent a non-human identity when authenticating to Temporal Cloud for operations automation or the Temporal SDKs and the Temporal CLI for Workflow Execution and management. > **💡 Tip:** > > Namespace Admins can now manage and create [Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped), regardless of their Account Role. > ## Manage Service Accounts Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) can manage Service Accounts by creating, viewing, updating, deleting Service Accounts using the following tools: - Temporal Cloud UI - Temporal CLI with the [Temporal Cloud extension](/cli/cloud) - Use `temporal cloud service-account --help` for a list of all service-account commands - Temporal Cloud CLI (tcld) - Use `tcld service-account --help` for a list of all service-account commands Account Owner and Global Admin [roles](/cloud/manage-access/users#account-level-roles) also have the ability to manage API Keys for Service Accounts. ### Prerequisites - A Cloud user account with Account Owner or Global Admin [role](/cloud/manage-access/users#account-level-roles) permissions - Access to the Temporal Cloud UI, the Temporal CLI with the [Temporal Cloud extension](/cli/cloud), or the Temporal Cloud CLI (tcld) - Enable access to API Keys for your Account - To manage Service Accounts using the Temporal Cloud CLI (tcld), upgrade to the latest version of tcld (v0.18.0 or higher) using `brew upgrade tcld`. - If using a version of tcld less than v0.31.0, enable Service Account commands with `tcld feature toggle-service-account`. ### Create a Service Account Create a Service Account using the Temporal Cloud UI or the CLI. While User identities are invited to Temporal Cloud, Service Accounts are created in Temporal Cloud. **Using the Cloud UI** 1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities) 2. Click the `Create Service Account` button located near the top of the `Identities` page 3. Provide the following information: - **Name** (required) - **Description** (optional) - **Account Level Role** (required) - **Namespace Permissions** (optional) - Use this section of the Create Service Account page to grant the Service Account access to individual Namespaces 4. Click `Create Service Account` at the bottom of the page - A status message is displayed at the bottom right corner of the screen and on the next screen - You will be prompted to create an API Key for the Service Account (optional) 5. (Optional) Create API Key - It is recommended to create an API Key for the Service Account right after you create the Service Account, though you can create/manage API Keys for Service Accounts at any time - See the API Key [documentation](/cloud/api-keys) for more information on creating and managing API Keys **Using the Temporal CLI** To create a Service Account, use the `temporal cloud service-account create` command: ``` temporal cloud service-account create --name "sa_test" --description "this is a test SA" --account-role read ``` This example creates a Service Account with the name `sa_test`, description `this is a test SA`, and a `read` Account Role. Creating a Service Account requires `--name`. You can optionally assign an Account Role with `--account-role` or grant Namespace Permissions with `--namespace-access`, in the format `namespace.account=permission`. Repeat `--namespace-access` to grant access to more than one Namespace. Creating a Service Account returns the Service Account ID, which you use to retrieve, update, or delete the Service Account. **Using tcld** To create a Service Account using tcld, use the `tcld service-account create` command: ``` tcld service-account create -n "sa_test" -d "this is a test SA" --ar "Read" ``` This example creates a Service Account with the name `"sa_test"`, description `"this is a test SA"`, and a `Read` Account Role. Creating a Service Account requires the following attributes: `name` and `account-role` (as above). You can also provide the Namespace Permissions for the Service Account using the `—-np` flag. Creating a Service Account returns the `ServiceAccountId` which is used to retrieve, update, or delete a Service Account. ### View Service Accounts View a single or all Service Account(s) using the Temporal Cloud UI or the CLI. **Using the Cloud UI** Service Accounts are listed on the `Identities` section of the `Settings` page, along with Users. To locate a Service Account: 1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities) 2. Select the `Service Accounts` filter **Using the Temporal CLI** To view all Service Accounts in your account, use the `temporal cloud service-account list` command: ``` temporal cloud service-account list ``` **Using tcld** To view all Service Accounts in your account using tcld, use the `tcld service-account list` command: ``` tcld service-account list ``` ### Delete a Service Account Delete a Service Account using the Temporal Cloud UI or the CLI. When you delete a Service Account, all associated API keys are automatically deleted as well. Therefore, you don't need to manually remove API keys after deleting a Service Account. **Using the Cloud UI** 1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities) 2. Find the relevant Service Account 3. Select the vertical ellipsis menu in the Service Account row 4. Select `Delete` 5. Confirm the delete action when prompted **Using the Temporal CLI** To delete a Service Account, use the `temporal cloud service-account delete` command: ``` temporal cloud service-account delete --service-account-id "e9d87418221548" ``` Run the Service Account list command to confirm the Service Account has been removed from the account. The Service Account is deleted when it is no longer visible in the output of the list command. **Using tcld** To delete a Service Account using tcld, use the `tcld service-account delete` command: ``` tcld service-account delete --service-account-id "e9d87418221548" ``` Use the tcld Service Account list command to validate the Service Account has been removed from the account. The Service Account is deleted when it is no longer visible in the output of the list command. ### Update a Service Account Update a Service Account's description using the Temporal Cloud UI or the CLI. **Using the Cloud UI** 1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities) 2. Find the relevant Service Account 3. Select the vertical ellipsis menu in the Service Account row 4. Select `Edit` 5. Make changes to the Service Account - You can change the Service Account's name, description, Account Level Role, and Namespace Permissions 6. Click the `Save` button located in the bottom left of the screen - A status message is displayed at the bottom right corner of the screen **Using the Temporal CLI** The `temporal cloud service-account update` command changes the name, description, Account Role, and Namespace access of a Service Account: ``` temporal cloud service-account update --service-account-id "2f68507677904e09b9bcdbf93380bb95" \ --description "new description" ``` Use `--account-role` to change the Account Role and `--namespace-access` to change Namespace access. For a Namespace-scoped Service Account, use `--namespace-permission` instead. **Using tcld** Three different commands exist to help users update a Service Account using tcld: - `tcld service-account update`: to update a Service Account's name or description field - `tcld service-account set-account-role`: to update a Service Account's Account Role - `tcld service-account set-namespace-permissions`: to update a Service Account's Namespace Permissions Example: ``` tcld service-account update --id "2f68507677904e09b9bcdbf93380bb95" -d "new description" ``` ## Namespace-scoped Service Accounts There is a special type of Service Account, called a Namespace-scoped Service Account, which shares the same functionality as the Service Accounts above, but is limited (or scoped) to a single namespace. In particular, a Namespace-scoped Service Account must _always_ have: - A `Read` Account Role - A single Namespace Permission Note that a Namespace-scoped Service Account cannot be reassigned to a different Namespace after creation, but its Namespace permission can be modified (for example, from `Read` to `Write`). Namespace-scoped Service Accounts are useful in situations when you need to restrict a client's access to a single Namespace. You can retrieve, update, and delete a Namespace-scoped Service Account using the same process and commands as above, but creation is slightly different. ### Permissions Unlike regular Service Accounts, which require a Global Admin or Account Owner role, Namespace-scoped Service Accounts can be created and managed by Namespace Admins. For example, an Account Developer with Namespace Admin for `test_ns` can create a Service Account scoped to `test_ns`. Global Admins and Account Owners can also create Namespace-scoped Service Accounts, as they implicitly have Namespace Admin rights for all Namespaces. ### Create a Namespace-scoped Service Account As with regular Service Accounts, Namespace-scoped Service Accounts can be created using the Temporal Cloud UI or the CLI. #### Using the Cloud UI Currently, creating a Namespace-scoped Service Account from the Temporal Cloud UI happens on an individual [Namespace](/cloud/namespaces#manage-namespaces) page. If the current Namespace has API key authentication enabled, then there will be a `Generate API Key` button as a banner on the top of the Namespace page or in the `Authentication` section. By clicking on the `Generate API Key` button, a Namespace-scoped Service Account will be automatically created for the given Namespace (if one does not already exist) and an associated API key will be displayed. This key will have the maximum expiration time, which is 2 years. The resulting Namespace-scoped Service Account will be named `-service-account` and will have an `Admin` Namespace permission by default. #### Using the Temporal CLI To create a Namespace-scoped Service Account, use the `temporal cloud service-account create-namespace-scoped` command: ``` temporal cloud service-account create-namespace-scoped --name "test-scoped-sa" \ --namespace "test-ns." --namespace-permission admin ``` #### Using tcld To create a Namespace-scoped Service Account with tcld, use the `tcld service-account create-scoped` command: ``` tcld service-account create-scoped -n "test-scoped-sa" --np "test-ns=Admin" ``` This example creates a Namespace-scoped Service Account for the Namespace `test-ns`, named `test-scoped-sa`, with `Admin` Namespace Permission. Note that the Account Role is omitted, since Namespace-scoped Service Accounts always have a `Read` Account Role. ### Lifecycle When a Namespace is deleted, all associated Namespace-scoped Service Accounts and their associated API keys are automatically deleted as well. Therefore, you do not need to manually remove Namespace-scoped Service Accounts and their API keys after deleting a Namespace. ## Project-scoped Service Accounts > **Pre-release** Project-scoped Service Accounts can be created and managed within a Project. Unlike Account-scoped Service Accounts, they can only access resources within their Project, making them useful for workloads that shouldn't have Account-wide permissions. For more information, see [Add resources to a Project](/cloud/projects/#add-resources-to-a-project). --- # Manage user groups Source: https://docs.temporal.io/cloud/manage-access/user-groups > Assign account-level roles and Namespace permissions to a group of users at once, and see how SCIM groups interact with them. ## What are user groups? User groups can be used to help manage sets of users that should have the same access. Instead of separately assigning the same role to individual users, a user group can be created, assigned the desired roles, and then users added to the user group. This eases the toil of managing individual user permissions and can simplify access management. When a new role is needed, it can be added to the group once and all users' access will reflect the new role. User groups can be assigned both [account-level roles](/cloud/manage-access/users#account-level-roles) and [namespace-level permissions](/cloud/manage-access/users#namespace-level-permissions). One user can be assigned to many groups. In the event that a user's group memberships have multiple roles for the same resource, the user will have an effective role of the most permissive of the permissions. For example if `Group A` grants a read-only role to a namespace, but `Group B` grants a write role to a namespace then a user that belongs to both `Group A` and `Group B` would have the write role to the namespace. [Service accounts](/cloud/manage-access/service-accounts) cannot be assigned to user groups. Only users with the Account Owner or Global Admin account-level [role](/cloud/manage-access/users#account-level-roles) can manage user groups. ## How SCIM groups work with user groups [SCIM groups](/cloud/manage-access/scim) work similarly to user groups with respect to role assignment. Unlike a user group, the lifecycle of a SCIM group is fully managed by the SCIM integration which means: 1. SCIM groups cannot be created except through the SCIM integration 1. SCIM groups cannot be deleted except through the SCIM integration 1. SCIM group membership is managed through the SCIM integration User groups and SCIM groups can be used simultaneously in a single Temporal Cloud account. One user may belong to multiple SCIM groups and to multiple user groups. Using user group and SCIM groups together can be useful when the groups defined in the identity provider (IDP) don't map cleanly to the access you need to grant in Temporal Cloud. Instead of having to update the IDP (which is often sensitive and time-consuming), you can use Temporal Cloud user groups to manage access. > **ℹ️ Info:** > > All user group administration requires an Account Owner or Global Admin account-level [role](/cloud/manage-access/users#account-level-roles). > ## How to create a user group in your Temporal Cloud account User group names must be 3-64 characters long and can only contain lowercase letters, numbers, hyphens, and underscores. **Web UI** 1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities) 1. Click the Create Group button 1. Name the group 1. Assign an account-level role to the group (you can assign namespace-level permissions after the group is created) 1. Click Save **Temporal CLI** Use the command that matches how the group is backed: - [`temporal cloud user-group create-cloud-group`](/cli/command-reference/cloud/user-group#create-cloud-group) for a group managed in Temporal Cloud - [`temporal cloud user-group create-google-group`](/cli/command-reference/cloud/user-group#create-google-group) for a group backed by a Google group - [`temporal cloud user-group create-scim-group`](/cli/command-reference/cloud/user-group#create-scim-group) for a group backed by SCIM **tcld** See the [`tcld` user-group create](/cloud/tcld/user-group/#create) command reference for details. **Terraform** See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details. ## How to assign roles to a user group **Web UI** To edit the account role of a group: 1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities) 1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table) 1. Click Edit Group 1. Click the Account Role dropdown 1. Select a new account role 1. Click Save To add namespace permissions to a group: 1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities) 1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table) 1. Click Edit Group 1. Click Add Namespaces 1. Under Grant Access to a Namespace, search for the namespace you’d like to add permissions for 1. Select the namespace 1. Click the pencil to edit the permissions for the selected namespace 1. Click Save To edit or remove namespace permissions from a group: 1. Click Edit Group 1. Click the pencil on a permission to edit it, or the trash can to delete it 1. Click Save **Temporal CLI** Account-level roles and Namespace-level permissions are set separately: - [`temporal cloud user-group set-account-role`](/cli/command-reference/cloud/user-group#set-account-role) - [`temporal cloud user-group set-namespace-permissions`](/cli/command-reference/cloud/user-group#set-namespace-permissions) **tcld** See the [`tcld` user-group set-access](/cloud/tcld/user-group/#set-access) command reference for details. **Terraform** See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details. ## How to manage users in a group **Web UI** To add users to the group: 1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities) 1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table) 1. Click Edit Group 1. Under Members, search for the user you’d like to add 1. Select the user 1. Click Save To remove a user from the group: 1. Click Edit Group 1. Under Members, click the X next to the user you’d like to remove 1. Click Save **Temporal CLI** See the [`temporal cloud user-group members add`](/cli/command-reference/cloud/user-group#members-add), [`temporal cloud user-group members remove`](/cli/command-reference/cloud/user-group#members-remove), and [`temporal cloud user-group members list`](/cli/command-reference/cloud/user-group#members-list) command reference for details. **tcld** See the [`tcld` user-group add-users](/cloud/tcld/user-group/#add-users) and the [`tcld` user-group remove-users](/cloud/tcld/user-group/#remove-users) command reference for details. **Terraform** See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details. ## Delete a user group **Web UI** 1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities) 1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table) 1. Click the dropdown next to the edit button 1. Click Delete 1. Confirm by clicking Delete **Temporal CLI** See the [`temporal cloud user-group delete`](/cli/command-reference/cloud/user-group#delete) command reference for details. **tcld** See the [`tcld` user-group delete](/cloud/tcld/user-group/#delete) command reference for details. **Terraform** See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details. --- # Manage users Source: https://docs.temporal.io/cloud/manage-access/users > Invite users, set account-level roles and Namespace-level permissions, and remove users from a Temporal Cloud account with the Web UI, the CLI, or the Cloud Ops API. - [How to invite users to your Temporal Cloud account](#invite-users) - [What are the account-level roles?](#account-level-roles) - [What are the Namespace-level permissions?](#namespace-level-permissions) - [How to update an account-level Role in Temporal Cloud](#update-roles) - [How to update Namespace-level permissions in Temporal Cloud](#update-permissions) - [How to delete a user from your Temporal Cloud account](#delete-users) - [How to troubleshoot account access issues](#troubleshoot-access) ## How to invite users to your Temporal Cloud account # User management > Learn how to manage user invitations for Temporal Cloud **Web UI** To invite users using the Temporal Cloud UI: 1. In Temporal Web UI, select **Settings** in the left portion of the window. 1. On the **Settings** page, select **Create Users** in the upper-right portion of the window. 1. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses. 1. In **Account-Level Role**, select a [Role](/cloud/manage-access/roles-and-permissions#account-level-roles). The Role applies to all users whose email addresses appear in **Email Addresses**. 1. If the account has any Namespaces, they are listed under **Grant access to Namespaces**. To add a permission, select the checkbox next to a Namespace, and then select a [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions). Repeat as needed. 1. When all permissions are assigned, select **Send Invite**. **Temporal CLI** Use the [`temporal cloud user invite`](/cli/command-reference/cloud/user#invite) command. Specify the user's email, an account-level role, and optionally one or more Namespace permissions. Available account roles: `owner` | `admin` | `developer` | `finance-admin` | `read` | `metrics-read`. Available Namespace permissions: `admin` | `write` | `read`. ```command temporal cloud user invite \ --email \ --account-role \ --namespace-access = ``` Repeat `--namespace-access` to grant permissions on more than one Namespace. `--email` takes a single address, so invite one user per command: ```command temporal cloud user invite \ --email user1@example.com \ --account-role developer \ --namespace-access ns1.my-account=admin \ --namespace-access ns2.my-account=write ``` **tcld** Use the [`tcld user invite`](/cloud/tcld/user/#invite) command. Specify the user's email, an account-level role, and optionally one or more Namespace permissions. Available account roles: `admin` | `developer` | `read`. Available Namespace permissions: `Admin` | `Write` | `Read`. ```command tcld user invite \ --user-email \ --account-role \ --namespace-permission = ``` You can invite multiple users and assign multiple Namespace permissions in a single command: ```command tcld user invite \ --user-email user1@example.com \ --user-email user2@example.com \ --account-role developer \ --namespace-permission ns1=Admin \ --namespace-permission ns2=Write ``` ### Frequently asked questions #### Can multiple Temporal Cloud accounts share the same email domain? Yes. Multiple Temporal Cloud accounts can coexist with users from the same email domain. Each account has its own independent SAML configuration, tied to its unique Account Id. We recommend configuring [SAML](/cloud/manage-access/saml) for each account independently. For the smoother login experience, you can configure SAML for each account separately and use IdP-initiated login: you click the relevant app tile in your identity provider's portal to access the Temporal Cloud account associated with your email address directly. #### Can the same email be used across different Temporal Cloud accounts? No. Each email address can only be associated with a single Temporal Cloud account. If you need access to multiple accounts, you’ll need a separate invite for each one using a different email address. #### Can I use Google or Microsoft SSO after signing up with email and password? If you originally signed up for Temporal Cloud using an email and password, you won’t be able to log in using Google or Microsoft single sign-on. If you prefer SSO, ask your Account Owner to delete your current user and send you a new invitation. During re-invitation, be sure to sign up using your preferred authentication method. Use the [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users) endpoint to invite a user. ``` POST /cloud/users ``` The request body includes a `spec` with the following fields: - `spec.email` — The email address of the user to invite. - `spec.access.account_access.role` — The account-level role to assign. - `spec.access.namespace_accesses` — A map of Namespace names to permissions. Available roles: `ROLE_ADMIN` | `ROLE_DEVELOPER` | `ROLE_READ` | `ROLE_OWNER` | `ROLE_FINANCE_ADMIN`. Available Namespace permissions: `PERMISSION_ADMIN` | `PERMISSION_WRITE` | `PERMISSION_READ`. The new users receive an email with a link to accept the invitation and complete their setup. The new user must use this link to sign up to be added to your account unless the account has a SAML configuration. If your account has a SAML configuration, the new user can sign in using their existing SAML credentials and be included in the account automatically. > **⚠️ Caution:** > > The new user must use the same authentication method they originally signed up with to sign in to Temporal Cloud. If > they used single sign-on (SSO), they must use the same SSO provider to sign in to Temporal Cloud. If they used email and > password authentication, they must use the same email and password to sign in to Temporal Cloud, and cannot use SSO, > even if the underlying email address is the same. > ## What are the account-level roles for users in Temporal Cloud? When an Account Owner or Global Admin invites a user to join an account, they select one of the following roles for that user: - **Global Admin** - Has full administrative permissions across the account, including users and usage - Can create and manage [Namespaces](/namespaces) and [Nexus Endpoints](/nexus/endpoints) - Has Namespace Admin [permissions](#namespace-level-permissions) on all Namespaces in the account. This permission cannot be revoked - **Developer** - Can create Namespaces - Is granted [Namespace Admin](/cloud/manage-access/users#namespace-level-permissions) permission for each Namespace they create. This permission can be revoked - Can create and manage Nexus Endpoints where they are a [Namespace Admin](/cloud/manage-access/users#namespace-level-permissions) on the Endpoint's target Namespace - **Read-Only** - Can read information - Can be granted Namespace [permissions](#namespace-level-permissions), for example to read or write Workflow state in a given Namespace - Can view all Nexus Endpoints in the account, which have separate [runtime access controls](/nexus/security#runtime-access-controls) In addition, there are two roles that the Global Admin cannot assign: - **Account Owner** - Has full administrative permissions across the account, including users, usage and [billing](/cloud/billing-and-usage) - Can create and manage Namespaces and Nexus Endpoints - Has Namespace Admin [permissions](#namespace-level-permissions) on all [Namespaces](/namespaces) in the account. This permission cannot be revoked - **Finance Admin** - Has permissions to view [billing](/cloud/billing-and-usage) information and update payment information - Otherwise, has the same permissions as Account Read-only users - Can be assigned to Service Accounts by a Global Admin, but otherwise can only be assigned by an Account Owner > **📝 Note:** > Default Role > > When the account is created, the initial user who logs in is automatically assigned the Account Owner role. > If your account does not have an Account Owner, please reach out to [Support](https://temporalsupport.zendesk.com/) to assign the appropriate individual to this role. > ## Using the Account Owner role The Account Owner role (that is, users with the Account Owner system role) holds the highest level of access in the system. This role configures account-level parameters and manages Temporal billing and payment information. It allows users to perform all actions within the Temporal Cloud account. > **💡 Tip:** > Best Practices > > Temporal strongly recommends the following precautions when assigning the Account Owner role to users: > > - Assign the role to at least two users in your organization. > Otherwise, limit the number of users with this role. > - Associate a person’s direct email address to the Account Owner, rather than a shared or generic address, so Temporal Support can contact the right person in urgent situations. > > This latter rule is useful for anyone on your team who may need to be contacted urgently, regardless of their Account role. > ## What are the Namespace-level permissions for users in Temporal Cloud? An Account Owner or Global Admin can assign permissions for any [Namespace](/namespaces) in an account. A Developer can assign permissions for a Namespace they create. For a Namespace, a user can have one of the following permissions: - **Namespace Admin:** - Can [manage the Namespace](/cloud/namespaces#manage-namespaces) including identities and permissions - Can create, rename, update, and delete [Workflows](/workflows) within the Namespace - **Write:** - Can create, rename, update, and delete [Workflows](/workflows) within the Namespace - **Read-Only:** - Can only read information from the Namespace ## How to update an account-level role in Temporal Cloud With Global Admin or Account Owner privileges, you can update any user's account-level [role](#account-level-roles) using either the Web UI or the CLI. The Account Owner role can only be granted by existing Account Owners. For security reasons, changes to the Account Owner role must be made through Temporal Support. To change or delete an Account Owner, you must submit a [support ticket](https://temporalsupport.zendesk.com/). ### How to update an account-level role using Web UI 1. In Temporal Web UI, select **Settings** in the left portion of the window. 1. On the **Settings** page, select the user. 1. On the user profile page, select **Edit User**. 1. On the **Edit User** page in **Account Level Role**, select the role. 1. Select **Save**. ### How to update an account-level role using the CLI **Temporal CLI** For details, see the [`temporal cloud user set-account-role`](/cli/command-reference/cloud/user#set-account-role) command. **tcld** For details, see the [tcld user set-account-role](/cloud/tcld/user/#set-account-role) command. ## How to update Namespace-level permissions in Temporal Cloud You can update Namespace-level [permissions](#namespace-level-permissions) by using either the Web UI or the CLI. ### How to use the Web UI to update a user's permissions across multiple Namespaces 1. In Temporal Web UI, select **Namespaces** in the left portion of the window. 1. On the **Namespaces** page, select the Namespace. 1. If necessary, scroll down to the list of permissions 1. On the user profile page in **Namespace permissions**, select the Namespace. 1. On the Namespace page in **Account Level Role**, select the role. 1. Select **Save**. ### How to use the Web UI to update permissions for multiple users within a single Namespace > **📝 Note:** > > A user with the Account Owner or Global Admin account-level [role](#account-level-roles) has Namespace Admin permissions for all Namespaces. > 1. In Temporal Web UI, select **Settings** in the left portion of the window. 1. On the **Settings** page in the **Users** tab, select the user. 1. On the user profile page, select **Edit User**. 1. On the **Edit User** page in **Namespace permissions**, change the permissions for one or more Namespaces. 1. Select **Save**. ### How to use the CLI to update Namespace-level permissions **Temporal CLI** For details, see the [`temporal cloud user set-namespace-permissions`](/cli/command-reference/cloud/user#set-namespace-permissions) command. **tcld** For details, see the [tcld user set-namespace-permissions](/cloud/tcld/user/#set-namespace-permissions) command. ## How to delete a user from your Temporal Cloud account You can delete a user from your Temporal Cloud Account by using either the Web UI or the CLI. > **ℹ️ Info:** > > To delete a user, a user must have the Account Owner or Global Admin account-level [role](#account-level-roles). > ### How to delete a user using Web UI 1. In Temporal Web UI, select **Settings** in the left portion of the window. 1. On the **Settings** page, find the user and, on the right end of the row, select **Delete**. 1. In the **Delete User** dialog, select **Delete**. You can delete a user in two other ways in Web UI: - User profile page: Select the down arrow next to **Edit User** and then select **Delete**. - **Edit User** page: Select **Delete User**. ### How to delete a user using the CLI **Temporal CLI** For details, see the [`temporal cloud user delete`](/cli/command-reference/cloud/user#delete) command. **tcld** For details, see the [tcld user delete](/cloud/tcld/user/#delete) command. ## Account-level roles and Namespace-level permissions Temporal account-level roles and Namespace-level permissions provide access to specific Temporal Workflow and Temporal Cloud operational APIs. The following table provides the API details associated with each account-level role and Namespace-level permission. #### Account-level role details This table provides API-level details for the permissions granted to a user through account-level roles. These permissions are configured per user. | Permission | Read-only | Developer | Finance Admin | Global Admin | Account Owner | | --------------------------------- | --------- | --------- | ------------- | ------------ | ------------- | | CountIdentities | ✅ | ✅ | ✅ | ✅ | ✅ | | CreateAccountAuditLogSink | | | | ✅ | ✅ | | CreateAPIKey | ✅ | ✅ | ✅ | ✅ | ✅ | | CreateNamespace | | ✅ | | ✅ | ✅ | | CreateNexusEndpoint | | ✅ | | ✅ | ✅ | | CreateServiceAccount | | | | ✅ | ✅ | | CreateServiceAccountAPIKey | | | | ✅ | ✅ | | CreateStripeCustomerPortalSession | | | ✅ | | ✅ | | CreateUser | | | | ✅ | ✅ | | DeleteAccountAuditLogSink | | | | ✅ | ✅ | | DeleteAPIKey | ✅ | ✅ | ✅ | ✅ | ✅ | | DeleteNexusEndpoint | | ✅ | | ✅ | ✅ | | DeleteServiceAccount | | | | ✅ | ✅ | | DeleteUser | | | | ✅ | ✅ | | GetAccount | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAccountAuditLogSink | | | | ✅ | ✅ | | GetAccountAuditLogSinks | | | | ✅ | ✅ | | GetAccountFeatureFlags | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAccountLimits | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAccountSettings | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAccountUsage | | | | ✅ | ✅ | | GetAPIKey | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAPIKeys | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAsyncOperation | ✅ | ✅ | ✅ | ✅ | ✅ | | GetAuditLogs | | | | ✅ | ✅ | | GetDecodedCertificate | ✅ | ✅ | ✅ | ✅ | ✅ | | GetIdentities | ✅ | ✅ | ✅ | ✅ | ✅ | | GetIdentity | ✅ | ✅ | ✅ | ✅ | ✅ | | GetNamespaces | ✅ | ✅ | ✅ | ✅ | ✅ | | GetNamespacesUsage | | | | ✅ | ✅ | | GetNexusEndpoint | ✅ | ✅ | ✅ | ✅ | ✅ | | GetNexusEndpoints | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRegion | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRegions | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRequestStatus | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRequestStatuses | | | | ✅ | ✅ | | GetRequestStatusesForNamespace | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRequestStatusesForUser | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRoles | ✅ | ✅ | ✅ | ✅ | ✅ | | GetRolesByPermissions | ✅ | ✅ | ✅ | ✅ | ✅ | | GetServiceAccount | ✅ | ✅ | ✅ | ✅ | ✅ | | GetServiceAccounts | ✅ | ✅ | ✅ | ✅ | ✅ | | GetStripeInvoice | | | ✅ | | ✅ | | GetUser[‡](/cloud/manage-access/permissions-reference#user-authorization-behavior) | ✅ | ✅ | ✅ | ✅ | ✅ | | GetUsers[‡](/cloud/manage-access/permissions-reference#user-authorization-behavior) | ✅ | ✅ | ✅ | ✅ | ✅ | | GetUsersWithAccountRoles | ✅ | ✅ | ✅ | ✅ | ✅ | | InviteUsers | | | | ✅ | ✅ | | ListCreditLedgerEntries | | | ✅ | | ✅ | | ListGrants | | | ✅ | | ✅ | | ListMetronomeInvoices | | | ✅ | | ✅ | | ListMetronomeInvoicesForNamespace | | | ✅ | | ✅ | | ListNamespaces | ✅ | ✅ | ✅ | ✅ | ✅ | | ListPromotionGrantBalances | | | ✅ | | ✅ | | ResendUserInvite | | | | ✅ | ✅ | | SetAccountSettings | | | | ✅ | ✅ | | SyncCurrentUserInvite | ✅ | ✅ | ✅ | ✅ | ✅ | | UpdateAccount | | | | ✅ | ✅ | | UpdateAccountAuditLogSink | | | | ✅ | ✅ | | UpdateAPIKey | ✅ | ✅ | ✅ | ✅ | ✅ | | UpdateNexusEndpoint | | ✅ | | ✅ | ✅ | | UpdateServiceAccount | | | | ✅ | ✅ | | UpdateUser | | | | ✅ | ✅ | | ValidateAccountAuditLogSink | | | | ✅ | ✅ | #### Namespace-level permissions details This table provides API-level details for the permissions granted to a user through Namespace-level permissions. These permissions are configured per Namespace per user. > **📝 Note:** > > Account Owners and Global Admins inherit Namespace Admin permissions on all Namespaces. > | Permission | Read | Write | Namespace Admin | | ---------------------------------- | ---- | ----- | --------------- | | CountWorkflowExecutions | ✅ | ✅ | ✅ | | CreateExportSink | | | ✅ | | CreateSchedule | | ✅ | ✅ | | DeleteExportSink | | | ✅ | | DeleteNamespace | | | ✅ | | DeleteSchedule | | ✅ | ✅ | | DescribeBatchOperation | ✅ | ✅ | ✅ | | DescribeNamespace | ✅ | ✅ | ✅ | | DescribeSchedule | ✅ | ✅ | ✅ | | DescribeTaskQueue | ✅ | ✅ | ✅ | | DescribeWorkflowExecution | ✅ | ✅ | ✅ | | FailoverNamespace | | | ✅ | | GetExportSink | ✅ | ✅ | ✅ | | GetExportSinks | ✅ | ✅ | ✅ | | GetNamespace | ✅ | ✅ | ✅ | | GetNamespaceUsage | ✅ | ✅ | ✅ | | GetReplicationStatus | ✅ | ✅ | ✅ | | GetSearchAttributes | ✅ | ✅ | ✅ | | GetUsersForNamespace | ✅ | ✅ | ✅ | | GetWorkerBuildIdCompatibility | ✅ | ✅ | ✅ | | GetWorkerTaskReachability | ✅ | ✅ | ✅ | | GetWorkflowExecutionHistory | ✅ | ✅ | ✅ | | GetWorkflowExecutionHistoryReverse | ✅ | ✅ | ✅ | | GlobalizeNamespace | | | ✅ | | ListBatchOperations | ✅ | ✅ | ✅ | | ListClosedWorkflowExecutions | ✅ | ✅ | ✅ | | ListExportSinks | ✅ | ✅ | ✅ | | ListFailoverHistoryByNamespace | ✅ | ✅ | ✅ | | ListOpenWorkflowExecutions | ✅ | ✅ | ✅ | | ListReplicaStatus | ✅ | ✅ | ✅ | | ListScheduleMatchingTimes | ✅ | ✅ | ✅ | | ListSchedules | ✅ | ✅ | ✅ | | ListTaskQueuePartitions | ✅ | ✅ | ✅ | | ListWorkflowExecutions | ✅ | ✅ | ✅ | | PatchSchedule | | ✅ | ✅ | | PollActivityTaskQueue | | ✅ | ✅ | | PollWorkflowTaskQueue | | ✅ | ✅ | | QueryWorkflow | ✅ | ✅ | ✅ | | RecordActivityTaskHeartbeat | | ✅ | ✅ | | RecordActivityTaskHeartbeatById | | ✅ | ✅ | | RenameCustomSearchAttribute | | | ✅ | | RequestCancelWorkflowExecution | | ✅ | ✅ | | ResetStickyTaskQueue | | ✅ | ✅ | | ResetWorkflowExecution | | ✅ | ✅ | | RespondActivityTaskCanceled | | ✅ | ✅ | | RespondActivityTaskCanceledById | | ✅ | ✅ | | RespondActivityTaskCompleted | | ✅ | ✅ | | RespondActivityTaskCompletedById | | ✅ | ✅ | | RespondActivityTaskFailed | | ✅ | ✅ | | RespondActivityTaskFailedById | | ✅ | ✅ | | RespondQueryTaskCompleted | | ✅ | ✅ | | RespondWorkflowTaskCompleted | | ✅ | ✅ | | RespondWorkflowTaskFailed | | ✅ | ✅ | | SetUserNamespaceAccess | | | ✅ | | SignalWithStartWorkflowExecution | | ✅ | ✅ | | SignalWorkflowExecution | | ✅ | ✅ | | StartBatchOperation | | ✅ | ✅ | | StartWorkflowExecution | | ✅ | ✅ | | StopBatchOperation | | ✅ | ✅ | | TerminateWorkflowExecution | | ✅ | ✅ | | UpdateExportSink | | | ✅ | | UpdateNamespace | | | ✅ | | UpdateSchedule | | ✅ | ✅ | | UpdateSearchAttributes | | | ✅ | | UpdateUserNamespacePermissions | | | ✅ | | ValidateExportSink | | | ✅ | | ValidateGlobalizeNamespace | | | ✅ | > **📝 Note:** > UpdateNamespace settings > > `UpdateNamespace` requires Namespace Admin permission and covers these settings: > - [Retention period](/temporal-service/temporal-server#retention-period) > - [API key auth](/cloud/api-keys#namespace-authentication) > - [mTLS certificates](/cloud/certificates) > - [Certificate filters](/cloud/certificates#manage-certificate-filters) > - [Codec server](/production-deployment/data-encryption) > - [Connectivity rules](/cloud/connectivity) > - [Custom Search Attributes](/search-attribute#custom-search-attribute) > - [Provisioned capacity (TRUs)](/cloud/capacity-modes#provisioned-capacity) > - [High Availability](/cloud/high-availability) > ## How to troubleshoot account access issues ### Why can't I sign in after my email domain changed? If your organization changed its email domain (for example, from `@oldcompany.com` to `@newcompany.com`), you may be unable to sign in to Temporal Cloud with your existing account. **Why this happens:** When you sign in using "Continue with Google" or "Continue with Microsoft", Temporal Cloud identifies your account by your email address. If your email address changes, Temporal Cloud sees this as a different identity and cannot match it to your existing account. **How to resolve this:** [Create a support ticket](/cloud/support#support-ticket) with the following information: - Your previous email address (the one originally used to access Temporal Cloud) - Your new email address - Your Temporal Cloud Account Id (if known) Temporal Support can update your account to use your new email address. > **💡 Tip:** > Use SAML for enterprise identity management > > If your organization frequently changes email domains or wants centralized control over user authentication, consider using [SAML authentication](/cloud/manage-access/saml). > With SAML, your identity provider (IdP) manages user identities, and email domain changes can be handled within your IdP without affecting Temporal Cloud access. > --- # Set up metrics for Temporal Cloud Source: https://docs.temporal.io/cloud/metrics > Monitor Temporal Cloud workloads with Cloud metrics and SDK metrics. Temporal offers two distinct sources of metrics: [Cloud Metrics](/cloud/metrics/openmetrics/metrics-reference) and [SDK Metrics](/references/sdk-metrics). Each source provides different levels of granularity, filtering options, monitoring-tool integrations, and configuration. - **SDK metrics** monitor individual Workers and your code's behavior from the perspective of your application. - **Cloud metrics** monitor Temporal Cloud's behavior from the perspective of the Temporal Service. When used together, Cloud and SDK metrics measure the health and performance of your full Temporal infrastructure, including the Temporal Cloud Service and user-supplied Temporal Workers. > **💡 Tip:** > New to Cloud metrics? > > Start with the [OpenMetrics Quickstart](/cloud/metrics/openmetrics#quickstart) to create a Service Account, generate an API key, and stream metrics into Datadog, Elastic, Grafana Cloud, New Relic, ClickStack, or self-hosted Prometheus in about 5 minutes. > ## Cloud Metrics Cloud metrics for all Namespaces in your account are available from the [OpenMetrics endpoint](/cloud/metrics/openmetrics), a Prometheus-compatible scrapable endpoint at `metrics.temporal.io`. Use the following rule of thumb when deciding which signal to rely on: | Question | Primary signal | |---|---| | Is Temporal Cloud accepting and serving work normally? | Cloud metrics | | Are Tasks backing up in a Task Queue? | Cloud metrics plus SDK Schedule-To-Start metrics | | Are my Workers saturated, under-provisioned, or misconfigured? | SDK metrics | | Is my application logic, downstream dependency, or Activity behavior unhealthy? | SDK metrics and traces | For a Worker-focused view of how to combine these signals, see [Monitor worker health](/cloud/worker-health). - [OpenMetrics overview](/cloud/metrics/openmetrics) - Getting started and key concepts - [Metrics integrations](/cloud/metrics/openmetrics/metrics-integrations) - Datadog, Grafana Cloud, New Relic, ClickStack, and more - [API reference](/cloud/metrics/openmetrics/api-reference) - Endpoint specification and advanced configuration - [Metrics reference](/cloud/metrics/openmetrics/metrics-reference) - Complete catalog of all `temporal_cloud_v1_*` metrics ## SDK Metrics SDK metrics are emitted by your Workers and Clients. For setup instructions, see [SDK metrics setup](/cloud/metrics/sdk-metrics-setup). ## PromQL Endpoint (Deprecated) > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint and its `temporal_cloud_v0_*` metrics were deprecated on April 2, 2026 and are no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > > Migrate to the [OpenMetrics endpoint](/cloud/metrics/openmetrics). > See the [migration guide](/cloud/metrics/openmetrics/migration-guide) for a complete v0-to-v1 metric mapping. > The legacy PromQL endpoint uses mTLS certificate authentication and exposes `temporal_cloud_v0_*` metrics via a Prometheus query API. - [PromQL endpoint](/cloud/metrics/promql) - [PromQL metrics reference](/cloud/metrics/reference) - [PromQL setup with Grafana](/cloud/metrics/prometheus-grafana) --- # General observability setup with metrics Source: https://docs.temporal.io/cloud/metrics/general-setup > Learn how to configure a metrics endpoint in Temporal Cloud using the UI or tcld CLI, assign certificates, and integrate with observability tools like Grafana. > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint and its `temporal_cloud_v0_*` metrics were deprecated on April 2, 2026 and are no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > > New users should set up the [OpenMetrics endpoint](/cloud/metrics/openmetrics) instead. > Existing users should follow the [migration guide](/cloud/metrics/openmetrics/migration-guide) to transition to the OpenMetrics endpoint. > You will learn how to do the following: - [Configure an endpoint using the UI](#configure-via-ui) - [Configure an endpoint using tcld](#configure-via-cli-tcld) - [Query for metrics with a PromQL endpoint](#query-promql) ## Configure using the UI **How to configure a metrics endpoint using Temporal Cloud UI** > **📝 Note:** > > To view and manage third-party integration settings, your user account must have the Account Owner or Global Admin [role](/cloud/manage-access/roles-and-permissions#account-level-roles). > To assign a certificate and generate your metrics endpoint, follow these steps: 1. Log in to Temporal Cloud UI with an Account Owner or Global Admin [role](/cloud/manage-access/roles-and-permissions#account-level-roles). 2. Go to **Settings** and select **Observability**. 4. Add your root CA certificate (.pem) and save it. Note that if an observability endpoint is already set up, you can append your root CA certificate here to use the generated observability endpoint in your observability tool. 5. To test your endpoint, run the following command on your host: ``` curl -v --cert --key "/api/v1/query?query=temporal_cloud_v0_state_transition_count" ``` If you have Workflows running on a Namespace in your Temporal Cloud instance, you should see some data as a result of running this command. After the page refreshes, the new metrics endpoint appears below **Endpoint**, in the form `https://.tmprl.cloud/prometheus`. Use the endpoint to configure your observability tool. For example, if you use Grafana, see [Grafana data source configuration](/cloud/metrics/prometheus-grafana#grafana-data-source-configuration). You can also query via the [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) at URLs like: ``` https://.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count ``` For example: ``` $ curl --cert client.pem --key client-key.pem "https://.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count" | jq . { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "temporal_cloud_v0_state_transition_count", "__rollup__": "true", "operation": "WorkflowContext", "temporal_account": "your-account", "temporal_namespace": "your-namespace.your-account-is", "temporal_service_type": "history" }, "value": [ 1672347471.2, "0" ] }, ... } ``` ## Configure endpoint using tcld **How to configure a metrics endpoint using the tcld CLI.** To add a certificate to a metrics endpoint, use [`tcld account metrics accepted-client-ca add`](/cloud/tcld/account#add). To enable a metrics endpoint, use [`tcld account metrics enable`](/cloud/tcld/account#enable). To disable a metrics endpoint, use [`tcld account metrics disable`](/cloud/tcld/account#disable). For more information, see [tcld account metrics command](/cloud/tcld/account#metrics). ## Query for metrics with a PromQL endpoint Temporal Cloud emits metrics in a Prometheus-supported format. Prometheus is an open-source toolkit for alerting and monitoring. The Temporal Service exposes Cloud metrics with a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/). Temporal Cloud metrics provide a compatible data source for visualizing, monitoring, and observability platforms. You can use functions like [rate](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) or [increase](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase) to calculate the rate of increase for a Temporal Cloud metric: ``` rate(temporal_cloud_v0_frontend_service_request_count[$__rate_interval]) ``` Or you might use Prometheus to calculate average latencies or histogram quartiles: ``` # Average latency rate(temporal_cloud_v0_service_latency_sum[$__rate_interval]) / rate(temporal_cloud_v0_service_latency_count[$__rate_interval]) # Approximate 99th percentile latency broken down by operation histogram_quantile(0.99, sum(rate(temporal_cloud_v0_service_latency_bucket[$__rate_interval])) by (le, operation)) ``` Metrics are scraped every 30 seconds and exposed to the metrics endpoint with a 1-minute lag.\ The endpoint returns data with a 15-second resolution, which results in displaying the same value twice. Set up Grafana with Temporal Cloud observability to view metrics by creating or getting your Prometheus endpoint for Temporal Cloud metrics and enabling SDK metrics. **Related:** - [How to set up Grafana with Temporal Cloud observability](/cloud/metrics/prometheus-grafana) - [How to monitor Worker Health with Temporal Cloud Metrics](/cloud/worker-health) - [How to monitor Service Health with Temporal Cloud Metrics](/cloud/service-health) --- # Set up Cloud metrics with OpenMetrics Source: https://docs.temporal.io/cloud/metrics/openmetrics > Export metrics from Temporal Cloud using the OpenMetrics standard and third party integrations. > **💡 Tip:** > PRICING > > Future pricing may apply to high-volume usage that exceeds standard [limits](/cloud/metrics/openmetrics/api-reference#api-limits). > Temporal Cloud's [OpenMetrics](https://openmetrics.io/) endpoint provides operational metrics for your Temporal Cloud workloads in industry-standard Prometheus format, enabling comprehensive monitoring across Namespaces, Workflows, and Task Queues with your existing observability stack. ## Quickstart Stream metrics from Temporal Cloud into your observability tool in about 5 minutes. **Prerequisites** - An **Account Owner** or **Global Admin** role on the Temporal Cloud account. The Metrics Read-Only role is an account-level role and can only be granted by these roles. A Namespace Admin cannot complete these steps. - An account in the observability tool you want to use, such as Datadog, Grafana Cloud, New Relic, ClickStack, or self-hosted Prometheus. **Steps** 1. **Create a Service Account with the Metrics Read-Only role.** In the Temporal Cloud UI, go to **Settings → Service Accounts → Create Service Account** and assign the **Metrics Read-Only** account-level role. 2. **Generate an API key for the Service Account.** Open the Service Account and create an API key. Copy the key and store it somewhere secure. It is shown only once. 3. **Verify the endpoint is reachable.** ```shell curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics ``` You should see OpenMetrics-formatted output beginning with `# TYPE temporal_cloud_v1_...`. > **📝 Note:** > `metrics.temporal.io` is for scrapers, not browsers > > The endpoint requires an `Authorization: Bearer ` header on every request. There is no browser UI. Opening `https://metrics.temporal.io` or `https://metrics.temporal.io/v1/metrics` directly in a browser returns `Jwt is missing`. Configure the endpoint inside your observability tool instead. > 4. **Configure your observability tool.** Paste the API key into the integration for your tool of choice. See [Metrics integrations](/cloud/metrics/openmetrics/metrics-integrations) for tool-specific setup: - [Datadog](/cloud/metrics/openmetrics/metrics-integrations#datadog) - [Elastic](/cloud/metrics/openmetrics/metrics-integrations#elastic) - [Grafana Cloud](/cloud/metrics/openmetrics/metrics-integrations#grafana-cloud) - [New Relic](/cloud/metrics/openmetrics/metrics-integrations#new-relic) - [ClickStack](/cloud/metrics/openmetrics/metrics-integrations#clickstack) - [SigNoz](/cloud/metrics/openmetrics/metrics-integrations#signoz) - [Self-hosted Prometheus or OpenTelemetry Collector](/cloud/metrics/openmetrics/metrics-integrations#prometheus-grafana) Metrics begin populating in your tool within a few minutes. ## Quick Links * [Integrations](/cloud/metrics/openmetrics/metrics-integrations) - Get started exporting metrics with common integrations * [API Documentation](/cloud/metrics/openmetrics/api-reference) - Endpoint specification and advanced configuration * [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference) - Complete catalog of all metrics with descriptions and labels * [Migration Guide](/cloud/metrics/openmetrics/migration-guide) - Transition from the deprecated Prometheus query endpoint to OpenMetrics ## Overview Temporal Cloud OpenMetrics exposes 50+ metrics covering workflow lifecycles, task queue operations, service performance, and system limits. All metrics are aggregated over one-minute windows and available for scraping within three minutes. Each scrape returns only the most recently completed one-minute window—configure your monitoring system to retain what it scrapes. * [Set up authentication and scraping](/cloud/metrics/openmetrics/api-reference#authentication) with the API documentation. * Browse the [complete metrics catalog](/cloud/metrics/openmetrics/metrics-reference) for descriptions and labels. * Teams using the query endpoint should review the [migration guide](/cloud/metrics/openmetrics/migration-guide). ## API key authentication Create a Service Account with the "Metrics Read-Only" role and generate an API key. See the [Quickstart](#quickstart) above for step-by-step instructions. API keys work with standard HTTPS, with no certificate rotation or distribution required. ## Global endpoint This is a single endpoint at `metrics.temporal.io` which serves all metrics across your entire account with API key authentication and standard HTTPS. ## Namespace and metric filtering You can use query parameters to enable selective scraping to manage data volume and costs, which support wildcards for flexible namespace selection and specific metric filtering. ## Dashboard templates Production-ready [Grafana dashboards](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) provide immediate visibility with pre-built queries and visualizations. --- # OpenMetrics API reference Source: https://docs.temporal.io/cloud/metrics/openmetrics/api-reference > Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. The Temporal Cloud OpenMetrics API provides actionable operational metrics about your Temporal Cloud deployment. This is a scrapable HTTP API that returns metrics in OpenMetrics format, suitable for ingestion by Prometheus-compatible monitoring systems. ## Available metrics reference Metrics descriptions are also available programmatically via the `/v1/descriptors` endpoint. You can see the Metrics Reference for a list of available metrics. ## Authentication Temporal uses API keys for integrating with the OpenMetrics endpoint. Applications must be authorized and authenticated before they can access metrics from Temporal Cloud. An API key is owned by a Service Account and inherits the permissions granted to the owner. ### Creating API keys API keys can be created using the [Temporal Cloud UI](https://cloud.temporal.io): 1. Navigate to Settings → Service Accounts 2. Create a service account with **"Metrics Read-Only"** Account Level Role 3. Generate an API key within the service account > **ℹ️ Info:** > > See the [docs](/cloud/api-keys#serviceaccount-api-keys) for more details on generating API keys. > ### Using API keys All API requests must be made over HTTPS. Calls made over plain HTTP will fail. API requests without authentication will also fail. ```shell curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics ``` ## Object model The object model for the Metrics API follows the [OpenMetrics](https://openmetrics.io/) standard. ### Metrics A metric is a numeric attribute measured at a specific point in time, labeled with contextual metadata gathered at the point of instrumentation. ### Metric types All Temporal Cloud metrics are exposed as *gauges* in OpenMetrics format, but represent different measurement types: * **Rate metrics**: Pre-computed per-second rates with delta temporality (for example, `temporal_cloud_v1_workflow_success_count` \- workflows completed per second) * **Value metrics**: Current or instantaneous values (for example, `temporal_cloud_v1_approximate_backlog_count` \- current number of tasks in queue) The list of metrics and their labels are available via the [List Descriptors](/cloud/metrics/openmetrics/api-reference#list-metric-descriptors) endpoint or in the [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference). ### Labels A label is a key-value attribute associated with a metric data point. Labels can be used to filter or aggregate metrics. Temporal SDKs and the Temporal Service call this same concept a [tag](/glossary#tag); it's called a label once scraped in OpenMetrics format. Common labels include: * `temporal_namespace`: The Temporal namespace * `temporal_account`: The Temporal account * `region`: The cloud region where the metric originated * `temporal_workflow_type`: The workflow type (where applicable) * `temporal_task_queue`: The task queue name (where applicable) Each metric has its own set of applicable labels. See the Metrics Reference for complete details. ### Metric family A [Metric Family](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#metricfamily) may have zero or more metrics. The set of metrics returned will vary based on actual system activity. Metrics only appear in a Metric Family if they were reported during the aggregation window. ## Client considerations ### Rate limiting To protect the stability of the API and keep it available to all users, Temporal employs multiple safeguards. When a rate limit is breached, an HTTP `429 Too Many Requests` error is returned with the following headers: | Header | Description | | ----- | ----- | | `Retry-After` | The time in seconds until the rate limit window resets | #### Rate limit scopes > **📝 Note:** > Rate limit scopes are subject to change. > | Scope | Limit | | ----- | ----- | | Account | 180 requests per hour | > **⚠️ Caution:** > Practical enforcement of this limit may allow bursts that exceed the limit at various times. > Do not rely on this headroom: design production clients against the documented limit. > ### Response completeness The `X-Completeness` header indicates whether the response contains all available data: * `complete`: The response contains all metrics requested * `limited`: Response truncated due to size limits (50k metric data points max). Use namespace or metric filtering to reduce the response size. * `unknown`: Completeness cannot be determined (possibly due to regional issues or timeouts). Clients are encouraged to retry. ### Retry logic Implement retry logic in your client to gracefully handle transient API failures. Use exponential backoff with jitter to avoid retry storms with reasonable retry intervals to avoid reaching rate limits. ### Data latency Metric data points are available for query within 3 minutes of their origination. This is in line with the freshest metrics [available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/). This latency should be accounted for when setting up monitoring alerts. ### Scrape window The endpoint exposes only the most recently completed one-minute aggregation window. Each scrape returns a snapshot of that window—there is no query interface for historical data. To retain historical metrics, configure your monitoring system to store what it scrapes. ## Endpoints > **ℹ️ Info:** > > All endpoints are served from: `metrics.temporal.io` > ### Get metrics `GET /v1/metrics` Returns metrics in OpenMetrics format suitable for scraping by Prometheus-compatible systems. #### Timestamp offset To account for metric data latency, this endpoint returns metrics from the current timestamp minus a fixed offset. The current offset is 3 minutes rounded down to the start of the minute. To accommodate this offset, the timestamps in the response should be honored when importing the metrics. For example, in Prometheus this can be controlled using the `honor\_timestamps` flag. #### Query parameters | Parameter | Type | Description | | ----- | ----- | ----- | | `namespaces` | string array | Filter to specific Namespaces. Supports wildcards (for example, `production-*`) | | `metrics` | string array | Filter to specific metrics | Array parameters use repeated keys. To pass multiple values, repeat the parameter name once per value: ```shell /v1/metrics?namespaces=prod-payments&namespaces=prod-orders ``` The `namespaces` and `metrics` parameters can be combined, and each may be repeated independently: ```shell /v1/metrics?namespaces=prod-payments&namespaces=prod-orders&metrics=temporal_cloud_v1_workflow_success_count&metrics=temporal_cloud_v1_approximate_backlog_count ``` #### Response headers | Header | Description | | ----- | ----- | | `X-Completeness` | Indicates the response status: `complete`, `limited`, or `unknown` | | `Content-Type` | `application/openmetrics-text` | > **ℹ️ Info:** > Example > > Request: > > ```shell > curl -H "Authorization: Bearer " \ > "https://metrics.temporal.io/v1/metrics?namespaces=production-*" > ``` > > Response: > ``` > # TYPE temporal_cloud_v1_workflow_success_count gauge > # HELP temporal_cloud_v1_workflow_success_count The number of successful workflows per second > temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="payment-processing",region="aws-us-west-2"} 42.0 1609459200000 > temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="order-fulfillment",region="aws-us-west-2"} 128.0 1609459200000 > > # TYPE temporal_cloud_v1_approximate_backlog_count gauge > # HELP temporal_cloud_v1_approximate_backlog_count Approximate number of tasks in a task queue > temporal_cloud_v1_approximate_backlog_count{temporal_namespace="production",temporal_task_queue="critical-queue",task_type="workflow", region="aws-us-west-2"} 15.0 1609459200000 > ``` > #### Summary of best practices * *Honor timestamps*: Set `honor_timestamps: true` in Prometheus * *Scrape interval*: Use 30s. Intervals longer than 60s may skip datapoints because metrics update once per minute. * *Timeout*: Set scrape timeout to 10 seconds for large responses * *Filtering*: Use query parameters to reduce response size ### List metric descriptors `GET /v1/descriptors` Lists all metric descriptors, including help text, available dimensions (labels), and release stages. Each descriptor includes a `release_stage` field that reports the metric's lifecycle status. The supported values are: | Value | Description | | ----- | ----- | | `public-preview` | The metric is in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) | | `general-availability` | The metric is in [General Availability](/evaluate/development-production-features/release-stages#general-availability) | | `deprecated` | The metric is deprecated | #### Query parameters | Parameter | Type | Description | | ----- | ----- | ----- | | `limit` | integer | Page size (1-100, default: 100\) | | `offset` | integer | Page offset | > **ℹ️ Info:** > Example > > Request: > > ```shell > curl -H "Authorization: Bearer " \ > "https://metrics.temporal.io/v1/descriptors" > ``` > > Response: > > ```json > { > "meta": { > "pagination": { > "total": 35, > "limit": 100, > "offset": 0 > } > }, > "descriptors": [ > { > "name": "temporal_cloud_v1_workflow_success_count", > "help": "The number of successful workflows per second", > "dimensions": [ > "temporal_namespace", > "temporal_workflow_type", > "temporal_task_queue", > "region" > ], > "release_stage": "general-availability" > } > ] > } > ``` > ## Managing high cardinality > **⚠️ Caution:** > > High-cardinality labels like `temporal_task_queue` and `temporal_workflow_type` can significantly increase metric volume and impact performance of your monitoring system. > ### Cardinality estimation To estimate your metric cardinality and see if this is an issue: ``` Total series = Base metrics × Namespaces × Task queues × Workflow types ``` Example: * 6 workflow metrics with both labels * 10 namespaces * 50 task queues * 20 workflow types * \= 6 × 10 × 50 × 20 \= 60,000 time series > **📝 Note:** > > 60,000 time series in the above example results in exceeding the 30,000 data points per scrape limit. > If the cardinality is too high or you are hitting API limits, consider the following strategies. ### Filtering at scrape time You can isolate only the metrics/namespaces you need. For example, the following shows examples of filtering by modifying the `metrics_path.` ```shell # Only specific namespaces matching the wildcard pattern /v1/metrics?namespaces=production-* # Multiple namespaces /v1/metrics?namespaces=prod-payments&namespaces=prod-orders # Only specific metrics /v1/metrics?metrics=temporal_cloud_v1_workflow_success_count # Combined filtering /v1/metrics?namespaces=prod-*&metrics=temporal_cloud_v1_approximate_backlog_count ``` > **ℹ️ Info:** > > In Prometheus, the `params` config can be set to match the same behavior as above. > > ```yaml > scrape_configs: > - job_name: 'temporal-cloud' > ... > static_configs: > - targets: ['metrics.temporal.io'] > metrics_path: '/v1/metrics' > params: > namespaces: ['prod-*'] > metrics: ['temporal_cloud_v1_approximate_backlog_count'] > > ``` > ### Label management #### Prometheus If using Prometheus, you can configure it to drop metrics with a specific label or even rename specific label values to reduce the cardinality. ```yaml metric_relabel_configs: # Consolidate non-critical task queues - source_labels: [temporal_task_queue] regex: '(critical-queue|payment-queue)' target_label: __tmp_keep_original replacement: 'true' - source_labels: [__tmp_keep_original] regex: '' target_label: temporal_task_queue replacement: 'other' - regex: '__tmp_keep_original' action: labeldrop ``` #### OpenTelemetry collector To accomplish the same as Prometheus, a filter can be used in the collector along with any other processors. ``` processors: filter: metrics: include: match_type: regexp expressions: # Only keep metrics with critical-queue or payment-queue - Label("temporal_task_queue") == nil or IsMatch(Label("temporal_task_queue"), "^(critical-queue|payment-queue)$") ``` ### Monitoring cardinality Cardinality can be monitored using this PromQL query. ```shell # Count the total number of series count({__name__=~"temporal_cloud_v1_.*"}) # Count the total number of series by metric count({__name__=~"temporal_cloud_v1_.*"}) by (__name__) ``` ## API limits | Limit | Impact | Mitigation | | ----- | ----- | ----- | | 50k total datapoints per scrape | Response may be truncated | Use namespace/metric filtering | | 180 requests per account per hour (~3 requests per minute) | HTTP 429 returned | Set scrape interval to 30s | > **⚠️ Caution:** > Practical enforcement of this limit may allow bursts that exceed the limit at various times. > Do not rely on this headroom: design production clients against the documented limit. > --- # Metrics integrations Source: https://docs.temporal.io/cloud/metrics/openmetrics/metrics-integrations > Integrating with the Temporal Cloud OpenMetrics endpoint. Metrics can be exported from Temporal Cloud using the OpenMetrics endpoint. This document describes configuring integrations that have third party support or are based on open standards. This document is for basic configuration only. For advanced concepts such as label management and high cardinality scenarios see the [general API reference](/cloud/metrics/openmetrics/api-reference). ## Integrations Before configuring any integration, complete the [Quickstart](/cloud/metrics/openmetrics#quickstart) to create a Service Account with the **Metrics Read-Only** role and generate an API key. This requires the **Account Owner** or **Global Admin** role - a Namespace Admin cannot grant the Metrics Read-Only role. ### Datadog Datadog provides a serverless integration with the OpenMetrics endpoint. It scrapes metrics, stores them in Datadog, and ships a default dashboard with built-in monitors. 1. In Datadog, open the [Integrations catalog](https://app.datadoghq.com/integrations) and search for **Temporal Cloud OpenMetrics**. Install the integration. 2. Click **Add Account** in the integration tile and paste your Temporal Cloud API key into the **API Key** field. 3. Save the configuration. The default Temporal Cloud dashboard appears in **Dashboards → Dashboards List** once data starts flowing (typically within a few minutes). For Datadog-side details, see the [Datadog integration page](https://docs.datadoghq.com/integrations/temporal-cloud-openmetrics/). Datadog also offers a separate [Cost Management integration](https://docs.datadoghq.com/integrations/temporal-cloud-costs/) that sends Temporal Cloud billing data to Datadog Cloud Cost Management for spend attribution by namespace and service type. For Datadog users, treat the OpenMetrics integration as the Cloud-side half of your observability setup: - Use OpenMetrics in Datadog to monitor Temporal Cloud behavior such as Task Queue backlog, poll success, and rate limiting. - Collect [SDK metrics](/cloud/metrics/sdk-metrics-setup) from your Workers separately to monitor saturation, Schedule-To-Start latency, slot availability, and sticky cache behavior. If you only ingest Cloud metrics, you will miss many worker-side bottlenecks. For recommended Worker monitors, see [Monitor worker health](/cloud/worker-health). ### Elastic The Elastic [Temporal Integration](https://www.elastic.co/docs/reference/integrations/temporal) scrapes the OpenMetrics endpoint, stores metrics in Elasticsearch, and automatically installs dashboards, alert templates, and SLO templates in Kibana. Setup 1. In Kibana, go to Management → Integrations and search for Temporal (OpenTelemetry). 2. Add the integration. The default Cloud endpoint (metrics.temporal.io:443) and path (/v1/metrics) are pre-configured. 3. Enter your Temporal Cloud API key in the Temporal Cloud API Key field. 4. Verify metrics are flowing in Discover using the filter data_stream.dataset: "temporal.cloud_metrics.otel". Dashboards, alert rules, and SLO templates are installed automatically. ### Grafana Cloud Grafana Cloud provides a serverless integration that scrapes the OpenMetrics endpoint, stores metrics in Grafana Cloud, and ships a prebuilt **Temporal overview** dashboard. 1. In Grafana Cloud, go to **Connections** and select the **Temporal Cloud** tile. 2. On the **Configuration** page, add a scrape job: give it a name, paste your Temporal Cloud API key, and set the scrape interval (default 1 minute - lower intervals increase data points per minute and cost). 3. Click **Test Connection** to verify authentication, then **Save** to start collecting metrics. 4. Click **Install** to deploy the prebuilt dashboards. Add a separate scrape job per account to monitor multiple accounts from one Grafana Cloud instance. If the dashboard shows no data after a few minutes, confirm the API key's Service Account has the **Metrics Read-Only** role and that the endpoint is reachable using the `curl` check from the [Quickstart](/cloud/metrics/openmetrics#quickstart). For full configuration, dashboards, and changelog, see the [Grafana Cloud integration page](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-temporal/). ### ClickStack ClickHouse provides an integration with the OpenMetrics endpoint for ClickStack. This integration uses an OpenTelemetry collector to read from the OpenMetrics endpoint, ingest data into ClickHouse, and includes a default dashboard to visualize the data with HyperDX. See the [integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics) for more details. 1. Save your Temporal Cloud API key to a local file named `temporal.key` (no trailing newline or spaces). 2. Create an OpenTelemetry collector config named `temporal-metrics.yaml` that uses a Prometheus receiver against `metrics.temporal.io` with Bearer token auth, a 60-second scrape interval, the `service.name: "temporal"` resource attribute, and the ClickHouse exporter. Copy the full template from the [ClickStack integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics). 3. Mount both files into your ClickStack collector and set the custom config env var. With Docker Compose: ```yaml volumes: - ./temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml - ./temporal.key:/etc/otelcol-contrib/temporal.key environment: CUSTOM_OTELCOL_CONFIG_FILE: /etc/otelcol-contrib/custom.config.yaml ``` 4. In HyperDX, open the **Metrics explorer** and confirm metrics with the `temporal` prefix are arriving. 5. Import the pre-built dashboard: in HyperDX click **Import Dashboard**, upload `temporal-metrics-dashboard.json` from the ClickStack integration page, then click **Finish Import**. ### New Relic The New Relic integration pulls metrics from the OpenMetrics endpoint via the `nri-flex` integration that runs alongside the New Relic infrastructure agent. > **📝 Note:** > Requires a host > > The integration runs on a host (Linux, Windows, or Kubernetes) with the New Relic infrastructure agent installed. The agent scrapes the endpoint and forwards metrics to New Relic. > 1. Install the **New Relic infrastructure agent** on a host. See the [agent install docs](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent/get-started/install-infrastructure-agent/) for platform-specific instructions. 2. Create `/etc/newrelic-infra/integrations.d/nri-flex-temporal-cloud-config.yml` using the template from the [New Relic integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/), and replace the `${TEMPORAL_API_KEY}` placeholder with your Temporal Cloud API key. 3. Restart the agent so the new config is picked up: ```shell sudo systemctl restart newrelic-infra.service ``` 4. In **one.newrelic.com**, go to **Integrations & Agents → Dashboards**, search for **Temporal Cloud**, and install the pre-built dashboard. Data appears within a few minutes. For New Relic-side details, see the [New Relic integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/). ### SigNoz SigNoz is an OpenTelemetry-native observability platform, available as a cloud service or self-hosted. The integration runs an OpenTelemetry Collector that scrapes the OpenMetrics endpoint and forwards metrics to SigNoz over OTLP. 1. Configure an OpenTelemetry Collector with a Prometheus receiver against `metrics.temporal.io` using Bearer token auth with your Temporal Cloud API key, and an OTLP exporter that sends to your SigNoz ingestion endpoint (using your SigNoz ingestion key and region). Copy the full template - including the Kubernetes Secret-based variant - from the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/). 2. Deploy the collector (VM/binary or Kubernetes) and confirm `temporal_cloud_v1_*` metrics arrive in the SigNoz **Metrics Explorer**, typically within a few minutes. 3. Import the pre-built dashboard JSON from the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/) to visualize Temporal Cloud metrics. For SigNoz-side details, see the [SigNoz integration page](https://signoz.io/docs/integrations/temporal-cloud-metrics/). ### Prometheus \+ Grafana Self hosted Prometheus can be used to scrape the OpenMetrics endpoint. 1. Add a new scrape job for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys). ```yaml scrape_configs: - job_name: 'temporal-cloud' scrape_interval: 30s scrape_timeout: 30s honor_timestamps: true scheme: https authorization: type: Bearer credentials: '' static_configs: - targets: ['metrics.temporal.io'] metrics_path: '/v1/metrics' ``` 2. Import the [Grafana dashboard](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) and configure your Prometheus datasource. ### OpenTelemetry Collector Configuration Collect metrics with a self-hosted OpenTelemetry Collector to ingest into the system of your choosing. 1. Add a new prometheus receiver for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys). ```yaml receivers: prometheus: config: scrape_configs: - job_name: 'temporal-cloud' scrape_interval: 30s scrape_timeout: 30s honor_timestamps: true scheme: https authorization: type: Bearer credentials_file: static_configs: - targets: ['metrics.temporal.io'] metrics_path: '/v1/metrics' processors: batch: exporters: otlphttp: endpoint: service: pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [otlphttp] ``` > **ℹ️ Info:** > > Examples for these integrations and more are [here](https://github.com/temporal-community/cloud-metrics-scrape-examples). > --- # OpenMetrics metrics reference Source: https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference > Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. This document describes all metrics available from the Temporal Cloud OpenMetrics endpoint. ## Metric Conventions ### Metric Types All metrics are exposed as OpenMetrics gauges, but represent different measurement types: * *Rate Metrics*: per-second rate of the aggregated values * *Value Metrics*: The most recent aggregate value within a look-back window (for example, backlogs, limits) * *Percentile Metrics*: Pre-calculated aggregated latency percentiles in seconds > **📝 Note:** > > All metrics are stored as 1 minute aggregates. Rate metrics are therefore per-second rates averaged over each minute, which smooths sub-minute bursts. A short spike can read below your limit even when it triggered throttling. See [Why does throttling occur when count metrics stay below the limit?](/cloud/service-health#why-does-throttling-occur-when-count-metrics-stay-below-the-limit) for a worked example. > > **📝 Note:** > Percentile metrics on low-traffic namespaces > > Percentile metrics (`*_p50` / `_p95` / `_p99`) are calculated from the requests observed in each 1-minute aggregation window. On a namespace with few requests per minute, that sample is small, so a single slow request dominates every percentile and p50, p95, and p99 converge toward the slowest observed request. Tail percentiles generally need roughly 20 or more samples per window before they are statistically meaningful; below that, values vary widely. > > For example, a low-volume namespace that starts one Workflow every few minutes can report a several-hundred-millisecond `StartWorkflowExecution` `temporal_cloud_v1_service_latency_p95` that reflects a single request, not systemic latency. When alerting on percentile latency for low-traffic namespaces, gate the alert on a minimum request count (for example, [`temporal_cloud_v1_service_request_count`](#temporal_cloud_v1_service_request_count)) so that windows with too few samples don't trigger it. These percentiles are pre-calculated per 1-minute window and cannot be re-aggregated into an accurate longer-window percentile, so widening your evaluation window does not by itself make a sparse sample meaningful. > ### Common Labels All metrics include these base labels: | Label | Description | | ----- | ----- | | `temporal_namespace` | The Temporal namespace | | `temporal_account` | The Temporal account identifier | | `region` | Cloud region where the metric originated | ### Opt-in Labels Some labels are **opt-in** due to their high cardinality. These labels are not included by default when you scrape the OpenMetrics endpoint. To enable an opt-in label, add it to the `labels` query parameter on your scrape URL. When an opt-in label is enabled, it is populated on **all metrics** that support it. | Label | Available on | Description | | ----- | ----- | ----- | | `temporal_activity_type` | Activity metrics | The activity type name | | `temporal_worker_deployment_name` | `temporal_cloud_v1_approximate_backlog_count` | The Worker Deployment name | | `temporal_worker_build_id` | `temporal_cloud_v1_approximate_backlog_count` | The Worker Deployment Version Build ID | For example, to include `temporal_activity_type` in your scrape results: ``` /v1/metrics?labels=temporal_activity_type ``` Enable multiple labels at the same time by concatenating multiple `labels` query parameters: ``` /v1/metrics?labels=temporal_worker_build_id&labels=temporal_worker_deployment_name ``` ## Metrics Catalog ### Frontend Service Metrics #### temporal\_cloud\_v1\_service\_request\_count gRPC requests received per second. | Label | Description | | ----- | ----- | | `operation` | The name of the RPC operation | **Type**: Rate #### temporal\_cloud\_v1\_service\_request\_throttled\_count gRPC requests throttled per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric. | Label | Description | | ----- | ----- | | `operation` | The name of the RPC operation | **Type**: Rate #### temporal\_cloud\_v1\_service\_error\_count gRPC errors per second. | Label | Description | | ----- | ----- | | `operation` | The name of the RPC operation | **Type**: Rate #### temporal\_cloud\_v1\_service\_pending\_requests The number of pollers that are actively long polling for a task. Use this to track against ``temporal_cloud_v1_poller_limit`` | Label | Description | | ----- | ----- | | `operation` | The name of the operation | **Type**: Value #### temporal\_cloud\_v1\_resource\_exhausted\_error\_count Resource exhaustion errors per second, incremented when a single resource receives a burst larger than it can absorb. SDKs retry these errors gracefully. This metric does not include throttling due to Namespace limits - see [`temporal_cloud_v1_total_action_throttled_count`](#temporal_cloud_v1_total_action_throttled_count) and related throttle metrics for rate limiting against account limits. See [Detecting Resource Exhaustion](/cloud/service-health#detecting-resource-exhaustion) for guidance on investigating non-zero values. | Label | Description | | ----- | ----- | | `operation` | The name of the operation | **Type**: Rate #### temporal\_cloud\_v1\_service\_latency\_p50 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 50th percentile latency of service requests in seconds | Label | Description | | ----- | ----- | | `operation` | The name of the operation | **Type**: Latency #### temporal\_cloud\_v1\_service\_latency\_p95 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 95th percentile latency of service requests in seconds | Label | Description | | ----- | ----- | | `operation` | The name of the operation | **Type**: Latency #### temporal\_cloud\_v1\_service\_latency\_p99 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions as the percentile won't be accurate. > The 99th percentile latency of service requests in seconds | Label | Description | | ----- | ----- | | `operation` | The name of the operation | **Type**: Latency ### Workflow Completion Metrics > **⚠️ Caution:** > High Cardinality > > These metrics could have high cardinality depending on number of workflow types and task queues. > #### temporal\_cloud\_v1\_workflow\_success\_count Successful workflow completions per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_failed\_count Workflow failures per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_timeout\_count Workflow timeouts per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_cancel\_count Workflow cancellations per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_terminate\_count Workflow terminations per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_continued\_as\_new\_count Workflows continued as new per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | **Type**: Rate #### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p50 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 50th percentile workflow schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_workflow_type` | The workflow type | **Type**: Latency #### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p95 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 95th percentile workflow schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_workflow_type` | The workflow type | **Type**: Latency #### temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p99 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 99th percentile workflow schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_workflow_type` | The workflow type | **Type**: Latency ### Activity Metrics > **⚠️ Caution:** > High Cardinality > > These metrics could have high cardinality depending on number of activity types, workflow types, and task queues. The `temporal_activity_type` label is [opt-in](#opt-in-labels) to help manage cardinality. > > **📝 Note:** > Standalone Activities > > Standalone Activities are Activity Executions that are started independently, without an associated Workflow. For Activity metrics that include the `temporal_workflow_type` label, Standalone Activities use the placeholder value `"__standalone_activity"`. > #### temporal\_cloud\_v1\_activity\_success\_count Successful activity completions per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Rate #### temporal\_cloud\_v1\_activity\_fail\_count Activity failures per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Rate #### temporal\_cloud\_v1\_activity\_timeout\_count Activity timeouts per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | | `timeout_type` | The timeout type | **Type**: Rate #### temporal\_cloud\_v1\_activity\_task\_fail\_count Activity task failures per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Rate #### temporal\_cloud\_v1\_activity\_task\_timeout\_count Activity task timeouts per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | | `timeout_type` | The timeout type | **Type**: Rate #### temporal\_cloud\_v1\_activity\_cancel\_count Activity cancellations per second. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Rate #### temporal\_cloud\_v1\_activity\_terminate\_count Activity terminations per second. This metric only applies to Standalone Activities. Regular Activities that run within a Workflow cannot be terminated independently. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `temporal_workflow_type` | The workflow type | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Rate > **ℹ️ Info:** > Activity latency labels > > Activity latency metrics include only the `temporal_activity_type` label. > Labels such as `temporal_task_queue` and `temporal_workflow_type` are intentionally excluded because pre-calculated percentile values cannot be accurately aggregated across additional dimensions. > #### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p50 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 50th percentile activity start-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency #### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p95 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 95th percentile activity start-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency #### temporal\_cloud\_v1\_activity\_start\_to\_close\_latency\_p99 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 99th percentile activity start-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency #### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p50 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 50th percentile activity schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency #### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p95 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 95th percentile activity schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency #### temporal\_cloud\_v1\_activity\_schedule\_to\_close\_latency\_p99 > **⚠️ Caution:** > > Avoid aggregating this metric across dimensions because the percentile won't be accurate. > The 99th percentile activity schedule-to-close latency in seconds. | Label | Description | | ----- | ----- | | `temporal_activity_type` | The activity type (opt-in) | **Type**: Latency ### Task Queue Metrics > **⚠️ Caution:** > High Cardinality > > These metrics could have high cardinality depending on number of task queues present. > #### temporal\_cloud\_v1\_approximate\_backlog\_count The approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue. > **📝 Note:** > Known accuracy limitations > This metric is approximate. > It can overcount because invalid or expired Tasks, like from cancelled, terminated, completed, or timed out Workflows, remain in the count until they reach the head of the queue and are processed and discarded. > > It can also reset to zero on an idle Task Queue. If no Worker polls, no new Tasks are added, and no other Task Queue calls occur (such as `DescribeTaskQueue` or `UpdateTaskQueueConfig`) for approximately 5 minutes. The Task Queue is unloaded from memory. > Infrequent metadata updates and database time-to-live settings can also cause this metric to drift at a smaller magnitude. > See [backlog accuracy limitations](/develop/worker-performance/task-queues#backlog-accuracy-limitations) for details. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `task_type` | Type of task: `workflow` or `activity` | | `task_priority` | The task priority | | `temporal_worker_deployment_name` | The Worker Deployment name (opt-in) | | `temporal_worker_build_id` | The Worker Deployment Version Build ID (opt-in) | **Type**: Value #### temporal\_cloud\_v1\_poll\_success\_count Successfully matched tasks per second. | Label | Description | | ----- | ----- | | `operation` | The poll operation name | | `task_type` | Type of task: `workflow` or `activity` | | `temporal_task_queue` | The task queue name | **Type**: Rate #### temporal\_cloud\_v1\_poll\_success\_sync\_count Tasks matched synchronously per second (no polling wait). | Label | Description | | ----- | ----- | | `operation` | The poll operation name | | `task_type` | Type of task: `workflow` or `activity` | | `temporal_task_queue` | The task queue name | **Type**: Rate #### temporal\_cloud\_v1\_poll\_timeout\_count The rate of poll requests that timed out without receiving a task. | Label | Description | | ----- | ----- | | `operation` | The poll operation name | | `task_type` | Type of task: `workflow` or `activity` | | `temporal_task_queue` | The task queue name | **Type**: Rate #### temporal\_cloud\_v1\_no\_poller\_tasks\_count The rate of tasks added to queues with no active pollers. | Label | Description | | ----- | ----- | | `temporal_task_queue` | The task queue name | | `task_type` | Type of task: `workflow` or `activity` | **Type**: Rate ### Namespace Metrics #### temporal\_cloud\_v1\_namespace\_open\_workflows The current number of open workflows in a namespace. **Type**: Value #### temporal\_cloud\_v1\_total\_action\_count The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``. | Label | Description | | ----- | ----- | | `is_background` | Whether the action was background: `true` or `false`. Background actions do not count toward the action rate limit | | `namespace_mode` | Indicates if actions are produced by an `active` or a `standby` Namespace | > **📝 Note:** > > Does not include the `region` label. Actions are scoped to the Namespace level. > #### temporal\_cloud\_v1\_billable\_action\_count The number of billable actions per second, broken down by action type and Workflow Type. Not all billable actions are included in this metric; see [Actions](/cloud/actions) for details on exceptions. | Label | Description | | ----- | ----- | | `action_type` | The [action](/cloud/actions) type | | `temporal_workflow_type` | The workflow type | > **⚠️ Caution:** > High Cardinality > > This metric could have high cardinality depending on number of action types and workflow types. > **Type**: Rate #### temporal\_cloud\_v1\_total\_action\_throttled\_count The total number of actions throttled per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric. **Type**: Rate #### temporal\_cloud\_v1\_operations\_count Operations performed per second. | Label | Description | | ----- | ----- | | `operation` | The name of the operation | | `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit | | `namespace_mode` | Indicates if operations are produced by an `active` or a `standby` Namespace | **Type**: Rate #### temporal\_cloud\_v1\_operations\_throttled\_count Operations throttled due to rate limits per second. See [Monitoring Trends Against Limits](/cloud/service-health#rps-aps-rate-limits) for guidance on setting alert thresholds against the corresponding limit metric. | Label | Description | | ----- | ----- | | `operation` | The name of the operation | | `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit | | `namespace_mode` | Indicates if actions are throttled in an `active` or a `standby` Namespace | **Type**: Rate ### Schedule Metrics #### temporal\_cloud\_v1\_schedule\_action\_success\_count Successful scheduled executions per second. **Type**: Rate #### temporal\_cloud\_v1\_schedule\_buffer\_overruns\_count Scheduled executions dropped per second because the Schedule's internal Action buffer is full. **Type**: Rate #### temporal\_cloud\_v1\_schedule\_missed\_catchup\_window\_count Scheduled executions permanently dropped per second because they fell outside the Catchup Window. This metric is not emitted for backfills. **Type**: Rate #### temporal\_cloud\_v1\_schedule\_rate\_limited\_count [DEPRECATED] Scheduled executions delayed per second due to rate limiting. This metric is in the process of being replaced by standard request per second rate limit monitoring with dynamic scaling. This metric will be removed when this change is rolled out to all of Temporal Cloud. **Type**: Rate #### temporal\_cloud\_v1\_schedule\_overlap\_skipped\_count Schedule executions dropped per second due to the overlap policy. This metric is emitted for `SKIP` when another execution is running or already selected, and for `BUFFER_ONE` when its single buffered slot is occupied. It can also be emitted during backfills that use `SKIP` or `BUFFER_ONE`. A sustained rate can indicate that executions are generated faster than they complete. | Label | Description | | ----- | ----- | | `schedule_overlap_policy` | The overlap policy that caused the execution to be dropped | **Type**: Rate ### Replication Metrics #### temporal\_cloud\_v1\_replication\_lag\_p50 The 50th percentile cross-region replication lag in seconds. **Type**: Latency #### temporal\_cloud\_v1\_replication\_lag\_p95 The 95th percentile cross-region replication lag in seconds. **Type**: Latency #### temporal\_cloud\_v1\_replication\_lag\_p99 The 99th percentile cross-region replication lag in seconds. **Type**: Latency ### Limit Metrics #### temporal\_cloud\_v1\_operations\_limit The current configured operations per second limit for a namespace. **Type**: Value #### temporal\_cloud\_v1\_action\_limit The current configured actions per second limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_total_action_count`` and `is_background=false`. **Type**: Value #### temporal\_cloud\_v1\_service\_request\_limit The current configured frontend service RPS limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_request_count`` **Type**: Value #### temporal\_cloud\_v1\_poller\_limit The current configured poller limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_pending_requests``. **Type**: Value #### temporal\_cloud\_v1\_action\_on\_demand\_envelope\_limit The on-demand envelope limit for actions per second. For Namespaces in provisioned capacity mode, this shows what the action limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_action_limit`. > **📝 Note:** > > Does not include the `region` label. Limits are scoped to the Namespace level. > **Type**: Value #### temporal\_cloud\_v1\_operations\_on\_demand\_envelope\_limit The on-demand envelope limit for operations per second. For Namespaces in provisioned capacity mode, this shows what the operations limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_operations_limit`. > **📝 Note:** > > Does not include the `region` label. Limits are scoped to the Namespace level. > **Type**: Value #### temporal\_cloud\_v1\_service\_request\_on\_demand\_envelope\_limit The on-demand envelope limit for service requests per second. For Namespaces in provisioned capacity mode, this shows what the service request limit would be if operating in on-demand mode. For Namespaces already in on-demand mode, this tracks the same value as `temporal_cloud_v1_service_request_limit`. > **📝 Note:** > > Does not include the `region` label. Limits are scoped to the Namespace level. > **Type**: Value #### temporal\_cloud\_v1\_provisioned\_capacity\_tru\_count The number of Temporal Resource Units (TRUs) provisioned for a Namespace. This is `0` for Namespaces that are not in [provisioned capacity mode](/cloud/capacity-modes). > **📝 Note:** > > Does not include the `region` label. Provisioned capacity is scoped to the Namespace level. > **Type**: Value --- # OpenMetrics migration guide Source: https://docs.temporal.io/cloud/metrics/openmetrics/migration-guide > Migrate from the Prometheus query endpoint to the new OpenMetrics endpoint in Temporal Cloud. Temporal Cloud is transitioning from our Prometheus query endpoint to an industry-standard OpenMetrics (Prometheus-compatible) endpoint for metrics collection. This migration represents a significant improvement in how you can monitor your Temporal Cloud workloads, bringing enhanced capabilities, better integration with observability tools, and access to high-cardinality metrics that were previously unavailable. > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint was deprecated on April 2, 2026 and is no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > Complete your migration to the OpenMetrics endpoint before this date. > ## Why We're Making This Change 1. **Industry-Standard Format**: Native compatibility with Prometheus and OpenTelemetry and all major observability platforms such as Datadog or New Relic without custom integrations. 2. **High-Cardinality Metrics**: Access to previously unavailable dimensions including: - `temporal_task_queue` labels on multiple metrics - `temporal_workflow_type` labels for workflow-specific monitoring - New task queue backlog metrics for better operational visibility 3. **Accurate Percentiles**: Our new system provides accurate percentile calculations for latency metrics, even in the presence of substantial outliers, unlike Prometheus-style histograms. 4. **Simplified Integration**: Direct scraping from your observability tools without intermediate translation layers. 5. **Enhanced Performance**: Optimized for high-cardinality data with built-in safeguards for system stability. Data is available to scrape three minutes from the time it was emitted, in line with the freshest metrics [available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/). ## What's Changing | Aspect | Current Query Endpoint | New OpenMetrics Endpoint | | ---------------------- | -------------------------------------------------- | ------------------------------------------- | | **Protocol** | Prometheus Query API (`/api/v1/query`) | OpenMetrics scrape endpoint (`/v1/metrics`) | | **Authentication** | mTLS certificates with customer-specific endpoints | API keys with global endpoint | | **Metric Temporality** | Cumulative counters | Delta temporality (pre-computed rates) | | **Query Requirement** | Direct queries supported | Requires observability platform | | **Cardinality** | Limited labels | High-cardinality labels available | | **Metric Naming** | `*_v0_*` metrics | `*_v1_*` metrics | ## Migration Timeline **April 2, 2026 - PromQL endpoint deprecated** - The PromQL endpoint is no longer accepting new users. - Existing users should begin migrating to the OpenMetrics endpoint. **October 5, 2026 - PromQL endpoint disabled** - The PromQL endpoint will be disabled for all users. - All metrics consumption must use the OpenMetrics endpoint by this date. > **❗ Important:** > Action Required > > Complete migration before October 5, 2026. > ## Notable Differences ### 1\. No longer use `rate()` in Prometheus queries Metrics are now pre-computed as per-second rates with delta temporality. **Before (Prometheus query endpoint)**: ``` rate(temporal_cloud_v0_frontend_service_request_count[1m]) ``` **After (OpenMetrics endpoint)**: ``` temporal_cloud_v1_service_request_count ``` ### 2\. Functions that no longer apply Metrics from OpenMetrics are already rates, therefore certain Prometheus functions no longer make sense. Below is a non-exhaustive list of some of the functions: - ❌ `rate()` \- Already computed - ❌ `increase()` \- Increase of a rate is meaningless - ❌ `irate()` \- Instant rate not applicable - ❌ `histogram_quantile()` \- Not applicable (explicit percentiles provided instead) - ✅ `sum()`, `avg()`, `max()`, `min()` \- Still work normally ### 3\. Percentile metrics The new endpoint provides explicit percentile metrics (p50, p95, p99) rather than histogram buckets: **Before (Prometheus query endpoint)**: Calculate percentiles using `histogram_quantile()` ```shell histogram_quantile(0.95, rate(temporal_cloud_v0_service_latency_bucket[5m])) ``` **After (OpenMetrics endpoint)**: Use pre-calculated percentiles directly ``` temporal_cloud_v1_service_latency_p95 ``` **Important Tradeoff**: While pre-calculated percentiles are more accurate for individual time series, they _cannot be accurately aggregated_. For example: - ❌ Cannot sum or average p95 values across Namespaces to get a global p95 - ❌ Cannot aggregate p95 values across regions or Task Queues - ✅ Can still view individual namespace/task queue percentiles accurately - ✅ More accurate percentile calculations for individual series, especially with outliers > **⚠️ Caution:** > Don't compare a v0 average against a v1 percentile > > The v0 latency metrics are a histogram, not a percentile. Dividing `temporal_cloud_v0_service_latency_sum` by > `temporal_cloud_v0_service_latency_count` yields an **average** (roughly a p50), and a single > `temporal_cloud_v0_service_latency_bucket{le="..."}` series only **counts** requests below a threshold. Neither is a p95 > or p99. > > If your v0 alert compared an average (or a raw `_sum` / `_bucket` value) against a latency threshold, switching to > `temporal_cloud_v1_service_latency_p95` / `_p99` reports higher values for identical traffic. This is a measurement > change, not a latency regression. > > To migrate latency alerts safely: > > - Compare like-for-like. To reproduce a former average-based alert, start on `temporal_cloud_v1_service_latency_p50`, > then move to `_p95` / `_p99` deliberately once you have set an appropriate threshold. > - Confirm which percentile your SLO targets. The Temporal Cloud [latency SLO](/cloud/service-availability#latency) is a > **p99**; alerting on p95 against a p99 threshold trips earlier than the SLO. > ### 4\. Authentication Setup **Before**: mTLS certificates with customer-specific endpoint ```shell curl --cert /path/to/client.pem \ --key /path/to/client.key \ --cacert /path/to/ca.pem \ "https://.tmprl.cloud/api/v1/query?query=rate(temporal_cloud_v0_frontend_service_request_count[5m])&time=2025-01-15T10:00:00Z" ``` **After**: API key with global endpoint ```shell curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics ``` ## Migration Steps ### Create an API Key Create a service account within the Temporal Cloud UI settings with the “Metrics Read-Only” Account Level Role. > **📝 Note:** > > As this is an account-level role, scoping it to specific namespaces has no effect as it will have access to the full > account’s metrics. > ![Create Service Account with Metrics Read-Only Role](/img/cloud/metrics/service-account-with-metrics-role.png) Once this is created, you can create an API key within this service account which will inherit the role. Save this API key in a secure location and use it to access the metrics APIs. To test that this works, curl the endpoint with your API Key. The output should resemble the following example: ```shell $ curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics # TYPE temporal_cloud_v1_service_error_count gauge # HELP temporal_cloud_v1_service_error_count The number of gRPC errors returned by frontend service # TYPE temporal_cloud_v1_service_pending_requests gauge # HELP temporal_cloud_v1_service_pending_requests The number of pollers that are waiting for a task # TYPE temporal_cloud_v1_service_request_count gauge # HELP temporal_cloud_v1_service_request_count The number of RPC requests received by the service.. ``` Now you are ready to scrape your metrics\! ### Migrate the Datadog integration 1. Install and configure the [Temporal Cloud - OpenMetrics integration](https://docs.datadoghq.com/integrations/temporal-cloud-openmetrics/). The new integration uses the `temporal.cloud.v1_*` metric prefix, while the deprecated integration uses `temporal.cloud.v0_*`, so both integrations can run during the migration without metric name collisions. 2. After you confirm that the new metrics are available in Datadog, optionally uninstall the deprecated [Temporal Cloud integration](https://docs.datadoghq.com/integrations/temporal-cloud/). Leaving the deprecated integration installed does not affect the new integration, but it stops receiving data when the PromQL endpoint is disabled and remains in a broken integration state. ### Configuring Grafana \+ Prometheus #### Update Prometheus Configuration Add a new scrape job for the OpenMetrics endpoint with your API key. ```yaml scrape_configs: - job_name: temporal-cloud static_configs: - targets: - 'metrics.temporal.io' scheme: https metrics_path: '/v1/metrics' honor_timestamps: true scrape_interval: 30s scrape_timeout: 30s authorization: type: Bearer credentials: 'API_KEY' ``` > **📝 Note:** > > This replaces the direct Grafana datasource configuration you used with the query endpoint. > #### Install New Dashboards - Download the new Grafana dashboard: [temporal_cloud_openmetrics.json](https://github.com/temporalio/dashboards/blob/master/cloud/temporal_cloud_openmetrics.json) - Import alongside existing dashboards during transition - Update any custom alerts and queries to use new metrics and remove `rate()` functions #### Other Observability Providers Consult the documentation for your observability system for how to configure it to scrape this endpoint and retrieve your metrics: - [NewRelic](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-openmetrics/configure-prometheus-openmetrics-integrations/) - [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/configuration/#receivers) Examples for all these integrations live [here](https://github.com/temporal-community/cloud-metrics-scrape-examples). ### Metric Mapping Reference Below is a template for mapping metrics from the old query endpoint to the new OpenMetrics endpoint. Note that all metrics follow the pattern of `v0` → `v1` version change, and the fundamental difference is the shift from cumulative counters to pre-computed rates for the majority of the metrics. Note that the labels below are only new labels added to the metrics. For the complete list of labels, see the /cloud/metrics/openmetrics/metrics-reference. #### Frontend Service Metrics | Old Metric (v0) | New Metric (v1) | New Labels | | -------------------------------------------------- | -------------------------------------------------- | ---------- | | `temporal_cloud_v0_frontend_service_error_count` | `temporal_cloud_v1_service_error_count` | `region` | | `temporal_cloud_v0_frontend_service_request_count` | `temporal_cloud_v1_service_request_count` | `region` | | `temporal_cloud_v0_resource_exhausted_error_count` | `temporal_cloud_v1_resource_exhausted_error_count` | `region` | | `temporal_cloud_v0_state_transition_count` | No direct equivalent | - | | `temporal_cloud_v0_total_action_count` | `temporal_cloud_v1_total_action_count` | `region` | > **📝 Note:** > State transition count removed > > `temporal_cloud_v0_state_transition_count` does not have an equivalent metric in the OpenMetrics endpoint. > To size workloads for Temporal Cloud (for example, when migrating from self-hosted), use action-based metrics (`temporal_cloud_v1_total_action_count`) and request-based metrics (`temporal_cloud_v1_service_request_count`) together instead. > #### Workflow Metrics | Old Metric (v0) | New Metric (v1) | New Labels | | --------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------- | | `temporal_cloud_v0_workflow_cancel_count` | `temporal_cloud_v1_workflow_cancel_count` | `region` `temporal_workflow_type` `temporal_task_queue` | | `temporal_cloud_v0_workflow_continued_as_new_count` | `temporal_cloud_v1_workflow_continued_as_new_count` | `region` `temporal_workflow_type` `temporal_task_queue` | | `temporal_cloud_v0_workflow_failed_count` | `temporal_cloud_v1_workflow_failed_count` | `region` `temporal_workflow_type` `temporal_task_queue` | | `temporal_cloud_v0_workflow_success_count` | `temporal_cloud_v1_workflow_success_count` | `region` `temporal_workflow_type` `temporal_task_queue` | | `temporal_cloud_v0_workflow_terminate_count` | `temporal_cloud_v1_workflow_terminate_count` | `region` `temporal_workflow_type` `temporal_task_queue` | | `temporal_cloud_v0_workflow_timeout_count` | `temporal_cloud_v1_workflow_timeout_count` | `region` `temporal_workflow_type` `temporal_task_queue` | #### Poll Metrics | Old Metric (v0) | New Metric (v1) | New Labels | | ------------------------------------------- | ------------------------------------------- | ------------------------------ | | `temporal_cloud_v0_poll_success_count` | `temporal_cloud_v1_poll_success_count` | `region` `temporal_task_queue` | | `temporal_cloud_v0_poll_success_sync_count` | `temporal_cloud_v1_poll_success_sync_count` | `region` `temporal_task_queue` | | `temporal_cloud_v0_poll_timeout_count` | `temporal_cloud_v1_poll_timeout_count` | `region` `temporal_task_queue` | #### Latency Metrics | Old Metric (v0) | New Metric (v1) | New Labels | | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ---------- | | `temporal_cloud_v0_service_latency_bucket temporal_cloud_v0_service_latency_count temporal_cloud_v0_service_latency_sum` | `temporal_cloud_v1_service_latency_p99 temporal_cloud_v1_service_latency_p95 temporal_cloud_v1_service_latency_p50` | `region` | | `temporal_cloud_v0_replication_lag_bucket temporal_cloud_v0_replication_lag_count temporal_cloud_v0_replication_lag_sum` | `temporal_cloud_v1_replication_lag_p99 temporal_cloud_v1_replication_lag_p95 temporal_cloud_v1_replication_lag_p50` | `region` | #### Schedule Metrics | Old Metric (v0) | New Metric (v1) | New Labels | | -------------------------------------------------------- | -------------------------------------------------------- | ---------- | | `temporal_cloud_v0_schedule_action_success_count` | `temporal_cloud_v1_schedule_action_success_count` | `region` | | `temporal_cloud_v0_schedule_buffer_overruns_count` | `temporal_cloud_v1_schedule_buffer_overruns_count` | `region` | | `temporal_cloud_v0_schedule_missed_catchup_window_count` | `temporal_cloud_v1_schedule_missed_catchup_window_count` | `region` | | `temporal_cloud_v0_schedule_rate_limited_count` | `temporal_cloud_v1_schedule_rate_limited_count` | `region` | In addition to these metrics, there are a number of new metrics provided by our OpenMetrics endpoint. > **ℹ️ Info:** > > See the [metrics reference](/cloud/metrics/openmetrics/metrics-reference) for an up-to-date list of all available metrics and their full descriptions. > ### Managing High-Cardinality The new endpoint provides access to high-cardinality labels that can significantly increase your metric volume: #### High-Cardinality Labels - `temporal_task_queue` - `temporal_workflow_type` #### Best Practices ##### Namespace/Metric filtering Namespace filtering can be used to ensure that metrics are scraped for relevant Namespaces, which reduces cardinality. ``` https://metrics.temporal.io/v1/metrics?namespaces=production-* ``` This can be taken further by only scraping relevant metrics for a given namespace which ensures that any new high cardinality metrics won’t be an issue for your observability system. ``` https://metrics.temporal.io/v1/metrics?metrics=temporal_cloud_v1_workflow_success_count?namespaces=production-* ``` ##### Relabeling If the above doesn’t work, consider dropping problematic labels post-scrape but pre-ingestion into your observability system. For example, in Prometheus this can be done via [relabeling rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). ```yaml metric_relabel_configs: - source_labels: [__name__] regex: 'temporal_cloud_v1_poll_success_count' action: labeldrop regex: 'temporal_task_queue' ``` Or you can even relabel certain label values in order to keep significant ones. For example, it’s possible to rename less important task queues to “unknown” while retaining important ones. ```yaml metric_relabel_configs: - source_labels: [temporal_task_queue] regex: '(critical-queue|payment-queue)' target_label: __tmp_keep_original replacement: 'true' # For anything without the keep flag, replace with "unknown" - source_labels: [__tmp_keep_original] regex: '' # empty/missing value target_label: temporal_task_queue replacement: 'unknown' # Clean up the temporary label - regex: '__tmp_keep_original' action: labeldrop ``` ## Limits See [API limits](/cloud/metrics/openmetrics/api-reference#api-limits) for details. ## FAQ ### What new metrics are available in OpenMetrics that are not available in PromQL? OpenMetrics provides visibility into several new areas - **Activity execution and latency**: completion outcomes (success, fail, timeout, cancel, terminate, plus activity-task-level fail/timeout) and latency percentiles (start-to-close, schedule-to-close), including Standalone Activities. - **Task queue health**: approximate backlog depth and a rate of tasks added to queues with no active pollers. - **Namespace utilization and rate limits**: operations counters, throttled-action and throttled-operation rates, a billable-action breakdown, an open-workflow gauge, and configured-limit and on-demand-envelope-limit gauges to chart usage against quotas. - **Usage against limits**: a request-throttled rate and a pending-requests (active long-pollers) gauge. - **Higher-dimensional breakdowns**: new labels including region, task queue, workflow type, activity type, worker deployment, and more. **New metrics:** - `temporal_cloud_v1_service_request_throttled_count` - `temporal_cloud_v1_service_pending_requests` - `temporal_cloud_v1_activity_success_count` - `temporal_cloud_v1_activity_fail_count` - `temporal_cloud_v1_activity_timeout_count` - `temporal_cloud_v1_activity_task_fail_count` - `temporal_cloud_v1_activity_task_timeout_count` - `temporal_cloud_v1_activity_cancel_count` - `temporal_cloud_v1_activity_terminate_count` - `temporal_cloud_v1_activity_start_to_close_latency_p50` / `_p95` / `_p99` - `temporal_cloud_v1_activity_schedule_to_close_latency_p50` / `_p95` / `_p99` - `temporal_cloud_v1_approximate_backlog_count` - `temporal_cloud_v1_no_poller_tasks_count` - `temporal_cloud_v1_namespace_open_workflows` - `temporal_cloud_v1_billable_action_count` - `temporal_cloud_v1_total_action_throttled_count` - `temporal_cloud_v1_operations_count` - `temporal_cloud_v1_operations_throttled_count` - `temporal_cloud_v1_action_limit` - `temporal_cloud_v1_operations_limit` - `temporal_cloud_v1_service_request_limit` - `temporal_cloud_v1_poller_limit` - `temporal_cloud_v1_action_on_demand_envelope_limit` - `temporal_cloud_v1_operations_on_demand_envelope_limit` - `temporal_cloud_v1_service_request_on_demand_envelope_limit` **New labels:** - `region` - `temporal_task_queue` - `temporal_workflow_type` - `task_type` - `task_priority` - `is_background` - `namespace_mode` - `action_type` - `timeout_type` - `temporal_activity_type` (opt-in) - `temporal_worker_deployment_name` (opt-in) - `temporal_worker_build_id` (opt-in) - `worker_version` (opt-in) For full descriptions, types, and per-metric labels, see the [OpenMetrics metrics reference](/cloud/metrics/openmetrics/metrics-reference). ### Will metrics match between promQL and OpenMetrics endpoints? No. The metrics will be approximately the same but due to aggregation differences and windowing, values likely won't match exactly between the two endpoints. Some metrics may be consistently different such as `temporal_cloud_v1_total_action_count` which includes History Export actions in the OpenMetrics endpoint. In the case of consistent differences the OpenMetrics endpoint is considered to be more accurate. ### Can I still query metrics directly (for example, with a Grafana dashboard)? Currently, the OpenMetrics endpoint requires an observability platform to collect and query metrics. Direct querying via API to return a time series of data is not supported. Supporting this type of query pattern is a future roadmap item. ### What happens to my existing dashboards and alerts? During the transition period, both endpoints remain active. ### Will historical data be preserved? Historical data from the query endpoint will remain in your observability platform. To maintain continuity: - Combine old (`v0`) and new (`v1`) metrics in your queries during transition - Consider using the PromQL `or` operator: `metric_v1 or metric_v0` ### Are there limits to how frequently I can scrape or how much data will be returned? The limits are documented [here](/cloud/metrics/openmetrics/api-reference#api-limits). ### Why are some metrics missing from my scrapes? I don’t see all the metrics documented. The OpenMetrics endpoint only returns metrics that were generated during the one-minute aggregation window. This is different from the query endpoint which might return zeros. **What this means:** - If no workflows failed in the last minute, `temporal_cloud_v1_workflow_failed_count` won't appear in that scrape. - If a specific task queue had no activity, its metrics will be absent. - The set of metrics returned varies between scrapes based on system activity. **This is normal behavior.** Unlike some metrics systems that populate zeros, the OpenMetrics endpoint follows a sparse reporting pattern \- metrics only appear when there's actual data to report. **How to handle this in queries:** ``` (temporal_cloud_v1_workflow_failed_count{namespace="production"} or vector(0)) ``` This ensures your dashboards and alerts work correctly even when metrics are temporarily absent due to no activity. --- # Set up the legacy PromQL endpoint with Prometheus and Grafana Source: https://docs.temporal.io/cloud/metrics/prometheus-grafana > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint and its `temporal_cloud_v0_*` metrics were deprecated on April 2, 2026 and are no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > > For Grafana setup with the OpenMetrics endpoint, see the [OpenMetrics integrations page](/cloud/metrics/openmetrics/metrics-integrations). > **How to set up Grafana with Temporal Cloud PromQL endpoint to view Cloud metrics.** Temporal Cloud emits metrics through a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/), which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform. > **📝 Note:** > > For setting up SDK metrics (emitted by your Workers and Clients), see > [SDK metrics setup](/cloud/metrics/sdk-metrics-setup). > The process for setting up Temporal Cloud PromQL to work with Grafana includes the following steps: 1. [Generate a Prometheus HTTP API endpoint](/cloud/metrics/general-setup) on Temporal Cloud using valid certificates. 2. Run Grafana and [set up a data source for Temporal Cloud metrics](#grafana-data-source-configuration) in Grafana. 3. [Create dashboards](#grafana-dashboards-setup) in Grafana to view Temporal Cloud metrics. Temporal provides [sample community-driven Grafana dashboards](https://github.com/temporalio/dashboards) for Cloud metrics that you can use and customize according to your requirements. If you're following through with the examples provided here, ensure that you have the following: - Root CA certificates and end-entity certificates. See [Certificate requirements](/cloud/certificates#certificate-requirements) for details. - Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. See Connect to a Temporal Service for details. - [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud) - [Java](/develop/java/client/temporal-client#connect-to-temporal-cloud) - [PHP](/develop/php/client/temporal-client#connect-to-a-dev-cluster) - [Python](/develop/python/client/temporal-client#connect-to-temporal-cloud) - [TypeScript](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) - [.NET](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) - Grafana installed. ## Temporal Cloud metrics setup Before you set up your Temporal Cloud metrics, ensure that you have the following: - Account Owner or Global Admin [role privileges](/cloud/manage-access/roles-and-permissions#account-level-roles) for the Temporal Cloud account. - [CA certificate and key](/cloud/certificates) for the Observability integration. You will need the certificate to set up the Observability endpoint in Temporal Cloud. The following steps describe how to set up Observability on Temporal Cloud to generate an endpoint: 1. Log in to Temporal Cloud UI with an Account Owner or Global Admin [role](/cloud/manage-access/roles-and-permissions#account-level-roles). 2. Go to **Settings** and select **Integrations**. 3. Select **Configure Observability** (if you're setting it up for the first time) or click **Edit** in the Observability section (if it was already configured before). 4. Add your root CA certificate (.pem) and save it. Note that if an observability endpoint is already set up, you can append your root CA certificate here to use the generated observability endpoint with your instance of Grafana. 5. To test your endpoint, run the following command on your host: ``` curl -v --cert --key "/api/v1/query?query=temporal_cloud_v0_state_transition_count" ``` If you have Workflows running on a Namespace in your Temporal Cloud instance, you should see some data as a result of running this command. 6. Copy the HTTP API endpoint that is generated (it is shown in the UI). This endpoint should be configured as a data source for Temporal Cloud metrics in Grafana. See [Grafana data source configuration](#grafana-data-source-configuration) for details. ## SDK metrics setup SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions. You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics. Each language development guide has details on how to set this up. - [Go SDK](/develop/go/platform/observability#metrics) - [Java SDK](/develop/java/platform/observability#metrics) - [TypeScript SDK](/develop/typescript/platform/observability#metrics) - [Python](/develop/python/platform/observability#metrics) - [.NET](/develop/dotnet/platform/observability#metrics) The following example uses the Java SDK to set the Prometheus registry and Micrometer stats reporter, set the scope, and expose an endpoint from which Prometheus can scrape the SDK metrics. ```java //You need the following packages to set up metrics in Java. //See the Developer's guide for packages required for other SDKs. //… import com.sun.net.httpserver.HttpServer; import com.uber.m3.tally.RootScopeBuilder; import com.uber.m3.tally.Scope; import com.uber.m3.util.Duration; import com.uber.m3.util.ImmutableMap; import io.micrometer.prometheus.PrometheusConfig; import io.micrometer.prometheus.PrometheusMeterRegistry; import io.temporal.common.reporter.MicrometerClientStatsReporter; import java.io.IOException; import java.io.OutputStream; import java.net.InetSocketAddress; import io.temporal.serviceclient.SimpleSslContextBuilder; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.io.FileInputStream; import java.io.InputStream; //… { // See the Micrometer documentation for configuration details on other supported monitoring systems. // Set up the Prometheus registry. PrometheusMeterRegistry yourRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); public static Scope yourScope(){ //Set up a scope, report every 10 seconds Scope yourScope = new RootScopeBuilder() .tags(ImmutableMap.of( "customtag1", "customvalue1", "customtag2", "customvalue2")) .reporter(new MicrometerClientStatsReporter(yourRegistry)) .reportEvery(Duration.ofSeconds(10)); //Start Prometheus scrape endpoint at port 8077 on your local host HttpServer scrapeEndpoint = startPrometheusScrapeEndpoint(yourRegistry, 8077); return yourScope; } /** * Starts HttpServer to expose a scrape endpoint. See * https://micrometer.io/docs/registry/prometheus for more info. */ public static HttpServer startPrometheusScrapeEndpoint( PrometheusMeterRegistry yourRegistry, int port) { try { HttpServer server = HttpServer.create(new InetSocketAddress(port), 0); server.createContext( "/metrics", httpExchange -> { String response = registry.scrape(); httpExchange.sendResponseHeaders(200, response.getBytes(UTF_8).length); try (OutputStream os = httpExchange.getResponseBody()) { os.write(response.getBytes(UTF_8)); } }); server.start(); return server; } catch (IOException e) { throw new RuntimeException(e); } } } //… // With your scrape endpoint configured, set the metrics scope in your Workflow service stub and // use it to create a Client to start your Workers and Workflow Executions. //… { //Create Workflow service stubs to connect to the Frontend Service. WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs( WorkflowServiceStubsOptions.newBuilder() .setMetricsScope(yourScope()) //set the metrics scope for the WorkflowServiceStubs .build()); //Create a Workflow service client, which can be used to start, signal, and query Workflow Executions. WorkflowClient yourClient = WorkflowClient.newInstance(service, WorkflowClientOptions.newBuilder().build()); } //… ``` To check whether your scrape endpoints are emitting metrics, run your code and go to [http://localhost:8077/metrics](http://localhost:8077/metrics) to verify that you see the SDK metrics. You can set up separate scrape endpoints in your Clients that you use to start your Workers and Workflow Executions. For more examples on setting metrics endpoints in other SDKs, see the metrics samples: - [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) - [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics) ## Grafana data source configuration **How to configure the Temporal Cloud metrics data source in Grafana.** Depending on how you use Grafana, you can either install and run it locally, run it as a Docker container, or log in to Grafana Cloud to set up your data sources. If you have installed and are running Grafana locally, go to [http://localhost:3000](http://localhost:3000) and sign in. To add the Temporal Cloud Prometheus HTTP API endpoint that we generated in the [Temporal Cloud metrics setup](/cloud/metrics/general-setup) section, do the following: 1. Go to **Configuration > Data sources**. 1. Select **Add data source > Prometheus**. 1. Enter a name for your Temporal Cloud metrics data source, such as _Temporal Cloud metrics_. 1. In the **Connection** section, paste the URL that was generated in the Observability section on the Temporal Cloud UI. 1. The **Authentication** section may be left as **No Authentication**. 1. In the **TLS Settings** section, select **TLS Client Authentication**: - Leave **ServerName** blank. This is not required. - Paste in your end-entity certificate and key. - Note that the end-entity certificate used here must be part of the certificate chain with the root CA certificates used in your [Temporal Cloud observability setup](/cloud/metrics/general-setup). ![Data source configuration in Grafana](/img/cloud/prometheus/add-prometheus-api-endpoint.png) 1. Click **Save and test** to verify that the data source is working. If you see issues in setting this data source, verify your CA certificate chain and ensure that you are setting the correct certificates in your Temporal Cloud observability setup and in the TLS authentication in Grafana. ### Grafana dashboards setup To set up dashboards in Grafana, you can use the UI or configure them directly in your Grafana deployment. > **💡 Tip:** > > Temporal provides community-driven > [example dashboards for Temporal Cloud](https://github.com/temporalio/dashboards/tree/master/cloud) that you can > customize to meet your needs. > To import a dashboard in Grafana: 1. In the left-hand navigation bar, select **Dashboards** > **Import dashboard**. 2. You can either copy and paste the JSON from the [Temporal Cloud sample dashboards](https://github.com/temporalio/dashboards/tree/master/cloud), or import the JSON files into Grafana. 3. Save the dashboard and review the metrics data in the graphs. To configure dashboards with the UI: 1. Go to **Create > Dashboard** and add an empty panel. 2. On the **Panel configuration** page, in the **Query** tab, select the "Temporal Cloud metrics" data source that you configured earlier. 3. Expand the **Metrics browser** and select the metrics you want. You can also select associated labels and values to sort the query data. The [PromQL documentation](/cloud/metrics/reference) lists all metrics emitted from PromQL in Temporal Cloud. 4. The graph should now display data based on your selected queries. --- # Legacy PromQL endpoint Source: https://docs.temporal.io/cloud/metrics/promql > Get detailed insights into your Temporal Cloud Namespace metrics using your own observability tool. Access data with a CA certificate and retain raw metrics for seven days. > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint and its `temporal_cloud_v0_*` metrics were deprecated on April 2, 2026 and are no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > > Migrate to the [OpenMetrics endpoint](/cloud/metrics/openmetrics). > See the [migration guide](/cloud/metrics/openmetrics/migration-guide) for a complete v0-to-v1 metric mapping. > Metrics for all Namespaces in your account are available from your metrics endpoint. Keep in mind that your Temporal Cloud metrics lag real-time performance by about one minute. Temporal Cloud also only retains raw metrics for seven days. To ensure security of your metrics, a CA certificate dedicated to observability is required. Only clients that use certificates signed by that CA, or that chain up to the CA, can query the metrics endpoint. For more information about CA certificates in Temporal Cloud, see [Certificate requirements](/cloud/certificates#certificate-requirements). - [General setup](/cloud/metrics/general-setup) - [Available metrics](/cloud/metrics/reference) - [Prometheus & Grafana setup](/cloud/metrics/prometheus-grafana) --- # Temporal Cloud metrics reference Source: https://docs.temporal.io/cloud/metrics/reference > Explore Temporal Cloud metrics to query with PromQL or scrape via OpenMetrics, supporting rate and latency calculations. > **🚨 Danger:** > PromQL endpoint deprecated > > The PromQL endpoint and its `temporal_cloud_v0_*` metrics were deprecated on April 2, 2026 and are no longer accepting new users. > The PromQL endpoint will be disabled for all users on **October 5, 2026**. > > Migrate to the [OpenMetrics endpoint](/cloud/metrics/openmetrics) which provides `temporal_cloud_v1_*` metrics with higher cardinality, accurate percentiles, and simplified authentication. > See the [migration guide](/cloud/metrics/openmetrics/migration-guide) for a complete v0-to-v1 metric mapping. > A metric is a measurement or data point that provides insights into the performance and health of a system. This document describes the `temporal_cloud_v0_*` metrics available from the deprecated Temporal Cloud PromQL endpoint. For the current metrics reference, see the [OpenMetrics metrics reference](/cloud/metrics/openmetrics/metrics-reference). This document describes: - **[Available Temporal Cloud metrics](#available-metrics)**: The metrics emitted by Temporal Cloud include counts of gRPC errors, requests, successful task matches to a poller, and more. - **[Metrics labels](#metrics-labels)**: Temporal Cloud metrics labels can filter metrics and help categorize and differentiate results. - **[Operations](#metrics-operations)**: An operation is a special type of label that categorizes the type of operation being performed when the metric was collected. > **ℹ️ Info:** > SDK METRICS > > This document discusses metrics emitted by [Temporal Cloud](/cloud). > Temporal SDKs also emit metrics, sourced from Temporal Clients and Worker processes. > You can find information about Temporal SDK metrics on its [dedicated page](/references/sdk-metrics). > > Please note: > > - SDK metrics start with the phrase `temporal_`. > - Temporal Cloud metrics start with `temporal_cloud_`. > ## Available Temporal Cloud metrics **What metrics are emitted from Temporal Cloud?** The following metrics are emitted for your Namespaces: ### Frontend Service metrics #### temporal_cloud_v0_frontend_service_error_count This is a count of gRPC errors returned aggregated by operation. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_frontend_service_request_count This is a count of gRPC requests received aggregated by operation. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_resource_exhausted_error_count gRPC requests received that were rate-limited by Temporal Cloud, aggregated by cause. Labels: temporal_account, temporal_namespace, resource_exhausted_cause #### temporal_cloud_v0_state_transition_count Count of state transitions for each Namespace. #### temporal_cloud_v0_total_action_count Approximate count of Temporal Cloud Actions. Labels: temporal_account, temporal_namespace, is_background, namespace_mode ### Poll metrics #### temporal_cloud_v0_poll_success_count Tasks that are successfully matched to a poller. Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type #### temporal_cloud_v0_poll_success_sync_count Tasks that are successfully sync matched to a poller. Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type #### temporal_cloud_v0_poll_timeout_count When no tasks are available for a poller before timing out. Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type ### Replication lag metrics #### temporal_cloud_v0_replication_lag_bucket A histogram of [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) during a specific time interval for a Namespace with high availability. Labels: temporal_account, temporal_namespace, le #### temporal_cloud_v0_replication_lag_count The [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) count during a specific time interval for a Namespace with high availability. Labels: temporal_account, temporal_namespace #### temporal_cloud_v0_replication_lag_sum The sum of [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) during a specific time interval for a Namespace with high availability. Labels: temporal_account, temporal_namespace ### Schedule metrics #### temporal_cloud_v0_schedule_action_success_count Successful execution of a Scheduled Workflow. Labels: temporal_account, temporal_namespace #### temporal_cloud_v0_schedule_buffer_overruns_count When average schedule run length is greater than average schedule interval while a `buffer_all` overlap policy is configured. Labels: temporal_account, temporal_namespace #### temporal_cloud_v0_schedule_missed_catchup_window_count Skipped Scheduled executions when Workflows were delayed longer than the catchup window. Labels: temporal_account, temporal_namespace #### temporal_cloud_v0_schedule_rate_limited_count Workflows that were delayed due to exceeding a rate limit. Labels: temporal_account, temporal_namespace ### Service latency metrics #### temporal_cloud_v0_service_latency_bucket Latency for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations. Labels: temporal_account, temporal_namespace, le, operation, temporal_service_type #### temporal_cloud_v0_service_latency_count Count of latency observations for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_service_latency_sum Sum of latency observation time for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations. Labels: temporal_account, temporal_namespace, operation, temporal_service_type ### Workflow metrics #### temporal_cloud_v0_workflow_cancel_count Workflows canceled before completing execution. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_workflow_continued_as_new_count Workflow Executions that were Continued-As-New from a past execution. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_workflow_failed_count Workflows that failed before completion. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_workflow_success_count Workflows that successfully completed. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_workflow_terminate_count Workflows terminated before completing execution. Labels: temporal_account, temporal_namespace, operation, temporal_service_type #### temporal_cloud_v0_workflow_timeout_count Workflows that timed out before completing execution. Labels: temporal_account, temporal_namespace, operation, temporal_service_type ## Metrics labels **What labels can you use to filter metrics?** Temporal Cloud metrics include key-value pairs called labels in their associated metadata. Labels help you categorize and differentiate metrics for precise filtering, querying, and aggregation. Use labels to filter specific attributes or compare values, such as numeric buckets in histograms. This added context enhances the monitoring and analysis capabilities, providing deeper insights into your data. Use the following labels to filter metrics: | Label | Explanation | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `le` | Less than or equal to (`le`) is used in histograms to categorize observations into buckets based on their value being less than or equal to a predefined upper limit. | | `operation` | This includes gRPC operations and general Cloud operations such as:
  • SignalWorkflowExecution
  • StartBatchOperation
  • StartWorkflowExecution
  • TaskQueueMgr
  • TerminateWorkflowExecution
  • UpdateNamespace
  • UpdateSchedule
See: [Metric Operations](#metrics-operations) and [Temporal Cloud Operation reference](/references/operation-list)| | `resource_exhausted_cause` | Cause for resource exhaustion. | | `task_type` | Activity, Workflow, or Nexus. | | `temporal_account` | Temporal Account. | | `temporal_namespace` | Temporal Namespace. | | `temporal_service_type` | Frontend or Matching or History or Worker. | | `is_background` | This label on `temporal_cloud_v0_total_action_count` indicates when actions are produced by a Temporal background job. | | `namespace_mode` | This label on `temporal_cloud_v0_total_action_count` indicates if actions are produced by an active vs a standby Namespace. For a regular Namespace, `namespace_mode` will always be “active”. | The following is an example of how you can filter metrics using labels: ```text temporal_cloud_v0_poll_success_count{__rollup__="true", operation="TaskQueueMgr", task_type="Activity", temporal_account="12345", temporal_namespace="your_namespace.12345", temporal_service_type="matching"} ``` ## Operations **What operation labels are captured by Temporal Cloud?** Operations are a special class of metrics label. They describe the context during which a metric was captured. Temporal Cloud includes the following operations labels: - AdminDescribeMutableState - AdminGetWorkflowExecutionRawHistory - AdminGetWorkflowExecutionRawHistoryV2 - AdminReapplyEvents - CountWorkflowExecutions - CreateSchedule - DeleteSchedule - DeleteWorkflowExecution - DescribeBatchOperation - DescribeNamespace - DescribeSchedule - DescribeTaskQueue - DescribeWorkflowExecution - GetWorkerBuildIdCompatibility - GetWorkerTaskReachability - GetWorkflowExecutionHistory - GetWorkflowExecutionHistoryReverse - ListBatchOperations - ListClosedWorkflowExecutions - OperatorDeleteNamespace - PatchSchedule - PollActivityTaskQueue - PollNexusTaskQueue - PollWorkflowExecutionHistory - PollWorkflowExecutionUpdate - PollWorkflowTaskQueue - QueryWorkflow - RecordActivityTaskHeartbeat - RecordActivityTaskHeartbeatById - RegisterNamespace - RequestCancelWorkflowExecution - ResetStickyTaskQueue - ResetWorkflowExecution - RespondActivityTaskCanceled - RespondActivityTaskCompleted - RespondActivityTaskCompletedById - RespondActivityTaskFailed - RespondActivityTaskFailedById - RespondNexusTaskCompleted - RespondNexusTaskFailed - RespondQueryTaskCompleted - RespondWorkflowTaskCompleted - RespondWorkflowTaskFailed - SignalWithStartWorkflowExecution - SignalWorkflowExecution - StartBatchOperation - StartWorkflowExecution - StopBatchOperation - TerminateWorkflowExecution - UpdateNamespace - UpdateSchedule - UpdateWorkerBuildIdCompatibility - UpdateWorkflowExecution As the following table shows, certain [metrics groups](#available-metrics) support [operations](#metrics-operations) for aggregation and filtering: | Metrics Group / Operations | All Operations | SignalWithStartWorkflowExecution / SignalWorkflowExecution / StartWorkflowExecution | TaskQueueMgr | CompletionStats | | ----------------------------------------------- | -------------- | ----------------------------------------------------------------------------------- | ------------ | --------------- | | **[Frontend Service Metrics](#frontend)** | X | | | | | **[Service Latency Metrics](#service-latency)** | | X | | | | **[Poll Metrics](#poll)** | | | X | | | **[Workflow Metrics](#workflow)** | | | | X | --- # Set up SDK metrics with Prometheus and Grafana Source: https://docs.temporal.io/cloud/metrics/sdk-metrics-setup > Set up Temporal SDK metrics with Prometheus and Grafana for monitoring Workers and Client performance. SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions. Unlike [Temporal Cloud metrics](/cloud/metrics/), which are exposed through a Prometheus HTTP API endpoint, SDK metrics require you to set up a Prometheus scrape endpoint in your application code for Prometheus to collect and aggregate. For a full list of available SDK metrics and their descriptions, see the [SDK metrics reference](/references/sdk-metrics). The process for setting up SDK metrics includes the following steps: 1. [Expose a metrics endpoint](#sdk-metrics-setup) in your application code where Prometheus can scrape SDK metrics. 2. [Configure Prometheus](#prometheus-configuration) to scrape your SDK metrics endpoints. 3. [Add an SDK metrics data source](#grafana-data-source-configuration) in Grafana. 4. [Set up dashboards](#grafana-dashboards-setup) to visualize SDK metrics. Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. Ensure Prometheus and Grafana are installed. - [Go](/develop/go/client/temporal-client#connect-to-temporal-cloud) - [Java](/develop/java/client/temporal-client#connect-to-temporal-cloud) - [Python](/develop/python/client/temporal-client#connect-to-temporal-cloud) - [TypeScript](/develop/typescript/client/temporal-client#connect-to-temporal-cloud) - [.NET](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) ## Expose a metrics endpoint You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics. Each language development guide has details on how to set this up. - [Go SDK](/develop/go/platform/observability#metrics) - [Java SDK](/develop/java/platform/observability#metrics) - [TypeScript SDK](/develop/typescript/platform/observability#metrics) - [Python](/develop/python/platform/observability#metrics) - [.NET](/develop/dotnet/platform/observability#metrics) For working examples of how to configure metrics in each SDK, see the metrics samples: - [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics) - [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) - [TypeScript SDK Samples](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) - [Python SDK Samples](https://github.com/temporalio/samples-python/tree/main/custom_metric) - [.NET SDK Samples](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry/DotNetMetrics) Some examples use OpenTelemtry to instrument metrics. It is useful to use a [Prometheus exporter with OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter) to expose metrics for scraping. ## Configure Prometheus For Temporal SDKs, you must have Prometheus running and configured to listen on the scrape endpoints exposed in your application code. For this example, you can run Prometheus locally or as a Docker container. In either case, ensure that you set the listen targets to the ports where you expose your scrape endpoints. This configuration assumes the scrape endpoint is set to port 8077 as in the [SDK metrics setup](#sdk-metrics-setup) example. ```yaml global: scrape_interval: 30s # Set the scrape interval to every 30 seconds. Default is every 1 minute. #... # Set your scrape configuration targets to the ports exposed on your endpoints in the SDK. scrape_configs: - job_name: 'temporalsdkmetrics' metrics_path: /metrics scheme: http static_configs: - targets: # This is the scrape endpoint where Prometheus listens for SDK metrics. - localhost:8077 # You can have multiple targets here, provided they are set up in your application code. ``` See the [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details on how you can run Prometheus locally or using Docker. To check whether Prometheus is receiving metrics from your SDK target, go to [http://localhost:9090](http://localhost:9090) and navigate to **Status > Targets**. The status of your target endpoint defined in your configuration appears here. ## Add an SDK metrics data source in Grafana Depending on how you use Grafana, you can either install and run it locally, run it as a Docker container, or log in to Grafana Cloud to set up your data sources. If you have installed and are running Grafana locally, go to [http://localhost:3000](http://localhost:3000) and sign in. To add the SDK metrics Prometheus endpoint as a data source, do the following: 1. Go to **Configuration > Data sources**. 2. Select **Add data source > Prometheus**. 3. Enter a name for your SDK metrics data source, such as _Temporal SDK metrics_. 4. In the **HTTP** section, enter your Prometheus endpoint in the URL field. If running Prometheus locally as described in the examples in this article, enter `http://localhost:9090`. 5. For this example, enable **Skip TLS Verify** in the **Auth** section. 6. Click **Save and test** to verify that the data source is working. If you see issues in setting this data source, check whether the endpoints set in your SDKs are showing metrics. If you don't see your SDK metrics at the scrape endpoints defined, check whether your Workers and Workflow Executions are running. If you see metrics on the scrape endpoints, but Prometheus shows your targets are down, then there is an issue with connecting to the targets set in your SDKs. Verify your Prometheus configuration and restart Prometheus. If you're running Grafana as a container, you can set your SDK metrics Prometheus data source in your Grafana configuration. See the example Grafana configuration described in the [Prometheus and Grafana setup for open-source Temporal Service](/self-hosted-guide/monitoring#grafana) article. ## Set up Grafana dashboards To set up SDK metrics dashboards in Grafana, you can use the UI or configure them directly in your Grafana deployment. > **💡 Tip:** > > Temporal provides community-driven [example dashboards for Temporal SDKs](https://github.com/temporalio/dashboards/tree/master/sdk) that you can customize to meet your needs. > To import a dashboard in Grafana: 1. In the navigation bar, select **Dashboards** > **Import dashboard**. 2. You can either copy and paste the JSON from the [Temporal SDK sample dashboards](https://github.com/temporalio/dashboards/tree/master/sdk), or import the JSON files into Grafana. 3. Save the dashboard and review the metrics data in the graphs. To configure dashboards with the UI: 1. Go to **Create > Dashboard** and add an empty panel. 2. On the **Panel configuration** page, in the **Query** tab, select the "Temporal SDK metrics" data source that you configured earlier. 3. Expand the **Metrics browser** and select the metrics you want. A list of Worker performance metrics is described in the [Developer's Guide - Worker performance](/develop/worker-performance). All SDK-related metrics are listed in the [SDK metrics](/references/sdk-metrics) reference. 4. The graph should now display data based on your selected queries. Note that SDK metrics will only show if you have Workflow Execution data and running Workers. If you don't see SDK metrics, run your Worker and Workflow Executions, then monitor the dashboard. --- # Migrate Source: https://docs.temporal.io/cloud/migrate > Migrate Temporal workflows with zero downtime, whether it's from self-hosted Temporal Server to Temporal Cloud, or between different regions or cloud providers within Temporal Cloud. Learn how to migrate your Temporal workflows with zero downtime: - [Automated Migration](/cloud/migrate/automated) - This process enables seamless transitions from self-hosted Temporal instances to Temporal Cloud. - [Manual Migration](/cloud/migrate/manual) - This process enables transitions from self-hosted Temporal instances to Temporal Cloud by updating clients and workflows to utilize new resources within Temporal Cloud. - [Migrate between regions](/cloud/migrate/migrate-within-cloud) - This process allows you to migrate a Temporal Cloud Namespace between regions or providers. - [Estimate Actions](/cloud/migrate/estimate-actions) - This process helps you estimate Actions and Actions per second from a self-hosted Temporal Service before migrating to Temporal Cloud. --- # Automated migration Source: https://docs.temporal.io/cloud/migrate/automated > **Pre-release** > Please contact your Temporal account executive prior to planning your migration project. ## Process Overview Automated migration is designed to provide a zero-downtime, secure means of migrating to Temporal Cloud. This guide outlines the current process for transitioning Workflows from self-hosted Temporal Server to a namespace hosted within Temporal Cloud. ### Namespace migration schedule When planning a migration it is highly recommended to migrate in the order of least-critical to most-critical Namespace. Ideally, the project will begin with Namespaces designated as "testing", where downtime is an acceptable outcome of the testing process. From there, prioritize migrations based on order of least potential impact. ![Temporal automated migration components](/img/cloud/migration/ns_migration_schedule.png) ### Project phases The migration process is separated into several phases, part of which involves coordinating with Temporal to create necessary cloud-side resources. Migration involves the following phases: 1. Prepare - Initial preparation involves collecting and evaluating key data points from the self-hosted clusters. Collected data will be evaluated by Temporal to ensure eligibility for migration. 2. Setup - Once eligibility has been verified, configurations for the self-hosted clusters will be modified to support the migration. All cloud-side components will be provisioned and the self-hosted [S2S Proxy](https://hub.docker.com/r/temporalio/s2s-proxy/tags) will be deployed. 3. Test - Once all required components are in place, the migration process will be validated using a test Namespace. 4. Initiate - At the conclusion of a successful testing process, migration of production Namespaces will begin. 5. Finalize - After the Namespace has been transferred to Temporal Cloud and validated, the migration will be finalized. Please review the [additional notes](#additional-notes) section prior to planning the migration. ## Phase 1: Prepare In preparation for migrating to Temporal Cloud, several data points must be collected and provided to Temporal via support ticket. ### Capture cluster configurations Depending on your server version, use one of the following methods to capture the configuration of each cluster. For server versions **above 1.28.1**, run the following command against each of your clusters: ``` temporal operator cluster describe --address --output json ``` For server versions **1.28.1 and prior**, use one of the following alternate methods: ``` tctl --address admin cluster describe ``` or ``` grpcurl -v -plaintext temporal.server.api.adminservice.v1.AdminService.DescribeCluster ``` ### Capture custom search attributes Temporal must ensure that your custom Search Attributes are compatible with Temporal Cloud. Capture any custom Search Attributes using one of the following commands: For Elasticsearch/OpenSearch ``` temporal operator search-attribute list ``` For SQL ``` temporal operator search-attribute list --namespace="your_namespace" ``` ### Capture Namespace metrics Metrics are used for cloud-side resource planning. For each Namespace, collect the following: - Total number of open/closed Workflows - Total storage used - Current retention policy. Note that this may differ from the [default retention policy](/cloud/limits#default-retention-period) in Temporal Cloud. - Peak [action per second](/glossary#actions-per-second-aps) (APS). For instructions on collecting these metrics, see [Estimate Actions for migration](/cloud/migrate/estimate-actions). When planning an automated migration, share the following estimates with your Temporal account team: - Peak APS for each Namespace. - Average APS for each Namespace, if available. - Fixed-range Action counts, such as total Actions over the last 30 days. - Representative Workflow Types and their estimated Actions per Workflow Execution. - Retention Periods and storage estimates. These estimates help Temporal plan capacity, Namespace limits, and migration timing. ### Capture schedules Capture a list of schedules on the system. This helps to ensure backwards compatibility. ``` temporal schedule list ``` ### Prepare mTLS certificates mTLS is used to secure the [S2S Proxy](#s2s-proxy-configuration) communications channel. Provide a single certificate using the process outlined [here](/cloud/certificates#issue-certificates). Verify your end-entity certificate with the following command. ``` openssl verify -CAfile ca.pem client-cert.pem ``` In the example above, the ca.pem file will be provided to Temporal where the client-cert.pem will be used by your S2S Proxy. ### Create Cloud Namespaces 1. Create your cloud-side Namespaces noting current [naming requirements](/cloud/namespaces#temporal-cloud-namespace-name). 2. Apply any required custom Search Attributes and adjust the [rate limits](/cloud/capacity-modes) as needed. Migration cannot proceed into a Namespace that is already in use. Please ensure that these Namespaces remain empty (no Workflows). If you are new to Temporal Cloud, consider your connectivity path to cloud. You may connect over the public internet or via [private connectivity](/cloud/connectivity). ### Report collected data Provide all collected data to Temporal via a [support ticket](/cloud/support#ticketing). In your ticket, please provide: * mTLS certificate - base64 encoded pem file works well for easy submission in the ticket * cluster configurations - json output for each cluster * custom search attributes - CLI output for each cluster/Namespace * list of schedules used * Namespace metrics * cluster/Namespace mappings - use CSV format (see below) Sample CSV file for reporting cluster/Namespace mappings. Use a separate file for each Temporal Cloud account. ``` cluster_name, cloud_region, source_namespace, cloud_namespace cluster1, us-east-1, default, use1.nnnnn cluster1, us-east-1, ns2, use2.nnnnn cluster2, us-central1, default, usc1.nnnnn ``` ## Phase 2: Setup Once the migration has been approved, the next step is to prepare both the self-hosted clusters and Temporal Cloud resources for the migration. > **⚠️ Warning:** > > Proceed only when your request has been approved by Temporal. > ### S2S Proxy configuration The [S2S Proxy](https://github.com/temporalio/s2s-proxy) requires a cloud-side inbound endpoint. Proceed with deployment only after receiving the endpoint from Temporal. The proxy provides API forwarding over a secure 2-way tunnel to Temporal Cloud. The self-hosted proxy will initiate an outbound connection (TCP 8233) to its cloud-side counterpart and establish the 2-way tunnel. If there are firewalls in-path, ensure that they permit this outbound connection. ![Temporal automated migration components](/img/cloud/migration/auto-migration-components.png) Use the following procedure to deploy the proxy: 1. Obtain the latest Docker image from the [temporalio/s2s-proxy repository](https://hub.docker.com/r/temporalio/s2s-proxy/tags). 2. Gather the mTLS certs generated in the previous step. 3. Deploy **3 replicas** of the s2s-proxy (minimum 4 CPU and 512mb memory). For Kubernetes users, use this [helm chart example](https://github.com/temporalio/s2s-proxy/blob/main/charts/s2s-proxy/README.md) as a reference. See the [example](https://github.com/temporalio/s2s-proxy/blob/main/charts/s2s-proxy/example.yaml) configuration file as a reference. Note that the number of replicas must match on both sides of the connection. If your replica count differs from 3, update Temporal with the actual number of replicas so that the same count can be configured on the cloud side. 4. Verify that you have included your self-hosted namespaces and their translations in your proxy configuration. Example: ```yaml aclPolicy:   allowedNamespaces:    - temporal-system # required - do not remove    - namespace1    - namespace2    - etc... namespaceTranslation:   mappings:   - local: "namespace1"     remote: "cloud-namespace1.wxyz"   - local: "namespace2"     remote: "cloud-namespace2.wxyz"   - etc... ``` 5. Test access using the command below. It should display the information of the cloud-side migration server. ``` temporal operator cluster describe --address {the-outbound-external-address-of-your-proxy} ``` There are multiple metrics available on the S2S proxy (prometheus endpoint: _proxy-pod-ip_:9090/metrics). These are helpful for monitoring the overall health of the proxy. In particular, the metric `temporal_s2s_proxy_mux_connection_active` will monitor connectivity to the cloud-side proxy. ### Modify cluster configuration > **⚠️ Warning:** > > Coordinate closely with Temporal before completing this process. > The [dynamic configuration](/references/dynamic-configuration) of your self-hosted cluster must be modified to facilitate the migration. Complete the following process. 1. Adjust the maximum connection keepalive time to match the setting in cloud. ```yaml frontend.keepAliveMaxConnectionAge: - value: '2h' ``` 2. For server versions 1.22.6 - 1.23.x, apply this [extra required setting](#special-dynamic-configuration-for-version-122---123). ```yaml history.enableReplicationStream: - value: true ``` 3. If [Global Namespace](/global-namespace) **is already enabled**, then skip to step 5. 4. If [Global Namespace](/global-namespace) **is not enabled**, then enable it and set _failoverVersionIncrement_ and _initialFailoverVersion_ to the values provided by Temporal. Pay close attention when setting these values. They **cannot be changed** once Global Namespace has been enabled. See the sample configuration below for a reference. ```yaml dcRedirectionPolicy: policy: 'all-apis-forwarding' clusterMetadata: enableGlobalNamespace: true # add this failoverVersionIncrement: CHANGEME # use value provided by Temporal masterClusterName: _NO_CHANGE_ currentClusterName: _NO_CHANGE_ clusterInformation: _NO_CHANGE_: enabled: true initialFailoverVersion: CHANGEME # use value provided by Temporal rpcName: _NO_CHANGE_ rpcAddress: _NO_CHANGE_ # for versions 1.22 - 1.23 only #history.enableReplicationStream: # - value: true ``` 5. Verify your configuration and restart all Temporal services (frontend, history, matching, worker), starting with the frontend. 6. After all services have restarted, verify the configuration using: ``` temporal operator cluster describe ``` The following sample output is expected: ```yaml "failoverVersionIncrement": "nnn", "initialFailoverVersion": "nnn" "isGlobalNamespaceEnabled": true ``` ### Verify current cluster utilization It is important to ensure that your production cluster has been allocated enough resources to support the migration. In particular, it is important to verify that your persistence/database layer has plenty of CPU and I/O capacity. ## Phase 3: Testing Once the proxy is deployed and the cluster configuration changes have been applied, then final testing may begin. ### Verifying the proxy Temporal will validate the proxy from the cloud side of the connection. Temporal will validate 1. the stability of the proxy connection 2. that all required permissions are allowed on the self-hosted proxy 3. that all cluster configuration changes have been applied 4. that all to-be-migrated Namespaces are visible 5. the presence of Custom Search Attributes Once the self-hosted setup has been verified, the next step will be to perform test migrations. ### Complete test migrations Migration testing should use either a newly created Namespace or else one that is considered to be non-production. It is ideal to have a mix of completed and running Workflows to use during testing. Testing uses the following process: 1. Create or identify a non-production Namespace that can tolerate data loss in the event of issues. 2. Create target cloud-side Namespace and add the Namespace definition to the S2S Proxy configuration. 3. Run test Workflows against the Namespace. 4. Perform a complete end-to-end migration for the Namespace (see remaining phases for full process). 5. Optionally, test [transferring clients](#transfer-clients-to-cloud) to the cloud. Testing is considered successful if all data from the self-hosted deployment is migrated to cloud. ## Phase 4: Initiate Review the section on [transferring clients](#transfer-clients-to-cloud) before proceeding. The sections below outline the process for initiating the migration. ### Migration start Temporal will generate the endpoint-id and initiate the migration. During this process, the self-hosted Namespace remains active while the cloud Namespace becomes passive. Workflows are replicated from the self-hosted Namespace to the cloud Namespace. Once the cloud Namespace has fully synced with self-hosted Namespace, migration is ready for handover. > **📝 Note:** > Billing > > Billing for the cloud Namespace does not begin until the migration is [confirmed](#confirm-complete). > The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-start-a-migration) is used to start the migration: ``` tcld migration start --endpoint-id --source-namespace --target-namespace ``` ### Monitor During the initial sync, it is important to monitor the overall process to ensure progress is being made. While Temporal will monitor from the cloud-side, progress may also be monitored from the self-hosted side using the _replication_stream_stuck_ metric. The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-get-a-migration) may also be used to monitor the migration progress: ``` tcld migration get --id ``` ### Handover-to-cloud Once the sync process has completed, Temporal will flip the roles of the self-hosted and cloud Namespace. At this point, the cloud becomes active and the self-hosted Namespace becomes passive. Workflows are then replicated from the cloud to the self-hosted server. The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-perform-handover-during-a-migration) is used to trigger the handover: ``` tcld migration handover --id --to-replica-id ``` When using this command, replace `` with `cloud` when handing over to Temporal Cloud. Replace `` with `on-prem` when handing back to the self-hosted setup. ## Phase 5: Finalize If you have not done so already, complete the process of [transferring clients](#transfer-clients-to-cloud) to the cloud Namespace. ### Final validation Use the following checklist prior to finalizing the migration: - Confirm that workers can access Namespaces. Either via public internet or [private connectivity](/cloud/connectivity). - Understand how to access metrics for your Namespace on Temporal Cloud. - Monitor general Workflow metrics (schedule to start latency, start v.s. completion rate, sync match rate, etc). - Learn how [capacity management](/cloud/capacity-modes) works in Temporal Cloud. - Plan for a worker tuning session - performance change between Temporal Cloud v.s. self-hosted cluster, which could lead to unexpected symptoms and optimizations. - Know how to reach out to your Temporal Solutions Architect (SA) and Account Executive (AE) for assistance. ### Confirm complete Once a Namespace has been transferred to the cloud and validated, the migration will be completed. Note that this step is final and may not be undone. Once performed, Workflow replication from the cloud Namespace to the self-hosted server is halted. The following [command](https://pkg.go.dev/github.com/temporalio/tcld#readme-confirm-a-migration) is used to confirm the migration: ``` tcld migration confirm --id ``` or to [abort](https://pkg.go.dev/github.com/temporalio/tcld#readme-abort-a-migration) and roll-back changes without impacting your Workflows, if needed: ``` tcld migration abort --id ``` ## Transfer clients to cloud There are two options for switching Temporal clients to the cloud. ### Option 1 (recommended) Deploy two sets of Temporal clients: one pointing to your Temporal server and one to the Cloud Namespace endpoint. This is the recommended option since your Workflows will continue to make progress during the handover, even if your cloud Temporal client is unable to access the cloud (due to misconfiguration, for example). The process is as follows: 1. Direct your cloud-based Temporal clients to the cloud Namespace endpoint. Initially, these clients will connect and send Poll requests but will not receive any tasks. 2. Start migration. Your self-hosted Namespace is active while your cloud Namespace is passive (or standby). Your cloud Temporal clients can begin receiving tasks, but all requests from cloud clients to the Cloud Namespace will automatically forward from the cloud to your self-hosted server. 3. Hand over Namespace to the cloud. Your cloud Namespace becomes active and your self-hosted Namespace becomes passive. All requests from your self-hosted Temporal clients will automatically forward from your server to the cloud. 4. Complete migration. Your self-hosted Temporal clients will no longer receive any tasks from your server, allowing you to stop these clients. ### Option 2 Deploy one set of Temporal clients and switch the Namespace endpoint during migration. With this option, if your workers are misconfigured during the switch, then it is possible that Workflows can stop making progress. It is important to ensure that all workers maintain connectivity to cloud to avoid this scenario. The process is as follows: 1. Start migration. 2. Switch your Temporal clients to point to the cloud Namespace endpoint. Requests from your Temporal clients will automatically forward from the cloud to your server. Alternatively, you may switch Temporal clients to the cloud Namespace endpoint after handover. 3. Hand over Namespace to the cloud. Requests from your Temporal clients will now be served by the cloud and will not be forwarded to your server. 4. Confirm migration completion. ## Additional notes ### Limitations The following are known limitations. - OSS server versions 1.22.6 or newer are required. Refer to the [upgrade](/self-hosted-guide/upgrade-server#upgrade-server) procedure as needed. - History shard counts must be a power of two (for example, 512 or 1024). - If you have multiple self-hosted servers and they are all configured with the same cluster name (by default Temporal uses 'active' as cluster name), they cannot be connected to a single migration server simultaneously due to cluster name collision. There are 2 available options: 1. Migrate one server at a time using a single migration server. 2. Create multiple migration servers (one for each self-hosted server) if you need to migrate all servers simultaneously. - If you are using multi-cluster replication in your self-hosted setup and have previously failed over Namespaces, then this may impact your eligibility for automated migration. Specifically, whenever Global Namespace has been previously enabled the following restrictions apply: 1. Initial Failover Version must be less than or equal to 1,000,000 2. Failover Version Increment must be a divisor of 1,000,000 (for example, 10) - OSS supports cross-Namespace commands (for example, parent-child, SignalExternal, CancelExternal) through the `system.enableCrossNamespaceCommands` configuration. This configuration is disabled on Temporal Cloud. The `system.enableCrossNamespaceCommands` configuration must be disabled, and code using cross-Namespace calls must be updated or removed prior to migration. ### Special dynamic configuration for Version 1.22 - 1.23 Temporal versions 1.22 and 1.23 include support for stream-based replication, but it is disabled by default. Since those releases, stream-based replication has been validated as more reliable than the poll-based replication that remained the default in 1.22 and 1.23. When preparing for an S2C migration on these versions, configure the following dynamic settings to enable stream-based replication: ```yaml history.enableReplicationStream: - value: true ``` Enabling this configuration will require a restart of your history pods. ## Frequently asked questions ### How many Temporal Cloud accounts do I need? If you are new to Temporal Cloud, then the most common recommendation is to create a single Account that contains your Namespaces. ### When should I opt for auto migration? The answer depends on your specific situation. However, automated migration is most likely to help if any of the following apply to you: * When safe transfer of workers is a concern - automated migration allows workers to run against both self-hosted and cloud environments simultaneously, allowing a gradual and lower-risk transition. * When there are a high number of long-running Workflows. * When there is a need to migrate closed Workflows. * When there is a need to migrate Schedules. In contrast, automated migration may not be the best solution if your self-hosted clusters do not meet the [minimum requirements](#limitations). ### Can I split Workflows from a single source Namespace into multiple cloud-side Namespaces? No. All Workflows will be migrated. ### Can I combine manual and auto migration? No. Auto migration requires a "fresh" target cloud-side Namespace (one that has never had a running Workflow). If Workflows were manually migrated to a cloud-side Namespace, then this Namespace would not be suitable as an auto-migration target. ### Why does it matter if custom search attributes are used? Custom search attributes must be mapped to a Namespace in Temporal Cloud. They matter because configurations in a self-hosted environment may not be directly compatible with Temporal Cloud, potentially requiring additional migration work. The exact process can also differ depending on the type of visibility data store used. ### What Workflows are migrated by default? All Workflows are migrated by default. For closed Workflows, you may specify a date range to be migrated. Your cloud-side Namespace must be configured with your desired retention period prior to starting the migration. ### What can I do to speed up an automated migration? The #1 speed optimization is to limit the time range for closed Workflows. This will reduce the amount of data required to be migrated and in many cases will dramatically reduce overall migration time. ### Is the migration of Schedules supported? Yes. Under the hood, Schedules are essentially Workflows. ### I have a long retention period for my Workflows. Is this compatible with Temporal Cloud? Occasionally, self-hosted [retention periods](/temporal-service/temporal-server#retention-period) are in excess of what is [supported](/cloud/limits#default-retention-period) in Temporal Cloud. In these cases it is recommended to utilize [archival](/temporal-service/archival) to store closed Workflows that cannot be migrated. In general, archival is recommended over large retention periods since the extra data can stress the persistence layer of the system. ### I am using payload encryption in my self-hosted Temporal cluster. Is this supported in cloud? Yes. If payloads are already [encrypted](/payload-codec#encryption) in your self-hosted server via data converter, then they will remain encrypted during and after migration. ### I would like to enable payload encryption as part of the migration. Is this supported? The automated migration tooling cannot add payload encryption. To encrypt payloads sent to Temporal Cloud, you must encrypt payloads in your cluster before starting the automated migration process. --- # Estimate Actions for migration Source: https://docs.temporal.io/cloud/migrate/estimate-actions > Estimate Temporal Cloud Actions and Actions per second from a self-hosted Temporal Service before migrating to Temporal Cloud. Before you migrate from a self-hosted Temporal Service to Temporal Cloud, you can measure your current Action usage to help predict your Cloud usage and costs. Use server metrics to estimate Actions per second (APS) and peak load. Use representative Workflow Execution Histories to understand how many Actions a typical Workflow Execution creates. Estimate storage separately from Actions. For details about which operations count as Actions in Temporal Cloud, see [Temporal Cloud Actions](/cloud/actions). ## Choose an estimation method Use one or more of the following methods depending on what you need to estimate. | Method | Use it to estimate | Notes | | --- | --- | --- | | Self-hosted server metrics | APS, peak APS, and Action counts over a fixed time range | Best for sizing Namespace limits and understanding usage patterns. | | Workflow Execution History samples | Actions per representative Workflow Execution | Best for understanding workload shape by Workflow Type. Some [Actions do not appear](/cloud/actions) in Event History. | | Storage estimates | Event History storage | Storage is [priced separately](/cloud/pricing#pricing-model) from Actions. | ## Estimate APS from self-hosted server metrics Temporal Server versions later than 1.17 provide an `action` metric. Use this metric to estimate Actions per second from a self-hosted Temporal Service. Temporal Server versions 1.22.3 and later provide an `action` metric that more closely reflects current Temporal Cloud Action pricing, including Local Activity metering. For Temporal Server versions from 1.17 through 1.22.2, use the `action` metric to understand server load, but do not treat it as a precise billable Action estimate. To calculate total APS, use the following PromQL query: ```promql sum(rate(action{service_name="frontend"}[1m])) ``` To calculate APS by Namespace, use the following PromQL query: ```promql sum(rate(action{service_name="frontend"}[1m])) by (exported_namespace) ``` Depending on your metrics exporter setup, the Namespace label might be `namespace` instead of `exported_namespace`. For a Grafana dashboard example, see the [`server-general.json`](https://github.com/temporalio/dashboards/blob/master/server/server-general.json) dashboard in the Temporal dashboards repository. For Datadog, use a query like the following to calculate Actions per second: ```text sum:io.temporal.server.action.count{$server-name}.as_rate() ``` ## Estimate Action counts over a fixed time range To estimate total Actions during a fixed time range, use `increase()` over the range you want to measure. For example, to estimate total Actions over 30 days, use the following PromQL query: ```promql sum(increase(action{service_name="frontend"}[30d])) ``` To estimate Actions for one Namespace, add the Namespace label: ```promql sum(increase(action{service_name="frontend", exported_namespace="default"}[30d])) ``` When you run fixed-range queries in Grafana, set the end of the dashboard time window to the end of the date range that you want to measure. For example, to measure Actions for the 30-day period ending March 31, set the dashboard end time to March 31. ## Estimate Actions from Workflow Execution Histories You can estimate Actions per Workflow Execution by counting billable events in representative Workflow Execution Histories. This method is useful when you need to understand how much Action usage a Workflow Type creates. 1. Choose representative Workflow Executions for each Workflow Type. 2. Download the Event History for each Workflow Execution from the Web UI or API. 3. Count the events that map to Temporal Cloud Actions. For the list of Action types and corresponding History Event types, see [Temporal Cloud Actions](/cloud/actions). 4. Multiply the Action count by the expected number of Workflow Executions in the period you want to estimate. This method produces an estimate. Some Actions do not appear directly in Event History, including Queries and some Activity Heartbeat Actions. If you use Global Namespaces, account for the additional Action and storage cost. Estimate storage separately from Workflow Action counts. ## Estimate storage separately Temporal Cloud storage is priced separately from Actions. To estimate storage, collect the following information: - Event History size for representative Workflow Executions. - Retention Period for closed Workflow Executions. - Duration for open Workflow Executions. - Expected number of Workflow Executions. For current storage pricing, see [Temporal Cloud pricing](/cloud/pricing). --- # Manual migration Source: https://docs.temporal.io/cloud/migrate/manual > Migrating to Temporal Cloud from self-hosted Temporal Service varies by Workflow requirements. This guide covers changing Client code, Workflow migration strategies, and necessary code adjustments. Migrating to Temporal Cloud from a self-hosted Temporal Service will have different requirements depending on your usage. This guide provides some guidance based on our experience helping customers of all sizes successfully migrate. ### What to expect from a migration Depending on your Workflows' requirements, the migration process may be as simple as changing a few parameters, or may require more extensive code changes. There are two aspects to consider when migrating: your Temporal Client connection code and your Workflow Executions. Here's a high-level overview of what you can expect: - **Introduce another Temporal Client to your Starter and Worker Processes:** Configure and deploy a new Temporal Client so that Temporal Cloud becomes responsible for new Workflow Executions. - **Migrate Workflow Executions:** There are different approaches for new, running, and completed Workflow Executions. - **New Workflow Executions:** When you no longer need to send Signals or Queries to your self-hosted Temporal Service, you can deprecate your old Client code. Until then, your self-hosted Temporal Service can receive relevant traffic, while new Workflow Executions are sent to Temporal Cloud. - **Running Workflow Executions:** Short-running Workflows can often be drained and then started again on Temporal Cloud. Long-running Workflows that cannot be drained might require you to implement more code changes to pass the state of the currently running Workflow to Temporal Cloud. - **Completed Workflow Executions:** Completed Workflow Execution History cannot be automatically migrated to Temporal Cloud. Refer to [Multi-Cluster Replication](#multi-cluster-replication) for more information. ### Updating Client connection code in your Workers Whether you're self-hosting Temporal or using Temporal Cloud, you manage runtime of your code. To migrate your Workflows to Temporal Cloud, you need to change some parameters in the Client connection code, such as updating the namespace and gRPC endpoint. The changes needed to direct your Workflow to your Temporal Cloud Namespace are only a few lines of code, including: - Add your [SSL certificate and private key](/cloud/certificates) or [API key](/cloud/api-keys) associated with your Namespace. - [Copy the Cloud-hosted endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint) from the Namespace detail Web page. The endpoint uses this format: `..tmprl.cloud:port`. - [Connect to Temporal Cloud](/cloud/get-started) with your Client. - [Configure tcld, the Cloud CLI](/cloud/tcld), with the same address, Namespace, and certificate used to create a Client through code. ### Migrating your Workflow Executions A Temporal Service stores the complete Event History for the entire lifecycle of a Workflow Execution. To migrate from a self-hosted Temporal Service to Temporal Cloud, take into account the current state, Event History, and any future expectations of your Workflow Executions. **New Workflows are automatically executed on Temporal Cloud.** Once you've made the code changes in Step 1, and your new code is deployed, new Workflow Executions will be sent to Temporal Cloud. Existing Workflows must receive Signals to migrate and re-execute on Cloud. If you maintain your self-hosted instance, you will still be able to use it to access any execution history from before your migration. You can also export JSON from your previous execution history, that you can then import into your own analytics system. **Running Workflows can either be drained or migrated.** If your Workflow can be completed before any compelling event which drives a move to Temporal Cloud, those Workflows can be automatically restarted on Temporal Cloud. If your Workflows need to run continuously, you must migrate Workflows while they are running. To accomplish this migration, cancel your current Workflow and pass the current state to a new Workflow in Temporal Cloud. Refer to [this repository](https://github.com/temporalio/temporal-migration) for an example of migrating running Workflows in Java. When performing a live migration, make sure your Worker capacity can support the migration load. Both a [Signal](/sending-messages#sending-signals) and a [Query](/sending-messages#sending-queries) will be executed during the course of the migration. Also, the Query API loads the entire history of Workflows into Workers to compute the result (if they are not already cached). That means that your self-hosted Temporal Service Worker capacity will need to support having those executions in memory to serve those requests. The volume of these requests might be high to execute against all the matches to a `ListFilter`. ### Considerations when resuming Workflows on a new Temporal Service or Namespace - **Skipping Steps:** If your Workflow steps cannot guarantee idempotency, determine whether you need to skip those steps when resuming the execution in the target Namespace. - **Elapsed Time:** If your Workflow is “resuming sleep” when in the target Namespace, determine how you will calculate the delta for the sleep invocation in the new execution. - **Child Relationships:** If your Workflow has Child Workflow relationships (other than Detached Parent Close Policy children), determine how you can pass the state of those children into the parent to execute the child in a resumed state. - **Heartbeat state:** If you have long running activities relying on heartbeat state, determine how you can resume these activities in the target Namespace. - Child Workflows with the same type as their Parent types are returned in List Filters used to gather relevant executions. Unless these are Detached `ParentClosePolicy` children, this is not what you want since the Parent/Child relationship will not be carried over to the target Namespace. - Long running activities that use heartbeat details will not receive the latest details in the target Namespace. - Duration between Awaitables inside a Workflow definition needs to be considered for elapsed time accuracy when resuming in the target Namespace. - When Signaling directly from one Workflow to another, make sure to handle `NotFound` executions in the target Namespace. The Workflows may resume out of order. ### Other considerations when migrating - Have you added an mTLS certificate to your Temporal Namespace? Review our [documentation for adding a certificate to your Temporal Cloud account](/cloud/certificates) for more information. - There are differences in how metrics are generated in self-hosted Temporal and Temporal Cloud. Review the [documentation on Temporal Cloud metrics](/cloud/metrics/) for more information. - Consider the implications for [security and access to your Temporal Service](/cloud/security). - Review your current load with [Action estimates](/cloud/migrate/estimate-actions) and speak to your Account Executive and Solutions Architect so we can set appropriate [Namespace limits](/cloud/limits). ### Multi-Cluster Replication [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) is an experimental feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters for backup and state reconstruction. Migrating Execution History from a self-hosted Temporal Service to Temporal Cloud is not currently supported. However, a migration tool based on Multi-Cluster Replication, which will enable this, is currently in development for Temporal Cloud. If you have used this feature locally or you are interested in using it to migrate to Temporal Cloud, [create a support ticket](/cloud/support) or watch this space for more information about public availability. --- # Migrate between regions Source: https://docs.temporal.io/cloud/migrate/migrate-within-cloud > Add a Namespace replica in a new region, wait for it to activate, then fail over for a zero-downtime Temporal Cloud migration. Temporal Cloud's [High Availability features](/cloud/high-availability) allow you to migrate a Temporal Cloud Namespace from one region or cloud provider to another with zero downtime. ## Preparing to migrate Namespaces using Export will need to stop Export and migrate the region configuration to the new region for Export jobs to continue after migration. See [failover scenarios](/cloud/export#failover-scenarios) for details. [Using High Availability features affects pricing](/cloud/pricing#high-availability-features). > **ℹ️ Info:** > Using AWS PrivateLink or GCP Private Service Connect? > > If the Namespace uses Private Connectivity, the steps below need additional DNS and VPC Endpoint work. Follow [How to migrate to another Temporal Cloud Region when using Private Connectivity](/cloud/high-availability/ha-connectivity#how-to-migrate-regions-with-private-connectivity) instead of (or alongside) the public steps below. > ## Steps to migrate 1. Add a Namespace replica in the region you want to migrate to. See [regions](/cloud/regions) for a list of available regions and supported multi-region and multi-cloud configurations. ![Add a namespace replica](/img/cloud/high-availability/migrate/1-add-replica.png) ![Choose the region for the replica](/img/cloud/high-availability/migrate/2-choose-region.png) 2. Wait for the replica to become active. The Cloud UI will display a time estimate, and namespace admins will receive an email when the replica is active. 3. If your Workers are using [Regional Endpoints](/cloud/namespaces#access-namespaces), ensure they are updated to use the Regional Endpoint of the [replica's region](/cloud/regions). 4. Trigger a failover to the new region. ![Initiate failover to the new region](/img/cloud/high-availability/migrate/3-failover.png) 5. Remove the Namespace replica in the region you are migrating from. ![Remove the replica for the original region](/img/cloud/high-availability/migrate/4-remove-replica.png) > **📝 Note:** > All replica changes are subject to a [cooldown period](/cloud/high-availability/enable#changing) before further replica changes can be made. > --- # Monitor Temporal Cloud Source: https://docs.temporal.io/cloud/monitor > Detect Task Queue backlogs, Worker capacity issues, and Temporal Cloud service errors, then route alerts to your monitoring tools. Monitor Temporal Cloud and the Workers connected to it to detect service issues, Task Queue backlogs, and Worker capacity problems. Temporal provides Cloud metrics, SDK metrics, health-monitoring guidance, and notifications. Use the following guides based on what you need to detect. ## Choose what to monitor | Goal | Guide | What it helps you detect | | --- | --- | --- | | Collect metrics from Temporal Cloud and your Workers | [Set up metrics](/cloud/metrics) | Service, Task Queue, Worker, and Client behavior | | Monitor Temporal Cloud service health | [Monitor service health](/cloud/service-health) | Service latency and errors, Workflow and Activity failure trends, throttling, and capacity limits | | Monitor Workers connected to Temporal Cloud | [Monitor Worker health](/cloud/worker-health) | Task backlogs, insufficient or excess Worker capacity, configuration problems, and Worker availability | | Receive operational updates | [Receive notifications](/cloud/notifications) | Temporal Cloud incidents, expiring credentials, billing events, and completed or failed failovers | ## Understand metric sources Temporal provides two complementary metric sources: - **Cloud metrics** show what the Temporal Service observes, including request latency, service errors, Task Queue behavior, throttling, and Namespace-level activity. - **SDK metrics** show what your Workers and Clients observe, including Schedule-To-Start latency, available Worker slots, cache behavior, and request failures. Use both sources to distinguish a Temporal Cloud service issue from a Worker or application issue. To begin collecting metrics: 1. [Set up Cloud metrics with OpenMetrics](/cloud/metrics/openmetrics). 2. [Connect Temporal Cloud to your observability tool](/cloud/metrics/openmetrics/metrics-integrations). 3. [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup). 4. Use the [service health](/cloud/service-health) and [Worker health](/cloud/worker-health) guides to create monitors and alerts. ## Monitor Nexus To monitor Nexus metrics and debug Nexus Operations, see [Temporal Nexus observability](/cloud/nexus/observability). For records of Control Plane changes, see [Audit Logs](/cloud/audit-logs). Audit Logs answer who changed a Temporal Cloud resource, what they changed, and when. They are separate from runtime health monitoring. --- # Namespaces Source: https://docs.temporal.io/cloud/namespaces > **ℹ️ Info:** > Temporal Cloud > This page covers namespace operations in **Temporal Cloud**. > For core namespace concepts, see [Temporal Namespace](/namespaces). > For open source Temporal, see [Managing Namespaces](/self-hosted-guide/namespaces). A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique identifiers, and gRPC endpoints in Temporal Cloud. - [Create a Namespace](#create-a-namespace) - [Access a Namespace](#access-namespaces) - [Manage Namespaces](#manage-namespaces) - [Delete a Namespace](#delete-a-namespace) - [Tag a Namespace](#tag-a-namespace) ## What is a Cloud Namespace Name? A Cloud Namespace Name is a customer-supplied name for a [Namespace](/namespaces) in Temporal Cloud. Each Namespace Name, such as `accounting-production`, is unique within the scope of a customer's account. It cannot be changed after the Namespace is provisioned. Each Namespace Name must conform to the following rules: - A Namespace Name must contain at least 2 characters and no more than 39 characters. - A Namespace Name must begin with a letter, end with a letter or number, and contain only letters, numbers, and the hyphen (-) character. - All letters in a Namespace Name must be lowercase. > **📝 Note:** > The Namespace Name alone is not sufficient to connect to or reference a Namespace. All Temporal Cloud operations — SDK connections, CLI commands, and API calls — require the full **Namespace ID**, which combines your Namespace Name with your Account ID: > > `.` > > For example: `accounting-production.123de` > > The full Namespace ID is shown in the Namespace list in the Temporal Cloud UI and in the output of the `namespace list` command. See [Cloud Namespace ID](#temporal-cloud-namespace-id) for details. ## What is a Temporal Cloud Account ID? A Temporal Cloud Account ID is a unique customer identifier assigned by Temporal Technologies. Each Id is a short string of numbers and letters like `f45a2`, at least five characters long. This account identifier is part of every Namespace ID in your account and is retained while you use Temporal Cloud. You can retrieve your Account ID from the [Temporal Cloud](https://cloud.temporal.io) Web UI or from the command line. Follow these steps. **Web UI** Follow these steps to retrieve your Account ID: 1. Log into Temporal Cloud. 1. Select your account avatar at the top right of the page. A profile dropdown menu appears. 1. Copy the Cloud Account ID from the menu. ![Temporal Cloud user profile dropdown menu. The dropdown includes a Cloud Account ID and copy button, with role, profile and logout options, and links to community resources.](/img/cloud/cloud-guide/cloud-account-id.png) In this example, the Account ID is `123de`. **Temporal CLI** 1. Use the Temporal Cloud extension to log into an account. ``` temporal cloud login ``` Complete the interactive login in the browser window that opens. 1. Return to the command line and list your Namespaces. ``` temporal cloud namespace list ``` Each Namespace uses an Account ID suffix, the same as in the Temporal Cloud Web UI Namespaces list. **tcld** 1. Use the `tcld` utility to log into an account. ``` tcld login ``` The `tcld` output presents a URL with an activation code at the end. Take note of this code. The utility blocks until the login/activation process completes. ``` Login via this url: https://login.tmprl.cloud/activate?user_code=KTGC-ZPWQ ``` A Web page automatically opens for authentication in your default browser. 1. Visit the browser. Ensure the user code shown by the CLI utility matches the code shown in the Web browser. Then, click Confirm in the browser to continue. After confirmation, Web feedback lets you know that the CLI "device" is now connected. 1. Return to the command line. Issue the following command. ``` tcld namespace list ``` The CLI tool returns a short JSON packet with your namespace information. This is the same list found in the Temporal Cloud Web UI Namespaces list. Like the browser version, each Namespace uses an Account ID suffix. ``` { "namespaces": [ "your-namespace.123de", "another-namespace.123de" ], "nextPageToken": "" } ``` Each Namespace automatically appends an Account ID suffix to its customer-supplied identifier. This five-character-or-longer string appears after the name, separated by a period. In this Namespace listing sample, the Account ID is 123de. ## What is a Cloud Namespace Id? A Cloud Namespace Id is a globally unique identifier for a [Namespace](/namespaces) in Temporal Cloud. A Namespace Id is formed by concatenating the following: 1. A [Namespace Name](#temporal-cloud-namespace-name) 1. A period (.) 1. The [Account ID](#temporal-cloud-account-id) to which the Namespace belongs For example, for the Account ID `123de` and Namespace Name `accounting-production`, the Namespace Id is `accounting-production.123de`. ## What is a Cloud gRPC Endpoint? Temporal Clients communicate between application code and a Temporal Server by sending and receiving messages via the gRPC protocol. gRPC is a Remote Procedure Call framework featuring low latency and high performance. gRPC provides Temporal with an efficient, language-agnostic communication framework. Every Temporal Namespace uses a gRPC endpoint for communication. When migrating to Temporal Cloud, you'll need to switch the gRPC endpoint in your code from your current hosting, whether self-hosted or locally-hosted, to Temporal Cloud. A gRPC endpoint appears on the detail page for each Cloud Namespace. Follow these steps to find it: 1. Log into your account on [cloud.temporal.io](https://cloud.temporal.io/namespaces). 2. Navigate to the Namespace list page from the left-side vertical navigation. 3. Tap or click on the Namespace Name to select and open the page for the Namespace whose endpoint you want to retrieve. 4. On the Namespace detail page, click on the "Connect" button in the top right corner of the page. 5. Click the copy icon next to the gRPC address to copy it to your clipboard. See [How to access a Namespace in Temporal Cloud](/cloud/namespaces/#access-namespaces) for more information on different gRPC endpoint types and how to access them. ## How to create a Namespace in Temporal Cloud > **ℹ️ Info:** > > The user who creates a [Namespace](/namespaces) is automatically granted > [Namespace Admin](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) permission for that Namespace. > > To create a Namespace, a user must have the Developer, Account Owner, or Global Admin account-level > [Role](/cloud/manage-access/roles-and-permissions#account-level-roles). > > **💡 Tip:** > > By default, each account starts with 10 Namespaces. This limit increases automatically when your existing Namespaces have scheduled or running Workflow Executions. For large-scale needs, open a [support ticket](/cloud/support#support-ticket). > ### Information needed to create a Namespace To create a Namespace in Temporal Cloud, gather the following information: - [Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name), region, and Cloud Provider - [Retention Period](/temporal-service/temporal-server#retention-period) for the [Event History](/workflow-execution/event#event-history) of closed [Workflow Executions](/workflow-execution). - [CA certificate](/cloud/certificates#certificate-requirements) for the Namespace, if you are using mTLS authentication. - [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) to show decoded payloads to users in the Event History for Workflow Executions in the Namespace. For details, see [Securing your data](/production-deployment/data-encryption). - [Permissions](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) for each user. **Web UI** ### Create a Namespace using Temporal Cloud UI 1. Gather the information listed earlier in [Information needed to create a Namespace](#information-needed-to-create-a-namespace). 1. Go to the Temporal Cloud UI and log in. 1. On the left side of the window, click **Namespaces**. 1. On the **Namespaces** page, click **Create Namespace** in the upper-right portion of the window. 1. On the **Create Namespace** page in **Name**, enter the Namespace Name. 1. In **Cloud Provider**, select the cloud provider in which to host this Namespace. 1. In **Region**, select the region in which to host this Namespace. 1. In **Retention Period**, specify a value from 1 to 90 days. When choosing this value, consider your needs for Event History versus the cost of maintaining that Event History. Typically, a development Namespace has a short retention period and a production Namespace has a longer retention period. The retention period of a namespace can be changed in the Temporal Cloud UI under the namespace's Settings tab or by using the Temporal CLI. 1. Select your authentication method: [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates). 1. If using mTLS authentication, paste the CA certificate for this Namespace. 1. Optional: In **Codec Server**, enter the HTTPS URL (including the port number) of your Codec Server endpoint. You may also enable "Pass the user access token with your endpoint" and "Include cross-origin credentials." For details, see [Hosting your Codec Server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli). 1. Click **Create Namespace**. **Temporal CLI** See the [`temporal cloud namespace create`](/cli/command-reference/cloud/namespace#create) command reference for details. **tcld** See the [`tcld` namespace create](/cloud/tcld/namespace/#create) command reference for details. ## What are some Namespace best practices? For guidance on how many Namespaces to create, how to split workloads across services or domains, and when to isolate tenants or teams, see [Namespace best practices](/best-practices/managing-namespace). This page focuses on Temporal Cloud namespace mechanics such as naming rules, provisioning, authentication, tagging, and accessing Namespace endpoints. ## How to access a Namespace in Temporal Cloud Temporal Cloud supports authentication to Namespaces using [API keys](/cloud/api-keys) _or_ [mTLS](/cloud/certificates). To migrate a Namespace from one authentication method to another, or to use both API key and mTLS authentication on the same Namespace, please contact [Support](/cloud/support#support-ticket). > **ℹ️ Info:** > > Using **both** API key and mTLS authentication on the **same** Namespace is in > [pre-release](/evaluate/development-production-features/release-stages) and doesn't support > [High Availability features](/cloud/high-availability) or authenticating with an API Key to a Namespace Endpoint. > Connecting to your Namespace requires a specific endpoint that works for the given Namespace. There are two types of gRPC endpoints for accessing a Namespace in Temporal Cloud: a Namespace endpoint and a regional endpoint. - **Namespace endpoint** (`..tmprl.cloud:7233`) — **Recommended** - This endpoint is unique to each Namespace. It always connects to the Namespace, no matter which region(s) the Namespace is using. - A Temporal Client that uses a Namespace endpoint doesn't have to be aware of which region the Namespace is in. - For Namespaces with [High Availability](/cloud/high-availability), the Namespace endpoint automatically directs traffic to the active region, so Workers and Clients don't need to change endpoints during a failover. - Regional endpoint (`..api.temporal.io:7233`) - Temporal Cloud has only one regional endpoint for each cloud region. The same regional endpoint can access any Namespace that is active in that region (or that has a [replica](/cloud/high-availability) in that region). - A Temporal Client can use a regional endpoint to ensure connection to a Namespace always happens within that region. This can be useful in advanced [High Availability](/cloud/high-availability) setups where you want explicit control over which region handles requests. - When using mTLS to authenticate, the Temporal Client must set the `server_name` property to `` in its request to the value of the Namespace endpoint. This tells the client to expect a different SNI header during the TLS handshake, since the request to the regional endpoint is redirected to the specific Namespace. > **⚠️ Caution:** > Do not take dependencies on Temporal Cloud endpoint DNS resolution > > In general, the IP addresses that Temporal Cloud endpoints resolve to are subject to change without notice. Do not configure Workers, Temporal Clients, or firewalls against the specific IPs you observe for an endpoint at a point in time. > > Temporal Cloud guarantees the DNS resolution behavior of the Namespace Endpoint in two cases: > > 1. **Stable IPs enabled.** The Namespace Endpoint resolves to one of the [Stable IPs](/cloud/connectivity/ip-addresses) for the Namespace's active region. > 2. **High Availability features with Private Connectivity.** The Namespace Endpoint resolves to a regional intermediary (`-.region.tmprl.cloud`) that you can override in a Route 53 private hosted zone or GCP private DNS zone to point at your VPC Endpoint. See [Connectivity for High Availability](/cloud/high-availability/ha-connectivity) for setup details. > ### Configuring a Temporal Client with API keys or mTLS To use API keys to connect with the [Temporal CLI](/cli), [Client SDK](/develop), [tcld](/cloud/tcld), [Cloud Ops API](/ops), and [Terraform](/cloud/terraform-provider), see [Use API keys to authenticate](/cloud/api-keys#using-apikeys). To use mTLS to connect with the [Temporal CLI](/cli) and [Client SDK](/develop), see [Configure Clients to use Client certificates](/cloud/certificates#configure-clients-to-use-client-certificates). ### Accessing the Temporal Web UI For accessing the Temporal Web UI, use the HTTPS endpoint in the form: `https://cloud.temporal.io/namespaces/.`. For example: `https://cloud.temporal.io/namespaces/accounting-production.f45a2`. ### Access Namespaces with encryption and private connectivity To ensure the security of your data, all traffic to and from your Namespace is encrypted with TLS 1.3. For enhanced protection: - Set up [private connectivity](/cloud/connectivity#private-network-connectivity-for-namespaces) to the Namespace. - Set up your allow list for outgoing network requests from your Clients and Workers. You have two options: - Use Temporal Cloud's [stable IPs configuration](/cloud/connectivity/ip-addresses) to get non-changing IP addresses for your Namespace endpoint, which you can then allowlist in your firewall rules. - Allowlist the entire Cloud Provider IP address ranges for the region in which your Namespace is located: - [AWS IP address ranges](https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html) - [GCP IP address ranges](https://cloud.google.com/compute/docs/faq#find_ip_range) ## How to manage Namespaces in Temporal Cloud ### Manage Namespaces in Temporal Cloud using Temporal Cloud UI To list Namespaces: - On the left side of the window, select **Namespaces**. To edit a Namespace (including custom Search Attributes, certificates, certificate filters, Codec Server endpoint, permissions, and users), find the Namespace and do either of the following: - On the right end of the Namespace row, select the three vertical dots (⋮). Click **Edit**. - Select the Namespace name. In the top-right portion of the page, select **Edit**. On the **Edit** page, you can do the following: - Add a [custom Search Attribute](/search-attribute#custom-search-attribute). - [Manage CA certificates](/cloud/certificates). - [Manage certificate filters](/cloud/certificates#manage-certificate-filters-using-temporal-cloud-ui). - Set [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) for all users on the Namespace. Each user on the Namespace has the option to [override this setting](/production-deployment/data-encryption#web-ui) in their browser. - Manage [Namespace-level permissions](/cloud/manage-access/roles-and-permissions#namespace-level-permissions). - Add users. To add a user to a Namespace, scroll to the bottom of the page and select **Add User**. After you make changes, select **Save** in the top-right or bottom-left portion of the page. ### Manage Namespaces in Temporal Cloud from the CLI To list Namespaces and get information about them, use the following commands: - [`temporal cloud namespace list`](/cli/command-reference/cloud/namespace#list) or [tcld namespace list](/cloud/tcld/namespace/#list) - [`temporal cloud namespace get`](/cli/command-reference/cloud/namespace#get) or [tcld namespace get](/cloud/tcld/namespace/#get) To manage certificates, use the [`temporal cloud namespace mtls cert-ca`](/cli/command-reference/cloud/namespace#mtls-cert-ca) or [tcld namespace accepted-client-ca](/cloud/tcld/namespace/#accepted-client-ca) commands. For more information, see [How to manage certificates in Temporal Cloud](/cloud/certificates). To manage certificate filters, use the [`temporal cloud namespace mtls cert-filter`](/cli/command-reference/cloud/namespace#mtls-cert-filter) or [tcld namespace certificate-filters](/cloud/tcld/namespace/#certificate-filters) commands. For more information, see [How to manage certificate filters in Temporal Cloud](/cloud/certificates#manage-certificate-filters). ## How to delete a Namespace in Temporal Cloud > **ℹ️ Info:** > > To delete a Namespace, a user must have Namespace Admin [permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) for that > Namespace. > ### Delete a Namespace using Temporal Cloud UI 1. Go to the Temporal Cloud UI and log in. 1. On the left side of the window, select **Namespaces**. 1. On the **Namespaces** page, select a Namespace Name. 1. On the Namespace page, select **Edit** in the upper-right portion of the window. 1. On the **Edit** Namespace page, select **Delete Namespace** in the upper-right portion of the window. 1. In the **Delete Namespace** dialog, type `DELETE` to confirm the deletion of that Namespace. 1. Select **Delete**. After deleting a Temporal Cloud Namespace, the Temporal Service immediately removes the Namespace's Workflow Executions and Task Queues. Make sure all Workflows have been completed, canceled, or terminated before removing a Namespace. The Namespace removal is permanent. Closed Workflow Histories remain in Temporal storage until the user-defined retention period expires. This period reflects the policy in effect when the Workflow Execution was closed. For further questions or concerns, contact [Support](/cloud/support#support-ticket). ### Delete a Namespace from the CLI See the [`temporal cloud namespace delete`](/cli/command-reference/cloud/namespace#delete) or [tcld namespace delete](/cloud/tcld/namespace/#delete) command reference for details. ### Namespace deletion protection To prevent accidental Namespace deletion, Temporal Cloud provides a protection feature. When you enable Deletion Protection for your production environment Namespace, you ensure that critical data won't be deleted unintentionally. Follow these steps: - Visit the [Namespaces page](https://cloud.temporal.io/namespaces) on Temporal Cloud. - Open your Namespace details page. - Select the Edit button. - Scroll down to Security and click the disclosure button (downward-facing caret). - Enable **Deletion Protection** ![Deletion Protection is enabled by toggling the switch](/img/cloud/namespace/deletion-protection.png) To enable or disable this feature from the CLI, use the following command. Set the value to `true` to enable or `false` to disable: **Temporal CLI** ``` temporal cloud namespace lifecycle set \ --namespace \ --enable-delete-protection ``` **tcld** ``` tcld namespace lifecycle set \ --namespace \ --enable-delete-protection ``` ## How to tag a Namespace in Temporal Cloud Tags are key-value metadata pairs that can be attached to namespaces in Temporal Cloud to help operators organize, track, and manage namespaces more easily. ### Tag structure and limits - Each namespace can have a maximum of 10 tags - Each key must be unique for a given namespace (for example, a namespace cannot have both `team:foo` and `team:bar` tags) - Keys and values must be 1-63 characters in length - Allowed characters: lowercase letters (`a-z`), numbers (`0-9`), periods (`.`), underscores (`_`), hyphens (`-`), and at signs (`@`) - Tags are not a secure storage mechanism and should not store PII or PHI - Tags will not change the behavior of the tagged resource - There is a soft limit of 1000 unique tag keys per account ### Permissions - Only [**Account Admins** and **Account Owners**](/cloud/manage-access/roles-and-permissions#account-level-roles) can create and edit tags - All users with access to a namespace can view its tags ### Temporal Cloud CLI See the [`temporal cloud namespace tag`](/cli/command-reference/cloud/namespace#tag) or [tcld namespace tags](/cloud/tcld/namespace/#tags) command reference for details. ### Terraform See the [Terraform provider](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/docs/resources/namespace_tags.md) for details. ### Web UI Tags can be viewed and managed through the Temporal Cloud web interface. When viewing a namespace, you'll see tags displayed and can add, edit, or remove them if you have the appropriate permissions. ![Tags appear in namespace details](/img/cloud/tags/Namespace-DetailsWithTags.png) ![Tags appear on the list of namespaces](/img/cloud/tags/Namespaces-IndexWithTags.png) ![Where to add tags during namespace creation](/img/cloud/tags/CreateNamespace-AddNewTag.png) ![After adding a tag during namespace creation](/img/cloud/tags/CreateNamespace-AddedTag.png) --- # Nexus Source: https://docs.temporal.io/cloud/nexus > Temporal Cloud adds global Nexus Registry, built-in access controls, audit logging, and multi-region connectivity on top of core Nexus. Temporal Cloud builds on the [core Nexus experience](/nexus) with: - **Global [Nexus Registry](/nexus/registry)** - Scoped to your entire Account across all Namespaces. Workers in any Namespace can host Nexus Services for others to use. - **Built-in [access controls](/nexus/registry#configure-runtime-access-controls)** - Restrict which caller Namespaces can use a Nexus Endpoint at runtime. - **[Audit logging](/cloud/audit-logs)** - Stream Nexus Registry actions (create, update, delete Endpoints) to your audit log integration. - **Multi-region connectivity** - Nexus requests route across Namespaces within and across AWS and GCP using a global mTLS-secured Envoy mesh. Compatible with Namespaces that have [High Availability](/cloud/high-availability) as Endpoint targets. - **[Terraform support](/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform)** - Manage Nexus Endpoints with the Temporal Cloud Terraform provider. ![Nexus Overview](/img/cloud/nexus/nexus-overview-short.png) ## Learn more - [Evaluate Nexus](/evaluate/nexus) | [Keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082) - [How Nexus works](/nexus) | [Deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4&t=934s) --- # Latency and Availability - Temporal Nexus Source: https://docs.temporal.io/cloud/nexus/latency-availability > Learn about Nexus latency and availability Nexus latency and availability in Temporal Cloud: - **SLOs and SLAs** - Nexus operations (for example, `RespondWorkflowTaskCompleted`, `PollNexusTaskQueue`, `RespondNexusTaskCompleted`, and `RespondNexusTaskFailed`) have the same [latency SLOs](/cloud/service-availability#latency) and [availability SLAs](/cloud/sla) as other Worker requests in both caller and handler Namespaces. - **[Nexus metrics](/nexus/metrics)** - SDK and Cloud latency metrics for monitoring Nexus performance. - **Cross-Namespace connectivity** - Traffic routes through a global mTLS-secured Envoy mesh. Same-region Namespaces have low latency; cross-region latency varies by provider. See [secure connectivity](/nexus/security#secure-connectivity). --- # Limits - Temporal Nexus Source: https://docs.temporal.io/cloud/nexus/limits > Learn about Nexus limits including rate limits, the maximum number of Endpoints, and handler request timeouts. Nexus limits are documented in [Temporal Cloud limits](/cloud/limits): - [Nexus rate limits](/cloud/limits#nexus-rate-limits) - Nexus requests count toward the Namespace RPS limit. - [Nexus Endpoint limits](/cloud/limits#nexus-endpoints-limits) - 100 Endpoints per Account (default). - [Nexus caller Namespace limits](/cloud/limits#nexus-endpoint-access-policy-limits) - 1,000 caller Namespaces per Endpoint (default). - [Per-Workflow Nexus Operation limits](/cloud/limits#per-workflow-nexus-operation-limits) - 30 in-flight Operations per Workflow. - [Nexus Operation request timeout](/cloud/limits#nexus-operation-request-timeout) - Less than 10 seconds for a handler to process a start or cancel request. - [Nexus Operation duration limits](/cloud/limits#nexus-operation-duration-limits) - 60-day maximum ScheduleToClose duration. - [Per-Workflow callback limits](/cloud/limits#per-workflow-callback-limits) - 2000 callbacks per Workflow. Governs how many Nexus callers can attach to a handler Workflow. --- # Observability - Temporal Nexus Source: https://docs.temporal.io/cloud/nexus/observability > Learn about integrated observability in Temporal Nexus including cloud metrics and audit log streaming. Nexus observability in Temporal Cloud: - **[Nexus metrics](/nexus/metrics)** - [SDK metrics](/nexus/metrics#sdk-metrics) emitted by Workers and [Cloud metrics](/nexus/metrics#cloud-metrics) emitted by Temporal Cloud. - **[Execution debugging](/nexus/execution-debugging)** - Bi-directional linking, pending Operations, pending callbacks, and tracing across Namespaces. - **[Audit logging](/cloud/audit-logs)** - `CreateNexusEndpoint`, `UpdateNexusEndpoint`, and `DeleteNexusEndpoint` actions streamed to your audit log integration. --- # Pricing for Temporal Nexus Source: https://docs.temporal.io/cloud/nexus/pricing > Learn about the pricing structure for using Nexus. Nexus pricing: - **One Action to start or cancel a Nexus Operation** in the caller Namespace. Underlying primitives (Workflows, Activities, Signals) and their retries created by the handler result in normal Actions. - **No Action for handling or retrying the Nexus Operation itself**. However, billable actions initiated by the handler (such as Activities) are charged if they fail and retry. See [Pricing](/cloud/pricing) for details. --- # Security - Temporal Nexus Source: https://docs.temporal.io/cloud/nexus/security > Learn about security in Temporal Nexus Nexus security in Temporal Cloud: - **[Runtime access controls](/nexus/security#runtime-access-controls)** - Endpoint allowlists restrict which caller Namespaces can use an Endpoint. See [configuring access controls](/nexus/registry#configure-runtime-access-controls). - **[Secure connectivity](/nexus/security#secure-connectivity)** - mTLS for all Nexus communication across cells and regions. Endpoints are only accessible within a Temporal Cloud Account. - **[Payload encryption](/nexus/security#payload-encryption-data-converter)** - Same Data Converter as Workflows and Activities, with three approaches for cross-Namespace encryption. - **[Registry roles and permissions](/nexus/registry#roles-and-permissions)** - Controls who can view, create, edit, and delete Endpoints. --- # Receive Temporal Cloud notifications Source: https://docs.temporal.io/cloud/notifications > Receive updates about Temporal Cloud including when certificates will expire, billing updates, and when a failover has completed. ## Get notified about Temporal Cloud status In the event of an incident, Temporal updates the [Temporal Cloud status page](https://status.temporal.io/) with important updates. Users can subscribe to updates in their preferred mode, such as email, Slack, or SMS, by visiting this page. ## Get notified about administrative events Temporal Cloud sends emails to notify users of important administrative events. | Reason for email | Who receives email | |------------------ | -------------------| | Certificate Expiring in 15 days | Global Administrator, Namespace Administrator, Account Owner | | Certificate Expiring in 10 days | Global Administrator, Namespace Administrator, Account Owner | | Certificate Expiring in 5 days | Global Administrator, Namespace Administrator, Account Owner | | API Key Expiring in 30 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | | API Key Expiring in 20 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | | API Key Expiring in 10 days | Global Administrator, Account Owner, individual user (if API Key has an owner) | | Sign up credit expiring in 30 days | Account Owner, Finance Administrator | | Sign up credit expiring in 14 days | Account Owner, Finance Administrator | | Sign up credit expiring in 7 days | Account Owner, Finance Administrator | | Sign up credit expiring in 1 days | Account Owner, Finance Administrator | | Sign up credit is 50% consumed | Account Owner, Finance Administrator | | Sign up credit is 90% consumed | Account Owner, Finance Administrator | | Account plan type changed | Global Administrator, Account Owner, Finance Administrator | | Namespace Failover Completed/Failed | Global Administrator, Namespace Administrator, Account Owner | To ensure that you receive email notifications, configure your junk-email filters to permit email from `noreply@temporal.io`. To provide feedback on notifications or request changes, [create a support ticket](/cloud/support#support-ticket). --- # Overview - Temporal Cloud Source: https://docs.temporal.io/cloud/overview > Temporal Cloud is a fully managed, globally distributed durable execution platform built on cell-based architecture. Available on AWS and GCP with consumption-based pricing and zero-downtime migration from self-hosted deployments. Temporal Cloud is a fully managed durable execution platform. It handles the complexity of running Temporal at scale—persistence, replication, upgrades, and availability—so you can focus on building applications. Your code runs in your environment. Temporal Cloud never sees your application logic or sensitive data. The platform stores encrypted Workflow state and orchestrates execution, while your Workers execute business logic wherever you deploy them. ## How Temporal Cloud works Temporal Cloud operates as the Control Plane for your distributed applications: 1. **Your environment**: You run Workers that execute your Workflow and Activity code. These can be deployed anywhere—Kubernetes, VMs, serverless, on-premises. 2. **Temporal Cloud**: Manages Workflow state, Event History, task queuing, and scheduling. All data is encrypted in transit and at rest. 3. **Temporal SDKs**: Your applications use the SDK to communicate with Temporal Cloud over secure gRPC connections. This separation means Temporal Cloud scales independently of your application. You control compute resources for your Workers; Temporal handles the orchestration layer. ## Architecture ### Cell-based infrastructure Temporal Cloud uses a cell-based architecture to achieve isolation and scalability. Each cell is a self-contained deployment unit with its own: - Dedicated cloud account and VPC - Kubernetes cluster running Temporal services - Primary database with synchronous replication across three availability zones - Elasticsearch for Workflow visibility and search - Load balancers and ingress management - Observability and certificate infrastructure Cells act as failure domains. If infrastructure within a cell experiences issues, only Namespaces in that cell are affected. This design limits blast radius and enables independent scaling. ### Data plane and control plane **Data plane**: Where your Workflows execute. Each cell processes Workflow operations, persists state, and manages task queues. The data plane is optimized for low latency and high throughput. **Control plane**: Manages provisioning, configuration, and lifecycle operations. When you create a Namespace, the Control Plane: 1. Selects an appropriate cell in your chosen region 2. Provisions database resources and roles 3. Generates and deploys mTLS certificates 4. Configures ingress routes and validates connectivity The control plane uses Temporal itself (durable execution) to orchestrate these operations reliably. ### Multi-cloud availability Temporal Cloud runs on both AWS and GCP: - AWS regions spanning North America, Europe, Asia Pacific, and South America - GCP regions in North America, Europe, and Asia Pacific You can create Namespaces in any supported region. For disaster recovery, you can replicate across regions within a cloud provider or across cloud providers entirely. See [Service regions](/cloud/regions) for the complete list of available regions. ## Built-in reliability Every Temporal Cloud Namespace includes baseline high availability: - **Three-zone replication**: Workflow state synchronously replicates across three availability zones before acknowledging writes - **Automatic failover**: If one zone becomes unavailable, operations continue on the remaining zones - **99.9% SLA**: Contractual uptime guarantee for standard Namespaces ### High Availability features For workloads requiring stronger guarantees, Temporal Cloud offers three replication options: | Deployment | Description | Use case | |------------|-------------|----------| | **Same-region** | Replicate across isolated cells within one region | Single-region applications needing cell-level isolation | | **Multi-region** | Replicate across regions within one cloud provider | Geographic redundancy and compliance requirements | | **Multi-cloud** | Replicate across cloud providers (AWS ↔️ GCP) | Maximum resilience against provider-level outages | High Availability Namespaces include: - **99.99% SLA**: Four-nines contractual uptime guarantee - **Sub-1-minute RPO**: Recovery Point Objective for data loss - **20-minute RTO**: Recovery Time Objective for failover completion - **Automatic or manual failover**: Choose your preferred failover strategy See [High Availability](/cloud/high-availability) for configuration details. ## Security model Temporal Cloud implements defense-in-depth security: ### Your code stays with you Temporal Cloud never executes your application code. Workers run in your environment, connecting to Temporal Cloud over encrypted channels. You control access to your compute resources and secrets. ### Client-side encryption The [Data Converter](/dataconversion) lets you encrypt payloads before they leave your Workers. Temporal Cloud stores ciphertext—if the service were compromised, your data remains encrypted. Deploy a [Codec Server](/production-deployment/data-encryption) to decrypt data in the Web UI without sharing keys. ### Network isolation - **mTLS authentication**: Per-Namespace certificate-based authentication for gRPC endpoints - **API key authentication**: Alternative to certificates for simpler key management - **Private connectivity**: AWS PrivateLink and GCP Private Service Connect for traffic that never traverses the public internet ### Compliance Temporal Technologies maintains SOC 2 Type 2 certification and complies with GDPR and HIPAA regulations. Audit logs capture supported operations in the Temporal Cloud Control Plane and can be exported to your security monitoring systems. See [audit logs](/cloud/audit-logs) for the supported operations and coverage details. See [Security model](/cloud/security) for complete details. ## Consumption-based pricing Temporal Cloud charges based on what you use: ### Actions The primary billing unit. Actions are billable operations like starting Workflows, sending Signals, recording Heartbeats, and completing Activities. Pricing starts at $50 per million Actions with volume discounts as you scale. ### Storage - **Active Storage**: Event History for running Workflows - **Retained Storage**: Event History for completed Workflows (configurable retention period up to 90 days) ### Plans Four tiers—Essentials, Business, Enterprise, and Mission Critical—with increasing support levels, included Actions/Storage, and features like SAML and SCIM. The Essentials plan starts at $100/month. Self-serve signup and plan management available at [cloud.temporal.io](https://cloud.temporal.io). See [Pricing](/cloud/pricing) for detailed rates and examples. ## Portability Temporal Cloud runs the same Temporal Server as the open-source distribution. This means: ### Zero code changes Applications built for self-hosted Temporal work on Temporal Cloud without modification. Update your connection configuration to point at your Cloud Namespace—that's it. ### Zero-downtime migration [Automated migration](/cloud/migrate/automated) uses Workflow replication to move running Workflows from self-hosted to Cloud (or between Cloud regions) without interruption. No Workflow restarts, no data loss, no downtime. [Manual migration](/cloud/migrate/manual) works by updating Clients and Workers to use new Namespace endpoints while existing Workflows complete naturally. ### Bidirectional Move workloads from self-hosted to Cloud, Cloud to self-hosted, or between Cloud regions and providers. The same migration tooling works in any direction. ## Self-serve operations Temporal Cloud is designed for self-service: - **Web UI**: Create Namespaces, manage users, configure settings at [cloud.temporal.io](https://cloud.temporal.io) - **[Temporal CLI Cloud extension](/cli/cloud)**: Automate operations from the Temporal CLI. - **CLI (`tcld`)**: Automate operations from the command line - **Terraform provider**: Infrastructure-as-code for Namespaces, users, and configuration - **Cloud Ops API**: Programmatic access for custom tooling and automation No support tickets required for standard operations. ## Getting started 1. [Sign up for Temporal Cloud](https://temporal.io/get-cloud) 2. [Create your first Namespace](/cloud/namespaces) 3. [Connect your Workers](/cloud/get-started#set-up-your-clients-and-workers) 4. [Run your first Workflow](/cloud/get-started#run-your-first-workflow) For existing Temporal users, see [Migration](/cloud/migrate) to move self-hosted workloads to Cloud. --- # Temporal Cloud pricing Source: https://docs.temporal.io/cloud/pricing > Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. Temporal Cloud is a consumption-based service. You pay only for what you use. Our pricing reflects your use of [_Actions_](#action), [_Storage_](#storage), and [_Support_](/cloud/support#support). It is flexible, transparent, and predictable, so you know your costs. This page describes the elements of Temporal Cloud pricing. It gives you the information you need to understand and estimate costs for your implementation. For more exact estimates, please reach out to [our team](https://pages.temporal.io/ask-an-expert). Billing and cost information is available directly in the Temporal Cloud UI. For more information, visit the [Billing](/cloud/billing) page. ## Temporal Cloud pricing model This section explains the basis of the Temporal Cloud pricing model and how it works. Your total invoice each calendar month is the combination of Temporal Cloud consumption ([Actions](#action) and [Storage](#storage)), and a [Temporal Cloud Plan](#base_plans) that includes [Support](/cloud/support#support). ### Temporal Cloud plans **How plans work** Each Temporal Cloud account includes a plan with Support, Actions, Active Storage, Retained Storage and platform features. Base allocations help you get started with the Temporal platform, so you can better estimate costs. - Temporal Cloud Plans are charged monthly. - Action and Storage allocations are reset each calendar month. Temporal offers four plans: Essentials, Business, Enterprise, Mission Critical. Prices are outlined in the following table: | | Essentials | Business | Enterprise | Mission Critical | | ----------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Support Targeting | Basic use | Production deployments
that scale | Enterprise deployments
w/ stringent uptime demands | Mission-critical applications
w/ the highest support needs | | Support Features | Access to Support | - P0 Response Times | - P0: \<30 Min, 24/7
- Private Slack | - P0: \<15 Min, 24/7
- Private Slack
- Dedicated Platform Architect | | Product Features | - 1M Actions
- 1 GB Active Storage
- 40 GB Retained Storage | - Commit discounts
- SAML included
- SCIM (Add-on)
- 2.5M Actions
- 2.5 GB Active Storage
- 100 GB Retained Storage | - Commit discounts
- SAML included
- SCIM included
- 10M Actions
- 10 GB Active Storage
- 400 GB Retained Storage | - Commit discounts
- SAML included
- SCIM included
- 10M Actions
- 10 GB Active Storage
- 400 GB Retained Storage | | Plan Pricing | Greater of
- $100/mo or
- 5% of Usage Spend | Greater of
- $500/mo or
- 10% of Usage Spend | Priced annually:
[contact Sales](mailto:sales@temporal.io) for details | Priced annually:
[contact Sales](mailto:sales@temporal.io) for details | | Usage Pricing | [Pay-As-You-Go](#payg) Pricing | Choose from
[Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing) | Choose from
[Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing) | Choose from
[Pay-As-You-Go](#payg) or [Commitment Pricing](#commitment-pricing) | Please note, partial months are prorated to the day. Find a complete description of Support offerings and response times in the [Support](/cloud/support) documentation. > **📝 Note:** > Converting GB to GBH > > Active and Retained Storage allocations are translated into GBh at a rate of 1GB equals 744GBh. > ### Actions **What are Temporal Actions?** Actions are the primary unit of consumption-based pricing for Temporal Cloud. They track billable operations within the Temporal Cloud Service, such as starting Workflows, recording a Heartbeat or sending messages. **Specific Billable Actions are discussed on the [Actions](/cloud/actions) page.** [Reach out](https://pages.temporal.io/contact-us) to our team for more information or to help size your number of Actions. ### Storage **How Workflow Storage works** A Workflow's execution might exist for a few seconds, a day, month, or even forever. The Temporal Service stores the Workflow Execution's [Event History](/workflow-execution/event#event-history). Under this framework, a Workflow Execution has only two states, open (Active Storage) or closed (Retained Storage). - _Active Storage_ measures the amount of storage used by active Workflows. - When the execution of a Workflow finishes, Temporal Cloud stores Event History for a defined [Retention Period](/temporal-service/temporal-server#retention-period), which is set by the user per Namespace. This is _Retained Storage_. Typical uses of Retained Storage include compliance, debugging, workload refresh, and business analytics. When closed Workflow Histories need to be retained for more than the 90-day maximum period on Temporal Cloud, we recommend using our [**Export**](/cloud/export) feature. Storage costs are measured in gigabyte-hours (GBh). ### Pricing options **How to Pay for Temporal Cloud** After you exceed your Actions and Storage allocations in your base tier, Temporal Cloud offers two payment options: Pay-As-You-Go and Commitments. Both models meter and bill for three primary components: [Actions](#action), [Storage](#storage), and [your Temporal Cloud Plan](/cloud/support#support). - With Pay-As-You-Go, you are invoiced each calendar month based on your consumption. Pay-As-You-Go pricing automatically applies volume prices as your Actions scale. - With Commitments, you pre-purchase your Temporal Cloud spend with Temporal Credits. Temporal Credits pay for your Temporal Cloud consumption, including Temporal Cloud Plan charges. ## Pay-As-You-Go **How does Pay-As-You-Go pricing work?** Pay-As-You-Go pricing is based on consumption. This section explains how you're billed each calendar month and gives examples. ### Action pricing Actions pricing starts at $50 per million Actions ($0.00005 per Action). You gain progressive volume discounts as you scale. Discounts are based on your account's total usage, metered and billed for each calendar month: | Actions | Price per Million Actions | | --------------------- | ----------------------------------------------------------------------------------------- | | First 5M | $50 | | Next 5M, up to 10M | $45 | | Next 10M, up to 20M | $40 | | Next 30M, up to 50M | $35 | | Next 50M, up to 100M | $30 | | Next 100M, up to 200M | $25 | | Over 200M | Contact [Sales](mailto:sales@temporal.io) for info
_More discounts, helpful humans_ | **Example** If you consume 11.25M Actions in excess of your Temporal Cloud Plan allocation in one calendar month, your bill for Actions will be: ``` 5M Actions ⨉ $50 Per Million Actions = $250 5M Actions ⨉ $45 Per Million Actions = $225 1.25M Actions ⨉ $40 Per Million Actions = $50 Actions $250 (First Tier) + $225 (Second Tier) + $50 (Third Tier) = $525 ``` ### Storage pricing Most accounts’ storage needs are met by our Temporal Cloud Plans. For additional storage within a calendar month, you are billed for Active and Retained Storage as follows: | **Storage** | **Price per GBh (USD)** | | ----------- | ----------------------- | | Retained | $0.00105 | | Active | $0.042 | > **💡 Tip:** > Storage costs are also affected by Temporal System Workflows that back features such as: > > - [Schedules](/schedule): Each Scheduled Workflow contributes to storage usage. Supplied inputs, outputs, and failures all account for the storage usage incurred from Scheduled Workflows. > - [Batch jobs](/cli/command-reference/batch): Batch Workflow executions also consume storage. > > These Workflow executions contribute to overall active and retained storage consumption. > **Example** If you have 720 GBh of Active Storage and 3,600 GBh of Retained Storage in excess of your Base Tier allocations in one calendar month, your bill will be: ``` 720 GBh Active Storage ⨉ $0.042 per GBh = $30.24 3,600 GBh Retained Storage ⨉ $0.00105 per GBh = $3.78 Total Storage Bill: $30.24 Active Storage + $3.78 Retained Storage = $34.02 ``` ## Temporal Cloud Plan pricing Your Temporal Cloud Plan pricing is the greater of the minimum monthly price or a percent (%) of your consumption spend: - The Essentials tier is priced at the greater of $100/month or 5% of your Temporal Cloud consumption. - The Business tier is priced at the greater of $500/month or 10% of Temporal Cloud consumption. - The Enterprise and Mission Critical Support plans must be paid annually. Contact [Sales](mailto:sales@temporal.io) to discuss your needs. Your Temporal Cloud consumption combines the costs of Actions and Storage. **Example** If you are signed up for Essentials, with $3,000 of monthly spend, your bill will be: ``` Greater of $100 or 5% ⨉ $3,000 = $150, so $150. ``` ## Commitment Pricing **Commitments with Temporal Credits** Temporal Cloud offers the option to commit to a minimum spend over a given timeframe. In exchange for this commitment you receive additional discounts. Key discount levers include: - Account Action volume over 200M Actions - Duration of your commitment (1, 2, or 3 years) Meet your commitments with any Temporal Cloud spend, including Actions, Storage and your Temporal Cloud Plan. After making a commitment, Temporal locks in your Actions price based on your expected volume and discounts your Active Storage costs. This price is used to bill your Actions and Active Storage across your account for the timeframe specified in your commitment. Commitments must be paid for with Temporal Credits. Temporal Credits are used to pay your Temporal Cloud consumption, including Temporal Cloud Plan charges. A Temporal Credit is equivalent to $1 USD. For example, a credit purchase of $20,000 results in 20,000 Temporal Credits. A minimum credit purchase equivalent to the first year of your commitment is required. For multi-year deals please contact [Sales](mailto:sales@temporal.io) for the most accurate pricing. ### Commitment Pricing Q&A **How do multi-year commitments work?** Our sales team works with you to match annual credit purchases in line with your expected spend. This aligns your payments to annual terms rather than one up-front expense. **What happens if I exhaust my commitment-based Temporal Credits before the end of my term?** You continue to receive the negotiated discounted prices for the remainder of your term. You'll be invoiced for another credit purchase based on your most recent calendar month's spend. This amount is multiplied by the months remaining in the annual portion of your term. If your previous month spend was $5,000, and you're 10 months through your annual term, you'll be invoiced for 10,000 Temporal Credits to cover the remaining two months. **What happens if I have unused Temporal Credits at the end of my term?** Commitments can be difficult to estimate. Temporal Cloud offers two ways to roll-over unused credits: - When you renew a commitment for the same or larger amount, Temporal Cloud rolls over any unused credits into your new commitment. - Should you need to downsize your commitment, Temporal Cloud rolls over up to 10% of your initial credit purchase amount into the new commitment. **How do I make a commitment and purchase Temporal Credits?** Contact our team at [sales@temporal.io](mailto:sales@temporal.io) or reach out to your dedicated account manager. You can also purchase Temporal Cloud commitments credits through [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) and [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go). ### Credit Balance Your Credit Balance is adjusted each calendar month based on your Credit Usage. **Example** An account has purchased credits with an annual spend commitment of $72,000. In the first calendar month, the account is billed $500 for a Business plan, $5,000 for Actions, $250 for Active Storage and $50 for Retained Storage, a total of $5800. Their invoice would state: ``` Beginning credit balance of 72,000 - 5800 credits used = 66,200, Temporal Credits remaining. ``` If you have additional questions about credits and volume-based pricing, please contact [sales](mailto:sales@temporal.io) or, if you're already a Temporal Cloud customer, reach out to your dedicated account executive. ## Other pricing Temporal Cloud has additional pricing for other elements of the platform. ### Capacity Modes **What are Capacity Modes?** Temporal offers On-Demand and Provisioned Capacity modes. On-Demand capacity is automatically adjusted based on past usage. Provisioned Capacity modes lets you define the capacity that is needed by your Workflow and is useful to handle traffic outside of the standard on-demand limits. See details on how capacity is set and the associated limits at [Capacity Modes](/cloud/capacity-modes). **How does pricing for Capacity Modes work?** The number of Actions accrued can be impacted by your capacity mode. For On-Demand Capacity Mode, Actions are accrued as usual. For Provisioned Capacity there is a minimum number of Actions that must be used per calendar hour for each [Temporal Resource Unit (TRU)](/cloud/capacity-modes) that is provisioned. If Action volume in a calendar hour does not exceed the minimum allocation, Actions of the subtype `ns_capacity:tru` will be recorded to the required volume. The minimum requirement for the first TRU is 0 Actions as it aligns with the default rate limits on Temporal. For each additional TRU, there is a minimum hourly requirement of 360,000 Actions Per Hour. 360,000 Actions per hour represents a 20% utilization of requested resources. This can be calculated as 500 APS Per TRU * 3600 Seconds (1 Hour) * 20%= 360,000 Actions. For example: If you have a namespace that requests 4 TRUs (that is, 2,000 APS) for 4 hours and have usage as follows: * Hour 1: 2,000,000 Actions * Hour 2: 5,000,000 Actions * Hour 3: 4,000,000 Actions * Hour 4: 500,000 Actions Then the cost of Provisioned Capacity would be calculated as: 4 TRUs Requested - 1 Default TRU included = 3 TRUs with a minimum usage requirement 3 TRUs * 360,000 Actions per Hour= 1,080,000 minimum Actions per Hour * Hour 1: 2,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued * Hour 2: 5,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued * Hour 3: 4,000,000 Actions > 1,080,000 minimum Actions = No additional Actions accrued * Hour 4: 500,000 Actions < 1,080,000 minimum Actions so 1,080,000-500,000 Actions = 580,000 additional actions will be added to the hour Total for 4 hours: 2,000,000+5,000,000+4,000,000+1,080,000=12,080,000 Actions If TRUs are changed multiple times within an hour, the highest value within that hour will be used to calculate the minimum actions required. Temporal’s approach to pricing Provisioned Capacity aligns with our goal to only charge for what you use. The minimum hourly allocation will have no impact on your price if you utilize the requested capacity above your default by 20% or more, and will only incur an additional charge if the requested capacity is not used. To avoid being charged for provisioned capacity it is advised to "return" capacity to the pool and switch back to the On-demand mode when usage is stable. ### High Availability feature pricing **How does the pricing for High Availability (HA) features work?** For workloads with stringent high availability requirements, Temporal Cloud provides same-region, multi-region, and multi-cloud replicas, which add a failover capability. Enabling HA features for Namespace automatically replicates Workflow Execution data and metadata to a replica in the same region or in a different region. This allows for a near-seamless failover when incidents or outages occur. The pricing for High Availability features aligns with the volume of your workloads. Actions and Storage in your Namespace contribute to your Actions and Storage consumption. To estimate costs for this deployment model, apply a 2x multiplier to the Actions and Storage in the Namespace you are replicating and include this scaling in your account's consumption. For pre-existing Namespaces that add a replica, the 2x multiplier only applies to Actions and Storage after the replica is active. Storage used and Actions completed before the replica's creation, or while it was being created but not yet in the 'active' status, do not incur the additional charges. > **💡 Tip:** > Future Pricing Update > > In the future, for Namespaces that incur high volumes of cross-cloud data transfer, Temporal will charge extra to cover the cost of egress charged by the cloud providers. > All Namespaces will receive a generous amount of network transfer for free, so the vast majority of Namespaces will have no charge. > When upgrading an existing Namespace, some points to consider: - Temporal won't charge for historical Actions completed prior to upgrading to a Namespace with High Availability features. Only ongoing (in-flight) and new Workflow Executions will generate consumption. - Temporal charges for all Actions of existing (ongoing) and new Workflows from the point of adding a replica. - Temporal charges for Replicated Storage of retained (historical), running (ongoing), and new Workflow Executions from the point of adding a new replica. ### Fairness pricing **How does pricing for Fairness work?** When [Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) is enabled on a Namespace, an additional `0.1` Action is charged per Action in that Namespace during each hour the feature is on, regardless of whether individual Workflows or Activities use fairness keys. The examples below use a Namespace, `your-namespace`, that normally generates 10,000 Actions per hour. **Fairness enabled, but no fairness keys in use** If Fairness is enabled but no Workflows or Activities send `fairness_key` metadata, all Tasks continue to route as before. Fairness Actions are still applied: billing is determined by whether Fairness is enabled on the Namespace. | Hour | Actions in Namespace | Fairness enabled? | Fairness Actions | Total billed Actions | |---------------|----------------------|-------------------|--------------------|----------------------| | 2:00–3:00 PM | 10,000 | Yes | +1,000 | 11,000 | **Mix of Workflows with and without fairness keys** Suppose `your-namespace` has two Task Queues: - `payment-queue`, where Workflows pass `fairness_key = "customer-id"` - `notification-queue`, where Workflows have no fairness keys and behave exactly as before If each queue generates 5,000 Actions in a given hour (10,000 total), Fairness Actions apply to the full 10,000, not only to the 5,000 from `payment-queue`. | Hour | Actions from payment-queue | Actions from notification-queue | Total Namespace Actions | Fairness | Total billed Actions | |---------------|----------------------------|---------------------------------|-------------------------|-----------------|----------------------| | 2:00–3:00 PM | 5,000 | 5,000 | 10,000 | +1,000 | 11,000 | ### SCIM and SSO via SAML pricing **What costs are associated with SSO/SAML use?** Single sign-on (SSO) integration using SAML is included for all customers on the Business, Enterprise, and Mission Critical Plans. **What costs are associated with SCIM?** To enable SCIM (System for Cross-domain Identity Management), you need an Enterprise or Mission Critical plan or an add-on for the Business plan (+$500/month in addition to the Business plan). Please note that you must configure SSO via SAML to use SCIM. ### Use case cost estimates Temporal Cloud uses a consumption-based pricing model based primarily on [Actions](#action) and [Storage](#storage). Each workload is different. You can estimate the cost of a specific Workflow by running it at a low volume. Use the resulting Storage and compute measurements to project your production scale cost. The examples below provide general estimates based on workload size. You can also use our calculator on the pricing page to build your estimate. Our team is always happy to [help you estimate costs](https://pages.temporal.io/contact-us) for your specific workloads and requirements. | Workload size | Cost (monthly) | Characteristics | Actions | Typical use cases | | ------------- | -------------- | ------------------------------------------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | Small | < $50.00 | Modest / transient throughput | < 1M / month 
_(< 0.38 actions per second)_ | General automation
Human dependent processes 
Data pipelines
Nightly batch processes | | Medium | < $2K | Steady or burst throughput | < 40M / month 
_(< 15 actions per second)_ | Transaction & order systems
Infrastructure automation
Payment Processing
Batch processes | | Large | < $15K | Sustained throughput or multiple use cases | < 400M / month 
_(< 150 actions per second)_ | Data processing / sync
Retail order system
KYC & fraud detection | | Web Scale | $20K+ | "Web scale" and / or numerous use cases | 1B+ / month 
_(400+ actions per second)_ | Social media application
SaaS application service | ## Billing Questions FAQs **What payment methods does Temporal accept?** You can pay with a credit card, ACH, or wire transfer. To pay for Temporal Cloud with an AWS or GCP Account. Sign up in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) or [GCP Marketplace](https://console.cloud.google.com/marketplace/product/temporal-public/temporal-cloud-pay-as-you-go) **How often will I be billed?** Temporal Cloud issues invoices for the previous month’s usage and costs. Invoices are issued on the 3rd of the month for the previous month. For example, invoices for May will be issued at midnight UTC on June 3rd. **Where can I view my usage and billing information?** Account Owners and Finance Admins can view their detailed billing data at any time. Visit the [Billing dashboards](/cloud/billing) in Temporal Cloud. **How do I purchase Temporal Cloud credits?** You can purchase Temporal Cloud credits by contacting our team at [sales@temporal.io](mailto:sales@temporal.io). **What's the minimum cost to run Temporal Cloud?** The Essentials plan starts at $100/month. Consumption in excess of your plan's allocations are billed on a consumption basis. **Can I purchase Temporal Cloud through my Amazon, Azure, or Google Cloud Platform Marketplace?** There are two ways to purchase Temporal Cloud through AWS Marketplace: - Pay-As-You-Go available [here](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo) - Credits: available via private offer, please contact our team at [sales@temporal.io](mailto:sales@temporal.io) To purchase Temporal Cloud on the Google Cloud Marketplace, please contact our team at [sales@temporal.io](mailto:sales@temporal.io). **How do I see how many Temporal Cloud credits are remaining?** To view remaining Temporal Cloud credits, Account Owners and Finance Admins can log in to Temporal Cloud and go to Settings > Billing. You need appropriate administrative permissions to access this section. **What happens if I exceed my available credits under a promotion such as the startup program?** Customers with free credits from the startup program or from a promotion are invoiced once their credit balance is exhausted at the end of that month. **Do promotional credits expire?** Credits received through the startup program or an offer have an expiry date. This date is stated as part of the sign-up process. **How do I update my payment information?** Account Owners and Finance Admins can update payment information at any time on the Temporal Cloud [Billing](https://cloud.temporal.io/billing) page under the Plan tab. You need appropriate administrative permissions to access this section. Select the "Manage Payment Method" button. See [this overview](/cloud/billing) for more details. **What happens if my payment fails?** Temporal will periodically send you email reminders to complete the payment. **How do I view my invoices and billing history?** Invoices are emailed to Account Owners or the designated billing contacts. Account Owners and Finance Admins can view their [detailed billing information](https://cloud.temporal.io/billing) at any time. See our [billing](/cloud/billing) page for details. You need appropriate administrative permissions to access this section. Alternatively, to view invoices and billing history, contact Temporal Finance at [ar@temporal.io](mailto: ar@temporal.io). **Does Temporal charge sales tax/VAT?** We charge applicable sales tax in US jurisdictions as required. **How do I cancel my account?** Account Owners can delete their account and cancel their subscription in the Plans tab in the billing center. See the [billing and cost](/cloud/billing) page for details on how to access the billing center. **Will I lose access immediately if I cancel my account?** Customers lose access to Temporal Cloud once Temporal completes the off-boarding process. Billing is independent of this process. **Can I reactivate my account after cancellation?** No. When your account is canceled, your account data is deleted and cannot be restored. To return to Temporal Cloud, you must sign up again. We will assign you a new Temporal account and consider you as a new customer. --- # Projects Source: https://docs.temporal.io/cloud/projects > Organize Temporal Cloud resources including Namespaces and Nexus Endpoints around your organizational structure > **Pre-release** > Features in pre-release are experimental. While Project-specific APIs and functionality may evolve, enabling Projects is designed to be backward compatible with existing resources, permissions, and workflows through the Default Project. Projects are an organizational layer within a Temporal Cloud Account that groups related resources into a single operational boundary. With Projects, you can organize [Namespaces](/cloud/namespaces), [Nexus endpoints](/nexus/endpoints), and [Connectivity Rules](/cloud/connectivity#connectivity-rules) around how your organization operates: by team, environment, service, or business unit. Projects make it easier to delegate ownership, manage access at scale, and organize resources without creating additional Cloud Accounts. As your organization grows, Projects help align your Temporal Cloud resources with your organizational structure while reducing operational overhead. Projects are for organization and authorization. They don't change how Workflows execute inside a Namespace. They also don't restrict which Namespaces can call a Nexus Endpoint. See [Use Nexus across Projects](#use-nexus-across-projects). ![Diagram of the Temporal Cloud resource hierarchy. An Account is the global container for all resources. Each Account contains one or more Projects, which organize and delegate access across teams, apps, or environments; the diagram shows a default Project created by Temporal alongside custom Projects named Agents and Evals, plus a control to create additional Projects. Each Project contains Namespaces, which are isolated, secure runtime environments for Workflow and other executions; the default Project contains NS-prod and NS-staging, Agents contains customer-support and sales-assistant, and Evals contains benchmarks and regression.](/img/cloud/projects/resource-hierarchy.png) ## Enable Projects on your Account While Projects are in pre-release, they are not enabled by default. [Contact Support](/cloud/support#support-ticket) to request enabling Projects for your account. Once Projects are enabled, existing Accounts receive a Default Project, and new Accounts are created with one automatically. For existing Accounts, all existing resources are automatically placed in the Default Project. Existing role assignments and effective permissions remain unchanged. For new Accounts, the Default Project is created automatically. New resources are automatically placed in the Default Project unless you specify otherwise. ## Default Project and Project ID Every Temporal Cloud Account includes a Default Project. The Default Project is not a special type of Project. It behaves like any other Project for managing resources. The main difference between the Default Project and any Projects you create is that the Default Project is automatically created for you as a starting point and provides backward-compatible behavior for existing resources and operations that do not specify a Project. Otherwise, it supports the same resource-management capabilities as other Projects. If you don't need the organizational boundaries offered by Projects, continue to use the Default Project for all your resources. You can always create a new Project later if you need it. Every Project has a unique, immutable `Project ID` and an editable `display name`. The Project ID is used by the Cloud Ops API, CLI, and Terraform to identify the Project when creating and managing Project-scoped resources. Namespaces cannot be moved between Projects currently, meaning all existing Namespaces will exist in the Default Project. You can create a new Project and place a new Namespace in that Project. For limitations on the number of Projects per Account and number of resources per Project, see [Project-level limitations](/cloud/limits#project-level). ## Create a new Project Create a Project when a set of resources has a distinct owner or access model, separate from the Default Project. A good Project represents a stable ownership boundary, not just a folder. Start with one Project for a team or application with clear ownership. Add more only when needed. **Cloud UI** To create a Project using the Temporal Cloud UI: 1. Select **Projects** at the top of the left navigation bar. 2. Click the **Create Project** button to create a new Project. 3. On the **Create Project** page that appears, give the Project a name (limited to 64 characters) and an optional description (limited to 255 characters). 4. Click **Create**. **Cloud Operations API** Use the [`CreateProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects) to create a Project. The structure of the `CreateProjectRequest` is defined by a JSON spec, as follows: ``` "spec": { "description": "", "displayName": "", "lifecycle": { "enableDeleteProtection": true } } ``` In this call, `description` is a string description of the Project, `displayName` is the Project name to be displayed in the UI, and `enableDeleteProtection` is a Boolean that indicates whether the Project can be deleted. After creating the Project, add new Namespaces and other supported resources within it. You cannot yet migrate resources from one Project to another. ## Add Resources to a Project After creating a Project, you can create Project-scoped resources such as Namespaces, Nexus Endpoints, or Connectivity Rules within it. **Cloud UI** In Temporal Cloud UI: 1. Open the Project. 2. Select **Namespaces** (or **Nexus Endpoints**) in the left navigation. 3. Click **Create Namespace** (or **Nexus Endpoints**) to add to the project. **Connectivity rules** cannot currently be created via the UI. Connectivity rules and the Namespace they attach to must reside in the same Project. **Cloud Operations API** The process for creating a [Namespace](/cloud/namespaces#create-a-namespace), [Nexus Endpoint](/nexus/registry#view-and-manage-nexus-endpoints), or [Connectivity Rule](/cloud/connectivity#creating-a-connectivity-rule) with the Cloud Operations API is the same as the standard process for creating those resources, with the addition of the `projectId` for the Project in which you're creating the resource. Use the [`CreateNamespace` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces) to create a new Namespace within a Project. For example: ``` curl -sS -X POST "$BASE/cloud/namespaces" \ -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \ -d '{ "projectId": "'"$PROJECT_ID"'", "spec": { "name": "payments-prod", "regions": ["aws-us-west-2"], "retentionDays": 7, "apiKeyAuth": { "enabled": true } } }' ``` Use the [`CreateCreateNexusEndpoint` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/nexus/POST/cloud/nexus/endpoints) to create a new Nexus Endpoint within a Project. For example: ``` curl -sS -X POST "$BASE/cloud/nexus/endpoints" \ -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \ -d '{ "projectId": "'"$PROJECT_ID"'", "spec": { "name": "order-fulfillment", "targetSpec": { "workerTargetSpec": { "namespaceId": "payments-prod.a1b2c3", "taskQueue": "nexus-handler" } }, "policySpecs": [ { "allowedCloudNamespacePolicySpec": { "namespaceId": "checkout-prod.a1b2c3" } } ] } }' ``` In this example, `payments-prod.a1b2c3` is the target Namespace (handler) and `checkout-prod.a1b2c3` is an allowed caller Namespace. Those Namespaces can be in different Projects from each other and from the Endpoint. Runtime access is the caller allowlist, not Project membership. See [Use Nexus across Projects](#use-nexus-across-projects). Use the [`CreateConnectivityRule` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/connectivity-rules/POST/cloud/connectivity-rules) to create a new Connectivity Rule within a Project. For example, for a private Connectivity Rule on AWS: ``` curl -sS -X POST "$BASE/cloud/connectivity-rules" \ -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \ -d '{ "projectId": "'"$PROJECT_ID"'", "spec": { "privateRule": { "connectionId": "vpce-0123456789abcdef0", "region": "aws-us-west-2" } } }' ``` Or for a public Connectivity Rule: ``` curl -sS -X POST "$BASE/cloud/connectivity-rules" \ -H "$H_AUTH" -H "$H_VER" -H "Content-Type: application/json" \ -d '{ "projectId": "'"$PROJECT_ID"'", "spec": { "publicRule": { "enableStableIps": true } } }' ``` Then attach the Connectivity Rule to a Namespace within the same project via [UpdateNamespace](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/namespaces/POST/cloud/namespaces/{namespace}). Creating the Connectivity Rule alone does not bind it to a Namespace. ## Use Nexus across Projects Projects group Nexus Endpoints for organization and authorization. They do not change how Nexus Operations run. - A Nexus Endpoint lives in one Project. That Project controls who can [create, list, update, and delete](/cloud/manage-access/permissions-reference#cloud-ops-api-permissions-2) the Endpoint. - Runtime access is unchanged: the Endpoint [allowlist of caller Namespaces](/nexus/security#runtime-access-controls). A caller Namespace in another Project can invoke the Endpoint if it is on that allowlist. - The target Namespace (handler) does not have to be in the same Project as the Endpoint. Unlike [Connectivity Rules](/cloud/connectivity#connectivity-rules), Nexus does not require the Endpoint and its Namespaces to share a Project. Example: an Endpoint in Project `payments`, a target Namespace in `payments`, and a caller Namespace in `checkout`. That call is allowed when the `checkout` Namespace is on the Endpoint allowlist. A user who can only list Endpoints in `checkout` will not see the Endpoint in `payments`. ## Manage Project Access Users, groups, and Account-scoped Service Accounts can be granted Project-level roles. Each [Project role](#project-roles) applies to the Project and, depending on the role, may grant inherited access to resources within it. **Cloud UI** To manage Project access in the Temporal Cloud UI: 1. Open the Project. 2. Click the **Project Identities** tab in the left navigation. The Project Identities page has three tabs, for Users, Service Accounts, and Groups. 3. Click the **Manage Identities** button to add (or remove) users, groups or service accounts, and assign desired Project Roles from the drop-down list. You can also modify or remove an existing principal's role in the same view. **Cloud Operations API** Use the [`SetUserProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/users/{userId}/access) to add Users to a Project. Use the [`SetUserGroupProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/user-groups/{groupId}/access) to add User Groups to a Project. Both calls take two required parameters: - `projectId` - The ID of the Project to set permissions for. - `userId` or `groupId` - The ID of the User or Group to set permissions for. Use the `role` attribute to set the role for the User or Group. See [Project roles](#project-roles) for more information. ``` "access": { "role": "" }, ``` Similar to [Namespace-scoped Service Accounts](/cloud/manage-access/service-accounts#scoped), a Project-scoped Service Account is an identity bound (or scoped) to a single Project. Use it for workers, CI/CD pipelines, Terraform, or operational automation that should not access resources outside that Project. Project-scoped Service Accounts are not created automatically with Project creation. Create them only when needed. **Cloud UI** To create a Project-scoped service account in this Project, click **Project Identities** in the left navigation. Then click the **Service Accounts** tab to create a Project-scoped service account. **Cloud Operations API** Use the [`SetServiceAccountProjectAccess` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}/service-accounts/{serviceAccountId}/access) to add Service Accounts to a Project. The call takes two required parameters: - `projectId` - The ID of the Project to set permissions for. - `serviceAccountId` - The ID of the Service Account to set permissions for. Use the `role` attribute to set the role for the User or Group. See [Project roles](#project-roles) for more information. ``` "access": { "role": "" }, ``` After creating the Service Account, you can optionally create one or more API keys for it. Each API key inherits only the permissions of its associated Service Account. If the Service Account is deleted, all of its API keys are deleted automatically.Project-scoped Service Accounts count toward the Project's Service Account limit. A Project-scope Service Account: - Can only bound (or scope to) exactly one Project. Each project can contain multiple Project-scoped Service Accounts. - Cannot be moved to another Project after it is created. - Counts toward the Project's Service Account limit. - Must be deleted before the Project can be deleted. Unlike an [Account-scoped Service Account](/cloud/manage-access/service-accounts), a Project-scoped Service Account cannot access resources outside its Project. For automation that needs access across multiple Projects, use an Account-scoped Service Account and grant it the appropriate Project roles. ## Project roles The following roles are available at the Project level for users, groups, and service accounts: - **Project Admin:** Complete access to the Project resource itself and all resources in it. - **Project Write:** Write access to all the Namespaces and other resources in a Project, but cannot add other users. - **Project Read:** Read-only access to all the resources in a Project, including viewing the actual contents of a Workflow. - **Project Contribute:** Can create new Namespaces, and list all Project resources, but without access to Workflow contents. - **Project List:** List-only access to metadata for all the resources in a Project, but without access to Workflow contents. - **Project Member:** Minimal Project-level access; used when a user already has explicit Namespace-level access and only needs to be represented at the Project level > **⚠️ Warning:** > Project Developer > > Project Developer is a compatibility role that is automatically inherited by users with the Account Developer role. It allows them to create Namespaces (and manage them) without automatically granting access to all existing Namespaces in the Project. This role cannot be assigned directly. For new role assignments, assign Project Write or Project Contribute as appropriate. > A principal automatically inherits a Project-level access from their Account-level role. For example, a Global Admin automatically inherits the Project Admin access for every Project in the Account. Project-level permissions are additive. Granting a Project-level role does not reduce permissions inherited from Account-level or Namespace-level roles. A principal's effective permissions are the union of all applicable Account-, Project-, and Namespace-level role assignments. ## Modify an existing Project **Cloud UI** To modify an existing Project, click **Project Settings** in the left navigation. The Project Settings page appears, with the General tab selected by default. Here, you can change the name or description of your Project. You can also access your Project's unique ID. You can copy the ID, but you can't change it. **Cloud Operations API** Use the [`UpdateProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/POST/cloud/projects/{projectId}) to add Service Accounts to a Project. The call takes one required parameter: - `projectId` - The ID of the Project to set permissions for. The structure of the `UpdateProjectRequest` is defined by a JSON spec, as follows: ``` "spec": { "description": "", "displayName": "", "lifecycle": { "enableDeleteProtection": true } } ``` In this call, `description` is a string description of the Project, `displayName` is the Project name to be displayed in the UI, and `enableDeleteProtection` is a Boolean that indicates whether the Project can be deleted. ## Delete a Project A Project must be empty, meaning it contains zero Namespaces, Nexus Endpoints, Connectivity Rules, or Project-scoped Service Accounts, to be deleted. An Account must always contain at least one Project. If the Account contains two or more Projects, you can delete the Default Project. If the Default Project is deleted, Temporal does not automatically designate a new Default Project. If the Default Project is deleted, API calls that do not specify a Project (by providing a Project ID) will fail. **Cloud UI** To delete a Project, click **Project Settings** in the left navigation and then click the **Delete Project** tab. Click the **Delete** button to permanently delete the Project. Use the **Deletion Prevention** slider to set the Deletion Protection flag for this Project. Once set, the Project can't be deleted unless the flag is deliberately disabled. **Cloud Operations API** Use the [`DeleteProject` API call](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/projects/DELETE/cloud/projects/{projectId}) to delete a Project. The call takes one required parameter: - `projectId` - The ID of the Project to delete. This call will not succeed if the delete protection flag is enabled. Projects support a delete protection flag, similar to Namespaces. When this flag is enabled, the Project cannot be deleted until the flag is disabled. This flag is enabled for the Default Project. The flag is disabled for any new Project created by a user. This flag is not inherited by the Namespaces within the Project. Namespace delete protection can be managed per-Namespace. If a Project is the only one remaining in an Account, that project cannot be deleted, regardless of the status of the Deletion Protection Flag. --- # Service regions - Temporal Cloud Source: https://docs.temporal.io/cloud/regions > Temporal Cloud offers high availability and low latency across multiple cloud provider regions with adjustable throughput limits and robust latency targets. Contact us for more details. You can access Temporal Cloud from anywhere with Internet connectivity, no matter where your Temporal Cloud Namespaces are physically located. Your applications can live in the cloud environment or data center of your choice. With that in mind, you _will_ reduce latency by creating Namespaces in a region close to where you host your Workers. This page enumerates the current regions supported by Temporal Cloud Namespaces. > **💡 Tip:** > Service Availability > > Visit [status.temporal.io](https://status.temporal.io) to check the status of our supported regions. > On that page, you can also subscribe to updates to receive email notifications whenever Temporal creates, updates or resolves an incident. > ## AWS Service Regions Temporal Cloud operates in the following Amazon Web Services (AWS) regions: ### Asia Pacific - Tokyo (`ap-northeast-1`) - **Cloud API Code**: `aws-ap-northeast-1` - **Regional Endpoint**: `ap-northeast-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Seoul (`ap-northeast-2`) - **Cloud API Code**: `aws-ap-northeast-2` - **Regional Endpoint**: `ap-northeast-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Mumbai (`ap-south-1`) - **Cloud API Code**: `aws-ap-south-1` - **Regional Endpoint**: `ap-south-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Hyderabad (`ap-south-2`) - **Cloud API Code**: `aws-ap-south-2` - **Regional Endpoint**: `ap-south-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-southeast-1` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Singapore (`ap-southeast-1`) - **Cloud API Code**: `aws-ap-southeast-1` - **Regional Endpoint**: `ap-southeast-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-2` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Asia Pacific - Sydney (`ap-southeast-2`) - **Cloud API Code**: `aws-ap-southeast-2` - **Regional Endpoint**: `ap-southeast-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - **Multi-Cloud Replication**: - `gcp-asia-south1` ### Europe - Frankfurt (`eu-central-1`) - **Cloud API Code**: `aws-eu-central-1` - **Regional Endpoint**: `eu-central-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-west-1` - `aws-eu-west-2` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### Europe - Ireland (`eu-west-1`) - **Cloud API Code**: `aws-eu-west-1` - **Regional Endpoint**: `eu-west-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-central-1` - `aws-eu-west-2` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### Europe - London (`eu-west-2`) - **Cloud API Code**: `aws-eu-west-2` - **Regional Endpoint**: `eu-west-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-eu-central-1` - `aws-eu-west-1` - **Multi-Cloud Replication**: - `gcp-europe-west3` ### North America - Central Canada (`ca-central-1`) - **Cloud API Code**: `aws-ca-central-1` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` - **Regional Endpoint**: `ca-central-1.aws.api.temporal.io:7233` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Northern Virginia (`us-east-1`) - **Cloud API Code**: `aws-us-east-1` - **Regional Endpoint**: `us-east-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-2` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Ohio (`us-east-2`) - **Cloud API Code**: `aws-us-east-2` - **Regional Endpoint**: `us-east-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-west-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### North America - Oregon (`us-west-2`) - **Cloud API Code**: `aws-us-west-2` - **Regional Endpoint**: `us-west-2.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` - **Same Region Replication**: Available - **Multi-Region Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - **Multi-Cloud Replication**: - `gcp-us-central1` - `gcp-us-west1` - `gcp-us-east4` ### South America - São Paulo (`sa-east-1`) - **Cloud API Code**: `aws-sa-east-1` - **Regional Endpoint**: `sa-east-1.aws.api.temporal.io:7233` - **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - None ## GCP Service Regions Temporal Cloud operates the following Google Cloud (GCP) regions: ### North America - Iowa (`us-central1`) - **Cloud API Code**: `gcp-us-central1` - **Regional Endpoint**: `us-central1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-west1` - `gcp-us-east4` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### North America - Oregon (`us-west1`) - **Cloud API Code**: `gcp-us-west1` - **Regional Endpoint**: `us-west1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-central1` - `gcp-us-east4` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### North America - Northern Virginia (`us-east4`) - **Cloud API Code**: `gcp-us-east4` - **Regional Endpoint**: `us-east4.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - `gcp-us-central1` - `gcp-us-west1` - **Multi-Cloud Replication**: - `aws-ca-central-1` - `aws-us-east-1` - `aws-us-east-2` - `aws-us-west-2` ### Europe - Frankfurt (`europe-west3`) - **Cloud API Code**: `gcp-europe-west3` - **Regional Endpoint**: `europe-west3.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - `aws-eu-central-1` - `aws-eu-west-1` - `aws-eu-west-2` ### Asia Pacific - Mumbai (`asia-south1`) - **Cloud API Code**: `gcp-asia-south1` - **Regional Endpoint**: `asia-south1.gcp.api.temporal.io:7233` - **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - `aws-ap-northeast-1` - `aws-ap-northeast-2` - `aws-ap-south-1` - `aws-ap-south-2` - `aws-ap-southeast-1` - `aws-ap-southeast-2` ### Asia Pacific - Jakarta (`asia-southeast2`) - **Cloud API Code**: `gcp-asia-southeast2` - **Regional Endpoint**: `gcp-asia-southeast2.region.tmprl.cloud` - **Private Service Connect Service Attachment URI**: `projects/prod-bsbyrfwqqq885qkcr3s43y524/regions/asia-southeast2/serviceAttachments/pl-c3ayi` - **Same Region Replication**: Not Available - **Multi-Region Replication**: - None - **Multi-Cloud Replication**: - None --- # Outages and Recovery Objectives (RTO / RPO) Source: https://docs.temporal.io/cloud/rpo-rto When a cloud outage disrupts a Namespace, Temporal Cloud takes measures to maintain the Namespace's availability and data durability. The time it takes to recover from the outage is called the _recovery time_. The _recovery point_ is how far back in time data must be recovered from after an outage. A durable system should have a low recovery time and a near recovery point. Temporal Cloud publishes goals for the recovery time and recovery point for each kind of outage. These goals are called the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). For details on how each is measured, see [How RTO and RPO are measured](#how-rto-and-rpo-are-measured). These objectives are complementary to Temporal Cloud's [Service Level Agreement (SLA)](/cloud/sla). The RTO and RPO for a Namespace depend on the type of outage and which [High Availability](/cloud/high-availability) features the Namespace has enabled. Your real-world recovery time also depends on your [Worker deployment pattern](/cloud/high-availability/architecture-patterns) — how quickly Workers resume processing in the new region after the Namespace fails over. ## RTO and RPO summary The following table summarizes the RTO and RPO targets for each type of outage. These targets apply to Namespaces that have automatic failovers enabled, which is the default. Automatic failovers are triggered by Temporal's tooling and on-call engineers without user action. Users can always initiate a failover independently. In an outage, a user-initiated failover will not cancel out or reverse an automatic failover. These targets are for unplanned cloud outages and do not apply to user-initiated failovers during healthy periods, such as DR drills. Read about [triggering a failover](/cloud/high-availability/failovers/manage#trigger-failover) to see how a Namespace failover performs during healthy periods. | Outage type | Applicable Namespaces | RPO | RTO | | ----------------------------------------------------- | --------------------------------------------------------------------- | -------------- | ---------------- | | [Availability Zone outage](#availability-zone-outage) | All Namespaces | Zero | Near-zero | | [Cell outage](#cell-outage) | Namespaces with Same-region, Multi-region, or Multi-cloud Replication | Under 1 minute | Under 20 minutes | | [Cloud Region outage](#cloud-region-outage) | Namespaces with Multi-region or Multi-cloud Replication | Under 1 minute | Under 20 minutes | | [Cloud-wide outage](#cloud-wide-outage) | Namespaces with Multi-cloud Replication | Under 1 minute | Under 20 minutes | > **💡 Tip:** > > Temporal highly recommends keeping automatic failovers enabled. When automatic failovers are > _disabled,_ Temporal Cloud cannot set an RPO and RTO for that Namespace, because it cannot control when or if the user > will trigger a failover. > As soon as a cloud outage resolves, Temporal's on-call engineers work to restore service to Namespaces that were not protected by High Availability. A cloud outage can leave lingering effects in Temporal's systems and applications, even after the cloud provider restores the underlying service. An affected Namespace's outage may last longer than the cloud provider's outage. All Namespaces are backed up every 4 hours. If an outage causes data loss on a Namespace that was not protected by High Availability, Temporal uses the backup to restore as much data as feasible. ## Outage types and their RTO/RPO The following sections explain each type of outage in more detail, including the blast radius, Temporal Cloud features that mitigate the outage, and whether the outage is included in the SLA calculation. ### Availability Zone outage An [Availability Zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones) (AZ) is akin to an isolated datacenter managed by a cloud hyperscaler, with independent power, networking, and cooling infrastructure. Each cloud region contains multiple AZs, and an individual AZ can fail due to events such as hardware failure, power loss, or a localized network partition. AZ outages are the most common type of outage, and Temporal Cloud has weathered many of them transparently. **Blast Radius:** A single Availability Zone within a single cloud region. Because every Namespace's components are spread across at least three AZs, the blast radius to Temporal Cloud users is typically zero — Namespaces stay operational with little to no downtime. > **⚠️ Caution:** > > While Temporal Cloud can withstand single AZ outages without disruption, if you have Workers that are deployed in the > impacted AZ, those Workers may be disrupted. To mitigate this risk, Temporal recommends deploying your Workers across > multiple AZs. > **Mitigation:** Every Namespace is automatically spread across at least three Availability Zones, and any Namespace can handle a single AZ failure without disruption to end-user Temporal operations. [High Availability](/cloud/high-availability) features are _not_ required to keep Temporal Cloud operations running through an AZ outage. **SLA inclusion:** Included in the [SLA](/cloud/sla) calculation. Any errors during an AZ outage count toward SLA credits, since AZ resilience is within Temporal's responsibility. If two AZs fail simultaneously, Temporal Cloud treats the event as a [Cloud Region outage](#cloud-region-outage). In that case, Namespaces in the region may be impacted, including those using [Same-region Replication](/cloud/high-availability#same-region-replication). > **ℹ️ Info:** > > When an AZ fails, Temporal may also trigger a failover on Namespaces that have High Availability enabled, as a > precaution in case the outage scope expands. For Multi-region and Multi-cloud Replication, you can opt out of this > behavior by [disabling automatic failovers](/cloud/high-availability/enable#automatic-failovers) on the Namespace. > Same-region Replication Namespaces always fail over automatically and cannot opt out. > #### RTO and RPO When using Temporal Cloud (no additional features required): - **Near-zero RTO.** When a single AZ fails, the remaining two AZs continue serving requests without a failover, so end users see little to no disruption. - **Zero RPO.** Writes to Workflow state are synchronously replicated across all three AZs before being acknowledged back to the Client, so an AZ failure cannot cause data loss. ### Cell outage Temporal Cloud runs on a [cell architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html). Each cell contains the software and services necessary to host a Namespace, and components within a cell are distributed across at least three Availability Zones. Cells provide a strong unit of isolation: a problem inside one cell does not propagate to other cells. A cell outage occurs when a cell becomes degraded or unavailable, disrupting the Namespaces hosted within it. **Blast Radius:** One cell, and the Namespaces within that cell, within a single region. Even though your Workers will remain healthy, they will not be able to process Workflows because the Namespace is down. **Mitigation:** [Multi-region Replication](/cloud/high-availability) and [Multi-cloud Replication](/cloud/high-availability) replicate a Namespace into another cell in a different region or different cloud provider. [Same-region Replication](/cloud/high-availability) replicates a Namespace into another cell within the same region. When any of these features are enabled for a Namespace, an outage that disrupts a single cell can be mitigated by failing the Namespace over to its replica. **SLA inclusion:** Included in the [SLA](/cloud/sla) calculation. Any errors during a cell outage count toward SLA credits, since mitigating cell outages is within Temporal's responsibility. Cell-level disruptions occur from time to time, and Temporal's replication and failover tooling has restored affected Namespaces in real-world incidents. #### RTO and RPO When using Same-region Replication, Multi-region Replication, or Multi-cloud Replication for automatic failover: - **RTO under 20 minutes.** Temporal detects the disruption and fails the Namespace over to its replica cell. - **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active cell. Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and Conflict Resolution process, which reconciles state between the active and replica when the outage is over. ### Cloud Region outage A cloud region as a whole can become degraded, with effects that span beyond any single cell or Availability Zone. **Blast Radius:** All Namespaces and Workers within a single cloud region are potentially affected. **Mitigation:** [Multi-region Replication](/cloud/high-availability) and [Multi-cloud Replication](/cloud/high-availability) place the replica outside the affected region, so a Namespace can fail over and continue serving Workflows. Same-region Replication does not protect against a Cloud Region outage, since the replica resides in the same region. **SLA inclusion:** Included in the [SLA](/cloud/sla) calculation only for Namespaces that have Multi-region Replication or Multi-cloud Replication enabled with automatic failovers — in those cases, Temporal can mitigate the outage. For Namespaces without these features, a Cloud Region outage is excluded from the SLA calculation, as it is beyond Temporal's control to mitigate. If two or more regions in the same cloud provider experience an outage simultaneously, Temporal Cloud treats the event as a [Cloud-wide outage](#cloud-wide-outage). Regional outages are less common than cell or AZ outages, but they do happen. During the [AWS us-east-1 incident on October 20, 2025](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025), Temporal Cloud's regional failover kept customer Namespaces running. #### RTO and RPO When using Multi-region Replication or Multi-cloud Replication for automatic failover: - **RTO under 20 minutes.** Temporal detects the regional disruption and fails the Namespace over to its replica in another region. - **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active region. Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and Conflict Resolution process, which reconciles state between the active and replica when a failover occurs. ### Cloud-wide outage On rare occasions, an issue affects two or more regions of a single cloud provider at once. Any simultaneous outage of two or more regions in the same cloud provider is treated as a cloud-wide outage. **Example causes:** a software bug rolled out to every region of a cloud provider that triggers cascading failures across the provider's infrastructure, or two or more regions in the same cloud experiencing independent regional outages at the same time. **Blast Radius:** Most or all regions of a single cloud provider. Every Namespace and every Worker hosted in that cloud is potentially affected. **Mitigation:** [Multi-cloud Replication](/cloud/high-availability) places the replica in a different cloud provider entirely, so the Namespace can fail over even when an entire cloud provider goes down. **SLA inclusion:** Included in the [SLA](/cloud/sla) calculation only for Namespaces that have Multi-cloud Replication enabled with automatic failovers — in those cases, Temporal can mitigate the outage. For Namespaces without this feature, a cloud-wide outage is excluded from the SLA calculation, as it is beyond Temporal's control to mitigate. Cloud-wide outages are the rarest category, but they [have occurred](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Multi-cloud Replication is designed to keep Namespaces running through such events. #### RTO and RPO When using Multi-cloud Replication for automatic failover: - **RTO under 20 minutes.** Temporal detects the cloud-wide disruption and fails the Namespace over to its replica in a different cloud provider. - **RPO under 1 minute.** Asynchronous replication keeps the replica close to the active region, even across cloud providers. Even though the RPO target is under 1 minute, data is virtually never "lost" thanks to Temporal's built-in Recovery and Conflict Resolution process, which reconciles state between the active and replica when a failover occurs. ## How RTO and RPO are measured Temporal Cloud achieves its RTO and RPO targets through [High Availability](/cloud/high-availability) replication. The following sections explain how each metric is measured and what factors can affect them. ### RPO Unlike a traditional database where data within the recovery point window may be permanently lost, Temporal Cloud durably persists all acknowledged data. After an outage resolves, Temporal's Recovery and Conflict Resolution process automatically syncs data back into the Namespace. The RPO therefore reflects the maximum data that may be _temporarily unavailable_ in the replica at the moment of failover, not data that is permanently lost. Temporal keeps replicas up to date using [asynchronous replication](https://youtu.be/mULBvv83dYM?si=RDeWb3gVsEtgGM4z&t=334), with monitoring, alerting, and internal SLOs on replication lag for every Namespace. User actions on a Namespace can affect the recovery point. For example, suddenly spiking into much higher throughput than a Namespace has seen before could create a period of replication lag where the replica falls behind the active. Temporal provides a [replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) metric for each Namespace. This metric approximates the recovery point the Namespace would achieve in a worst-case failure at that moment. Temporal recommends monitoring the replication lag and alerting if it rises above 1 minute. ### RTO The Recovery Time for a given incident is measured from the moment the incident begins to cause abnormal Namespace operation — for example, when unavailability or error rates rise above an acceptable level — to the moment the Namespace is restored to full functionality. For most incidents, the vast majority of the Recovery Time is spent detecting the incident, determining the affected boundary (a single cell, a region, or an entire cloud), and deciding to fail Namespaces over to their replicas. The actual time to complete the failover is usually a very small piece of the Recovery Time. This Recovery Time covers only the Temporal Namespace. Your application's overall Recovery Time also depends on having enough healthy Workers that can reach the Namespace and process Workflows. Maintaining sufficient Worker capacity that can reach the replica region (or replica cloud) during a failover is your responsibility. You are also responsible for failing over any other regional dependencies your application relies on, such as replicated application databases. ## Tips for a lower Recovery Time To achieve the lowest possible recovery times, Temporal recommends that you: - Keep automatic failovers enabled on your Namespace (the default) - Invest in a process to detect outages and trigger a manual failover. You can trigger manual failovers on your Namespaces even if automatic failovers are enabled. There are several benefits to combining a manual failover process with automatic failovers: - You can detect outages that Temporal doesn't. In the cloud, regional outages don't affect all services equally. It's possible that Temporal--and the services it depends on--are unaffected by the outage, even while your Workers or other cloud infrastructure are disrupted. If you [monitor services in your critical path](https://sre.google/sre-book/monitoring-distributed-systems/) and alert on unusual error rates, you may catch outages before Temporal Cloud does. - You can sequence your failovers in a particular order. Your cloud infrastructure probably contains more pieces than just your Temporal Namespace: Temporal Workers, compute pools, data stores, and other cloud services. If you manually fail over, you can choose the order in which these pieces switch to the replica region. You can then test that ordering with failover drills and ensure it executes smoothly without data consistency issues or bottlenecks. - You can proactively fail over more aggressively than Temporal. While the 20-minute RTO should be sufficient for most use cases, some may strive to hit an even lower RTO. For workloads like high frequency trading, auctions, or popular sporting events, an outage at the wrong time could cause tremendous lost revenue per minute. You can adopt a posture that fails over more eagerly than Temporal does. For example, you could trigger a manual failover at the first sign of a possible disruption, before knowing whether there's a true regional outage. - Even if you have robust tooling to detect an outage and trigger a failover, leaving automatic failovers enabled provides a "safety net" in case your automation misses an outage. It also gives Temporal leeway to preemptively fail over your Namespace if we detect that it may be disrupted soon, for example, by a rolling failure that has impacted other Namespaces but not yours, yet. ## Comparing RTO and SLA Temporal has both a Recovery Time Objective (RTO) and a Service Level Agreement (SLA). They serve complementary purposes and apply in different situations. | Aspect | RTO | SLA | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | What is it? | An objective, or high-priority goal, for the total time that an outage disrupts a Namespace. | A contractual agreement that sets an upper bound on the service error rate, with financial repercussions. | | How is it measured? | The achieved recovery time is measured in terms of minutes per outage. | The achieved service error rate is measured in terms of error rate per month. | | How is the calculation performed? | The achieved recovery time in a given outage is the total time between when a disruption to a Namespace began and when the Namespace was restored to full functionality, either after a failover to a healthy region or after the outage has been mitigated. | Temporal measures the percentage of requests to Temporal Cloud that fail, and applies a [formula](/cloud/sla) to get the final percentage for the month. | | Do partial degradations count? | Most outages contain periods of **partial degradation** where some percentage of Namespace operations fail while the rest complete as normal. When they disrupt a Namespace, periods of partial degradation count in the calculation of the recovery time. | Partial degradations only partially count for the service error rate calculation. A 5-minute window with a 10% error rate would count less than a 5-minute window with a 100% error rate. | | What is excluded? | For partial degradations, what counts as a disruption to a Namespace is subject to Temporal's expert judgment, but a good rule of thumb is a service error rate >=10%. | We exclude outages that are out of Temporal's control to mitigate, for example, a failure of the underlying cloud provider infrastructure that affects a Namespace without High Availability and automatic failovers enabled. If a Namespace has the relevant High Availability feature and has automatic failovers enabled, then Temporal can act to mitigate the outage and it does usually count against the SLA. Full exclusions on the [SLA page](/cloud/sla). | The following examples illustrate the RTO and SLA calculations for different types of outages in a regional outage. These hypothetical Namespaces are based on actual Temporal Cloud performance in a [real-world outage](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025). Suppose that region `middle-earth-1` experienced a cascading failure starting at 10:00:00 UTC, causing various instances and machines to fail over time. Temporal's automatic failover triggered for all Namespaces and completed at 10:15:00 UTC. - Namespace 0 was in the region but its cell was not affected by the outage. The only downtime it had was for a few seconds during the failover operation. It experienced a near-zero Recovery Time, and its service error rate was negligible. Graceful failover was successful, and this Namespace achieved a recovery point of 0. - Namespace 1_A was in the region and its cell experienced a partial degradation that caused 10% of requests to fail in the first 5 minutes, 25% in the second five minutes, and 50% in the third five minutes. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( (1 - 10%) + (1 - 25%) + (1 - 50%) + 8925 \* 100% ) / 8928 = 99.990%. (Note: there are 8928 5-minute periods in a 31-day month.) Graceful failover was successful, and this Namespace achieved a recovery point of 0. - Namespace 1*B was in the same cell as Namespace 2_A, so it also experienced a partial degradation that caused 10% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually fail over at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 * (1 - 10%) + 8927 \_ 100% ) / 8928 = 99.998%. Graceful failover was successful, and this Namespace achieved a recovery point of 0. - Namespace 2*A was in the region and its cell was fully network partitioned at the start of the outage, causing 100% of requests to fail. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( 3 * (1 - 100%) + 8928 \_ 100% ) / 8640 5-minute periods per month = 99.97%. Because the Namespace was network partitioned, graceful failover did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the time of the network partition, which was a few seconds. - Namespace 2*B was in the region and was fully network partitioned, causing 100% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually fail over at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 5-minute periods * (1 - 100%) + 8639 5-minute periods \_ 100% ) / 8640 5-minute periods per month = 99.99%. Because the Namespace was network partitioned, graceful failover did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the time of the network partition, which was a few seconds. All of the above Namespaces were in the affected region and beat the 1-minute RPO. But they achieved varying recovery times and service error rates. - Notice how Namespace 1_A and Namespace 2_A were both automatically failed over with **the same recovery time but different service error rates**. Notice how Namespace 2_B and Namespace 1_A happen to have the **same service error rate but different recovery times**. This illustrates how RTO and SLA can differ, even in the same outage. Both are valuable tools for Temporal Cloud users to measure the availability of their Namespaces. - Notice how the Namespaces that were manually failed over (Namespace 1_B and Namespace 2_B) achieved lower recovery times than the Namespaces that were automatically failed over (Namespace 1_A and Namespace 2_A). This illustrates how **proactive, aggressive manual failover can achieve a better recovery time than automatic failover**. --- # Security model - Temporal Cloud Source: https://docs.temporal.io/cloud/security > Temporal Cloud provides robust security for applications, data, and its platform with features like mTLS, client-side encryption, PrivateLink, and SOC 2 Type 2 compliance. **What kind of security does Temporal Cloud provide?** The security model of [Temporal Cloud](/cloud) encompasses applications, data, and the Temporal Cloud platform itself. > **ℹ️ Info:** > General platform security > > For information about the general security features of Temporal, see our [Platform security page](/security). > ## Application and data **What is the security model for applications and data in Temporal Cloud?** ### Code execution boundaries Temporal Cloud provides the capabilities of Temporal Server as a managed service; it does not manage your applications or [Workers](/workers#worker). Applications and services written using [Temporal SDKs](/encyclopedia/architecture/temporal-sdks) run in your computing environment, such as containers (Docker, Kubernetes) or virtual machines (in any hosting environment). You have full control over how you secure your applications and services. ### Data Converter: Client-side encryption The optional [Data Conversion](/dataconversion) capability of the Temporal Platform lets you transparently encrypt data before it's sent to Temporal Cloud and decrypt it when it comes out. Data Conversion runs on your Temporal Workers and [Clients](/encyclopedia/temporal-client); Temporal Cloud cannot see or decrypt your data. If you use this feature, data stored in Temporal Cloud remains encrypted even if the service itself is compromised. By deploying a [Codec Server](/production-deployment/data-encryption) you can securely decrypt data in the [Temporal Web UI](/web-ui) without sharing encryption keys with Temporal. ## The platform **What is the security model for the Temporal Cloud platform?** ### Namespace isolation The base unit of isolation in a Temporal environment is a [Namespace](/namespaces). Each Temporal Cloud account can have multiple Namespaces, and each Namespace is isolated to ensure your workloads remain secure and performant. #### Authentication Each Namespace is secured with your choice of authentication method: - **mTLS certificates** - Namespace-specific X.509 certificates for mutual TLS authentication - **API keys** - Namespace-scoped API keys for authentication See [API Keys](/cloud/api-keys) and [mTLS Certificates](/cloud/certificates) for more details on configuring authentication for your Namespace. #### Rate limiting Temporal Cloud protects each Namespace with separate rate limits to prevent noisy neighbor problems: - **Actions Per Second (APS)** - Limits the rate of [actions](/best-practices/managing-aps-limits) performed in your Workflows - **Operations Per Second (OPS)** - Limits the rate of all [operations](/references/operation-list) that create load on Temporal Server These per-Namespace rate limits ensure that one Namespace experiencing a traffic spike cannot impact the performance or reliability of other Namespaces, whether those Namespaces belong to a single Temporal Cloud account or separate ones. See [Rate limiting](/cloud/limits) for more information about Temporal Cloud limits, and [Monitoring trends against limits](/cloud/service-health#rps-aps-rate-limits) for monitoring best practices. #### Inter-Namespace communication Namespaces are isolated by default. The only way for Workflows in one Namespace to interact with Workflows in another Namespace is through [Temporal Nexus](/nexus), which provides controlled, secure cross-Namespace communication via Nexus Endpoints. See [Nexus Security](/nexus/security) for details on how Nexus enables secure inter-Namespace communication. #### Logical segregation Temporal Cloud is a multi-tenant service. Namespaces in the same environment are logically segregated. Namespaces do not share data processing or data storage across regional boundaries. ### Private Connectivity Temporal Cloud supports private connectivity to enable you to connect to Temporal Cloud from a secured network. See the [Connectivity](/cloud/connectivity) page for more information and details about using AWS PrivateLink and GCP Private Service Connect with Temporal Cloud. ### Temporal Nexus Like Namespaces, the Nexus Registry is account-scoped and global within a Temporal Cloud account. Nexus Endpoint names remain unique Account-wide, but each Endpoint is Project-scoped for management. An Account Developer (or higher), or a user with Project Contribute (or higher) on the Endpoint's Project, can manage (create, update, or delete) an Endpoint only when they also have Namespace Admin on its target Namespace. All users with a Read-only role (or higher) in an account, can view and browse the full list of Endpoints. Runtime access from a Workflow in a caller Namespace to a Nexus Endpoint is controlled by an allowlist policy (of caller Namespaces) for each Endpoint in the Nexus API registry. Workers authenticate with Temporal Cloud as they do today with mTLS certificates or API keys as allowed by the Namespace configuration. Nexus requests are sent from the caller’s Namespace to the handler’s Namespace over a secure multi-region mTLS Envoy mesh. For payload encryption, the DataConverter works the same for a Nexus Operation as it does for other payloads sent between a Worker and Temporal Cloud. See [Nexus Security](/nexus/security) for more information. ### Encryption > **💡 Tip:** > TLS vs mTLS > > **TLS** (Transport Layer Security) encrypts data in transit. **mTLS** (mutual TLS) is an authentication method where both client and server present certificates to verify identity. All Temporal Cloud connections use TLS encryption. When you choose "mTLS authentication," you're choosing how to prove your identity, not whether your connection is encrypted. > **In transit**: All connections to Temporal Cloud use TLS 1.3 encryption, regardless of your authentication method ([API keys](/cloud/api-keys) or [mTLS certificates](/cloud/certificates)). **At rest**: Data is stored in two locations: an Elasticsearch instance (used when filtering Workflows in SDK clients, the [CLI](/cloud/tcld), or the Web UI) and the core Temporal Cloud persistence layer. Both are encrypted at rest with AES-256-GCM. ### Identity Authentication to Temporal Cloud gRPC endpoints supports two methods: - **[API keys](/cloud/api-keys)**: Identity-based authentication using bearer tokens. Recommended for most use cases. - **[mTLS certificates](/cloud/certificates)**: Mutual TLS authentication using client certificates issued by your CA. Both methods provide secure, encrypted connections to Temporal Cloud. Choose based on your organization's security requirements and key management preferences. For user authentication to the Temporal Cloud UI, see [How to manage SAML authentication with Temporal Cloud](/cloud/manage-access/saml). ### Access Authorization is managed at the account and Namespace level. Users and systems are assigned one or more preconfigured roles. Users hold [account-level Roles](/cloud/manage-access/roles-and-permissions#account-level-roles) of administrators, developers, and read-only users. Systems and applications processes hold their own distinct roles. ### Monitoring In addition to extensive system monitoring for operational and availability requirements, we collect and monitor audit logs from the AWS environment and all calls to the gRPC API (which is used by the SDKs, CLI, and Web UI). These audit logs can be made available for ingestion into your security monitoring system. ### Testing We contract with a third party to perform a full-scope pentest (with the exception of social engineering) annually. Additionally, we perform targeted third-party and internal testing on an as-needed basis, such as when a significant feature is being released. ### Internal Temporal access We restrict access to production systems to the small team of employees who maintain our production infrastructure. We log all access to production systems; shared accounts are not allowed. Access to all production systems is through SSO, with MFA enabled. Access to our cloud environments is granted only for limited periods of time, with a maximum of 8 hours. (For more information, see the blog post [Rolling out access hours at Temporal](https://temporal.io/blog/rolling-out-access-hours-at-temporal).) All Temporal engineering systems are secured by GitHub credentials, which require both membership in the Temporal GitHub organization and MFA. Access grants are reviewed quarterly. ### Compliance Temporal Technologies is SOC 2 Type 2 certified and compliant with GDPR and HIPAA regulations. Compliance audits are available by request through our [Contact](https://pages.temporal.io/contact-us) page. --- # Service availability - Temporal Cloud Source: https://docs.temporal.io/cloud/service-availability > Temporal Cloud offers high availability and low latency across multiple cloud provider regions with adjustable throughput limits and robust latency targets. Contact us for more details. The operating envelope of Temporal Cloud includes throughput, latency, and limits. Service regions are listed on [this page](/cloud/regions). If you need more details, [contact us](https://pages.temporal.io/contact-us). ## Throughput expectations **What kind of throughput can I get with Temporal Cloud?** Each Namespace in Temporal has a rate limit, which is measured in [Actions](/cloud/pricing#action) per second. Temporal offers two different modes for adjusting capacity: On-Demand Capacity or Provisioned Capacity. With On-Demand Capacity, Namespace capacity is increased automatically along with usage. With Provisioned Capacity, you can control your capacity limits by requesting Temporal Resource Units (TRUs). ## Latency Service Level Objective (SLO) **What kind of latency can I expect from Temporal Cloud?** Temporal Cloud has a p99 latency SLO of 200ms per region. The same SLO for normal Worker requests (commands and polling) apply to Nexus in both the caller and handler Namespaces. ### Historical latency data Latency over a week-long period for starting and signaling Workflow Executions was as follows: #### August 2026 | Operation | p50 | p90 | p99 | | :--------------------------------- | :--: | :--: | ---: | | `StartWorkflowExecution` | 20ms | 32ms | 78ms | | `SignalWorkflowExecution` | 19ms | 42ms | 91ms | | `SignalWithStartWorkflowExecution` | 30ms | 47ms | 109ms | #### January 2026 | Operation | p50 | p90 | p99 | | :--------------------------------- | :----: | :--: | ---: | | `StartWorkflowExecution` | 14ms | 21ms | 69ms | | `SignalWorkflowExecution` | 11ms | 19ms | 46ms | | `SignalWithStartWorkflowExecution` | 19ms | 37ms | 95ms | #### March 2024 | Operation | p90 | p99 | | :--------------------------------- | :--: | ---: | | `StartWorkflowExecution` | 24ms | 54ms | | `SignalWorkflowExecution` | 14ms | 40ms | | `SignalWithStartWorkflowExecution` | 24ms | 61ms | Latency observed from the Temporal Client is influenced by other system components like the Codec Server, egress proxy, and the network itself. Also, concurrent operations on the same Workflow Execution may result in higher latency. --- # Monitor Temporal Cloud service health Source: https://docs.temporal.io/cloud/service-health > Use Temporal Cloud metrics to monitor service latency, errors, failures, throttling, and capacity limits. Temporal Cloud metrics help you monitor the service health of production deployments. This page covers recommended signals for monitoring Temporal Cloud. ## Monitor availability issues When you see a sudden drop in Worker resource utilization, verify whether Temporal Cloud's API is showing increased latency and error rates. ### Reference Metrics - [temporal\_cloud\_v1\_service\_latency\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_latency_p99) This metric measures latency for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations. These operations are mission critical and never [throttled](/cloud/service-availability#throughput). This metric is a good indicator of your lowest possible latency for the 99th percentile of requests. ### Workflow execution latency To monitor end-to-end Workflow execution time (not just the service API latency above), use the workflow schedule-to-close latency metrics: - [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p50](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p50) - [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p95](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p95) - [temporal\_cloud\_v1\_workflow\_schedule\_to\_close\_latency\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_workflow_schedule_to_close_latency_p99) These measure the time from when a Workflow is scheduled until it closes, including all Activity execution time. A sudden increase may indicate Worker capacity issues, downstream service degradation, or retry storms. ## Monitor Temporal Service errors Check for Temporal Service gRPC API errors. Note that Service API errors are not equivalent to guarantees mentioned in the [Temporal Cloud SLA](/cloud/sla). ### Reference Metrics - [temporal\_cloud\_v1\_service\_error\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_error_count) - [temporal\_cloud\_v1\_service\_request\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_count) ### Prometheus Query for this Metric Measure your daily average success rate over 10-minute windows. OpenMetrics v1 metrics are pre-computed rates. Use `sum()` to aggregate across dimensions rather than `increase()` or `rate()`. ``` avg_over_time(( ( ( sum(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}) - sum(temporal_cloud_v1_service_error_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}) ) / sum(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}) ) or vector(1) )[1d:1m]) ``` ## Detecting Activity and Workflow Failures The metrics `temporal_cloud_v1_activity_fail_count` and `temporal_cloud_v1_workflow_failed_count` together provide failure detection for Temporal applications. These metrics work in tandem to give you both granular component-level visibility and high-level workflow health insights. ### Activity failure cascade If not using infinite retry policies, Activity failures can lead to Workflow failures: ``` Activity Failure --> Retry Logic --> More Activity Failures --> Workflow Decision --> Potential Workflow Failure ``` Activity failures are often recoverable and expected. Workflow failures represent terminal states requiring immediate attention. A spike in activity failures may precede workflow failures. Generally Temporal recommends that Workflows should be designed to always succeed. If an Activity fails more than its retry policy allows, we suggest having the Workflow handle Activity failure and take action to notify a human to take corrective action or be aware of the error. ### Ratio-based monitoring #### Failure conversion rate Monitor the ratio of workflow failures to activity failures: ``` workflow_failure_rate = temporal_cloud_v1_workflow_failed_count / temporal_cloud_v1_activity_fail_count ``` What to watch for: - High ratio (greater than 0.1): Poor error handling - activities failing are causing workflow failures - Low ratio (less than 0.01): Good resilience - activities fail but workflows recover - Sudden spikes: May indicate systematic issues #### Activity success rate ``` activity_success_rate = temporal_cloud_v1_activity_success_count / (temporal_cloud_v1_activity_success_count + temporal_cloud_v1_activity_fail_count) ``` Target: >95% for most applications. Lower success rate can be a sign of system troubles. See also: - [Crafting an Error Handling Strategy](https://learn.temporal.io/courses/errstrat/) - [Temporal Failures reference](/references/failures) - [Detecting Workflow failures](/encyclopedia/detecting-workflow-failures) ## Monitor replication lag for Namespaces with High Availability features Replication lag refers to the transmission delay of Workflow updates and history events from the primary Namespace to the replica. Always check the [metric replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) before initiating a failover. A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. **Who owns the replication lag?** Temporal owns replication lag. **What guarantees are available?** There is no SLA for replication lag. Temporal recommends that customers do not trigger failovers except for testing or emergency situations. High Availability feature's four-9 guarantee SLA means Temporal will handle failovers and ensure high availability. Temporal also monitors replication lag. Customers who decide to trigger failovers should look at this metric before moving forward. **If the lag is high, what should you do?** We don't expect users to failover. Please contact Temporal support if you feel you have a pressing need. **Where can you read more?** See [operations and metrics](/cloud/high-availability) for Namespaces with High Availability features. ### Reference Metrics - [temporal\_cloud\_v1\_replication\_lag\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) - [temporal\_cloud\_v1\_replication\_lag\_p95](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p95) - [temporal\_cloud\_v1\_replication\_lag\_p50](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p50) ## Monitoring Trends Against Limits Tracking trends against your account limits is the most important throttling signal to monitor. Unlike [Resource Exhaustion](#detecting-resource-exhaustion), which usually self-heals through retries, hitting a limit slows or stalls progress until the workload backs off or your capacity is increased. The set of [limit metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) provide a time series of values for limits. Use these metrics with their corresponding count metrics to monitor general trends against limits and set alerts when limits are exceeded. Use the corresponding throttle metrics to determine the severity of any active rate limiting. | Limit Metric | Count Metric | Throttle Metric | | ------------ | ------------ | --------------- | | `temporal_cloud_v1_action_limit` | `temporal_cloud_v1_total_action_count` | `temporal_cloud_v1_total_action_throttled_count` | | `temporal_cloud_v1_service_request_limit` | `temporal_cloud_v1_service_request_count` | `temporal_cloud_v1_service_request_throttled_count` | | `temporal_cloud_v1_operations_limit` | `temporal_cloud_v1_operations_count` | `temporal_cloud_v1_operations_throttled_count` | ### On-demand envelope limits For Namespaces using provisioned capacity, the following metrics show what your limits would be under on-demand mode. Compare these against your current provisioned limits to evaluate capacity mode choices: | On-Demand Envelope Metric | Equivalent Limit Metric | | ------------------------- | ----------------------- | | `temporal_cloud_v1_action_on_demand_envelope_limit` | `temporal_cloud_v1_action_limit` | | `temporal_cloud_v1_operations_on_demand_envelope_limit` | `temporal_cloud_v1_operations_limit` | | `temporal_cloud_v1_service_request_on_demand_envelope_limit` | `temporal_cloud_v1_service_request_limit` | For Namespaces already in on-demand mode, these metrics track the same values as their equivalent limit metrics. The [Grafana dashboard example](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) includes a Usage & Quotas section that creates demo charts for these limits and count metrics respectively. The limit metrics, throttle metrics, and count metrics are already directly comparable as per second rates. Keep in mind that each `count` metric is represented as a per second rate averaged over each minute. For example, to get the total count of Actions, you must multiply this metric by 60. When setting alerts against limits, consider if your workload is spiky or sensitive to throttling (for example, does latency matter?). If your workload is sensitive, consider alerting for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_cloud_v1_action_limit`. If your workload is not sensitive, consider an alert at 90% of this threshold or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive. ### Provisioned capacity utilization For Namespaces in [provisioned capacity](/cloud/capacity-modes#provisioned-capacity) mode, the limit and count metrics also reveal the lower bound: how much of your reserved capacity you are actually using. This matters because provisioned capacity is reserved by the hour whether or not you use it. Each TRU beyond the first carries a minimum hourly Action charge of 360,000 Actions, which is 20% of that TRU's 500 APS, so a Namespace that runs well below its limit can accrue charges for capacity it never uses. See [Capacity Mode Pricing](/cloud/pricing#capacity-modes-pricing) for how the minimum is calculated. Utilization is the ratio of `temporal_cloud_v1_total_action_count` to `temporal_cloud_v1_action_limit`. Sustained utilization well below 20% while provisioned indicates you may be paying for reserved capacity you are not using; consider alerting when the ratio stays under 20% over a sustained window, such as several hours, so you can reduce TRUs or return to on-demand mode. The [Grafana dashboard example](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) includes provisioned-capacity panels for utilization and limits. Low utilization is not always a problem. If your traffic is spiky or unpredictable, you may intentionally keep capacity provisioned so it is ready the moment you need it. > **📝 Note:** > > These metrics are approximate measures of TRU capacity based on real-time TRU quotas, and do not capture the fact that TRUs are billed by the calendar hour. > Please use [Usage](/cloud/actions-usage) and [Billing](/cloud/billing) as the source of truth for TRU and action accounting. > ### Why does throttling occur when count metrics stay below the limit? For spiky workloads, the throttle metric can be non-zero even though the count metric never rises above the limit. This looks contradictory, but both values are correct. They describe the workload at different time resolutions. Count and limit metrics are per-second rates **averaged over a 1-minute window** (see [Metric conventions](/cloud/metrics/openmetrics/metrics-reference#metric-types)). A short burst is smoothed across all 60 seconds, so the count metric can sit well below the limit even though the instantaneous request rate exceeded it and triggered throttling. The throttle metric, not the count-versus-limit comparison, is what tells you throttling actually occurred. #### Example: a spiky Actions workload Assume an Actions per second (APS) limit of 2,000 (`temporal_cloud_v1_action_limit` = 2000). A workload submits 60,000 Actions in a single second, then stays idle for the rest of the minute: - The rate limiter admits about 2,000 Actions in that second and throttles the remaining ~58,000. The SDK retries throttled Actions, which drain through at the 2,000 APS limit over the next ~30 seconds. They complete, but delayed. When throttling persists, delayed completions such as `RespondWorkflowTaskCompleted` can push Workflow Tasks and Activities past their timeouts and cause retries. - `temporal_cloud_v1_total_action_throttled_count` reflects the throttling: ~58,000 Actions throttled over the minute, or ~967 per second. - `temporal_cloud_v1_total_action_count` reports 60,000 Actions / 60 seconds = **1,000 APS** — half the 2,000 limit. Read in isolation, the count metric (1,000) against the limit (2,000) suggests plenty of headroom and no throttling. The throttle metric tells the true story: the burst exceeded the limit and ~58,000 Actions were delayed. #### Monitor count, limit, and throttle together To understand a spiky workload, always read all three metrics in the row as a set: | Metric | What it tells you | | ------ | ----------------- | | Count (for example, `temporal_cloud_v1_total_action_count`) | Average demand over the minute | | Limit (for example, `temporal_cloud_v1_action_limit`) | Your provisioned ceiling | | Throttle (for example, `temporal_cloud_v1_total_action_throttled_count`) | Whether the limit was actually hit | A non-zero throttle value means throttling occurred during that window, even when the count sits comfortably below the limit. Most often this reflects a sub-minute burst in your own workload. If you cannot find a matching burst in your SDK metrics, the cause may be a shared limit or another Cloud-side condition rather than your namespace. In that case, contact [Temporal Support](/cloud/support#support-ticket). For spiky or latency-sensitive workloads, alert on the throttle metric directly (any value greater than zero) rather than relying only on a count-versus-limit threshold, which can hide sub-minute bursts. The same logic applies to all three limit types — Actions (APS), service requests (RPS), and operations — using each row of the [limit / count / throttle table](#rps-aps-rate-limits) above. ## Detecting Resource Exhaustion Resource exhaustion happens when a single resource (a Namespace, Task Queue, or Workflow ID) receives a burst of operations larger than that resource can absorb in the moment. The Cloud metric `temporal_cloud_v1_resource_exhausted_error_count` increments and `ResourceExhausted` gRPC errors are returned to the client. SDKs retry these errors gracefully, so workflow progress is rarely impacted. Persistent non-zero values are unexpected and indicate a hot resource. Use the `operation` label to identify which RPC is hitting the burst limit. For example, `StartWorkflowExecution` increments here when the same Workflow Id is started more than once per second. See [Per-primitive Id reuse rate limits](/cloud/limits#per-primitive-id-reuse-rate-limits). Resource exhaustion is distinct from rate limiting against your account limits. For workloads that are throttled because they exceed their provisioned capacity, see [Monitoring Trends Against Limits](#rps-aps-rate-limits). Limits-driven throttling slows or stalls a workload, so it is generally the more important signal to monitor. ### Workflow lock contention (BusyWorkflow) The most common cause of resource exhaustion is Workflow lock contention. Every operation that mutates a single Workflow Execution (starting it, sending a Signal, etc.) is serialized under a per-Workflow lock. When operations reach one Execution faster than that lock can be acquired, the Service rejects the excess with a `ResourceExhausted` error. In Service logs this appears as `Workflow is busy.` This is contention on a single Execution, not an account limit. Increasing your Actions, Requests, or Operations per second limits does not resolve it. To confirm lock contention: 1. Rule out account-limit throttling first. If the throttle metrics described in [Monitoring Trends Against Limits](#rps-aps-rate-limits) are elevated, address that throttling first. 2. If you are within your limits but `temporal_cloud_v1_resource_exhausted_error_count` is still non-zero, break it down by the `operation` label. Lock contention concentrates on operations that target individual executions. 3. Match the operation to the guidance below. | `operation` | What it indicates | What to do | | ----------- | ----------------- | ---------- | | `StartWorkflowExecution`, `SignalWithStartWorkflowExecution` | The same Workflow ID was started again within a short de-duplication window (about one second). The first start succeeded; the duplicate was rejected. | Usually safe to ignore. Do not aggressively retry. There may be a client path firing the duplicate start. | | `SignalWorkflowExecution` | Too high of a rate of Signals to one execution. | Batch or coalesce Signals (for example, one Signal per N events), shard work across more executions, or buffer Signals and drain them in the main Workflow loop. | | `UpdateWorkflowExecution` | More than the per-execution in-flight Update limit (10) are outstanding. | Cap concurrent in-flight Updates on the client, then back off and retry. | | `RecordActivityTaskHeartbeat` | Too many Activities are heartbeating to the same execution. | Increase the heartbeat timeout and interval, and reduce the number of Activities heartbeating into one execution concurrently. | | `RespondWorkflowTaskCompleted` | A single Workflow schedules a large batch of Activities or Child Workflows in parallel, each taking the lock. | Limit concurrent operations to 500 or fewer per execution. Process the batch in smaller groups using a sliding-window or plain batching pattern instead of scheduling everything at once. | | `QueryWorkflow` | Too many concurrent Queries against one execution, or a side effect of repeated Workflow Task retries. | Reduce concurrent Queries to that execution. If it correlates with Workflow Task failures or timeouts, resolve those first. | At low, brief rates this error is benign because clients retry it and no progress is lost. Investigate when the rate is sustained or correlates with rising latency on the affected operations. For the per-execution limits referenced above, see [Per Workflow Execution concurrency limits](/cloud/limits#per-workflow-execution-concurrency-limits). --- # Service Level Agreement (SLA) - Temporal Cloud Source: https://docs.temporal.io/cloud/sla > Temporal Cloud offers two availability levels; 99.99% uptime for standard and High Availability feature deployments, with SLAs guaranteeing 99.9% and 99.99% against service errors, respectively. **What is Temporal Cloud's Service Level Agreement? SLA?** Temporal Cloud provides two availability levels: the [service availability](https://en.wikipedia.org/wiki/Reliability,_availability_and_serviceability) and the contractual [service level agreement](https://en.wikipedia.org/wiki/Service-level_agreement) (SLA). These levels are set by your deployment mode: - **Temporal Cloud with standard single-region deployment**: Standard Temporal Cloud deployment provides 99.99% availability and a contractual service level agreement (SLA) of 99.9% guarantee against service errors. - **Temporal Cloud with High Availability feature Namespace deployment**: Temporal Cloud Namespaces that use the High Availability feature provide 99.99% availability and contractual service level agreement (SLA) of 99.99% guarantee against service errors. The same SLA for normal Worker requests (commands and polling) apply to Nexus in both the caller and handler Namespaces. To calculate the service-error rate, Temporal Cloud captures all requests that arrive in a Namespace during a five-minute interval. We record the number of gRPC service errors that occurred. For each Namespace, we calculate the service-error rate as 1 - (count of errors / count of requests). Rates are averaged per month and reset quarterly. Errors recorded against the SLA are service errors, such as the `UNAVAILABLE` [gRPC status code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). The following errors are _not_ counted against the SLA: - `ClientVersionNotSupported` - `InvalidArgument` - `NamespaceAlreadyExists` - `NamespaceInvalidState` - `NamespaceNotActive` - `NamespaceNotFound` - `NotFound` - `PermissionDenied` - `QueryFailed` - `RetryReplication` - `StickyWorkerUnavailable` - `TaskAlreadyStarted` - `Throttling (resources exhausted; triggers retry)` - `WorkflowExecutionAlreadyStarted` - `WorkflowNotReady` Our internal alerting system is based on a [service level objective](https://en.wikipedia.org/wiki/Service-level_objective) (SLO) for all errors, not just errors that count against the SLA. When we receive an alert that an SLO is not being met, we page our on-call engineers, which often means that issues are resolved before they become noticeable. Internally, our components are distributed across a minimum of three availability zones per region. We implement a cell architecture. Each cell contains the software and services necessary to host a Namespace. Within each cell, the components are distributed across a minimum of three availability zones per region. For current system status and information about recent incidents, see [Temporal Status](https://status.temporal.io). --- # Services, support, and training - Temporal Cloud Source: https://docs.temporal.io/cloud/support > Temporal Cloud offers support, services, and training for seamless onboarding, efficient app design, and scaling. Services include technical onboarding, design/code reviews, pre-production optimization, and load tests. Temporal Cloud includes the right level of technical support and guidance, services and training needed to onboard you successfully, assist with design and deployment of your application efficiently and at scale. Our team has extensive knowledge of Temporal, and a broad set of skills to help you succeed with any project. Temporal Cloud provides several levels of support, from assisting with break/fix scenarios to issues and services to helping with onboarding, design/code reviews for your application, and pre-production optimizations and operational readiness. > **📝 Note:** > > The content of this page applies to Temporal Cloud customers only. > ## Services offered by Temporal Cloud | | Essentials | Business | Enterprise | Mission Critical | | --------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | Support Staff | Trained staff providing break-fix support and general guidance. | Trained staff providing break-fix support and general guidance. | Developer experts who provide advanced support | Developer experts who provide advanced support | | Technical Guidance | Core platform config, platform access, documented features, and basic inquiries | Advanced technical support, Workflow troubleshooting, SDK implementations, and Worker configuration, Quarterly code review or design implementation best practices. | Business+ expert-led code reviews and design implementation best practices, available as needed | Enterprise+ expert guidance on Workflow latency monitoring and optimization; performance recommendations based on real time tests | | Billing & Cost Optimization | Generic Billing Questions | Generic Billing Questions | Quarterly review of spend | Quarterly review of spend, proactive cost optimization | ## Temporal Cloud support guarantees Temporal endeavors to ensure you are successful with Temporal Cloud. We offer explicit guarantees for support. Temporal Cloud customers get break/fix support with an agreed-upon set of SLAs for prioritized issues. We use a ticketing system for entering, tracking, and closing these issues. If an issue occurs, the team also provides support through a dedicated Slack channel, forums, and a knowledge base. We offer two levels of support defined by their availability and SLAs in the following table: | | Essentials | Business | Enterprise | Mission Critical | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | | **Availability**
(Based on
Time-zones) | **P0–3**: 9–5 Mon–Fri | **P0–3**: 9–5 Mon–Fri | **P0**: 24×7,
(On Page Service)
**P1–3**: 9–5 Mon–Fri | **P0**: 24×7 (On Page Service)
**P1**: 9-5, 7 days/week
**P2–3**: Mon–Fri | | **Response Time** | **P0**: 1 business day
**P1**: 1 business day
**P2**: 1 business day
**P3**: 2 business days | **P0**: 2 business hours
**P1**: 2 business hours
**P2**: 1 business day
**P3**: 2 business days | **P0**: 30 minutes
**P1**: 1 business hour
**P2**: 4 business hours
**P3**: 1 business day | **P0**: 15 minutes
**P1**: 1 business hour
**P2**: 4 business hours
**P3**: 1 business day | | **Dedicated Platform Architect** | - | - | Add-on | Included (1 Unit) | | **Channels** | Community
Temporal Support Portal | Community
Temporal Support Portal | Community
Temporal Support Portal
Private Slack | Community
Temporal Support Portal
Private Slack | > **ℹ️ Info:** > Business Hours Timezones > > Business Hours will be specified in your contract, including one of three locations: US Pacific time, European Central time, Australia Eastern time > **Priority definitions** - **P0 - Critical** (Production impacted) - The Temporal Cloud service is unavailable or degraded with a significant impact. - **P1 - High** (Production issue) - An issue related to production workloads running on the Temporal Cloud service, or a significant project is blocked. - **P2 - Normal** (General issues) - General Temporal Cloud service or other issues where there is no production impact, or a workaround exists to mitigate the impact. - **P3 - Low** (General guidance) - Questions or an issue with the Temporal Cloud service that is not impacting system availability or functionality. > **📝 Note:** > On Page Service > > P0: 24×7 (On Page Service) is offered for Enterprise and Mission Critical accounts. > For pricing details of these support levels, please visit our [pricing page](/cloud/pricing). ## Temporal Dedicated Platform Architect Customers on the Mission Critical Plan and (by opting in) Enterprise customers receive access to a Dedicated Support Engineer. We offer: - Direct access to a senior developer expert, who becomes part of your Temporal account team, adding deep technical expertise. - Our high-touch engagement model goes beyond traditional support to deliver transformative value through hands-on collaboration, proactive optimization, implementation design and operations. - Faster issue resolution with direct assistance from someone who already knows your implementation. - Focused advisory on best practices and development pairing to ensure high-quality code and scalability. - Optimizations through regular checks and recommendations to improve performance and efficiency. - Priority access to a senior engineer for up to 20 hours per month, providing expert guidance and proactive support for one business unit or major group, specifically within a single region. Our Services focus on local time zone alignment to ensure optimal responsiveness and efficiency. Additional service units for this service can be purchased to cover additional groups or regions at $6,000/Mo/Unit. One unit of Mission Critical Support includes: - Up to 20 hours per month - One major group or business unit - Limited to one region - Quarterly onsite visits ## Ticketing Temporal offers a ticketing system for Temporal Cloud customers. We have an active [community Slack](https://temporalio.slack.com) and an active [community Discourse forum](https://community.temporal.io/) where you can post questions and ask for help. > **ℹ️ Info:** > > The Temporal Support Portal is for Cloud customers only. > Other Temporal users (non-cloud) have full community access excluding the "#support-cloud" channel. > All Cloud customers pay for support as part of their plan. > ### Access Temporal Support 1. Go to [support.temporal.io](https://support.temporal.io/). 2. If prompted, log in to Temporal Cloud using the same method you normally use (for example, Google, Microsoft, email-password, or other methods). 3. You will be presented with a screen where you can view open and closed tickets for your Temporal account, as well as submit a new ticket. To request assistance from Temporal Support, see [Create a ticket](#support-ticket). ### Create a Ticket > **ℹ️ Info:** > > This procedure applies only to Temporal Cloud customers whose contracts include paid support. > If you need assistance and don't have paid support, post your request in the [Temporal Community Forum](https://community.temporal.io) or the `#support-cloud` channel of the [Temporal workspace](https://t.mp/slack) in Slack. > To create a ticket in the Temporal Support Portal: 1. Go to [support.temporal.io](https://support.temporal.io/). 2. If prompted, log in to Temporal Cloud using the same method you normally use (for example, Google, Microsoft, email-password, or other methods). 3. Click the **Create Ticket** button in the top right corner. 4. On the **Submit a ticket** page, enter the details of your request into the form. **Name**, **Subject**, and **Description** are required. 5. At the bottom of the form, choose **Submit**. ## Developer resources Temporal offers developer resources and a variety of hands-on tutorials to get you started and learn more advanced Temporal concepts. - [Get started with Temporal](https://learn.temporal.io/getting_started): Start your journey with Temporal with this guide that helps you set up your development environment, run an existing Temporal app, and then build your first app from scratch using our SDKs. - [Courses](https://learn.temporal.io/courses): Learn and apply Temporal concepts in our free, self-paced, hands-on courses. - [Tutorials](https://learn.temporal.io/tutorials): Apply Temporal concepts to build real-world applications with these hands-on tutorials. - [Example applications](https://learn.temporal.io/examples): Explore example applications that use Temporal and gain a clearer understanding of how Temporal concepts work in a complex application. --- # tcld command reference Source: https://docs.temporal.io/cloud/tcld > The Temporal Cloud CLI (tcld) is a command-line tool for interacting with Temporal Cloud, offering commands for account management, login, namespace, and more. Install via Homebrew or build from source. The Temporal Cloud CLI (tcld) is a command-line tool that you can use to interact with Temporal Cloud. > **💡 Tip:** > > The Temporal CLI also manages Temporal Cloud through its > [Temporal Cloud extension](/cli/cloud), which adds `temporal cloud` commands for Namespaces, users, API keys, and Nexus > Endpoints. See the [`temporal cloud` command reference](/cli/command-reference/cloud) for the equivalent commands. > - [How to install tcld](#install-tcld) ### tcld commands - [tcld account](/cloud/tcld/account) - [tcld apikey](/cloud/tcld/apikey) - [tcld connectivity-rule](/cloud/tcld/connectivity-rule) - [tcld feature](/cloud/tcld/feature) - [tcld generate-certificates](/cloud/tcld/generate-certificates) - [tcld login](/cloud/tcld/login) - [tcld logout](/cloud/tcld/logout) - [tcld migration](/cloud/tcld/migration) - [tcld namespace](/cloud/tcld/namespace) - [tcld nexus](/cloud/tcld/nexus) - [tcld request](/cloud/tcld/request) - [tcld service-account](/cloud/tcld/service-account) - [tcld user](/cloud/tcld/user) - [tcld user-group](/cloud/tcld/user-group) - [tcld version](/cloud/tcld/version) ### Global modifiers #### --auto_confirm Automatically confirm all prompts. You can specify the value for this modifier by setting the AUTO_CONFIRM environment variable. The default value is `false`. ## How to install tcld You can install [tcld](/cloud/tcld) in two ways. ### Install tcld by using Homebrew ```bash brew install temporalio/brew/tcld ``` ### Build tcld from source 1. Verify that you have Go 1.18 or later installed. ```bash go version ``` If Go 1.18 or later is not installed, follow the [Download and install](https://go.dev/doc/install) instructions on the Go website. 1. Clone the tcld repository and run make. ```bash git clone https://github.com/temporalio/tcld.git cd tcld make ``` 1. Copy the tcld executable to any directory that appears in the PATH environment variable, such as `/usr/local/bin`. ```bash cp tcld /usr/local/bin/tcld ``` 1. Verify that tcld is installed. ```bash tcld version ``` --- # tcld account command reference Source: https://docs.temporal.io/cloud/tcld/account > Account operations `tcld account`: Account operations. Alias: `a` - [tcld account get](#get) - [tcld account list-regions](#list-regions) - [tcld account metrics](#metrics) - [tcld account audit-log](#audit-log) ### get `tcld account get`: Get account information. Alias: `g` ### list-regions `tcld account list-regions`: Lists all regions where the account can provision namespaces. Alias: `l` ### metrics `tcld account metrics`: Configures the metrics endpoint for the Temporal Cloud Account. Alias: `m` - [tcld account metrics enable](#enable) - [tcld account metrics disable](#disable) - [tcld account metrics accepted-client-ca](#accepted-client-ca) #### enable `tcld account metrics enable`: Enables the metrics endpoint. CA Certificates *must* be configured prior to enabling the endpoint. #### disable `tcld account metrics disable`: Disables the metrics endpoint. #### accepted-client-ca `tcld account metrics accepted-client-ca`: Manages configuration of ca certificates for the external metrics endpoint. Alias: `ca` - [tcld account metrics accepted-client-ca add](#add) - [tcld account metrics accepted-client-ca remove](#remove) - [tcld account metrics accepted-client-ca set](#set) - [tcld account metrics accepted-client-ca list](#list) ##### add `tcld account metrics accepted-client-ca add`: Add a new ca accepted client ca certificate. Alias: `a` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --ca-certificate The base64 encoded ca certificate Alias: `c` ###### --ca-certificate-file The path to the ca pem file Alias: `f` ##### remove `tcld account metrics accepted-client-ca remove`: Remove existing certificates. Alias: `r` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --ca-certificate The base64 encoded ca certificate Alias: `c` ###### --ca-certificate-file The path to the ca pem file Alias: `f` ###### --ca-certificate-fingerprint The fingerprint of to the ca certificate Alias: `fp` ##### set `tcld account metrics accepted-client-ca set`: Set the accepted client ca certificate. Alias: `s` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --ca-certificate The base64 encoded ca certificate Alias: `c` ###### --ca-certificate-file The path to the ca pem file Alias: `f` ##### list `tcld account metrics accepted-client-ca list`: List the accepted client ca certificates currently configured for the account metrics endpoint. Alias: `l` ### audit-log `tcld account audit-log`: audit log commands. Alias: `al` - [tcld account audit-log kinesis](#kinesis) - [tcld account audit-log pubsub](#pubsub) #### kinesis `tcld account audit-log kinesis`: Manage Kinesis audit log sink. Alias: `k` - [tcld account audit-log kinesis create](#create) - [tcld account audit-log kinesis validate](#validate) - [tcld account audit-log kinesis update](#update) - [tcld account audit-log kinesis get](#get) - [tcld account audit-log kinesis delete](#delete) - [tcld account audit-log kinesis list](#list) ##### create `tcld account audit-log kinesis create`: Create a kinesis audit log sink. Alias: `c` ###### --sink-name Provide a name for the sink ###### --role-name The role name to use to write to the sink Alias: `rn` ###### --destination-uri The destination URI of the audit log sink Alias: `du` ###### --region The region to use for the request Alias: `re` ##### validate `tcld account audit-log kinesis validate`: Validate kinesis audit log sink. Alias: `v` ###### --sink-name Provide a name for the sink ###### --role-name The role name to use to write to the sink Alias: `rn` ###### --destination-uri The destination URI of the audit log sink Alias: `du` ###### --region The region to use for the request Alias: `re` ##### update `tcld account audit-log kinesis update`: Update a kinesis audit log sink. Alias: `u` ###### --sink-name Provide a name for the sink ###### --enabled Whether the sink is enabled ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --role-name The role name to use to write to the sink Alias: `rn` ###### --destination-uri The destination URI of the audit log sink Alias: `du` ###### --region The region to use for the request Alias: `re` ##### get `tcld account audit-log kinesis get`: Get audit log sink. Alias: `g` ###### --sink-name Provide a name for the sink ##### delete `tcld account audit-log kinesis delete`: Delete audit log sink. Alias: `d` ###### --sink-name Provide a name for the sink ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### list `tcld account audit-log kinesis list`: List audit log sinks. Alias: `l` ###### --page-size The page size for list operations ###### --page-token The page token for list operations #### pubsub `tcld account audit-log pubsub`: Manage PubSub audit log sink. Alias: `ps` - [tcld account audit-log pubsub create](#create) - [tcld account audit-log pubsub validate](#validate) - [tcld account audit-log pubsub update](#update) - [tcld account audit-log pubsub get](#get) - [tcld account audit-log pubsub delete](#delete) - [tcld account audit-log pubsub list](#list) ##### create `tcld account audit-log pubsub create`: Create a pubsub audit log sink. Alias: `c` ###### --sink-name Provide a name for the sink ###### --service-account-email The service account email to impersonate to write to the sink Alias: `sae` ###### --topic-name The topic name to write to the sink Alias: `tn` ##### validate `tcld account audit-log pubsub validate`: Validate pubsub audit log sink. Alias: `v` ###### --sink-name Provide a name for the sink ###### --service-account-email The service account email to impersonate to write to the sink Alias: `sae` ###### --topic-name The topic name to write to the sink Alias: `tn` ##### update `tcld account audit-log pubsub update`: Update a pubsub audit log sink. Alias: `u` ###### --sink-name Provide a name for the sink ###### --enabled Whether the sink is enabled ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --service-account-email The service account email to impersonate to write to the sink Alias: `sae` ###### --topic-name The topic name to write to the sink Alias: `tn` ##### get `tcld account audit-log pubsub get`: Get audit log sink. Alias: `g` ###### --sink-name Provide a name for the sink ##### delete `tcld account audit-log pubsub delete`: Delete audit log sink. Alias: `d` ###### --sink-name Provide a name for the sink ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### list `tcld account audit-log pubsub list`: List audit log sinks. Alias: `l` ###### --page-size The page size for list operations ###### --page-token The page token for list operations --- # tcld apikey command reference Source: https://docs.temporal.io/cloud/tcld/apikey > APIKey operations `tcld apikey`: APIKey operations. Alias: `ak` - [tcld apikey create](#create) - [tcld apikey get](#get) - [tcld apikey list](#list) - [tcld apikey delete](#delete) - [tcld apikey disable](#disable) - [tcld apikey enable](#enable) ### create `tcld apikey create`: Create an apikey. Make sure to copy the secret or else you will not be able to retrieve it again. Alias: `c` #### --name the display name of the apikey Alias: `n` #### --description the description of the apikey Alias: `desc` #### --duration the duration from now when the apikey will expire, will be ignored if expiry flag is set, examples: '1.5y', '30d', '4d12h' Alias: `d` #### --expiry Alias: `e` #### --service-account-id setting this flag will create an api key for a service account, not a user Alias: `si` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### get `tcld apikey get`: Get an apikey. Alias: `g` #### --id The id of the apikey to get Alias: `i` ### list `tcld apikey list`: List API keys. Alias: `l` #### --owner-id Filter API keys by owner ID Alias: `oid` #### --owner-type Filter API keys by owner type (that is, 'user', 'service-account') Alias: `ot` ### delete `tcld apikey delete`: Delete an apikey. Alias: `d` #### --id The id of the apikey to delete Alias: `i` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### disable `tcld apikey disable`: Disable an apikey. Alias: `da` #### --id The id of the apikey to disable Alias: `i` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### enable `tcld apikey enable`: Enable a disabled apikey. Alias: `ea` #### --id The id of the apikey to enable Alias: `i` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` --- # tcld connectivity-rule command reference Source: https://docs.temporal.io/cloud/tcld/connectivity-rule > Connectivity rule operations `tcld connectivity-rule`: Connectivity rule operations. Alias: `cr` - [tcld connectivity-rule create](#create) - [tcld connectivity-rule get](#get) - [tcld connectivity-rule list](#list) - [tcld connectivity-rule delete](#delete) ### create `tcld connectivity-rule create`: Create a connectivity rule. Alias: `c` #### --connectivity-type The type of connectivity, currently only support 'private' and 'public' Alias: `ct` #### --connection-id The connection ID of the private connection Alias: `ci` #### --region The region of the connection Alias: `r` #### --gcp-project-id The GCP project ID of the connection, required if the cloud provider is 'gcp' Alias: `gpi` ### get `tcld connectivity-rule get`: Get a connectivity rule. Alias: `g` #### --connectivity-rule-id The connectivity rule ID Alias: `id` ### list `tcld connectivity-rule list`: list connectivity rules. Alias: `l` #### --namespace The namespace hosted on temporal cloud Alias: `n` ### delete `tcld connectivity-rule delete`: Delete a connectivity rule. Alias: `d` #### --connectivity-rule-id The connectivity rule ID Alias: `id` --- # tcld feature command reference Source: https://docs.temporal.io/cloud/tcld/feature > feature commands `tcld feature`: feature commands. Alias: `f` - [tcld feature get](#get) ### get `tcld feature get`: get all feature flags Value. Alias: `g` --- # tcld generate-certificates command reference Source: https://docs.temporal.io/cloud/tcld/generate-certificates > Commands for generating certificate authority and end-entity TLS certificates `tcld generate-certificates`: Commands for generating certificate authority and end-entity TLS certificates. Alias: `gen` - [tcld generate-certificates certificate-authority-certificate](#certificate-authority-certificate) - [tcld generate-certificates end-entity-certificate](#end-entity-certificate) ### certificate-authority-certificate `tcld generate-certificates certificate-authority-certificate`: Generate a certificate authority certificate. Alias: `ca` #### --organization The name of the organization Alias: `org` #### --validity-period The duration for which the certificate is valid for. example: 30d10h (30 days and 10 hrs) Alias: `d` #### --ca-certificate-file The path where the generated x509 certificate will be stored Alias: `ca-cert` #### --ca-key-file The path where the certificate's private key will be stored Alias: `ca-key` #### --rsa-algorithm Generates a 4096-bit RSA keypair instead of an ECDSA P-384 keypair (the recommended default) for the certificate (optional) Alias: `rsa` ### end-entity-certificate `tcld generate-certificates end-entity-certificate`: Generate an end-entity certificate. Alias: `leaf` #### --organization The name of the organization Alias: `org` #### --organization-unit The name of the organizational unit (optional) #### --common-name The common name (optional) #### --validity-period The duration for which the end entity certificate is valid for. example: 30d10h (30 days and 10 hrs). By default the generated certificate expires 24 hours before the certificate authority expires (optional) Alias: `d` #### --ca-certificate-file The path of the x509 certificate for the certificate authority Alias: `ca-cert` #### --ca-key-file The path of the private key for the certificate authority Alias: `ca-key` #### --certificate-file The path where the generated x509 certificate will be stored Alias: `cert` #### --key-file The path where the certificate's private key will be stored Alias: `key` --- # tcld login command reference Source: https://docs.temporal.io/cloud/tcld/login > Login as user `tcld login`: Login as user. Alias: `l` ### --disable-pop-up disable browser pop-up --- # tcld logout command reference Source: https://docs.temporal.io/cloud/tcld/logout > Logout current user `tcld logout`: Logout current user. Alias: `lo` ### --disable-pop-up disable browser pop-up --- # tcld migration command reference Source: https://docs.temporal.io/cloud/tcld/migration > (private preview) Manage migrations between self-hosted Temporal and Temporal cloud `tcld migration`: (private preview) Manage migrations between self-hosted Temporal and Temporal cloud. Alias: `m` - [tcld migration get](#get) - [tcld migration list](#list) - [tcld migration start](#start) - [tcld migration handover](#handover) - [tcld migration confirm](#confirm) - [tcld migration abort](#abort) ### get `tcld migration get`: Get a migration. Alias: `g` #### --id Migration id Alias: `i` ### list `tcld migration list`: List migrations. Alias: `l` ### start `tcld migration start`: Start a new migration. Alias: `s` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --endpoint-id Migration endpoint id Alias: `e` #### --source-namespace Source namespace name Alias: `s` #### --target-namespace Target namespace name Alias: `t` ### handover `tcld migration handover`: Handover the namespace from on-prem to cloud, or from cloud back to on-prem. Alias: `s` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --id Migration id Alias: `i` #### --to-replica-id The id of the replica to make active Alias: `rp` ### confirm `tcld migration confirm`: Confirm the migration. Alias: `c` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --id Migration id Alias: `i` ### abort `tcld migration abort`: Abort the migration. Alias: `a` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --id Migration id Alias: `i` --- # tcld namespace command reference Source: https://docs.temporal.io/cloud/tcld/namespace > Namespace operations `tcld namespace`: Namespace operations. Alias: `n` - [tcld namespace create](#create) - [tcld namespace add-region](#add-region) - [tcld namespace delete-region](#delete-region) - [tcld namespace lifecycle](#lifecycle) - [tcld namespace delete](#delete) - [tcld namespace list](#list) - [tcld namespace get](#get) - [tcld namespace accepted-client-ca](#accepted-client-ca) - [tcld namespace auth-method](#auth-method) - [tcld namespace certificate-filters](#certificate-filters) - [tcld namespace update-codec-server](#update-codec-server) - [tcld namespace retention](#retention) - [tcld namespace search-attributes](#search-attributes) - [tcld namespace failover](#failover) - [tcld namespace update-high-availability](#update-high-availability) - [tcld namespace tags](#tags) - [tcld namespace capacity](#capacity) - [tcld namespace export](#export) - [tcld namespace set-connectivity-rules](#set-connectivity-rules) ### create `tcld namespace create`: Create a temporal namespace. Alias: `c` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --ca-certificate The base64 encoded ca certificate Alias: `c` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --region Create namespace in specified regions; if multiple regions are selected, the first one will be the active region. See 'tcld account list-regions' to get a list of available regions for your account Alias: `re` #### --retention-days The retention of the namespace in days Alias: `rd` #### --auth-method The authentication method to use for the namespace (for example, 'mtls', 'api_key') #### --ca-certificate-file The path to the ca pem file Alias: `cf` #### --certificate-filter-file Path to a JSON file that defines the certificate filters that will be added to the namespace. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `cff` #### --certificate-filter-input JSON that defines the certificate filters that will be added to the namespace. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `cfi` #### --search-attribute Flag can be used multiple times; value must be "name=type"; valid types are: [Keyword Text Int Double Datetime Bool KeywordList] Alias: `sa` #### --user-namespace-permission Flag can be used multiple times; value must be "email=permission"; valid permissions are: [Admin Write Read] Alias: `p` #### --enable-delete-protection Enable delete protection on the namespace Alias: `edp` #### --endpoint The codec server endpoint to decode payloads for all users interacting with this Namespace, must be https Alias: `e` #### --pass-access-token Pass the user access token to the remote endpoint Alias: `pat` #### --include-credentials Include cross-origin credentials Alias: `ic` #### --cloud-provider Cloud provider for the namespace to be created for, currently support [aws, gcp]. For this version, if not specified, we default to aws Alias: `cp` #### --tag Add tags to the namespace (format: key=value). Flag can be used multiple times. Alias: `t` #### --connectivity-rule-ids The list of connectivity rule IDs, can be used in create namespace and update namespace. example: --ids id1 --ids id2 --ids id3 Alias: `ids` ### add-region `tcld namespace add-region`: Add a new region to a namespace. #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --region New region to add to the namespace. Alias: `re` #### --cloud-provider The cloud provider of the region. Default: aws ### delete-region `tcld namespace delete-region`: Delete a region from a namespace. #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --region The region to remove from a namespace. Alias: `re` #### --cloud-provider The cloud provider of the region. Default: aws ### lifecycle `tcld namespace lifecycle`: Enable delete protection on a temporal namespace. Alias: `lc` - [tcld namespace lifecycle get](#get) - [tcld namespace lifecycle set](#set) #### get `tcld namespace lifecycle get`: Get the lifecycle spec for the namespace. ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --namespace The namespace hosted on temporal cloud Alias: `n` #### set `tcld namespace lifecycle set`: Set the lifecycle spec for the namespace. ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --enable-delete-protection Enable delete protection on the namespace, value must be true or false Alias: `edp` ### delete `tcld namespace delete`: Delete a temporal namespace. Alias: `d` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --namespace The namespace hosted on temporal cloud Alias: `n` ### list `tcld namespace list`: List all known namespaces. Alias: `l` #### --page-token The page token for list operations #### --page-size Number of namespaces to list per page ### get `tcld namespace get`: Get namespace information. Alias: `g` #### --namespace The namespace hosted on temporal cloud Alias: `n` ### accepted-client-ca `tcld namespace accepted-client-ca`: Manage client ca certificate used to verify client connections. Alias: `ca` - [tcld namespace accepted-client-ca list](#list) - [tcld namespace accepted-client-ca add](#add) - [tcld namespace accepted-client-ca remove](#remove) - [tcld namespace accepted-client-ca set](#set) #### list `tcld namespace accepted-client-ca list`: List the accepted client ca certificates currently configured for the namespace. Alias: `l` ##### --namespace The namespace hosted on temporal cloud Alias: `n` #### add `tcld namespace accepted-client-ca add`: Add a new ca accepted client ca certificate. Alias: `a` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --ca-certificate The base64 encoded ca certificate Alias: `c` ##### --ca-certificate-file The path to the ca pem file Alias: `f` #### remove `tcld namespace accepted-client-ca remove`: Remove existing certificates. Alias: `r` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --ca-certificate The base64 encoded ca certificate Alias: `c` ##### --ca-certificate-file The path to the ca pem file Alias: `f` ##### --ca-certificate-fingerprint The fingerprint of to the ca certificate Alias: `fp` ##### --all If set, all existing certificates will be removed #### set `tcld namespace accepted-client-ca set`: Set the accepted client ca certificate. Alias: `s` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --ca-certificate The base64 encoded ca certificate Alias: `c` ##### --ca-certificate-file The path to the ca pem file Alias: `f` ### auth-method `tcld namespace auth-method`: Manage the authentication method for the namespace. Alias: `am` - [tcld namespace auth-method set](#set) - [tcld namespace auth-method get](#get) #### set `tcld namespace auth-method set`: Set the authentication method for the namespace. ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --auth-method The authentication method used for the namespace (that is, 'restricted', 'mtls', 'api_key', 'api_key_or_mtls') Alias: `am` #### get `tcld namespace auth-method get`: Retrieve the authentication method for namespace. ##### --namespace The namespace hosted on temporal cloud Alias: `n` ### certificate-filters `tcld namespace certificate-filters`: Manage optional certificate filters used by namespace to authorize client certificates based on distinguished name fields. Alias: `cf` - [tcld namespace certificate-filters import](#import) - [tcld namespace certificate-filters export](#export) - [tcld namespace certificate-filters clear](#clear) - [tcld namespace certificate-filters add](#add) #### import `tcld namespace certificate-filters import`: Sets the certificate filters on the namespace. Existing filters will be replaced. Alias: `imp` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --certificate-filter-file Path to a JSON file that defines the certificate filters that will be configured on the namespace. This will replace the existing filter configuration. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `file`, `f` ##### --certificate-filter-input JSON that defines the certificate filters that will be configured on the namespace. This will replace the existing filter configuration. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `input`, `i` #### export `tcld namespace certificate-filters export`: Exports existing certificate filters on the namespace. Alias: `exp` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --certificate-filter-file Path to a JSON file where tcld will export the certificate filter configuration to Alias: `file`, `f` #### clear `tcld namespace certificate-filters clear`: Clears all certificate filters on the namespace. Note that this will allow *any* client certificate that chains up to a configured CA in the bundle to connect to the namespace. Alias: `c` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### add `tcld namespace certificate-filters add`: Adds additional certificate filters to the namespace. Alias: `a` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --certificate-filter-file Path to a JSON file that defines the certificate filters that will be added to the namespace. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `file`, `f` ##### --certificate-filter-input JSON that defines the certificate filters that will be added to the namespace. Sample JSON: { "filters": [ { "commonName": "test1" } ] } Alias: `input`, `i` ### update-codec-server `tcld namespace update-codec-server`: Update codec server config used to decode encoded payloads through remote endpoint. Alias: `ucs` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --endpoint The codec server endpoint to decode payloads for all users interacting with this Namespace, must be https Alias: `e` #### --pass-access-token Pass the user access token to the remote endpoint Alias: `pat` #### --include-credentials Include cross-origin credentials Alias: `ic` ### retention `tcld namespace retention`: Manages configuration of the length of time (in days) a closed workflow will be preserved before deletion. Alias: `r` - [tcld namespace retention set](#set) - [tcld namespace retention get](#get) #### set `tcld namespace retention set`: Set the length of time (in days) a closed workflow will be preserved before deletion for a given namespace. Alias: `s` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --retention-days The retention of the namespace in days Alias: `rd` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### get `tcld namespace retention get`: Retrieve the length of time (in days) a closed workflow will be preserved before deletion for a given namespace. Alias: `g` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ### search-attributes `tcld namespace search-attributes`: Manage search attributes used by namespace. Alias: `sa` - [tcld namespace search-attributes add](#add) - [tcld namespace search-attributes remove](#remove) - [tcld namespace search-attributes rename](#rename) #### add `tcld namespace search-attributes add`: Add a new namespace custom search attribute. Alias: `a` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --search-attribute Flag can be used multiple times; value must be "name=type"; valid types are: [Keyword Text Int Double Datetime Bool KeywordList] Alias: `sa` #### remove `tcld namespace search-attributes remove`: Remove an existing namespace custom search attribute. Alias: `rm` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --search-attribute The name of the search attribute to remove Alias: `sa` #### rename `tcld namespace search-attributes rename`: Update the name of an existing custom search attribute. Alias: `rn` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --existing-name The name of an existing search attribute Alias: `en` ##### --new-name The new name for the search attribute Alias: `nn` ### failover `tcld namespace failover`: Failover a temporal namespace. Alias: `fo` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --region The region to failover to Alias: `re` #### --cloud-provider The cloud provider of the region. Default: aws ### update-high-availability `tcld namespace update-high-availability`: Update Temporal namespace high availability setting. Alias: `uha` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --disable-auto-failover Disable Temporal-managed failover on a replicated namespace (use --disable-auto-failover=false to enable) ### tags `tcld namespace tags`: Manage namespace tags. Alias: `t` - [tcld namespace tags upsert](#upsert) - [tcld namespace tags remove](#remove) #### upsert `tcld namespace tags upsert`: Add new tags or update existing tag values. Alias: `u` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --tag Add new or update existing namespace tags (format: key=value). Flag can be used multiple times. Alias: `t` #### remove `tcld namespace tags remove`: Remove existing tags by key. Alias: `rm` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --tag-key Remove namespace tags by key. Flag can be used multiple times. Alias: `tk` ### capacity `tcld namespace capacity`: Manage namespace capacity. Alias: `cap` - [tcld namespace capacity get](#get) - [tcld namespace capacity update](#update) #### get `tcld namespace capacity get`: Get namespace capacity information. Alias: `g` ##### --namespace The namespace hosted on temporal cloud Alias: `n` #### update `tcld namespace capacity update`: Set the capacity of a given namespace. Alias: `u` ##### --namespace The namespace hosted on temporal cloud Alias: `n` ##### --capacity-mode The capacity mode to use for the namespace. Valid values are 'on_demand' and 'provisioned' Alias: `cm` ##### --capacity-value The capacity value to use for the namespace. Required if capacity mode is 'provisioned', ignored otherwise Alias: `cv` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ### export `tcld namespace export`: Manage export. Alias: `es` - [tcld namespace export s3](#s3) - [tcld namespace export gcs](#gcs) #### s3 `tcld namespace export s3`: Manage S3 export sink. - [tcld namespace export s3 create](#create) - [tcld namespace export s3 validate](#validate) - [tcld namespace export s3 update](#update) - [tcld namespace export s3 get](#get) - [tcld namespace export s3 delete](#delete) - [tcld namespace export s3 list](#list) ##### create `tcld namespace export s3 create`: Create export sink. Alias: `c` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --role-arn Provide role arn for the IAM Role ###### --s3-bucket-name Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to ###### --kms-arn Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly. ###### --region The region to use for the request, if not set the server will use the namespace's region Alias: `re` ##### validate `tcld namespace export s3 validate`: Validate export sink. Alias: `v` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --role-arn Provide role arn for the IAM Role ###### --s3-bucket-name Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to ###### --kms-arn Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly. ###### --region The region to use for the request, if not set the server will use the namespace's region Alias: `re` ##### update `tcld namespace export s3 update`: Update export sink. Alias: `u` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --enabled Whether the sink is enabled ###### --role-arn Provide role arn for the IAM Role ###### --s3-bucket-name Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --kms-arn Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, user must create the IAM Role with KMS or modify the created IAM Role accordingly. ##### get `tcld namespace export s3 get`: Get export sink. Alias: `g` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ##### delete `tcld namespace export s3 delete`: Delete export sink. Alias: `d` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### list `tcld namespace export s3 list`: List export sinks. Alias: `l` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --page-size The page size for list operations ###### --page-token The page token for list operations #### gcs `tcld namespace export gcs`: Manage GCS export sink. - [tcld namespace export gcs create](#create) - [tcld namespace export gcs update](#update) - [tcld namespace export gcs validate](#validate) - [tcld namespace export gcs get](#get) - [tcld namespace export gcs delete](#delete) - [tcld namespace export gcs list](#list) ##### create `tcld namespace export gcs create`: Create export sink. Alias: `c` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --service-account-email Service account that has access to the sink ###### --gcs-bucket GCS bucket of the sink ##### update `tcld namespace export gcs update`: Update export sink. Alias: `u` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --enabled Whether the sink is enabled ###### --service-account-email Service account that has access to the sink ###### --gcs-bucket GCS bucket of the sink ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### validate `tcld namespace export gcs validate`: Validate export sink. Alias: `v` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --service-account-email Service account that has access to the sink ###### --gcs-bucket GCS bucket of the sink ##### get `tcld namespace export gcs get`: Get export sink. Alias: `g` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ##### delete `tcld namespace export gcs delete`: Delete export sink. Alias: `d` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --sink-name Provide a name for the sink ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### list `tcld namespace export gcs list`: List export sinks. Alias: `l` ###### --namespace The namespace hosted on temporal cloud Alias: `n` ###### --page-size The page size for list operations ###### --page-token The page token for list operations ### set-connectivity-rules `tcld namespace set-connectivity-rules`: set the connectivity rules for a namespace. Alias: `scrs` #### --namespace The namespace hosted on temporal cloud Alias: `n` #### --connectivity-rule-ids The list of connectivity rule IDs, can be used in create namespace and update namespace. example: --ids id1 --ids id2 --ids id3 Alias: `ids` #### --remove-all Acknowledge that all connectivity rules will be removed, enabling connectivity from any source --- # tcld nexus command reference Source: https://docs.temporal.io/cloud/tcld/nexus > Reference for tcld nexus commands Alias: `nxs` - [tcld nexus endpoint](#endpoint) ### endpoint `tcld nexus endpoint`: Commands for managing Nexus Endpoints (EXPERIMENTAL). Alias: `ep` - [tcld nexus endpoint get](#get) - [tcld nexus endpoint list](#list) - [tcld nexus endpoint create](#create) - [tcld nexus endpoint update](#update) - [tcld nexus endpoint allowed-namespace](#allowed-namespace) - [tcld nexus endpoint delete](#delete) #### get `tcld nexus endpoint get`: Get a Nexus Endpoint by name (EXPERIMENTAL). Alias: `g` ##### --name Endpoint name Alias: `n` #### list `tcld nexus endpoint list`: List Nexus Endpoints (EXPERIMENTAL). Alias: `l` #### create `tcld nexus endpoint create`: Create a new Nexus Endpoint (EXPERIMENTAL). Alias: `c` ##### --name Endpoint name Alias: `n` ##### --description Endpoint description in markdown format (optional) Alias: `d` ##### --description-file Endpoint description file in markdown format (optional) Alias: `df` ##### --target-namespace Namespace in which a handler worker will be polling for Nexus tasks on Alias: `tns` ##### --target-task-queue Task Queue in which a handler worker will be polling for Nexus tasks on Alias: `ttq` ##### --allow-namespace Namespace that is allowed to call this endpoint (optional) Alias: `ans` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### update `tcld nexus endpoint update`: Update an existing Nexus Endpoint (EXPERIMENTAL). Alias: `u` ##### --name Endpoint name Alias: `n` ##### --description Endpoint description in markdown format (optional) Alias: `d` ##### --description-file Endpoint description file in markdown format (optional) Alias: `df` ##### --unset-description Unset endpoint description ##### --target-namespace Namespace in which a handler worker will be polling for Nexus tasks on (optional) Alias: `tns` ##### --target-task-queue Task Queue in which a handler worker will be polling for Nexus tasks on (optional) Alias: `ttq` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### allowed-namespace `tcld nexus endpoint allowed-namespace`: Allowed namespace operations for a Nexus Endpoint (EXPERIMENTAL). Alias: `an` - [tcld nexus endpoint allowed-namespace add](#add) - [tcld nexus endpoint allowed-namespace list](#list) - [tcld nexus endpoint allowed-namespace set](#set) - [tcld nexus endpoint allowed-namespace remove](#remove) ##### add `tcld nexus endpoint allowed-namespace add`: Add allowed namespaces to a Nexus Endpoint (EXPERIMENTAL). Alias: `a` ###### --name Endpoint name Alias: `n` ###### --namespace Namespace that is allowed to call this endpoint Alias: `ns` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### list `tcld nexus endpoint allowed-namespace list`: List allowed namespaces of a Nexus Endpoint (EXPERIMENTAL). Alias: `l` ###### --name Endpoint name Alias: `n` ##### set `tcld nexus endpoint allowed-namespace set`: Set allowed namespaces of a Nexus Endpoint (EXPERIMENTAL). Alias: `s` ###### --name Endpoint name Alias: `n` ###### --namespace Namespace that is allowed to call this endpoint Alias: `ns` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ##### remove `tcld nexus endpoint allowed-namespace remove`: Remove allowed namespaces from a Nexus Endpoint (EXPERIMENTAL). Alias: `r` ###### --name Endpoint name Alias: `n` ###### --namespace Namespace that is allowed to call this endpoint Alias: `ns` ###### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ###### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### delete `tcld nexus endpoint delete`: Delete a Nexus Endpoint (EXPERIMENTAL). Alias: `d` ##### --name Endpoint name Alias: `n` ##### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` ##### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` --- # tcld request command reference Source: https://docs.temporal.io/cloud/tcld/request > Manage asynchronous requests `tcld request`: Manage asynchronous requests. Alias: `r` - [tcld request get](#get) ### get `tcld request get`: Get the request status. Alias: `g` #### --request-id The request-id of the asynchronous request Alias: `r` --- # tcld service-account command reference Source: https://docs.temporal.io/cloud/tcld/service-account > Service Account management operations `tcld service-account`: Service Account management operations. Alias: `sa` - [tcld service-account create](#create) - [tcld service-account create-scoped](#create-scoped) - [tcld service-account list](#list) - [tcld service-account get](#get) - [tcld service-account update](#update) - [tcld service-account delete](#delete) - [tcld service-account set-account-role](#set-account-role) - [tcld service-account set-namespace-permissions](#set-namespace-permissions) ### create `tcld service-account create`: Create a service account. Alias: `c` #### --description The service account description Alias: `d` #### --name The service account name Alias: `n` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --account-role The account role to set on the service account; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read] Alias: `ar` #### --namespace-permission Flag can be used multiple times; value must be "<namespace>=<permission>"; valid types are: [Admin Read Write] Alias: `np` ### create-scoped `tcld service-account create-scoped`: Create a scoped service account (service account restricted to a single namespace). Alias: `cs` #### --description The service account description Alias: `d` #### --name The service account name Alias: `n` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --namespace-permission Value must be "<namespace>=<permission>"; valid types are: [Admin Read Write] Alias: `np` ### list `tcld service-account list`: List service accounts. Alias: `l` #### --page-token Page token for paging list service accounts request Alias: `p` #### --page-size Page size for paging list service accounts request Alias: `s` ### get `tcld service-account get`: Get service account information. Alias: `g` #### --service-account-id The service account id Alias: `id` ### update `tcld service-account update`: Update service account from Temporal Cloud. Alias: `u` #### --service-account-id The service account id Alias: `id` #### --description The service account description Alias: `d` #### --name The service account name Alias: `n` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### delete `tcld service-account delete`: Delete service account from Temporal Cloud. Alias: `d` #### --service-account-id The service account id Alias: `id` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### set-account-role `tcld service-account set-account-role`: Set account role for a service account. Alias: `sar` #### --service-account-id The service account id Alias: `id` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --account-role The account role to set on the service account; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read] Alias: `ar` ### set-namespace-permissions `tcld service-account set-namespace-permissions`: Set entirely new set of namespace permissions for a service account. Alias: `snp` #### --service-account-id The service account id Alias: `id` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --namespace-permission Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write] Alias: `p` --- # tcld user command reference Source: https://docs.temporal.io/cloud/tcld/user > User management operations `tcld user`: User management operations. Alias: `u` - [tcld user list](#list) - [tcld user get](#get) - [tcld user invite](#invite) - [tcld user resend-invite](#resend-invite) - [tcld user delete](#delete) - [tcld user set-account-role](#set-account-role) - [tcld user set-namespace-permissions](#set-namespace-permissions) ### list `tcld user list`: List users. Alias: `l` #### --namespace List users that have permissions to the namespace Alias: `n` #### --page-token Page token for paging list users request Alias: `p` #### --page-size Page size for paging list users request Alias: `s` ### get `tcld user get`: Get user information. Alias: `g` #### --user-id The user id Alias: `id` #### --user-email The user email address of the user Alias: `e` ### invite `tcld user invite`: Invite users to Temporal Cloud. Alias: `i` #### --user-email The email address of the user, you can supply this flag multiple times to invite multiple users in a single request Alias: `e` #### --account-role The account role to set on the user; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read] Alias: `ar` #### --namespace-permission Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write] Alias: `p` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### resend-invite `tcld user resend-invite`: Resend invitation to a user on Temporal Cloud. Alias: `ri` #### --user-id The user id Alias: `id` #### --user-email The user email address of the user Alias: `e` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### delete `tcld user delete`: Delete user from Temporal Cloud. Alias: `d` #### --user-id The user id Alias: `id` #### --user-email The user email address of the user Alias: `e` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### set-account-role `tcld user set-account-role`: Set account role for a user. Alias: `sar` #### --user-id The user id Alias: `id` #### --user-email The user email address of the user Alias: `e` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --account-role The account role to set on the user; valid types are: [Admin Developer FinanceAdmin MetricsRead Owner Read] Alias: `ar` ### set-namespace-permissions `tcld user set-namespace-permissions`: Set entirely new set of namespace permissions for a user. Alias: `snp` #### --user-id The user id Alias: `id` #### --user-email The user email address of the user Alias: `e` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` #### --resource-version The resource-version (etag) to update from, if not set the cli will use the latest (optional) Alias: `v` #### --namespace-permission Flag can be used multiple times; value must be "namespace=permission"; valid types are: [Admin Read Write] Alias: `p` --- # tcld user-group command reference Source: https://docs.temporal.io/cloud/tcld/user-group > User group management operations `tcld user-group`: User group management operations. Alias: `ug` - [tcld user-group list](#list) - [tcld user-group get](#get) - [tcld user-group create](#create) - [tcld user-group set-access](#set-access) - [tcld user-group add-users](#add-users) - [tcld user-group remove-users](#remove-users) - [tcld user-group list-members](#list-members) - [tcld user-group delete](#delete) ### list `tcld user-group list`: List groups. Alias: `l` #### --page-token list groups starting from this page token Alias: `p` #### --page-size number of groups to list Alias: `s` ### get `tcld user-group get`: Get group. Alias: `g` #### --group-id group ID Alias: `id` ### create `tcld user-group create`: Create a new user group. Alias: `c` #### --display-name display name for the group #### --account-role account role (admin, read, developer, owner, financeadmin, none) #### --namespace-role namespace roles Alias: `nr` ### set-access `tcld user-group set-access`: Set group access. Alias: `sa` #### --group-id group ID Alias: `id` #### --account-role account role Alias: `ar` #### --namespace-role namespace roles Alias: `nr` #### --append append namespace roles, cannot be used with remove flag, cannot set account role Alias: `a` #### --remove remove namespace roles, cannot be used with append flag, cannot set account role Alias: `r` ### add-users `tcld user-group add-users`: Add users to a group. Alias: `au` #### --group-id group ID Alias: `id` #### --user-email The email address of the user, you can supply this flag multiple times to add multiple users in a single request Alias: `e` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### remove-users `tcld user-group remove-users`: Remove users from a group. Alias: `ru` #### --group-id group ID Alias: `id` #### --user-email The email address of the user, you can supply this flag multiple times to remove multiple users in a single request Alias: `e` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` ### list-members `tcld user-group list-members`: List all members of a group. Alias: `lm` #### --group-id group ID Alias: `id` ### delete `tcld user-group delete`: Delete a user group. Alias: `d` #### --group-id group ID Alias: `id` #### --request-id The request-id to use for the asynchronous operation, if not set the server will assign one (optional) Alias: `r` --- # tcld version command reference Source: https://docs.temporal.io/cloud/tcld/version > Version information `tcld version`: Version information. Alias: `v` --- # Temporal Cloud Terraform provider Source: https://docs.temporal.io/cloud/terraform-provider The Terraform Temporal Cloud provider allows you to use Terraform to manage resources for Temporal Cloud. The Terraform tool manages infrastructure as code (IaC). With this provider, you can use Terraform to automate Temporal Cloud resource management, including Namespaces, Users, Service Accounts, API Keys and more. > **📝 Note:** > Terraform Management > > Once a resource is managed by Terraform, you should only use Terraform to manage that resource. > Resources: - The [Temporal Cloud Terraform provider](https://registry.terraform.io/providers/temporalio/temporalcloud/latest) is available in the Terraform Registry, where you can find detailed documentation on the Provider's supported resources and data sources. - The GitHub repository for the Terraform provider is [terraform-provider-temporalcloud](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main), where you can report bugs, provide feature requests, and [contribute](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/CONTRIBUTING.md) to the provider. We encourage your input as we develop the provider with the community. - To view the list of available Temporal Cloud resources supported by Terraform provider, visit the resources section of the Terraform documentation in Hashi's [registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs). ### Prerequisites To use the Terraform provider, you'll need the following: - The [Terraform CLI](https://developer.hashicorp.com/terraform/cli) - An [API Key](/cloud/api-keys): an API Key is required to use the Terraform provider. - See [the API docs](/cloud/api-keys#generate-an-api-key) for instructions on generating an API Key. > **📝 Note:** > OpenTofu Registry > > Our Terraform Provider is registered with [OpenTofu](https://opentofu.org), but that registration is not maintained or > managed by Temporal Technologies. > ## Setup Generate an [API Key](/cloud/api-keys#generate-an-api-key) to authenticate Terraform operations with your Temporal Cloud account or a Service Account. Then, either use an environment variable or pass the API Key into the provider manually to manage your Temporal Cloud Terraform resources. Follow these examples to use an environment variable to pass in your API Key to the provider. **macOS** Export your environment variable for secure access to the API Keys. ```bash # Replace with the token output from `temporal cloud apikey create-for-me` or `tcld apikey create`. export TEMPORAL_CLOUD_API_KEY= ``` > **💡 Tip:** > ENVIRONMENT VARIABLES > > Do not confuse environment variables, set with your shell, with temporal env options. > **Windows** Export your environment variable for secure access to the API Keys. ```bash # Replace with the token output from `temporal cloud apikey create-for-me` or `tcld apikey create`. set TEMPORAL_CLOUD_API_KEY= ``` > **💡 Tip:** > ENVIRONMENT VARIABLES > > Do not confuse environment variables, set with your shell, with temporal env options. > Or, pass it in manually in your .tf file using the provider code block ```yml provider "temporalcloud" { api_key = "my-temporalcloud-api-key" } ``` ## Manage Temporal Cloud Namespaces with Terraform Terraform is a great way to automate the management of Temporal Namespaces. It doesn't matter whether you want management to be centralized within a platform team or federated to different product teams. The provider allows you to import, create, update, and delete Namespaces with Terraform. You must use an Identity with Temporal Cloud Namespace management privileges. This includes the Account Owner, Global Admin, or Developer Account Role. For more detailed examples on how to manage Namespaces via Terraform, check the [Terraform Registry documentation for Namespaces](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace). **How do I create a Namespace with Terraform?** 1. Create a Terraform configuration file (`terraform.tf`) to define a Namespace. ```hcl terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } } provider "temporalcloud" { } resource "temporalcloud_namespace" "namespace" { name = "terraform" regions = ["aws-us-east-1"] accepted_client_ca = base64encode(file("ca.pem")) retention_days = 14 } ``` In this example, you create a Temporal Cloud Namespace named `terraform`, specifying the AWS region `aws-us-east-1`, and specifying the path to the CA certificate. 1. Initialize the Terraform provider. Run the following command to initialize the Terraform provider. ```bash terraform init ``` 1. Apply the Terraform configuration. Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account. ```bash terraform apply ``` Follow the onscreen prompts. Upon completion, you'll see a success message indicating your Namespace is created. ```bash temporalcloud_namespace.terraform: Creation complete after 2m17s [id=] ``` You can find more examples of Namespace management in the Terraform Provider docs located on HashiCorp's [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace). The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace with API Key based authentication. **How do I validate the creation of the Namespace?** You can validate the creation of the Namespace through the Temporal Web UI or through the CLI. **Using the Temporal Web UI** 1. Log into the Temporal Cloud Web UI. 1. Navigate to the Namespaces page. 1. Search for the Namespace you created. **Using the CLI** Validate the creation of your Namespace through the Terraform provider. To validate see your Namespace in the Cloud UI or through the `namespace get` command. Run the command and pass in your [Cloud Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name) and [Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id): **Temporal CLI** ```bash temporal cloud namespace get -n "." ``` **tcld** ```bash tcld namespace get -n "." ``` **How do I update a Temporal Cloud Namespace?** Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal. For example, change the retention period setting in the Terraform file from the previous example and watch Terraform apply the change without any additional steps required by you. 1. Set the retention period to 30 days. ```hcl terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" version = ">= 0.0.6" } } } provider "temporalcloud" { } resource "temporalcloud_namespace" "namespace" { name = "terraform" regions = ["aws-us-east-1"] accepted_client_ca = base64encode(file("ca.pem")) retention_days = 30 } ``` 1. Apply your configuration. When prompted, answer yes to continue: ```command terraform apply ``` Upon completion, you will see a success message indicating your Namespace has been updated. It may take several minutes to update a Namespace. ```text temporalcloud_namespace.namespace: Modifications complete after 10s [id=terraform.a1bb2] ``` **How do I delete a Temporal Cloud Namespace?** To delete a Namespace, remove the `temporalcloud_namespace` resource and all dependent resource configurations from your Terraform files and run the `terraform apply` command. Upon completion, you will see a success message indicating the resource has been destroyed: ```text temporalcloud_namespace.my_namespace: Destruction complete after 3s Apply complete! Resources: 0 added, 0 changed, 1 destroyed. ``` > **📝 Note:** > Preventing Deletion > > You can prevent deletion of any Terraform resource by including the `prevent_destroy` argument in the Terraform > configuration file. > **How do I import a Temporal Cloud Namespace?** If you have an existing Namespace in Temporal Cloud, you can import it into Terraform to manage the Namespace from Terraform using the `terraform import` command. 1. Provide a configuration placeholder in your Terraform configuration. ```yml resource "temporalcloud_namespace" "namespace" { } ``` 1. Run the `terraform import` command from the command line and pass in the Namespace ID. Your Namespace ID is available at the top of the Namespace's page in the Temporal Cloud UI and is in the format `namespaceid.acctid`. ```bash terraform import temporalcloud_namespace.terraform namespaceid.acctid ``` The Namespace is now a part of the Terraform state and all changes to the Namespace should be managed by Terraform. > **⚠️ Caution:** > > Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors > on subsequent Terraform operations. > ## Manage Temporal Cloud Nexus Endpoints with Terraform Terraform provides a great way to automate the management of [Nexus Endpoints](/nexus/endpoints). The provider allows you to import, create, update, and delete Nexus Endpoints with Terraform. You must use an Identity with [Developer role (or higher)](/cloud/manage-access/roles-and-permissions#account-level-roles) and [Namespace Admin permission](/cloud/manage-access/roles-and-permissions#namespace-level-permissions) on the Endpoint's target Namespace. **How do I create a Nexus Endpoint with Terraform?** 1. Create a Terraform configuration file (`terraform.tf`) to define a Nexus Endpoint. From the [example in the Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint): ```yml terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } } provider "temporalcloud" { } resource "temporalcloud_namespace" "target_namespace" { name = "terraform-target-namespace" regions = ["aws-us-west-2"] api_key_auth = true retention_days = 14 timeouts { create = "10m" delete = "10m" } } resource "temporalcloud_namespace" "caller_namespace" { name = "terraform-caller-namespace" regions = ["aws-us-east-1"] api_key_auth = true retention_days = 14 timeouts { create = "10m" delete = "10m" } } resource "temporalcloud_namespace" "caller_namespace_2" { name = "terraform-caller-namespace-2" regions = ["gcp-us-central1"] api_key_auth = true retention_days = 14 timeouts { create = "10m" delete = "10m" } } resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { name = "terraform-nexus-endpoint" description = <<-EOT Service Name: my-hello-service Operation Names: echo say-hello Input / Output arguments are in the following repository: https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go EOT worker_target = { namespace_id = temporalcloud_namespace.target_namespace.id task_queue = "terraform-task-queue" } allowed_caller_namespaces = [ temporalcloud_namespace.caller_namespace.id, temporalcloud_namespace.caller_namespace_2.id, ] } ``` In this example, 3 Namespaces are created: - target Namespace for a Nexus Endpoint - Nexus requests will be routed to a Worker that polls the target Namespace. - caller Namespace(s) - Nexus Operations are invoked from caller Namespace, for example from a caller Workflow. These Namespaces are referenced in the [Nexus Endpoint](/nexus/endpoints) configuration: - `worker_target` (Namespace and Task Queue) - currently only a single worker_target is supported. - `allowed_caller_namespaces` - used to enforce Nexus Endpoint [runtime access controls](/nexus/security#runtime-access-controls). 1. Initialize the Terraform provider. Run the following command to initialize the Terraform provider. ```bash terraform init ``` 1. Apply the Terraform configuration. Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account. ```bash terraform apply ``` Follow the onscreen prompts. Upon completion, you'll see a success message indicating 3 Namespaces and a Nexus Endpoint are created. ```bash temporalcloud_nexus_endpoint.nexus_endpoint: Creation complete after 2s [id=b158063be978471fa1d200569b03834d] ``` You can find more examples of Nexus Endpoint management in the Terraform Provider docs located on HashiCorp's [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint). The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace with API Key based authentication. **How do I validate the creation of the Nexus Endpoint?** You can validate the creation of the Nexus Endpoint through the Temporal Web UI or through the CLI. **Using the Temporal Web UI** 1. Log into the Temporal Cloud Web UI. 1. Navigate to [the Nexus page](https://cloud.temporal.io/nexus). 1. Search for the Nexus Endpoint you created, using only the Nexus Endpoint Name (without an account suffix). **Using the CLI** Validate the creation of your Nexus Endpoint through the Terraform provider. To validate see your Nexus Endpoint in the Cloud UI or through the `nexus endpoint get` command. Run the below command using your Nexus Endpoint Name. Do not use the account ID suffix with this endpoint name: **Temporal CLI** ```bash temporal cloud nexus endpoint get --name "" ``` **tcld** ```bash tcld nexus endpoint get -n "" ``` **How do I update a Nexus Endpoint?** Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal. For example, to change the allowed caller Namespaces on a Nexus Endpoint: 1. Add or remove allowed caller Namespaces by updating the Nexus Endpoint configuration, for example by removing `caller_namespace_2` from the configuration above: ```yml resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { name = "terraform-nexus-endpoint" description = <<-EOT Service Name: my-hello-service Operation Names: echo say-hello Input / Output arguments are in the following repository: https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go EOT worker_target = { namespace_id = temporalcloud_namespace.target_namespace.id task_queue = "terraform-task-queue" } allowed_caller_namespaces = [ temporalcloud_namespace.caller_namespace.id ] } ``` 1. Apply your configuration. When prompted, answer yes to continue: ```command terraform apply ``` Upon completion, you will see a success message indicating your Nexus Endpoint has been updated. It may take several seconds to update a Nexus Endpoint in the Control Plane which is visible from the Temporal UI or the CLI. Propagation of Nexus Endpoint changes to the data plane may take longer, but usually complete in less than one minute. ```text temporalcloud_nexus_endpoint.nexus_endpoint: Modifications complete after 1s [id=b158063be978471fa1d200569b03834d] ``` **How do I delete a Nexus Endpoint?** To delete a Nexus Endpoint, remove the `temporalcloud_nexus_endpoint` resource configuration from your Terraform files and run the `terraform apply` command. Upon completion, you will see a success message indicating the resource has been destroyed: ```text temporalcloud_nexus_endpoint.my_nexus_endpoint: Destruction complete after 3s Apply complete! Resources: 0 added, 0 changed, 1 destroyed. ``` **How do I import a Temporal Cloud Nexus Endpoint?** If you have an existing Nexus Endpoint in Temporal Cloud, you can import it into Terraform to manage the Nexus Endpoint from Terraform using the `terraform import` command. 1. Initialize the Terraform provider in a new directory. Run the following command to initialize the Terraform provider. ```bash terraform init ``` 1. Provide a configuration placeholder in your Terraform configuration and ensure you've included your [API key](#setup). ```yml terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } } provider "temporalcloud" { } resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { } ``` 1. Run the `terraform import` command from the command line and pass in the Nexus Endpoint ID. ```bash terraform import temporalcloud_nexus_endpoint ``` Your Nexus Endpoint ID is available at the top of the Nexus Endpoint's page in the [Temporal Cloud UI](https://cloud.temporal.io/nexus). ![Nexus Endpoint ID](/img/cloud/nexus/nexus-endpoint-id.png) Upon completion, you will see a success message indicating the Nexus Endpoint was imported. ```text temporalcloud_nexus_endpoint.nexus_endpoint: Refreshing state... [id=3c0c75ccfa8144b092c13ce632463761] Import successful! ``` The Nexus Endpoint is now a part of the Terraform state and all changes to the Nexus Endpoint should be managed by Terraform. > **⚠️ Caution:** > > Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors > on subsequent Terraform operations. > ## Manage Temporal Cloud Users with Terraform Manage Temporal Cloud Users with the same process you use to manage Namespaces with Terraform. The following examples create, update, delete, and import Temporal Cloud Users with `terraform apply` commands on the Terraform configuration file. > **📝 Note:** > User Management > > Cautions about Temporal User management: > > - Terraform can't manage the Temporal Account Owner role. While you can import an Account Owner to Terraform, you cannot > create, update, or delete an Account Owner with Terraform. > - Right now, you can't manage a user's access to a Namespace from the Namespace resource. You must manage Namespace > access from the User resource. This is also true for Service Accounts. > - Account Owners and Global Admins automatically gain access to all Namespaces in Temporal. Therefore, you cannot > specify Namespace access for these roles. This is also true for Service Accounts. > - Follow Terraform best practices for resource management. Manage a specific user in one and only one .tf file. There's > a risk that you may overwrite a user's permissions if you don't. > - To Import a user, you'll need the User's ID which is currently not available in the Temporal Cloud UI. You can fetch > current User ID by running the `temporal cloud user list` or `tcld user list` command. > For more detailed examples on how to manage Namespaces via Terraform, check the Terraform Registry documentation for [provisioning a Temporal Cloud user](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/user). **How do I create a Temporal Cloud User with Terraform?** 1. Add a Terraform User resources configuration to your Terraform file. ```hcl terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } } provider "temporalcloud" { } resource "temporalcloud_namespace" "namespace" { name = "terraform" regions = ["aws-us-east-1"] accepted_client_ca = base64encode(file("ca.pem")) retention_days = 14 } # Global admins automatically have access to all namespaces. resource "temporalcloud_user" "global_admin" { email = "admin@example.com" account_access = "Admin" } # Developers can be granted explicit namespace permissions. resource "temporalcloud_user" "namespace_admin" { email = "developer@example.com" account_access = "Developer" namespace_accesses = [{ namespace_id = temporalcloud_namespace.namespace.id permission = "Write" }] } ``` Replace the email and domain values with your Temporal Cloud User email and domain. 1. Apply your configuration. When prompted, answer yes to continue: ```command terraform apply ``` Upon completion, you will see a success message indicating your User has been created. ```text temporalcloud_user.namespace_admin: Creation complete after 1s [id=12a34bc5678910d38d9e8390636e7412] Apply complete! Resources: 2 added, 0 changed, 0 destroyed. ``` **How do I update a Temporal Cloud User with Terraform?** To update a User with Terraform, follow the same steps used to create a User. **How do I delete a Temporal Cloud User with Terraform?** To delete a User with Terraform, remove the Terraform User resources configuration from your Terraform file and run the `terraform apply` command. 1. Remove the Terraform User resources configuration from your Terraform file. ```hcl terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" version = ">= 0.0.6" } } } provider "temporalcloud" { } resource "temporalcloud_namespace" "namespace" { name = "terraform" regions = ["aws-us-east-1"] accepted_client_ca = base64encode(file("ca.pem")) retention_days = 14 } # This user will be deleted after running `terraform apply` resource "temporalcloud_user" "global_admin" { email = "admin@example.com" account_access = "Admin" } # The following user resource has been removed (or commented out), # so Terraform will delete it. # resource "temporalcloud_user" "namespace_admin" { # email = "developer@example.com" # account_access = "Developer" # # namespace_accesses = [{ # namespace_id = temporalcloud_namespace.namespace.id # permission = "Write" # }] # } ``` 1. Run the `terraform apply` command. When prompted, answer yes to continue: ```command terraform apply ``` Upon completion, you will see a success message indicating your User has been deleted. ```text temporalcloud_user.namespace_admin: Destruction complete after 2s Apply complete! Resources: 0 added, 0 changed, 1 destroyed. ``` **How do I import a Temporal User?** If you have an existing User in Temporal Cloud, you can import it into Terraform using the `terraform import` command. 1. Provide a configuration placeholder in your Terraform configuration. ```yml resource "temporalcloud_user" "user" { } ``` 1. Run the `terraform import` command and pass in the User ID. Your User ID is available by running `temporal cloud user list` or `tcld u l`. ```bash terraform import temporalcloud_user.user 72360058153949edb2f1d47019c1e85f ``` The User is now a part of the Terraform state and all changes to the User should be managed by Terraform. ## Manage Temporal Cloud Service Accounts with Terraform The process and steps to managing a Service Account with Terraform are very similar to managing a User with Terraform with a few small differences: - Service Accounts use the Service Account Terraform resource not the User resource. - Service Accounts do not have email addresses, they have names instead. This means you should specify a name for a Service Account instead of an email. Everything else about managing Services Accounts with Terraform follows the same process, guidance, and limitations of managing Users with Terraform. ## Manage Temporal Cloud API Keys with Terraform You can manage your own, personal API Keys and Service Account API Keys with Terraform. The process and steps to managing an API Key with Terraform are very similar to managing other resources with Terraform. You can create, delete, update and import API Keys with Terraform. One difference between working with API Keys as a Terraform resource compared to other Temporal Cloud resources is the need to access an API Keys secure token output from Terraform. Walk through the process of securely accessing the API Key Token in the Create section of this guide. > **📝 Note:** > Limits and Best Practices > > - See the API Key [documentation](/cloud/api-keys) for information about the limits and best > practices for managing API Keys. > - See Terraform's documentation on working with > [sensitive data](https://www.terraform.io/docs/language/values/variables.html#sensitive-values) for more information > on how to manage sensitive data in Terraform. > **How do I create a Temporal Cloud API Key with Terraform?** From the example in the [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/apikey): 1. Add a Terraform API Key resources configuration to your Terraform file. ```hcl terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } } provider "temporalcloud" { } resource "temporalcloud_service_account" "global_service_account" { name = "admin" account_access = "Admin" } resource "temporalcloud_apikey" "global_apikey" { display_name = "admin" owner_type = "service-account" owner_id = temporalcloud_service_account.global_service_account.id expiry_time = "2024-11-01T00:00:00Z" disabled = false } ``` Make sure to: - Replace the display_name, expiry_time, and disabled values with your Temporal Cloud API Key configuration. - Replace the owner_type and owner_id values with your Temporal Cloud Service Account or other Identity information. 1. Create an output.tf file and add the following code to output the API Key Token. ```hcl output "apikey_token" { value = temporalcloud_apikey.global_apikey.token sensitive = true } ``` 1. Apply your configuration. When prompted, answer yes to continue: ```command terraform apply ``` Upon completion, you will see a success message indicating the API Key has been created. ```text temporalcloud_apikey.global_apikey: Creation complete after 1s [id=kayBf38JIWkMPmnfr59iEIaEk2L7uqR4] ``` 1. Access the API Key Token securely. You'll notice that if you view the state for the API Key resource, the token value is not displayed. ```bash terraform state show temporalcloud_apikey.global_apikey # temporalcloud_apikey.global_apikey: resource "temporalcloud_apikey" "global_apikey" { disabled = false display_name = "adminKey3" expiry_time = "2024-12-01T00:00:00Z" id = "kayBf38JIWkMPmnfr59iEIaEk2L7uqR4" owner_id = "b81336a6097449cba75c2e5500df3d31" owner_type = "service-account" state = "active" token = (sensitive value) } ``` To access the token, you can use the Terraform output command. ```bash terraform output -json apikey_token ``` This will display the token value in the terminal. > **ℹ️ Info:** > Security and API Keys > > Remember, keep your Terraform state files secure if you're managing API Keys with Terraform. The state file contains > sensitive information, like the API Key Token, that should not be shared or exposed. > **How do I update a Temporal Cloud API Key with Terraform?** To update an API Key with Terraform, follow the same steps used to create an API Key. > **📝 Note:** > Editing Fields > > You can only edit an API Key's name or description field. Updating an API Key does not generate a new secure token > **How do I delete a Temporal Cloud API Key with Terraform?** To delete an API Key with Terraform, remove the Terraform API Key resources configuration from your Terraform and output.tf files and run the `terraform apply` command. **How do I Import a Temporal API Key?** You cannot import an API Key into Terraform. Once created, the API Key secret isn't stored and can't be retrieved, so you can't access it using import. Instead, Temporal recommends creating a new API Key using Terraform directly. ## Data Sources - Regions and Namespaces The Terraform provider also supports 2 data sources that provide you access to the available Regions and Namespaces in your Temporal Cloud account. > **📝 Note:** > Terraform Data Sources > > See Terraform [documentation](https://developer.hashicorp.com/terraform/language/data-sources) to learn more about > Terraform Data Sources > For example, to retrieve a list of regions available for your account, you can use the regions data_source ```hcl data "temporalcloud_regions" "regions" {} output "regions" { value = data.temporalcloud_regions.regions.regions } ``` ## Community Involvement Do you have feedback about the provider? Want to report a bug or request a feature? We'd love to hear from you. - Please reach out to us in the Temporal Community [Slack](https://join.slack.com/t/temporalio/shared_invite/zt-2u2ey8ilu-LRxnd3PSoAk9GZ94UuzoBA) in the #terraform channel - Feel free to create issues and contribute PRs in the Temporal Terraform [GitHub repository](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main) --- # Monitor Worker health Source: https://docs.temporal.io/cloud/worker-health > Detect and configure for Task backlogs, greedy Worker resources, misconfigured Workers, and Sticky cache settings. Optimize alert systems and get actionable insights on metrics like Schedule-To-Start latency, Sync Match Rate, and Poll Success Rate for improved application health. This page is a guide to monitoring a Temporal Worker fleet and covers the following scenarios: - [Configuring minimal observations](#minimal-observations) - [How to detect a backlog of Tasks](#detect-task-backlog) - [How to detect greedy Worker resources](#detect-greedy-workers) - [How to detect misconfigured Workers](#detect-misconfigured-workers) - [How to configure Sticky cache](#configure-sticky-cache) This page assumes you are monitoring both Worker-side SDK metrics and Cloud-side metrics. Use SDK metrics to understand what your Workers are doing, and Cloud metrics to understand what Temporal Cloud is seeing at the Task Queue and service level. For an overview of how these signals fit together, see [Temporal Cloud metrics](/cloud/metrics). > **💡 Tip:** > > You can also inspect Workers and the Workers assigned to a Task Queue directly in the Temporal UI. See [Visualize Workers in the UI](/develop/worker-performance#visualize-workers). > ## Minimal Observations These alerts should be configured and understood first to gain intelligence into your application health and behaviors. 1. Create monitors and alerts for Schedule To Start latency SDK metrics (both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency)). See [Detect Task backlog section](#detect-task-backlog) to explore [sample queries](#prometheus-query-samples) and appropriate responses that accompany these values. - Alert at >200ms for your p99 value - Plot >100ms for your p95 value 2. Create a [Grafana](/cloud/metrics/prometheus-grafana) panel called Sync Match Rate. See the [Sync Match Rate section](#sync-match-rate) to explore example queries and appropriate responses that accompany these values. - Alert at \<95% for your p99 value - Plot \<99% for your p95 value 3. Create a [Grafana](/cloud/metrics/prometheus-grafana) panel called Poll Success Rate. See the [Detect greedy Workers section](#detect-greedy-workers) for example queries and appropriate responses that accompany these values. - Alert at \<90% for your p99 value - Plot \<95% for your p95 value The following alerts build on the above to dive deeper into specific potential causes for Worker related issues you might be experiencing. 1. Create monitors and alerts for the [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available) SDK metric. See the [Detect misconfigured Workers section](#detect-misconfigured-workers) for appropriate responses based on the value. - Alert at 0 for your p99 value 2. Create monitors for the [temporal_sticky_cache_size](/references/sdk-metrics#sticky_cache_size) SDK metric. See the [Configure Sticky Cache section](#configure-sticky-cache) for more details on this configuration. - Plot at \{value\} > \{WorkflowCacheSize.Value\} 3. Create monitors for the [temporal_sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction) SDK metric. This metric is available in the Go SDK, and the Java SDK only. See the [Configure Sticky Cache section](#configure-sticky-cache) for more details and appropriate responses. - Alert at >\{predetermined_high_number\} 4. Create monitors for the [temporal_cloud_v1_approximate_backlog_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) Cloud metric. This metric provides a server-side view of how many Tasks are waiting in a Task Queue and complements the SDK Schedule To Start latency metrics. - Alert when the value is growing over time for a given Task Queue ## Detect Task Backlog ### Symptoms of high Task backlog If the Task backlog is too high, you will find that tasks are waiting to find Workers to run on. This can cause a delay in Workflow execution. Detecting a growing Task backlog is possible by watching the Schedule To Start latency, sync match rate, and approximate backlog count. Metrics to monitor: - **SDK metric**: [workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency) - **SDK metric**: [activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency) - **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_count) - **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_sync_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_sync_count) - **Temporal Cloud metric**: [temporal_cloud_v1_approximate_backlog_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) ### Schedule To Start latency The Schedule To Start metric represents how long Tasks are staying unprocessed in the Task Queues. It is the time between when a Task is enqueued and when it is started by a Worker. This time being long (likely) means that your Workers can't keep up - either increase the number of Workers (if the host load is already high) or increase the number of pollers per Worker. If your Schedule To Start latency alert triggers or is high, check the [Sync Match Rate](#sync-match-rate) to decide if you need to adjust your Worker or fleet, or contact Temporal Cloud support. If your Sync Match Rate is low, contact [Temporal Cloud support](/cloud/support#support-ticket). If your Sync Match Rate is low, you can contact Temporal Cloud support. The schedule_to_start_latency SDK metric for both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency) should have alerts. #### Prometheus query samples **Workflow Task Latency, 99th percentile** ``` histogram_quantile(0.99, sum(rate(temporal_workflow_task_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue)) ``` **Workflow Task Latency, average** ``` sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue) / sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue) ``` **Activity Task Latency, 99th percentile** ``` histogram_quantile(0.99, sum(rate(temporal_activity_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue)) ``` **Activity Task Latency, average** ``` sum(increase(temporal_activity_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue) / sum(increase(temporal_activity_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue) ``` **Target** This latency should be very low, close to zero. Any higher value indicates a bottleneck. ### Sync Match Rate The sync match rate measures the rate of Tasks that are delivered to workers without having to be persisted (workers are up and available to pick them up) to the rate of all delivered tasks. A sync match is when a task is immediately matched to a Worker via the Sticky Queue. An async match is when a Task cannot be matched to the Sticky Queue for a Worker. This can happen when no Worker has cached the Workflow, or if the Task times out during processing. In this case, the Task returns to the general Task Queue. **Calculate Sync Match Rate** ``` temporal_cloud_v1_poll_success_sync_count / temporal_cloud_v1_poll_success_count = N ``` #### Prometheus query samples **sync_match_rate query** ``` sum by(temporal_namespace) ( temporal_cloud_v1_poll_success_sync_count{temporal_namespace=~"$namespace"} ) / sum by(temporal_namespace) ( temporal_cloud_v1_poll_success_count{temporal_namespace=~"$namespace"} ) ``` **Target** The Sync Match Rate should be at least >95%, but preferably >99%. ### Handling Task backlog issues Once you have detected the condition of a high Task backlog, consider the scenarios below to take action. #### High Schedule To Start latency and high sync match rate There are three typical causes for this: - There are not enough workers to perform work - Each worker is either under resourced, or is misconfigured, to handle enough work - There is congestion caused by the environment (for example, network) hosting the worker(s) and Temporal Cloud Consider - Increasing either the number of available workers - Verifying that your worker hosts are appropriately resourced - Increasing the worker configuration value for concurrent pollers for workers/task executions (if your worker resources can accommodate the increased load) - Doing some combination of these #### High Schedule To Start latency and low sync match rate Verify that you have not set a value for `ScheduleToStartTimeout` in your Activity Options. This may skew your observations and note that this timeout is non-retryable. It may be acceptable for your use case to have low sync match rate. For example, if you have known workloads or you intentionally throttle tasks. In this case it's also important to understand the fill and drain rates of the async tasks are during these windows: Successful async polls ``` temporal_cloud_v1_poll_success_count - temporal_cloud_v1_poll_success_sync_count = N ``` ``` sum by(temporal_namespace, task_type) ( temporal_cloud_v1_poll_success_count{temporal_namespace=~"$namespace"} ) - sum by(temporal_namespace, task_type) ( temporal_cloud_v1_poll_success_sync_count{temporal_namespace=~"$namespace"} ) ``` You can also monitor the [approximate backlog count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) to observe Task Queue depth directly: ``` temporal_cloud_v1_approximate_backlog_count{temporal_namespace=~"$namespace", temporal_task_queue=~"$task_queue"} ``` **Actions** - Verify that your Worker setup is optimized for your instance: - Check the system CPU usage against `task_slots` and adjust `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` settings as necessary. - Check the system memory usage against `sticky_cache_size` and adjust sticky cache size as necessary. - For a detailed explanation of settings, see the [Worker Performance](/develop/worker-performance/task-queues#task-queues-processing-tuning) section. - Increase the Worker config for concurrent pollers for Workflow or Activity `task_slots`, if your Worker resources can accommodate the increased load. - Reference [Worker Performance > Poller Count](/develop/worker-performance/task-queues#poller-count). - Increase the number of available Workers. > **⚠️ Warning:** > > Setting the [Schedule To Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) in your Activity Options can skew your observations. > Avoid setting a Schedule To Start Timeout when load testing for latency. > ## Detect greedy Worker resources **How to detect greedy Worker resources.** You can have too many Workers. If you see the Poll Success Rate showing low numbers, you might have too many resources polling Temporal Cloud. Metrics to monitor: - **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_count) - **Temporal Cloud metric**: [temporal_cloud_v1_poll_success_sync_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_success_sync_count) - **Temporal Cloud metric**: [temporal_cloud_v1_poll_timeout_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_poll_timeout_count) - **SDK metric**: [temporal_workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency) - **SDK metric**: [temporal_activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency) **Calculate Poll Success Rate** ``` (temporal_cloud_v1_poll_success_count) / (temporal_cloud_v1_poll_success_count + temporal_cloud_v1_poll_timeout_count) ``` **Target** Poll Success Rate should be >90% in most cases of systems with a steady load. For high volume and low latency, try to target >95%. **Interpretation** There may be too many Pollers for the amount of Workers. If you see all of the following at the same time then you might have too many Workers: - Low poll success rate - Low Schedule To Start latency - Low worker host resource utilization **Actions** Consider sizing down your Workers by either: - Reducing the number of Workers polling the impacted Task Queue, OR - Reducing the concurrent pollers per Worker, OR - Both of the above ## Detect misconfigured Workers **How to detect misconfigured Workers.** Worker configuration can negatively affect Task processing efficiency. Metrics to monitor: - **SDK metric**: [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available) - **SDK metric**: [sticky_cache_size](/references/sdk-metrics#sticky_cache_size) - **SDK metric**: [sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction) **Execution Size Configuration** The `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` define the number of total available slots for the Worker. If this is set too low, the Worker will not be able to keep up processing Tasks. **Target** The `temporal_worker_task_slots_available` metric should always be >0. #### Prometheus query samples **Over Time** ``` avg_over_time(temporal_worker_task_slots_available{namespace="$namespace",worker_type="WorkflowWorker"}[10m]) ``` **Current Time** ``` temporal_worker_task_slots_available{namespace="default", worker_type="WorkflowWorker", task_queue="$task_queue_name"} ``` **Interpretation** You are likely experiencing a Task backlog if you are seeing inadequate slot counts frequently. The work is not getting processed as fast as it should/can. **Action** Increase the `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` values and keep an eye on your Worker resource metrics (CPU utilization, etc) to make sure you haven't created a new issue. ### Configure Sticky Execution Cache Sticky Execution means that a Worker caches a Workflow Execution Event History and creates a dedicated Task Queue to listen on. It significantly improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories. **Target** The `sticky_cache_size` should report less than or equal to your `WorkflowCacheSize` value. Also, sticky_cache_total_forced_eviction should not be reporting high numbers (relative). **Action** If you see a high eviction count, verify there are no other inefficiencies in your Worker configuration or resource provisioning (backlog). If you see the cache size metric exceed the `WorkflowCacheSize`, increase this value if your Worker resources can accommodate it or provision more Workers. Finally, take time to review [the Worker performance guide](/develop/worker-performance) and see if it addresses other potential cache issues. #### Prometheus query samples **Sticky Cache Size** ``` max_over_time(temporal_sticky_cache_size{namespace="$namespace"}[10m]) ``` **Sticky Cache Evictions** ``` rate(temporal_sticky_cache_total_forced_eviction_total{namespace="$namespace"}[5m])) ``` ## Manage Worker Heartbeating > **Public Preview** Workers send a heartbeat to Temporal Server every 60 seconds by default. This heartbeat serves to provide liveness and configuration data from the Worker to the Server. Specific data sent can be found in the [API](https://github.com/temporalio/api/blob/main/temporal/api/worker/v1/message.proto). By providing a consistent heartbeat from Workers, the Server can obtain an accurate count of Workers, understand Worker performance, and respond to Worker heartbeats with commands. Some examples of how this is useful: - understanding the difference between a Worker that is down and a Worker that is processing tasks for a long time - identifying a Worker with high CPU usage from the Server point of view Use the Temporal CLI to view information about all Workers connected to Temporal Server. Use `temporal worker describe` to see details of a specific Worker. Use `temporal worker list` to get a complete list of all connected Workers. If you wish to disable Worker heartbeating or set heartbeating to be more frequent than every 60 seconds (allowed range is 1s to 60s), set the configuration relevant to your SDK. The server needs Worker heartbeating to understand Worker status accurately. Disabling the Worker heartbeat will cause features that provide the list of active Workers and information about those Workers to show missing or inaccurate information. **Go** _Available since Go SDK v1.41.0_ Set the `WorkerHeartbeatInterval` field on [`client.Options`](https://pkg.go.dev/go.temporal.io/sdk/client#Options) to adjust the heartbeat interval. Set it to a negative value to disable heartbeating. #### Enable host resource reporting By default, the Go SDK reports `0` for CPU and memory usage in Worker heartbeats. Set `SysInfoProvider` on [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/worker#Options) to enable host resource reporting. Host resource reporting is not included in the core SDK module. Add the [sysinfo](https://pkg.go.dev/go.temporal.io/sdk/contrib/sysinfo) contrib package to your imports - it provides a [gopsutil](https://github.com/shirou/gopsutil)-based implementation that supports cgroup metrics in containerized Linux environments: ```go import ( "go.temporal.io/sdk/contrib/sysinfo" "go.temporal.io/sdk/worker" ) w := worker.New(c, "my-task-queue", worker.Options{ SysInfoProvider: sysinfo.SysInfoProvider(), }) ``` You can also implement the `worker.SysInfoProvider` interface to provide your own resource metrics. **Python** _Available since Python SDK v1.20.0_ Use `TelemetryConfig()` to adjust heartbeat settings. See the [Python SDK documentation](https://python.temporal.io/temporalio.bridge.runtime.RuntimeOptions.html#worker_heartbeat_interval_millis) for more details. **TypeScript** _Available since TypeScript SDK v1.14.0_ Set the `workerHeartbeatInterval` property on [`RuntimeOptions`](https://typescript.temporal.io/api/interfaces/worker.RuntimeOptions) to adjust the heartbeat interval. Set it to `0` to disable heartbeating. **.NET** _Available since .NET SDK v1.10.0_ Set the `WorkerHeartbeatInterval` property on [`TemporalRuntimeOptions`](https://dotnet.temporal.io/api/Temporalio.Runtime.TemporalRuntimeOptions.html) to adjust the heartbeat interval. Set it to `null` to disable heartbeating. **Java** _Available since Java SDK v1.35.0_ Set the heartbeat interval on [`WorkflowClientOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClientOptions.Builder.html) with `setWorkerHeartbeatInterval(Duration)`. Set it to a negative `Duration` to disable heartbeating. **Ruby** _Available since Ruby SDK v1.1.0_ Add configurations to `Runtime()` to adjust heartbeat settings. See the [Ruby SDK documentation](https://ruby.temporal.io/Temporalio/Runtime.html) for more details. --- # Codec Server Source: https://docs.temporal.io/codec-server A Codec Server is an HTTP/HTTPS server that you host and operate. It runs your [Payload Codec](/payload-codec) logic to encode and decode [Payloads](/dataconversion#payload) on behalf of the Temporal CLI and Web UI. The Codec Server is independent of the Temporal Service. Encryption keys and codec logic remain in your environment. For setup instructions, see [Codec Server setup](/production-deployment/data-encryption#codec-server-setup). ## Why use a Codec Server When you apply a custom [Payload Codec](/payload-codec) for encryption or compression, data stored in the Temporal Service is encoded. The Temporal Service never has access to your encryption keys, so it cannot decode this data. Without a Codec Server, the Web UI and CLI display raw encoded payloads. A Codec Server solves this by giving the Web UI and CLI a way to decode payloads on demand, without exposing keys to the Temporal Service. Common reasons to run a Codec Server include: - **Debugging Workflows.** View decoded Workflow inputs, outputs, and Event History in the Web UI instead of reading base64-encoded or encrypted blobs. - **Operating from the CLI.** Use commands like `temporal workflow show` and `temporal workflow execute` with readable data, even when payloads are encrypted at rest. - **Encoding inputs from the UI and CLI.** When you start or signal a Workflow from the Web UI or CLI, the Codec Server can encode the input before it reaches the Temporal Service, so the Temporal Service never sees plaintext (the input still travels from your browser or CLI to the Codec Server, which is why HTTPS matters in any non-loopback deployment). - **Compliance and access control.** Because the Codec Server runs in your environment, you control who can decode payloads and under what conditions. You can layer authorization on top of the decode endpoint to restrict access per user or per Namespace. ## How a Codec Server works A Codec Server follows the Temporal [Codec Server Protocol](https://github.com/temporalio/samples-go/tree/main/codec-server#codec-server-protocol). It exposes two HTTP POST endpoints: - **`/encode`** accepts plaintext payloads and returns encoded payloads. Used for sending payloads. - **`/decode`** accepts encoded payloads and returns decoded payloads. Used for retrieving payloads. Both endpoints receive and respond with a JSON body containing a `payloads` array of [Payload](/dataconversion#payload) objects. The Codec Server passes each payload through your [Payload Codec](/payload-codec), which applies the same encoding or decoding logic that your Workers use. ![Codec Server](/diagrams/codec-server.svg) When the Web UI or CLI needs to display decoded data, it sends the encoded payloads to your Codec Server's `/decode` endpoint. The Codec Server decodes the payloads and returns them to the client. The Temporal Service never sees the decoded data. The `/encode` endpoint works in the other direction. When you start a Workflow or send a Signal from the Web UI or CLI, the input is sent to the Codec Server's `/encode` endpoint first, so data reaches the Temporal Service in its encoded form. Your Codec Server should use the same Payload Codec implementation as your Workers to ensure consistent encoding and decoding. ## Codec Server with External Storage When your Workers and Clients use [External Storage](/external-storage), your storage drivers replace some payloads in the Event History with small references that point to data in an external store like Amazon S3. The Temporal Service and the Web UI only see these references, not the actual payload data. This is further complicated by setups where you run Codecs in proxy that encode payloads after the Data Converter has returned on the Worker. Your Codec Server must be able to handle downloading and decoding in the correct order for you to be able to view the Workflow data in the UI or CLI. To support External Storage, create a handler using `NewPayloadHTTPHandler` with `PayloadHTTPHandlerOptions`. The options accept your storage drivers, your pre-storage codecs (the Payload Codecs configured in your Worker's Data Converter), and any post-storage codecs (codecs applied by a proxy after external storage). The handler applies them in the correct order across all endpoints automatically. When you configure the handler with storage drivers, the existing endpoints become storage-aware and a new `/download` endpoint becomes available: > **⚠️ Caution:** > > `NewPayloadHTTPHandler` runs the full encode-store-encode and decode-retrieve-decode pipeline. Do not use it as a target > for a remote Data Converter or remote codec on your Workers. For remote codecs, use `NewPayloadCodecHTTPHandler` > separately. If you need both, set up `NewPayloadHTTPHandler` for the Web UI and CLI alongside > `NewPayloadCodecHTTPHandler` for your Workers, and configure both with the same codecs. > - **`/download`** retrieves the actual payload data from external storage and decodes it through the Payload Codec. This endpoint is used internally by `/decode` when it encounters storage references, but you can also call it directly from the Web UI to retrieve the decoded payload. The Temporal Web UI uses this endpoint when you click to view the full payload for a storage reference. - **`/decode`** still decodes encoded payloads, but also handles storage references. By default, `/decode` uses the download logic internally to retrieve and decode any storage references in the request alongside regular payloads. With the `?preserveStorageRefs=true` query parameter, `/decode` skips retrieval and returns storage references as-is. - **`/encode`** applies the Payload Codec, then uploads payloads that exceed the size threshold to external storage and replaces them with reference tokens. ![Codec Server with External Storage](/diagrams/codec-server-with-external-storage.svg) The following example walks through how all three endpoints work together: 1. A user starts a Workflow from the CLI with a plaintext input. The CLI sends the input to the Codec Server's `/encode` endpoint. 2. The Codec Server encodes the payload through the Payload Codec. The encoded payload exceeds the storage threshold, so the Codec Server uploads it to external storage and returns a small reference token. 3. The CLI sends the reference token to the Temporal Service, which stores it in the Event History. 4. Later, a user views the Workflow in the Web UI. The Web UI retrieves the Event History from the Temporal Service and sends the payloads to the Codec Server's `/decode` endpoint with the `?preserveStorageRefs=true` query parameter. 5. The Codec Server decodes any non-reference payloads through the Payload Codec, but returns storage references as-is. The Web UI displays the reference metadata, indicating the payload is stored externally. 6. The user clicks to view the full payload. The Web UI sends the storage reference to the `/download` endpoint. 7. The Codec Server retrieves the encoded payload from external storage, decodes it through the Payload Codec, and returns the plaintext result to the Web UI. ## Codec Server vs. Payload Codec A Codec Server runs a [Payload Codec](/payload-codec) internally, so the two are directly connected. The difference is where the codec logic runs and who calls it. | | Payload Codec | Codec Server | | --------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | **Purpose** | Encodes and decodes Payloads. Applies encryption, compression, or other byte-level transformations. | Hosts a Payload Codec as an HTTP service so the Web UI and CLI can encode and decode Payloads remotely. | | **Runs where** | In-process, inside your Workers and Clients. Also runs inside the Codec Server. | As a standalone HTTP service in your environment, with a Payload Codec inside it. | | **Called by** | The Temporal SDK, automatically on every serialization and deserialization. | The Web UI and CLI, over HTTP, when a user views or submits Payload data. | | **Has access to encryption keys** | Yes. Keys are available in the Worker or Client process. | Yes. Must be configured with the same keys the Payload Codec uses. | You implement the transformation logic once in a Payload Codec, then host that logic in a Codec Server so the Web UI and CLI can use it remotely. ## Securing a Codec Server Because a Codec Server can decode sensitive data, treat it with the same trust as a Worker. Anyone who can call it has effective decrypt access. Use HTTPS for any deployment that is not strictly loopback (`localhost`). ### Network-level restrictions Restrict network access to the Codec Server. The Web UI can communicate with a Codec Server that is only accessible on `localhost`, so running the Codec Server locally is a viable security pattern. For team access, place the Codec Server behind a VPN. ### Authentication When the Codec Server is accessible beyond `localhost`, authenticate requests to verify the identity of the caller. The Web UI supports two approaches: **Include cross-origin credentials (recommended).** Enable **Include cross-origin credentials** in the Web UI Codec Server settings. The browser sends cookies scoped to the Codec Server's domain with each request. Your Codec Server must have its own authentication mechanism (its own login page and session cookies), so the user must have independently authenticated with the Codec Server. This is the recommended approach because the Codec Server maintains its own auth boundary, separate from the Temporal UI. **Pass access token.** Enable **Pass access token** in the Web UI Codec Server settings. The Web UI includes the same JSON Web Token (JWT) the user used to log into the Temporal UI in the `Authorization` header of each request. Your Codec Server validates the token signature against the OpenID Connect (OIDC) provider's JSON Web Key Set (JWKS) endpoint. On Temporal Cloud, verify against the [Temporal Cloud JWKS endpoint](https://login.tmprl.cloud/.well-known/jwks.json). On a self-hosted Temporal Service, the token comes from whatever auth provider you have [configured for the Web UI](/references/web-ui-configuration#auth). This approach requires less setup but reuses the same token across the Temporal UI and the Codec Server. > **📝 Note:** > > If you have a step in your process for token validation to ensure access isn't granted to the wrong token, you can validate the `audience` claim with: > > ```bash > "aud": [ > "https://saas-api.tmprl.cloud" > ] > ``` > ### Namespace-level authorization Authentication identifies the caller, but does not confirm the caller is authorized to decode payloads for a specific Namespace. Each request from the Web UI includes an `X-Namespace` header identifying the Namespace. To enforce Namespace-level access control, your Codec Server must enforce an additional check on whether the authenticated user has permissions for the requested Namespace. This applies regardless of which authentication approach you use. ### Key management You may also need [key management infrastructure](/key-management) to share encryption keys between your Workers and the Codec Server. ## SDK Codec Server samples Most Temporal SDKs provide example Codec Server implementations: - [Go](https://github.com/temporalio/samples-go/tree/main/codec-server) - [Java](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder) - [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py) | [Python with External Storage](https://github.com/temporalio/samples-python/tree/main/external_storage) - [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts) - [.NET](https://github.com/temporalio/samples-dotnet/blob/main/src/Encryption/CodecServer/Program.cs) - [Ruby](https://github.com/temporalio/samples-ruby/blob/main/encryption/codec_server.rb) --- # Temporal Cron Job Source: https://docs.temporal.io/cron-job > A Cron Schedule repeats a Workflow Execution like a unix cron job; see why Schedules are now the recommended alternative. This page discusses [Cron Job](#temporal-cron-job) including [Cron Schedules](#cron-schedules), [Time Zones](#cron-job-time-zones), and [how to stop a Cron Schedule](#stop-cron-schedules). ## What is a Temporal Cron Job? > **📝 Note:** > > We recommend using [Schedules](/schedule) instead of Cron Jobs. > Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules. > A Temporal Cron Job is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. - [How to set a Cron Schedule using the Go SDK](/develop/go/workflows/schedules#temporal-cron-jobs) - [How to set a Cron Schedule using the Java SDK](/develop/java/workflows/schedules#cron-schedule) - [How to set a Cron Schedule using the PHP SDK](/develop/php/workflows/schedules#temporal-cron-jobs) - [How to set a Cron Schedule using the Python SDK](/develop/python/workflows/schedules#temporal-cron-jobs) - [How to set a Cron Schedule using the TypeScript SDK](/develop/typescript/workflows/schedules#temporal-cron-jobs) ![Temporal Cron Job timeline](/diagrams/temporal-cron-job.svg) A Temporal Cron Job is similar to a classic unix cron job. Just as a unix cron job accepts a command and a schedule on which to execute that command, a Cron Schedule can be provided with the call to spawn a Workflow Execution. If a Cron Schedule is provided, the Temporal Server will spawn an execution for the associated Workflow Type per the schedule. Each Workflow Execution within the series is considered a Run. - Each Run receives the same input parameters as the initial Run. - Each Run inherits the same Workflow Options as the initial Run. The Temporal Server spawns the first Workflow Execution in the chain of Runs immediately. However, it calculates and applies a backoff (`firstWorkflowTaskBackoff`) so that the first Workflow Task of the Workflow Execution does not get placed into a Task Queue until the scheduled time. After each Run Completes, Fails, or reaches the [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout), the same thing happens: the next run will be created immediately with a new `firstWorkflowTaskBackoff` that is calculated based on the current Server time and the defined Cron Schedule. The Temporal Server spawns the next Run only after the current Run has Completed, Failed, or has reached the Workflow Run Timeout. This means that, if a Retry Policy has also been provided, and a Run Fails or reaches the Workflow Run Timeout, the Run will first be retried per the Retry Policy until the Run Completes or the Retry Policy has been exhausted. If the next Run, per the Cron Schedule, is due to spawn while the current Run is still Open (including retries), the Server automatically starts the new Run after the current Run completes successfully. The start time for this new Run and the Cron definitions are used to calculate the `firstWorkflowTaskBackoff` that is applied to the new Run. A [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) is used to limit how long a Workflow can be executing (have an Open status), including retries and any usage of Continue As New. The Cron Schedule runs until the Workflow Execution Timeout is reached or you terminate the Workflow. ![Temporal Cron Job Run Failure with a Retry Policy](/diagrams/temporal-cron-job-failure-with-retry.svg) ## Cron Schedules Cron Schedules are interpreted in UTC time by default. The Cron Schedule is provided as a string and must follow one of two specifications: **Classic specification** This is what the "classic" specification looks like: ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) │ │ │ │ │ │ │ │ │ │ * * * * * ``` For example, `15 8 * * *` causes a Workflow Execution to spawn daily at 8:15 AM UTC. Use the [crontab guru site](https://crontab.guru/) to test your cron expressions. ### `robfig` predefined schedules and intervals You can also pass any of the [predefined schedules](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-Predefined_schedules) or [intervals](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-Intervals) described in the [`robfig/cron` documentation](https://pkg.go.dev/github.com/robfig/cron/v3). ``` | Schedules | Description | Equivalent To | | ---------------------- | ------------------------------------------ | ------------- | | @yearly (or @annually) | Run once a year, midnight, Jan. 1st | 0 0 1 1 * | | @monthly | Run once a month, midnight, first of month | 0 0 1 * * | | @weekly | Run once a week, midnight between Sat/Sun | 0 0 * * 0 | | @daily (or @midnight) | Run once a day, midnight | 0 0 * * * | | @hourly | Run once an hour, beginning of hour | 0 * * * * | ``` For example, "@weekly" causes a Workflow Execution to spawn once a week at midnight between Saturday and Sunday. Intervals just take a string that can be accepted by [time.ParseDuration](http://golang.org/pkg/time/#ParseDuration). ``` @every ``` ## Time zones _This feature only applies in Temporal 1.15 and up_ You can change the time zone that a Cron Schedule is interpreted in by prefixing the specification with `CRON_TZ=America/New_York` (or your [desired time zone from tz](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)). `CRON_TZ=America/New_York 15 8 * * *` therefore spawns a Workflow Execution every day at 8:15 AM New York time, subject to caveats listed below. Consider that using time zones in production introduces a surprising amount of complexity and failure modes! **If at all possible, we recommend specifying Cron Schedules in UTC (the default)**. If you need to use time zones, here are a few edge cases to keep in mind: - **Beware Daylight Saving Time:** If a Temporal Cron Job is scheduled around the time when daylight saving time (DST) begins or ends (for example, `30 2 * * *`), **it might run zero, one, or two times in a day**! The Cron library that we use does not do any special handling of DST transitions. Avoid schedules that include times that fall within DST transition periods. - For example, in the US, DST begins at 2 AM. When you "fall back," the clock goes `1:59 … 1:00 … 1:01 … 1:59 … 2:00 … 2:01 AM` and any Cron jobs that fall in that 1 AM hour are fired again. The inverse happens when clocks "spring forward" for DST, and Cron jobs that fall in the 2 AM hour are skipped. - In other time zones like Chile and Iran, DST "spring forward" is at midnight. 11:59 PM is followed by 1 AM, which means `00:00:00` never happens. - **Self Hosting note:** If you manage your own Temporal Service, you are responsible for ensuring that it has access to current `tzdata` files. The official Docker images are built with [tzdata](https://docs.w3cub.com/go/time/tzdata/index) installed (provided by Alpine Linux), but ultimately you should be aware of how tzdata is deployed and updated in your infrastructure. - **Updating Temporal:** If you use the official Docker images, note that an upgrade of the Temporal Service may include an update to the tzdata files, which may change the meaning of your Cron Schedule. You should be aware of upcoming changes to the definitions of the time zones you use, particularly around daylight saving time start/end dates. - **Absolute Time Fixed at Start:** The absolute start time of the next Run is computed and stored in the database when the previous Run completes, and is not recomputed. This means that if you have a Cron Schedule that runs very infrequently, and the definition of the time zone changes between one Run and the next, the Run might happen at the wrong time. For example, `CRON_TZ=America/Los_Angeles 0 12 11 11 *` means "noon in Los Angeles on November 11" (normally not in DST). If at some point the government makes any changes (for example, move the end of DST one week later, or stay on permanent DST year-round), the meaning of that specification changes. In that first year, the Run happens at the wrong time, because it was computed using the older definition. ## How to stop a Temporal Cron Job A Temporal Cron Job does not stop spawning Runs until it has been Terminated or until the [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) is reached. A Cancellation Request affects only the current Run. Use the Workflow Id in any requests to Cancel or Terminate. --- # How does Temporal handle application data? Source: https://docs.temporal.io/dataconversion > This guide explores Data Converters in the Temporal Platform, detailing how they handle serialization and encoding for Workflow inputs and outputs, ensuring data stays secure and manageable. This guide provides an overview of data handling using a Data Converter on the Temporal Platform. Data Converters in Temporal are SDK components that handle the serialization and encoding of data entering and exiting a Temporal Service. Workflow inputs and outputs need to be serialized and deserialized so they can be sent as JSON to a Temporal Service. ![Data Converter encodes and decodes data](/diagrams/default-data-converter.svg) The Data Converter encodes data from your application to a [Payload](/dataconversion#payload) before it is sent to the Temporal Service in the Client call. When the Temporal Server sends the encoded data back to the Worker, the Data Converter decodes it for processing within your application. This ensures that all your sensitive data exists in its original format only on hosts that you control. Data Converter steps are followed when data is sent to a Temporal Service (as input to a Workflow) and when it is returned from a Workflow (as output). Due to how Temporal provides access to Workflow output, this implementation is asymmetric: - Data encoding is performed automatically using the default converter provided by Temporal or your custom Data Converter when passing input to a Temporal Service. For example, plain text input is usually serialized into a JSON object. - Data decoding may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Service. Instead, they are stored encoded on the Temporal Service, and you need to provide an additional parameter when using [`temporal workflow show`](/cli/command-reference/workflow#show) or when browsing the Web UI to view output. Each piece of data (like a single argument or return value) is encoded as a [Payload](/dataconversion#payload), which consists of binary data and key-value metadata. For details, see the API references: - [Go](https://pkg.go.dev/go.temporal.io/sdk/converter#DataConverter) - [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html) - [Python](https://python.temporal.io/temporalio.converter.DataConverter.html) - [TypeScript](https://typescript.temporal.io/api/interfaces/common.DataConverter) ### What is a Payload? A [Payload](https://api-docs.temporal.io/#temporal.api.common.v1.Payload) represents binary data such as input and output from Activities and Workflows. Payloads also contain metadata that describe their data type or other parameters for use by custom encoders/converters. When processed through the SDK, the [default Data Converter](/default-custom-data-converters#default-data-converter) serializes your data/value to a Payload before sending it to the Temporal Server. The default Data Converter processes supported type values to Payloads. You can create a custom [Payload Converter](/payload-converter) to apply different conversion steps. You can additionally apply [custom codecs](/payload-codec), such as for encryption or compression, on your Payloads. --- # Default and Custom Data Converters Source: https://docs.temporal.io/default-custom-data-converters > Learn about the default Data Converter in Temporal SDKs and how to implement a custom Data Converter for custom serialization and encoding needs. This page discusses the following: - [Default Data Converter](#default-data-converter) - [Custom Data Converter](#custom-data-converter) ## What is a default Data Converter? Each Temporal SDK includes and uses a default Data Converter. The default Data Converter converts objects to bytes using a series of Payload Converters and supports binary, Protobufs, and JSON formats. It encodes values in the following order: - Null - Byte array - Protobuf JSON - JSON In SDKs that cannot determine parameter types at runtime (for example, TypeScript), Protobufs aren't included in the default converter. For example: - If a value is an instance of a Protobuf message, it is encoded with [proto3 JSON](https://developers.google.com/protocol-buffers/docs/proto3#json). - If a value isn't null, binary, or a Protobuf, it is encoded as JSON. Most common input types — including strings, integers, floating point numbers, and booleans — are serializable as JSON. If any part of it is not serializable as JSON, an error is thrown. The default Data Converter serializes objects based on their root type, rather than nested types. The JSON serializers of some SDKs cannot process lists with Protobuf children objects without implementing a [custom Data Converter](#custom-data-converter). ## What is a custom Data Converter? A custom Data Converter extends the default Data Converter with custom logic for [Payload](/dataconversion#payload) conversion or encoding. You can create a custom Data Converter to alter formats (for example, using [MessagePack](https://msgpack.org/) instead of JSON) or add compression and encryption. A Payload Codec encodes and decodes [Payloads](/dataconversion#payload), with bytes-to-bytes conversion. To use custom encryption or compression logic, create a custom Payload Codec with your encryption/compression logic in the `encode` function and your decryption/decompression logic in the `decode` function. To implement a custom Payload Codec, you can override the default Data Converter, or create a customized Data Converter that defines its own Payload Converter. Custom Data Converters are not applied to all data; for example, [Search Attributes](/search-attribute) are persisted unencoded so they can be indexed for searching. A customized Data Converter can have the following three components: - [Payload Converter](/payload-converter) - [Payload Codec](/payload-codec) - [Failure Converter](/failure-converter) For details on how to implement custom encryption and compression in your SDK, see [Data Encryption](/production-deployment/data-encryption). --- # Visualize an Activity Retry Policy with timeouts Source: https://docs.temporal.io/demos/activity-retry-simulator > Visualize Activity Execution times, experiment with timeouts, Retry Policies, and simulate scenarios with our tool. Configure retries and see their impact on success or failure. Use this tool to visualize total Activity Execution times and experiment with different Activity timeouts and Retry Policies. For a list of Activity Task Execution times, use [this calculator](https://temporal-time.netlify.app/?initialInterval=1&maxInterval=100&maxReties=10&backoffCoefficient=2). The simulator is based on a common Activity use-case, which is to call a third party HTTP API and return the results. See the example code snippets below. Use the Activity Retries settings to configure how long the API request takes to succeed or fail. There is an option to generate scenarios. The _Task Time in Queue_ simulates the time the Activity Task might be waiting in the Task Queue. Use the Activity Timeouts and Retry Policy settings to see how they impact the success or failure of an Activity Execution. --- # Task Queue Priority and Fairness - Interactive Walkthrough Source: https://docs.temporal.io/demos/priority-fairness-walkthrough > Interactively explore how Task Queue Priority and Fairness control dispatch order. Step through scenarios, see which task gets picked next and why, then grab the SDK code. --- # Standalone Activities Demo Source: https://docs.temporal.io/demos/standalone-activities > An interactive overview of Temporal Standalone Activities. > **Public Preview** — Go, Python, Java, .NET, TypeScript, Ruby > Available in [Temporal Cloud](/standalone-activity#temporal-cloud-support) and in the [Temporal CLI](/standalone-activity#temporal-cli-support) v1.7.0 or higher with Temporal Server v1.31.0 or higher. Java SDK support is in [Pre-release](/evaluate/development-production-features/release-stages#pre-release). Standalone Activities let you run a single Activity straight from your application without writing a Workflow. Your code uses the Temporal Client to send the request to the Server, the Server durably enqueues the request for a Worker to pick up, and the result comes back through a handle that your code can wait on or check later. Try the demo below to walk through the full flow, tweak the retry and timeout settings, and watch the SDK code and CLI command update as you go. --- ## How it works When you call `client.execute_activity()` (or the equivalent in your applicable SDK) from your application, the following happens: 1. **Connect**: Your application opens a connection to the Temporal Server using a Temporal Client configured with your namespace and credentials. 2. **Schedule**: The Server durably persists the Activity Task on the specified Task Queue so that the request survives Worker restarts and network interruptions. 3. **Poll**: A Worker that is polling that Task Queue picks up the Activity Task and prepares to execute it. 4. **Execute**: The Worker runs your Activity function with the provided arguments and reports the outcome back to the Server. 5. **Return**: The Server stores the result and returns it to the original caller, either directly or via a handle, depending on which SDK method you use. ### Standalone vs Workflow Activities | | Workflow Activity | Standalone Activity | |---|---|---| | Orchestrated by | A Workflow Definition | Your application code (via the Temporal Client) | | Started with | `workflow.execute_activity()` (or the equivalent in your applicable SDK) from inside a Workflow Definition | `client.execute_activity()` (or the equivalent in your applicable SDK) from your application code | | Retry policy | Set when calling the Activity from inside a Workflow | Set when calling the Activity from your application | | Visibility | Shown in the Workflow's Event History | Shown in the Standalone Activity list and count views | | Use case | Multi-step orchestration with multiple Activities | Single, independent jobs like sending an email or processing a webhook | The Activity function and Worker registration are **identical** for both approaches, and only the execution path that triggers the Activity differs between them. If the Activity fails, the Server automatically retries it according to the Retry Policy you configure. --- ## Next steps For complete API reference and advanced usage, see the SDK-specific guides: - [Standalone Activities - Go](/develop/go/activities/standalone-activities) - [Standalone Activities - Java](/develop/java/activities/standalone-activities) - [Standalone Activities - Python](/develop/python/activities/standalone-activities) - [Standalone Activities - Ruby](/develop/ruby/activities/standalone-activities) - [Standalone Activities - TypeScript](/develop/typescript/activities/standalone-activities) - [Standalone Activities - .NET](/develop/dotnet/activities/standalone-activities) --- # Temporal Design Patterns Source: https://docs.temporal.io/design-patterns > A catalog of common, reusable, and proven design patterns for Temporal Workflows, organized by problem domain. Temporal provides a set of durable execution primitives that you can compose into common, reusable, and proven patterns. Having these patterns in your toolbox helps you solve recurring problems in a battle-tested way. ## Task orchestration patterns - [Child Workflows](/design-patterns/child-workflows): Decomposes complex Workflows into smaller, reusable units. Each child has an independent Workflow ID, history, and lifecycle. - [Parallel Execution](/design-patterns/parallel-execution): Executes multiple Activities concurrently for maximum throughput with error handling and controlled parallelism. - [Pick First (Race)](/design-patterns/pick-first): Starts multiple Activities in parallel and uses the first result, cancelling the rest. ## Workflow messaging patterns - [Signal with Start](/design-patterns/signal-with-start): Starts a Workflow when Signaling it if it does not already exist. If already running, it receives the Signal directly. - [Request-Response via Updates](/design-patterns/request-response-via-updates): Synchronous request-response with validation. Updates modify state and return results directly. ## Entity & lifecycle patterns - [Entity Workflow](/design-patterns/entity-workflow): Models long-lived business entities as individual Workflows that persist for the entity's entire lifetime, handling all state transitions through Signals and Updates. - [Continue-As-New](/design-patterns/continue-as-new): Prevents unbounded history growth by completing the current execution and starting a new one with fresh history. - [Updatable Timer](/design-patterns/updatable-timer): Dynamically adjustable timers that respond to Signals or Updates. Extend, shorten, or cancel timers based on external events. ## External interaction patterns - [Polling External Services](/design-patterns/polling): Strategies for polling external resources with varying frequencies: frequent, infrequent, and periodic patterns. - [Long-Running Activity](/design-patterns/long-running-activity): Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support. - [Delayed Start](/design-patterns/delayed-start): Creates Workflows immediately but defers execution until a specified delay expires. Fits one-time scheduled operations and grace periods. - [Delayed Callback (Webhooks)](/design-patterns/delayed-callback): Integrates webhooks durably: receive inbound webhooks via Signals, fire delayed outbound callbacks with durable timers, and complete Activities asynchronously via task tokens. - [Approval](/design-patterns/approval): Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata. ## Distributed transaction patterns - [Saga Pattern](/design-patterns/saga-pattern): Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail. - [Early Return](/design-patterns/early-return): Synchronous initialization with asynchronous completion. Returns results immediately while processing continues in the background. ## Error handling & retry patterns - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource. - [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bound the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many individual attempts occur. - [Non-Retryable Errors](/design-patterns/non-retryable-errors): Mark error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying indefinitely. - [Delayed Retry](/design-patterns/delayed-retry): Override the next retry interval for a specific failure using nextRetryDelay on ApplicationFailure. Use when an error carries information about how long to wait before retrying. - [Fast/Slow Retries](/design-patterns/fast-slow-retries): Try aggressively with a short interval first, then shift to a long interval when fast retries are exhausted, keeping the Workflow alive until the downstream system recovers. - [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emit a custom metric from inside the Activity when the attempt count crosses a threshold, surfacing silent persistent failures to on-call teams before an SLA breach. - [Resumable Activity](/design-patterns/resumable-activity): Park the Workflow after retries are exhausted and wait for a human to signal a correction, then resume execution from where it left off. ## Batch processing patterns - [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows): Distributes a large record set across parallel Child Workflows for concurrent processing with automatic scaling. - [Batch Iterator](/design-patterns/batch-iterator): Pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees. - [Sliding Window](/design-patterns/sliding-window): Maintains a fixed number of concurrently active Child Workflows, starting a new one each time an existing one completes. - [MapReduce Tree](/design-patterns/mapreduce-tree): Recursively splits a dataset into a binary tree of Child Workflows, processes leaves in parallel, then aggregates results back up the tree. ## QoS & throughput patterns - [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting): Caps Activity execution rate against a downstream service by routing throttled Activities to a dedicated Task Queue backed by Workers configured with a throughput limit. - [Priority Task Queues](/design-patterns/priority-task-queues): Assigns a priority level to Workflows and Activities so that time-sensitive work executes ahead of lower-priority work within a single Task Queue. - [Fairness](/design-patterns/fairness): Distributes Worker capacity evenly across tenants or users so that a burst from one caller does not starve the others. ## Performance & latency patterns - [Local Activities](/design-patterns/local-activities): Run Activity functions in-process inside the Workflow Task, eliminating all server scheduling round-trips. Best for short, idempotent Activities on a latency-sensitive path. - [Early Return + Local Activities](/design-patterns/early-return-local-activities): Extends Early Return by running Phase 1 Activities as Local Activities. The client receives its response after Phase 1 completes entirely in-process, achieving the lowest possible first-response latency. - [Eager Workflow Start](/design-patterns/eager-workflow-start): Dispatch the first Workflow Task directly to a co-located Worker, bypassing the Temporal Matching Service. Requires the starter and Worker to share the same process and client connection. ## Worker configuration patterns - [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue): Routes Activities to specific Workers using unique Task Queues for Worker affinity and host-specific processing. - [Activity Dependency Injection](/design-patterns/activity-dependency-injection): Injects external dependencies into Activities at Worker startup, keeping Workflow code deterministic and Activities testable. --- # Activity Dependency Injection Source: https://docs.temporal.io/design-patterns/activity-dependency-injection > Injects external dependencies into Activities at Worker startup, keeping Workflow code deterministic and Activities testable. ## Overview The Activity Dependency Injection pattern separates the creation of external dependencies (database connections, API clients, configuration) from Activity business logic by injecting them at Worker startup. This approach keeps Workflow code deterministic, makes Activities testable in isolation, and ensures expensive resources are initialized once per Worker process rather than once per Activity execution. ## Problem Activities often need access to external resources such as database connection pools, HTTP clients, third-party API credentials, or shared caches. Without a structured approach, you face several challenges: - **Reinitializing resources per execution.** Creating a new database connection or API client on every Activity invocation wastes resources and increases latency. - **Hardcoded dependencies.** Embedding connection logic directly inside Activity functions couples business logic to infrastructure, making it difficult to swap implementations across environments. - **Difficult testing.** When Activities construct their own dependencies internally, you cannot substitute test doubles without modifying production code. - **Non-determinism risk.** Passing dependencies directly into Workflow code breaks Temporal's determinism guarantees, because dependency state can change between replays. ## Solution You define Activities as methods on a struct or class that holds dependencies as fields. At Worker startup, you instantiate the struct or class with real implementations and register it with the Worker. The Workflow references Activity methods without knowing about the underlying dependencies. ```mermaid flowchart LR subgraph Worker Startup D[Dependencies
DB, API Client, Config] --> S[Activity Struct / Class] S --> R[Register with Worker] end subgraph Workflow Execution W[Workflow] -->|"execute activity"| A[Activity Method] A -->|"uses"| S end subgraph Testing M[Mock / Stub] --> T[Test Environment] T -->|"execute activity"| A2[Activity Method] end ``` The following describes each path in the diagram: 1. At Worker startup, you create dependency instances (database pools, API clients) and inject them into an Activity struct or class, which you then register with the Worker. 2. During Workflow execution, the Workflow calls Activity methods by reference. The Temporal runtime routes the call to the registered instance on the Worker, where the method accesses the injected dependencies. 3. During testing, you substitute mock or stub implementations into the same Activity struct or class, allowing you to verify behavior without external services. ## Implementation ### Define Activities with dependencies Define Activities as methods on a struct or class that accepts dependencies through its constructor or fields. Each method acts as a separate Activity Type. **Go** ```go // activities.go package payment import ( "context" "go.temporal.io/sdk/activity" ) type Activities struct { DBClient DBClient EmailClient EmailClient } func (a *Activities) ChargeCustomer(ctx context.Context, orderID string, amount int) (string, error) { logger := activity.GetLogger(ctx) logger.Info("Charging customer", "orderID", orderID, "amount", amount) receiptID, err := a.DBClient.ProcessPayment(orderID, amount) if err != nil { return "", err } return receiptID, nil } func (a *Activities) SendReceipt(ctx context.Context, email string, receiptID string) error { return a.EmailClient.Send(email, "Payment Receipt", receiptID) } ``` **Python** ```python # activities.py from dataclasses import dataclass from temporalio import activity @dataclass class PaymentActivities: db_client: DBClient email_client: EmailClient @activity.defn async def charge_customer(self, order_id: str, amount: int) -> str: activity.logger.info( "Charging customer", extra={"order_id": order_id, "amount": amount} ) receipt_id = await self.db_client.process_payment(order_id, amount) return receipt_id @activity.defn async def send_receipt(self, email: str, receipt_id: str) -> None: await self.email_client.send(email, "Payment Receipt", receipt_id) ``` **Java** ```java // PaymentActivities.java @ActivityInterface public interface PaymentActivities { String chargeCustomer(String orderId, int amount); void sendReceipt(String email, String receiptId); } // PaymentActivitiesImpl.java public class PaymentActivitiesImpl implements PaymentActivities { private final DBClient dbClient; private final EmailClient emailClient; public PaymentActivitiesImpl(DBClient dbClient, EmailClient emailClient) { this.dbClient = dbClient; this.emailClient = emailClient; } @Override public String chargeCustomer(String orderId, int amount) { return dbClient.processPayment(orderId, amount); } @Override public void sendReceipt(String email, String receiptId) { emailClient.send(email, "Payment Receipt", receiptId); } } ``` **TypeScript** ```typescript // activities.ts export interface DB { processPayment(orderId: string, amount: number): Promise; } export interface EmailClient { send(to: string, subject: string, body: string): Promise; } export const createActivities = (db: DB, emailClient: EmailClient) => ({ async chargeCustomer(orderId: string, amount: number): Promise { const receiptId = await db.processPayment(orderId, amount); return receiptId; }, async sendReceipt(email: string, receiptId: string): Promise { await emailClient.send(email, 'Payment Receipt', receiptId); }, }); ``` Each SDK uses a different mechanism to group Activities with their dependencies: - **Go**: Methods on a struct. The struct fields hold dependencies. - **Python**: A `@dataclass` with `@activity.defn` methods. Fields hold dependencies. - **Java**: An `@ActivityInterface` with a separate implementation class. Dependencies are passed through the constructor. - **TypeScript**: A factory function that closes over dependencies and returns an object of Activity functions. ### Reference Activities from the Workflow The Workflow references Activity methods without any knowledge of the injected dependencies. Each SDK provides a type-safe way to call Activities. **Go** ```go // workflow.go package payment import ( "time" "go.temporal.io/sdk/workflow" ) func PaymentWorkflow(ctx workflow.Context, orderID string, amount int, email string) error { ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) // Use a nil struct pointer to reference Activity methods. // This provides compile-time type safety without instantiating the struct. var a *Activities var receiptID string err := workflow.ExecuteActivity(ctx, a.ChargeCustomer, orderID, amount).Get(ctx, &receiptID) if err != nil { return err } return workflow.ExecuteActivity(ctx, a.SendReceipt, email, receiptID).Get(ctx, nil) } ``` **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import PaymentActivities @workflow.defn class PaymentWorkflow: @workflow.run async def run(self, order_id: str, amount: int, email: str) -> None: receipt_id = await workflow.execute_activity_method( PaymentActivities.charge_customer, args=[order_id, amount], start_to_close_timeout=timedelta(seconds=30), ) await workflow.execute_activity_method( PaymentActivities.send_receipt, args=[email, receipt_id], start_to_close_timeout=timedelta(seconds=30), ) ``` **Java** ```java // PaymentWorkflowImpl.java public class PaymentWorkflowImpl implements PaymentWorkflow { private final PaymentActivities activities = Workflow.newActivityStub( PaymentActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build() ); @Override public void processPayment(String orderId, int amount, String email) { String receiptId = activities.chargeCustomer(orderId, amount); activities.sendReceipt(email, receiptId); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type { createActivities } from './activities'; // Use ReturnType to extract the Activity types from the factory function const { chargeCustomer, sendReceipt } = proxyActivities< ReturnType >({ startToCloseTimeout: '30s', }); export async function paymentWorkflow( orderId: string, amount: number, email: string ): Promise { const receiptId = await chargeCustomer(orderId, amount); await sendReceipt(email, receiptId); } ``` Key points for each SDK: - **Go**: A nil pointer of the Activity struct type (`var a *Activities`) provides compile-time method references without instantiating the struct. The Temporal runtime resolves the actual registered instance at execution time. - **Python**: `workflow.execute_activity_method` references the class method directly and resolves to the registered instance on the Worker. - **Java**: `Workflow.newActivityStub` creates a typed proxy from the Activity interface. The Temporal runtime routes calls to the registered implementation. - **TypeScript**: `proxyActivities>` infers the Activity types from the factory function's return type. Activities are always referenced by name at runtime. ### Register Activities with the Worker At Worker startup, you instantiate the Activity struct or class with real dependency implementations and register it. **Go** ```go // worker/main.go package main import ( "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" "example/payment" ) func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "payment", worker.Options{}) w.RegisterWorkflow(payment.PaymentWorkflow) // Inject real dependencies at Worker startup w.RegisterActivity(&payment.Activities{ DBClient: payment.NewPostgresClient("postgres://localhost:5432/payments"), EmailClient: payment.NewSMTPClient("smtp://mail.example.com"), }) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` **Python** ```python # worker.py import asyncio from temporalio.client import Client from temporalio.worker import Worker from activities import PaymentActivities from workflows import PaymentWorkflow async def main(): client = await Client.connect("localhost:7233") # Inject real dependencies at Worker startup payment_activities = PaymentActivities( db_client=PostgresClient("postgres://localhost:5432/payments"), email_client=SMTPClient("smtp://mail.example.com"), ) worker = Worker( client, task_queue="payment", workflows=[PaymentWorkflow], activities=[ payment_activities.charge_customer, payment_activities.send_receipt, ], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` **Java** ```java // PaymentWorker.java public class PaymentWorker { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("payment"); worker.registerWorkflowImplementationTypes(PaymentWorkflowImpl.class); // Inject real dependencies at Worker startup worker.registerActivitiesImplementations( new PaymentActivitiesImpl( new PostgresClient("postgres://localhost:5432/payments"), new SMTPClient("smtp://mail.example.com") ) ); factory.start(); } } ``` **TypeScript** ```typescript // worker.ts import { Worker } from '@temporalio/worker'; import { createActivities } from './activities'; async function run() { // Initialize dependencies at Worker startup const db = new PostgresClient('postgres://localhost:5432/payments'); const emailClient = new SMTPClient('smtp://mail.example.com'); const worker = await Worker.create({ taskQueue: 'payment', workflowsPath: require.resolve('./workflows'), // Inject dependencies through the factory function activities: createActivities(db, emailClient), }); await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); ``` Dependencies are initialized once when the Worker process starts. All Activity executions on that Worker share the same instances, which is appropriate for thread-safe resources like connection pools and HTTP clients. ### Inject a circuit breaker A circuit breaker is a natural fit for Worker-level injection because it is a *stateful* dependency. It tracks recent failures for a downstream service and, once the failure rate crosses a threshold, trips to an open state that rejects calls immediately for a cool-down period before allowing a probe. That state only protects the service if it is shared across every Activity execution on the Worker. Constructing a new breaker inside each Activity method would reset the counters on every call, so the breaker would never trip. By injecting a single breaker instance alongside the client it guards, all executions of the Activity feed the same failure window. When the downstream service degrades, the breaker opens and Activities fail fast instead of piling up latency against an unhealthy dependency, which in turn lets Temporal's retry policy back off rather than hammering the service. The following examples wrap an outbound payment API call in a circuit breaker, using an established library for each language: **Go** ```go // activities.go — github.com/sony/gobreaker/v2 package payment import ( "context" "github.com/sony/gobreaker/v2" "go.temporal.io/sdk/activity" ) type Activities struct { PaymentAPI PaymentAPI Breaker *gobreaker.CircuitBreaker[string] } func (a *Activities) ChargeCustomer(ctx context.Context, orderID string, amount int) (string, error) { logger := activity.GetLogger(ctx) logger.Info("Charging customer", "orderID", orderID, "amount", amount) // The breaker rejects the call immediately when it is open, // returning gobreaker.ErrOpenState without touching the API. return a.Breaker.Execute(func() (string, error) { return a.PaymentAPI.Charge(orderID, amount) }) } ``` **Python** ```python # activities.py — pybreaker import pybreaker from dataclasses import dataclass from temporalio import activity @dataclass class PaymentActivities: payment_api: PaymentAPI breaker: pybreaker.CircuitBreaker @activity.defn async def charge_customer(self, order_id: str, amount: int) -> str: activity.logger.info( "Charging customer", extra={"order_id": order_id, "amount": amount} ) # Raises pybreaker.CircuitBreakerError when the breaker is open. return await self.breaker.call_async( self.payment_api.charge, order_id, amount ) ``` **Java** ```java // PaymentActivitiesImpl.java — resilience4j import io.github.resilience4j.circuitbreaker.CircuitBreaker; public class PaymentActivitiesImpl implements PaymentActivities { private final PaymentAPI paymentApi; private final CircuitBreaker breaker; public PaymentActivitiesImpl(PaymentAPI paymentApi, CircuitBreaker breaker) { this.paymentApi = paymentApi; this.breaker = breaker; } @Override public String chargeCustomer(String orderId, int amount) { // Throws CallNotPermittedException when the breaker is open. return breaker.executeSupplier(() -> paymentApi.charge(orderId, amount)); } } ``` **TypeScript** ```typescript // activities.ts — opossum import CircuitBreaker from 'opossum'; export const createActivities = (breaker: CircuitBreaker) => ({ async chargeCustomer(orderId: string, amount: number): Promise { // Rejects with an EOPENBREAKER error when the breaker is open. return breaker.fire(orderId, amount) as Promise; }, }); ``` You construct the breaker at Worker startup and inject it the same way as any other dependency, so its state lives for the lifetime of the Worker process. **Go** ```go // worker/main.go package main import ( "log" "time" "github.com/sony/gobreaker/v2" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" "example/payment" ) func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "payment", worker.Options{}) w.RegisterWorkflow(payment.PaymentWorkflow) // Construct the breaker once at Worker startup so its failure // counters are shared across all Activity executions. breaker := gobreaker.NewCircuitBreaker[string](gobreaker.Settings{ Name: "payment-api", MaxRequests: 3, // probes allowed while half-open Interval: 60 * time.Second, // window for counting failures Timeout: 30 * time.Second, // cool-down before half-open }) w.RegisterActivity(&payment.Activities{ PaymentAPI: payment.NewPaymentAPI("https://api.example.com"), Breaker: breaker, }) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` **Python** ```python # worker.py import asyncio import pybreaker from temporalio.client import Client from temporalio.worker import Worker from activities import PaymentActivities, PaymentAPI from workflows import PaymentWorkflow async def main(): client = await Client.connect("localhost:7233") # Construct the breaker once at Worker startup so its failure # counters are shared across all Activity executions. breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30) payment_activities = PaymentActivities( payment_api=PaymentAPI("https://api.example.com"), breaker=breaker, ) worker = Worker( client, task_queue="payment", workflows=[PaymentWorkflow], activities=[ payment_activities.charge_customer, payment_activities.send_receipt, ], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` **Java** ```java // PaymentWorker.java import io.github.resilience4j.circuitbreaker.CircuitBreaker; import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig; import io.temporal.client.WorkflowClient; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; import java.time.Duration; public class PaymentWorker { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("payment"); worker.registerWorkflowImplementationTypes(PaymentWorkflowImpl.class); // Construct the breaker once at Worker startup so its failure // counters are shared across all Activity executions. CircuitBreakerConfig config = CircuitBreakerConfig.custom() .failureRateThreshold(50) // percent .waitDurationInOpenState(Duration.ofSeconds(30)) // cool-down .slidingWindowSize(20) .build(); CircuitBreaker breaker = CircuitBreaker.of("payment-api", config); worker.registerActivitiesImplementations( new PaymentActivitiesImpl(new PaymentAPI("https://api.example.com"), breaker) ); factory.start(); } } ``` **TypeScript** ```typescript // worker.ts import { Worker } from '@temporalio/worker'; import CircuitBreaker from 'opossum'; import { createActivities } from './activities'; import { PaymentAPI } from './payment-api'; async function run() { // Construct the breaker once at Worker startup so its failure // counters are shared across all Activity executions. const paymentApi = new PaymentAPI('https://api.example.com'); const breaker = new CircuitBreaker( (orderId: string, amount: number) => paymentApi.charge(orderId, amount), { errorThresholdPercentage: 50, resetTimeout: 30000 }, // cool-down in ms ); const worker = await Worker.create({ taskQueue: 'payment', workflowsPath: require.resolve('./workflows'), activities: createActivities(breaker), }); await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); ``` Circuit breaker libraries by language: - **Go**: [`sony/gobreaker`](https://github.com/sony/gobreaker) — the generic (`v2`) API returns the wrapped result type directly. - **Python**: [`pybreaker`](https://github.com/danielfm/pybreaker) — `call_async` wraps coroutine-based clients. - **Java**: [`resilience4j`](https://resilience4j.readme.io/docs/circuitbreaker) — the successor to Netflix Hystrix, built around functional decorators. - **TypeScript**: [`opossum`](https://nodeshift.dev/opossum/) — wraps a single async action bound at construction time. Because the breaker counts failures, size the Activity retry policy accordingly. A breaker that opens after a burst of failures pairs well with a retry policy that backs off, so an open breaker returns fast errors that Temporal retries later rather than each attempt waiting on a timeout. ## When to use This pattern is a good fit when your Activities access external services such as databases, message queues, or third-party APIs. It is appropriate when you want to initialize expensive resources once per Worker process, when you need to test Activity logic without connecting to real services, or when you operate in multiple environments (development, staging, production) that require different dependency configurations. This pattern is not necessary for Activities that are pure functions with no external dependencies, or for Activities that only use Temporal-provided context like heartbeating and logging. ## Benefits and trade-offs Injecting dependencies at the Worker level provides several advantages. Resources like connection pools are initialized once and shared across all Activity executions, reducing overhead. Substituting mock implementations in tests requires no changes to Activity or Workflow code. Switching between environments involves changing only the Worker configuration. The trade-off is that all Activity executions on a given Worker share the same dependency instances. If an Activity requires per-execution isolation (for example, a database transaction scoped to a single Activity), you need to manage that within the Activity method itself. Dependencies must also be thread-safe, since multiple Activity executions may run concurrently on the same Worker. ## Best practices - **Keep dependencies thread-safe.** Multiple Activity executions run concurrently on the same Worker. Use connection pools rather than single connections, and avoid mutable shared state. - **Define dependencies as interfaces.** In Go, Python, and Java, using interfaces (or protocols in Python) for dependencies makes it possible to swap implementations for testing or different environments. - **Do not inject dependencies into Workflows.** Workflow code must remain deterministic. If a Workflow needs configuration, retrieve it through a Local Activity so the value gets recorded in the Event History. - **Initialize dependencies before Worker startup.** Create and validate all connections before calling `worker.Run()` or its equivalent. This ensures that the Worker does not start accepting tasks until all dependencies are ready. - **Group related Activities on a single struct or class.** Activities that share the same dependencies belong together. If two groups of Activities have different dependencies, use separate structs or classes for each group. ## Common pitfalls - **Constructing dependencies inside Activity methods.** Creating a new database connection or API client per execution leads to resource exhaustion and increased latency. - **Injecting dependencies into Workflows.** This breaks determinism because dependency state can change between the original execution and a replay. The Temporal Java SDK documentation explicitly warns against this. - **Using non-thread-safe dependencies.** A single mutable object shared across concurrent Activity executions causes race conditions. Use connection pools and ensure all injected objects are safe for concurrent use. - **Registering class methods as static in Python.** If you register `BotService.send_message` (the unbound method) instead of `bot_service.send_message` (a method on an instance), the `self` parameter is not bound, causing a missing argument error at runtime. - **Forgetting to bind methods in TypeScript.** When using a class instead of a factory function, class methods must be defined as arrow functions or explicitly bound in the constructor. Otherwise, `this` is `undefined` when Temporal invokes the Activity. ## Related ### Patterns - **[Entity Workflow](/design-patterns/entity-workflow)**: Long-lived Workflows that manage stateful entities, often using Activities with injected dependencies. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Routing Activities to specific Workers, which can have different injected dependencies. ### Sample code ### Go - [Greetings](https://github.com/temporalio/samples-go/tree/main/greetings) — Activities as struct methods with injected dependencies. - [Large Payload Fixture](https://github.com/temporalio/samples-go/tree/main/temporal-fixtures/largepayload) — The reference sample using struct-based Activity dependency injection. ### Python - [Hello Activity Method](https://github.com/temporalio/samples-python/blob/main/hello/hello_activity_method.py) — Activities defined as class methods with dependency injection. ### Java - [Hello World](https://github.com/temporalio/hello-world-project-template-java) — Activity interface and implementation with Worker registration. ### TypeScript - [Activities Dependency Injection](https://github.com/temporalio/samples-typescript/tree/main/activities-dependency-injection) — Factory function pattern for sharing dependencies between Activities. --- # Approval Pattern Source: https://docs.temporal.io/design-patterns/approval > Human-in-the-loop Workflows that block until external approval decisions are made. Uses Signals to capture approval data with metadata. ## Overview The Approval pattern implements human-in-the-loop Workflows where execution blocks until an external decision is made. It uses Workflow Signals with custom input data to unblock Workflows, enabling approval processes, manual reviews, and decision gates in automated business processes. ## Problem In many business processes, you need Workflows that wait for human approval before proceeding. These Workflows must capture approval decisions along with metadata such as the approver's identity, a reason, and a timestamp. They must also support multiple outcomes — approval, rejection, or escalation — and handle timeout scenarios when no decision arrives. Without a structured approval pattern, you are forced to poll external systems for approval status, implement complex state machines by hand, and manage race conditions between timeouts and incoming approvals. You also risk losing approval context and metadata, and you must build custom audit logging to meet compliance requirements. ## Solution The Approval pattern uses a blocking wait with timeout to pause execution until a Signal is received. The Signal carries custom data — the approval decision, approver details, and comments — that the Workflow captures and uses to determine next steps. ```mermaid sequenceDiagram participant Requester participant Workflow participant Approver Requester->>+Workflow: Start approval request activate Workflow Workflow->>Workflow: Wait with timeout Note over Workflow: Waiting for approval... alt Approval received Approver->>Workflow: Signal: submitApproval(data) Workflow->>Workflow: Process approval data Workflow-->>Requester: Approved else Timeout Note over Workflow: Timeout expires Workflow-->>Requester: Timeout/Rejected end deactivate Workflow ``` The following describes each step in the diagram: 1. The requester starts the Workflow with an approval request. 2. The Workflow blocks execution with a timeout — using `Workflow.await()` in Java, `condition()` in TypeScript, `workflow.wait_condition()` in Python, or `workflow.AwaitWithTimeout()` in Go. 3. If an approver sends a Signal before the timeout expires, the Workflow receives the approval data, processes the decision, and returns the result to the requester. 4. If the timeout expires before any Signal arrives, the Workflow unblocks and follows the timeout path, which typically results in rejection or escalation. ## Implementation ### Basic approval with timeout The basic implementation captures the approval decision in a structured data object rather than a plain boolean. Define a type to hold the approver's identity, the decision, any comments, and a timestamp: **Python** ```python # models.py from dataclasses import dataclass @dataclass class ApprovalData: approver: str decision: str # "APPROVED", "REJECTED", "ESCALATED" comments: str timestamp: int ``` **Go** ```go // types.go type ApprovalData struct { Approver string Decision string // "APPROVED", "REJECTED", "ESCALATED" Comments string Timestamp int64 } ``` **Java** ```java // ApprovalData.java public class ApprovalData { private String approver; private String decision; // "APPROVED", "REJECTED", "ESCALATED" private String comments; private long timestamp; // Constructor, getters, setters } ``` **TypeScript** ```typescript // types.ts export interface ApprovalData { approver: string; decision: 'APPROVED' | 'REJECTED' | 'ESCALATED'; comments: string; timestamp: number; } ``` This type gives you a structured way to pass rich context through the Signal rather than a plain boolean. The implementation ties everything together. The Workflow blocks until either the approval data arrives via Signal or the timeout expires: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow from models import ApprovalData @workflow.defn class ApprovalWorkflow: def __init__(self) -> None: self.approval_data: ApprovalData | None = None self.status = "PENDING" @workflow.run async def run(self, request_id: str, timeout_seconds: int) -> str: try: # Block until the Signal sets approval_data, or raise on timeout await workflow.wait_condition( lambda: self.approval_data is not None, timeout=timedelta(seconds=timeout_seconds), ) self.status = self.approval_data.decision return f"Request {request_id} {self.status} by {self.approval_data.approver}" except asyncio.TimeoutError: self.status = "TIMEOUT" return f"Request {request_id} timed out" # Signal handler: an external approver submits the decision @workflow.signal def submit_approval(self, data: ApprovalData) -> None: self.approval_data = data # Query handler: read current status without mutating state @workflow.query def get_status(self) -> str: return self.status ``` **Go** ```go // workflow.go func ApprovalWorkflow(ctx workflow.Context, requestId string, timeout time.Duration) (string, error) { var approvalData *ApprovalData status := "PENDING" // Query handler: expose current status without mutating state err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) { return status, nil }) if err != nil { return "", err } // Receive the approval signal in a goroutine workflow.Go(ctx, func(ctx workflow.Context) { signalChan := workflow.GetSignalChannel(ctx, "submitApproval") signalChan.Receive(ctx, &approvalData) }) // Block until the signal sets approvalData, or time out (ok=false) approved, err := workflow.AwaitWithTimeout(ctx, timeout, func() bool { return approvalData != nil }) if err != nil { return "", err } if approved { status = approvalData.Decision return fmt.Sprintf("Request %s %s by %s", requestId, status, approvalData.Approver), nil } status = "TIMEOUT" return fmt.Sprintf("Request %s timed out", requestId), nil } ``` **Java** ```java // ApprovalWorkflowImpl.java public class ApprovalWorkflowImpl implements ApprovalWorkflow { private ApprovalData approvalData; private String status = "PENDING"; @Override public String execute(String requestId, Duration timeout) { // Block until the Signal sets approvalData, or time out (returns false) boolean approved = Workflow.await(timeout, () -> approvalData != null); if (approved) { status = approvalData.getDecision(); return "Request " + requestId + " " + status + " by " + approvalData.getApprover(); } else { status = "TIMEOUT"; return "Request " + requestId + " timed out"; } } // Signal handler: an external approver submits the decision @Override public void submitApproval(ApprovalData data) { this.approvalData = data; } // Query handler: read current status without mutating state @Override public String getStatus() { return status; } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import { ApprovalData } from './types'; export const submitApprovalSignal = wf.defineSignal<[ApprovalData]>('submitApproval'); export const getStatusQuery = wf.defineQuery('getStatus'); export async function approvalWorkflow( requestId: string, timeout: string | number, // ms or Duration string ): Promise { let approvalData: ApprovalData | undefined; let status = 'PENDING'; // Signal handler: an external approver submits the decision wf.setHandler(submitApprovalSignal, (data: ApprovalData) => { approvalData = data; }); // Query handler: read current status without mutating state wf.setHandler(getStatusQuery, () => status); // Block until the Signal sets approvalData, or time out (returns false) const approved = await wf.condition(() => approvalData !== undefined, timeout); if (approved) { status = approvalData!.decision; return `Request ${requestId} ${status} by ${approvalData!.approver}`; } else { status = 'TIMEOUT'; return `Request ${requestId} timed out`; } } ``` Each SDK uses a different mechanism to block with a timeout, but the core pattern is the same. In Java, `Workflow.await()` takes a timeout and a condition lambda, returning `false` on timeout. In TypeScript, `condition()` takes a predicate and a timeout, returning `false` on timeout. In Python, `workflow.wait_condition()` takes a lambda and a timeout, raising `asyncio.TimeoutError` on timeout. In Go, `workflow.AwaitWithTimeout()` takes a timeout and a condition function, returning `ok=false` on timeout. The condition is evaluated on every state transition, so it must not call blocking operations, mutate Workflow state, or use time-based checks. When the Signal handler sets the approval data, the condition evaluates to `true` and the Workflow unblocks. ### Multi-level approval chain Some business processes require approvals from multiple levels of authority in sequence. The following implementation iterates through a list of required approval levels, waiting for a Signal at each level before proceeding to the next: **Python** ```python # models.py from dataclasses import dataclass @dataclass class MultiLevelApprovalData: level: str # "L1", "L2", "L3" approver: str decision: str comments: str ``` **Go** ```go // types.go type MultiLevelApprovalData struct { Level string // "L1", "L2", "L3" Approver string Decision string Comments string } ``` **Java** ```java // MultiLevelApprovalData.java public class MultiLevelApprovalData { private String level; // "L1", "L2", "L3" private String approver; private String decision; private String comments; } ``` **TypeScript** ```typescript // types.ts export interface MultiLevelApprovalData { level: 'L1' | 'L2' | 'L3'; approver: string; decision: string; comments: string; } ``` This data type extends the basic approval data with a `level` field that identifies which approval tier the decision belongs to. **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow from models import MultiLevelApprovalData @workflow.defn class MultiLevelApprovalWorkflow: def __init__(self) -> None: self.approvals: list[MultiLevelApprovalData] = [] @workflow.run async def run(self, request_id: str, timeout_per_level_seconds: int) -> str: required_levels = ["L1", "L2", "L3"] timeout = timedelta(seconds=timeout_per_level_seconds) # Require approval at each level in sequence for level in required_levels: try: # Wait for a Signal carrying this level's approval await workflow.wait_condition( lambda lv=level: any(a.level == lv for a in self.approvals), timeout=timeout, ) except asyncio.TimeoutError: return f"Timeout at {level}" approval = next(a for a in self.approvals if a.level == level) # Stop the chain early if any level rejects if approval.decision == "REJECTED": return f"Rejected at {level} by {approval.approver}" return "Fully approved through all levels" # Signal handler: collect each level's approval as it arrives @workflow.signal def submit_approval(self, data: MultiLevelApprovalData) -> None: self.approvals.append(data) ``` **Go** ```go // workflow.go func MultiLevelApprovalWorkflow(ctx workflow.Context, requestId string, timeoutPerLevel time.Duration) (string, error) { var approvals []MultiLevelApprovalData requiredLevels := []string{"L1", "L2", "L3"} // Collect every approval Signal as it arrives workflow.Go(ctx, func(ctx workflow.Context) { signalChan := workflow.GetSignalChannel(ctx, "submitApproval") for { var data MultiLevelApprovalData signalChan.Receive(ctx, &data) approvals = append(approvals, data) } }) // Require approval at each level in sequence for _, level := range requiredLevels { lv := level // Wait for a Signal carrying this level's approval ok, err := workflow.AwaitWithTimeout(ctx, timeoutPerLevel, func() bool { for _, a := range approvals { if a.Level == lv { return true } } return false }) if err != nil { return "", err } if !ok { return fmt.Sprintf("Timeout at %s", lv), nil } var approval MultiLevelApprovalData for _, a := range approvals { if a.Level == lv { approval = a break } } // Stop the chain early if any level rejects if approval.Decision == "REJECTED" { return fmt.Sprintf("Rejected at %s by %s", lv, approval.Approver), nil } } return "Fully approved through all levels", nil } ``` **Java** ```java // MultiLevelApprovalWorkflowImpl.java public class MultiLevelApprovalWorkflowImpl implements ApprovalWorkflow { private List approvals = new ArrayList<>(); private String[] requiredLevels = {"L1", "L2", "L3"}; @Override public String execute(String requestId, Duration timeoutPerLevel) { // Require approval at each level in sequence for (String level : requiredLevels) { // Wait for a Signal carrying this level's approval boolean received = Workflow.await( timeoutPerLevel, () -> hasApprovalForLevel(level)); if (!received) { return "Timeout at " + level; } MultiLevelApprovalData approval = getApprovalForLevel(level); // Stop the chain early if any level rejects if (approval.getDecision().equals("REJECTED")) { return "Rejected at " + level + " by " + approval.getApprover(); } } return "Fully approved through all levels"; } // Signal handler: collect each level's approval as it arrives @Override public void submitApproval(MultiLevelApprovalData data) { approvals.add(data); } private boolean hasApprovalForLevel(String level) { return approvals.stream().anyMatch(a -> a.getLevel().equals(level)); } private MultiLevelApprovalData getApprovalForLevel(String level) { return approvals.stream() .filter(a -> a.getLevel().equals(level)) .findFirst() .orElse(null); } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import { MultiLevelApprovalData } from './types'; export const submitApprovalSignal = wf.defineSignal<[MultiLevelApprovalData]>('submitApproval'); export async function multiLevelApprovalWorkflow( requestId: string, timeoutPerLevelMs: number, ): Promise { const approvals: MultiLevelApprovalData[] = []; const requiredLevels = ['L1', 'L2', 'L3'] as const; // Signal handler: collect each level's approval as it arrives wf.setHandler(submitApprovalSignal, (data: MultiLevelApprovalData) => { approvals.push(data); }); // Require approval at each level in sequence for (const level of requiredLevels) { // Wait for a Signal carrying this level's approval const received = await wf.condition( () => approvals.some((a) => a.level === level), timeoutPerLevelMs, ); if (!received) { return `Timeout at ${level}`; } const approval = approvals.find((a) => a.level === level)!; // Stop the chain early if any level rejects if (approval.decision === 'REJECTED') { return `Rejected at ${level} by ${approval.approver}`; } } return 'Fully approved through all levels'; } ``` The Workflow loops through each required level and waits with a per-level timeout. The helper logic checks whether a Signal has arrived for the current level. If a timeout occurs at any level, the Workflow exits with a timeout result. If any level returns a rejection, the Workflow exits immediately without proceeding to subsequent levels. ### Approval with escalation When an initial approval times out, you may want to escalate the request to a manager rather than rejecting it outright. The following implementation adds an escalation step with an extended timeout: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow from models import ApprovalData with workflow.unsafe.imports_passed_through(): from activities import send_escalation_email @workflow.defn class EscalatingApprovalWorkflow: def __init__(self) -> None: self.approval_data: ApprovalData | None = None self.escalated = False @workflow.run async def run(self, request_id: str, initial_timeout_seconds: int) -> str: try: # Wait for the first approval within the initial timeout await workflow.wait_condition( lambda: self.approval_data is not None, timeout=timedelta(seconds=initial_timeout_seconds), ) except asyncio.TimeoutError: # No response in time: escalate to a manager, then wait longer self.escalated = True await workflow.execute_activity( send_escalation_email, start_to_close_timeout=timedelta(seconds=10), ) try: # Extended wait for the escalated approval await workflow.wait_condition( lambda: self.approval_data is not None, timeout=timedelta(hours=24), ) except asyncio.TimeoutError: return "Escalation timeout - auto-rejected" decision = self.approval_data.decision approver = self.approval_data.approver escalation_note = " (escalated)" if self.escalated else "" return f"{decision} by {approver}{escalation_note}" @workflow.signal def submit_approval(self, data: ApprovalData) -> None: self.approval_data = data ``` **Go** ```go // workflow.go func EscalatingApprovalWorkflow(ctx workflow.Context, requestId string, initialTimeout time.Duration) (string, error) { var approvalData *ApprovalData escalated := false workflow.Go(ctx, func(ctx workflow.Context) { signalChan := workflow.GetSignalChannel(ctx, "submitApproval") signalChan.Receive(ctx, &approvalData) }) // Wait for the first approval within the initial timeout ok, err := workflow.AwaitWithTimeout(ctx, initialTimeout, func() bool { return approvalData != nil }) if err != nil { return "", err } if !ok { // No response in time: escalate to a manager, then wait longer escalated = true ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } actCtx := workflow.WithActivityOptions(ctx, ao) // Notify the manager via an Activity err = workflow.ExecuteActivity(actCtx, SendEscalationEmail).Get(ctx, nil) if err != nil { return "", err } // Extended wait for the escalated approval ok, err = workflow.AwaitWithTimeout(ctx, 24*time.Hour, func() bool { return approvalData != nil }) if err != nil { return "", err } if !ok { return "Escalation timeout - auto-rejected", nil } } escalationNote := "" if escalated { escalationNote = " (escalated)" } return fmt.Sprintf("%s by %s%s", approvalData.Decision, approvalData.Approver, escalationNote), nil } ``` **Java** ```java // EscalatingApprovalWorkflowImpl.java public class EscalatingApprovalWorkflowImpl implements ApprovalWorkflow { private ApprovalData approvalData; private boolean escalated = false; @Override public String execute(String requestId, Duration initialTimeout) { // Wait for the first approval within the initial timeout boolean received = Workflow.await(initialTimeout, () -> approvalData != null); if (!received) { // No response in time: escalate to a manager, then wait longer escalated = true; sendEscalationNotification(); // Extended wait for the escalated approval received = Workflow.await( Duration.ofHours(24), () -> approvalData != null); if (!received) { return "Escalation timeout - auto-rejected"; } } String decision = approvalData.getDecision(); String approver = approvalData.getApprover(); String escalationNote = escalated ? " (escalated)" : ""; return decision + " by " + approver + escalationNote; } @Override public void submitApproval(ApprovalData data) { this.approvalData = data; } private void sendEscalationNotification() { ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .build(); NotificationActivities activities = Workflow.newActivityStub(NotificationActivities.class, options); activities.sendEscalationEmail(); } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; import { ApprovalData } from './types'; const { sendEscalationEmail } = wf.proxyActivities({ startToCloseTimeout: '10 seconds', }); export const submitApprovalSignal = wf.defineSignal<[ApprovalData]>('submitApproval'); export async function escalatingApprovalWorkflow( requestId: string, initialTimeoutMs: number, ): Promise { let approvalData: ApprovalData | undefined; let escalated = false; wf.setHandler(submitApprovalSignal, (data: ApprovalData) => { approvalData = data; }); // Wait for the first approval within the initial timeout let received = await wf.condition( () => approvalData !== undefined, initialTimeoutMs, ); if (!received) { // No response in time: escalate to a manager, then wait longer escalated = true; await sendEscalationEmail(); // Extended wait for the escalated approval received = await wf.condition( () => approvalData !== undefined, '24 hours', ); if (!received) { return 'Escalation timeout - auto-rejected'; } } const { decision, approver } = approvalData!; const escalationNote = escalated ? ' (escalated)' : ''; return `${decision} by ${approver}${escalationNote}`; } ``` The Workflow first waits for the initial timeout. If no Signal arrives, it sets the `escalated` flag, executes a notification Activity to alert the manager, and then waits again with a 24-hour extended timeout. The notification Activity uses a short start-to-close timeout, since sending an email should complete quickly. The final result includes an escalation note so the caller knows the request was escalated before approval. ## When to use The Approval pattern is a good fit for purchase order approvals, expense report reviews, code deployment gates, contract signing Workflows, manual quality checks, compliance reviews, budget authorization, and access request approvals. It is not a good fit for fully automated processes that require no human input, real-time decisions that need synchronous API responses, or processes that require sub-second response times. If you only need a boolean yes/no without any context, a plain boolean Signal may be sufficient. ## Benefits and trade-offs The Approval pattern captures rich context — approver identity, reasons, and timestamps — alongside each decision. All approval data is recorded in the Workflow history as Signal events, giving you a built-in audit trail. Timeout handling is automatic: you define the maximum wait time and the Workflow handles the fallback. The pattern supports multi-level, conditional, and escalating approval chains, and you can check approval status at any time through Query methods without modifying Workflow state. Because all decisions are recorded in the event history, the Workflow is deterministic and replay-safe. The trade-offs to consider are that the pattern requires an external system to send approval Signals, which means you need a separate approval interface. The Workflow blocks until the approval arrives or the timeout expires, so you must define a maximum wait time. Large approval data objects increase the size of the Workflow history. ## Comparison with alternatives | Approach | Rich data | Built-in wait | Caller gets result | Complexity | Use case | | :--- | :--- | :--- | :--- | :--- | :--- | | Signal with data | Yes | Yes | No | Low | Approval Workflows | | Update | Yes | No | Yes | Low | Synchronous validation with immediate confirmation | | Boolean Signal | No | Yes | No | Low | Yes/no decisions | | Polling Activity | Yes | Yes | Yes | High | External approval systems | Signals are fire-and-forget: the caller receives an acknowledgement from the server but cannot wait for the Workflow to process the Signal or receive a result. Updates are synchronous: the caller blocks until the handler completes and can receive a return value or error. If the approver's interface needs immediate confirmation that the approval was accepted and valid, consider using an Update with a validator instead of a Signal. ## Best practices - **Use custom data objects.** Capture rich approval context — approver identity, comments, timestamps — rather than a plain boolean. - **Set reasonable timeouts.** Balance responsiveness with the time approvers realistically need to respond. - **Add Query methods.** Expose the current approval status so external systems can check progress without sending a Signal. - **Validate Signal data.** Verify approver permissions and data completeness before accepting an approval. - **Log approval events.** Record each decision for audit trails and compliance. - **Handle timeouts gracefully.** Define clear timeout behavior such as rejection, escalation, or notification. - **Support cancellation.** Allow Workflows to be cancelled if the request is withdrawn. - **Ensure idempotency.** Handle duplicate approval Signals safely so that re-delivery does not corrupt state. Signals [may be duplicated in rare cases](/handling-messages#exactly-once-message-processing), so use idempotency keys when necessary. - **Include timestamps.** Record when each approval was submitted to support time-based auditing. - **Expose approval history.** Provide a Query method that returns all approval attempts, not only the final decision. ## Common pitfalls - **No timeout.** Without a timeout, the Workflow waits indefinitely for an approval that may never arrive. - **Missing validation.** Accepting approvals from unauthorized users compromises the integrity of the process. - **Lost context.** Failing to capture the approver's identity or reason makes audit trails incomplete. - **Assuming non-deterministic races.** Temporal processes events in a deterministic, single-threaded order, so a Signal and a timer cannot truly "race." However, if the Signal arrives after the timer fires in the event history, the wait will have already returned with a timeout result. Design your timeout path to account for late-arriving Signals. - **No audit trail.** Skipping approval logging makes it difficult to meet compliance requirements. - **Tight timeouts.** Setting the timeout too short causes legitimate approvals to be rejected. - **Boolean-only Signals.** Using a plain boolean instead of a rich data object limits your ability to capture decision context. - **No status Query.** Without a Query method, external systems have no way to check approval progress. - **No duplicate handling.** Receiving multiple approval Signals without deduplication can overwrite earlier decisions. - **No escalation path.** Without a fallback when the initial approval times out, requests stall or are silently rejected. ## Related ### Patterns - [Signal-Based Event Handling](/design-patterns/signal-with-start): Receiving external events through Signals. - [Updatable Timer](/design-patterns/updatable-timer): Extending approval deadlines dynamically. - [Saga Pattern](/design-patterns/saga-pattern): Executing compensating actions on rejection. ### Guides - [Reliable document approvals](/guides/reliable-document-approvals): A complete Python implementation with SLA timers, automatic escalation, resubmission loops, and audit logging. ### Sample code **Java** - [Hello Signal](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSignal.java) — Basic Signal handling in a Workflow. - [Safe Message Passing](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Signal handling with validation. **TypeScript** - [Signals and Queries](https://github.com/temporalio/samples-typescript/tree/main/signals-queries) — Signal and Query usage in a Workflow. - [Message Passing](https://github.com/temporalio/samples-typescript/tree/main/message-passing) — Introduction to message passing with Signals, Queries, and Updates. **Python** - [Hello Signal](https://github.com/temporalio/samples-python/tree/main/hello/hello_signal.py) — Basic Signal handling in a Workflow. - [Message Passing](https://github.com/temporalio/samples-python/tree/main/message_passing/introduction) — Introduction to message passing with Signals, Queries, and Updates. **Go** - [Await Signals](https://github.com/temporalio/samples-go/tree/main/await-signals) — Waiting for Signals with timeout using `AwaitWithTimeout`. - [Message Passing](https://github.com/temporalio/samples-go/tree/message-passing/message-passing-intro) — Introduction to message passing with Signals, Queries, and Updates. --- # Batch Iterator Source: https://docs.temporal.io/design-patterns/batch-iterator > Pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees. > **ℹ️ TLDR:** > **Process one page at a time** and call Continue-as-New with the next offset after each page so the Workflow's event history never grows without bound. With this method you can process infinite pages. Use this when your record set is arbitrarily large, you need a durable checkpoint after every page, and sequential page-by-page throughput is acceptable. ## Overview The Batch Iterator pattern processes a large record set one page at a time. Each Workflow run processes a single page and then calls Continue-as-New with the next offset, producing a chain of short-lived runs that together cover the entire record set without accumulating unbounded event history. ## Problem A single Workflow run is limited to 50,000 history events (aim for 2,000) and 2,000 in-flight Activities. Processing millions of records in one run is not possible within these bounds. You need a way to process an arbitrarily large record set reliably, with the ability to resume from a checkpoint if the Workflow is interrupted, and without overwhelming downstream systems with a burst of concurrent requests. ## Solution Each Workflow run fetches one page of records using a persistent `offset` parameter, processes each record sequentially, and then calls `continueAsNew` with the incremented offset. The next run picks up exactly where the previous one left off. Because each run processes only a bounded number of records, history stays well within limits. The offset acts as a durable checkpoint: if the Workflow is interrupted mid-page, the next run replays only from the start of the current page. ```mermaid flowchart TD DB[("Data Source\n(paginated)")] WF1["Workflow Run 1\n(offset=0)"] WF2["Workflow Run 2\n(offset=PAGE_SIZE)"] WF3["Workflow Run N\n(offset=N×PAGE_SIZE)"] Done(["Complete"]) DB -->|"fetch page 1"| WF1 WF1 -->|"processRecord ×PAGE_SIZE"| Acts1["Activities"] WF1 -->|"continueAsNew\n(offset=PAGE_SIZE)"| WF2 DB -->|"fetch page 2"| WF2 WF2 -->|"processRecord ×PAGE_SIZE"| Acts2["Activities"] WF2 -->|"continueAsNew\n(offset=N×PAGE_SIZE)"| WF3 DB -->|"fetch page N"| WF3 WF3 -->|"processRecord ×PAGE_SIZE"| Acts3["Activities"] WF3 -->|"last page → return"| Done ``` The following describes each step in the diagram: 1. The Workflow starts with `offset=0` and calls `fetchPage(offset, pageSize)` to retrieve the first page of records. 2. It processes each record in the page by executing the `processRecord` Activity. 3. After the page is fully processed, it calls `continueAsNew` with `offset + pageSize`, passing the updated offset to the next run. 4. The next run begins with a clean history and repeats the same steps for the next page. 5. When `fetchPage` returns fewer records than `pageSize`, the Workflow knows it has reached the last page and returns normally. ## Implementation The following examples show how each SDK implements the Batch Iterator pattern. **TypeScript** ```typescript // workflows.ts import { continueAsNew, log, proxyActivities } from "@temporalio/workflow"; import type * as activities from "./activities"; import { PAGE_SIZE } from "./shared"; const { fetchPage, processRecord } = proxyActivities({ startToCloseTimeout: "10 seconds", }); export async function batchIteratorWorkflow( offset: number = 0, totalProcessed: number = 0 ): Promise { const page = await fetchPage(offset, PAGE_SIZE); for (const record of page) { await processRecord(record); totalProcessed++; } log.info(`Processed page at offset ${offset} (${page.length} records, running total: ${totalProcessed})`); if (page.length === PAGE_SIZE) { await continueAsNew(offset + PAGE_SIZE, totalProcessed); } return totalProcessed; } ``` **Python** ```python # workflows.py from temporalio import workflow from temporalio.workflow import continue_as_new from datetime import timedelta from activities import fetch_page, process_record from shared import PAGE_SIZE @workflow.defn class BatchIteratorWorkflow: @workflow.run async def run(self, offset: int = 0, total_processed: int = 0) -> int: page = await workflow.execute_activity( fetch_page, args=[offset, PAGE_SIZE], start_to_close_timeout=timedelta(seconds=10), ) for record in page: await workflow.execute_activity( process_record, record, start_to_close_timeout=timedelta(seconds=10), ) total_processed += 1 workflow.logger.info( f"Processed page at offset {offset} ({len(page)} records, running total: {total_processed})" ) if len(page) == PAGE_SIZE: continue_as_new(args=[offset + PAGE_SIZE, total_processed]) return total_processed ``` **Go** ```go // workflows.go package main import ( "time" "go.temporal.io/sdk/workflow" ) func BatchIteratorWorkflow(ctx workflow.Context, offset int, totalProcessed int) (int, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var page []Record if err := workflow.ExecuteActivity(ctx, FetchPage, offset, PageSize).Get(ctx, &page); err != nil { return totalProcessed, err } for _, record := range page { if err := workflow.ExecuteActivity(ctx, ProcessRecord, record).Get(ctx, nil); err != nil { return totalProcessed, err } totalProcessed++ } workflow.GetLogger(ctx).Info("Processed page", "offset", offset, "pageSize", len(page), "totalProcessed", totalProcessed) if len(page) == PageSize { return totalProcessed, workflow.NewContinueAsNewError(ctx, BatchIteratorWorkflow, offset+PageSize, totalProcessed) } return totalProcessed, nil } ``` **Java** ```java // BatchIteratorWorkflow.java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.*; import java.time.Duration; import java.util.List; @WorkflowInterface public interface BatchIteratorWorkflow { @WorkflowMethod int run(int offset, int totalProcessed); } // BatchIteratorWorkflowImpl.java public class BatchIteratorWorkflowImpl implements BatchIteratorWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .build() ); @Override public int run(int offset, int totalProcessed) { List page = activities.fetchPage(offset, Shared.PAGE_SIZE); for (Record record : page) { activities.processRecord(record); totalProcessed++; } Workflow.getLogger(BatchIteratorWorkflowImpl.class).info( "Processed page at offset " + offset + " (" + page.size() + " records, total: " + totalProcessed + ")" ); if (page.size() == Shared.PAGE_SIZE) { BatchIteratorWorkflow next = Workflow.newContinueAsNewStub(BatchIteratorWorkflow.class); next.run(offset + Shared.PAGE_SIZE, totalProcessed); } return totalProcessed; } } ``` ## Best practices - **Choose a page size that keeps history under 2,000 events.** Each page produces roughly `3 × pageSize` history events (`ActivityTaskScheduled` + `ActivityTaskStarted` + `ActivityTaskCompleted`). A page size of 500–800 records is a safe target. - **Include `totalProcessed` (or a similar counter) in the `continueAsNew` args.** This lets you observe overall progress via the Workflow input visible in the UI without querying internal state. - **Fetch inside an Activity, not the Workflow.** The `fetchPage` call must be an Activity — not inline Workflow code — so it can interact with external systems and be retried independently. - **Make `processRecord` idempotent.** Activities have at-least-once execution semantics. If a worker crashes after an Activity completes externally but before the completion is recorded in history, Temporal will retry it. Your downstream system must tolerate receiving the same record more than once. - **Avoid accumulating large local state between pages.** `continueAsNew` does not carry over in-memory state; only the arguments you pass are available in the next run. ## Common pitfalls - **Forgetting `continueAsNew` on the last page.** If you call `continueAsNew` unconditionally, the Workflow loops forever even when the data source is exhausted. Check whether the returned page is shorter than `pageSize` before continuing. - **Passing unnecessary state into `continueAsNew`.** All arguments are serialized and stored in history. Pass only the minimal state needed (offset, counters) — not accumulated result lists or large collections that grow with each page. - **Sequential processing bottlenecks.** The default implementation processes one record at a time per page. You can fan out Activities concurrently within a page using the SDK's async primitives for higher per-page throughput — note this increases per-page event count accordingly. If record-set-wide throughput matters more than rate limiting, consider [Sliding Window](/design-patterns/sliding-window) or [MapReduce Tree](/design-patterns/mapreduce-tree). ## Related ### Patterns - [Continue-as-New pattern](/design-patterns/continue-as-new) — core concepts for history management via `continueAsNew` - [Sliding Window](/design-patterns/sliding-window) — bounded concurrency that progresses at the rate of the fastest processor - [MapReduce Tree](/design-patterns/mapreduce-tree) — fully parallel processing for maximum speed - [Temporal limits reference](/cloud/limits) - [Batch samples (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/iterator) --- # Batch Processing Patterns Source: https://docs.temporal.io/design-patterns/batch-processing-patterns > Compare Fan-Out, Batch Iterator, Sliding Window, and MapReduce Tree patterns for processing large record sets reliably at scale. These patterns process large volumes of records reliably, at scale, and without overwhelming downstream systems. Choose based on your throughput requirements, record set size, and whether you need rate limiting or maximum parallelism. ## When to use which pattern | Pattern | Record set size | Parallelism model | Workflow-based rate control | |---|---|---|---| | [Basic Workflow](#basic-workflow-single-tier-fan-out) | Small (up to a few hundred records) | Sequential or parallel activities in one Workflow | No | | [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows) | Up to ~4M records | Fixed concurrency (one child per chunk) | No | | [Batch Iterator](/design-patterns/batch-iterator) | Unlimited | Limited (activities per page) | Yes — fixed page rate | | [Sliding Window](/design-patterns/sliding-window) | Unlimited | Bounded window of concurrent children | Yes — configurable window | | [MapReduce Tree](/design-patterns/mapreduce-tree) | Unlimited | Fully parallel recursive tree | No — maximum speed | ## Patterns in this section - [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows): Splits a record set into fixed-size chunks and assigns each to an independent child Workflow. Direct to reason about; best for record sets up to ~4M items. - [Batch Iterator](/design-patterns/batch-iterator): Processes one page of records per Workflow run and continues as new with the next page offset. Handles unlimited record sets while controlling downstream traffic. - [Sliding Window](/design-patterns/sliding-window): Maintains a fixed-size window of concurrent child Workflows. As each child completes it signals the parent, which immediately starts a replacement — maximizing throughput within a concurrency budget. - [MapReduce Tree](/design-patterns/mapreduce-tree): Recursively splits a record set into chunks, fans out to leaf Workflows for parallel processing, and signals results back up the tree. Maximizes speed for embarrassingly parallel workloads. --- ## Schedules Schedules allow Workflows to be executed on a recurring basis — think of them as a more flexible cron with start, pause, stop, update, and backfill controls. - Supports `start` / `pause` / `stop` / `update` / `backfill` of scheduled Workflow executions - Configurable **Overlap Policies** control what happens when the previous run is still running - Full execution history visibility in the Temporal UI - Schedules can be created via the UI, CLI, or SDK ```bash temporal schedule create \ --schedule-id 'your-schedule-id' \ --workflow-id 'your-workflow-id' \ --task-queue 'your-task-queue' \ --workflow-type 'YourWorkflowType' ``` **References:** - [Temporal Schedules](/schedule) - [CLI schedule commands](/cli/command-reference/schedule) --- ## Basic Workflow (single-tier fan-out) The most direct form of batch processing: the Workflow fetches or receives record IDs and executes one Activity per record. - Activities can be executed sequentially or concurrently (using the SDK's async primitives) - **Limit: 2,000 in-flight Activities per Workflow run** (aim for 500) - If total event count is likely to exceed 2,000 (hard limit: 51,200), use the [Batch Iterator](/design-patterns/batch-iterator) instead **Pros:** Minimal code and orchestration overhead **Cons:** Hard cap on concurrent Activities; all-or-nothing failure model; can overwhelm downstream systems ```mermaid flowchart TD Records["📋 Record IDs\n(fetched or passed in)"] WF["Workflow"] A1["Activity"] A2["Activity"] AN["Activity ..."] Records --> WF WF --> A1 WF --> A2 WF --> AN ``` --- ## Batch signalling The Temporal CLI lets you signal, reset, cancel, or terminate multiple Workflows with a single command using a visibility query. - 1 running batch job per namespace - 50 Workflows per second per batch ```bash # Signal all running Workflows of a given type temporal workflow signal \ --name MySignal \ --input '{"Input": "As-JSON"}' \ --query 'ExecutionStatus = "Running" AND WorkflowType="YourWorkflow"' \ --reason "Testing" # Terminate all running Workflows of a given type temporal workflow terminate \ --query 'ExecutionStatus = "Running" AND WorkflowType="SomeWorkflowType"' \ --reason "Terminate Test Workflows" ``` **Reference:** [CLI batch commands](/cli/command-reference/batch) --- ## Key limits Full reference: [Temporal Cloud limits](/cloud/limits) | Limit | Value | |---|---| | Unfinished actions per Workflow | 2,000 max (aim for 500). Includes Activities, Signals, Child Workflows, cancellation requests | | Events per Workflow history | 51,200 events max (aim for a few thousand) **or** 50 MB total history size; warns at 10,240 events / 10 MB | | Signals per Workflow | 10,000 | | Updates per Workflow | 10 in-flight, 2,000 total | | Batch Signalling | 1 batch job per namespace; 50 Workflows/sec per batch | ## Related sections - [Task Orchestration Patterns](/design-patterns/task-orchestration-patterns) — the child-Workflow and parallel primitives these patterns scale up - [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — rate-limit and prioritize the work a batch generates - [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — reduce the latency of each record's processing --- # Child Workflows Pattern Source: https://docs.temporal.io/design-patterns/child-workflows > Decomposes complex Workflows into smaller, reusable units. Each child has an independent Workflow ID, history, and lifecycle. ## Overview Child Workflows enable decomposition of complex business logic into smaller, reusable Workflow units. Each child executes as an independent Workflow with its own Workflow ID, event history (50K event limit), and lifecycle. Unlike Activities which execute code, Child Workflows orchestrate processes and provide Workflow-level semantics: independent tracking, querying, timeouts, and the ability to outlive the parent. Key capabilities: - **Independent identity**: Each child has a unique Workflow ID visible in the UI for tracking and querying. - **Separate history**: Each child maintains its own event history, preventing parent history bloat. - **Flexible invocation**: Synchronous (blocking) or asynchronous (non-blocking) execution. - **Lifecycle control**: Parent close policies (TERMINATE, ABANDON, REQUEST_CANCEL) determine child behavior when the parent completes. - **Task Queue routing**: Children can execute on different Task Queues with specialized Workers. - **Reusability**: The same Child Workflow logic can be invoked by multiple different parent Workflows. ## Problem In distributed systems, you often need Workflows that break down complex processes into modular, reusable components, execute sub-processes that may outlive the parent Workflow, coordinate multiple independent Workflows with different lifecycles, isolate failure domains while maintaining orchestration control, and reuse Workflow logic across different parent Workflows. Without Child Workflows, you must implement all logic in a single monolithic Workflow, manually coordinate separate Workflows via Signals and Queries, duplicate Workflow logic across multiple implementations, and manage complex state machines for sub-process coordination. ## Solution You invoke Child Workflows from within parent Workflows using the SDK's Child Workflow API. You can call them synchronously (blocking until completion) or asynchronously (fire-and-forget). The `ParentClosePolicy` determines what happens to children when the parent completes. ```mermaid sequenceDiagram participant Parent participant Child1 participant Child2 Parent->>+Child1: Start (sync) activate Parent Child1->>Child1: Execute Child1-->>-Parent: Result Parent->>+Child2: Start (async) Note over Parent,Child2: Parent continues immediately Parent->>Parent: Do other work Child2->>Child2: Execute independently alt Parent completes first Parent->>Parent: Complete deactivate Parent Note over Child2: Policy: ABANDON
Child continues Child2->>Child2: Keep running Child2-->>-Child2: Complete end ``` The following describes each step in the diagram: 1. The parent starts Child 1 synchronously and blocks until it completes. 2. The parent starts Child 2 asynchronously and continues doing other work immediately. 3. If the parent completes before Child 2, the ABANDON policy allows Child 2 to continue running independently. ### Synchronous Child Workflow The following example creates a Child Workflow and calls it synchronously. The parent blocks until the child completes and returns a result: **Python** ```python # workflows.py from temporalio import workflow from child_workflows import ChildWorkflow @workflow.defn class ParentWorkflow: @workflow.run async def run(self, input: str) -> str: # Synchronous call - awaits until child completes result = await workflow.execute_child_workflow( ChildWorkflow.run, input, id=f"child-{workflow.uuid4()}", ) return f"Parent received: {result}" ``` **Go** ```go // parent_workflow.go func ParentWorkflow(ctx workflow.Context, input string) (string, error) { cwo := workflow.ChildWorkflowOptions{} ctx = workflow.WithChildOptions(ctx, cwo) // Synchronous call - blocks until child completes var result string err := workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, input).Get(ctx, &result) if err != nil { return "", err } return "Parent received: " + result, nil } ``` **Java** ```java // ParentWorkflowImpl.java @WorkflowInterface public interface ParentWorkflow { @WorkflowMethod String execute(String input); } public class ParentWorkflowImpl implements ParentWorkflow { @Override public String execute(String input) { ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class); // Synchronous call - blocks until child completes String result = child.processData(input); return "Parent received: " + result; } } ``` **TypeScript** ```typescript // workflows.ts import { executeChild } from '@temporalio/workflow'; import { childWorkflow } from './child-workflows'; export async function parentWorkflow(input: string): Promise { // Synchronous call - awaits until child completes const result = await executeChild(childWorkflow, { args: [input], }); return `Parent received: ${result}`; } ``` In Java, `Workflow.newChildWorkflowStub()` creates a typed stub and calling a method on it blocks the parent. In TypeScript, `executeChild()` starts the child and awaits its completion. In Python, `workflow.execute_child_workflow()` starts the child and awaits its completion. In Go, `workflow.ExecuteChildWorkflow()` returns a `ChildWorkflowFuture`, and calling `.Get()` blocks until the child completes. ### Asynchronous Child Workflow The following example starts a Child Workflow asynchronously with an ABANDON policy. The parent receives the child's execution info without waiting for completion. Select a concept to highlight the matching lines: **Python** annotations={[ { label: 'Parent Close Policy', description: 'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.', lines: [14], }, { label: 'Workflow Id', description: 'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.', lines: [13], }, { label: 'Async start', description: 'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.', lines: [10, 11, 12, 13, 14, 15], }, ]} > ```python from temporalio import workflow from temporalio.workflow import ParentClosePolicy from child_workflows import ChildWorkflow @workflow.defn class ParentWorkflow: @workflow.run async def run(self, input: str) -> str: handle = await workflow.start_child_workflow( ChildWorkflow.run, input, id=f"child-{workflow.uuid4()}", parent_close_policy=ParentClosePolicy.ABANDON, ) return handle.id ``` **Go** annotations={[ { label: 'Parent Close Policy', description: 'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.', lines: [3], }, { label: 'Workflow Id', description: 'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.', lines: [14], }, { label: 'Async start', description: 'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.', lines: [7], }, ]} > ```go func ParentWorkflow(ctx workflow.Context, input string) (string, error) { cwo := workflow.ChildWorkflowOptions{ ParentClosePolicy: enumspb.PARENT_CLOSE_POLICY_ABANDON, } ctx = workflow.WithChildOptions(ctx, cwo) childFuture := workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, input) var childWE workflow.Execution if err := childFuture.GetChildWorkflowExecution().Get(ctx, &childWE); err != nil { return "", err } return childWE.ID, nil } ``` **Java** annotations={[ { label: 'Parent Close Policy', description: 'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.', lines: [6], }, { label: 'Workflow Id', description: 'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.', lines: [5], }, { label: 'Async start', description: 'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.', lines: [11], }, ]} > ```java public class ParentWorkflowImpl implements ParentWorkflow { @Override public WorkflowExecution execute(String input) { ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder() .setWorkflowId("child-" + Workflow.randomUUID()) .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON) .build(); ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class, options); Async.function(child::processData, input); Promise childExecution = Workflow.getWorkflowExecution(child); return childExecution.get(); } } ``` **TypeScript** annotations={[ { label: 'Parent Close Policy', description: 'Controls what happens to the Child Workflow when the parent closes. ABANDON lets the child keep running after the parent completes.', lines: [7], }, { label: 'Workflow Id', description: 'Gives the Child Workflow a stable identity for tracking, querying, and deduplication in the UI.', lines: [10], }, { label: 'Async start', description: 'Starts the Child Workflow without waiting for it to finish. The parent continues after the child has started.', lines: [5, 6, 7, 8], }, ]} > ```typescript import { startChild, ParentClosePolicy } from '@temporalio/workflow'; import { childWorkflow } from './child-workflows'; export async function parentWorkflow(input: string): Promise { const childHandle = await startChild(childWorkflow, { args: [input], parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON, }); return childHandle.workflowId; } ``` In Java, `Async.function()` starts the child asynchronously. `Workflow.getWorkflowExecution(child)` returns a Promise that resolves when the child starts (not when it completes). In TypeScript, `startChild()` returns a handle once the child has started. In Python, `workflow.start_child_workflow()` returns a handle once the child has started, without waiting for completion. In Go, `childFuture.GetChildWorkflowExecution().Get()` blocks until the child has started. The ABANDON policy ensures the child continues running even if the parent completes first. ## Parent Close Policy The `ParentClosePolicy` determines Child Workflow behavior when the parent closes: | Policy | Behavior | Use case | | :--- | :--- | :--- | | `TERMINATE` | Child is terminated when parent closes | Tightly coupled processes | | `ABANDON` | Child continues independently | Fire-and-forget, long-running tasks | | `REQUEST_CANCEL` | Child receives cancellation request | Graceful cleanup | ## Implementation ### Parallel Child Workflows The following example starts multiple Child Workflows in parallel and waits for all of them to complete: **Python** ```python # workflows.py import asyncio from temporalio import workflow from child_workflows import ChildWorkflow @workflow.defn class ParallelParentWorkflow: @workflow.run async def run(self, items: list[str]) -> str: # Start all children concurrently using asyncio.gather results = await asyncio.gather( *[ workflow.execute_child_workflow( ChildWorkflow.run, item, id=f"child-{workflow.uuid4()}", ) for item in items ] ) return ", ".join(results) ``` **Go** ```go // parallel_parent_workflow.go func ParallelParentWorkflow(ctx workflow.Context, items []string) (string, error) { cwo := workflow.ChildWorkflowOptions{} ctx = workflow.WithChildOptions(ctx, cwo) // Start all children - ExecuteChildWorkflow returns immediately var futures []workflow.ChildWorkflowFuture for _, item := range items { futures = append(futures, workflow.ExecuteChildWorkflow(ctx, ChildWorkflow, item)) } // Wait for all children to complete var results []string for _, future := range futures { var result string if err := future.Get(ctx, &result); err != nil { return "", err } results = append(results, result) } return strings.Join(results, ", "), nil } ``` **Java** ```java // ParallelParentWorkflowImpl.java public class ParallelParentWorkflowImpl implements ParentWorkflow { @Override public String execute(List items) { List> promises = new ArrayList<>(); for (String item : items) { ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class); promises.add(Async.function(child::process, item)); } // Wait for all children to complete Promise.allOf(promises).get(); return promises.stream() .map(Promise::get) .collect(Collectors.joining(", ")); } } ``` **TypeScript** ```typescript // workflows.ts import { executeChild } from '@temporalio/workflow'; import { childWorkflow } from './child-workflows'; export async function parallelParentWorkflow(items: string[]): Promise { // Start all children concurrently using Promise.all const results = await Promise.all( items.map((item) => executeChild(childWorkflow, { args: [item], }) ) ); return results.join(', '); } ``` In Java, each child starts asynchronously via `Async.function()`, and `Promise.allOf(promises).get()` blocks until every child completes. In TypeScript, `Promise.all()` starts all children concurrently and awaits all results. In Python, `asyncio.gather()` starts all children concurrently and awaits all results. In Go, `workflow.ExecuteChildWorkflow()` returns a Future immediately without blocking, so starting all children in a loop launches them in parallel. Calling `.Get()` on each Future afterward collects the results. ### Fire-and-forget The following example starts a Child Workflow with the ABANDON policy and returns immediately without waiting: **Python** ```python # workflows.py from temporalio import workflow from temporalio.workflow import ParentClosePolicy from child_workflows import LongRunningChildWorkflow @workflow.defn class FireAndForgetParentWorkflow: @workflow.run async def run(self, data: str) -> None: # Start child with ABANDON policy - child survives parent completion await workflow.start_child_workflow( LongRunningChildWorkflow.run, data, id=f"child-{workflow.uuid4()}", parent_close_policy=ParentClosePolicy.ABANDON, ) # start_child_workflow resolves once the child has started # Parent completes, child continues independently ``` **Go** ```go // fire_and_forget_workflow.go import ( enumspb "go.temporal.io/api/enums/v1" "go.temporal.io/sdk/workflow" ) func FireAndForgetParentWorkflow(ctx workflow.Context, data string) error { cwo := workflow.ChildWorkflowOptions{ ParentClosePolicy: enumspb.PARENT_CLOSE_POLICY_ABANDON, } ctx = workflow.WithChildOptions(ctx, cwo) childFuture := workflow.ExecuteChildWorkflow(ctx, LongRunningChildWorkflow, data) // Wait for child to start before parent completes if err := childFuture.GetChildWorkflowExecution().Get(ctx, nil); err != nil { return err } // Parent completes, child continues independently return nil } ``` **Java** ```java // FireAndForgetParentWorkflowImpl.java public class FireAndForgetParentWorkflowImpl implements ParentWorkflow { @Override public void execute(String data) { ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder() .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON) .build(); ChildWorkflow child = Workflow.newChildWorkflowStub(ChildWorkflow.class, options); // Start child and don't wait for completion Async.function(child::longRunningProcess, data); // Wait for child to start before parent completes Workflow.getWorkflowExecution(child).get(); // Parent completes, child continues independently } } ``` **TypeScript** ```typescript // workflows.ts import { startChild, ParentClosePolicy } from '@temporalio/workflow'; import { longRunningChildWorkflow } from './child-workflows'; export async function fireAndForgetParentWorkflow(data: string): Promise { // Start child with ABANDON policy - child survives parent completion await startChild(longRunningChildWorkflow, { args: [data], parentClosePolicy: ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON, }); // startChild resolves once the child has started // Parent completes, child continues independently } ``` You must wait for the child to start before the parent completes. Without this, the parent could complete before the child is scheduled, and the child would never execute. The ABANDON policy ensures the child continues running after the parent completes. ### Conditional child execution The following example conditionally starts different Child Workflows based on business logic: **Python** ```python # workflows.py from temporalio import workflow from child_workflows import ApprovalWorkflow, FulfillmentWorkflow @workflow.defn class ConditionalParentWorkflow: @workflow.run async def run(self, order: Order) -> str: if order.requires_approval: approved = await workflow.execute_child_workflow( ApprovalWorkflow.run, order, id=f"approval-{order.id}", ) if not approved: return "Order rejected" return await workflow.execute_child_workflow( FulfillmentWorkflow.run, order, id=f"fulfillment-{order.id}", ) ``` **Go** ```go // conditional_parent_workflow.go func ConditionalParentWorkflow(ctx workflow.Context, order Order) (string, error) { cwo := workflow.ChildWorkflowOptions{} ctx = workflow.WithChildOptions(ctx, cwo) if order.RequiresApproval { var approved bool err := workflow.ExecuteChildWorkflow(ctx, ApprovalWorkflow, order).Get(ctx, &approved) if err != nil { return "", err } if !approved { return "Order rejected", nil } } var result string err := workflow.ExecuteChildWorkflow(ctx, FulfillmentWorkflow, order).Get(ctx, &result) if err != nil { return "", err } return result, nil } ``` **Java** ```java // ConditionalParentWorkflowImpl.java public class ConditionalParentWorkflowImpl implements ParentWorkflow { @Override public String execute(Order order) { if (order.requiresApproval()) { ApprovalWorkflow approval = Workflow.newChildWorkflowStub(ApprovalWorkflow.class); boolean approved = approval.requestApproval(order); if (!approved) { return "Order rejected"; } } FulfillmentWorkflow fulfillment = Workflow.newChildWorkflowStub(FulfillmentWorkflow.class); return fulfillment.fulfill(order); } } ``` **TypeScript** ```typescript // workflows.ts import { executeChild } from '@temporalio/workflow'; import { approvalWorkflow, fulfillmentWorkflow } from './child-workflows'; export async function conditionalParentWorkflow(order: Order): Promise { if (order.requiresApproval) { const approved = await executeChild(approvalWorkflow, { args: [order], }); if (!approved) { return 'Order rejected'; } } return await executeChild(fulfillmentWorkflow, { args: [order], }); } ``` The parent checks whether the order requires approval and only starts the approval Child Workflow when needed. ## When to use Child Workflows and Activities serve different purposes. Use Child Workflows when: - You need a separate Workflow ID for tracking and querying. - The operation may outlive the parent Workflow. - You need to reuse Workflow logic across multiple parents. - You want to execute Workflows on different Task Queues. - You need independent history and event limits. - You want to apply different timeouts or retry policies at the Workflow level. Use Activities when: - You are executing external operations (API calls, database queries). - The operation is short-lived. - You do not need independent Workflow tracking. - The operation is tightly coupled to the parent Workflow lifecycle. - Lower overhead is important. The key distinction is that Activities are for executing code (especially external operations), while Child Workflows are for orchestrating processes that benefit from independent Workflow semantics. ## Benefits and trade-offs Child Workflows provide modularity by breaking complex logic into reusable units. Each child is a first-class Workflow with its own ID for tracking, its own 50K event history limit, and its own execution timeout configuration. Children can outlive parents with the ABANDON policy, and you can start multiple children concurrently. Child failures do not automatically fail the parent, and the same Child Workflow can be reused by multiple parents. The trade-offs to consider are that each child is a separate Workflow execution with its own history (overhead). There are more moving parts than a single Workflow. Child execution details are not in the parent history (but are queryable independently). Async children require explicit synchronization if needed. More Workflow executions mean higher resource usage. Starting a Child Workflow has more overhead than starting an Activity. ## Comparison with alternatives | Approach | Modularity | Independent history | Can outlive parent | Overhead | Separate Workflow ID | | :--- | :--- | :--- | :--- | :--- | :--- | | Child Workflow | High | Yes | Yes (ABANDON) | Medium | Yes | | Activity | Medium | No | No | Low | No | | Separate Workflow + Signals | High | Yes | Yes | High | Yes | | Async Lambda | Low | No | No | Very Low | No | ## Best practices - **Use unique Workflow IDs.** Generate unique IDs for Child Workflows to avoid conflicts. - **Choose the appropriate policy.** Use TERMINATE for tightly coupled children, ABANDON for independent children. - **Handle child failures.** Catch and handle Child Workflow exceptions appropriately. - **Limit parallelism.** Do not spawn unlimited children; use batch patterns for large datasets. - **Consider Activities first.** Use Activities for operations that do not need independent Workflow tracking. - **Set timeouts.** Configure appropriate Workflow execution timeouts for children. - **Use typed stubs.** Prefer typed stubs over untyped for compile-time safety. - **Monitor child executions.** Track Child Workflow IDs for observability and debugging. ## Common pitfalls - **Treating Child Workflows like Activities.** Child Workflows are for orchestration, not for executing external code. If you only need to call an API or run a function, use an Activity instead. - **Spawning unbounded children in a loop.** Starting thousands of Child Workflows without batching can overwhelm the Temporal Service and bloat the parent's event history. Use fixed-size batches or a sliding window. - **Ignoring the Parent Close Policy.** The default policy is TERMINATE, which kills children when the parent closes. If children must outlive the parent, set the policy to ABANDON explicitly. - **Using synchronous calls when async is needed.** Calling a Child Workflow synchronously blocks the parent until the child completes. For long-running children, use the async API (`Async.function()` in Java, `startChild()` in TypeScript, `start_child_workflow()` in Python, or collect Futures without calling `.Get()` in Go) to avoid stalling the parent. - **Omitting Workflow IDs.** Without explicit Workflow IDs, you lose the ability to deduplicate or look up Child Workflows by a meaningful identifier. Generate deterministic IDs based on business keys. - **Not handling child failures.** Child Workflow failures propagate to the parent as a Child Workflow Failure (`ChildWorkflowFailure` in TypeScript and Java, `ChildWorkflowError` in Python, `ChildWorkflowExecutionError` in Go), with the underlying cause in its `cause` field. If you do not catch and handle them, the parent Workflow fails as well. ## Related ### Patterns - **[Parallel Execution](/design-patterns/parallel-execution)**: Running multiple children concurrently. - **[Continue-As-New](/design-patterns/continue-as-new)**: Child Workflows can use Continue-As-New independently. - **[Saga Pattern](/design-patterns/saga-pattern)**: Children as compensatable transactions. ### Sample code **Java:** - [HelloChild](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloChild.java) — Basic synchronous Child Workflow. - [Async Child Workflow](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/asyncchild) — Asynchronous child with ABANDON policy. - [Async Untyped Child](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/asyncuntypedchild) — Untyped async Child Workflow. **TypeScript:** - [Child Workflows](https://github.com/temporalio/samples-typescript/tree/main/child-workflows) — Parent and child Workflow using `executeChild` and `startChild`. **Python:** - [Child Workflows](https://github.com/temporalio/samples-python/tree/main/hello/hello_child_workflow.py) — Basic Child Workflow using `execute_child_workflow`. **Go:** - [Child Workflow](https://github.com/temporalio/samples-go/tree/main/child-workflow) — Synchronous and async Child Workflow patterns. --- # Continue-As-New Pattern Source: https://docs.temporal.io/design-patterns/continue-as-new > Prevents unbounded history growth by completing the current execution and starting a new one with fresh history. ## Overview The Continue-As-New pattern allows long-running Workflows to reset their event history by completing the current execution and immediately starting a new one with fresh state. This prevents Workflows from hitting Temporal's event history limits while maintaining logical continuity, making it essential for periodic tasks, infinite loops, and Workflows that process unbounded data streams. By archiving old event history and starting fresh, Continue-As-New also reduces active storage costs — only the current execution's history remains in active storage while previous runs are moved to cheaper archived storage. ## Problem In long-running Workflows, you often need to execute periodic tasks indefinitely, process unbounded streams of data without accumulating history, implement infinite loops that run for months or years, avoid hitting the 50,000 event history limit, and maintain Workflow state across logical restarts. Without Continue-As-New, you must manually stop and restart Workflows (losing continuity), risk hitting history limits and Workflow failures, implement external orchestration to manage Workflow lifecycle, and accept degraded performance as history grows large. ## Solution Continue-As-New completes the current Workflow execution and atomically starts a new one with the same Workflow ID. The new execution begins with a fresh event history while preserving logical continuity. You pass state as arguments to the new execution. ```mermaid flowchart LR Start([Start]) --> Exec1[Execution 1
Process batch] Exec1 --> Check1{History
large?} Check1 -->|Yes| CAN1[Continue-As-New] Check1 -->|No| More1{More
data?} More1 -->|Yes| Exec1 More1 -->|No| End1([Complete]) CAN1 -.->|Fresh history
Same Workflow ID| Exec2[Execution 2
Process batch] Exec2 --> Check2{History
large?} Check2 -->|Yes| CAN2[Continue-As-New] Check2 -->|No| More2{More
data?} More2 -->|Yes| Exec2 More2 -->|No| End2([Complete]) CAN2 -.-> Etc[...] classDef highlight stroke-width:1px class CAN1,CAN2 highlight ``` The following describes each step in the diagram: 1. The Workflow starts Execution 1 and processes a batch of data. 2. After each batch, the Workflow checks whether the history is getting large. 3. If the history is large, the Workflow calls Continue-As-New, which starts Execution 2 with a fresh history and the same Workflow ID. 4. If the history is not large and more data remains, the Workflow loops and processes the next batch. 5. If no more data remains, the Workflow completes normally. The following implementation shows a data processor that passes a cursor and a running total across executions. When the batch is full (indicating more data to process), the Workflow calls Continue-As-New with the updated state: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import fetchBatch, process BATCH_SIZE = 100 @workflow.defn class DataProcessorWorkflow: @workflow.run async def run(self, cursor: str, total_processed: int = 0) -> None: batch = await workflow.execute_activity( fetchBatch, args=[cursor, BATCH_SIZE], start_to_close_timeout=timedelta(seconds=60), ) for record in batch: await workflow.execute_activity( process, record, start_to_close_timeout=timedelta(seconds=60), ) total_processed += 1 cursor = record.id if len(batch) == BATCH_SIZE: # More data to process - continue as new with updated state workflow.continue_as_new(args=[cursor, total_processed]) # Otherwise complete normally ``` **Go** ```go // data_processor_workflow.go const BatchSize = 100 func DataProcessorWorkflow(ctx workflow.Context, cursor string, totalProcessed int) error { ao := workflow.ActivityOptions{StartToCloseTimeout: time.Minute} ctx = workflow.WithActivityOptions(ctx, ao) var batch []Record err := workflow.ExecuteActivity(ctx, FetchBatch, cursor, BatchSize).Get(ctx, &batch) if err != nil { return err } for _, record := range batch { err = workflow.ExecuteActivity(ctx, Process, record).Get(ctx, nil) if err != nil { return err } totalProcessed++ cursor = record.ID } if len(batch) == BatchSize { // More data to process - continue as new with updated state return workflow.NewContinueAsNewError(ctx, DataProcessorWorkflow, cursor, totalProcessed) } // Otherwise complete normally return nil } ``` **Java** ```java // DataProcessorWorkflowImpl.java @WorkflowInterface public interface DataProcessorWorkflow { @WorkflowMethod void processData(String cursor, int totalProcessed); } public class DataProcessorWorkflowImpl implements DataProcessorWorkflow { private static final int BATCH_SIZE = 100; @Override public void processData(String cursor, int totalProcessed) { List batch = activities.fetchBatch(cursor, BATCH_SIZE); for (Record record : batch) { activities.process(record); totalProcessed++; cursor = record.getId(); } if (batch.size() == BATCH_SIZE) { // More data to process - continue as new with updated state DataProcessorWorkflow continueAsNew = Workflow.newContinueAsNewStub(DataProcessorWorkflow.class); continueAsNew.processData(cursor, totalProcessed); } // Otherwise complete normally } } ``` **TypeScript** ```typescript // workflows.ts import { continueAsNew, proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { fetchBatch, process } = proxyActivities({ startToCloseTimeout: '1 minute', }); const BATCH_SIZE = 100; export async function dataProcessorWorkflow( cursor: string, totalProcessed: number = 0 ): Promise { const batch = await fetchBatch(cursor, BATCH_SIZE); for (const record of batch) { await process(record); totalProcessed++; cursor = record.id; } if (batch.length === BATCH_SIZE) { // More data to process - continue as new with updated state await continueAsNew(cursor, totalProcessed); } // Otherwise complete normally } ``` The Workflow fetches a batch of records, processes each one, and updates the cursor. If the batch is full, the Workflow triggers Continue-As-New with the updated cursor and total. In Java, `Workflow.newContinueAsNewStub()` creates a typed stub. In TypeScript, `continueAsNew()` throws a special error that the runtime intercepts. In Python, `workflow.continue_as_new()` immediately stops the current execution and starts a new one. In Go, `workflow.NewContinueAsNewError()` returns a special error that signals the runtime to continue as new. If the batch is smaller than `BATCH_SIZE`, no more data remains and the Workflow completes. ## Implementation ### Use the Continue-As-New suggestion Instead of tracking iteration counts manually, you can use the SDK's built-in suggestion to let Temporal tell you when the history is getting large: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow @workflow.defn class DataProcessorWorkflow: @workflow.run async def run(self, cursor: str, total_processed: int = 0) -> None: batch = await workflow.execute_activity( fetchBatch, args=[cursor, BATCH_SIZE], start_to_close_timeout=timedelta(seconds=60), ) for record in batch: await workflow.execute_activity( process, record, start_to_close_timeout=timedelta(seconds=60), ) total_processed += 1 cursor = record.id # Check if history is getting large if workflow.info().is_continue_as_new_suggested(): workflow.continue_as_new(args=[cursor, total_processed]) # Continue processing or complete ``` **Go** ```go // data_processor_workflow.go func DataProcessorWorkflow(ctx workflow.Context, cursor string, totalProcessed int) error { ao := workflow.ActivityOptions{StartToCloseTimeout: time.Minute} ctx = workflow.WithActivityOptions(ctx, ao) var batch []Record err := workflow.ExecuteActivity(ctx, FetchBatch, cursor, BatchSize).Get(ctx, &batch) if err != nil { return err } for _, record := range batch { err = workflow.ExecuteActivity(ctx, Process, record).Get(ctx, nil) if err != nil { return err } totalProcessed++ cursor = record.ID // Check if history is getting large if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { return workflow.NewContinueAsNewError(ctx, DataProcessorWorkflow, cursor, totalProcessed) } } // Continue processing or complete return nil } ``` **Java** ```java // DataProcessorWorkflowImpl.java public class DataProcessorWorkflowImpl implements DataProcessorWorkflow { @Override public void processData(String cursor, int totalProcessed) { List batch = activities.fetchBatch(cursor, BATCH_SIZE); for (Record record : batch) { activities.process(record); totalProcessed++; cursor = record.getId(); // Check if history is getting large if (Workflow.getInfo().isContinueAsNewSuggested()) { DataProcessorWorkflow continueAsNew = Workflow.newContinueAsNewStub(DataProcessorWorkflow.class); continueAsNew.processData(cursor, totalProcessed); return; } } // Continue processing or complete } } ``` **TypeScript** ```typescript // workflows.ts import { continueAsNew, workflowInfo, proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { fetchBatch, process } = proxyActivities({ startToCloseTimeout: '1 minute', }); export async function dataProcessorWorkflow( cursor: string, totalProcessed: number = 0 ): Promise { const batch = await fetchBatch(cursor, BATCH_SIZE); for (const record of batch) { await process(record); totalProcessed++; cursor = record.id; // Check if history is getting large if (workflowInfo().continueAsNewSuggested) { await continueAsNew(cursor, totalProcessed); } } // Continue processing or complete } ``` Each SDK provides a method to check if the history is approaching the limit: - Java: `Workflow.getInfo().isContinueAsNewSuggested()` - TypeScript: `workflowInfo().continueAsNewSuggested` - Python: `workflow.info().is_continue_as_new_suggested()` - Go: `workflow.GetInfo(ctx).GetContinueAsNewSuggested()` This approach is more reliable than a fixed iteration count because it accounts for the actual number of events generated per iteration. ## When to use The Continue-As-New pattern is a good fit for periodic Workflows running indefinitely (cron-like behavior), processing unbounded data streams, long-running Workflows with repetitive patterns, Workflows that accumulate state over many iterations, and preventing event history from growing too large. It is not a good fit for short-lived Workflows (under 1,000 events), Workflows that naturally complete, one-time batch processing, or Workflows that require full history for audit purposes. ## Benefits and trade-offs Continue-As-New allows you to run Workflows indefinitely without history limits. Fresh history keeps Workflow execution fast. It reduces active storage costs by archiving old event history — more aggressive iteration limits mean more frequent archiving, keeping active storage minimal. The transition is atomic with no gap between old and new execution. You pass state as arguments to the new execution, and the Workflow ID remains the same, maintaining logical continuity for Queries and Signals. The trade-offs to consider are that previous execution history is archived separately. You must explicitly pass state as arguments (manual state management). Queries only see the current execution's state. Debugging requires tracing across multiple execution runs. You cannot undo Continue-As-New once triggered. ## Comparison with alternatives | Approach | History reset | State continuity | Use case | | :--- | :--- | :--- | :--- | | Continue-As-New | Yes | Manual | Long-running periodic | | Child Workflows | Per child | Automatic | Parallel processing | | Cron Schedule | Yes | None | Fixed schedule tasks | | Manual Restart | Yes | None | One-time Workflows | ## Best practices - **Use the continue-as-new suggestion.** Check the SDK's built-in suggestion (`isContinueAsNewSuggested()` in Java, `continueAsNewSuggested` in TypeScript, `is_continue_as_new_suggested()` in Python, `GetContinueAsNewSuggested()` in Go) to automatically detect when history is large. - **Set aggressive iteration limits.** Continue as new every 100–1000 iterations to prevent history buildup and reduce storage costs. Balance frequency with the overhead of creating new executions. - **Pass minimal state.** Only pass necessary state to keep arguments small. - **Add exit Signals.** Allow graceful termination via Signals. - **Log transitions.** Log when continuing as new for observability. - **Version carefully.** Ensure new code can handle state from old executions. - **Monitor history size.** Track event count and continue before hitting limits. - **Use typed APIs.** In Java, prefer `newContinueAsNewStub()` over untyped `continueAsNew()`. In TypeScript, use the generic `continueAsNew()` for type safety. - **Consider cron.** For fixed Schedules, use Temporal Schedules instead. - **Test state transfer.** Verify state correctly passes between executions. ## Common pitfalls - **Passing too much state.** Continue-As-New arguments are serialized into the first event of the new execution. Large payloads slow down startup and increase storage costs. Pass only the minimal state needed. - **Forgetting to drain Signals before continuing.** Any Signals received but not yet processed are lost when Continue-As-New starts a fresh execution. Drain your Signal channel and carry pending Signals forward as arguments. - **Using a fixed iteration count instead of the built-in suggestion.** Different Workflow paths generate different numbers of events per iteration. A fixed count may continue too early or too late. Use the SDK's built-in continue-as-new suggestion for accurate detection. - **Not versioning state arguments.** When you change the Workflow method signature or state shape, in-flight executions may continue as new into code that cannot deserialize the old arguments. Use versioning or backward-compatible argument types. - **Calling Continue-As-New from a Signal handler.** Triggering Continue-As-New inside a Signal handler can cause Signal loss because the handler may preempt other pending Signals. Always set a flag in the Signal handler and call Continue-As-New from the main Workflow thread, where all Signal handlers are guaranteed to have run first. - **Not accounting for Child Workflows.** Continue-As-New closes the current Workflow Execution, which triggers the Parent Close Policy on all Child Workflows. By default, children are terminated. If children must survive, set `ParentClosePolicy` to `ABANDON` and pass their Workflow IDs to the new execution so you can interact with them via external handles. - **Caching Run IDs for external interaction.** Continue-As-New creates a new Run ID. If external callers cache the old Run ID for Signals or Queries, they will get a "workflow execution already completed" error. Always use Workflow ID without a Run ID (or an empty Run ID) so the request routes to the currently running execution. - **Catching the Continue-As-New exception.** In TypeScript and Python, Continue-As-New is implemented by throwing a special exception. Wrapping it in a try-catch or try-except can suppress the transition and cause unexpected behavior. Let the exception propagate unhandled. In Go, return the `ContinueAsNewError` from the Workflow function without wrapping it. ## Related ### Patterns - **[Entity Workflow](/design-patterns/entity-workflow)**: Long-lived Workflows that model business entities, relying on Continue-As-New to prevent unbounded history. - **[Child Workflows](/design-patterns/child-workflows)**: Decomposing work into sub-Workflows. Consider Parent Close Policy when combining with Continue-As-New. - **[Signal with Start](/design-patterns/signal-with-start)**: Idempotent Workflow start with an initial Signal — use Workflow ID without Run ID to interact with continued executions. ### Guides - [Track customer loyalty points with durable Workflows](/guides/entity-pattern-loyalty-points): Uses Continue-As-New to keep a long-lived customer loyalty account within Event History limits, including an upgrade path at the Continue-As-New boundary for Worker Versioning. - [Player Sessions That Survive Anything](/guides/durable-gaming-sessions): Uses Continue-As-New so a multiplayer game session can run for weeks without unbounded history growth. ### Sample code **Java:** - [Cron Workflow](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java) — Periodic Workflow using Continue-As-New. - [Heartbeating Activity Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/heartbeatingactivity) — Batch processing with Continue-As-New for large datasets. **TypeScript:** - [Continue-As-New](https://github.com/temporalio/samples-typescript/tree/main/continue-as-new) — Basic Continue-As-New with `continueAsNew()`. - [Safe Message Handlers](https://github.com/temporalio/samples-typescript/tree/main/message-passing/safe-message-handlers) — Entity Workflow with Continue-As-New and `continueAsNewSuggested`. **Python:** - [Safe Message Handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Entity Workflow with Continue-As-New and `is_continue_as_new_suggested()`. **Go:** - [Safe Message Handlers](https://github.com/temporalio/samples-go/tree/main/safe_message_handler) — Entity Workflow with Continue-As-New and `GetContinueAsNewSuggested()`. ### References - [Continue-As-New](/workflow-execution/continue-as-new) — Concept reference for the Continue-As-New mechanism. - [Temporal Blog: How to Keep a Workflow Running Indefinitely Long](https://temporal.io/blog/very-long-running-workflows) — Detailed guidance on managing Workflows that run forever. - [Temporal Blog: Workflows as Actors](https://temporal.io/blog/workflows-as-actors-is-it-really-possible) — Using Continue-As-New with the Entity Workflow / Actor pattern. --- # Delayed Callback (Webhooks) Source: https://docs.temporal.io/design-patterns/delayed-callback > Integrates webhooks durably: receive inbound webhooks via Signals, fire delayed outbound callbacks with durable timers, and complete Activities asynchronously via task tokens. ## Overview Delayed Callback patterns in Temporal manage delayed completion notification between two systems. They use durable timers, and you define the behavior entirely in code. ### Webhooks You build and configure [Webhooks](https://www.redhat.com/en/topics/automation/what-is-a-webhook) with the Delayed Callback patterns. There are two recommended patterns for integrating Temporal Workflows to communicate through HTTP callbacks: - waiting and receiving inbound webhooks - firing delayed outbound callbacks Included is a pattern for completing activities asynchronously via a callback token and some guidance on when to use it. Temporal lets you build durable, observable webhook-based integrations without ad hoc queues, cron jobs, or fragile state machines. ## Problem Waiting for another system without Durable Execution is hard. You must implement your own: - durable timers (for example, with a cron per timer) - retry queues - state stores - reconciliation jobs All to support a delayed callback. Webhook-based integrations have several failure modes that are difficult to handle without durable infrastructure: - An inbound message fires before the target system is ready or at the proper state, causing the message to be ignored or lost. - An outbound HTTP callback fails midway through a multi-step cross-system process, and there is no record of what was sent, retried, or skipped. - An external job (payment processor, ML pipeline, etc.) completes and calls back, but the in-process state that was waiting for the callback has been lost to poor state management or an application restart. - A delayed callback is scheduled via a cron job or message queue, but the scheduling system and the application process have no shared recovery mechanism. ## Solution Temporal solves these problems with [durable timers](/workflow-execution/timers-delays) in a Workflow. You use [Signals or Updates](/encyclopedia/workflow-message-passing) to send events to a Workflow. All of these are Temporal code — no extra infrastructure to deploy or manage. - **Pattern 1 — Inbound Callback:** Route the incoming HTTP request to a Temporal Signal-with-Start. The Workflow is the durable recipient; if it is not running yet, Temporal creates it and delivers the Signal atomically. - **Pattern 2 — Delayed Outbound Callbacks:** Use a durable `workflow.sleep()` to set the proper delay before executing the outbound HTTP activity. The sleep timer survives worker and server restarts; the activity retries automatically on failure. - **Pattern 3 — Async Activity Completion:** The activity records a task token before returning, and your callback endpoint uses that token to complete the activity from the outside. The Workflow resumes with the result as if the activity had returned normally. ### Pattern 1 — inbound webhooks ```mermaid sequenceDiagram participant E as External Service participant A as Your API Handler participant T as Temporal Cluster participant W as Workflow E->>A: POST /webhook (payload) A->>T: signal_with_start(workflow_id,
"payment_received", payload) Note over T: Atomic: start if not running, then signal T->>W: Start workflow (if not running)
+ deliver signal W->>W: Wake up, process payload W->>W: Execute follow-on activities ``` The following describes each step in the diagram: 1. An external service sends an HTTP POST to your API handler — this is the inbound webhook. 2. Your handler calls `signal_with_start` on the Temporal client with the Workflow ID and payload. The handler can return an HTTP 200 immediately after this call; Temporal takes responsibility for delivery. 3. Temporal atomically starts the Workflow if it is not already running, then delivers the Signal — no race condition between "start" and "signal." 4. The Workflow wakes up exactly where it was blocked waiting (or begins execution if newly created) and processes the payload. ### Pattern 2 — delayed outbound callbacks ```mermaid sequenceDiagram participant C as Client participant W as Workflow participant T as Temporal Cluster participant E as External Service C->>T: Start DelayedCallbackWorkflow
(url, data, delay) T->>W: Schedule first task W->>T: workflow.sleep(delay) Note over T: Durable timer (survives restarts) T->>W: Timer fires after delay W->>W: execute_activity(send_callback, url, data) W->>E: POST callback_url (data) E-->>W: HTTP 200 W-->>C: Workflow complete ``` The following describes each step in the diagram: 1. The client starts the Workflow with a target URL, payload, and delay duration. 2. The Workflow calls `workflow.sleep()`. This stores a durable timer in the Temporal cluster — not in process memory. 3. If any worker restarts during the delay, the timer continues. When it fires, Temporal schedules the next Workflow Task on a healthy worker. 4. The Workflow executes an activity that performs the outbound HTTP POST. If the POST fails, Temporal retries it with the configured retry policy. ### Pattern 3 — async Activity Completion ```mermaid sequenceDiagram participant W as Workflow participant A as Activity participant E as External Service participant CB as Your Callback API W->>A: execute_activity(submit_job) A->>E: Submit job, record task_token A-->>W: Return (workflow paused waiting) Note over W: Waiting for external completion... E->>CB: POST /callback (result) CB->>W: complete_async_activity
(task_token, result) W->>W: Resume with result ``` The following describes each step in the diagram: 1. The Workflow executes an activity that submits a job to an external system. 2. The activity records its task token (an opaque handle Temporal provides) alongside the submitted job ID — for example, in a database row. 3. The activity returns without waiting; the Workflow is now paused waiting for the activity to complete externally. 4. When the external system finishes, it calls your callback endpoint with the result. 5. Your callback handler retrieves the task token from the database and completes the activity through the Temporal client using that token. The Workflow resumes immediately with the result. ## Implementation ### Pattern 1 — inbound webhooks via Signal-With-Start The following examples show an `OrderWorkflow` that waits for a `payment_received` Signal. The starter uses Signal-with-Start to atomically create the Workflow and deliver the payment signal in one call — exactly what your HTTP handler would do on a real POST. **Python** ```python # workflows.py import asyncio from dataclasses import dataclass from datetime import timedelta from typing import Optional from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import process_payment from shared import OrderInput, PaymentPayload @workflow.defn class OrderWorkflow: def __init__(self) -> None: self._payment: Optional[PaymentPayload] = None @workflow.run async def run(self, order: OrderInput) -> str: workflow.logger.info(f"Order {order.order_id}: waiting for payment webhook") # Block until the inbound webhook signal arrives (or timeout after 24 hours) await workflow.wait_condition( lambda: self._payment is not None, timeout=timedelta(hours=24), ) if self._payment is None: return f"Order {order.order_id}: timed out waiting for payment" result = await workflow.execute_activity( process_payment, self._payment, start_to_close_timeout=timedelta(seconds=30), ) return result @workflow.signal async def payment_received(self, payload: PaymentPayload) -> None: workflow.logger.info(f"Payment signal received: {payload.payment_id}") self._payment = payload ``` **Java** ```java // OrderWorkflow.java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.SignalMethod; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; import java.time.Duration; @WorkflowInterface public interface OrderWorkflow { @WorkflowMethod String run(Shared.OrderInput order); @SignalMethod void paymentReceived(Shared.PaymentPayload payload); final class Impl implements OrderWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); private Shared.PaymentPayload payment = null; @Override public String run(Shared.OrderInput order) { System.out.println("Order " + order.orderId() + ": waiting for payment webhook"); // Block until the inbound webhook signal arrives (or timeout after 24 hours) boolean received = Workflow.await(Duration.ofHours(24), () -> payment != null); if (!received) { return "Order " + order.orderId() + ": timed out waiting for payment"; } return activities.processPayment(payment); } @Override public void paymentReceived(Shared.PaymentPayload payload) { System.out.println("Payment signal received: " + payload.paymentId()); this.payment = payload; } } } ``` **Go** ```go // workflows.go package main import ( "time" "go.temporal.io/sdk/workflow" ) func OrderWorkflow(ctx workflow.Context, order OrderInput) (string, error) { workflow.GetLogger(ctx).Info("Order waiting for payment webhook", "order_id", order.OrderID) var payment *PaymentPayload // Block until the inbound webhook signal arrives (or timeout after 24 hours) selector := workflow.NewSelector(ctx) timerFired := false timerCtx, cancelTimer := workflow.WithCancel(ctx) timer := workflow.NewTimer(timerCtx, 24*time.Hour) signalCh := workflow.GetSignalChannel(ctx, SignalName) selector.AddReceive(signalCh, func(ch workflow.ReceiveChannel, more bool) { ch.Receive(ctx, &payment) cancelTimer() }) selector.AddFuture(timer, func(f workflow.Future) { if err := f.Get(ctx, nil); err == nil { timerFired = true } }) selector.Select(ctx) if timerFired || payment == nil { return "Order " + order.OrderID + ": timed out waiting for payment", nil } ao := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, }) var result string err := workflow.ExecuteActivity(ao, ProcessPayment, payment).Get(ao, &result) return result, err } ``` The Starter uses Signal-with-Start to atomically create the Workflow (if needed) and deliver the simulated webhook payload: **Python** ```python # starter.py import asyncio import time from temporalio.client import Client from shared import TASK_QUEUE, OrderInput, PaymentPayload from workflows import OrderWorkflow async def main() -> None: client = await Client.connect("localhost:7233") order_id = f"order-{int(time.time() * 1000)}" order = OrderInput(order_id=order_id, amount=99.99) payment = PaymentPayload(payment_id=f"pay-{int(time.time() * 1000)}", amount=99.99) print(f"Sending webhook for order {order_id}") # Signal-with-Start: atomically starts the workflow (if not running) and # delivers the payment signal — this is exactly what your HTTP handler would do. handle = await client.start_workflow( OrderWorkflow.run, order, id=f"order-{order_id}", task_queue=TASK_QUEUE, start_signal="payment_received", start_signal_args=[payment], ) print(f"Webhook signal sent: {payment.payment_id}") result = await handle.result() print(f"Order completed: {result}") if __name__ == "__main__": asyncio.run(main()) ``` **Java** ```java // Starter.java import io.temporal.client.BatchRequest; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowOptions; import io.temporal.client.WorkflowStub; import io.temporal.serviceclient.WorkflowServiceStubs; public class Starter { public static void main(String[] args) throws Exception { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); String orderId = "order-" + System.currentTimeMillis(); Shared.PaymentPayload payment = new Shared.PaymentPayload( "pay-" + System.currentTimeMillis(), 99.99); OrderWorkflow workflow = client.newWorkflowStub( OrderWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(Shared.TASK_QUEUE) .setWorkflowId("order-" + orderId) .build()); System.out.println("Sending webhook for order " + orderId); // Signal-with-Start: atomically starts the workflow (if not running) and // delivers the payment signal — this is exactly what your HTTP handler would do. BatchRequest request = client.newSignalWithStartRequest(); request.add(workflow::run, new Shared.OrderInput(orderId, 99.99)); request.add(workflow::paymentReceived, payment); client.signalWithStart(request); System.out.println("Webhook signal sent: " + payment.paymentId()); // Wait for the workflow to complete String result = WorkflowStub.fromTyped(workflow).getResult(String.class); System.out.println("Order completed: " + result); System.exit(0); } } ``` **Go** ```go // starter.go package main import ( "context" "fmt" "log" "time" "go.temporal.io/sdk/client" ) func main() { c, err := client.Dial(client.Options{HostPort: "localhost:7233"}) if err != nil { log.Fatalln("Unable to create client:", err) } defer c.Close() ctx := context.Background() orderID := fmt.Sprintf("order-%d", time.Now().UnixMilli()) workflowID := "order-" + orderID order := OrderInput{OrderID: orderID, Amount: 99.99} payment := PaymentPayload{ PaymentID: fmt.Sprintf("pay-%d", time.Now().UnixMilli()), Amount: 99.99, } fmt.Printf("Sending webhook for order %s\n", orderID) // Signal-with-Start: atomically starts the workflow (if not running) and // delivers the payment signal — this is exactly what your HTTP handler would do. we, err := c.SignalWithStartWorkflow( ctx, workflowID, SignalName, payment, client.StartWorkflowOptions{ ID: workflowID, TaskQueue: TaskQueue, }, OrderWorkflow, order, ) if err != nil { log.Fatalln("SignalWithStart failed:", err) } fmt.Printf("Webhook signal sent: %s\n", payment.PaymentID) var result string if err := we.Get(ctx, &result); err != nil { log.Fatalln("Workflow result failed:", err) } fmt.Printf("Order completed: %s\n", result) } ``` ### Pattern 2 — delayed outbound callbacks Use a durable `workflow.sleep()` before the outbound activity. The timer is stored in the Temporal cluster, not in process memory — it survives any number of worker restarts. **Python** ```python # delayed_callback_workflow.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import send_webhook_callback from shared import CallbackInput @workflow.defn class DelayedCallbackWorkflow: @workflow.run async def run(self, input: CallbackInput) -> str: workflow.logger.info( f"Sleeping {input.delay_seconds}s before calling {input.callback_url}" ) # Durable sleep — survives worker restarts, server restarts, everything await workflow.sleep(timedelta(seconds=input.delay_seconds)) # Fire the outbound callback; Temporal retries on HTTP failure result = await workflow.execute_activity( send_webhook_callback, input, start_to_close_timeout=timedelta(minutes=5), ) workflow.logger.info(f"Callback delivered to {input.callback_url}") return result ``` **Java** ```java // DelayedCallbackWorkflow.java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; import java.time.Duration; @WorkflowInterface public interface DelayedCallbackWorkflow { @WorkflowMethod void run(Shared.CallbackInput input); final class Impl implements DelayedCallbackWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(5)) .build()); @Override public void run(Shared.CallbackInput input) { System.out.println("Sleeping " + input.delaySeconds() + "s before calling " + input.callbackUrl()); // Durable sleep — survives worker restarts, server restarts, everything Workflow.sleep(Duration.ofSeconds(input.delaySeconds())); // Fire the outbound callback; Temporal retries on HTTP failure activities.sendWebhookCallback(input); System.out.println("Callback delivered to " + input.callbackUrl()); } } } ``` **Go** ```go // delayed_callback.go (add to workflows.go) package main import ( "time" "go.temporal.io/sdk/workflow" ) func DelayedCallbackWorkflow(ctx workflow.Context, input CallbackInput) error { workflow.GetLogger(ctx).Info("Sleeping before callback", "delay", input.DelaySeconds, "url", input.CallbackURL) // Durable sleep — survives worker restarts, server restarts, everything if err := workflow.Sleep(ctx, time.Duration(input.DelaySeconds)*time.Second); err != nil { return err } // Fire the outbound callback; Temporal retries on HTTP failure ao := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 5 * time.Minute, }) return workflow.ExecuteActivity(ao, SendWebhookCallback, input).Get(ao, nil) } ``` ### Pattern 3 — async Activity Completion The activity records a task token before returning. Your callback endpoint later uses that token to complete the activity and unblock the Workflow. **Python** ```python # async_completion_activities.py import asyncio from temporalio import activity from temporalio.client import Client from shared import JobInput, JobResult @activity.defn async def submit_job(input: JobInput) -> str: """Submit job to external system and return immediately. The activity completes asynchronously when the callback arrives.""" # Get the task token — this is the claim ticket task_token = activity.info().task_token # Submit the job to the external system, persisting the task token # so your callback handler can retrieve it later job_id = await external_service.submit( payload=input.payload, callback_url=f"https://your-api.example.com/callback", task_token_hex=task_token.hex(), # store alongside job_id ) activity.logger.info(f"Job {job_id} submitted; waiting for async callback") # Tell Temporal not to mark the activity complete on return; the external # callback will complete it later using the task token. activity.raise_complete_async() # In your webhook callback handler (e.g., FastAPI route): async def handle_callback(result: str, task_token_hex: str) -> None: token = bytes.fromhex(task_token_hex) client = await Client.connect("localhost:7233") handle = client.get_async_activity_handle(task_token=token) await handle.complete(result) # Workflow resumes with `result` immediately ``` **Java** ```java // AsyncCompletionActivity.java — submit side import io.temporal.activity.Activity; import io.temporal.activity.ActivityExecutionContext; import io.temporal.activity.ActivityInterface; import io.temporal.activity.ActivityMethod; import io.temporal.client.ActivityCompletionClient; @ActivityInterface public interface AsyncJobActivity { @ActivityMethod String submitJob(Shared.JobInput input); } public class AsyncJobActivityImpl implements AsyncJobActivity { private final ActivityCompletionClient completionClient; public AsyncJobActivityImpl(ActivityCompletionClient completionClient) { this.completionClient = completionClient; } @Override public String submitJob(Shared.JobInput input) { ActivityExecutionContext context = Activity.getExecutionContext(); // Get the task token — this is the claim ticket byte[] taskToken = context.getTaskToken(); // Submit to external system, persisting the task token so the callback can retrieve it String jobId = ExternalService.submit(input.payload(), taskToken); System.out.println("Job " + jobId + " submitted; waiting for async callback"); // Tell Temporal not to mark the activity complete yet context.doNotCompleteOnReturn(); return null; // ignored } } // In your callback handler: // completionClient.complete(taskToken, result); ``` **Go** ```go // async_completion.go package main import ( "context" "encoding/hex" "fmt" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/client" ) // SubmitJob submits a job and returns immediately; the activity completes // when the external callback arrives and calls CompleteAsyncActivity. func SubmitJob(ctx context.Context, input JobInput) (string, error) { info := activity.GetInfo(ctx) // Persist the task token so the callback handler can retrieve it by job ID taskToken := info.TaskToken jobID := fmt.Sprintf("job-%d", info.StartedTime.UnixMilli()) if err := persistTaskToken(jobID, hex.EncodeToString(taskToken)); err != nil { return "", err } fmt.Printf("Job %s submitted; waiting for async callback\n", jobID) // Return ErrResultPending to tell Temporal not to mark the activity complete yet return "", activity.ErrResultPending } // CompleteJob is called by your webhook callback handler to unblock the workflow. func CompleteJob(ctx context.Context, c client.Client, jobID string, result string) error { tokenHex, err := loadTaskToken(jobID) if err != nil { return err } taskToken, _ := hex.DecodeString(tokenHex) return c.CompleteActivity(ctx, taskToken, result, nil) } ``` ## When to use | Scenario | Pattern | | :--- | :--- | | External service POSTs a webhook (Workflow may or may not be running) | Signal-with-Start (Pattern 1) | | Fire an outbound HTTP callback after a delay (seconds to years) | `workflow.sleep()` + activity (Pattern 2) | | Submit a job to an external system; wait for its completion webhook | Async activity completion (Pattern 3) | | Poll an external system that does not support webhooks | [Polling External Services](/design-patterns/polling) pattern | **Do not use** Pattern 2 for delays shorter than one second. Timer durations range from one second to several years, and your Workflows should not rely on [sub-second accuracy for Timers](/workflow-execution/timers-delays). ## Benefits and trade-offs **Benefits** - Retries and backoff on outbound HTTP calls come for free via Temporal's retry policy — no custom retry queues needed. - Workflow state survives worker restarts, deploys, and infrastructure failures; durable timers continue without a running process. - Every in-flight delayed callback is visible in the Temporal UI with its scheduled time, payload, and retry count. - Signal-with-Start eliminates the race condition between "does the Workflow exist?" and "deliver the event." - Async activity completion decouples job submission from job completion without polling. **Trade-offs** - Your inbound webhook handler requires a Temporal client; you need the client library in the service receiving webhooks. - Task tokens for async completion must be persisted outside Temporal (for example, in a database); if that store is unavailable the callback cannot complete. - Workflow IDs must be deterministic and stable across webhook deliveries (order ID, user ID, etc.) so that Signal-with-Start routes to the correct instance. ## Comparison with alternatives | Approach | Durability | Retries | Observability | Complexity | | :--- | :--- | :--- | :--- | :--- | | Temporal Signals + Workflows | Durable — survives restarts | Built-in, configurable | Full Temporal UI | Low — primitives compose naturally | | Message queue (SQS, Kafka) | Durable (queue level) | Limited, manual DLQ | Requires external tooling | Medium — must handle ordering, DLQ | | Redis `SET` + cron job | In-memory/volatile | Manual | None | High — cron + polling + error handling | | Direct HTTP retry loops | Process lifetime only | Manual with `time.sleep` | None | High — fragile without process supervisor | ## Best practices - Use stable, business-meaningful Workflow IDs (for example, `order-{order_id}`) so that Signal-with-Start and queries always route to the right Workflow. - Return HTTP 200 from your inbound webhook handler as soon as you have called `signal_with_start`; do not wait for the Workflow to process the payload. - Set a realistic `start_to_close_timeout` on outbound callback activities — long enough for the destination to respond, short enough to surface failures quickly. - For async activity completion, persist the task token in a transactional write alongside the job submission so you never lose the token. - Add a timeout to the `workflow.wait_condition` / `Workflow.await` call in inbound webhook Workflows so they do not wait indefinitely if the webhook is never delivered. - For Pattern 3, use `heartbeat` if the external job takes longer than the activity heartbeat timeout to report back — heartbeating keeps the activity lease alive. ## Common pitfalls - **Sending a plain Signal to a Workflow that does not exist** causes an error. Use Signal-with-Start when the Workflow may not be running. - **Using `time.sleep()` (non-durable)** in Pattern 2 instead of `workflow.sleep()`. A process sleep disappears on restart; only Temporal's timer is durable. - **Non-deterministic Workflow IDs** — generating IDs from timestamps or random values means Signal-with-Start creates a new Workflow on every webhook delivery instead of routing to the existing one. - **Losing the task token** in Pattern 3. If the service storing task tokens is unavailable when the callback arrives, the activity can never complete. Store the token durably (database, not in-process cache). - **Forgetting to signal async completion** in Pattern 3 (`raise_complete_async()` in Python, `ErrResultPending` in Go, `doNotCompleteOnReturn()` in Java). Without this, Temporal marks the activity as completed immediately when the function returns, before the external callback arrives. ## Related ### Patterns - [Signal with Start](/design-patterns/signal-with-start) — deeper coverage of the Signal-with-Start API for entity Workflows - [Approval](/design-patterns/approval) — blocking wait for an external human decision using Signals - [Polling External Services](/design-patterns/polling) — alternative to callbacks when the external system does not support webhooks - [Delayed Start](/design-patterns/delayed-start) — defer Workflow execution to a future time without `workflow.sleep()` - [Long-Running Activity](/design-patterns/long-running-activity) — heartbeating pattern for activities that run for extended periods - **In the future** - Org-to-Org Nexus, stay tuned. --- # Delayed Retry Source: https://docs.temporal.io/design-patterns/delayed-retry > Override the next retry interval for a specific failure using nextRetryDelay on ApplicationFailure. Use when an error carries information about how long to wait before retrying. > **ℹ️ TLDR:** > Throw an `ApplicationFailure` with `nextRetryDelay` set inside the Activity to **delay the next retry for a fixed time.** Use this when an error carries its own timing information — such as an HTTP 429 `Retry-After` header or a known maintenance window — so Temporal waits exactly as long as needed instead of following the generic backoff schedule. ## Overview The Delayed Retry pattern overrides the next retry interval for a specific failure by throwing an `ApplicationFailure` with a `nextRetryDelay` field set from inside the Activity. Use it when a particular error carries information about how long to wait before retrying — such as a rate-limit response with a `Retry-After` header, or a known maintenance window with a fixed end time. ## Problem A `RetryPolicy` applies a single backoff schedule to all failures from an Activity. This works well for generic transient errors, but some errors carry specific information about how long the caller must wait: - An HTTP 429 response includes a `Retry-After: 60` header telling you exactly when the quota resets. - A downstream system returns an error message saying "maintenance until 02:00 UTC" — a precise, known delay. - A database error includes a lock timeout duration that indicates when the resource will be available. With a global `RetryPolicy`, you have two options, neither of which is what you need: set a short interval and retry too early (wasting quota and adding load), or set a long interval and wait longer than necessary. What you need is to set the next retry delay *specific to this failure*, based on the information the error itself provides. ## Solution Throw an `ApplicationFailure` with the `nextRetryDelay` field set from inside the Activity. Temporal replaces the RetryPolicy-calculated interval for that single retry with the value you specify. Subsequent retries (if the next attempt also fails) return to the normal RetryPolicy schedule unless you set `nextRetryDelay` again. ```mermaid sequenceDiagram participant Workflow participant Temporal as Temporal Service participant API as Rate-Limited API Workflow->>Temporal: Schedule activity
(RetryPolicy: initialInterval=1s) Temporal->>+API: Attempt 1 API-->>-Temporal: HTTP 429 — Retry-After: 60s Note over Temporal: Activity throws
ApplicationFailure(nextRetryDelay=60s) Note over Temporal: Override: wait 60s
(ignoring RetryPolicy interval) Temporal->>+API: Attempt 2 API-->>-Temporal: Success Temporal-->>Workflow: Result ``` The following describes each step: 1. The Activity calls the API. It receives an HTTP 429 with a `Retry-After: 60` header. 2. The Activity extracts the retry delay from the response and throws `ApplicationFailure` with `nextRetryDelay=60s`. 3. Temporal ignores the RetryPolicy's calculated interval for this retry and waits exactly 60 seconds instead. 4. The next attempt succeeds and Temporal delivers the result to the Workflow. ## Implementation ### Override the retry delay from the response Extract the wait duration from the error or response and pass it to `ApplicationFailure`. The RetryPolicy's `MaximumAttempts` and `ScheduleToCloseTimeout` still apply — only the interval for the next retry is overridden. **Java** ```java // RateLimitedActivityImpl.java import io.temporal.activity.Activity; import io.temporal.failure.ApplicationFailure; import java.time.Duration; public class RateLimitedActivityImpl implements RateLimitedActivity { @Override public String callApi(String endpoint) { ApiResponse response = httpClient.get(endpoint); if (response.getStatusCode() == 429) { int retryAfterSeconds = response.getHeaderInt("Retry-After", 0); if (retryAfterSeconds > 0) { throw ApplicationFailure.newFailureWithCauseAndDelay( "Rate limited — retrying after " + retryAfterSeconds + "s", "RateLimitError", null, Duration.ofSeconds(retryAfterSeconds) ); } throw ApplicationFailure.newFailure("Rate limited — retrying per RetryPolicy", "RateLimitError"); } return response.getBody(); } } ``` **TypeScript** ```typescript // activities.ts import { ApplicationFailure } from '@temporalio/activity'; export async function callApi(endpoint: string): Promise { const response = await fetch(endpoint); if (response.status === 429) { const retryAfterHeader = response.headers.get('Retry-After'); const retryAfterSeconds = retryAfterHeader != null ? parseInt(retryAfterHeader, 10) : undefined; throw ApplicationFailure.create({ message: retryAfterSeconds != null ? `Rate limited — retrying after ${retryAfterSeconds}s` : 'Rate limited — retrying per RetryPolicy', type: 'RateLimitError', // Only override the interval when the header is present; fall back to RetryPolicy otherwise nextRetryDelay: retryAfterSeconds != null ? `${retryAfterSeconds}s` : undefined, }); } return response.text(); } ``` ### Attempt-proportional delay You can also set the delay dynamically based on the attempt number — for example, to implement a custom backoff that differs from exponential, or to add a known base delay on top of the standard backoff. **Java** ```java // BackoffActivityImpl.java import io.temporal.activity.Activity; import io.temporal.failure.ApplicationFailure; import java.time.Duration; public class BackoffActivityImpl implements BackoffActivity { @Override public String process(String input) { int attempt = Activity.getExecutionContext().getInfo().getAttempt(); try { return downstreamService.call(input); } catch (ServiceUnavailableException e) { // Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …) throw ApplicationFailure.newFailureWithCauseAndDelay( "Service unavailable on attempt " + attempt, "ServiceUnavailable", e, Duration.ofSeconds(3L * attempt) ); } } } ``` **TypeScript** ```typescript // activities.ts import { ApplicationFailure, activityInfo } from '@temporalio/activity'; export async function process(input: string): Promise { const { attempt } = activityInfo(); try { return await downstreamService.call(input); } catch (e) { // Custom delay: 3 seconds × attempt number (3s, 6s, 9s, …) throw ApplicationFailure.create({ message: `Service unavailable on attempt ${attempt}`, type: 'ServiceUnavailable', cause: e as Error, nextRetryDelay: `${3 * attempt}s`, }); } } ``` ### Workflow configuration The Workflow sets a normal `RetryPolicy`. The `nextRetryDelay` set in the Activity overrides the interval only for the retry following that specific failure — subsequent attempts fall back to the RetryPolicy schedule if `nextRetryDelay` is not set again. **Java** ```java // ApiWorkflowImpl.java public class ApiWorkflowImpl implements ApiWorkflow { private final RateLimitedActivity activities = Workflow.newActivityStub( RateLimitedActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setBackoffCoefficient(2.0) .setMaximumAttempts(10) .build()) .build() ); @Override public String run(String endpoint) { return activities.callApi(endpoint); } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; const { callApi } = wf.proxyActivities({ startToCloseTimeout: '10s', retry: { initialInterval: '1s', backoffCoefficient: 2, maximumAttempts: 10, }, }); export async function apiWorkflow(endpoint: string): Promise { return await callApi(endpoint); } ``` ## Best practices - **Use the error's own delay information when available.** HTTP 429 `Retry-After`, database lock timeouts, and API-provided backoff hints are more accurate than any value you could configure statically. - **Fall back to the RetryPolicy for unknown errors.** Only set `nextRetryDelay` for error types where you have reliable delay information. Let the RetryPolicy handle all other failures normally. - **Still set a meaningful RetryPolicy.** `nextRetryDelay` overrides the interval for a single retry; the RetryPolicy still governs maximum attempts and the intervals for attempts where `nextRetryDelay` is not set. Also ensure `scheduleToCloseTimeout` is long enough to accommodate the maximum possible `nextRetryDelay` value — a tight budget can cause the Activity to expire before the delayed retry executes. - **Surface the delay in the failure message.** Include the delay value and its source in the `ApplicationFailure` message (for example, `"Rate limited — retrying after 60s (Retry-After header)"`) so it appears directly in the Workflow history - Activity failure details. This makes it clear why the Activity waited an unusual amount of time without requiring separate log correlation. ## Common pitfalls - **Assuming `nextRetryDelay` persists across all retries.** It only applies to the immediate next retry. If the following attempt also fails without setting `nextRetryDelay`, the RetryPolicy interval resumes. - **Setting `nextRetryDelay` longer than `ScheduleToCloseTimeout`.** If the override delay exceeds the remaining `ScheduleToCloseTimeout` budget, the retry will never execute — Temporal will expire the Activity before the delay elapses. ## Related ### Patterns - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap total attempts to prevent unbounded retry cost. - [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Enforce a total elapsed time budget across all attempts. - [Fast/Slow Retries](/design-patterns/fast-slow-retries): Shift from a fast retry cadence to a slow one after initial attempts are exhausted. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### References - [Per-error next Retry delay](/encyclopedia/retry-policies#per-error-next-retry-delay) --- # Delayed Start Pattern Source: https://docs.temporal.io/design-patterns/delayed-start > Creates Workflows immediately but defers execution until a specified delay expires. Fits one-time scheduled operations and grace periods. ## Overview The Delayed Start pattern enables Workflows to be created immediately but begin execution after a specified delay. The Workflow execution is registered in Temporal right away, but the first Workflow Task is scheduled to run only after the delay period expires, making it suitable for scheduled operations, grace periods, and deferred processing. ## Problem In business processes, you often need Workflows that start execution at a future time, are created immediately for tracking but execute later, avoid external scheduling systems or cron jobs for one-time delays, and maintain Workflow identity and queryability before execution begins. Without delayed start, you must use external schedulers to trigger Workflow creation later, start Workflows immediately and sleep as the first operation (which wastes resources), implement complex queueing systems for deferred execution, or use Temporal Schedules for one-time delays (which is more than you need). ## Solution The Delayed Start uses a start delay option in WorkflowOptions to defer the first Workflow Task. The Workflow execution is created immediately with a `firstWorkflowTaskBackoff` set to the delay duration, but no Workflow code runs until the delay expires. ```mermaid sequenceDiagram participant Client participant Temporal participant Workflow Client->>Temporal: Start with setStartDelay(30s) Temporal->>Temporal: Create execution Note over Temporal: Execution visible
but not running Temporal-->>Client: Workflow ID Note over Temporal: Delay period (30s)... opt During delay Client->>Temporal: Signal-With-Start Note over Temporal: Bypasses remaining delay end Note over Temporal: Delay expires (if not bypassed) Temporal->>+Workflow: Schedule first task Workflow->>Workflow: Execute Workflow-->>-Temporal: Complete ``` The following describes each step in the diagram: 1. The client starts the Workflow with a 30-second delay. Temporal creates the execution immediately. 2. The execution is visible and queryable, but no Workflow code runs during the delay. 3. If the client sends a Signal-With-Start or Update-With-Start during the delay, the remaining delay is bypassed and a Workflow Task is dispatched immediately. Regular Signals do not interrupt the delay. 4. After the delay expires, Temporal schedules the first Workflow Task and the Workflow begins execution. The following example creates a Workflow with a 30-second start delay: **Python** ```python # client.py from datetime import timedelta handle = await client.start_workflow( DelayedStartWorkflow.run, id=WORKFLOW_ID, task_queue=TASK_QUEUE, start_delay=timedelta(seconds=30), ) # Created now, executes in 30 seconds ``` **Go** ```go // starter/main.go workflowOptions := client.StartWorkflowOptions{ ID: WorkflowID, TaskQueue: TaskQueue, StartDelay: 30 * time.Second, } we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, DelayedStartWorkflow) // Created now, executes in 30 seconds ``` **Java** ```java // Client.java DelayedStartWorkflow workflow = client.newWorkflowStub( DelayedStartWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId(WORKFLOW_ID) .setTaskQueue(TASK_QUEUE) .setStartDelay(Duration.ofSeconds(30)) .build()); workflow.start(); // Created now, executes in 30 seconds ``` **TypeScript** ```typescript // client.ts const handle = await client.workflow.start(delayedStartWorkflow, { workflowId: WORKFLOW_ID, taskQueue: TASK_QUEUE, startDelay: '30 seconds', }); // Created now, executes in 30 seconds ``` The start delay option sets the `firstWorkflowTaskBackoff` on the execution. The Workflow is created and visible in the UI immediately, but the Worker does not receive a Task until the delay expires. ## Implementation ### Basic delayed notification The following implementation sends a notification after a one-hour delay. The Workflow code runs only after the delay expires: **Python** ```python # workflows.py from temporalio import workflow @workflow.defn class NotificationWorkflow: @workflow.run async def run(self, message: str) -> None: workflow.logger.info(f"Sending notification: {message}") # client.py from datetime import timedelta handle = await client.start_workflow( NotificationWorkflow.run, "Your trial expires soon", task_queue=TASK_QUEUE, start_delay=timedelta(hours=1), ) ``` **Go** ```go // workflow.go func NotificationWorkflow(ctx workflow.Context, message string) error { logger := workflow.GetLogger(ctx) logger.Info("Sending notification: " + message) return nil } // starter/main.go workflowOptions := client.StartWorkflowOptions{ TaskQueue: TaskQueue, StartDelay: 1 * time.Hour, } we, err := c.ExecuteWorkflow( context.Background(), workflowOptions, NotificationWorkflow, "Your trial expires soon", ) ``` **Java** ```java // NotificationWorkflowImpl.java @WorkflowInterface public interface NotificationWorkflow { @WorkflowMethod void sendNotification(String message); } public class NotificationWorkflowImpl implements NotificationWorkflow { @Override public void sendNotification(String message) { Workflow.getLogger(NotificationWorkflowImpl.class) .info("Sending notification: " + message); } } // Client.java NotificationWorkflow workflow = client.newWorkflowStub( NotificationWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(TASK_QUEUE) .setStartDelay(Duration.ofHours(1)) .build()); workflow.sendNotification("Your trial expires soon"); ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; export async function notificationWorkflow(message: string): Promise { wf.log.info(`Sending notification: ${message}`); } // client.ts const handle = await client.workflow.start(notificationWorkflow, { args: ['Your trial expires soon'], taskQueue: TASK_QUEUE, startDelay: '1 hour', }); ``` The Workflow is created immediately, but the notification logic does not execute until one hour later. ### Cancellable delayed execution The following implementation adds Signal handlers for cancellation and a Query for status. You can cancel the Workflow before it runs or check its status during the delay: **Python** ```python # workflows.py from temporalio import workflow @workflow.defn class DelayedOrderWorkflow: def __init__(self) -> None: self._cancelled = False self._status = "SCHEDULED" @workflow.run async def run(self, order_id: str) -> None: if self._cancelled: self._status = "CANCELLED" return self._status = "PROCESSING" # Process order logic self._status = "COMPLETED" @workflow.signal async def cancel(self) -> None: self._cancelled = True @workflow.query def get_status(self) -> str: return self._status ``` **Go** ```go // workflow.go func DelayedOrderWorkflow(ctx workflow.Context, orderID string) error { logger := workflow.GetLogger(ctx) cancelled := false status := "SCHEDULED" // Register Signal handler for cancellation cancelCh := workflow.GetSignalChannel(ctx, "cancel") // Drain any pending signals without blocking for { var signal interface{} ok := cancelCh.ReceiveAsync(&signal) if !ok { break } cancelled = true } // Register Query handler for status err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) { return status, nil }) if err != nil { return err } if cancelled { logger.Info("Order cancelled before processing", "orderId", orderID) return nil } status = "PROCESSING" // Process order logic status = "COMPLETED" return nil } ``` **Java** ```java // DelayedOrderWorkflowImpl.java @WorkflowInterface public interface DelayedOrderWorkflow { @WorkflowMethod void processOrder(String orderId); @SignalMethod void cancel(); @QueryMethod String getStatus(); } public class DelayedOrderWorkflowImpl implements DelayedOrderWorkflow { private boolean cancelled = false; private String status = "SCHEDULED"; @Override public void processOrder(String orderId) { if (cancelled) { status = "CANCELLED"; return; } status = "PROCESSING"; // Process order logic status = "COMPLETED"; } @Override public void cancel() { cancelled = true; } @Override public String getStatus() { return status; } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; const cancelSignal = wf.defineSignal('cancel'); const getStatusQuery = wf.defineQuery('getStatus'); export async function delayedOrderWorkflow(orderId: string): Promise { let cancelled = false; let status = 'SCHEDULED'; wf.setHandler(cancelSignal, () => { cancelled = true; }); wf.setHandler(getStatusQuery, () => status); if (cancelled) { status = 'CANCELLED'; return; } status = 'PROCESSING'; // Process order logic status = 'COMPLETED'; } ``` The `cancel` Signal handler sets a flag that the Workflow checks when it starts executing. Note that Signal handlers and Query handlers only run after the delay expires and the first Workflow Task is dispatched. To cancel before execution, use `Signal-With-Start` to bypass the delay, or cancel the Workflow Execution directly. ## When to use The Delayed Start pattern is a good fit for scheduled one-time operations (send a reminder in 24 hours), grace periods before processing (cancel a subscription in 7 days), delayed notifications and alerts, deferred batch processing, and trial expiration Workflows. It is not a good fit for recurring Schedules (use Temporal Schedules), immediate execution with internal delays (use Workflow sleep — `Workflow.sleep()` in Java, `wf.sleep()` in TypeScript, `workflow.sleep()` in Python, `workflow.Sleep()` in Go), complex scheduling logic (use Schedules with cron), or sub-second delays (minimal benefit). ## Benefits and trade-offs The Workflow is queryable before execution starts (immediate visibility). No Worker resources are consumed during the delay. You can cancel the Workflow Execution before it runs. A Signal-With-Start or Update-With-Start bypasses the remaining delay. Regular Signals sent during the delay do not interrupt it. The API is a single configuration option with no external schedulers needed. The delay is managed by Temporal, ensuring deterministic behavior. The trade-offs to consider are that you cannot dynamically adjust the delay after creation (use the Updatable Timer pattern for that). The pattern is for one-time delays only — for recurring Schedules, use Temporal Schedules. Very short delays (sub-second) provide minimal benefit — Temporal does not guarantee sub-second timer accuracy, and the delay is rounded up to account for scheduling latency. The delay is time-based only, not condition-based. Regular Signals sent during the delay are not delivered until the first Workflow Task fires, so Query and Signal handlers are not available until execution begins. ## Comparison with alternatives | Approach | Immediate visibility | Resource usage | Cancellable | Use case | | :--- | :--- | :--- | :--- | :--- | | Delayed Start | Yes | None during delay | Yes | One-time future execution | | Workflow sleep | Yes | Worker resources | Yes | Internal delays | | Temporal Schedules | Yes | None | Yes | Recurring Schedules | | External Scheduler | No | External system | Depends | Complex scheduling | ## Best practices - **Use for one-time delays.** For recurring Schedules, use Temporal Schedules instead. - **Set Workflow ID.** Always set an explicit Workflow ID for tracking and cancellation. - **Add Query methods.** Expose status via Queries to check state during the delay. - **Enable cancellation.** Add Signal handlers to cancel before execution. - **Validate delay duration.** Ensure the delay is reasonable (not too short or too long). - **Monitor backoff.** Check `firstWorkflowTaskBackoff` in history for verification. - **Consider time zones.** Use absolute timestamps if the delay depends on a specific time. - **Document behavior.** Clearly indicate that the Workflow does not execute immediately. ## Common pitfalls - **Using Signals during the delay.** Regular Signals do not interrupt the Start Delay. Only Signal-With-Start or Update-With-Start bypass the delay. Signals sent to a delayed Workflow are buffered but the Workflow code has not started, so there is no handler to process them until the delay expires. - **Querying before the Workflow starts.** Queries have no state to return during the delay because no Workflow code has executed yet. Clients may receive errors or empty results. - **Relying on sub-second delays.** Temporal does not guarantee sub-second timer accuracy, and the delay is rounded up due to scheduling latency. Treat the configured duration as a minimum, not an exact value. - **Forgetting that the Workflow ID is reserved.** A delayed Workflow reserves its Workflow ID immediately. Starting another Workflow with the same ID will fail depending on the ID reuse policy. ## Related ### Patterns - **Temporal Schedules**: For recurring Workflow execution. - **[Updatable Timer](/design-patterns/updatable-timer)**: For dynamically adjustable delays within Workflows. - **[Signal with Start](/design-patterns/signal-with-start)**: Interacting with Workflows before execution. ### Sample code - [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloDelayedStart.java) — Delayed start with `setStartDelay()`. - [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/start-delay) — Delayed start with `startDelay` option. - [Python Sample](https://github.com/temporalio/samples-python/tree/main/start_delay) — Delayed start with `start_delay` parameter. - [Go Sample](https://github.com/temporalio/samples-go/tree/main/start-delay) — Delayed start with `StartDelay` option. --- # Distributed Transaction Patterns Source: https://docs.temporal.io/design-patterns/distributed-transaction-patterns > Pattern selection guide for distributed transactions, with a decision tree for choosing between Saga and Early Return. Distributed transactions span multiple services that each own their own data, with no shared database transaction to roll back. These patterns coordinate the steps, undo completed work when a later step fails, and keep external side effects correct under retries. ## Patterns in this section - [Saga Pattern](/design-patterns/saga-pattern): Manages a distributed transaction as a sequence of local steps, where each step defines a compensating action that undoes its effect if a later step fails. - [Early Return](/design-patterns/early-return): Returns a result to the caller as soon as initialization succeeds, while the remaining work continues asynchronously in the background. ## Choosing a pattern **You need to undo completed steps when a later step fails**: use the [Saga Pattern](/design-patterns/saga-pattern) and define a compensation for every step that has an external effect. **You need to respond to the caller before the transaction finishes**: use [Early Return](/design-patterns/early-return) to acknowledge after initialization and continue processing in the background. ## Related sections - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — control how each step retries before a compensation triggers - [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Update-with-Start mechanism behind Early Return - [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — combine Early Return with Local Activities for the lowest first-response latency --- # Downstream Rate Limiting Source: https://docs.temporal.io/design-patterns/downstream-rate-limiting > Caps Activity execution rate against a downstream service by routing throttled Activities to a dedicated Task Queue backed by Workers configured with a throughput limit. > **ℹ️ TLDR:** > Used to **rate limit outbound requests to a downstream service**. Use this to limit the rate of requests, such as to a third-party API, or payment processor, or other external system, that concurrent Workflows would otherwise exceed. ## Overview The Downstream Rate Limiting pattern, also known as Task Queue rate limiting, caps how many Activities execute per second against a downstream service. You place throttled Activities on a dedicated Task Queue backed by Workers configured with `MaxTaskQueueActivitiesPerSecond`. The Temporal matching service enforces this limit before dispatching tasks, so the downstream service receives a controlled request rate regardless of how many Worker instances or Workflow executions are running concurrently. ## Problem Many downstream systems — LLM providers, payment processors, third-party REST APIs — enforce requests-per-second limits. Some systems cannot handle more than a defined level of requests per second. When many Temporal Workflows schedule Activities concurrently, the resulting burst can saturate those limits, causing request failures, cascading retries, and increased latency for all callers. Without centralized throttling, each Activity implementation must manage backpressure independently, which scatters policy across the codebase and provides no enforcement at the Temporal scheduling layer. ## Solution You assign rate-limited Activities to a dedicated Task Queue and Worker set and configure the Workers on that queue with a throughput cap. Because the limit applies to the Task Queue, it is enforced before any Worker executes an Activity, and it holds across all Worker replicas without coordination. The Workflow routes the throttled Activity to the dedicated queue by specifying an explicit `task_queue` override in the Activity options. ```mermaid flowchart LR subgraph Workflows WA[Workflow A] WB[Workflow B] WC[Workflow C] end subgraph Temporal Server TQ["rate-limited-tq\n─────────────\ntask 1\ntask 2\ntask 3\ntask 4\n…"] end subgraph Workers WK1["Worker 1\nMaxTaskQueueActivitiesPerSecond\n= 5 RPS"] WK2["Worker 2\nMaxTaskQueueActivitiesPerSecond\n= 5 RPS"] end DS["Downstream API\n(rate limit: 5 RPS)"] WA -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ WB -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ WC -->|"schedule callApi\ntask_queue=rate-limited-tq"| TQ TQ -->|"dispatch ≤5/sec across both"| WK1 TQ --> WK2 WK1 -->|"≤5 req/sec combined"| DS WK2 --> DS ``` The following describes each step in the diagram: 1. Any number of Workflows schedule `callApi` Activities to the dedicated `rate-limited-tq` Task Queue via an explicit `task_queue` override in their Activity options. 2. The Temporal server holds tasks in `rate-limited-tq`. The queue depth grows if submission rate exceeds dispatch capacity. 3. Two Workers poll the queue. Each is configured with `MaxTaskQueueActivitiesPerSecond = 5`, so together they dispatch at most 5 Activity tasks per second — matching the downstream API's rate limit. 4. The downstream API receives a steady, controlled request rate regardless of how many Workflows are running concurrently. ## Implementation ### Worker configured with a throughput cap **Python** ```python # worker.py # This is a dedicated worker for rate-limited activities. # You will also need a separate worker registered on your workflow task queue. from temporalio.worker import Worker from activities import call_api async def run_worker(client): worker = Worker( client, task_queue="rate-limited-tq", activities=[call_api], max_task_queue_activities_per_second=5.0, ) await worker.run() ``` **Go** ```go // main.go // This is a dedicated worker for rate-limited activities. // You will also need a separate worker registered on your workflow task queue. w := worker.New(c, "rate-limited-tq", worker.Options{ TaskQueueActivitiesPerSecond: 5.0, }) w.RegisterActivity(CallApi) if err := w.Run(worker.InterruptCh()); err != nil { log.Fatalf("worker error: %v", err) } ``` **Java** ```java // WorkerSetup.java // This is a dedicated worker for rate-limited activities. // You will also need a separate worker registered on your workflow task queue. WorkerOptions rateLimitedOptions = WorkerOptions.newBuilder() .setMaxTaskQueueActivitiesPerSecond(5.0) .build(); Worker rateLimitedWorker = factory.newWorker("rate-limited-tq", rateLimitedOptions); rateLimitedWorker.registerActivitiesImplementations(new RateLimitedActivitiesImpl()); factory.start(); ``` ### Activity Definition **Python** ```python # activities.py from temporalio import activity @activity.defn async def call_api(input: str) -> str: return await downstream_api.call(input) ``` **Go** ```go // activities.go func CallApi(ctx context.Context, input string) (string, error) { return downstreamApi.Call(input) } ``` **Java** ```java // RateLimitedActivities.java @ActivityInterface public interface RateLimitedActivities { @ActivityMethod String callApi(String input); } public class RateLimitedActivitiesImpl implements RateLimitedActivities { @Override public String callApi(String input) { return downstreamApi.call(input); } } ``` ### Workflow routing to the rate-limited queue **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from activities import call_api @workflow.defn class MyWorkflow: @workflow.run async def run(self, input: str) -> str: return await workflow.execute_activity( call_api, input, task_queue="rate-limited-tq", start_to_close_timeout=timedelta(seconds=30), ) ``` **Go** ```go // workflow.go func MyWorkflow(ctx workflow.Context, input string) (string, error) { ao := workflow.ActivityOptions{ TaskQueue: "rate-limited-tq", StartToCloseTimeout: 30 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, CallApi, input).Get(ctx, &result) return result, err } ``` **Java** ```java // MyWorkflowImpl.java public class MyWorkflowImpl implements MyWorkflow { private final RateLimitedActivities rateLimitedActivities = Workflow.newActivityStub(RateLimitedActivities.class, ActivityOptions.newBuilder() .setTaskQueue("rate-limited-tq") .setStartToCloseTimeout(Duration.ofSeconds(30)) .build() ); @Override public String run(String input) { return rateLimitedActivities.callApi(input); } } ``` ## When to use This pattern is a good fit when your Workflow calls a downstream service with explicit requests-per-second limits, when you need throughput enforcement that holds across many concurrent Workflow instances without per-Activity logic, and when only a subset of Activity types require throttling and others should run without restriction. It is not a good fit when you need concurrency limits rather than throughput limits (see [Priority Task Queues](/design-patterns/priority-task-queues)), when the downstream system has no rate limit and throughput is bounded only by Workflow logic, or when all Activities require the same limit and a single shared queue suffices. ## Benefits and trade-offs Centralizing rate limiting at the Task Queue ensures enforcement even when any number of Workflow instances run in parallel. Because the Temporal server controls dispatch, the limit holds regardless of how many Worker replicas are running — provided you account for Worker count when setting the per-worker cap. Dedicated Task Queues require operating additional Workers. If the throughput cap is set too low relative to demand, the queue depth grows and scheduling latency increases. You must size the Worker pool so that slot availability does not become the bottleneck before the rate limit is reached. ## Comparison with alternatives | Approach | Enforcement point | Works across Workers | Runtime adjustable | Complexity | | :--- | :--- | :--- | :--- | :--- | | `MaxTaskQueueActivitiesPerSecond` | Temporal matching service (server-side) | Yes | No (requires redeploy) | Low | | `MaxWorkerActivitiesPerSecond` | Worker SDK poller (worker-side) | No — per-worker only | No (requires redeploy) | Low | | Concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkflowTaskExecutionSize`, `MaxConcurrentLocalActivityExecutionSize`) | Worker executor | No — per-worker only | No (requires redeploy) | Low | | Sleep-based throttle in Workflow | Workflow scheduler | No | Via signal | Low | | Client-side token bucket in Activity | Activity execution | Per-worker only | No | Medium | | API gateway rate limiting | Network layer | Yes | Yes | High | Three distinct layers of worker-side control exist alongside the server-side queue limit. `MaxWorkerActivitiesPerSecond` instructs the SDK to self-throttle its polling — the Worker will not request a new Activity task if doing so would push it over this rate. Because the limit is per-process, multiple Workers on the same queue each apply it independently, so the effective queue throughput is the per-worker cap multiplied by Worker count. By contrast, `MaxTaskQueueActivitiesPerSecond` is a server-side instruction: the Temporal matching service slows dispatch for the entire queue regardless of how many Workers are polling, making it the correct tool for protecting a shared downstream service. The concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkflowTaskExecutionSize`, `MaxConcurrentLocalActivityExecutionSize`) are not throughput limits but define the number of execution slots available on a Worker. A Worker will not accept more tasks than it has open slots, so a low slot count acts as an indirect throughput ceiling. ## Best practices - **Use a separate Task Queue for each rate limit.** `MaxTaskQueueActivitiesPerSecond` applies to every Activity on the queue. Mixing rate-limited and unrestricted Activities on the same queue will throttle the unrestricted ones too. - **Run at least two Worker processes per queue for availability.** A single Worker process is a single point of failure. Because `MaxTaskQueueActivitiesPerSecond` is a server-side per-queue limit rather than a per-worker one, set the same value on every Worker that polls the queue. Set each Worker to the target RPS — for example, 5 on each of two Workers yields a combined queue limit of 5, not 10. If Workers report different values, the server applies the value from the last Worker that polled. - **Monitor queue depth and schedule latency.** Track the `temporal_activity_schedule_to_start_latency` metric on the rate-limited queue; sustained growth signals that demand consistently exceeds the configured cap. You can also query the Task Queue's `ApproximateBacklogCount` via the `DescribeTaskQueue` API — a steadily growing backlog count is a direct indicator that the configured RPS cap is too low for the current submission rate. ## Common pitfalls - **Forgetting to override the task queue in Activity options.** If the Workflow does not explicitly specify `task_queue` in the Activity options, the Activity runs on the Workflow's default queue and bypasses the rate-limited Worker entirely. - **Setting conflicting MaxTaskQueueActivitiesPerSecond limits in workers.** This setting is set in Workers and sent to the Task Queue when a Worker polls. If you have multiple Workers with conflicting settings, the Workers will overwrite each other as they poll. - **Confusing throughput limits with concurrency limits.** `MaxTaskQueueActivitiesPerSecond` controls starts per second; `MaxConcurrentActivityExecutionSize` controls simultaneous executions. Long-running Activities that hold slots for minutes may exhaust concurrency before the RPS cap applies. - **Setting the cap far below actual demand.** A cap much lower than actual submission rate causes the queue to grow unboundedly. Monitor queue depth and raise the cap or add more Workers when throughput requirements grow. - **Expecting a perfectly even per-second rate.** The limit is enforced across the queue's partitions, default four. The server maintains the configured rate as an average over time but can dispatch a short burst above it, up to roughly the rate divided across partitions. If the downstream service rejects any momentary overshoot, set the cap below the hard limit to leave headroom, or reduce the partition count for the queue. ## Related ### Patterns - **[Priority Task Queues](/design-patterns/priority-task-queues)**: Route work to separate queues by urgency, with different concurrency budgets per tier. - **[Fairness](/design-patterns/fairness)**: Give each tenant an equal throughput share when multiple tenants share capacity. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. ### Guides - [Rate-limit downstream APIs with separate Task Queues](/guides/rate-limit-downstream-apis): A Python walkthrough covering multiple rate-limited APIs (SendGrid, Stripe, OpenAI), 429 handling with `Retry-After`, and backlog draining strategies. ### References - **Python** — [`max_task_queue_activities_per_second`](https://python.temporal.io/temporalio.worker.WorkerConfig.html#max_task_queue_activities_per_second) on [`Worker`](https://python.temporal.io/temporalio.worker.Worker.html) - **Go** — [`TaskQueueActivitiesPerSecond`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) in [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/worker#Options) - **Java** — [`setMaxTaskQueueActivitiesPerSecond`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html) on `WorkerOptions.Builder` - **Temporal Community** — [Rate limit configuration and best practices](https://community.temporal.io/t/rate-limit-configuration-and-best-practices/5498) --- # Eager Workflow Start Source: https://docs.temporal.io/design-patterns/eager-workflow-start > Dispatch the first Workflow Task directly to a co-located Worker, bypassing the Temporal Matching Service. Requires the starter and Worker to share the same process and client connection. > **ℹ️ TLDR:** > **Bypass the Temporal Matching Service by dispatching the first Workflow Task directly to a co-located Worker.** The Worker and the client that starts the Workflow must share the same process and server connection. Eager Workflow Start eliminates the Matching Service round-trip, saving approximately 30–50 ms per Workflow start. When combined with Local Activities, this pattern achieves ~265 ms total-workflow latency (vs ~850 ms baseline). The TypeScript SDK does not support Eager Workflow Start. ## Overview When you call `ExecuteWorkflow`, the Temporal server normally stores the new Workflow execution, then routes the first Workflow Task through its Matching Service to an available Worker. **Eager Workflow Start** short-circuits this routing: the server returns the first Workflow Task inline in the `StartWorkflowExecution` response, and the co-located Worker processes it immediately—without a separate polling round-trip. ```mermaid sequenceDiagram participant CW as Client + Worker (same process) participant S as Temporal Server rect rgb(230, 235, 250) Note over CW,S: Normal Workflow Start CW->>S: StartWorkflowExecution S->>S: Matching Service routes
task to available Worker S-->>CW: WorkflowTask dispatched via polling CW->>CW: Execute WorkflowTask CW->>S: Complete WorkflowTask end rect rgb(220, 245, 225) Note over CW,S: Eager Workflow Start (co-located Worker) CW->>S: StartWorkflowExecution (EnableEagerStart=true) S-->>CW: WorkflowTask returned inline
— no Matching step CW->>CW: Execute WorkflowTask immediately CW->>S: Complete WorkflowTask end ``` **Numbered walkthrough:** 1. In a normal start, the server queues the Workflow execution and the Matching Service waits for an available Worker slot. The Worker polls, picks up the task, runs it, and reports back—adding an extra server round-trip. 2. With Eager Workflow Start enabled, the server detects that the requesting client has a co-located Worker with an available slot. Instead of queuing the task, the server attaches the first Workflow Task to the `StartWorkflowExecution` response. 3. The Worker processes the Workflow Task immediately upon receiving the response. No separate poll is required. 4. If the server cannot fulfill the eager request (for example, no local slot is available), it falls back silently to normal dispatch. Your code does not need to handle this case explicitly. ## Problem Even with Local Activities eliminating per-Activity server round-trips, the Workflow's first Workflow Task still requires a scheduling round-trip through the Temporal Matching Service. This adds latency that is unavoidable in a distributed deployment where Workers are separate from the caller. For applications where the starter and Worker share the same deployment unit—such as a request-handling service that also runs Workers—this Matching overhead can be eliminated. ## Solution Start a Worker in the same process as the workflow starter, using the same client connection. Set `EnableEagerStart: true` (Go), `setDisableEagerExecution(false)` (Java), or `request_eager_start=True` (Python) on the `StartWorkflowOptions`. The SDK signals to the server that a local Worker is available, and the server returns the first Workflow Task inline. > **⚠️ Feature flag for self-hosted Temporal:** > On self-hosted Temporal Server, Eager Workflow Start may require enabling a dynamic config flag: > > ``` > --dynamic-config-value system.enableEagerWorkflowStart=true > ``` > > Temporal Cloud and recent versions of the open-source server may enable this by default. Check your server's release notes or documentation to confirm. **Python** ```python # starter.py — starts the Worker in the same process, then executes the Workflow eagerly import asyncio from temporalio.client import Client from temporalio.worker import Worker from workflows import TransactionWorkflow from activities import validate_transaction, settle_transaction from shared import TASK_QUEUE, TransactionRequest async def main(): client = await Client.connect("localhost:7233") # The Worker must share this process and client for eager dispatch to work. async with Worker( client, task_queue=TASK_QUEUE, workflows=[TransactionWorkflow], activities=[validate_transaction, settle_transaction], ): result = await client.execute_workflow( TransactionWorkflow.run, TransactionRequest(amount=100.00, currency="USD"), id="eager-workflow-start-demo", task_queue=TASK_QUEUE, request_eager_start=True, # Dispatch first WorkflowTask inline ) print(f"Transaction complete: ID={result.id} Status={result.status}") if __name__ == "__main__": asyncio.run(main()) ``` **Go** ```go // starter.go — starts the Worker in the same process, then executes the Workflow eagerly func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create Temporal client:", err) } defer c.Close() // Start the Worker non-blocking — it must share this process and client. w := worker.New(c, TaskQueue, worker.Options{}) w.RegisterWorkflow(TransactionWorkflow) w.RegisterActivity(ValidateTransaction) w.RegisterActivity(SettleTransaction) if err := w.Start(); err != nil { log.Fatalln("Unable to start worker:", err) } defer w.Stop() run, err := c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{ ID: "eager-workflow-start-demo", TaskQueue: TaskQueue, EnableEagerStart: true, // Dispatch first WorkflowTask inline }, TransactionWorkflow, TransactionRequest{Amount: 100.00, Currency: "USD"}) if err != nil { log.Fatalln("Failed to start workflow:", err) } var result Transaction if err := run.Get(context.Background(), &result); err != nil { log.Fatalln("Workflow failed:", err) } fmt.Printf("Transaction complete: ID=%s Status=%s\n", result.ID, result.Status) } ``` **Java** ```java // Starter.java — starts the Worker in the same process, then executes the Workflow eagerly public class Starter { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); // The Worker must share this process and client for eager dispatch. WorkerFactory factory = WorkerFactory.newInstance(client); io.temporal.worker.Worker worker = factory.newWorker(Shared.TASK_QUEUE); worker.registerWorkflowImplementationTypes(TransactionWorkflow.Impl.class); worker.registerActivitiesImplementations(new Activities.Impl()); factory.start(); TransactionWorkflow workflow = client.newWorkflowStub( TransactionWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(Shared.TASK_QUEUE) .setWorkflowId("eager-workflow-start-demo") .setDisableEagerExecution(false) // false = enable eager dispatch .build() ); Shared.Transaction result = workflow.processTransaction( new Shared.TransactionRequest(100.00, "USD")); System.out.printf("Transaction complete: ID=%s Status=%s%n", result.id(), result.status()); factory.shutdown(); } } ``` > **ℹ️ TypeScript SDK:** > The TypeScript SDK does not currently support Eager Workflow Start. Use [Local Activities](/design-patterns/local-activities) or [Early Return + Local Activities](/design-patterns/early-return-local-activities) for latency-sensitive TypeScript workflows. ## When to use **Good fit:** - The workflow starter and Worker run in the same deployment unit (for example, a single service that both handles API requests and runs Workers) - You need the absolute minimum total-workflow latency and are already using Local Activities - The language is Go, Java, or Python **Poor fit:** - Workers are deployed independently from starters (the eager request falls back to normal dispatch, which is harmless but provides no benefit) - You are using the TypeScript SDK - First-response latency matters more than total latency—combine with [Early Return](/design-patterns/early-return) or [Early Return + Local Activities](/design-patterns/early-return-local-activities) for that use case ## Benefits and trade-offs | | Normal Start | Eager Workflow Start | |---|---|---| | Matching Service round-trip | Yes (~30–50 ms) | No (eliminated) | | Worker co-location required | No | Yes (same process + client) | | Fallback behavior | N/A | Graceful fallback to normal dispatch | | TypeScript SDK support | Yes | No | | Configuration required | None | `EnableEagerStart`/`request_eager_start`/`setDisableEagerExecution(false)` | | Self-hosted server flag | N/A | May need `system.enableEagerWorkflowStart=true` | ## Best practices - **Combine with Local Activities.** Eager Workflow Start eliminates the Matching overhead on the first Workflow Task; Local Activities eliminate server round-trips within each Workflow Task. Together they provide the greatest total latency reduction. - **Use a non-blocking Worker start.** Start the Worker before executing the Workflow so it has an available slot. In Go, use `w.Start()` and defer `w.Stop()`. In Python, use `async with Worker(...)`. In Java, call `factory.start()` before creating the workflow stub. - **Do not rely on eager dispatch always firing.** The server falls back to normal dispatch if no local slot is available (for example, the Worker is at capacity). Design the Workflow to work correctly in both cases. - **Share the same client and connection.** The Worker and the workflow starter must use the same `WorkflowClient` instance (Java), `client.Client` (Go), or `Client` (Python). A Worker using a different connection cannot receive eager tasks from another client. - **Be mindful of resource sharing in co-located deployments.** When a Worker runs in the same process as a request handler, they share CPU, memory, and failure domains. A spike in activity execution can slow request handling, and vice versa. Monitor Worker CPU, Workflow Task execution latency, and task queue depth to ensure Worker load does not affect client-facing latency. ## Common pitfalls - **Starting the Worker after `ExecuteWorkflow`.** If the Worker is not registered and running before the eager start call, no local slot exists and the request falls back to normal dispatch. - **Expecting eager dispatch in distributed deployments.** If the process that calls `ExecuteWorkflow` is not the same process running the Worker, eager dispatch will never succeed. The call still works, but it provides no latency benefit. - **Missing the feature flag on self-hosted servers.** If the server dynamic config flag is not set, eager dispatch requests are silently ignored and the execution falls back to normal dispatch. Verify the flag is set if you do not observe the expected latency improvement. - **Using TypeScript.** The TypeScript SDK does not support Eager Workflow Start. Switch to Python, Go, or Java for this optimization. ## Related ### Patterns - [Local Activities](/design-patterns/local-activities) — eliminates per-Activity server round-trips; pairs naturally with Eager Workflow Start - [Early Return + Local Activities](/design-patterns/early-return-local-activities) — minimum first-response latency via Update-with-Start plus Local Activities - [Early Return](/design-patterns/early-return) — returns early to the client via Update-with-Start --- # Early Return (Update with Start) Source: https://docs.temporal.io/design-patterns/early-return > Synchronous initialization with asynchronous completion. Returns results immediately while processing continues in the background. ## Overview The Early Return pattern returns initialization results to the caller immediately while continuing asynchronous processing in the background. ## Problem Clients need immediate feedback on whether an operation can proceed, but the full operation takes significant time to complete. Blocking the client for the entire operation duration creates a poor user experience and ties up resources. ## Solution You use Update-with-Start to split operations into two phases: a fast synchronous initialization phase that validates and returns results immediately, and a slower asynchronous completion phase that runs in the background. The Workflow uses local Activities for quick initialization, Signals completion via Update handlers, then either completes or cancels the operation based on initialization success. ```mermaid sequenceDiagram participant Client participant Workflow participant Activity Client->>+Workflow: Update-with-Start activate Workflow Workflow->>+Activity: Phase 1: Init (fast) Activity-->>-Workflow: Result Workflow-->>Client: Init Result (early return) deactivate Workflow Note over Workflow: Workflow continues executing Workflow->>+Activity: Phase 2: Complete (slow) Activity-->>-Workflow: Done deactivate Workflow ``` The following describes each step in the diagram: 1. The client sends an Update-with-Start request to the Workflow. 2. The Workflow executes a fast initialization Activity (Phase 1) and returns the result to the client immediately. 3. The client receives the initialization result while the Workflow continues executing. 4. The Workflow executes the slower completion Activity (Phase 2) in the background. ## Implementation The following examples show how each SDK implements this pattern. The Workflow registers an Update handler that blocks until initialization completes, then returns the result to the caller. The client receives the initialization result in a single round trip while the Workflow continues processing. **Python** ```python # workflow.py from dataclasses import dataclass from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import init_transaction, complete_transaction, cancel_transaction @dataclass class TransactionRequest: amount: float currency: str @dataclass class Transaction: id: str status: str @workflow.defn class TransactionWorkflow: def __init__(self) -> None: self.tx: Transaction | None = None self.init_done = False self.init_err: Exception | None = None @workflow.run async def run(self, tx_request: TransactionRequest) -> Transaction | None: # Phase 1: Fast synchronous initialization (local activity) try: self.tx = await workflow.execute_local_activity( init_transaction, tx_request, schedule_to_close_timeout=timedelta(seconds=5), ) except Exception as e: self.init_err = e finally: self.init_done = True # Signal update handler # Phase 2: Slow asynchronous completion if self.init_err is not None: await workflow.execute_activity( cancel_transaction, self.tx, start_to_close_timeout=timedelta(seconds=30), ) return None await workflow.execute_activity( complete_transaction, self.tx, start_to_close_timeout=timedelta(seconds=30), ) return self.tx @workflow.update async def return_init_result(self) -> Transaction: await workflow.wait_condition(lambda: self.init_done) if self.init_err is not None: raise self.init_err return self.tx # client.py from temporalio.client import ( Client, WithStartWorkflowOperation, WorkflowUpdateStage, ) from temporalio.common import WorkflowIDConflictPolicy client = await Client.connect("localhost:7233") start_op = WithStartWorkflowOperation( TransactionWorkflow.run, tx_request, id="transaction-123", task_queue="transactions", id_conflict_policy=WorkflowIDConflictPolicy.FAIL, ) update_handle = await client.start_update_with_start_workflow( TransactionWorkflow.return_init_result, wait_for_stage=WorkflowUpdateStage.COMPLETED, start_workflow_operation=start_op, ) # Get initialization result immediately tx = await update_handle.result() # Use transaction ID immediately while workflow continues print(f"Transaction initialized: {tx.id}") ``` **Go** ```go // workflow.go func Workflow(ctx workflow.Context, txRequest TransactionRequest) (*Transaction, error) { var tx *Transaction var initDone bool var initErr error // Register update handler that waits for initialization workflow.SetUpdateHandler(ctx, UpdateName, func(ctx workflow.Context) (*Transaction, error) { workflow.Await(ctx, func() bool { return initDone }) return tx, initErr }, ) // Phase 1: Fast synchronous initialization (local activity) localOpts := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{ ScheduleToCloseTimeout: 5 * time.Second, }) initErr = workflow.ExecuteLocalActivity(localOpts, txRequest.Init).Get(ctx, &tx) initDone = true // Signal update handler // Phase 2: Slow asynchronous completion activityCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, }) if initErr != nil { // Cancel on initialization failure return nil, workflow.ExecuteActivity(activityCtx, CancelTransaction, tx).Get(ctx, nil) } // Complete on initialization success return tx, workflow.ExecuteActivity(activityCtx, CompleteTransaction, tx).Get(ctx, nil) } // client.go startOp := client.NewWithStartWorkflowOperation( client.StartWorkflowOptions{ ID: "transaction-123", TaskQueue: "transactions", WorkflowIDConflictPolicy: enumspb.WORKFLOW_ID_CONFLICT_POLICY_FAIL, }, Workflow, txRequest, ) updateHandle, err := client.UpdateWithStartWorkflow(ctx, client.UpdateWithStartWorkflowOptions{ StartWorkflowOperation: startOp, UpdateOptions: client.UpdateWorkflowOptions{ UpdateName: UpdateName, WaitForStage: client.WorkflowUpdateStageCompleted, }, }, ) // Get initialization result immediately var tx Transaction err = updateHandle.Get(ctx, &tx) if err != nil { return err } // Use transaction ID immediately while workflow continues fmt.Printf("Transaction initialized: %s\n", tx.ID) ``` **Java** ```java // TransactionWorkflowImpl.java public class TransactionWorkflowImpl implements TransactionWorkflow { private boolean initDone = false; private Transaction tx; private Exception initError = null; @Override public TxResult processTransaction(TransactionRequest txRequest) { this.tx = activities.mintTransactionId(txRequest); // Phase 1: Fast synchronous initialization try { this.tx = activities.initTransaction(this.tx); } catch (Exception e) { initError = e; } finally { initDone = true; // Signal update handler } // Phase 2: Slow asynchronous completion if (initError != null) { activities.cancelTransaction(this.tx); return new TxResult("", "Transaction cancelled."); } else { activities.completeTransaction(this.tx); return new TxResult(this.tx.getId(), "Transaction completed successfully."); } } @Override public TxResult returnInitResult() { Workflow.await(() -> initDone); // Wait for initialization if (initError != null) { throw Workflow.wrap(initError); } return new TxResult(tx.getId(), "Initialization successful"); } } // Client.java TransactionWorkflow workflow = client.newWorkflowStub( TransactionWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId("transaction-123") .setTaskQueue("transactions") .setWorkflowIdConflictPolicy( WorkflowIdConflictPolicy.WORKFLOW_ID_CONFLICT_POLICY_FAIL) .build()); WorkflowUpdateHandle updateHandle = WorkflowClient.startUpdateWithStart( workflow::returnInitResult, UpdateOptions.newBuilder().build(), new WithStartWorkflowOperation<>(workflow::processTransaction, txRequest)); // Get initialization result immediately TxResult result = updateHandle.getResultAsync().get(); // Use transaction ID immediately while workflow continues System.out.println("Transaction initialized: " + result.getId()); ``` **TypeScript** ```typescript // workflow.ts import { defineUpdate, setHandler, condition } from '@temporalio/workflow'; import * as activities from './activities'; const { initTransaction, completeTransaction, cancelTransaction } = proxyLocalActivities({ scheduleToCloseTimeout: '5s', }); export const returnInitResultUpdate = defineUpdate('returnInitResult'); export async function transactionWorkflow(txRequest: TransactionRequest): Promise { let tx: Transaction | undefined; let initDone = false; let initError: Error | undefined; // Register update handler that waits for initialization setHandler(returnInitResultUpdate, async () => { await condition(() => initDone); if (initError) { throw initError; } return tx!; }); // Phase 1: Fast synchronous initialization (local activity) try { tx = await initTransaction(txRequest); } catch (err) { initError = err as Error; } finally { initDone = true; // Signal update handler } // Phase 2: Slow asynchronous completion if (initError) { await cancelTransaction(tx!); throw initError; } await completeTransaction(tx); return tx; } // client.ts const startWorkflowOperation = new WithStartWorkflowOperation( transactionWorkflow, { workflowId: 'transaction-123', args: [txRequest], taskQueue: 'transactions', workflowIdConflictPolicy: 'FAIL', }, ); const tx = await client.workflow.executeUpdateWithStart( returnInitResultUpdate, { startWorkflowOperation }, ); const wfHandle = await startWorkflowOperation.workflowHandle(); // Use transaction ID immediately while workflow continues console.log(`Transaction initialized: ${tx.id}`); // Optionally wait for the workflow to complete const finalResult = await wfHandle.result(); ``` The key points across all SDKs are: - **Update-with-Start** is a single API call that starts the Workflow and returns the initialization result. - The Workflow uses an Update handler with a condition/await to block until initialization completes. - The client receives the result immediately while the Workflow continues executing in the background. - A `WorkflowIdConflictPolicy` must be specified. For early return, use `FAIL` to assert a new Workflow is created. - Update-with-Start is **not atomic**. If the Update cannot be delivered (for example, no Worker is available), the Workflow Execution will still start. The SDKs will retry the Update request, but there is no guarantee the Update will succeed. ## When to use The Early Return pattern is a good fit when clients need immediate feedback but operations take time to complete, validation or initialization can be done quickly (under 5 seconds), the operation can be safely cancelled if initialization fails, and the initialization result determines whether to proceed or abort. Common use cases include e-commerce payment processing (immediate authorization while settlement runs in the background), user onboarding and KYC verification (quick user ID return while background checks continue), resource provisioning (fast validation results while infrastructure is set up), document processing (immediate receipt confirmation while OCR and content analysis continue), and order processing (fast inventory check while fulfillment runs in the background). It is not a good fit for fully automated processes that require no intermediate feedback, operations that cannot be split into fast and slow phases, or fire-and-forget operations where no immediate response is needed (use Signals). ## Benefits and trade-offs The Early Return pattern provides immediate client feedback via Update-with-Start in a single round trip. Clients do not wait for full operation completion. Local Activities avoid extra server roundtrips during initialization, and there is a clear separation between validation and execution phases. Automatic cancellation handling runs on initialization failure. The trade-offs to consider are that you must tune timeouts carefully for local Activities. There is a concurrent Update limit (10 per Workflow) that can bottleneck high-throughput scenarios requiring multiple simultaneous Updates. Clients must handle asynchronous completion separately, and initialization must complete within a single Workflow Task. The pattern is limited to operations that you can split into fast and slow phases. ## Comparison with alternatives | Approach | Immediate response | Consistency | Complexity | Use case | | :--- | :--- | :--- | :--- | :--- | | Early Return (Update-with-Start) | Yes (typed) | Strong | Medium | Synchronous init + async completion | | Signal + Query polling | Yes (eventual) | Eventual | High | Fire-and-forget with status checks | | Child Workflow split | Yes (Workflow ID) | Strong | High | Separate init and completion Workflows | | Blocking until completion | Yes (final) | Strong | Low | Short operations only | ## Best practices - **Set WorkflowIdConflictPolicy to FAIL.** For early return, use `FAIL` to assert a new Workflow is created per request. Use `USE_EXISTING` only for lazy initialization patterns. - **Use Workflow.await in the Update handler.** Keep the Update handler lightweight — block on a condition flag (`workflow.Await` in Go, `Workflow.await` in Java, `condition` in TypeScript, `workflow.wait_condition` in Python) and let the main Workflow method do the real work. - **Use local Activities for initialization.** Local Activities avoid extra server roundtrips, keeping the synchronous phase fast (under 5 seconds). - **Handle Update-with-Start non-atomicity.** Update-with-Start is not atomic. The Workflow may start even if the Update fails. Ensure Workers are running and handle the case where the Update is not delivered. - **Set a timeout on the Update result.** Use a timeout when waiting for the Update result to avoid blocking the client indefinitely if the Worker is unavailable. - **Be aware of the concurrent Update limit.** The default `maxInFlightUpdates` is 10 per Workflow. If you expect high concurrency, design accordingly or use separate Workflows. - **Provide a unique Update ID.** Use a unique Update ID for idempotency so retried requests attach to the same Update rather than creating duplicates. - **Avoid Workflow timeouts.** Do not set Workflow Execution timeouts when using early return, as the background phase may take longer than expected. ## Common pitfalls - **Assuming Update-with-Start is atomic.** Unlike Signal-with-Start, Update-with-Start is not atomic. The Workflow may start even if the Update fails (for example, if no Worker is available). Handle this by checking Workflow state after the call. - **Missing WorkflowIdConflictPolicy.** Update-with-Start requires a `WorkflowIdConflictPolicy`. Omitting it causes an error. Use `FAIL` for early return (one Workflow per request) or `USE_EXISTING` for lazy initialization. - **Blocking too long in the Update handler.** The Update handler should return quickly. Perform long-running work in the main Workflow method and use `Workflow.await` in the Update handler to wait for a result. - **Swallowing the ContinueAsNew exception.** In TypeScript, `continueAsNew` throws a special exception. Catching it in a try-catch without re-throwing (or returning in a `finally` block) silently prevents Continue-As-New. ## Related ### Patterns - **[Saga Pattern](/design-patterns/saga-pattern)**: You can combine this with the Saga pattern to add compensation for failed completions. - **[Signal with Start](/design-patterns/signal-with-start)**: For fire-and-forget operations that do not need an immediate response. - **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: For synchronous state modifications on already-running Workflows. ### Sample code - [Python Sample](https://github.com/temporalio/samples-python/tree/main/early_return) — Early return with Update-with-Start. - [Go Sample](https://github.com/temporalio/samples-go/tree/main/early-return) — Early return with Update-with-Start. - [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/early-return) — Early return with local Activities. - [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/earlyreturn) — Early return with Update handler. --- # Early Return + Local Activities Source: https://docs.temporal.io/design-patterns/early-return-local-activities > Extends Early Return by running Phase 1 Activities as Local Activities. The client receives its response after Phase 1 completes entirely in-process, achieving the lowest possible first-response latency. > **ℹ️ TLDR:** > **Combine Update-with-Start with Local Activities in the synchronous Phase 1 to reduce first-response latency from ~265 ms to ~160 ms.** Phase 1 (initialization) runs as Local Activities with no server round-trips. Phase 2 (settlement) runs as regular Activities in the background. The client receives its response as soon as Phase 1 completes, so the in-process execution speed of Local Activities directly improves the time-to-first-byte. ## Overview The [Early Return](/design-patterns/early-return) pattern uses Update-with-Start to send the client an early response after a fast Phase 1 completes, while slow Phase 2 work continues in the background. This pattern extends Early Return by running Phase 1 Activities as **Local Activities**, eliminating all server round-trips on the synchronous hot path. Because the client waits only for Phase 1, and Phase 1 now runs entirely in-process, the end-to-end first-response time drops from approximately 265 ms (Early Return with regular Activities) to approximately 160 ms. ```mermaid sequenceDiagram participant C as Client participant S as Temporal Server participant W as Worker C->>S: UpdateWithStart (start workflow + deliver update) S->>W: Schedule WorkflowTask rect rgb(220, 245, 225) Note over S,W: Phase 1 — Local Activities
(in-process, zero server round-trips) W->>W: validateTransaction (local) W->>W: initTransaction (local) W->>S: Complete WorkflowTask
(Phase 1 done, Update result ready) end S-->>C: Update result returned (~160 ms first response) rect rgb(230, 235, 250) Note over S,W: Phase 2 — Regular Activities (background settlement) W->>S: Schedule completeTransaction S-->>W: Dispatch completeTransaction W->>W: Execute (slow settlement) W->>S: Complete ActivityTask end Note over C,W: Client has its response and Phase 2 continues in the background ``` **Numbered walkthrough:** 1. The client sends a single `UpdateWithStart` RPC, which atomically starts the Workflow and delivers the Update in one server call. 2. The Worker picks up the first Workflow Task and executes Phase 1. Because all Phase 1 Activities are Local Activities, they run in-process with no additional server calls. The entire phase completes inside a single Workflow Task. 3. When the Workflow Task completes, the server marks the Update as fulfilled and returns the result to the waiting client. This happens as soon as Phase 1 finishes—approximately 160 ms after the initial request. 4. The Workflow continues in a new Workflow Task to execute Phase 2 using regular Activities. These run in the background. The client is not blocked by this work. ## Problem The plain Early Return pattern reduces first-response latency significantly compared to waiting for the full Workflow. However, if Phase 1 uses regular Activities, each Activity still incurs server scheduling overhead (~50 ms per call on Temporal Cloud). With two or three Phase 1 Activities, this overhead alone can account for 100–150 ms of the first-response time. ## Solution Run all Phase 1 Activities as Local Activities. They execute in-process within the first Workflow Task, so their results are available to the Update handler as soon as the task completes—with no additional server calls. Phase 2 Activities remain regular Activities, which is acceptable because Phase 2 runs in the background after the client has already received its response. **Python** ```python # workflows.py from temporalio import workflow from datetime import timedelta from activities import validate_transaction, init_transaction, complete_transaction, cancel_transaction LOCAL_TIMEOUT = timedelta(seconds=5) ACTIVITY_TIMEOUT = timedelta(seconds=30) @workflow.defn class TransactionWorkflow: def __init__(self) -> None: self._tx: Transaction | None = None self._phase1_done = False self._phase1_error: Exception | None = None @workflow.update async def get_result(self, req: TransactionRequest) -> Transaction: # Wait for Phase 1 to finish before returning to the caller. await workflow.wait_condition(lambda: self._phase1_done) if self._phase1_error: raise self._phase1_error return self._tx @workflow.run async def run(self, req: TransactionRequest) -> None: try: # Phase 1: Local Activities — zero server round-trips on the hot path. tx = await workflow.execute_local_activity( validate_transaction, req, schedule_to_close_timeout=LOCAL_TIMEOUT, ) self._tx = await workflow.execute_local_activity( init_transaction, tx, schedule_to_close_timeout=LOCAL_TIMEOUT, ) except Exception as e: self._phase1_error = e finally: self._phase1_done = True if self._phase1_error: if self._tx is not None: await workflow.execute_activity( cancel_transaction, self._tx, start_to_close_timeout=ACTIVITY_TIMEOUT, ) return # Phase 2: Regular Activities — background settlement (client already has response). await workflow.execute_activity( complete_transaction, self._tx, start_to_close_timeout=ACTIVITY_TIMEOUT, ) ``` **TypeScript** ```typescript // workflows.ts import { proxyLocalActivities, proxyActivities, defineUpdate, setHandler, condition } from "@temporalio/workflow"; import type * as activities from "./activities"; import type { TransactionRequest, Transaction } from "./shared"; // Phase 1: local activities — no server round-trips on the hot path. const { validateTransaction, initTransaction } = proxyLocalActivities({ scheduleToCloseTimeout: "5s" }); // Phase 2: regular activities — background settlement. const { completeTransaction, cancelTransaction } = proxyActivities({ startToCloseTimeout: "30s" }); export const getResultUpdate = defineUpdate("getResult"); export async function transactionWorkflow(req: TransactionRequest): Promise { let tx: Transaction | undefined; let phase1Done = false; let phase1Error: unknown; setHandler(getResultUpdate, async () => { // The Update handler waits for Phase 1 before returning. await condition(() => phase1Done); if (phase1Error) throw phase1Error; return tx!; }); try { // Phase 1: Local Activities run in-process. tx = await validateTransaction(req); tx = await initTransaction(tx); } catch (err) { phase1Error = err; } finally { phase1Done = true; } if (phase1Error) { if (tx !== undefined) { await cancelTransaction(tx); } return; } // Phase 2: Regular Activity runs in the background. await completeTransaction(tx!); } ``` **Go** ```go // workflows.go func TransactionWorkflow(ctx workflow.Context, req TransactionRequest) error { var tx Transaction var initDone bool var initErr error // Register Update handler — returns to the client as soon as Phase 1 is done. if err := workflow.SetUpdateHandler(ctx, "getResult", func(ctx workflow.Context, r TransactionRequest) (Transaction, error) { _ = workflow.Await(ctx, func() bool { return initDone }) return tx, initErr }); err != nil { return err } // Phase 1: Local Activities — in-process, zero server round-trips. localCtx := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{ ScheduleToCloseTimeout: 5 * time.Second, }) if err := workflow.ExecuteLocalActivity(localCtx, ValidateTransaction, req).Get(localCtx, &tx); err == nil { initErr = workflow.ExecuteLocalActivity(localCtx, InitTransaction, tx).Get(localCtx, &tx) } else { initErr = err } initDone = true activityCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, }) if initErr != nil { // Phase 2 (cancel): regular Activity runs in the background. return workflow.ExecuteActivity(activityCtx, CancelTransaction, tx).Get(activityCtx, nil) } // Phase 2 (complete): regular Activity runs in the background. return workflow.ExecuteActivity(activityCtx, CompleteTransaction, tx).Get(activityCtx, nil) } ``` **Java** ```java // TransactionWorkflow.java public class Impl implements TransactionWorkflow { // Phase 1: local activities — zero server round-trips on the hot path. private final Activities localActivities = Workflow.newLocalActivityStub( Activities.class, LocalActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) .build() ); // Phase 2: regular activities — background settlement. private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build() ); private Shared.Transaction tx; private boolean phase1Done = false; private RuntimeException phase1Error = null; @Override public Shared.Transaction getResult(Shared.TransactionRequest req) { // Update handler: wait for Phase 1 before returning. Workflow.await(() -> phase1Done); if (phase1Error != null) throw phase1Error; return tx; } @Override public void processTransaction(Shared.TransactionRequest req) { try { tx = localActivities.validateTransaction(req); tx = localActivities.initTransaction(tx); } catch (RuntimeException e) { phase1Error = e; } finally { phase1Done = true; } if (phase1Error != null) { activities.cancelTransaction(tx); return; } // Phase 2: regular activity in the background. activities.completeTransaction(tx); } } ``` ## When to use **Good fit:** - User-facing workflows where the first response is more latency-critical than total execution time - Phase 1 consists of short, idempotent validation and initialization steps that fit naturally as Local Activities - Phase 2 is slow (network I/O, external systems) and does not need to be on the client's critical path - You already use or plan to use the Early Return pattern **Poor fit:** - Phase 1 Activities are long-running or require heartbeating—Local Activities cannot heartbeat - The Workflow's total latency matters more than first-response latency - Phase 1 and Phase 2 cannot be cleanly separated ## Benefits and trade-offs | Pattern | First Response | Total Latency | Complexity | |---|---|---|---| | Synchronous workflow | Same as total | ~850 ms | Low | | Early Return (regular activities) | ~265 ms | ~850 ms | Medium | | Local Activities only | Same as total | ~275 ms | Medium | | **Early Return + Local Activities** | **~160 ms** | **~275 ms** | **Medium** | | Eager Workflow Start + Local Activities | ~160 ms | ~265 ms | High | ## Best practices - **Keep Phase 1 Local Activities short.** Each must complete well within the Workflow Task timeout (default 10 seconds). Aim for under 5 seconds total for all Phase 1 work. - **Design Phase 1 for at-least-once execution.** If the Workflow Task that runs Phase 1 fails and retries, all Phase 1 Local Activities re-execute. Phase 1 operations must be idempotent. - **Separate Phase 1 and Phase 2 concerns cleanly.** The Update handler should wait only on the Phase 1 sentinel flag, not on any Phase 2 state. Phase 2 should be independent enough to proceed without client involvement. - **Set appropriate timeouts for Phase 2.** Phase 2 regular Activities run in the background and should have a `startToCloseTimeout` that reflects the maximum acceptable settlement time. ## Common pitfalls - **Putting slow operations in Phase 1.** If any Phase 1 Local Activity takes too long, the Workflow Task times out and retries. The client also waits longer for its early response, defeating the purpose of the pattern. - **Non-idempotent Phase 1.** A retried Workflow Task re-executes all Local Activities in that task. Ensure Phase 1 operations (for example, creating a record in an external system) are safe to re-run. - **Ignoring Phase 1 errors in Phase 2.** Always check Phase 1 error state before proceeding to Phase 2. If Phase 1 failed, Phase 2 should run a compensating Activity (cancel, rollback) rather than complete. - **Mixing Local and regular Activity stubs incorrectly.** In Java, `Workflow.newLocalActivityStub` and `Workflow.newActivityStub` return distinct objects. Make sure Phase 1 uses the local stub and Phase 2 uses the regular stub. ## Related ### Patterns - [Early Return](/design-patterns/early-return) — the baseline Update-with-Start pattern without Local Activity optimization - [Local Activities](/design-patterns/local-activities) — using Local Activities for full-workflow latency reduction without early return - [Eager Workflow Start](/design-patterns/eager-workflow-start) — eliminates the Matching step when starting the Workflow for additional total latency improvement --- # Entity & Lifecycle Patterns Source: https://docs.temporal.io/design-patterns/entity-lifecycle-patterns > Pattern selection guide for modeling long-lived stateful entities and managing Workflow history growth over time. These patterns model long-lived business entities as Workflows and keep those Workflows healthy as they run for days, months, or indefinitely. They cover how an entity holds and mutates state, how you bound Workflow history growth, and how you manage timers that change over time. ## Patterns in this section - [Entity Workflow](/design-patterns/entity-workflow): Models a long-lived business entity as a single Workflow that persists for the entity's entire lifetime, handling every state transition through Signals and Updates. - [Continue-As-New](/design-patterns/continue-as-new): Prevents unbounded history growth by completing the current execution and starting a fresh one that carries forward the current state. - [Updatable Timer](/design-patterns/updatable-timer): Provides a timer you can extend, shorten, or cancel in response to Signals or Updates while the Workflow waits. ## Choosing a pattern **You are modeling something with an ongoing lifecycle** — an account, a device, a subscription: use an [Entity Workflow](/design-patterns/entity-workflow) as the single source of truth for that entity. **Your Workflow runs long enough to grow a large history**: apply [Continue-As-New](/design-patterns/continue-as-new) to reset history while preserving state. **You need a wait that responds to new information**: use an [Updatable Timer](/design-patterns/updatable-timer) instead of a fixed sleep. ## Related sections - [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Signals and Updates that drive entity state transitions - [Task Orchestration Patterns](/design-patterns/task-orchestration-patterns) — decompose a large entity into child Workflows --- # Entity Workflow Pattern Source: https://docs.temporal.io/design-patterns/entity-workflow > Models long-lived business entities as individual Workflows that persist for the entity's entire lifetime, handling all state transitions through Signals and Updates. ## Overview The Entity Workflow pattern models long-lived business entities (users, accounts, devices, orders) as individual Workflows that persist for the entity's entire lifetime — potentially months or years. Each entity gets its own Workflow instance identified by the entity ID, handling all state transitions and operations through Signals and Updates. ## Problem Many business domains have entities that exist for extended periods, undergo multiple state transitions over their lifetime, need to maintain consistent state across operations, require audit trails of all changes, and must handle concurrent operations safely. Traditional approaches struggle with these requirements: - **Database-centric**: Complex locking, race conditions, scattered business logic. - **Event Sourcing**: Requires rebuilding state from events, complex infrastructure. - **Stateless Services**: No built-in consistency, must coordinate state externally. - **Short-lived Workflows**: Do not model the full entity lifecycle. ## Solution You create one Workflow per entity, using the entity ID as the Workflow ID. The Workflow runs for the entity's entire lifetime, maintaining state in Workflow variables and handling operations via Signals and Updates. You use Continue-As-New periodically to prevent unbounded history growth. ```mermaid sequenceDiagram participant Client participant UserWorkflow participant NotificationWorkflow participant Activities Client->>UserWorkflow: Start(userId="user-123") activate UserWorkflow Note over UserWorkflow: State: ACTIVE Client->>UserWorkflow: Update: updateProfile(data) UserWorkflow->>Activities: validateProfile(data) Activities-->>UserWorkflow: valid UserWorkflow->>Activities: updateDatabase(userId, data) Activities-->>UserWorkflow: success Note over UserWorkflow: Profile updated UserWorkflow-->>Client: Success Client->>UserWorkflow: Signal: suspend() Note over UserWorkflow: State: SUSPENDED UserWorkflow->>NotificationWorkflow: Start child workflow activate NotificationWorkflow NotificationWorkflow->>Activities: sendEmail(userId, "suspended") deactivate NotificationWorkflow Client->>UserWorkflow: Update: reactivate() Note over UserWorkflow: State: ACTIVE UserWorkflow-->>Client: Success Note over UserWorkflow: After 1000 operations... UserWorkflow->>UserWorkflow: Continue-As-New Note over UserWorkflow: Fresh history, same state Client->>UserWorkflow: Signal: delete() Note over UserWorkflow: State: DELETED UserWorkflow-->>UserWorkflow: Complete deactivate UserWorkflow ``` The following describes each step in the diagram: 1. The client starts the Workflow with a user ID. The Workflow initializes in the ACTIVE state. 2. The client sends an Update to modify the profile. The Workflow validates the data via an Activity, persists the change, and returns success. 3. The client sends a Signal to suspend the account. The Workflow transitions to SUSPENDED and starts a Child Workflow to send a notification email. 4. The client sends an Update to reactivate the account. The Workflow transitions back to ACTIVE. 5. After 1000 operations, the Workflow calls Continue-As-New to reset its history while preserving state. 6. The client sends a Signal to delete the account. The Workflow transitions to DELETED and completes. ## Implementation The following examples show how each SDK implements the Entity Workflow pattern. Each implementation defines Update handlers for synchronous operations, Signal handlers for asynchronous events, and Query handlers for state inspection. **Python** ```python # workflows.py from dataclasses import dataclass from datetime import datetime, timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import validate_profile @dataclass class UserState: status: str = "ACTIVE" profile: ProfileData | None = None pending_email: str | None = None created_at: datetime | None = None updated_at: datetime | None = None @dataclass class UserAccountInput: user_id: str # Unset for the original caller; carries state across Continue-As-New state: UserState | None = None @workflow.defn class UserAccountWorkflow: def __init__(self) -> None: self.state = UserState(created_at=datetime.utcnow()) self.deleted = False self.operation_count = 0 @workflow.run async def run(self, input: UserAccountInput) -> None: # On Continue-As-New, restore the state carried from the previous run if input.state is not None: self.state = input.state # Block until deleted or Continue-As-New is suggested await workflow.wait_condition( lambda: self.deleted or workflow.info().is_continue_as_new_suggested() ) if not self.deleted and workflow.info().is_continue_as_new_suggested(): await workflow.wait_condition(workflow.all_handlers_finished) # Carry current state forward so it is not reset on the new run workflow.continue_as_new( UserAccountInput(user_id=input.user_id, state=self.state) ) self.state.status = "DELETED" @workflow.update async def update_profile(self, data: ProfileData) -> None: if self.deleted: raise ValueError("User account is deleted") await workflow.execute_activity( validate_profile, data, start_to_close_timeout=timedelta(seconds=30), ) self.state.profile = data self.state.updated_at = datetime.utcnow() self.operation_count += 1 @workflow.update async def suspend(self) -> None: if not self.deleted and self.state.status != "SUSPENDED": self.state.status = "SUSPENDED" self.state.updated_at = datetime.utcnow() self.operation_count += 1 @workflow.update async def reactivate(self) -> None: if not self.deleted and self.state.status == "SUSPENDED": self.state.status = "ACTIVE" self.state.updated_at = datetime.utcnow() self.operation_count += 1 @workflow.signal def delete(self) -> None: self.deleted = True @workflow.query def get_state(self) -> UserState: return self.state ``` **Go** ```go // workflow.go type UserAccountWorkflow struct{} type UserState struct { Status string Profile ProfileData PendingEmail string CreatedAt time.Time UpdatedAt time.Time } type UserAccountInput struct { UserID string // Nil for the original caller; carries state across Continue-As-New State *UserState } func (w *UserAccountWorkflow) Run(ctx workflow.Context, input UserAccountInput) error { // On Continue-As-New, restore the state carried from the previous run var state UserState if input.State != nil { state = *input.State } else { state = UserState{ Status: "ACTIVE", CreatedAt: workflow.Now(ctx), } } deleted := false operationCount := 0 err := workflow.SetUpdateHandler(ctx, "updateProfile", func(ctx workflow.Context, data ProfileData) error { if deleted { return errors.New("user account is deleted") } if err := workflow.ExecuteActivity(ctx, ValidateProfile, data).Get(ctx, nil); err != nil { return err } state.Profile = data state.UpdatedAt = workflow.Now(ctx) operationCount++ return nil }) if err != nil { return err } err = workflow.SetUpdateHandler(ctx, "suspend", func(ctx workflow.Context) error { if !deleted && state.Status != "SUSPENDED" { state.Status = "SUSPENDED" state.UpdatedAt = workflow.Now(ctx) operationCount++ } return nil }) if err != nil { return err } // Block until deleted or Continue-As-New is suggested for { selector := workflow.NewSelector(ctx) selector.AddReceive(workflow.GetSignalChannel(ctx, "delete"), func(c workflow.ReceiveChannel, more bool) { c.Receive(ctx, nil) deleted = true }) selector.Select(ctx) if deleted { state.Status = "DELETED" return nil } if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { // Carry current state forward so it is not reset on the new run return workflow.NewContinueAsNewError(ctx, w.Run, UserAccountInput{UserID: input.UserID, State: &state}) } } } ``` **Java** ```java // UserAccountWorkflow.java // userId is always set; state is empty for the original caller and // carries state across Continue-As-New. public record UserAccountInput(String userId, Optional state) {} @WorkflowInterface public interface UserAccountWorkflow { @WorkflowMethod void run(UserAccountInput input); @UpdateMethod void updateProfile(ProfileData data); @UpdateMethod void changeEmail(String newEmail); @SignalMethod void suspend(); @SignalMethod void reactivate(); @SignalMethod void delete(); @QueryMethod UserState getState(); } public class UserAccountWorkflowImpl implements UserAccountWorkflow { private String userId; private UserState state = new UserState(); private boolean deleted = false; private int operationCount = 0; private static final int CONTINUE_AS_NEW_THRESHOLD = 1000; @Override public void run(UserAccountInput input) { this.userId = input.userId(); // On Continue-As-New, restore the state carried from the previous run if (input.state().isPresent()) { this.state = input.state().get(); } else { state.setStatus("ACTIVE"); state.setCreatedAt(Workflow.currentTimeMillis()); } // Run until deleted or Continue-As-New is needed Workflow.await(() -> deleted || Workflow.getInfo().isContinueAsNewSuggested()); if (!deleted && Workflow.getInfo().isContinueAsNewSuggested()) { // Carry current state forward so it is not reset on the new run Workflow.continueAsNew(new UserAccountInput(userId, Optional.of(state))); } state.setStatus("DELETED"); state.setDeletedAt(Workflow.currentTimeMillis()); } @Override public void updateProfile(ProfileData data) { validateNotDeleted(); Activities.validateProfile(data); state.setProfile(data); state.setUpdatedAt(Workflow.currentTimeMillis()); incrementOperationCount(); } @Override public void changeEmail(String newEmail) { validateNotDeleted(); Activities.sendVerificationEmail(userId, newEmail); state.setPendingEmail(newEmail); state.setUpdatedAt(Workflow.currentTimeMillis()); incrementOperationCount(); } @Override public void suspend() { if (!deleted && !"SUSPENDED".equals(state.getStatus())) { state.setStatus("SUSPENDED"); state.setUpdatedAt(Workflow.currentTimeMillis()); incrementOperationCount(); } } @Override public void reactivate() { if (!deleted && "SUSPENDED".equals(state.getStatus())) { state.setStatus("ACTIVE"); state.setUpdatedAt(Workflow.currentTimeMillis()); incrementOperationCount(); } } @Override public void delete() { deleted = true; } @Override public UserState getState() { return state; } private void validateNotDeleted() { if (deleted) { throw new IllegalStateException("User account is deleted"); } } private void incrementOperationCount() { operationCount++; } } ``` **TypeScript** ```typescript // workflow.ts import { condition, allHandlersFinished, defineUpdate, defineSignal, defineQuery, setHandler, continueAsNew, workflowInfo, proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { validateProfile } = proxyActivities({ startToCloseTimeout: '30s', }); interface UserState { status: string; profile?: ProfileData; pendingEmail?: string; createdAt: number; updatedAt: number; } interface UserAccountInput { userId: string; // Unset for the original caller; carries state across Continue-As-New state?: UserState; } export const updateProfileUpdate = defineUpdate('updateProfile'); export const suspendSignal = defineSignal('suspend'); export const deleteSignal = defineSignal('delete'); export const getStateQuery = defineQuery('getState'); export async function userAccountWorkflow(input: UserAccountInput): Promise { // On Continue-As-New, restore the state carried from the previous run const state: UserState = input.state ?? { status: 'ACTIVE', createdAt: Date.now(), updatedAt: Date.now(), }; let deleted = false; let operationCount = 0; setHandler(updateProfileUpdate, async (data: ProfileData) => { if (deleted) { throw new Error('User account is deleted'); } await validateProfile(data); state.profile = data; state.updatedAt = Date.now(); operationCount++; }); setHandler(suspendSignal, () => { if (!deleted && state.status !== 'SUSPENDED') { state.status = 'SUSPENDED'; state.updatedAt = Date.now(); operationCount++; } }); setHandler(deleteSignal, () => { deleted = true; }); setHandler(getStateQuery, () => state); // Block until deleted or Continue-As-New is suggested await condition(() => deleted || workflowInfo().continueAsNewSuggested); if (!deleted && workflowInfo().continueAsNewSuggested) { await condition(allHandlersFinished); // Carry current state forward so it is not reset on the new run await continueAsNew({ userId: input.userId, state }); } state.status = 'DELETED'; } ``` The Workflow blocks on `Workflow.await(() -> deleted)` (Java), `condition(() => deleted)` (TypeScript), `workflow.wait_condition(lambda: self.deleted)` (Python), or `selector.Select(ctx)` (Go) until the delete Signal arrives. All state transitions happen through Signal and Update handlers, ensuring that every operation on the entity goes through a single Workflow with no race conditions. Continue-As-New is triggered from the main Workflow method (not from handlers) when `isContinueAsNewSuggested()` returns true. The Workflow takes a single input object that carries both the entity ID and the current state, so state is passed forward on Continue-As-New rather than reset. The state field is unset for the original caller and populated only on continuation; the new run restores it before processing further operations. All SDK docs explicitly warn: do not call Continue-As-New from Update or Signal handlers. Instead, handlers set state, and the main Workflow method checks whether to Continue-As-New. ## When to use The Entity Workflow pattern is a good fit for user accounts and profiles, IoT devices and sensors, customer relationships (CRM), shopping carts and orders, financial accounts, subscription management, device provisioning and lifecycle, and multi-tenant resources. It is not a good fit for short-lived processes (use regular Workflows), stateless operations (use Activities), high-frequency updates (more than 100 per second per entity), or entities with only CRUD operations (use a database). ## Benefits and trade-offs Benefits: - All operations on an entity go through a single Workflow, eliminating race conditions. - The Workflow history provides a complete audit trail of all state changes. - All entity logic lives in one place, and state survives process crashes and restarts. - Temporal provides exactly-once execution and automatic retries. - You can inspect current state through Queries without side effects. Trade-offs: - You must use Continue-As-New to prevent unbounded history growth. - A single Workflow handles all operations for one entity, which limits throughput. - State is kept in Workflow memory, so you should use Activities for large data. - One Workflow per entity means you should consider costs at scale. - The first operation after an idle period may have latency. ## Comparison with alternatives | Approach | Consistency | Audit trail | Complexity | Scalability | | :--- | :--- | :--- | :--- | :--- | | Entity Workflow | Strong | Complete | Low | High (per entity) | | Database + Locks | Eventual | Manual | High | Very High | | Event Sourcing | Strong | Complete | High | High | | Stateless Service | Weak | Manual | Medium | Very High | ## Best practices - **Use entity ID as Workflow ID.** This ensures uniqueness and idempotent starts. - **Implement Continue-As-New.** Use `isContinueAsNewSuggested()` to check when to continue. Always call Continue-As-New from the main Workflow method, never from handlers. Wait for all handlers to finish before continuing. - **Validate in Updates.** Use Updates for operations that require validation and a return value. - **Use Signals for events.** Use Signals for asynchronous notifications that do not need responses. - **Keep state minimal.** Store large data externally and reference it in the Workflow. - **Add Queries.** Expose state for monitoring and debugging. - **Handle deletion.** Implement an explicit deletion or decommission Signal. - **Version carefully.** Use Worker versioning for Workflow code changes. - **Set timeouts.** Use Workflow execution timeout as a safety net. - **Monitor history size.** Alert when approaching the Continue-As-New threshold. ## Common pitfalls - **Calling Continue-As-New from Signal or Update handlers.** Continue-As-New must be called from the main Workflow method, never from inside a handler. Calling it from a handler causes non-determinism errors. - **Not waiting for handlers to finish before Continue-As-New.** Use `allHandlersFinished` (TypeScript), `Workflow.isEveryHandlerFinished()` (Java), or `workflow.all_handlers_finished()` (Python) to ensure in-flight handlers complete before transitioning. - **Losing Update ID deduplication across Continue-As-New.** Update IDs are scoped to a single Workflow Execution. After Continue-As-New, the same Update ID can be accepted again. Carry processed IDs in the Continue-As-New input if deduplication is needed. - **Exceeding the 2 MB payload limit on Continue-As-New input.** State passed to Continue-As-New is subject to the same 2 MB blob size limit as Workflow inputs. Use external storage for large state. - **Using a hardcoded counter instead of `isContinueAsNewSuggested`.** The SDK provides `isContinueAsNewSuggested()` which accounts for actual history size. Hardcoded thresholds may be too aggressive or too lenient. ## Related ### Patterns - **[Continue-As-New](/design-patterns/continue-as-new)**: Essential for preventing unbounded history. - **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: Synchronous operations with validation. - **[Signal with Start](/design-patterns/signal-with-start)**: Idempotent Workflow start with an initial Signal. ### Guides - [Track customer loyalty points with durable Workflows](/guides/entity-pattern-loyalty-points): A complete Python implementation of this pattern, including tier calculation, Continue-As-New, and Worker Versioning for accounts that span years. - [Player Sessions That Survive Anything](/guides/durable-gaming-sessions): Extends the Entity Workflow into an Actor Workflow that executes game actions — joining rooms, resolving combat — rather than only holding state. ### Sample code **Python:** - [Safe Message Handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Entity Workflow with Updates, Signals, and Continue-As-New. **Go:** - [Safe Message Handlers](https://github.com/temporalio/samples-go/tree/main/safe_message_handler) — Entity Workflow with Updates, Signals, and Continue-As-New. **Java:** - [Safe Message Handlers](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Entity Workflow with Updates, Signals, and Continue-As-New. **TypeScript:** - [Safe Message Handlers](https://github.com/temporalio/samples-typescript/tree/main/message-passing/safe-message-handlers) — Entity Workflow with Updates, Signals, and Continue-As-New. ### References - [Temporal Blog: Very Long-Running Workflows](https://temporal.io/blog/very-long-running-workflows) — Guidance on managing Workflows that run for extended periods. - [Continue-As-New](/workflow-execution/continue-as-new) — Concept reference for the Continue-As-New mechanism. --- # Error Handling & Retry Patterns Source: https://docs.temporal.io/design-patterns/error-handling-patterns > Pattern selection guide and decision tree for choosing the right retry strategy based on your error type, cost constraints, and recovery requirements. These patterns control how Temporal retries Activities, surfaces persistent failures, and recovers from errors that require human intervention. ## Patterns in this section - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Caps the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource. - [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bounds the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many attempts occur. - [Non-Retryable Errors](/design-patterns/non-retryable-errors): Marks error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying. - [Delayed Retry](/design-patterns/delayed-retry): Override the next retry interval for a specific failure using nextRetryDelay on ApplicationFailure. Use when an error carries information about how long to wait before retrying. - [Fast/Slow Retries](/design-patterns/fast-slow-retries): Retries aggressively with a short interval first, then shifts to a long interval when fast retries are exhausted, keeping the Workflow alive until the downstream system recovers. - [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emits a custom metric from inside the Activity when the attempt count crosses a threshold, surfacing silent persistent failures to on-call teams before an SLA breach. - [Resumable Activity](/design-patterns/resumable-activity): Parks the Workflow after retries are exhausted and waits for a human to correct the data or approve continuing, then resumes from where it left off. ## Choosing a pattern The following decision tree helps you select the appropriate retry strategy for your use case. ```mermaid flowchart TD Start([Activity failing]) --> Q1{Each attempt\ncosts money or\nconsumes quota?} Q1 -->|Yes| FixedCount[Fixed Count of Retries\nCap MaximumAttempts] Q1 -->|No| Q2{Will this error\never succeed\nautomatically?} Q2 -->|No| Q2a{Can a human\ncorrect and retry?} Q2a -->|Yes| Resumable[Resumable Activity\nPark and await signal] Q2a -->|No| NonRetryable[Non-Retryable Errors\nFail fast] Q2 -->|Yes| Q3{Downstream has\na predictable\nunavailability window?} Q3 -->|Yes| Delayed[Delayed Retry\nFixed interval backoff] Q3 -->|No| Q4{Must resolve\nwithin a\ntime budget?} Q4 -->|Yes| WallTime[Fixed Wall-Time Retries\nScheduleToCloseTimeout] Q4 -->|No| Q5{Want aggressive\ninitial retries then\npatient recovery?} Q5 -->|Yes| FastSlow[Fast/Slow Retries\nTwo-phase retry policy] Q5 -->|No| Metrics[Retry Alerting via Metrics\nEmit metrics at attempt threshold] ``` The following describes each decision point: 1. If each attempt consumes a paid API call, a rate-limited token, or another scarce resource, use **Fixed Count of Retries** to cap total consumption. 2. If the error is structural — a missing record, invalid input, or authorization failure — and cannot be corrected automatically, ask whether a human can fix it: if so, use **Resumable Activity** to park the Workflow and await a correction signal; otherwise use **Non-Retryable Errors** to fail fast. 3. If the downstream system has a scheduled maintenance window and you know approximately how long it will be unavailable, use **Delayed Retry** with a fixed interval. 4. If the process must resolve (one way or another) within a business SLA window such as 24 hours, use **Fixed Wall-Time Retries** with `ScheduleToCloseTimeout`. 5. If you want to recover from transient errors quickly but also wait indefinitely for the downstream system to come back, use **Fast/Slow Retries**. 6. For any long-running retry scenario, add **Retry Alerting via Metrics** to surface persistent failures before they breach an SLA. ## How Temporal retries work Temporal's default `RetryPolicy` retries Activities indefinitely with exponential backoff. Unless you configure a policy, a failing Activity will keep retrying until the `ScheduleToCloseTimeout` or the Workflow itself completes. The key `RetryPolicy` fields are: | Field | Default | Effect | | :--- | :--- | :--- | | `MaximumAttempts` | 0 (unlimited) | Caps total attempts including the first | | `InitialInterval` | 1 second | Delay before the first retry | | `BackoffCoefficient` | 2.0 | Multiplier applied after each retry | | `MaximumInterval` | 100× InitialInterval | Upper bound on the backoff delay | | `NonRetryableErrorTypes` | `[]` | Error types that skip retries entirely | `ScheduleToCloseTimeout` is set on the Activity call options, not in `RetryPolicy`. It caps the total wall-clock time from when the Activity is first scheduled to when it must complete — across all retry attempts. ## Related sections - [External Interaction Patterns](/design-patterns/external-interaction-patterns) — heartbeating, polling, and approval gates for the external calls that fail - [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — compensate completed steps when retries finally give up - [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — where retry configuration fits in the latency budget ## References - [Temporal Retry Policies](/encyclopedia/retry-policies) - [Understanding Workflow Retries and Failures](https://community.temporal.io/t/understanding-workflow-retries-and-failures/122) - [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice) --- # Event Accumulator Pattern Source: https://docs.temporal.io/design-patterns/event-accumulator > Durably collect and deduplicate signals from multiple senders, then process the batch after a sliding inactivity timeout. > **ℹ️ TL;DR:** > Use the Event Accumulator pattern to **durably collect and process events from multiple senders over an unlimited time.** The workflow accumulates signals, deduplicates by a stable item key, and processes the batch after a sliding inactivity timeout — no external coordination, no lost events on retry. ## Overview The Accumulator pattern groups a stream of incoming events by a key and processes them together as a batch. A *group key* is a stable, domain-specific identifier — for example, an order ID, customer ID, or session token — that logically binds related events belonging to the same accumulation window. A single workflow instance per group key receives signals as events arrive, deduplicates them, and waits with a sliding inactivity timer. When no new events arrive within the timeout window, the workflow calls a batch processing activity and completes. ## Problem In distributed systems, events for the same logical entity arrive asynchronously from multiple producers at unpredictable rates. Processing each event individually wastes downstream resources and makes throughput harder to control. Ensuring exactly one active collector per group key — even under concurrent producers — requires coordination logic that is difficult to build reliably without distributed state. Without the Accumulator pattern, you must: - Handle race conditions when multiple producers try to start the same collection workflow simultaneously. - Implement deduplication externally, because at-least-once delivery is common in event streams. - Manage a reset timer that extends the collection window each time a new event arrives, without a reliable durable timer primitive. - Persist collection state externally across restarts and failures. - Handle gracefully the case where a long accumulation period grows the workflow history beyond safe limits. ## Solution You assign each group a deterministic workflow ID derived from the group key (for example, `accumulator-order-123`). Producers call Signal-With-Start so the workflow is created on first use and receives additional signals on subsequent calls — without any client-side coordination. Inside the workflow, `Workflow.await()` with a timeout implements a sliding inactivity window: each arriving signal resets the countdown. When the countdown expires — or when an explicit flush signal is sent — the workflow passes all accumulated, deduplicated events to a batch processing activity and completes. If the accumulation period is long enough to grow the workflow history near its limit, the workflow uses Continue-As-New to carry its state forward into a fresh run. ```mermaid sequenceDiagram participant P as Producers participant T as Temporal participant W as Accumulator Workflow participant A as Batch Activity P->>T: SignalWithStart(order-A, item-1) T->>W: Start workflow + deliver add-item(item-1) activate W Note over W: Accumulate item-1 · reset timer P->>T: SignalWithStart(order-A, item-2) T->>W: Deliver add-item(item-2) Note over W: Accumulate item-2 · reset timer P->>T: SignalWithStart(order-A, item-1) ← duplicate T->>W: Deliver add-item(item-1) Note over W: Deduplicate: item-1 already seen Note over W: Timer expires — no new signals W->>A: processItems(order-A, [item-1, item-2]) A-->>W: "Order order-A: 2 items fulfilled" deactivate W ``` The following describes each step in the diagram: 1. A producer calls Signal-With-Start with `order-A` as the bucket key. Temporal starts `accumulator-order-A` and delivers the first `add-item` signal. The workflow adds `item-1` to its list and starts the inactivity timer. 2. A second producer calls Signal-With-Start for the same order. Temporal finds the workflow already running and delivers the signal — no new instance is created. The workflow adds `item-2` and resets the timer. 3. The first producer retries `item-1` due to at-least-once delivery. The workflow checks its deduplication set, finds `item-1` already recorded, and discards the duplicate. Alternatively, you can replace the existing record with the new payload — useful when a producer may resend an updated version of the same event under the same key. 4. No new signals arrive within the inactivity window. The `Workflow.await()` condition times out. 5. The workflow calls `processItems` with all accumulated items. The activity returns a result, and the workflow completes. ## Implementation The following examples highlight the key mechanics of the accumulator loop. Full implementations including the worker, starter, and shared types are available in the runner above. **TypeScript** ```typescript // workflows.ts export async function accumulatorWorkflow( bucketKey: string, accumulated: OrderItem[] = [], seenKeys: string[] = [], ): Promise { const seenSet = new Set(seenKeys); const items: OrderItem[] = [...accumulated]; const unprocessed: OrderItem[] = []; let flushRequested = false; setHandler(addItemSignal, (item: OrderItem) => { unprocessed.push(item); }); setHandler(flushSignal, () => { flushRequested = true; }); do { // Sliding window: wait for a signal or let the inactivity timer fire const timedOut = !(await condition( () => unprocessed.length > 0 || flushRequested, "10 seconds", )); // Drain and deduplicate incoming signals while (unprocessed.length > 0) { const item = unprocessed.shift()!; if (item.orderId === bucketKey && !seenSet.has(item.itemId)) { seenSet.add(item.itemId); items.push(item); } } if (timedOut || flushRequested) { const result = await processItems(bucketKey, items); if (unprocessed.length === 0) return result; // More signals arrived after timeout/flush — loop to process them } } while (unprocessed.length > 0 || !workflowInfo().continueAsNewSuggested); // History growing large — continue as new, carrying accumulated state forward await continueAsNew(bucketKey, items, [...seenSet]); return ""; // unreachable } ``` **Python** ```python # workflows.py @workflow.defn class AccumulatorWorkflow: def __init__(self) -> None: self._unprocessed: deque[OrderItem] = deque() self._flush_requested = False @workflow.signal(name="add-item") async def add_item(self, item: OrderItem) -> None: self._unprocessed.append(item) @workflow.signal(name="flush") async def flush(self) -> None: self._flush_requested = True @workflow.run async def run( self, bucket_key: str, accumulated: list[OrderItem] | None = None, seen_keys: list[str] | None = None, ) -> str: items = list(accumulated or []) seen_set = set(seen_keys or []) while True: # Sliding window: wait for a signal or let the inactivity timer fire timed_out = not await workflow.wait_condition( lambda: bool(self._unprocessed) or self._flush_requested, timeout=timedelta(seconds=10), ) # Drain and deduplicate the signal queue while self._unprocessed: item = self._unprocessed.popleft() if item.order_id == bucket_key and item.item_id not in seen_set: seen_set.add(item.item_id) items.append(item) if timed_out or self._flush_requested: result = await workflow.execute_activity( process_items, args=[bucket_key, items], start_to_close_timeout=timedelta(seconds=10), ) if not self._unprocessed: return result # More signals arrived after timeout/flush — loop to process them if not self._unprocessed and workflow.info().is_continue_as_new_suggested(): workflow.continue_as_new(args=[bucket_key, items, sorted(seen_set)]) ``` **Go** ```go // workflows.go func AccumulatorWorkflow(ctx workflow.Context, bucketKey string, items []OrderItem, seenKeys []string) (string, error) { seenSet := make(map[string]bool) for _, k := range seenKeys { seenSet[k] = true } accumulated := append([]OrderItem{}, items...) addItemCh := workflow.GetSignalChannel(ctx, "add-item") flushCh := workflow.GetSignalChannel(ctx, "flush") flushRequested := false for { // Drain any signals buffered before this iteration for { var item OrderItem if !addItemCh.ReceiveAsync(&item) { break } if item.OrderID == bucketKey && !seenSet[item.ItemID] { seenSet[item.ItemID] = true accumulated = append(accumulated, item) } } var voidFlush interface{} if flushCh.ReceiveAsync(&voidFlush) { flushRequested = true } if flushRequested { break } if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { keys := make([]string, 0, len(seenSet)) for k := range seenSet { keys = append(keys, k) } sort.Strings(keys) // deterministic order for replay return "", workflow.NewContinueAsNewError(ctx, AccumulatorWorkflow, bucketKey, accumulated, keys) } // Sliding window: wait for a signal or let the inactivity timer fire timedOut := false timerCtx, cancelTimer := workflow.WithCancel(ctx) timer := workflow.NewTimer(timerCtx, maxAwaitTime) selector := workflow.NewSelector(ctx) selector.AddFuture(timer, func(f workflow.Future) { timedOut = true }) selector.AddReceive(addItemCh, func(c workflow.ReceiveChannel, _ bool) { var item OrderItem c.Receive(ctx, &item) if item.OrderID == bucketKey && !seenSet[item.ItemID] { seenSet[item.ItemID] = true accumulated = append(accumulated, item) } }) selector.AddReceive(flushCh, func(c workflow.ReceiveChannel, _ bool) { var void interface{} c.Receive(ctx, &void) flushRequested = true }) selector.Select(ctx) cancelTimer() // no-op if timer already fired; cancels timer if a signal arrived if timedOut || flushRequested { break } } ao := workflow.ActivityOptions{StartToCloseTimeout: 10 * time.Second} actCtx := workflow.WithActivityOptions(ctx, ao) var result string if err := workflow.ExecuteActivity(actCtx, ProcessItems, bucketKey, accumulated).Get(ctx, &result); err != nil { return "", err } workflow.GetLogger(ctx).Info("Processed order batch", "bucketKey", bucketKey, "count", len(accumulated)) return result, nil } ``` **Java** ```java // AccumulatorWorkflow.java @WorkflowInterface public interface AccumulatorWorkflow { @WorkflowMethod String accumulate(String bucketKey, List items, List seenKeys); @SignalMethod void addItem(Shared.OrderItem item); @SignalMethod void flush(); class Impl implements AccumulatorWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Shared.MAX_AWAIT_TIME.plusSeconds(10)) .build()); private final ArrayDeque unprocessed = new ArrayDeque<>(); private boolean flushRequested = false; @Override public String accumulate(String bucketKey, List itemsInput, List seenKeysInput) { List items = new ArrayList<>(itemsInput); Set seenSet = new HashSet<>(seenKeysInput); do { // Sliding window: wait for a signal or let the inactivity timer fire boolean timedOut = !Workflow.await( Shared.MAX_AWAIT_TIME, () -> !unprocessed.isEmpty() || flushRequested); // Drain and deduplicate the signal queue while (!unprocessed.isEmpty()) { Shared.OrderItem item = unprocessed.removeFirst(); if (item.orderId.equals(bucketKey) && seenSet.add(item.itemId)) { items.add(item); } } if (timedOut || flushRequested) { String result = activities.processItems(bucketKey, items); if (unprocessed.isEmpty()) return result; // More signals arrived after timeout/flush — loop to process them } } while (!unprocessed.isEmpty() || !Workflow.getInfo().isContinueAsNewSuggested()); // History growing large — continue as new, carrying accumulated state forward AccumulatorWorkflow continueAsNew = Workflow.newContinueAsNewStub(AccumulatorWorkflow.class); List seenKeysList = new ArrayList<>(seenSet); java.util.Collections.sort(seenKeysList); // deterministic order for replay continueAsNew.accumulate(bucketKey, items, seenKeysList); return ""; // unreachable } @Override public void addItem(Shared.OrderItem item) { unprocessed.add(item); } @Override public void flush() { flushRequested = true; } } } ``` The key differences between SDKs: - **TypeScript**: Uses `setHandler` to register signal handlers and `condition()` with a string duration for the sliding window. `continueAsNew()` carries accumulated state to the next run. - **Python**: Uses `@workflow.signal(name=...)` decorators and `workflow.wait_condition()` with a `timedelta` timeout. `workflow.continue_as_new()` accepts the new run arguments. - **Go**: Uses `workflow.GetSignalChannel` to obtain named signal channels, then builds a `Selector` per iteration with a `NewTimer` future and two channel receivers. Cancels the old timer whenever a signal arrives before it fires. - **Java**: Uses `@SignalMethod` annotations and `Workflow.await(Duration, Supplier)`. `Workflow.newContinueAsNewStub` carries state to the next run. Producers in all SDKs call Signal-With-Start to atomically start the workflow if not running and deliver the first signal without any client-side coordination. ## When to use The Accumulator pattern is well suited when events related to the same entity or group arrive from multiple producers — including a single consumer (for example, a Kafka consumer) polling a topic that carries events for multiple groups simultaneously — and can be processed together as batches, when downstream systems prefer batched calls rather than one call per event, and when at-least-once event delivery makes deduplication necessary at the collection layer. It is not a good fit for use cases that require processing every event individually in _strict order_, for cases where the batch size is known in advance and all events arrive within a short deterministic window (a standard workflow is sufficient), or when events for different keys must be correlated at processing time (consider fan-in with for example Child Workflows or multiple levels of Accumulator). ## Benefits and trade-offs The Accumulator pattern reduces downstream load by consolidating many individual events into a single batch and activity call. Signal-With-Start eliminates client-side coordination logic for starting or locating the collector workflow. Temporal's durable execution guarantees that accumulated state survives Worker restarts, and Continue-As-New prevents history growth from becoming a long-term problem. The trade-offs to consider are that events are not processed until the inactivity timeout fires or a flush signal is sent, introducing intentional latency. The timeout value is a domain-specific trade-off between latency and batch size. If producers stop sending signals but never send a flush, the workflow holds open resources until the timeout fires. ## Best practices - **Use a deterministic workflow ID per bucket key.** Encode the group key and, optionally, an accumulation period (for example, `accumulator-order-123-2026-05-14`) to control when a new window starts. - **Always use Signal-With-Start from producers.** Calling start and signal separately is not atomic: a signal sent between the two calls can be lost if the workflow has not yet started. - **Include a deduplication key in every signal payload.** At-least-once delivery is common in event streams; without a dedup key, retried events add duplicate entries to the batch. - **Size the inactivity timeout to your domain's quiet period.** The timeout should reflect how long you are confident no more events will arrive for this batch. Tune it based on observed producer behavior, not an arbitrary constant. - **Pass accumulated state as workflow arguments into each Continue-As-New run.** Workflow state does not survive a Continue-As-New transition automatically; always pass items and the seen-keys list as arguments to the new run. If you omit the seen-keys list, any signal delivered to both the old and new run during the Continue-As-New handoff will be processed again in the new run, producing duplicate entries in the batch. - **Keep signal handlers fast and side-effect free.** Signal handlers must not call activities or yield to the scheduler. Buffer incoming signals and process them in the main workflow loop. - **Add a flush signal for testing and operational runbooks.** An explicit flush signal lets you trigger early batch processing without waiting for the timeout, which is useful for end-to-end tests and manual intervention. - **Keep producer signal rate below 5/sec per workflow instance.** Each signal briefly locks the workflow execution. A sustained rate above roughly 5 signals/second causes workflow task backlog, limits throughput, and can eventually prevent Continue-As-New from completing. If your producer rate is higher, partition by a finer-grained key so each accumulator workflow receives a manageable share of the total signal volume. - **Account for the 10,000-signal-per-run limit on Temporal Cloud.** A single workflow run in Temporal Cloud can receive at most 10,000 signals. If your accumulation window is long and producers are active, ensure your Continue-As-New trigger fires well before the per-run signal count reaches this limit. ## Common pitfalls - **Non-deterministic or random workflow IDs.** If the workflow ID is not derived deterministically from the group key, multiple accumulator instances are created for the same group, splitting the batch. - **Calling start followed by a separate signal.** These are not atomic. A signal sent between the two calls will be lost if the workflow has not yet started. Use Signal-With-Start instead. - **Omitting a deduplication key.** Retried or re-delivered events add duplicate entries to the batch. Every signal payload must carry a stable, unique key. - **Timeout set too short.** The workflow processes a partial batch while more events are still in flight, forcing producers to re-send unprocessed events. Profile your producer arrival rate before choosing a timeout. - **Forgetting to pass accumulated state into Continue-As-New.** The new run starts with empty state and re-processes events from scratch, producing duplicate batches. - **Calling activities inside signal handlers.** Signal handlers run synchronously in the workflow thread and must not block or call activities. Buffer the item and let the main loop handle activity calls. - **Assuming workflow completion means all events were captured.** Producers that send signals after the workflow completes will start a new accumulator instance. Decide whether this is intentional (a new accumulation window) or an error. - **Signal rate too high to allow Continue-As-New to complete.** Continue-As-New requires a brief window (~100 ms) with no unhandled signals. If producers send signals continuously without pause, the workflow can never enter that window, history grows without bound, and Temporal will eventually terminate the workflow. Partition by a finer-grained key, throttle producers, or batch multiple events into a single signal payload to keep the per-instance signal rate low enough for CAN to succeed. - **Not draining the signal queue before calling Continue-As-New.** Any signal that arrives between the CAN decision and the actual CAN execution can be lost if the signal buffer is not empty when CAN fires. All SDK implementations in this pattern guard against this by re-checking the unprocessed queue before continuing; do not remove that guard or call CAN unconditionally on the history-size trigger. ## Related ### Patterns - **[Signal with Start](/design-patterns/signal-with-start)** — the atomic start-and-signal primitive this pattern uses to create or locate the accumulator workflow. - **[Continue-As-New](/design-patterns/continue-as-new)** — used to reset workflow history when the accumulation period is long. - **[Updatable Timer](/design-patterns/updatable-timer)** — an alternative approach for a resettable timer that does not require signals. - **[Entity Workflow](/design-patterns/entity-workflow)** — a broader pattern for long-lived, keyed workflow instances. ### Sample code - [Java Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloAccumulator.java) — the canonical accumulator example from the Temporal Java samples repository. --- # External Interaction Patterns Source: https://docs.temporal.io/design-patterns/external-interaction-patterns > Pattern selection guide for waiting on or interacting with systems and actors outside the Workflow. These patterns cover how a Workflow waits on or interacts with the world outside it — external APIs, human decisions, scheduled delays, and inbound or outbound callbacks — while staying durable across failures. ## Patterns in this section - [Polling External Services](/design-patterns/polling): Checks an external resource on a schedule until it reaches the state you need, with frequent, infrequent, and periodic variants. - [Long Running Activity](/design-patterns/long-running-activity): Reports progress via heartbeats and resumes after failures, with cancellation support, for Activities that run for minutes to hours. - [Approval](/design-patterns/approval): Blocks the Workflow until an external decision arrives, capturing the approval and its metadata through a Signal. - [Delayed Start](/design-patterns/delayed-start): Creates the Workflow immediately but defers execution until a delay expires. - [Delayed Callback (Webhooks)](/design-patterns/delayed-callback): Integrates webhooks durably: receive inbound webhooks via Signals, fire delayed outbound callbacks with durable timers, and complete Activities asynchronously via task tokens. ## Choosing a pattern **The external system offers no notification**: use [Polling External Services](/design-patterns/polling) and tune the interval to the expected latency. **One Activity runs for a long time**: use a [Long Running Activity](/design-patterns/long-running-activity) with heartbeats so failures resume instead of restarting. **A person or external system must decide**: use [Approval](/design-patterns/approval) to block on a Signal. **Execution should begin later**: use [Delayed Start](/design-patterns/delayed-start). **You send or receive HTTP callbacks**: use [Delayed Callback (Webhooks)](/design-patterns/delayed-callback). ## Related sections - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — retry strategies for the external calls these patterns make - [Workflow Messaging Patterns](/design-patterns/workflow-messaging-patterns) — the Signals that deliver external decisions into a Workflow --- # Fairness Source: https://docs.temporal.io/design-patterns/fairness > Distributes Worker capacity evenly across tenants or users so that a burst from one caller does not starve the others. > **ℹ️ TLDR:** > Assign a `FairnessKey` and weight to Workflows and Activities so each tenant or group receives the **correct proportional share of Worker capacity** on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants without requiring separate queues per tenant. ## Overview The Fairness pattern distributes Worker capacity proportionally across tenants or user groups within a single Task Queue so that a burst from one caller cannot starve others. Each group is assigned a fairness key and an optional weight; the Temporal matching service dispatches tasks in weighted round-robin order across all keys. ## Problem When multiple tenants (for example, customers) share a single Task Queue, a high-volume tenant can fill the queue and occupy all Worker slots. Other tenants receive no service until the dominant tenant's backlog drains. This starvation violates throughput guarantees and makes latency for lower-volume tenants unpredictable under burst conditions. The classic workaround—assigning one Task Queue per tenant—scales poorly: each new tenant requires a new Worker deployment, idle capacity on low-traffic tenants cannot be used by busy ones, and queue management complexity grows with tenant count. ## Solution Temporal's native Fairness feature lets you assign a `FairnessKey` (a string identifier such as a tenant name or tier) and an optional `FairnessWeight` (a positive float, default 1.0) to Workflows, Activities, and Child Workflows. The Temporal matching service creates a virtual queue for each key and dispatches tasks in proportion to their weights. A single shared Worker pool serves all keys; no extra queues or routing logic is required. For example, assigning weights of 5.0, 3.0, and 2.0 to `premium`, `basic`, and `free` tiers causes 50% of dispatched tasks to come from `premium`, 30% from `basic`, and 20% from `free`—regardless of backlog depth. Within a single fairness key, tasks are dispatched in FIFO order. ```mermaid flowchart TD WA["Workflow\nfairness_key=tenant-big\n(weight 1.0)"] --> TQ["my-task-queue"] WB["Workflow\nfairness_key=tenant-mid\n(weight 1.0)"] --> TQ WC["Workflow\nfairness_key=tenant-small\n(weight 1.0)"] --> TQ TQ --> VQ1["Virtual Queue\ntenant-big"] TQ --> VQ2["Virtual Queue\ntenant-mid"] TQ --> VQ3["Virtual Queue\ntenant-small"] VQ1 -->|round-robin| W["Shared Workers"] VQ2 -->|round-robin| W VQ3 -->|round-robin| W W --> DS["Downstream\nService"] ``` The following describes each step in the diagram: 1. Workflows start with a `FairnessKey` matching the tenant or group identity. 2. The Temporal matching service routes each task to the corresponding virtual queue inside the single Task Queue. 3. Workers poll the Task Queue and receive tasks in weighted round-robin order across all fairness keys. 4. Tenant-big's large backlog does not prevent tenant-mid or tenant-small from receiving service. ## Implementation ### Enable fairness **Temporal Cloud:** Navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Fairness is a paid feature in Temporal Cloud. **Self-hosted Temporal:** Set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) for the relevant Task Queues or Namespaces. ### Set fairness key and weight at Workflow start **Python** ```python from temporalio.common import Priority handle = await client.start_workflow( ProcessOrder.run, id="process-order-wf", task_queue="my-task-queue", priority=Priority( fairness_key="tenant-a", fairness_weight=2.0, ), ) ``` **Go** ```go we, err := c.ExecuteWorkflow( context.Background(), client.StartWorkflowOptions{ ID: "process-order-wf", TaskQueue: "my-task-queue", Priority: temporal.Priority{ FairnessKey: "tenant-a", FairnessWeight: 2.0, }, }, ProcessOrder, ) ``` **Java** ```java WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("process-order-wf") .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder() .setFairnessKey("tenant-a") .setFairnessWeight(2.0f) .build()) .build(); ProcessOrder workflow = client.newWorkflowStub(ProcessOrder.class, options); WorkflowClient.start(workflow::run); ``` ### Set fairness key and weight on Activities Activities inherit the parent Workflow's fairness key and weight. Override them in `ActivityOptions` when an Activity should belong to a different fairness group than its Workflow. Each field (`priority_key`, `fairness_key`, `fairness_weight`) is resolved independently in this order: Task Queue weight overrides (highest precedence), value set explicitly in the options, value inherited from the calling Workflow, then the default. Workflows started with Continue-As-New inherit the current execution's priority values unless you pass explicit values. See [Inheritance](/develop/task-queue-priority-fairness#inheritance) in the Temporal docs for the full resolution diagram. **Python** ```python from temporalio.common import Priority # inside the workflow result = await workflow.execute_activity( process_for_tenant, tenant_request, start_to_close_timeout=timedelta(minutes=1), priority=Priority( fairness_key="tenant-a", fairness_weight=2.0, ), ) ``` **Go** ```go ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{ FairnessKey: "tenant-a", FairnessWeight: 2.0, }, } ctx = workflow.WithActivityOptions(ctx, ao) err := workflow.ExecuteActivity(ctx, ProcessForTenant, req).Get(ctx, nil) ``` **Java** ```java ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder() .setFairnessKey("tenant-a") .setFairnessWeight(2.0f) .build()) .build(); TenantActivity activity = Workflow.newActivityStub(TenantActivity.class, options); activity.processForTenant(request); ``` ### Set queue-level and per-key rate limits via CLI You can rate-limit the entire Task Queue and set a default per-fairness-key limit. The per-key limit is scaled by the fairness weight for that key, so a key with weight 2.5 and a default per-key limit of 10 gets an effective limit of 25 tasks/second. ```sh temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --queue-rps-limit 500 \ --queue-rps-limit-reason "overall limit" \ --fairness-key-rps-limit-default 33.3 \ --fairness-key-rps-limit-reason "per-key limit" ``` ### Override fairness weights via CLI When it is more convenient to manage weights through configuration than to embed them in client code, you can override weights for up to 1000 keys per Task Queue. Overrides take precedence over the weight attached to a task's options and can be updated without a code deploy. ```sh temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type workflow \ --namespace my-namespace \ --fairness-key-weight premium=5.0 \ --fairness-key-weight basic=3.0 \ --fairness-key-weight free=2.0 ``` ### Use priority and fairness together Priority and Fairness can be combined. Priority determines which sub-queue (1–5) a task enters; Fairness determines the dispatch order within each priority level. Set both `PriorityKey` and `FairnessKey` on the same options object. **Python** ```python from temporalio.common import Priority handle = await client.start_workflow( ChargeCustomer.run, id="charge-customer-wf", task_queue="my-task-queue", priority=Priority( priority_key=1, fairness_key="tenant-a", fairness_weight=2.0, ), ) ``` **Go** ```go we, err := c.ExecuteWorkflow( context.Background(), client.StartWorkflowOptions{ ID: "charge-customer-wf", TaskQueue: "my-task-queue", Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "tenant-a", FairnessWeight: 2.0, }, }, ChargeCustomer, ) ``` **Java** ```java WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("charge-customer-wf") .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder() .setPriorityKey(1) .setFairnessKey("tenant-a") .setFairnessWeight(2.0f) .build()) .build(); ``` ## When to use This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants, for workloads that need proportional capacity allocation across groups without hard rate limits, and when the set of tenants or groups is dynamic (new keys can be introduced without deploying new Workers). For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns). It is not a good fit when absolute throughput isolation is required (dedicated queues per tenant remain or [task queue priorities](/design-patterns/priority-task-queues) are the appropriate choice). ## Benefits and trade-offs A single Worker pool serves all tenants; idle capacity from a low-traffic tenant automatically benefits high-traffic tenants rather than going to waste. New tenants require no Worker deployment—add a fairness key and Temporal starts dispatching their tasks immediately. Weights can be updated via CLI without redeploying application code. Fairness requires explicit enablement on Temporal Cloud and self-hosted deployments. Accuracy can degrade with a very large number of fairness keys. Fairness weight applies at schedule time, not dispatch time: changing a weight does not retroactively reorder tasks already in the backlog. ## Comparison with alternatives | Approach | Tenant isolation | Dynamic tenants | Shares idle capacity | Complexity | | :--- | :--- | :--- | :--- | :--- | | Temporal FairnessKey (native) | Soft | Yes | Yes | Low | | Dedicated queue per tenant | Hard | No | No | Medium | | Single shared queue (no control) | None | Yes | Yes | Lowest | | External queue with per-tenant consumer groups | Hard | Yes | No | High | ## Best practices - **Use stable, consistent naming for fairness keys.** Use account IDs or tenant slugs rather than display names. Key names cannot be changed retroactively on tasks already in the backlog. - **Combine Priority and Fairness for multi-class, multi-tenant workloads.** Priority separates urgent from batch work; Fairness prevents any single tenant from dominating within each priority level. - **Monitor queue depth by fairness key.** Sustained backlog growth for a particular key means its weight fraction of Worker capacity cannot drain its submission rate. ## Common pitfalls - **Expecting Fairness to reorder the existing backlog.** Fairness weight is evaluated at schedule time. Enabling Fairness on a Namespace with an existing backlog drains that backlog in its original order first; the fairness-aware dispatch mode takes effect only for newly submitted tasks. - **Using Fairness as a hard rate limiter.** Fairness controls proportional dispatch but does not cap the absolute throughput of any one key. For hard throughput caps, combine Fairness with per-fairness-key RPS limits via the CLI. - **Unkeyed tasks bypassing Fairness.** Tasks without a `FairnessKey` are grouped under an implicit empty-string key and participate in round-robin dispatch alongside named keys with a weight of 1.0. They do not bypass Fairness and compete as one group. - **Task Queue partitioning reducing accuracy.** Task Queues are internally partitioned and tasks are distributed to partitions randomly, which can interfere with fair dispatch proportions. If your workload requires higher accuracy, contact Temporal Support to configure a single-partition Task Queue. - **Assuming Fairness applies across Worker Versioning boundaries.** When using Worker Versioning and moving Workflows between versions, Priority still applies across versions but Fairness is only guaranteed within tasks originally queued on the same Worker version. Tasks moved from one version to another may not dispatch in fairness order relative to tasks on the destination version. - **Expecting consistent fairness immediately after a server restart.** Fairness ordering is preserved across restarts for the most active keys. Less active keys may briefly dispatch new tasks ahead of their existing backlog until ordering normalizes. - **Expecting the running task mix to immediately reflect fair dispatch.** Fairness governs which task is dispatched next; it does not account for tasks already running on Workers. The mix of in-flight tasks at any moment may not match the configured weight ratios. ## Related ### Patterns - **[Priority Task Queues](/design-patterns/priority-task-queues)**: Order tasks by urgency level within the same Task Queue using `PriorityKey`. - **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap absolute throughput to a downstream service with a queue RPS setting. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. --- # Fan-Out with Child Workflows Source: https://docs.temporal.io/design-patterns/fanout-child-workflows > Distributes a large record set across parallel Child Workflows for concurrent processing with automatic scaling. > **ℹ️ TLDR:** > Split your record set into fixed-size chunks and start **one child Workflow per chunk** so that each chunk's history stays within Temporal's limits. Use this when you want maximum concurrency with no rate control and you can pre-compute how many chunks you need before the job starts. Keep the number of in-flight children per parent well under the default limit of 2,000; use [Sliding Window](/design-patterns/sliding-window) or [Batch Iterator](/design-patterns/batch-iterator) for larger workloads. ## Overview The Fan-Out pattern distributes a large record set across multiple independent child Workflows, each responsible for processing a fixed-size chunk. The parent Workflow assigns work by offset and length so that no record IDs need to be passed over the wire — only two integers per child. ## Problem A single Workflow run can have at most 2,000 in-flight Activities (aim for 500) and at most 50,000 history events. Processing millions of records in a single Workflow run is therefore not possible. You need a way to partition a large record set, process each partition independently, and coordinate the overall job while keeping each Workflow's history within safe bounds. ## Solution You split the total record count into fixed-size chunks and start one child Workflow per chunk. Each child is given an `offset` and a `length` so it knows which slice of the record set to fetch and process independently. The parent Workflow starts all children concurrently and waits for them all to complete. If a child fails the parent can retry that child without re-processing the records handled by other children. ```mermaid flowchart TD Records["📋 Total record set\n(N records)"] Parent["Parent Workflow\n(fanOutWorkflow)"] C1["Child Workflow\n(offset=0, length=chunk)"] C2["Child Workflow\n(offset=chunk, length=chunk)"] C3["Child Workflow\n(offset=2×chunk, length=chunk)"] Records --> Parent Parent -->|"start child 1"| C1 Parent -->|"start child 2"| C2 Parent -->|"start child 3"| C3 C1 --> A1["processRecord ×chunk"] C2 --> A2["processRecord ×chunk"] C3 --> A3["processRecord ×chunk"] A1 -->|"done"| Parent A2 -->|"done"| Parent A3 -->|"done"| Parent ``` The following describes each step in the diagram: 1. The parent Workflow receives the total record count and a configured chunk size. 2. It divides the total into chunks and starts one child Workflow per chunk, passing only `offset` and `length`. 3. Each child independently fetches its slice of records (using the offset and length) and calls `processRecord` for each one. 4. Each child completes and returns its result to the parent. 5. The parent blocks until all children have completed, then returns the aggregated result. ## Implementation The following examples show how each SDK implements the Fan-Out pattern. **TypeScript** ```typescript // workflows.ts import { executeChild, proxyActivities, workflowInfo, } from "@temporalio/workflow"; import type * as activities from "./activities"; import { TASK_QUEUE, CHUNK_SIZE } from "./shared"; const { processRecord } = proxyActivities({ startToCloseTimeout: "10 seconds", }); export async function fanOutWorkflow( totalRecords: number, chunkSize: number = CHUNK_SIZE ): Promise { const children: Promise[] = []; for (let offset = 0; offset < totalRecords; offset += chunkSize) { const length = Math.min(chunkSize, totalRecords - offset); children.push( executeChild(recordBatchWorkflow, { args: [offset, length], taskQueue: TASK_QUEUE, workflowId: `${workflowInfo().workflowId}/batch-${offset}`, }) ); } const results = await Promise.all(children); return results.reduce((sum, n) => sum + n, 0); } export async function recordBatchWorkflow( offset: number, length: number ): Promise { let processed = 0; for (let i = offset; i < offset + length; i++) { await processRecord(i); processed++; } return processed; } ``` **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.workflow import ChildWorkflowHandle import asyncio from activities import process_record from shared import TASK_QUEUE, CHUNK_SIZE @workflow.defn class RecordBatchWorkflow: @workflow.run async def run(self, offset: int, length: int) -> int: processed = 0 for i in range(offset, offset + length): await workflow.execute_activity( process_record, i, start_to_close_timeout=timedelta(seconds=10), ) processed += 1 return processed @workflow.defn class FanOutWorkflow: @workflow.run async def run(self, total_records: int, chunk_size: int = CHUNK_SIZE) -> int: handles: list[ChildWorkflowHandle] = [] parent_id = workflow.info().workflow_id offset = 0 while offset < total_records: length = min(chunk_size, total_records - offset) handle = await workflow.start_child_workflow( RecordBatchWorkflow.run, args=[offset, length], id=f"{parent_id}/batch-{offset}", task_queue=TASK_QUEUE, ) handles.append(handle) offset += chunk_size results = await asyncio.gather(*handles) return sum(results) ``` **Go** ```go // workflows.go package main import ( "fmt" "time" "go.temporal.io/sdk/workflow" ) func FanOutWorkflow(ctx workflow.Context, totalRecords int, chunkSize int) (int, error) { if chunkSize <= 0 { chunkSize = ChunkSize } var futures []workflow.Future parentID := workflow.GetInfo(ctx).WorkflowExecution.ID for offset := 0; offset < totalRecords; offset += chunkSize { length := chunkSize if offset+chunkSize > totalRecords { length = totalRecords - offset } off := offset // capture loop variable cwo := workflow.ChildWorkflowOptions{ WorkflowID: parentID + "/batch-" + fmt.Sprintf("%d", off), TaskQueue: TaskQueue, } cctx := workflow.WithChildOptions(ctx, cwo) futures = append(futures, workflow.ExecuteChildWorkflow(cctx, RecordBatchWorkflow, off, length)) } total := 0 for _, f := range futures { var n int if err := f.Get(ctx, &n); err != nil { return total, err } total += n } return total, nil } func RecordBatchWorkflow(ctx workflow.Context, offset int, length int) (int, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) processed := 0 for i := offset; i < offset+length; i++ { if err := workflow.ExecuteActivity(ctx, ProcessRecord, i).Get(ctx, nil); err != nil { return processed, err } processed++ } return processed, nil } ``` **Java** ```java // FanOutWorkflow.java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.*; import java.time.Duration; import java.util.ArrayList; import java.util.List; @WorkflowInterface public interface FanOutWorkflow { @WorkflowMethod int run(int totalRecords, int chunkSize); } // FanOutWorkflowImpl.java public class FanOutWorkflowImpl implements FanOutWorkflow { @Override public int run(int totalRecords, int chunkSize) { if (chunkSize <= 0) chunkSize = Shared.CHUNK_SIZE; List> promises = new ArrayList<>(); String parentId = Workflow.getInfo().getWorkflowId(); for (int offset = 0; offset < totalRecords; offset += chunkSize) { int length = Math.min(chunkSize, totalRecords - offset); ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder() .setWorkflowId(parentId + "/batch-" + offset) .setTaskQueue(Shared.TASK_QUEUE) .build(); RecordBatchWorkflow child = Workflow.newChildWorkflowStub(RecordBatchWorkflow.class, opts); promises.add(Async.function(child::run, offset, length)); } int total = 0; for (Promise p : promises) { total += p.get(); } return total; } } // RecordBatchWorkflow.java @WorkflowInterface public interface RecordBatchWorkflow { @WorkflowMethod int run(int offset, int length); } // RecordBatchWorkflowImpl.java public class RecordBatchWorkflowImpl implements RecordBatchWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .build() ); @Override public int run(int offset, int length) { int processed = 0; for (int i = offset; i < offset + length; i++) { activities.processRecord(i); processed++; } return processed; } } ``` ## Best practices - **Use offset and length, not explicit IDs.** Pass only two integers to each child rather than a full slice of IDs. The child fetches its own records. This keeps history events small. - **Size chunks to stay under the Activity limit.** Each child Workflow can have at most 2,000 in-flight Activities. Aim for chunks of 500 records or fewer if each record maps to one Activity. - **Cap concurrent children in the parent.** Starting thousands of child Workflows simultaneously puts pressure on the namespace. Consider batching child starts or using [Sliding Window](/design-patterns/sliding-window) if you need tighter concurrency control. - **Set `PARENT_CLOSE_POLICY_ABANDON`** for fire-and-forget fan-outs where the parent does not need to collect results. With the default `TERMINATE` policy, cancelling or timing out the parent will terminate all in-flight children. - **Give each child a deterministic Workflow ID** (`parentId/batch-`). This makes it safe to re-run the parent: Temporal deduplicates child starts by Workflow ID, so already-completed children are not re-executed. ## Common pitfalls - **Starting too many children at once.** Each child start adds to the parent's history. Temporal enforces a default limit of 2,000 pending (in-flight) child Workflows per parent; keep well under it. See [Temporal guidance](/child-workflows#when-to-use-child-workflows). If you need more children, switch to [MapReduce Tree](/design-patterns/mapreduce-tree) or [Sliding Window](/design-patterns/sliding-window). - **Passing large lists of IDs.** Workflow inputs are stored in event history. Passing millions of record IDs as a list will blow the history size limit. Use offset + length instead. - **Ignoring child failures.** A failed child does not automatically fail the parent unless you await all results. Always await child handles and handle errors explicitly. ## Related ### Patterns - [Child Workflows pattern](/design-patterns/child-workflows) — core concepts for parent/child Workflow coordination - [Batch Iterator](/design-patterns/batch-iterator) — unbounded record sets with Continue-as-New pagination - [Sliding Window](/design-patterns/sliding-window) — bounded concurrency with maximum throughput - [Temporal limits reference](/cloud/limits) --- # Fast/Slow Retries Source: https://docs.temporal.io/design-patterns/fast-slow-retries > Try aggressively with a short interval first, then shift to a long interval when fast retries are exhausted, keeping the Workflow alive until the downstream system recovers. > **ℹ️ TLDR:** > Orchestrate two retry phases in the Workflow: a fast phase with short intervals and bounded attempts for transient errors, followed by a slow phase with long intervals and unlimited retries for extended outages. **Use this when a single `RetryPolicy` should not cover both brief blips and hour-long outages or maintenance windows.** ## Overview The Fast/Slow Retries pattern runs an Activity through two distinct retry phases: a fast phase with a short interval and bounded attempt count, followed by an unlimited slow phase with a long fixed interval managed by the Temporal Service. Use it when transient errors are common and worth recovering from quickly, but the downstream system may also suffer extended outages that require patient, indefinite waiting. ## Problem Conventional retry policies force you to choose between: - **Low `MaximumAttempts`**: Recovers from transient errors quickly but abandons the request when the downstream system has a longer outage. - **Unlimited or High `MaximumAttempts` with short interval**: Floods a degraded downstream system with retries and accumulates noisy failures in logs. Incurs processing cost with each attempt. - **Long fixed interval with unlimited retries**: Recovers from outages eventually, but is too slow to recover from transient errors that would have resolved in seconds. None of these options handles both scenarios well — a downstream system that sometimes has brief 503 errors *and* occasionally goes down for an extended maintenance window. ## Solution Use the Workflow itself as a retry orchestrator across two phases: **Phase 1 — Fast retries**: Execute the Activity with a short `InitialInterval` and a bounded `MaximumAttempts`. This phase recovers from transient errors within seconds or minutes. **Phase 2 — Slow retries**: When the fast retry policy is exhausted, catch the `ActivityError` in the Workflow and execute the Activity again with a long `InitialInterval` and unlimited `MaximumAttempts`. The Temporal Service owns the slow retry management; the Workflow blocks until the Activity eventually succeeds. This design is invisible in conventional retry libraries because it requires the retry orchestrator to be a durable, resumable process — exactly what a Temporal Workflow is. ```mermaid flowchart TD Start([Workflow starts]) --> Phase1 subgraph Phase1 [Phase 1 — Fast Retries] F1[Execute Activity\ninitialInterval=1s\nmaxAttempts=10] -->|Success| Done F1 -->|Failure| FCheck{Attempts\nexhausted?} FCheck -->|No| F1 FCheck -->|Yes| Log[Log: switching to slow phase] end Log --> Phase2 subgraph Phase2 [Phase 2 — Slow Retries, Unlimited] S1[Execute Activity\ninitialInterval=5m\nunlimited attempts] -->|Success| Done S1 -->|Failure| SWait[Temporal waits 5m\nthen retries] SWait --> S1 end Done([Return result]) ``` The following describes each step: 1. The Workflow first tries the Activity with a fast policy: 1-second initial interval and a maximum of 10 total attempts. 2. If the Activity succeeds during the fast phase, the Workflow returns the result immediately. 3. If all fast attempts are exhausted, the Workflow logs a warning and transitions to the slow phase. 4. In the slow phase, the Workflow executes the Activity with a 5-minute fixed interval and unlimited retries. The Temporal Service manages the wait between attempts. 5. When the Activity eventually succeeds — after the downstream system recovers — the Workflow returns the result. ## Implementation ### Two-phase workflow retry management The key change between phases is the retry interval and attempt count. In Phase 1, the Temporal Service manages a fast set of retries: short interval, bounded attempts. In Phase 2, the Temporal Service manages a slow set of retries: long fixed interval, unlimited attempts. **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy from temporalio.exceptions import ActivityError import activities @workflow.defn class FastSlowRetryWorkflow: @workflow.run async def run(self, request: str) -> str: # Phase 1: fast retries fast_policy = RetryPolicy( initial_interval=timedelta(seconds=1), backoff_coefficient=1.5, maximum_interval=timedelta(seconds=30), maximum_attempts=10, ) try: return await workflow.execute_activity( activities.call_downstream, request, start_to_close_timeout=timedelta(seconds=30), retry_policy=fast_policy, ) except ActivityError: workflow.logger.warning( "Fast retries exhausted — switching to slow retry phase", extra={"request": request}, ) # Phase 2: slow retries slow_policy = RetryPolicy( initial_interval=timedelta(minutes=5), backoff_coefficient=1.0, ) return await workflow.execute_activity( activities.call_downstream, request, start_to_close_timeout=timedelta(seconds=30), retry_policy=slow_policy, ) ``` **Go** ```go // workflow.go package downstream import ( "time" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/workflow" ) func FastSlowRetryWorkflow(ctx workflow.Context, request string) (string, error) { log := workflow.GetLogger(ctx) // Phase 1: fast retries fastCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 1.5, MaximumInterval: 30 * time.Second, MaximumAttempts: 10, }, }) var result string err := workflow.ExecuteActivity(fastCtx, CallDownstream, request).Get(fastCtx, &result) if err != nil { log.Warn("Fast retries exhausted — switching to slow retry phase", "request", request) // Phase 2: slow retries slowCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{ InitialInterval: 5 * time.Minute, BackoffCoefficient: 1.0, // MaximumAttempts defaults to 0 (unlimited) }, }) err = workflow.ExecuteActivity(slowCtx, CallDownstream, request).Get(slowCtx, &result) } return result, err } ``` **Java** ```java // FastSlowRetryWorkflowImpl.java import io.temporal.activity.ActivityOptions; import io.temporal.common.RetryOptions; import io.temporal.failure.ActivityFailure; import io.temporal.workflow.Workflow; import java.time.Duration; public class FastSlowRetryWorkflowImpl implements FastSlowRetryWorkflow { @Override public String run(String request) { // Phase 1: fast retries DownstreamActivities fastActivities = Workflow.newActivityStub( DownstreamActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setBackoffCoefficient(1.5) .setMaximumInterval(Duration.ofSeconds(30)) .setMaximumAttempts(10) .build()) .build() ); try { return fastActivities.callDownstream(request); } catch (ActivityFailure e) { Workflow.getLogger(getClass()).warn( "Fast retries exhausted — switching to slow retry phase: " + request ); // Phase 2: slow retries DownstreamActivities slowActivities = Workflow.newActivityStub( DownstreamActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofMinutes(5)) .setBackoffCoefficient(1.0) // setMaximumAttempts not set — defaults to unlimited .build()) .build() ); return slowActivities.callDownstream(request); } } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; const fastDownstream = wf.proxyActivities({ startToCloseTimeout: '30s', retry: { initialInterval: '1s', backoffCoefficient: 1.5, maximumInterval: '30s', maximumAttempts: 10, }, }); const slowDownstream = wf.proxyActivities({ startToCloseTimeout: '30s', retry: { initialInterval: '5m', backoffCoefficient: 1, // maximumAttempts defaults to unlimited }, }); export async function fastSlowRetryWorkflow(request: string): Promise { // Phase 1: fast retries try { return await fastDownstream.callDownstream(request); } catch { wf.log.warn('Fast retries exhausted — switching to slow retry phase', { request }); // Phase 2: slow retries return await slowDownstream.callDownstream(request); } } ``` ## Tuning the phases Adjust the phase parameters to match the characteristics of your downstream system: | Parameter | Typical value | Rationale | | :--- | :--- | :--- | | Phase 1 `InitialInterval` | 1–5 seconds | Recover from transient errors within seconds | | Phase 1 `BackoffCoefficient` | 1.5–2.0 | Spread retries to avoid overwhelming a briefly degraded system | | Phase 1 `MaximumAttempts` | 5–20 | Enough attempts to cover a short transient period | | Phase 2 `InitialInterval` | 1–15 minutes | Long enough to avoid hammering a down system; short enough to recover promptly | | Phase 2 `BackoffCoefficient` | 1.0 | Keeps the interval fixed; the default 2.0 would exponentially increase delays between slow-phase attempts | Phase 2 runs indefinitely by default. If the business process has a maximum wait time, add a `ScheduleToCloseTimeout` or use a Workflow execution timeout to impose an outer bound. ## Best practices - **Log the phase transition.** The transition from fast to slow is a meaningful signal that the downstream system may have a sustained problem. Log it with enough context — request identifier, attempt count, timestamp — to aid diagnosis. - **Leave `MaximumAttempts` unset in Phase 2.** Omitting `MaximumAttempts` (or setting it to 0) gives the slow phase unlimited retries. The Temporal Service manages the wait between attempts via `InitialInterval`; the Workflow blocks until the Activity eventually succeeds. - **Combine with Retry Alerting via Metrics.** Add a metric counter inside the Activity to surface slow-phase attempts to on-call teams. See [Retry Alerting via Metrics](/design-patterns/retry-metrics). ## Common pitfalls - **Catching too broadly in Phase 1.** Catch `ActivityError` specifically. Catching all exceptions in Phase 1 may swallow errors that should propagate immediately (such as `CancelledError` in Python or a `PanicError` in Go). - **Setting `MaximumAttempts` in Phase 2.** If you set a finite `MaximumAttempts` in the slow phase, it will eventually exhaust and propagate a failure to the Workflow. Only add a limit if the business process has a defined maximum wait time; in that case, pair it with a `ScheduleToCloseTimeout` to make the budget explicit. - **Using exponential backoff in Phase 2.** The default `BackoffCoefficient` is 2.0, which doubles the interval with each attempt. Set `BackoffCoefficient=1.0` in the slow phase to keep the interval fixed and predictable. ## Related ### Patterns - [Retry Alerting via Metrics](/design-patterns/retry-metrics): Emit a metric in the slow-phase Activity to surface sustained failures to on-call teams. - [Delayed Retry](/design-patterns/delayed-retry): Override the retry interval per error type using `nextRetryDelay` on `ApplicationFailure`. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### References - [Understanding Workflow Retries and Failures](https://community.temporal.io/t/understanding-workflow-retries-and-failures/122) - [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice) --- # Fixed Count of Retries Source: https://docs.temporal.io/design-patterns/fixed-count-retries > Cap the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource. > **ℹ️ TLDR:** > Set `MaximumAttempts` on the `RetryPolicy` to **cap how many times Temporal will attempt an Activity**. Use this when each attempt consumes a paid API call, a rate-limited token, or any scarce resource where unbounded retries translate directly to unbounded cost. ## Overview The Fixed Count of Retries pattern caps the total number of Activity execution attempts by setting `MaximumAttempts` on the `RetryPolicy`. Use it when each attempt consumes a paid API call, a rate-limited token, or any scarce resource where unbounded retries translate directly to unbounded cost. ## Problem Temporal's default retry policy retries Activities indefinitely with exponential backoff. This is appropriate for most infrastructure failures, but it creates problems when the Activity calls a paid third-party API: - A credit card authorization that fails due to a transient network error will be retried dozens of times, each attempt charging a per-call fee. - A generative AI API with a per-token pricing model will accumulate costs silently while the Workflow waits. - A rate-limited partner API will exhaust its quota across all callers if one Workflow retries without bound. Without a cap, a single stuck Workflow can generate costs that are orders of magnitude larger than the intended spend. ## Solution Set `MaximumAttempts` on the `RetryPolicy` passed to the Activity call. Temporal counts the initial attempt and each retry toward the limit. When the limit is reached, Temporal stops retrying and delivers an `ActivityError` to the Workflow. The Workflow can catch that error and decide whether to fail, alert, or escalate. ```mermaid sequenceDiagram participant Workflow participant Temporal as Temporal Service participant API as Payment API Workflow->>Temporal: Schedule activity (MaximumAttempts=3) Temporal->>+API: Attempt 1 API-->>-Temporal: Failure Note over Temporal: Retry 1 of 2 Temporal->>+API: Attempt 2 API-->>-Temporal: Failure Note over Temporal: Retry 2 of 2 Temporal->>+API: Attempt 3 API-->>-Temporal: Failure Note over Temporal: MaximumAttempts reached — no more retries Temporal-->>Workflow: ActivityError Workflow->>Workflow: Handle failure (alert, compensate, or escalate) ``` The following describes each step: 1. The Workflow schedules the Activity with a `RetryPolicy` that caps attempts at 3. 2. The Temporal Service executes the Activity. On failure, it schedules a retry. 3. After 3 total attempts (1 initial + 2 retries), Temporal delivers an `ActivityError` to the Workflow. 4. The Workflow catches the error and handles it — logging, compensating, or escalating — rather than accumulating further cost. ## Implementation ### Capping attempts Set `maximum_attempts` (Python), `MaximumAttempts` (Go / Java), or `maximumAttempts` (TypeScript) on the retry policy. The count includes the initial attempt, so `maximum_attempts=3` means one attempt plus two retries. **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy from temporalio.exceptions import ActivityError, RetryState import activities @workflow.defn class PaymentWorkflow: @workflow.run async def run(self, order_id: str) -> str: try: return await workflow.execute_activity( activities.charge_payment_api, order_id, start_to_close_timeout=timedelta(seconds=10), retry_policy=RetryPolicy(maximum_attempts=3), ) except ActivityError as e: if e.retry_state == RetryState.MAXIMUM_ATTEMPTS_REACHED: # All retries exhausted — handle the failure here. # Options: alert on-call, trigger a compensation activity, or escalate to a human. workflow.logger.error( "Payment failed: all 3 attempts exhausted", extra={"order_id": order_id}, ) raise ``` **Go** ```go // workflow.go package payments import ( "errors" "time" enumspb "go.temporal.io/api/enums/v1" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/workflow" ) func PaymentWorkflow(ctx workflow.Context, orderID string) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, RetryPolicy: &temporal.RetryPolicy{ MaximumAttempts: 3, }, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, ChargePaymentAPI, orderID).Get(ctx, &result) if err != nil { var actErr *temporal.ActivityError if errors.As(err, &actErr) && actErr.RetryState() == enumspb.RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED { // All retries exhausted — handle the failure here. // Options: alert on-call, trigger a compensation activity, or escalate to a human. workflow.GetLogger(ctx).Error("Payment failed: all 3 attempts exhausted", "orderID", orderID) } return "", err } return result, nil } ``` **Java** ```java // PaymentWorkflowImpl.java import io.temporal.activity.ActivityOptions; import io.temporal.api.enums.v1.RetryState; import io.temporal.common.RetryOptions; import io.temporal.failure.ActivityFailure; import io.temporal.workflow.Workflow; import java.time.Duration; public class PaymentWorkflowImpl implements PaymentWorkflow { private final PaymentActivities activities = Workflow.newActivityStub( PaymentActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(3) .build()) .build() ); @Override public String run(String orderId) { try { return activities.chargePaymentApi(orderId); } catch (ActivityFailure e) { if (e.getRetryState() == RetryState.RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED) { // All retries exhausted — handle the failure here. // Options: alert on-call, trigger a compensation activity, or escalate to a human. Workflow.getLogger(getClass()).error( "Payment failed: all 3 attempts exhausted: " + orderId, e); } throw e; } } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; const { chargePaymentApi } = wf.proxyActivities({ startToCloseTimeout: '10s', retry: { maximumAttempts: 3 }, }); export async function paymentWorkflow(orderId: string): Promise { try { return await chargePaymentApi(orderId); } catch (err) { if (err instanceof wf.ActivityFailure && err.retryState === wf.RetryState.MAXIMUM_ATTEMPTS_REACHED) { // All retries exhausted — handle the failure here. // Options: alert on-call, trigger a compensation activity, or escalate to a human. wf.log.error('Payment failed: all 3 attempts exhausted', { orderId }); } throw err; } } ``` ### Disabling retries entirely Set `maximum_attempts=1` to disable retries. The Activity starts once and any failure is immediately delivered to the Workflow. This is appropriate when the operation is not idempotent and a second attempt would cause a duplicate side effect such as a double charge or a duplicate email. **Python** ```python # workflows.py result = await workflow.execute_activity( activities.send_welcome_email, user_id, start_to_close_timeout=timedelta(seconds=10), retry_policy=RetryPolicy(maximum_attempts=1), ) ``` **Go** ```go // workflow.go ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, RetryPolicy: &temporal.RetryPolicy{ MaximumAttempts: 1, }, } ``` **Java** ```java // Workflow.java ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(1) .build()) .build() ``` **TypeScript** ```typescript // workflows.ts const { sendWelcomeEmail } = wf.proxyActivities({ startToCloseTimeout: '10s', retry: { maximumAttempts: 1 }, }); ``` If a Worker crashes after the API call succeeds but before the result is recorded, Temporal will not retry — the call is lost. An idempotency key (a stable identifier derived from the Workflow and Activity IDs) lets the downstream system detect and discard duplicates if a retry is needed in future. If you have idempotency keys, there's little need to cap retries at 1. ## Best practices - **Match the cap to the cost model.** If the API charges per call, set `maximum_attempts` to the maximum number of calls you are willing to pay for per Workflow execution. - **Combine with `StartToCloseTimeout`.** A per-attempt timeout prevents a slow response from consuming the entire retry budget on a single hanging call. - **Catch `ActivityError` in the Workflow.** Handle the exhausted-retries case explicitly — log, alert, compensate, or escalate — rather than letting it fail the Workflow silently. - **Use idempotency keys.** When retrying, it's vital to have downstream systems detect and discard duplicate calls to avoid duplicate downstream effects. - **Prefer non-retryable errors for structural failures.** If the failure is not transient (for example, invalid input), mark it as non-retryable rather than relying solely on `maximum_attempts`. ## Common pitfalls - **Confusing `MaximumAttempts` with allowed retry count.** `MaximumAttempts=3` means 3 total attempts (1 initial + 2 retries), not 3 retries after the initial attempt. - **Setting no timeout alongside a low attempt cap.** Without `StartToCloseTimeout`, a single hanging attempt can block all retries for minutes or hours. - **Ignoring the `ActivityError` in the Workflow.** Exhausted retries raise an error in the Workflow. If you do not catch it, the Workflow fails without any compensation or alerting. - **Disabling retries on operations without safeguards.** `maximum_attempts=1` on a call means any failure — including a Worker crash after the API responded — results in a permanent gap. ## Related ### Patterns - [Non-Retryable Errors](/design-patterns/non-retryable-errors): Fail immediately for errors that will never succeed regardless of how many times you try. - [Fixed Wall-Time Retries](/design-patterns/fixed-wall-time-retries): Bound by total elapsed time rather than attempt count. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### References - [Temporal Retry Policies](/encyclopedia/retry-policies) - [Idempotency and Durable Execution](https://temporal.io/blog/idempotency-and-durable-execution) --- # Fixed Wall-Time Retries Source: https://docs.temporal.io/design-patterns/fixed-wall-time-retries > Bound the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many individual attempts occur. > **ℹ️ TLDR:** > Set `ScheduleToCloseTimeout` on the Activity call to enforce a hard time budget across all retry attempts. Use this when a business SLA requires the Activity to **succeed or fail within a defined window**, regardless of how many individual attempts occur. ## Overview The Fixed Wall-Time Retries pattern enforces a maximum total elapsed time across all Activity retry attempts using `ScheduleToCloseTimeout`. Use it when a business process must _succeed or fail_ within a defined time budget, regardless of how many individual attempts occur. ## Problem `StartToCloseTimeout` limits how long a single Activity attempt may run before Temporal cancels it and schedules a retry. It does not limit how long retries collectively may run. A process with `StartToCloseTimeout=5m` and the default unlimited retry policy can run for days — each attempt times out at 5 minutes, then Temporal waits for the backoff delay and tries again, indefinitely. When a business SLA exists and violating that SLA is a failure such as a payment must charge in two minutes or less, an authorization check must complete within 30 seconds — you need a hard outer boundary that Temporal enforces automatically without requiring the Workflow to track elapsed time itself. ## Solution Set `ScheduleToCloseTimeout` on the Activity call options. It starts when the Activity is first scheduled and expires when the clock runs out, regardless of how many attempts have occurred. When the timeout expires, the Temporal Service marks the Activity Execution as timed out and delivers an `ActivityError` to the Workflow. No further retries are scheduled. A timeout does not forcibly stop Activity code that is already running, so an Activity that runs past the budget must heartbeat and handle cancellation to stop cooperatively. ```mermaid sequenceDiagram participant Workflow participant Temporal as Temporal Service participant Service as Downstream Service Note over Temporal: ScheduleToCloseTimeout = 2m starts Workflow->>Temporal: Schedule activity Temporal->>+Service: Attempt 1 (StartToClose = 30s) Service-->>-Temporal: Failure Note over Temporal: Backoff delay (5s) Temporal->>+Service: Attempt 2 (StartToClose = 30s) Service-->>-Temporal: Failure Note over Temporal: Backoff delay (5s), ...retries continue Note over Temporal: 2m elapsed — ScheduleToClose exceeded Temporal-->>Workflow: ActivityError (schedule-to-close timeout) Workflow->>Workflow: Handle SLA breach ``` The following describes each step: 1. The two minute budget clock starts the moment the Workflow schedules the Activity. 2. Each attempt runs up to 30 seconds (`StartToCloseTimeout`). On failure, Temporal waits the backoff delay and retries. 3. Retries continue until either the Activity succeeds or the two minute budget is exhausted. 4. When the budget expires, Temporal delivers an `ActivityError` to the Workflow, which can log, alert, or compensate. ## Implementation ### Enforcing a 2-minute SLA Set both `schedule_to_close_timeout` (the total budget) and `start_to_close_timeout` (the per-attempt cap). The retry policy controls the interval between attempts. Temporal stops retrying automatically when the budget runs out. **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy from temporalio.exceptions import ActivityError, TimeoutError, TimeoutType import activities @workflow.defn class PaymentAuthWorkflow: @workflow.run async def run(self, transaction_id: str) -> str: try: return await workflow.execute_activity( activities.authorize_transaction, transaction_id, schedule_to_close_timeout=timedelta(minutes=2), # total budget start_to_close_timeout=timedelta(seconds=30), # per attempt retry_policy=RetryPolicy( initial_interval=timedelta(seconds=5), backoff_coefficient=1.5, maximum_interval=timedelta(seconds=30), ), ) except ActivityError as e: cause = e.__cause__ if isinstance(cause, TimeoutError) and cause.type == TimeoutType.SCHEDULE_TO_CLOSE: workflow.logger.error( "Authorization failed — 2-minute SLA breached", extra={"transaction_id": transaction_id}, ) raise ``` **Go** ```go // workflow.go package shipment import ( "errors" "time" enumspb "go.temporal.io/api/enums/v1" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/workflow" ) func PaymentAuthWorkflow(ctx workflow.Context, transactionID string) (string, error) { ao := workflow.ActivityOptions{ ScheduleToCloseTimeout: 2 * time.Minute, // total budget StartToCloseTimeout: 30 * time.Second, // per attempt RetryPolicy: &temporal.RetryPolicy{ InitialInterval: 5 * time.Second, BackoffCoefficient: 1.5, MaximumInterval: 30 * time.Second, }, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, AuthorizeTransaction, transactionID).Get(ctx, &result) if err != nil { var timeoutErr *temporal.TimeoutError if errors.As(err, &timeoutErr) && timeoutErr.TimeoutType() == enumspb.TIMEOUT_TYPE_SCHEDULE_TO_CLOSE { workflow.GetLogger(ctx).Error( "Authorization failed — 2-minute SLA breached", "transactionID", transactionID, ) } return "", err } return result, nil } ``` **Java** ```java // ShipmentNotificationWorkflowImpl.java import io.temporal.activity.ActivityOptions; import io.temporal.api.enums.v1.TimeoutType; import io.temporal.common.RetryOptions; import io.temporal.failure.ActivityFailure; import io.temporal.failure.TimeoutFailure; import io.temporal.workflow.Workflow; import java.time.Duration; public class PaymentAuthWorkflowImpl implements PaymentAuthWorkflow { private final PaymentActivities activities = Workflow.newActivityStub( PaymentActivities.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofMinutes(2)) // total budget .setStartToCloseTimeout(Duration.ofSeconds(30)) // per attempt .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(5)) .setBackoffCoefficient(1.5) .setMaximumInterval(Duration.ofSeconds(30)) .build()) .build() ); @Override public String run(String transactionId) { try { return activities.authorizeTransaction(transactionId); } catch (ActivityFailure e) { if (e.getCause() instanceof TimeoutFailure tf && tf.getTimeoutType() == TimeoutType.TIMEOUT_TYPE_SCHEDULE_TO_CLOSE) { Workflow.getLogger(getClass()).error( "Authorization failed — 2-minute SLA breached: " + transactionId, e ); } throw e; } } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; const { authorizeTransaction } = wf.proxyActivities({ scheduleToCloseTimeout: '2m', // total budget startToCloseTimeout: '30s', // per attempt retry: { initialInterval: '5s', backoffCoefficient: 1.5, maximumInterval: '30s', }, }); export async function paymentAuthWorkflow(transactionId: string): Promise { try { return await authorizeTransaction(transactionId); } catch (err) { if (err instanceof wf.ActivityFailure) { const cause = err.cause; if (cause instanceof wf.TimeoutFailure && cause.type === wf.TimeoutType.SCHEDULE_TO_CLOSE) { wf.log.error('Authorization failed — 2-minute SLA breached', { transactionId }); } } throw err; } } ``` ### Short SLA without a per-attempt timeout For tighter budgets — such as a 30 second authorization window — you may omit `StartToCloseTimeout` and let `ScheduleToCloseTimeout` act as the only bound. Temporal requires at least one timeout to be set; `ScheduleToCloseTimeout` alone satisfies that requirement. **Python** ```python # workflows.py result = await workflow.execute_activity( activities.authorize_transaction, transaction_id, schedule_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy( initial_interval=timedelta(seconds=3), backoff_coefficient=1.5, ), ) ``` **Go** ```go // workflow.go ao := workflow.ActivityOptions{ ScheduleToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{ InitialInterval: 3 * time.Second, BackoffCoefficient: 1.5, }, } ``` **Java** ```java // Workflow.java ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(3)) .setBackoffCoefficient(1.5) .build()) .build() ``` **TypeScript** ```typescript // workflows.ts const { authorizeTransaction } = wf.proxyActivities({ scheduleToCloseTimeout: '30s', retry: { initialInterval: '3s', backoffCoefficient: 1.5, }, }); ``` ## Best practices - **Set both timeouts for clarity.** Use `ScheduleToCloseTimeout` as the total SLA and `StartToCloseTimeout` as a per-attempt safety valve. Omitting `StartToCloseTimeout` means a single slow response can consume the entire budget. - **Cap `MaximumInterval` well below the SLA.** If `MaximumInterval` is 2 hours and the SLA is 24 hours, only 12 retries are possible. Tune the interval so the backoff plateaus at a value that allows meaningful retries within the budget. - **Handle `ActivityError` explicitly.** When the SLA expires, Temporal delivers an error to the Workflow. Catch it to send an alert, trigger a compensation, or record a breach in an audit log. - **Distinguish SLA breaches from transient errors.** Inspect the error cause — check that the `ActivityError`'s cause is a `TimeoutError` with `TimeoutType.SCHEDULE_TO_CLOSE` (Python) or a `TimeoutFailure` with `TimeoutType.SCHEDULE_TO_CLOSE` (TypeScript) or `TIMEOUT_TYPE_SCHEDULE_TO_CLOSE` (Go/Java) to separate an SLA breach from an application failure. This lets you log or alert specifically on SLA violations rather than treating all activity errors the same way. ## Common pitfalls - **Not accounting for `ScheduleToStart` delay in the budget.** `ScheduleToCloseTimeout` begins when the Activity is first scheduled, which includes the time the task waits in the queue before a Worker picks it up. Under high load or insufficient Worker capacity, tasks can sit in the queue for seconds or minutes before the first attempt starts — consuming SLA budget before any work is done. Provision Workers with enough capacity for peak traffic, or use autoscaling, to keep `ScheduleToStart` latency negligible relative to the SLA window. - **Using `StartToCloseTimeout` alone for SLA enforcement.** A downstream system that responds slowly but never fully times out can keep resetting the per-attempt clock indefinitely. - **Setting `ScheduleToCloseTimeout` shorter than `StartToCloseTimeout`.** If the total budget is shorter than a single attempt's maximum, the first attempt cannot finish within the budget — the Temporal Service times out the Activity Execution and returns an error before any attempt can succeed. - **Ignoring the breach in the Workflow.** Letting the `ActivityError` propagate without handling it means SLA breaches go unlogged and uncompensated. - **Not accounting for backoff delays in the budget.** The total time includes both attempt durations and the backoff delays between them. A 1-hour budget with a 30-minute initial interval and coefficient 2.0 leaves room for only one or two attempts. ## Related ### Patterns - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Bound by attempt count rather than elapsed time. - [Delayed Retry](/design-patterns/delayed-retry): Fixed-interval retry when the downstream unavailability window is known. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### References - [Activity Timeouts](https://temporal.io/blog/activity-timeouts) - [Temporal Retry Policies](/encyclopedia/retry-policies) --- # Local Activities Source: https://docs.temporal.io/design-patterns/local-activities > Run Activity functions in-process inside the Workflow Task, eliminating all server scheduling round-trips. Best for short, idempotent Activities on a latency-sensitive path. > **ℹ️ TLDR:** > **Replace regular Activities with Local Activities to eliminate Temporal server round-trips and reduce per-Activity overhead to near zero.** This is best for short-lived, idempotent Activities that complete well within the Workflow Task timeout. Each Activity you convert saves approximately 50 ms of scheduling overhead on Temporal Cloud. If you haven't measured a latency problem, start with regular Activities—they are easier to debug, rate-limit, and monitor. ## Overview A **Local Activity** executes the Activity function directly inside the Worker process that is currently running the Workflow Task. The result is recorded as part of the same Workflow Task completion event, so no additional server calls occur between Activity invocations. ```mermaid sequenceDiagram participant W as Worker participant S as Temporal Server rect rgb(230, 235, 250) Note over W,S: Regular Activity —
server round-trips on each call W->>S: Schedule ActivityTask S-->>W: Dispatch ActivityTask W->>W: Execute activity W->>S: Complete ActivityTask S-->>W: Resume WorkflowTask end rect rgb(220, 245, 225) Note over W,S: Local Activity —
runs in-process, zero server calls W->>W: Execute activity function directly Note over W: Result bundled in WorkflowTask completion end ``` **Numbered walkthrough:** 1. A regular Activity follows a five-step exchange with the Temporal server: schedule, dispatch, execute, complete, then resume the Workflow Task. On Temporal Cloud, each round-trip adds approximately 50 ms. 2. A Local Activity bypasses all of that. The Workflow Task scheduler on the Worker invokes the Activity function directly. The result is folded into the WorkflowTask completion event sent at the end of the task. 3. For a Workflow with three serial Activities, switching all three to Local Activities can save 150 ms or more while keeping the exact same business logic. ## Problem Workflows that call several short Activities in sequence accumulate significant latency from server scheduling overhead. A validation-and-reservation flow with three Activities may complete in 850 ms even when the actual computation takes only milliseconds. Every regular Activity incurs at least two server round-trips (schedule + complete), which becomes a bottleneck on latency-sensitive paths. ## Solution Use `workflow.execute_local_activity` (Python), `proxyLocalActivities` (TypeScript), `workflow.ExecuteLocalActivity` (Go), or `Workflow.newLocalActivityStub` with `LocalActivityOptions` (Java) to run Activity functions in-process. The Activity executes inside the same Workflow Task and its result is available immediately—no scheduling overhead. Local Activities are subject to the Workflow Task timeout (default 10 seconds) rather than an independent start-to-close timeout. Always configure the `scheduleToCloseTimeout` option (not `startToCloseTimeout`) to set an upper bound. **Python** ```python # workflows.py from temporalio import workflow from datetime import timedelta from activities import validate_transaction, reserve_funds, settle_transaction LOCAL_ACTIVITY_TIMEOUT = timedelta(seconds=10) @workflow.defn class TransactionWorkflow: @workflow.run async def run(self, req: TransactionRequest) -> Transaction: # All three activities run in-process — no server round-trips. tx = await workflow.execute_local_activity( validate_transaction, req, schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT, ) tx = await workflow.execute_local_activity( reserve_funds, tx, schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT, ) return await workflow.execute_local_activity( settle_transaction, tx, schedule_to_close_timeout=LOCAL_ACTIVITY_TIMEOUT, ) ``` **TypeScript** ```typescript // workflows.ts import { proxyLocalActivities } from "@temporalio/workflow"; import type * as activities from "./activities"; const { validateTransaction, reserveFunds, settleTransaction } = proxyLocalActivities({ scheduleToCloseTimeout: "10s" }); export async function transactionWorkflow( req: TransactionRequest, ): Promise { // All three activities run in-process — no server round-trips. let tx = await validateTransaction(req); tx = await reserveFunds(tx); return settleTransaction(tx); } ``` **Go** ```go // workflows.go func TransactionWorkflow(ctx workflow.Context, req TransactionRequest) (Transaction, error) { localCtx := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{ ScheduleToCloseTimeout: 10 * time.Second, }) // All three activities run in-process — no server round-trips. var tx Transaction if err := workflow.ExecuteLocalActivity(localCtx, ValidateTransaction, req).Get(localCtx, &tx); err != nil { return Transaction{}, err } if err := workflow.ExecuteLocalActivity(localCtx, ReserveFunds, tx).Get(localCtx, &tx); err != nil { return Transaction{}, err } if err := workflow.ExecuteLocalActivity(localCtx, SettleTransaction, tx).Get(localCtx, &tx); err != nil { return Transaction{}, err } return tx, nil } ``` **Java** ```java // TransactionWorkflow.java public class Impl implements TransactionWorkflow { // All activities run as local — no server round-trips. private final Activities activities = Workflow.newLocalActivityStub( Activities.class, LocalActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .build() ); @Override public Shared.Transaction processTransaction(Shared.TransactionRequest req) { Shared.Transaction tx = activities.validateTransaction(req); tx = activities.reserveFunds(tx); return activities.settleTransaction(tx); } } ``` ## When to use **Good fit:** - Short-lived Activities that complete in milliseconds or a few seconds - Idempotent operations safe to re-execute if a Workflow Task fails - Hot-path Workflows where end-to-end latency is a product requirement - CPU-bound or in-memory computations that do not need a separate worker pool **Poor fit:** - Activities that may run longer than the Workflow Task timeout (default 10 seconds) - Activities that require heartbeating to detect stuck executions - Operations with long retry back-off intervals—retry timers still schedule server events, reducing the latency benefit - Non-idempotent operations where re-execution on Worker crash would cause harm - Operations that need rate limiting or routing through task queue capacity controls—Local Activities bypass the task queue entirely and cannot be throttled by the server ## Benefits and trade-offs | | Regular Activity | Local Activity | |---|---|---| | Server round-trips | 2–4 per call | 0 | | Latency overhead | ~50 ms per call (Temporal Cloud) | Near zero | | Heartbeat support | Yes | No | | Execution timeout | `StartToCloseTimeout` | `ScheduleToCloseTimeout` | | Retry semantics | Independent per attempt | Entire Workflow Task re-executes on failure | | Dedicated worker pool | Yes (separate poller) | No (shares Workflow Task thread) | | Rate limiting / routing | Yes (task queue capacity) | No (bypasses task queue) | | Visible in Temporal UI | Full Activity history event | Recorded in Workflow Task event; no standalone Activity task in UI | ## Best practices - **Design for at-least-once execution.** If a Workflow Task fails after a Local Activity completes but before the task is persisted, all Local Activities in that task re-execute on the next attempt. Your Activity logic must tolerate this. - **Keep each Local Activity short.** Aim for well under 5 seconds to leave headroom for retries within the same Workflow Task, which has a 10-second default timeout. - **Avoid blocking signal and update handlers.** While a Local Activity executes, the Workflow Task is occupied. Incoming signals and updates accumulate in the server buffer and are not processed until the next task begins. - **Set retry policy carefully.** Large `initialInterval` or `maximumInterval` values in a retry policy still cause the SDK to schedule server-side timer events, which partially defeats the latency benefit. ## Common pitfalls - **Exceeding the Workflow Task timeout.** If a Local Activity takes longer than the Workflow Task timeout (default 10 seconds), the entire task times out and retries—including any Local Activities that already completed in memory during that task. - **Assuming exactly-once semantics.** Unlike regular Activities, a Local Activity does not get its own persisted history event until the Workflow Task completes. A crashed Worker causes the whole task to re-run. This compounds when Local Activities are chained: if a Worker crashes after the third of five sequential Local Activities, all five re-execute on the next attempt. If you need a durable checkpoint between each step, use regular Activities instead. - **Long retry intervals.** Each retry attempt with back-off creates a server-side timer event. For truly short Activities, use a tight `scheduleToCloseTimeout` and allow immediate retries rather than spaced-out back-off. ## Related ### Patterns - [Early Return + Local Activities](/design-patterns/early-return-local-activities) — adds an Update-with-Start early-response path on top of Local Activities for minimum first-response latency - [Early Return](/design-patterns/early-return) — returns a response to the caller before the Workflow finishes, independent of Local Activities - [Eager Workflow Start](/design-patterns/eager-workflow-start) — eliminates the server Matching step when starting a Workflow for additional latency reduction - [Long Running Activity](/design-patterns/long-running-activity) — the right choice when Activities need heartbeating and long execution windows --- # Long-Running Activity - Tracking Progress and Handling Cancellation with Heartbeats Source: https://docs.temporal.io/design-patterns/long-running-activity > Long-running Activities report progress via heartbeats and enable resumption after failures with cancellation support. ## Overview The Activity Heartbeat pattern enables long-running Activities to report progress, handle cancellation gracefully, and resume from the last checkpoint after failures. Heartbeats inform Temporal that the Activity is still alive and allow storing progress details that survive Worker restarts. ## Problem In long-running operations, you often need Activities that process large datasets or perform time-consuming operations (minutes to hours), report progress to avoid appearing stuck or timing out, resume from the last checkpoint after Worker crashes or restarts, handle cancellation requests gracefully and clean up resources, and avoid reprocessing already-completed work. Without heartbeats, you must set very long Activity timeouts that delay failure detection, reprocess entire batches from the beginning on failures, accept no visibility into Activity progress, risk zombie Activities that appear alive but are stuck, and implement custom checkpointing and recovery logic. ## Solution Activity heartbeats periodically report progress to the Temporal Service. The heartbeat details are persisted and available to retry attempts, enabling resumption from the last checkpoint. Heartbeat timeouts detect stuck Activities faster than execution timeouts. ```mermaid sequenceDiagram participant Workflow participant Activity participant Temporal Workflow->>+Activity: Start (with heartbeat timeout) loop Process items Activity->>Activity: Process item Activity->>Temporal: heartbeat(progress) Note over Temporal: Store progress end alt Activity completes Activity-->>-Workflow: Result else Worker crashes Note over Activity: Heartbeat timeout expires Temporal->>+Activity: Retry on new worker Activity->>Temporal: getHeartbeatDetails() Temporal-->>Activity: Last progress Activity->>Activity: Resume from checkpoint Activity-->>-Workflow: Result end ``` The following describes each step in the diagram: 1. The Workflow starts the Activity with a heartbeat timeout. 2. The Activity processes items in a loop, heartbeating progress after each batch. 3. If the Activity completes normally, it returns the result to the Workflow. 4. If the Worker crashes, the heartbeat timeout expires and Temporal retries the Activity on a new Worker. The new attempt retrieves the last heartbeat details and resumes from the checkpoint. ## Implementation ### Basic progress tracking The following implementation processes a large file line by line, heartbeating every 100 lines. On retry, it retrieves the last processed line number and skips ahead: **Python** ```python # activities.py from temporalio import activity @activity.defn async def process_large_file(file_path: str) -> None: details = activity.info().heartbeat_details start_line = details[0] if details else 0 with open(file_path, "r") as f: for i, line in enumerate(f): if i < start_line: continue process_line(line) if (i + 1) % 100 == 0: activity.heartbeat(i + 1) ``` **Go** ```go // activities.go func ProcessLargeFile(ctx context.Context, filePath string) error { startLine := 0 if activity.HasHeartbeatDetails(ctx) { if err := activity.GetHeartbeatDetails(ctx, &startLine); err != nil { return err } } file, err := os.Open(filePath) if err != nil { return err } defer file.Close() scanner := bufio.NewScanner(file) currentLine := 0 for scanner.Scan() { if currentLine < startLine { currentLine++ continue } processLine(scanner.Text()) currentLine++ if currentLine%100 == 0 { activity.RecordHeartbeat(ctx, currentLine) } } return scanner.Err() } ``` **Java** ```java // FileProcessingActivityImpl.java @ActivityInterface public interface FileProcessingActivity { void processLargeFile(String filePath); } public class FileProcessingActivityImpl implements FileProcessingActivity { @Override public void processLargeFile(String filePath) { ActivityExecutionContext context = Activity.getExecutionContext(); Optional lastProcessedLine = context.getHeartbeatDetails(Integer.class); int startLine = lastProcessedLine.orElse(0); try (BufferedReader reader = new BufferedReader(new FileReader(filePath))) { for (int i = 0; i < startLine; i++) { reader.readLine(); } String line; int currentLine = startLine; while ((line = reader.readLine()) != null) { processLine(line); currentLine++; if (currentLine % 100 == 0) { context.heartbeat(currentLine); } } } } } ``` **TypeScript** ```typescript // activities.ts import { heartbeat, activityInfo } from '@temporalio/activity'; import { createReadStream } from 'fs'; import { createInterface } from 'readline'; export async function processLargeFile(filePath: string): Promise { const startLine = activityInfo().heartbeatDetails ?? 0; const rl = createInterface({ input: createReadStream(filePath), }); let currentLine = 0; for await (const line of rl) { if (currentLine < startLine) { currentLine++; continue; } processLine(line); currentLine++; if (currentLine % 100 === 0) { heartbeat(currentLine); } } } ``` The heartbeat details call retrieves the last heartbeat value from a previous attempt. If this is the first attempt, there are no details and the Activity starts from line 0. The Activity heartbeats every 100 lines, storing the current line number as the checkpoint. ### Handle cancellation The following implementation adds cancellation support. The Activity checks for cancellation on each heartbeat and cleans up resources before exiting: **Python** ```python # activities.py import asyncio from temporalio import activity @activity.defn async def process_large_file(file_path: str) -> None: details = activity.info().heartbeat_details current_line = details[0] if details else 0 try: with open(file_path, "r") as f: for i, line in enumerate(f): if i < current_line: continue activity.heartbeat(i) process_line(line) current_line = i + 1 except asyncio.CancelledError: cleanup_resources() raise ``` **Go** ```go // activities.go func ProcessLargeFile(ctx context.Context, filePath string) error { currentLine := 0 if activity.HasHeartbeatDetails(ctx) { if err := activity.GetHeartbeatDetails(ctx, ¤tLine); err != nil { return err } } file, err := os.Open(filePath) if err != nil { return err } defer file.Close() scanner := bufio.NewScanner(file) for scanner.Scan() { if currentLine > 0 { currentLine-- continue } activity.RecordHeartbeat(ctx, currentLine) // Check if the Activity has been cancelled select { case <-ctx.Done(): cleanupResources() return ctx.Err() default: } processLine(scanner.Text()) currentLine++ } return scanner.Err() } ``` **Java** ```java // FileProcessingActivityImpl.java public class FileProcessingActivityImpl implements FileProcessingActivity { @Override public void processLargeFile(String filePath) { ActivityExecutionContext context = Activity.getExecutionContext(); Optional lastProcessedLine = context.getHeartbeatDetails(Integer.class); int currentLine = lastProcessedLine.orElse(0); try (BufferedReader reader = new BufferedReader(new FileReader(filePath))) { for (int i = 0; i < currentLine; i++) { reader.readLine(); } String line; while ((line = reader.readLine()) != null) { context.heartbeat(currentLine); processLine(line); currentLine++; } } catch (ActivityCompletionException e) { cleanupResources(); throw e; } } } ``` **TypeScript** ```typescript // activities.ts import { heartbeat, activityInfo, sleep } from '@temporalio/activity'; import { CancelledFailure } from '@temporalio/common'; import { createReadStream } from 'fs'; import { createInterface } from 'readline'; export async function processLargeFile(filePath: string): Promise { const startLine = activityInfo().heartbeatDetails ?? 0; const rl = createInterface({ input: createReadStream(filePath), }); let currentLine = 0; try { for await (const line of rl) { if (currentLine < startLine) { currentLine++; continue; } heartbeat(currentLine); processLine(line); currentLine++; } } catch (err) { if (err instanceof CancelledFailure) { cleanupResources(); } throw err; } } ``` Cancellation is delivered to the Activity when it heartbeats. In Java, the next `heartbeat()` call throws an `ActivityCompletionException` (an `ActivityCanceledException` for cancellation). In TypeScript, cancellation is delivered as a `CancelledFailure` via `sleep()` or `Context.current().cancelled`. In Python, cancellation is delivered as an `asyncio.CancelledError`. In Go, the context is cancelled and `ctx.Done()` becomes readable. The catch/error handling block performs cleanup before re-throwing the error. ### Complex progress state The following implementation tracks multiple progress fields -- processed count, failed count, and the last processed ID: **Python** ```python # activities.py from dataclasses import dataclass from temporalio import activity @dataclass class ProgressState: processed_count: int = 0 failed_count: int = 0 last_processed_id: str = "" @activity.defn async def process_batch(item_ids: list[str]) -> dict: details = activity.info().heartbeat_details progress = details[0] if details else ProgressState() start_index = ( item_ids.index(progress.last_processed_id) + 1 if progress.last_processed_id else 0 ) for i in range(start_index, len(item_ids)): item_id = item_ids[i] try: await process_item(item_id) progress.processed_count += 1 except Exception: progress.failed_count += 1 progress.last_processed_id = item_id activity.heartbeat(progress) return { "processed_count": progress.processed_count, "failed_count": progress.failed_count, } ``` **Go** ```go // activities.go type ProgressState struct { ProcessedCount int `json:"processedCount"` FailedCount int `json:"failedCount"` LastProcessedID string `json:"lastProcessedId"` } type BatchResult struct { ProcessedCount int `json:"processedCount"` FailedCount int `json:"failedCount"` } func ProcessBatch(ctx context.Context, itemIDs []string) (BatchResult, error) { progress := ProgressState{} if activity.HasHeartbeatDetails(ctx) { if err := activity.GetHeartbeatDetails(ctx, &progress); err != nil { return BatchResult{}, err } } startIndex := 0 if progress.LastProcessedID != "" { for i, id := range itemIDs { if id == progress.LastProcessedID { startIndex = i + 1 break } } } for i := startIndex; i < len(itemIDs); i++ { itemID := itemIDs[i] if err := processItem(ctx, itemID); err != nil { progress.FailedCount++ } else { progress.ProcessedCount++ } progress.LastProcessedID = itemID activity.RecordHeartbeat(ctx, progress) } return BatchResult{ ProcessedCount: progress.ProcessedCount, FailedCount: progress.FailedCount, }, nil } ``` **Java** ```java // BatchProcessingActivityImpl.java public class BatchProcessingActivityImpl implements BatchProcessingActivity { static class ProgressState { int processedCount; int failedCount; String lastProcessedId; } @Override public BatchResult processBatch(List itemIds) { ActivityExecutionContext context = Activity.getExecutionContext(); Optional details = context.getHeartbeatDetails(ProgressState.class); ProgressState progress = details.orElse(new ProgressState()); int startIndex = itemIds.indexOf(progress.lastProcessedId) + 1; for (int i = startIndex; i < itemIds.size(); i++) { String itemId = itemIds.get(i); try { processItem(itemId); progress.processedCount++; } catch (Exception e) { progress.failedCount++; } progress.lastProcessedId = itemId; context.heartbeat(progress); } return new BatchResult(progress.processedCount, progress.failedCount); } } ``` **TypeScript** ```typescript // activities.ts import { heartbeat, activityInfo } from '@temporalio/activity'; interface ProgressState { processedCount: number; failedCount: number; lastProcessedId: string; } export async function processBatch(itemIds: string[]): Promise { const saved: ProgressState = activityInfo().heartbeatDetails ?? { processedCount: 0, failedCount: 0, lastProcessedId: '', }; const startIndex = saved.lastProcessedId ? itemIds.indexOf(saved.lastProcessedId) + 1 : 0; const progress = { ...saved }; for (let i = startIndex; i < itemIds.length; i++) { const itemId = itemIds[i]; try { await processItem(itemId); progress.processedCount++; } catch { progress.failedCount++; } progress.lastProcessedId = itemId; heartbeat(progress); } return { processedCount: progress.processedCount, failedCount: progress.failedCount }; } ``` The progress state object stores all the checkpoint data needed to resume. On retry, the Activity finds the index of the last processed ID and starts from the next item. Each heartbeat stores the full progress state, so the next attempt has everything it needs to resume. ## When to use The Heartbeat pattern is a good fit for batch processing of large datasets, file uploads and downloads with progress tracking, database migrations or bulk operations, long-running computations (ML training, video encoding), external API polling with multiple attempts, and any Activity running longer than 30 seconds. It is not a good fit for quick operations (under 10 seconds), operations that cannot be checkpointed, Activities requiring exact-once semantics without idempotency, or real-time streaming (use Workflows instead). ## Benefits and trade-offs Heartbeats enable fault tolerance by resuming from the last checkpoint after failures. Heartbeat timeouts detect stuck Activities faster than execution timeouts. You gain visibility into Activity progress in real-time. Activities can handle cancellation gracefully and clean up resources. Completed work is not reprocessed, and Activities can move between Workers. The trade-offs to consider are that frequent heartbeats increase network traffic. You must implement checkpointing logic and state management. You must handle partial reprocessing of the last checkpoint (idempotency). You need to balance heartbeat frequency between responsiveness and overhead. Heartbeat details have size limits, so you should avoid large objects. ## Comparison with alternatives | Approach | Progress tracking | Resumable | Cancellation | Complexity | | :--- | :--- | :--- | :--- | :--- | | Heartbeat | Yes | Yes | Graceful | Medium | | Long Timeout | No | No | Delayed | Low | | Child Workflows | Yes | Yes | Immediate | High | | Local Activity | No | No | N/A | Low | ## Best practices - **Set heartbeat timeout.** Configure to 2-3x the expected heartbeat interval. - **Heartbeat at regular intervals.** Balance between responsiveness (every 10-30 seconds) and overhead. - **Checkpoint strategically.** Save progress at meaningful boundaries (records, pages, chunks). - **Keep details small.** Store minimal state (IDs, offsets, counts), not full objects. - **Handle idempotency.** Ensure reprocessing the last checkpoint is safe. - **Check cancellation.** Heartbeat regularly to detect cancellation quickly. - **Clean up on cancel.** Handle cancellation errors appropriately: catch `ActivityCompletionException` (Java), `CancelledFailure` (TypeScript), `asyncio.CancelledError` (Python), or check `ctx.Done()` (Go). - **Log progress.** Log heartbeat details for debugging and monitoring. - **Test resumption.** Verify Activities resume correctly after simulated failures. - **Avoid heartbeat spam.** Do not heartbeat on every iteration of tight loops. ## Common pitfalls - **Missing HeartbeatTimeout.** Without a HeartbeatTimeout, Temporal cannot detect a stuck or crashed Worker until the StartToCloseTimeout expires. Always set HeartbeatTimeout shorter than StartToCloseTimeout. - **Heartbeating too infrequently.** Cancellation is only delivered on the next heartbeat. If the Activity heartbeats every 5 minutes, cancellation takes up to 5 minutes to propagate. - **Not resuming from heartbeat progress on retry.** When an Activity retries, retrieve the last heartbeat details -- `context.getHeartbeatDetails()` (Java), `activityInfo().heartbeatDetails` (TypeScript), `activity.info().heartbeat_details` (Python), or `activity.GetHeartbeatDetails()` (Go) -- and resume from the last checkpoint instead of restarting from scratch. - **Catching the wrong exception for cancellation.** Cancellation is SDK-specific. Inside the Activity, the `heartbeat()` call throws an `ActivityCompletionException` (Java), cancellation surfaces as a `CancelledFailure` (TypeScript) or an `asyncio.CancelledError` (Python), and the context reports `ctx.Err()` returning `context.Canceled` (Go). The `CanceledFailure` type is what the Workflow observes as the cause of the resulting `ActivityFailure`, not what the Activity body catches. ## Related ### Patterns - **[Saga Pattern](/design-patterns/saga-pattern)**: Compensating transactions with long-running steps. - **[Polling](/design-patterns/polling)**: Heartbeating Activity for frequent polling. ### Sample code ### Java - [Heartbeating Activity Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/heartbeatingactivity) -- Complete batch processing implementation. - [Auto-Heartbeating](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/autoheartbeat) -- Automatic heartbeating via interceptor. ### TypeScript - [Activities Cancellation and Heartbeating](https://github.com/temporalio/samples-typescript/tree/main/activities-cancellation-heartbeating) -- Activity cancellation and heartbeat-based resumption. ### Python - [Hello Cancellation](https://github.com/temporalio/samples-python/blob/main/hello/hello_cancellation.py) -- Activity heartbeating with cancellation handling. - [Custom Decorator Heartbeat](https://github.com/temporalio/samples-python/blob/main/custom_decorator/activity_utils.py) -- Automatic heartbeating via decorator. ### Go - [Cancellation](https://github.com/temporalio/samples-go/tree/main/cancellation) -- Workflow and Activity cancellation with heartbeating. --- # MapReduce Tree Source: https://docs.temporal.io/design-patterns/mapreduce-tree > Recursively splits a dataset into a binary tree of Child Workflows, processes leaves in parallel, then aggregates results back up the tree. > **ℹ️ TLDR:** > Recursively split a record set into a tree of child Workflows — each node fans out to N sub-slices (two by default) — process every leaf in parallel, and signal results back up the tree to the root. Use this when you need **maximum throughput** for an embarrassingly parallel workload and downstream systems can absorb an unbounded burst of concurrent requests. ## Overview The MapReduce Tree pattern processes a large record set with maximum parallelism by recursively splitting it into smaller chunks and distributing each chunk to a child Workflow. Results are signalled back up the tree to the parent. It is best suited for embarrassingly parallel workloads where speed matters more than rate limiting. ## Problem Both the [Batch Iterator](/design-patterns/batch-iterator) and [Sliding Window](/design-patterns/sliding-window) patterns bound concurrency, which limits throughput. When you need to process a large record set as fast as possible and downstream systems can handle the load, you want to fan out work across as many concurrent processors as possible without a fixed window. You also need a way to handle record sets larger than what a single Workflow's concurrency limits allow, without pre-partitioning data into fixed chunks before the job starts. ## Solution A Node Workflow receives a slice of records. If the slice is small enough (at or below a configurable `leafThreshold`), it starts one Leaf Workflow per record. Otherwise it splits the slice into `n` sub-slices and starts `n` Node child Workflows recursively. Each Leaf Workflow runs the actual processing Activity and signals its result back to its parent Node. Each Node aggregates the results it receives and signals them up to its own parent. The Root Node returns the final aggregated result. ```mermaid flowchart TD Records["📋 Full record set"] Root["Root Node Workflow\n(depth=0)"] Node1["Node Workflow\n(depth=1, chunk 1)"] Node2["Node Workflow\n(depth=1, chunk 2)"] L1["Leaf Workflow\n(record A)"] L2["Leaf Workflow\n(record B)"] L3["Leaf Workflow\n(record C)"] L4["Leaf Workflow\n(record D)"] L5["Leaf Workflow\n(record E)"] L6["Leaf Workflow\n(record F)"] Records --> Root Root -->|"split → chunk 1"| Node1 Root -->|"split → chunk 2"| Node2 Node1 --> L1 Node1 --> L2 Node1 --> L3 Node2 --> L4 Node2 --> L5 Node2 --> L6 L1 -->|"Signal result"| Node1 L2 -->|"Signal result"| Node1 L3 -->|"Signal result"| Node1 L4 -->|"Signal result"| Node2 L5 -->|"Signal result"| Node2 L6 -->|"Signal result"| Node2 Node1 -->|"Signal result"| Root Node2 -->|"Signal result"| Root ``` The following describes each step in the diagram: 1. The Root Node Workflow receives the full record set and `depth=0`. 2. Because the record set is larger than `leafThreshold`, the Root splits it into N chunks and starts N child Node Workflows (two in this example, but the split factor is configurable). 3. Each Node Workflow receives its chunk and checks its size against `leafThreshold`. In this example, each chunk is small enough, so each Node starts one Leaf Workflow per record. 4. Each Leaf Workflow calls the `processLeaf` Activity and, when complete, signals its result back to its parent Node using `signalExternalWorkflow`. 5. Each Node collects all leaf results via signal handlers, aggregates them, and signals the aggregated result back to the Root. 6. The Root collects both node results and returns the final aggregate. ## Implementation The following examples show how each SDK implements the MapReduce Tree pattern. **TypeScript** ```typescript // workflows.ts import { condition, defineSignal, getExternalWorkflowHandle, proxyActivities, setHandler, startChild, workflowInfo, } from "@temporalio/workflow"; import type * as activities from "./activities"; import { TASK_QUEUE, LEAF_THRESHOLD, MAX_DEPTH, RESULT_SIGNAL } from "./shared"; const { processLeaf } = proxyActivities({ startToCloseTimeout: "30 seconds", }); interface ResultPayload { id: string; results: string[]; } export const resultSignal = defineSignal<[ResultPayload]>(RESULT_SIGNAL); export async function leafWorkflow( record: string, parentWorkflowId: string ): Promise { const result = await processLeaf(record); // Signal result back to parent node. const parent = getExternalWorkflowHandle(parentWorkflowId); await parent.signal(resultSignal, { id: record, results: [result] }); } export async function nodeWorkflow( records: string[], depth: number = 0, parentWorkflowId: string = "" ): Promise { if (depth > MAX_DEPTH) { throw new Error(`Tree depth exceeded ${MAX_DEPTH}`); } const myId = workflowInfo().workflowId; const collectedResults: string[] = []; let received = 0; let expected = 0; setHandler(resultSignal, (payload: ResultPayload) => { collectedResults.push(...payload.results); received++; }); if (records.length <= LEAF_THRESHOLD) { // Start one leaf per record. expected = records.length; for (const record of records) { void startChild(leafWorkflow, { args: [record, myId], workflowId: `${myId}/leaf-${record}`, taskQueue: TASK_QUEUE, }); } } else { // Split and recurse. const mid = Math.floor(records.length / 2); const chunks = [records.slice(0, mid), records.slice(mid)]; expected = chunks.length; for (let i = 0; i < chunks.length; i++) { void startChild(nodeWorkflow, { args: [chunks[i], depth + 1, myId], workflowId: `${myId}/node-d${depth + 1}-${i}`, taskQueue: TASK_QUEUE, }); } } // Wait until all expected signals have arrived. await condition(() => received >= expected); // Signal aggregated results up to parent (if this is not the root). if (parentWorkflowId) { const parent = getExternalWorkflowHandle(parentWorkflowId); await parent.signal(resultSignal, { id: myId, results: collectedResults }); } return collectedResults; } ``` **Python** ```python # workflows.py from dataclasses import dataclass from datetime import timedelta from temporalio import workflow from activities import process_leaf from shared import TASK_QUEUE, LEAF_THRESHOLD, MAX_DEPTH, RESULT_SIGNAL @dataclass class ResultPayload: id: str results: list[str] @workflow.defn class LeafWorkflow: @workflow.run async def run(self, record: str, parent_workflow_id: str) -> None: result = await workflow.execute_activity( process_leaf, record, start_to_close_timeout=timedelta(seconds=30), ) parent = workflow.get_external_workflow_handle(parent_workflow_id) await parent.signal(RESULT_SIGNAL, ResultPayload(id=record, results=[result])) @workflow.defn class NodeWorkflow: def __init__(self) -> None: self._collected: list[str] = [] self._received = 0 @workflow.signal(name=RESULT_SIGNAL) def node_result(self, payload: ResultPayload) -> None: self._collected.extend(payload.results) self._received += 1 @workflow.run async def run( self, records: list[str], depth: int = 0, parent_workflow_id: str = "", ) -> list[str]: if depth > MAX_DEPTH: raise RuntimeError(f"Tree depth exceeded {MAX_DEPTH}") my_id = workflow.info().workflow_id expected = 0 if len(records) <= LEAF_THRESHOLD: expected = len(records) for record in records: await workflow.start_child_workflow( LeafWorkflow.run, args=[record, my_id], id=f"{my_id}/leaf-{record}", task_queue=TASK_QUEUE, ) else: mid = len(records) // 2 chunks = [records[:mid], records[mid:]] expected = len(chunks) for i, chunk in enumerate(chunks): await workflow.start_child_workflow( NodeWorkflow.run, args=[chunk, depth + 1, my_id], id=f"{my_id}/node-d{depth+1}-{i}", task_queue=TASK_QUEUE, ) await workflow.wait_condition(lambda: self._received >= expected) if parent_workflow_id: parent = workflow.get_external_workflow_handle(parent_workflow_id) await parent.signal( RESULT_SIGNAL, ResultPayload(id=my_id, results=self._collected) ) return self._collected ``` **Go** ```go // workflows.go package main import ( "fmt" "time" "go.temporal.io/sdk/workflow" ) // ResultSignal and ResultPayload are declared in shared.go. func LeafWorkflow(ctx workflow.Context, record string, parentWorkflowID string) error { ao := workflow.ActivityOptions{StartToCloseTimeout: 30 * time.Second} ctx = workflow.WithActivityOptions(ctx, ao) var result string if err := workflow.ExecuteActivity(ctx, ProcessLeaf, record).Get(ctx, &result); err != nil { return err } payload := ResultPayload{ID: record, Results: []string{result}} return workflow.SignalExternalWorkflow(ctx, parentWorkflowID, "", ResultSignal, payload).Get(ctx, nil) } func NodeWorkflow(ctx workflow.Context, records []string, depth int, parentWorkflowID string) ([]string, error) { if depth > MaxDepth { return nil, fmt.Errorf("tree depth exceeded %d", MaxDepth) } myID := workflow.GetInfo(ctx).WorkflowExecution.ID resultCh := workflow.GetSignalChannel(ctx, ResultSignal) var collected []string expected := 0 if len(records) <= LeafThreshold { expected = len(records) for _, record := range records { cwo := workflow.ChildWorkflowOptions{ WorkflowID: myID + "/leaf-" + record, TaskQueue: TaskQueue, } workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), LeafWorkflow, record, myID) } } else { mid := len(records) / 2 chunks := [][]string{records[:mid], records[mid:]} expected = len(chunks) for i, chunk := range chunks { cwo := workflow.ChildWorkflowOptions{ WorkflowID: fmt.Sprintf("%s/node-d%d-%d", myID, depth+1, i), TaskQueue: TaskQueue, } workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), NodeWorkflow, chunk, depth+1, myID) } } for i := 0; i < expected; i++ { var payload ResultPayload resultCh.Receive(ctx, &payload) collected = append(collected, payload.Results...) } if parentWorkflowID != "" { payload := ResultPayload{ID: myID, Results: collected} if err := workflow.SignalExternalWorkflow(ctx, parentWorkflowID, "", ResultSignal, payload).Get(ctx, nil); err != nil { return collected, err } } return collected, nil } ``` **Java** ```java // NodeWorkflow.java import io.temporal.workflow.*; import java.util.*; @WorkflowInterface public interface NodeWorkflow { @WorkflowMethod List run(List records, int depth, String parentWorkflowId); @SignalMethod void nodeResult(String id, List results); } // NodeWorkflowImpl.java public class NodeWorkflowImpl implements NodeWorkflow { private final List collected = new ArrayList<>(); private int received = 0; @Override public void nodeResult(String id, List results) { collected.addAll(results); received++; } @Override public List run(List records, int depth, String parentWorkflowId) { if (depth > Shared.MAX_DEPTH) { throw new RuntimeException("Tree depth exceeded " + Shared.MAX_DEPTH); } String myId = Workflow.getInfo().getWorkflowId(); int expected; if (records.size() <= Shared.LEAF_THRESHOLD) { for (String record : records) { ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder() .setWorkflowId(myId + "/leaf-" + record) .setTaskQueue(Shared.TASK_QUEUE) .build(); LeafWorkflow leaf = Workflow.newChildWorkflowStub(LeafWorkflow.class, opts); Async.procedure(leaf::run, record, myId); } expected = records.size(); } else { int mid = records.size() / 2; List> chunks = List.of(records.subList(0, mid), records.subList(mid, records.size())); for (int i = 0; i < chunks.size(); i++) { ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder() .setWorkflowId(String.format("%s/node-d%d-%d", myId, depth + 1, i)) .setTaskQueue(Shared.TASK_QUEUE) .build(); NodeWorkflow child = Workflow.newChildWorkflowStub(NodeWorkflow.class, opts); Async.function(child::run, chunks.get(i), depth + 1, myId); } expected = chunks.size(); } final int exp = expected; Workflow.await(() -> received >= exp); if (parentWorkflowId != null && !parentWorkflowId.isEmpty()) { ExternalWorkflowStub parent = Workflow.newUntypedExternalWorkflowStub(parentWorkflowId, ""); parent.signal(Shared.RESULT_SIGNAL, myId, new ArrayList<>(collected)); } return collected; } } ``` ## Best practices - **Set a `leafThreshold` to control tree depth.** A threshold of 3–10 records per leaf is typical. Too small a threshold creates excessive Workflow overhead; too large prevents full parallelism. - **Set a `MAX_DEPTH` guard.** Recursive fan-out without a depth limit can produce extremely deep trees for large record sets. Fail fast if depth exceeds your expected maximum (for example, `log2(totalRecords / leafThreshold) + 2`). - **Avoid external writes in Node Workflows.** Node Workflows only aggregate results from children. Leaf Workflows perform the actual work. Keeping the roles separate prevents duplicate external writes if a Node is retried. - **Use signals for result aggregation, not return values.** A parent cannot directly await a child started in a previous Workflow run. Signals decouple the result delivery from the parent-child lifetime, making the pattern resilient to replays. - **Skip the reduce phase if results are not needed.** If you only need the side effects of processing each record (writes to a database, messages sent), omit the signal-back entirely and set `PARENT_CLOSE_POLICY_ABANDON` on all children. - **Consider replacing Leaf Workflows with Activities for lighter workloads.** Leaf Workflows give each record its own Event History, independent cancellation, and dedicated visibility in the UI — useful when per-record observability matters. If those properties are not required, executing the `processLeaf` Activity directly from a Node Workflow reduces overhead. A successful child workflow and a successful Activity each add three Events to the parent's history, but a Leaf Workflow also maintains a separate Event History of its own and emits an extra signal back to the Node, so child workflows produce more overall Events than Activities. The Temporal documentation recommends starting with Activities and adopting child workflows only when there is a clear need. ## Common pitfalls - **Thundering herd.** The MapReduce Tree fans out exponentially. For large record sets, all leaf Activities start nearly simultaneously. Ensure your downstream system can absorb the burst, or switch to [Sliding Window](/design-patterns/sliding-window) for rate limiting. - **Signal storms.** If thousands of leaves all signal a single Node at the same time, the Node's signal queue can become a bottleneck. A two-level tree (Root → Nodes → Leaves) distributes this load; a deeper tree helps even more. - **History bloat in the Root Workflow.** Each child start and signal received adds events to the Root's history. For very large record sets, consider adding an extra tree level to keep the Root from receiving too many direct signals. - **Attempting external/downstream writes from Node Workflows.** Nodes may be retried. Any external write in a Node Workflow will be executed multiple times. Keep all side effects in Leaf Workflows (or Activities called by Leaves). ## Related ### Patterns - [Fan-Out with Child Workflows](/design-patterns/fanout-child-workflows) — simpler flat fan-out for smaller record sets - [Sliding Window](/design-patterns/sliding-window) — bounded concurrency with rate limiting - [Child Workflows pattern](/design-patterns/child-workflows) — core concepts for parent/child coordination - [Temporal limits reference](/cloud/limits) --- # Non-Retryable Errors Source: https://docs.temporal.io/design-patterns/non-retryable-errors > Mark error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying indefinitely. > **ℹ️ TLDR:** > Raise a non-retryable `ApplicationError` from the Activity — or list error types in the `RetryPolicy` — so **the Temporal Activity fails fast instead of retrying indefinitely**. Use this for permanent failures such as invalid input, missing records, or authorization errors where repeating the same call will never succeed. ## Overview The Non-Retryable Errors pattern marks specific error types so Temporal stops retrying immediately when one is raised. Use it for failures where the root cause is structural — invalid input, a missing record, an authorization problem — where repeating the same call will never produce a different result. ## Problem Temporal retries all Activity failures by default. For transient infrastructure errors such as network timeouts or service restarts, this is the right behavior. But some failures are permanent: no amount of retrying will fix them. Retrying a permanent failure wastes time and resources: - A transfer to a non-existent account number will fail on attempt 1, 2, and 3 in exactly the same way. - An API call with a malformed request body will be rejected every time. - A request from a revoked API key will receive an authorization error on every attempt. With the default unlimited retry policy, the Workflow waits through exponential backoff delays — minutes to hours — before eventually delivering the error to the Workflow, when it could have failed in milliseconds. ## Solution Raise a non-retryable error from inside the Activity when the failure is known to be permanent. Temporal detects the error type and skips all remaining retries, delivering the failure to the Workflow immediately. There are two complementary mechanisms: 1. **Mark the error as non-retryable at the throw site** — the Activity explicitly signals that this specific failure should not be retried. 2. **Register non-retryable error types in the `RetryPolicy`** — the Workflow declares which error type names should never be retried, regardless of how the Activity raises them. Both mechanisms can be used together. ```mermaid flowchart TD Workflow -->|Schedule Activity| Activity Activity -->|Raises error| Check{Is error type\nnon-retryable?} Check -->|Yes — marked at throw site\nor listed in RetryPolicy| Fail([Deliver ActivityError\nto Workflow immediately]) Check -->|No| Retry[Schedule retry\nwith backoff] Retry --> Activity Fail --> Handle[Workflow handles error:\nlog, compensate, or escalate] ``` The following describes each path: 1. The Activity raises an error. Temporal inspects whether the error type is non-retryable. 2. If non-retryable — either because the Activity flagged it or the `RetryPolicy` lists the type — Temporal delivers the `ActivityError` to the Workflow without delay. 3. If retryable, Temporal schedules another attempt after the configured backoff. 4. The Workflow catches the `ActivityError` and handles it according to the business logic. ## Implementation ### Marking an error as non-retryable at the throw site Use the SDK's `ApplicationError` (or equivalent) with the non-retryable flag. Temporal propagates the error type name and the flag to the Workflow without retrying. **Python** ```python # activities.py from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def process_order(order_id: str) -> str: response = await http_client.post(f"/orders/{order_id}/process") if response.status_code == 404: raise ApplicationError( f"Order {order_id} not found", type="OrderNotFoundError", non_retryable=True, ) if response.status_code == 422: raise ApplicationError( f"Order {order_id} rejected: {response.json().get('detail', 'validation error')}", type="ValidationError", non_retryable=True, ) response.raise_for_status() return response.json()["confirmation_id"] ``` **Go** ```go // activities.go package orders import ( "context" "fmt" "go.temporal.io/sdk/temporal" ) func ProcessOrder(ctx context.Context, orderID string) (string, error) { resp, err := httpClient.Post(fmt.Sprintf("/orders/%s/process", orderID)) if err != nil { return "", err } if resp.StatusCode == 404 { return "", temporal.NewNonRetryableApplicationError( fmt.Sprintf("order %s not found", orderID), "OrderNotFoundError", nil, ) } if resp.StatusCode == 422 { return "", temporal.NewNonRetryableApplicationError( fmt.Sprintf("order %s rejected: %s", orderID, resp.ErrorDetail), "ValidationError", nil, ) } return resp.ConfirmationID, nil } ``` **Java** ```java // ProcessOrderActivityImpl.java import io.temporal.failure.ApplicationFailure; public class ProcessOrderActivityImpl implements ProcessOrderActivity { @Override public String processOrder(String orderId) { HttpResponse response = httpClient.post("/orders/" + orderId + "/process"); if (response.getStatusCode() == 404) { throw ApplicationFailure.newNonRetryableFailure( "Order " + orderId + " not found", "OrderNotFoundError" ); } if (response.getStatusCode() == 422) { throw ApplicationFailure.newNonRetryableFailure( "Order " + orderId + " rejected: " + response.getErrorDetail(), "ValidationError" ); } return response.getConfirmationId(); } } ``` **TypeScript** ```typescript // activities.ts import { ApplicationFailure } from '@temporalio/activity'; export async function processOrder(orderId: string): Promise { const response = await fetch(`/orders/${orderId}/process`, { method: 'POST' }); if (response.status === 404) { throw ApplicationFailure.nonRetryable( `Order ${orderId} not found`, 'OrderNotFoundError', ); } if (response.status === 422) { const body = await response.json(); throw ApplicationFailure.nonRetryable( `Order ${orderId} rejected: ${body.detail ?? 'validation error'}`, 'ValidationError', ); } if (!response.ok) throw new Error(`API error ${response.status}`); const body = await response.json(); return body.confirmation_id; } ``` ### Declaring non-retryable types in the RetryPolicy Alternatively, list error type names in the `RetryPolicy` at the Workflow call site. Temporal stops retrying when the Activity raises an error whose type matches any name in the list. This approach separates the retry decision from the Activity code, which is useful when the Activity is shared and the non-retryable classification depends on the caller's context. The Activity raises a standard `ApplicationError` with a type name but without the non-retryable flag — the retry decision is delegated to the Workflow's `RetryPolicy`: **Python** ```python # activities.py @activity.defn async def process_order(order_id: str) -> str: response = await http_client.post(f"/orders/{order_id}/process") if response.status_code == 404: # No non_retryable=True — the RetryPolicy in the Workflow controls retry behavior raise ApplicationError(f"Order {order_id} not found", type="OrderNotFoundError") if response.status_code == 422: raise ApplicationError( f"Order {order_id} rejected: {response.json().get('detail', 'validation error')}", type="ValidationError", ) response.raise_for_status() return response.json()["confirmation_id"] ``` **Go** ```go // activities.go func ProcessOrder(ctx context.Context, orderID string) (string, error) { resp, err := httpClient.Post(fmt.Sprintf("/orders/%s/process", orderID)) if err != nil { return "", err } if resp.StatusCode == 404 { // Use NewApplicationError, not NewNonRetryableApplicationError return "", temporal.NewApplicationError( fmt.Sprintf("order %s not found", orderID), "OrderNotFoundError", ) } if resp.StatusCode == 422 { return "", temporal.NewApplicationError( fmt.Sprintf("order %s rejected: %s", orderID, resp.ErrorDetail), "ValidationError", ) } return resp.ConfirmationID, nil } ``` **Java** ```java // ProcessOrderActivityImpl.java public String processOrder(String orderId) { HttpResponse response = httpClient.post("/orders/" + orderId + "/process"); if (response.getStatusCode() == 404) { // Use newFailure, not newNonRetryableFailure throw ApplicationFailure.newFailure("Order " + orderId + " not found", "OrderNotFoundError"); } if (response.getStatusCode() == 422) { throw ApplicationFailure.newFailure( "Order " + orderId + " rejected: " + response.getErrorDetail(), "ValidationError"); } return response.getConfirmationId(); } ``` **TypeScript** ```typescript // activities.ts export async function processOrder(orderId: string): Promise { const response = await fetch(`/orders/${orderId}/process`, { method: 'POST' }); if (response.status === 404) { // Use ApplicationFailure.create, not .nonRetryable throw ApplicationFailure.create({ message: `Order ${orderId} not found`, type: 'OrderNotFoundError' }); } if (response.status === 422) { const body = await response.json(); throw ApplicationFailure.create({ message: `Order ${orderId} rejected: ${body.detail ?? 'validation error'}`, type: 'ValidationError', }); } if (!response.ok) throw new Error(`API error ${response.status}`); const body = await response.json(); return body.confirmation_id; } ``` The Workflow lists which error type names to never retry: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy import activities @workflow.defn class OrderWorkflow: @workflow.run async def run(self, order_id: str) -> str: return await workflow.execute_activity( activities.process_order, order_id, start_to_close_timeout=timedelta(seconds=10), retry_policy=RetryPolicy( non_retryable_error_types=["OrderNotFoundError", "ValidationError"], ), ) ``` **Go** ```go // workflow.go ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, RetryPolicy: &temporal.RetryPolicy{ NonRetryableErrorTypes: []string{"OrderNotFoundError", "ValidationError"}, }, } ctx = workflow.WithActivityOptions(ctx, ao) ``` **Java** ```java // OrderWorkflowImpl.java private final ProcessOrderActivity activities = Workflow.newActivityStub( ProcessOrderActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setRetryOptions(RetryOptions.newBuilder() .setDoNotRetry("OrderNotFoundError", "ValidationError") .build()) .build() ); ``` **TypeScript** ```typescript // workflows.ts const { processOrder } = wf.proxyActivities({ startToCloseTimeout: '10s', retry: { nonRetryableErrorTypes: ['OrderNotFoundError', 'ValidationError'], }, }); ``` ### Handle non-retryable errors in the Workflow Catch the `ActivityError` in the Workflow to distinguish between permanent failures and transient ones. Inspect the underlying cause to route to the appropriate compensation or escalation path. **Python** ```python # workflows.py from temporalio.exceptions import ActivityError, ApplicationError @workflow.defn class OrderWorkflow: @workflow.run async def run(self, order_id: str) -> str: try: return await workflow.execute_activity( activities.process_order, order_id, start_to_close_timeout=timedelta(seconds=10), retry_policy=RetryPolicy( non_retryable_error_types=["OrderNotFoundError", "ValidationError"], ), ) except ActivityError as e: cause = e.__cause__ if isinstance(cause, ApplicationError): if cause.type == "ValidationError": return f"Order rejected: {cause}" if cause.type == "OrderNotFoundError": return f"Order not found: {cause}" raise ``` **Go** ```go // workflow.go var result string err := workflow.ExecuteActivity(ctx, ProcessOrder, orderID).Get(ctx, &result) if err != nil { var appErr *temporal.ApplicationError if errors.As(err, &appErr) { switch appErr.Type() { case "ValidationError": return "", fmt.Errorf("order rejected: %w", appErr) case "OrderNotFoundError": return "", fmt.Errorf("order not found: %w", appErr) } } return "", err } ``` **Java** ```java // OrderWorkflowImpl.java try { return activities.processOrder(orderId); } catch (ActivityFailure e) { if (e.getCause() instanceof ApplicationFailure appFailure) { switch (appFailure.getType()) { case "ValidationError" -> { return "Order rejected: " + appFailure.getMessage(); } case "OrderNotFoundError" -> { return "Order not found: " + appFailure.getMessage(); } } } throw e; } ``` **TypeScript** ```typescript // workflows.ts try { return await processOrder(orderId); } catch (err) { if (err instanceof wf.ActivityFailure && err.cause instanceof wf.ApplicationFailure) { if (err.cause.type === 'ValidationError') { return `Order rejected: ${err.cause.message}`; } if (err.cause.type === 'OrderNotFoundError') { return `Order not found: ${err.cause.message}`; } } throw err; } ``` ## Best practices - **Validate input before scheduling the Activity.** If the Workflow can detect invalid input upfront — using an `Update` validator or by inspecting the input data — fail fast in the Workflow rather than paying the cost of an Activity execution. - **Use specific error type names.** Generic names like `"Error"` or `"Failure"` match too broadly. Use domain-specific names like `"OrderNotFoundError"` or `"InsufficientFundsError"` so the Workflow can distinguish between failure causes. - **Reserve non-retryable for truly permanent failures.** A rate-limit error (HTTP 429) is transient — the same call will succeed after a delay. A not-found error (HTTP 404) is typically permanent. Match the non-retryable classification to the nature of the error. - **Combine both mechanisms for defence in depth.** Mark the error as non-retryable at the throw site so the Activity is self-describing, and also list the type in the `RetryPolicy` so the classification is enforced even if the Activity code changes. ## Common pitfalls - **Marking transient errors as non-retryable.** Network timeouts and service unavailability are transient. Marking them non-retryable removes Temporal's ability to recover automatically. - **Using the error message instead of a type name.** `RetryPolicy.NonRetryableErrorTypes` matches on type names, not message strings. Without a type name, the policy cannot identify the error. - **Swallowing the `ActivityError` without logging.** Non-retryable errors fail fast and silently if you do not catch and log them. Always log the failure before re-raising or returning an error result. - **Confusing non-retryable errors with Workflow failures.** A non-retryable `ActivityError` fails the Activity and delivers the error to the Workflow. The Workflow itself does not fail unless it re-raises the error without catching it. ## Related ### Patterns - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Limit retries for transient errors that are worth retrying a bounded number of times. - [Resumable Activity](/design-patterns/resumable-activity): Park the Workflow and accept a corrected input via Signal when retries are exhausted. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### Guides - [Recover business processes without restarting](/guides/recover-without-restart): Uses `ApplicationFailure.nonRetryable()` throughout a loan pipeline to distinguish permanent failures that need a human fix from transient ones the default Retry Policy already handles. ### References - [Temporal Retry Policies](/encyclopedia/retry-policies) - [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice) --- # Parallel Execution Source: https://docs.temporal.io/design-patterns/parallel-execution > Executes multiple Activities concurrently for maximum throughput with error handling and controlled parallelism. ## Overview The Parallel Execution pattern enables concurrent execution of multiple Activities or Child Workflows to maximize throughput and minimize total execution time. Using Temporal's async APIs, you can launch multiple operations asynchronously and wait for their completion. ## Problem In sequential execution, operations run one after another, causing unnecessary delays when multiple independent operations could run simultaneously. Total execution time equals the sum of all operation durations, resources sit idle while waiting, and batch processing takes hours when it could take minutes. Without parallel execution, you must accept slow sequential processing, implement complex threading or async logic manually, risk inconsistent state management across threads, and handle thread safety and synchronization issues. ## Solution Each SDK provides its own mechanism for launching Activities concurrently and waiting for the results: - **Java**: `Async.function()` schedules Activities that return `Promise` objects. `Promise.allOf()` waits for all of them. - **TypeScript**: Activity proxy functions return native `Promise` objects. `Promise.all()` waits for all of them. - **Python**: `workflow.execute_activity()` returns awaitables. `asyncio.gather()` waits for all of them. - **Go**: `workflow.ExecuteActivity()` returns `Future` objects. You call `.Get()` on each Future to collect results. ```mermaid sequenceDiagram participant Workflow participant Activity1 participant Activity2 participant Activity3 Workflow->>+Activity1: Start async Workflow->>+Activity2: Start async Workflow->>+Activity3: Start async Note over Workflow: Returns immediately with Futures/Promises par Parallel Execution Activity1->>Activity1: Execute Activity2->>Activity2: Execute Activity3->>Activity3: Execute end Activity1-->>-Workflow: Result 1 Activity2-->>-Workflow: Result 2 Activity3-->>-Workflow: Result 3 Workflow->>Workflow: Await all results Note over Workflow: Collect all results ``` The following describes each step in the diagram: 1. The Workflow starts three Activities asynchronously, which returns immediately with Futures or Promises. 2. All three Activities execute in parallel on available Workers. 3. As each Activity completes, its Future or Promise resolves with the result. 4. The Workflow waits until all results are available, then collects them. ## Implementation ### Basic parallel Activities The following implementation starts one Activity per item in a list and waits for all of them to complete: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import process @workflow.defn class ParallelWorkflow: @workflow.run async def run(self, items: list[str]) -> list[str]: results = await asyncio.gather( *[ workflow.execute_activity( process, item, start_to_close_timeout=timedelta(seconds=30), ) for item in items ] ) return list(results) ``` **Go** ```go // parallel_workflow.go func ProcessInParallel(ctx workflow.Context, items []string) ([]string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) futures := make([]workflow.Future, len(items)) for i, item := range items { futures[i] = workflow.ExecuteActivity(ctx, Process, item) } results := make([]string, len(items)) for i, future := range futures { if err := future.Get(ctx, &results[i]); err != nil { return nil, err } } return results, nil } ``` **Java** ```java // ParallelWorkflowImpl.java @WorkflowInterface public interface ParallelWorkflow { @WorkflowMethod List processInParallel(List items); } public class ParallelWorkflowImpl implements ParallelWorkflow { private final ProcessingActivity activity = Workflow.newActivityStub(ProcessingActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); @Override public List processInParallel(List items) { List> promises = new ArrayList<>(); for (String item : items) { Promise promise = Async.function(activity::process, item); promises.add(promise); } Promise.allOf(promises).get(); return promises.stream().map(Promise::get).collect(Collectors.toList()); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { process } = proxyActivities({ startToCloseTimeout: '30s', }); export async function processInParallel(items: string[]): Promise { const promises = items.map((item) => process(item)); return await Promise.all(promises); } ``` Each SDK schedules all Activities before waiting for any results. In Java, `Async.function()` returns a `Promise`; in TypeScript, calling the activity proxy without `await` returns a native `Promise`; in Python, `workflow.execute_activity()` returns an awaitable; and in Go, `workflow.ExecuteActivity()` returns a `Future`. The Workflow then waits for all of them to complete and collects the results. ### Controlled parallelism The following implementation limits the number of concurrent Activities by processing items in batches: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import process @workflow.defn class BatchWorkflow: @workflow.run async def run(self, items: list[str], max_parallel: int) -> list[str]: results: list[str] = [] for i in range(0, len(items), max_parallel): batch = items[i : i + max_parallel] batch_results = await asyncio.gather( *[ workflow.execute_activity( process, item, start_to_close_timeout=timedelta(seconds=30), ) for item in batch ] ) results.extend(batch_results) return results ``` **Go** ```go // batch_workflow.go func ProcessBatch(ctx workflow.Context, items []string, maxParallel int) ([]string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var results []string for i := 0; i < len(items); i += maxParallel { end := i + maxParallel if end > len(items) { end = len(items) } batch := items[i:end] futures := make([]workflow.Future, len(batch)) for j, item := range batch { futures[j] = workflow.ExecuteActivity(ctx, Process, item) } for _, future := range futures { var result string if err := future.Get(ctx, &result); err != nil { return nil, err } results = append(results, result) } } return results, nil } ``` **Java** ```java // BatchWorkflowImpl.java public class BatchWorkflowImpl implements BatchWorkflow { private final ProcessingActivity activity = Workflow.newActivityStub(ProcessingActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); @Override public BatchResult processBatch(List items, int maxParallel) { List results = new ArrayList<>(); for (int i = 0; i < items.size(); i += maxParallel) { int end = Math.min(i + maxParallel, items.size()); List batch = items.subList(i, end); List> promises = batch.stream() .map(item -> Async.function(activity::process, item)) .collect(Collectors.toList()); Promise.allOf(promises).get(); results.addAll(promises.stream().map(Promise::get).collect(Collectors.toList())); } return new BatchResult(results); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { process } = proxyActivities({ startToCloseTimeout: '30s', }); export async function processBatch( items: string[], maxParallel: number ): Promise { const results: string[] = []; for (let i = 0; i < items.length; i += maxParallel) { const batch = items.slice(i, i + maxParallel); const batchResults = await Promise.all(batch.map((item) => process(item))); results.push(...batchResults); } return results; } ``` The Workflow processes items in chunks of `maxParallel`. Each chunk runs in parallel, and the Workflow waits for the entire chunk to complete before starting the next one. This prevents overwhelming Workers or external services. ### Error handling The following implementation wraps each Activity in error handling so that individual failures do not prevent other Activities from completing: **Python** ```python # workflows.py import asyncio from dataclasses import dataclass from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import process @dataclass class Result: item: str output: str | None = None error: str | None = None @workflow.defn class ResilientParallelWorkflow: @workflow.run async def run(self, items: list[str]) -> list[Result]: tasks = [ workflow.execute_activity( process, item, start_to_close_timeout=timedelta(seconds=30), ) for item in items ] outcomes = await asyncio.gather(*tasks, return_exceptions=True) results: list[Result] = [] for item, outcome in zip(items, outcomes): if isinstance(outcome, BaseException): results.append(Result(item=item, error=str(outcome))) else: results.append(Result(item=item, output=outcome)) return results ``` **Go** ```go // resilient_parallel_workflow.go func ProcessWithErrorHandling(ctx workflow.Context, items []string) ([]Result, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) futures := make([]workflow.Future, len(items)) for i, item := range items { futures[i] = workflow.ExecuteActivity(ctx, Process, item) } results := make([]Result, len(items)) for i, future := range futures { var output string if err := future.Get(ctx, &output); err != nil { results[i] = Result{Item: items[i], Error: err.Error()} } else { results[i] = Result{Item: items[i], Output: output} } } return results, nil } ``` **Java** ```java // ResilientParallelWorkflowImpl.java public class ResilientParallelWorkflowImpl implements ParallelWorkflow { private final ProcessingActivity activity = Workflow.newActivityStub(ProcessingActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); @Override public ProcessingReport processWithErrorHandling(List items) { List> promises = new ArrayList<>(); for (String item : items) { Promise promise = Async.function(() -> { try { return activity.process(item); } catch (Exception e) { return Result.failed(item, e.getMessage()); } }); promises.add(promise); } Promise.allOf(promises).get(); List results = promises.stream().map(Promise::get).collect(Collectors.toList()); return new ProcessingReport(results); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { process } = proxyActivities({ startToCloseTimeout: '30s', }); interface Result { item: string; output?: string; error?: string; } export async function processWithErrorHandling(items: string[]): Promise { const settled = await Promise.allSettled(items.map((item) => process(item))); return settled.map((outcome, i) => { if (outcome.status === 'fulfilled') { return { item: items[i], output: outcome.value }; } return { item: items[i], error: String(outcome.reason) }; }); } ``` Each Activity handles its own errors so that the Workflow can collect results from all Activities, including those that failed. In TypeScript, `Promise.allSettled()` is especially convenient for this pattern. In Python, `asyncio.gather()` with `return_exceptions=True` captures errors alongside successes. In Go, each `Future.Get()` call is checked individually for errors. ## When to use The Parallel Execution pattern is a good fit for processing independent items in batches, calling multiple external services simultaneously, fan-out/fan-in patterns, parallel data transformations, concurrent API requests, and multi-step pipelines with independent stages. It is not a good fit for operations with dependencies between them, resource-constrained environments (use controlled parallelism), operations requiring strict ordering, or a single fast operation (the overhead is not worth it). ## Benefits and trade-offs Parallel execution reduces total execution time dramatically and maximizes Worker and external service usage. Temporal handles retries and failures per operation, and you do not need manual thread management or synchronization. The trade-offs to consider are that more concurrent operations require more Workers. Error handling across parallel operations is harder. Parallel execution makes tracing more difficult. You may overwhelm external services without throttling, and storing many Futures or Promises consumes Workflow memory. ## Comparison with alternatives | Approach | Parallelism | Complexity | Control | Use case | | :--- | :--- | :--- | :--- | :--- | | Async Activities | High | Low | Medium | Independent operations | | Sequential | None | Very Low | Full | Dependent operations | | Child Workflows | High | Medium | High | Complex sub-processes | | ContinueAsNew | None | Medium | Full | Large iterations | ## Best practices - **Limit concurrency.** Use batching to avoid overwhelming Workers or external services. - **Handle failures.** Wrap operations in error handling or use Activity retry policies. - **Set timeouts.** Configure appropriate Activity timeouts for parallel operations. - **Monitor resources.** Ensure sufficient Workers for the desired parallelism. - **Aggregate carefully.** Consider memory when collecting large result sets. - **Use Child Workflows.** For complex parallel operations with their own state. - **Test scalability.** Verify performance with realistic parallel loads. - **Rate limit.** Implement throttling for external API calls. - **Support partial results.** Consider returning partial results on some failures. - **Avoid premature blocking.** Schedule all Activities before waiting for any results. ## Common pitfalls - **Exceeding the pending Activities limit.** A single Workflow Execution can have at most 2,000 pending (concurrently running) Activities. Scheduling more causes Workflow Task failures. Batch Activities or use child Workflows for higher concurrency. - **Ignoring errors from individual Activities.** Waiting for all results (for example, `Promise.allOf()` in Java, `Promise.all()` in TypeScript, `asyncio.gather()` in Python) fails on the first error by default. If you need partial results, catch errors inside each async function or use `Promise.allSettled()` / `return_exceptions=True` / per-Future error checking. - **Blowing the 4 MB gRPC message limit.** Scheduling hundreds of Activities in a single Workflow Task can exceed the 4 MB gRPC message size limit if their combined inputs are large. Batch scheduling across multiple Workflow Tasks. - **Not using Continue-As-New for large fan-outs.** Each Activity adds events to history. Hundreds of parallel Activities can quickly approach the 50K event limit. Use Continue-As-New or child Workflows to partition work. ## Related ### Patterns - **[Child Workflows](/design-patterns/child-workflows)**: For complex parallel operations with their own state. - **[Saga Pattern](/design-patterns/saga-pattern)**: Parallel operations with compensation. ### Sample code **Java:** - [HelloParallelActivity](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloParallelActivity.java) — Basic parallel Activity execution. - [HelloAsync](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloAsync.java) — Async execution with Promises. - [Sliding Window Batch](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow) — Controlled parallel Child Workflows. **TypeScript:** - [activities-examples](https://github.com/temporalio/samples-typescript/tree/main/activities-examples) — Activity patterns including parallel execution with `Promise.all()`. **Python:** - [hello_parallel_activity](https://github.com/temporalio/samples-python/blob/main/hello/hello_parallel_activity.py) — Basic parallel Activity execution with `asyncio.gather()`. **Go:** - [splitmerge-future](https://github.com/temporalio/samples-go/tree/main/splitmerge-future) — Parallel Activity execution with Futures. - [splitmerge-selector](https://github.com/temporalio/samples-go/tree/main/splitmerge-selector) — Parallel Activities with Selector for first-completion handling. --- # Performance & Latency Patterns Source: https://docs.temporal.io/design-patterns/performance-latency-patterns > Pattern selection guide for reducing Workflow latency, with a comparison of the round-trips each pattern removes and their combined effect. Temporal Workflows are durable and reliable, but a default implementation—using regular Activities scheduled through the Temporal server—carries inherent latency. Each regular Activity incurs multiple server round-trips, and each new Workflow begins with a Matching Service routing step. On Temporal Cloud, this baseline can reach 850 ms or more for a typical three-Activity workflow. This section covers three complementary patterns that each target a different source of latency. They can be applied individually or combined depending on your requirements. ## Latency sources in a typical Workflow ```mermaid flowchart LR A[Client\nExecuteWorkflow] --> B[Matching Service\nroutes first WFT] B --> C[Worker\nexecutes WFT] C --> D[Server\nschedules Activity] D --> E[Worker\nexecutes Activity] E --> F[Server\nrecords completion] F --> G[Worker\nresumes WFT] G --> H[Repeat per\nActivity] ``` | Source | Overhead | Pattern that removes it | |---|---|---| | Matching Service (first Workflow Task) | ~30–50 ms | [Eager Workflow Start](/design-patterns/eager-workflow-start) | | Activity scheduling round-trip | ~50 ms per Activity | [Local Activities](/design-patterns/local-activities) | | Client waiting for full workflow | Total duration | [Early Return](/design-patterns/early-return) | ## Pattern comparison The numbers below are approximate benchmarks based on a three-Activity transaction workflow running on Temporal Cloud. Actual results vary by region, Activity implementation, and server load. | Pattern | First Response | Total Latency | SDK Support | |---|---|---|---| | Baseline (regular Activities) | ~850 ms | ~850 ms | All | | [Early Return](/design-patterns/early-return) | ~265 ms | ~850 ms | All | | [Local Activities](/design-patterns/local-activities) | ~275 ms | ~275 ms | All | | [Early Return + Local Activities](/design-patterns/early-return-local-activities) | ~160 ms | ~275 ms | All | | [Eager Workflow Start](/design-patterns/eager-workflow-start) + Local Activities | ~265 ms | ~265 ms | Go, Java, Python | | Early Return + Local Activities + Eager Start | ~160 ms | ~265 ms | Go, Java, Python | **First Response** is the time until the client receives an actionable result. **Total Latency** is the time until the Workflow fully completes. > **💡 TypeScript users:** > Eager Workflow Start is not available in the TypeScript SDK, but the latency gap is small (~30–50 ms per Workflow start). [Local Activities](/design-patterns/local-activities) and [Early Return + Local Activities](/design-patterns/early-return-local-activities) are fully supported and achieve competitive results: ~275 ms total latency and ~160 ms first-response latency respectively. ## Patterns in this section - [Local Activities](/design-patterns/local-activities): Run Activity functions in-process inside the Workflow Task, eliminating all server scheduling round-trips. Best for short, idempotent Activities on a latency-sensitive path. - [Early Return + Local Activities](/design-patterns/early-return-local-activities): Extends Early Return by running Phase 1 Activities as Local Activities. The client receives its response after Phase 1 completes entirely in-process, achieving the lowest possible first-response latency. - [Eager Workflow Start](/design-patterns/eager-workflow-start): Dispatch the first Workflow Task directly to a co-located Worker, bypassing the Temporal Matching Service. Requires the starter and Worker to share the same process and client connection. ## Choosing a pattern **You only care about total workflow latency** (not first-response time): use [Local Activities](/design-patterns/local-activities). If co-location is feasible, add [Eager Workflow Start](/design-patterns/eager-workflow-start) for the maximum reduction. **You care most about first-response latency**: use [Early Return + Local Activities](/design-patterns/early-return-local-activities). The client gets its response in ~160 ms; background work continues independently. **You are using TypeScript**: use [Local Activities](/design-patterns/local-activities) and [Early Return + Local Activities](/design-patterns/early-return-local-activities). Eager Workflow Start is not available in the TypeScript SDK. **You want to start simple**: begin with [Local Activities](/design-patterns/local-activities). It requires minimal structural change and provides the most straightforward per-Activity improvement. ## Related sections - [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — the [Early Return](/design-patterns/early-return) pattern lives there, describing the Update-with-Start mechanism in detail - [Worker Configuration Patterns](/design-patterns/worker-configuration-patterns) — tuning Worker concurrency and task queue assignments that affect throughput - [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — rate limiting and fairness patterns for high-volume workloads --- # Pick First Pattern Source: https://docs.temporal.io/design-patterns/pick-first > Starts multiple Activities in parallel and uses the first result, cancelling the rest. ## Overview The Pick First pattern executes multiple Activities in parallel and returns the result of whichever completes first, then cancels the remaining Activities. It is suitable for racing multiple approaches to the same task, implementing timeout alternatives, or optimizing for fastest response when multiple options are available. ## Problem In distributed systems, you often need Workflows that execute multiple Activities that can accomplish the same goal, return as soon as any one succeeds (fastest wins), cancel remaining Activities to avoid wasted resources, and handle scenarios where speed matters more than trying all options. Without the Pick First pattern, you must wait for all Activities to complete even when only one result is needed, manually track which Activity finished first, implement complex cancellation logic for remaining Activities, and waste compute resources on Activities whose results will not be used. ## Solution The Pick First pattern races multiple Activities simultaneously, captures the first result, then cancels remaining Activities using a cancellation mechanism provided by each SDK. ```mermaid sequenceDiagram participant Workflow participant Activity1 participant Activity2 participant Activity3 Workflow->>+Activity1: Start (shared ctx) Workflow->>+Activity2: Start (shared ctx) Workflow->>+Activity3: Start (shared ctx) Note over Workflow: Selector waits for first par Race Activity1->>Activity1: Execute (slow) Activity2->>Activity2: Execute (fast) Activity3->>Activity3: Execute (medium) end Activity2-->>Workflow: Result (FIRST!) Note over Workflow: Selector returns Workflow->>Workflow: cancelHandler() Workflow->>Activity1: Cancel Workflow->>Activity3: Cancel Activity1-->>-Workflow: Cancelled Activity3-->>-Workflow: Cancelled deactivate Activity2 ``` The following describes each step in the diagram: 1. The Workflow starts three Activities in parallel using a shared cancellable context. 2. The Workflow waits for the first Activity to complete. 3. Activity 2 completes first. The Workflow captures its result. 4. The Workflow cancels the shared context, which cancels Activities 1 and 3. The following implementation shows the core pattern. The Workflow creates a cancellable context, starts two Activities, and captures the first result: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import sample_activity @workflow.defn class PickFirstWorkflow: @workflow.run async def run(self) -> str: task1 = asyncio.create_task( workflow.execute_activity( sample_activity, "option1", start_to_close_timeout=timedelta(minutes=2), heartbeat_timeout=timedelta(seconds=10), ) ) task2 = asyncio.create_task( workflow.execute_activity( sample_activity, "option2", start_to_close_timeout=timedelta(minutes=2), heartbeat_timeout=timedelta(seconds=10), ) ) done, pending = await workflow.wait( [task1, task2], return_when=asyncio.FIRST_COMPLETED ) for task in pending: task.cancel() return done.pop().result() ``` **Go** ```go // workflow.go func PickFirstWorkflow(ctx workflow.Context) (string, error) { selector := workflow.NewSelector(ctx) var firstResponse string childCtx, cancelHandler := workflow.WithCancel(ctx) childCtx = workflow.WithActivityOptions(childCtx, activityOptions) f1 := workflow.ExecuteActivity(childCtx, Activity, "option1") f2 := workflow.ExecuteActivity(childCtx, Activity, "option2") selector.AddFuture(f1, func(f workflow.Future) { _ = f.Get(ctx, &firstResponse) }).AddFuture(f2, func(f workflow.Future) { _ = f.Get(ctx, &firstResponse) }) selector.Select(ctx) // Blocks until first completes cancelHandler() // Cancel remaining activities return firstResponse, nil } ``` **Java** ```java // PickFirstWorkflow.java public class PickFirstWorkflowImpl implements PickFirstWorkflow { private final SampleActivities activities = Workflow.newActivityStub( SampleActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(2)) .setHeartbeatTimeout(Duration.ofSeconds(10)) .build()); @Override public String pickFirst() { List> results = new ArrayList<>(); CancellationScope scope = Workflow.newCancellationScope( () -> { results.add(Async.function(activities::sampleActivity, "option1")); results.add(Async.function(activities::sampleActivity, "option2")); }); scope.run(); String firstResponse = Promise.anyOf(results).get(); scope.cancel(); return firstResponse; } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities, CancellationScope } from '@temporalio/workflow'; import type * as activities from './activities'; const { sampleActivity } = proxyActivities({ startToCloseTimeout: '2m', heartbeatTimeout: '10s', }); export async function pickFirstWorkflow(): Promise { return await CancellationScope.cancellable(async () => { const p1 = sampleActivity('option1'); const p2 = sampleActivity('option2'); const firstResponse = await Promise.race([p1, p2]); CancellationScope.current().cancel(); return firstResponse; }); } ``` Each SDK provides a different mechanism for racing Activities and cancelling the rest: - **Go** uses `workflow.NewSelector()` with `AddFuture()` callbacks and `workflow.WithCancel(ctx)` for cancellation. - **TypeScript** uses `Promise.race()` within a `CancellationScope.cancellable()`. Calling `CancellationScope.current().cancel()` cancels all Activities started in that scope. - **Python** uses `asyncio.create_task()` to start Activities concurrently, then `workflow.wait()` with `return_when=asyncio.FIRST_COMPLETED` to get the first result. Pending tasks are cancelled explicitly. - **Java** uses `Async.function()` to start Activities inside a `CancellationScope`, then `Promise.anyOf()` to wait for the first result. Calling `scope.cancel()` cancels the remaining Activities. ## Implementation ### Activity with cancellation support For the Pick First pattern to work efficiently, Activities must detect cancellation via heartbeats and respond to the cancellation signal provided by their SDK: **Python** ```python # activities.py import asyncio from temporalio import activity @activity.defn async def sample_activity(branch_id: str) -> str: try: for elapsed in range(60): await asyncio.sleep(1) activity.heartbeat("status-report") return f"Branch {branch_id} completed" except asyncio.CancelledError: activity.logger.info(f"Branch {branch_id} cancelled") raise ``` **Go** ```go // activity.go func SampleActivity(ctx context.Context, branchID int, duration time.Duration) (string, error) { logger := activity.GetLogger(ctx) elapsed := time.Nanosecond for elapsed < duration { time.Sleep(time.Second) elapsed += time.Second activity.RecordHeartbeat(ctx, "status-report") select { case <-ctx.Done(): msg := fmt.Sprintf("Branch %d cancelled", branchID) logger.Info(msg) return msg, ctx.Err() default: // Continue working } } return fmt.Sprintf("Branch %d completed", branchID), nil } ``` **Java** ```java // SampleActivityImpl.java public class SampleActivityImpl implements SampleActivities { @Override public String sampleActivity(String branchID) { for (int elapsed = 0; elapsed < 60; elapsed++) { try { Thread.sleep(1000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw Activity.wrap(e); } Activity.getExecutionContext().heartbeat("status-report"); // Heartbeat throws CanceledFailure if cancellation was requested } return "Branch " + branchID + " completed"; } } ``` **TypeScript** ```typescript // activities.ts import { Context, heartbeat } from '@temporalio/activity'; export async function sampleActivity(branchID: string): Promise { for (let elapsed = 0; elapsed < 60; elapsed++) { await new Promise((resolve) => setTimeout(resolve, 1000)); heartbeat('status-report'); Context.current().cancellationSignal.throwIfAborted(); } return `Branch ${branchID} completed`; } ``` The Activity heartbeats on each iteration, which allows the Temporal Server to deliver cancellation notifications promptly. When cancellation is detected, the Activity performs any necessary cleanup and exits. ### Wait for cancellation completion The following implementation waits for all Activities to finish their cleanup before returning: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy with workflow.unsafe.imports_passed_through(): from activities import sample_activity @workflow.defn class PickFirstWithCleanup: @workflow.run async def run(self) -> str: task1 = asyncio.create_task( workflow.execute_activity( sample_activity, "branch1", start_to_close_timeout=timedelta(minutes=2), heartbeat_timeout=timedelta(seconds=10), cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED, ) ) task2 = asyncio.create_task( workflow.execute_activity( sample_activity, "branch2", start_to_close_timeout=timedelta(minutes=2), heartbeat_timeout=timedelta(seconds=10), cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED, ) ) done, pending = await workflow.wait( [task1, task2], return_when=asyncio.FIRST_COMPLETED ) for task in pending: task.cancel() # Wait for all activities to finish cancellation for task in pending: try: await task except asyncio.CancelledError: pass return done.pop().result() ``` **Go** ```go // workflow.go func PickFirstWithCleanup(ctx workflow.Context) (string, error) { selector := workflow.NewSelector(ctx) var firstResponse string childCtx, cancelHandler := workflow.WithCancel(ctx) childCtx = workflow.WithActivityOptions(childCtx, workflow.ActivityOptions{ StartToCloseTimeout: 2 * time.Minute, WaitForCancellation: true, }) f1 := workflow.ExecuteActivity(childCtx, Activity, "branch1") f2 := workflow.ExecuteActivity(childCtx, Activity, "branch2") pendingFutures := []workflow.Future{f1, f2} selector.AddFuture(f1, func(f workflow.Future) { _ = f.Get(ctx, &firstResponse) }).AddFuture(f2, func(f workflow.Future) { _ = f.Get(ctx, &firstResponse) }) selector.Select(ctx) cancelHandler() // Wait for all activities to finish cancellation for _, f := range pendingFutures { _ = f.Get(ctx, nil) } return firstResponse, nil } ``` **Java** ```java // PickFirstWithCleanupImpl.java public class PickFirstWithCleanupImpl implements PickFirstWorkflow { private final SampleActivities activities = Workflow.newActivityStub( SampleActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(2)) .setHeartbeatTimeout(Duration.ofSeconds(10)) .setCancellationType( ActivityCancellationType.WAIT_CANCELLATION_COMPLETED) .build()); @Override public String pickFirst() { List> results = new ArrayList<>(); CancellationScope scope = Workflow.newCancellationScope( () -> { results.add(Async.function(activities::sampleActivity, "branch1")); results.add(Async.function(activities::sampleActivity, "branch2")); }); scope.run(); String firstResponse = Promise.anyOf(results).get(); scope.cancel(); // Wait for all activities to finish cancellation for (Promise activityResult : results) { try { activityResult.get(); } catch (ActivityFailure e) { if (!(e.getCause() instanceof CanceledFailure)) { throw e; } } } return firstResponse; } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities, CancellationScope, ActivityCancellationType, isCancellation, } from '@temporalio/workflow'; import type * as activities from './activities'; const { sampleActivity } = proxyActivities({ startToCloseTimeout: '2m', heartbeatTimeout: '10s', cancellationType: ActivityCancellationType.WAIT_CANCELLATION_COMPLETED, }); export async function pickFirstWithCleanup(): Promise { return await CancellationScope.cancellable(async () => { const p1 = sampleActivity('branch1'); const p2 = sampleActivity('branch2'); const firstResponse = await Promise.race([p1, p2]); CancellationScope.current().cancel(); // Wait for all activities to finish cancellation const results = [p1, p2]; for (const p of results) { try { await p; } catch (err) { if (!isCancellation(err)) throw err; } } return firstResponse; }); } ``` Each SDK provides a way to wait for cancelled Activities to finish their cleanup: - **Go** sets `WaitForCancellation: true` in the Activity options, then calls `Get` on all futures after cancelling. - **TypeScript** sets `cancellationType: ActivityCancellationType.WAIT_CANCELLATION_COMPLETED` in the Activity options, then awaits all promises while catching cancellation errors with `isCancellation()`. - **Python** sets `cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED`, then awaits pending tasks while catching `asyncio.CancelledError`. - **Java** sets `setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED)`, then calls `get()` on all promises while catching `ActivityFailure` with a `CanceledFailure` cause. ## When to use The Pick First pattern is a good fit for racing multiple data sources (primary vs backup), trying multiple algorithms and picking the fastest, implementing fallback strategies with timeout, optimizing for latency when multiple options exist, and testing multiple service endpoints for fastest response. It is not a good fit when you need results from all Activities (use parallel execution), Activities have side effects that should not be cancelled, order matters (use sequential execution), or all Activities must complete. ## Benefits and trade-offs The pattern returns as soon as the fastest option completes, optimizing for latency. Unnecessary work is cancelled automatically. Each SDK's race mechanism ensures replay consistency, and cancellation cleanup is handled properly. The trade-offs to consider are that cancelled Activities may have done partial work. Activities need heartbeats to detect cancellation quickly. Activities do not cancel instantly (they wait for the next heartbeat). You must implement proper cancellation handling in Activities. Only the first result is used; others are discarded. ## Comparison with alternatives | Approach | Returns first | Cancels others | Complexity | Use case | | :--- | :--- | :--- | :--- | :--- | | Pick First | Yes | Yes | Medium | Race for fastest | | Parallel Execution | No | No | Low | All must complete | | Sequential | No | N/A | Low | Order matters | | Split/Merge | No | No | Medium | Aggregate results | ## Best practices - **Use heartbeats.** Activities must heartbeat to detect cancellation quickly. - **Configure cancellation wait behavior.** Decide if the Workflow should wait for cleanup to complete before returning. - **Handle cancellation in Activities.** Activities must check for cancellation signals and exit cleanly. - **Use a shared cancellable context.** Use a single cancellable context or scope for all raced Activities. - **Track futures or tasks.** Keep references to all futures or tasks if waiting for cleanup. - **Set Activity timeouts.** Configure appropriate StartToCloseTimeout and HeartbeatTimeout. - **Log cancellations.** Log when Activities are cancelled for observability. - **Design idempotent Activities.** Ensure Activities handle cancellation safely. ## Common pitfalls - **Missing heartbeats in Activities.** Activities must heartbeat to detect cancellation. Without heartbeats, cancelled Activities continue running until their StartToCloseTimeout expires, wasting resources. - **Not waiting for cancellation cleanup.** Without configuring the cancellation type to wait for completion (for example, `WaitForCancellation: true` in Go, `WAIT_CANCELLATION_COMPLETED` in other SDKs), fetching a cancelled Activity's result returns a cancellation error immediately, before the Activity has finished cleanup. Configure this setting if you need to wait for cleanup to complete. - **Ignoring errors from the winning Activity.** The first Activity to complete might return an error. Always check the result for errors rather than assuming success. - **Forgetting to cancel remaining Activities.** If you forget to cancel the shared context or scope after receiving the first result, the remaining Activities continue running indefinitely. ## Related ### Patterns - **[Parallel Execution](/design-patterns/parallel-execution)**: Execute in parallel and combine all results. ### Sample code - [Go Sample](https://github.com/temporalio/samples-go/tree/main/pickfirst) — Complete implementation with Worker and starter. - [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/activities-cancellation-heartbeating) — Activities with cancellation and heartbeating. - [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello) — Hello samples including cancellation scope patterns. - [Python Sample](https://github.com/temporalio/samples-python) — Python SDK samples with async patterns. --- # Polling External Services Source: https://docs.temporal.io/design-patterns/polling > Strategies for polling external resources with varying frequencies: frequent, infrequent, and periodic patterns. ## Overview The Polling External Services pattern implements strategies for periodically checking external systems until a desired state is reached. It enables Workflows to wait for asynchronous operations in third-party services that do not support callbacks, making it essential for integrating with REST APIs, job queues, and batch processing systems. ## Problem In distributed systems, you often need Workflows that wait for external jobs to complete, poll REST APIs that do not provide webhooks, check the status of long-running operations in third-party systems, handle varying poll frequencies, and avoid overwhelming external services with requests. Without proper polling strategies, you must implement complex retry logic manually, risk unbounded Workflow history growth, choose between responsiveness and resource efficiency, and handle heartbeating and timeout management yourself. ## Solution You can use Temporal to implement three distinct polling strategies, each optimized for different polling frequencies and requirements: 1. **Frequent Polling (1 second or faster)**: Loop inside an Activity with heartbeats. 2. **Infrequent Polling (1 minute or slower)**: Use Activity retries with fixed backoff. 3. **Periodic Sequence**: Use Child Workflows for complex polling sequences. ```mermaid flowchart TD Start([Polling Required]) --> Freq{Polling
Frequency?} Freq -->|≤1 second| Fast[Frequent Polling] Freq -->|≥1 minute| Slow[Infrequent Polling] Freq -->|Complex| Complex[Periodic Sequence] Fast --> FastImpl[Activity with loop
+ heartbeats] Slow --> SlowImpl[Activity retries
backoffCoefficient=1] Complex --> ComplexImpl[Child workflow
+ Continue-As-New] ``` The following describes each path in the diagram: 1. If you need polling at 1-second intervals or faster, use frequent polling with an Activity loop and heartbeats. 2. If you need polling at 1-minute intervals or slower, use infrequent polling with Activity retries and a fixed backoff coefficient. 3. If you need complex multi-step polling or changing parameters between attempts, use a periodic sequence with Child Workflows and Continue-As-New. ## Implementation ### Frequent polling (fast response required) For polling intervals of 1 second or faster, implement the polling loop inside the Activity with heartbeats. The heartbeat reports progress and enables Temporal to detect stuck Activities: **Python** ```python # activities.py from temporalio import activity import asyncio @activity.defn async def do_poll() -> str: while True: activity.heartbeat() result = await external_service.check_status() if result == "COMPLETED": return result await asyncio.sleep(1) ``` **Go** ```go // activities.go func DoPoll(ctx context.Context) (string, error) { for { activity.RecordHeartbeat(ctx) result, err := externalService.CheckStatus() if err != nil { return "", err } if result == "COMPLETED" { return result, nil } select { case <-ctx.Done(): return "", ctx.Err() case <-time.After(1 * time.Second): } } } ``` **Java** ```java // FrequentPollingActivityImpl.java @ActivityInterface public interface PollingActivities { String doPoll(); } public class FrequentPollingActivityImpl implements PollingActivities { @Override public String doPoll() { while (true) { Activity.getExecutionContext().heartbeat(null); String result = externalService.checkStatus(); if (result.equals("COMPLETED")) { return result; } try { Thread.sleep(1000); } catch (InterruptedException e) { throw Activity.wrap(e); } } } } ``` **TypeScript** ```typescript // activities.ts import { heartbeat, sleep } from '@temporalio/activity'; export async function doPoll(): Promise { while (true) { heartbeat(); const result = await externalService.checkStatus(); if (result === 'COMPLETED') { return result; } await sleep('1s'); } } ``` The Activity loops indefinitely, heartbeating on each iteration. If the Worker crashes, the heartbeat timeout expires and Temporal retries the Activity on another Worker. The Workflow configures the Activity with a heartbeat timeout shorter than the start-to-close timeout: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import do_poll @workflow.defn class FrequentPollingWorkflow: @workflow.run async def run(self) -> str: return await workflow.execute_activity( do_poll, start_to_close_timeout=timedelta(seconds=60), heartbeat_timeout=timedelta(seconds=2), ) ``` **Go** ```go // workflow.go func FrequentPollingWorkflow(ctx workflow.Context) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 60 * time.Second, HeartbeatTimeout: 2 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result) return result, err } ``` **Java** ```java // FrequentPollingWorkflowImpl.java public class FrequentPollingWorkflowImpl implements PollingWorkflow { @Override public String exec() { ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(60)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build(); PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options); return activities.doPoll(); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { doPoll } = proxyActivities({ startToCloseTimeout: '60s', heartbeatTimeout: '2s', }); export async function frequentPollingWorkflow(): Promise { return await doPoll(); } ``` The heartbeat timeout (2 seconds) is shorter than the start-to-close timeout (60 seconds). If the Activity misses a heartbeat, Temporal detects the failure and retries the Activity. ### Infrequent polling (resource efficient) For polling intervals of 1 minute or slower, use Activity retries with a backoff coefficient of 1. The Activity throws an exception when the external service is not ready, and Temporal retries after the configured interval: **Python** ```python # activities.py from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def do_poll() -> str: result = await external_service.check_status() if result != "COMPLETED": raise ApplicationError("Service not ready, will retry") return result ``` **Go** ```go // activities.go func DoPoll(ctx context.Context) (string, error) { result, err := externalService.CheckStatus() if err != nil { return "", err } if result != "COMPLETED" { return "", fmt.Errorf("service not ready, will retry") } return result, nil } ``` **Java** ```java // InfrequentPollingActivityImpl.java public class InfrequentPollingActivityImpl implements PollingActivities { @Override public String doPoll() { String result = externalService.checkStatus(); if (!result.equals("COMPLETED")) { throw new RuntimeException("Service not ready, will retry"); } return result; } } ``` **TypeScript** ```typescript // activities.ts import { ApplicationFailure } from '@temporalio/activity'; export async function doPoll(): Promise { const result = await externalService.checkStatus(); if (result !== 'COMPLETED') { throw ApplicationFailure.retryable('Service not ready, will retry'); } return result; } ``` The Activity performs a single poll and throws if the service is not ready. Temporal handles the retry scheduling. The Workflow configures the retry policy with a fixed interval: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy with workflow.unsafe.imports_passed_through(): from activities import do_poll @workflow.defn class InfrequentPollingWorkflow: @workflow.run async def run(self) -> str: return await workflow.execute_activity( do_poll, start_to_close_timeout=timedelta(seconds=2), retry_policy=RetryPolicy( backoff_coefficient=1, initial_interval=timedelta(seconds=60), ), ) ``` **Go** ```go // workflow.go func InfrequentPollingWorkflow(ctx workflow.Context) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 2 * time.Second, RetryPolicy: &temporal.RetryPolicy{ BackoffCoefficient: 1, InitialInterval: 60 * time.Second, }, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result) return result, err } ``` **Java** ```java // InfrequentPollingWorkflowImpl.java public class InfrequentPollingWorkflowImpl implements PollingWorkflow { @Override public String exec() { ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(2)) .setRetryOptions( RetryOptions.newBuilder() .setBackoffCoefficient(1) .setInitialInterval(Duration.ofSeconds(60)) .build()) .build(); PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options); return activities.doPoll(); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { doPoll } = proxyActivities({ startToCloseTimeout: '2s', retry: { backoffCoefficient: 1, initialInterval: '60s', }, }); export async function infrequentPollingWorkflow(): Promise { return await doPoll(); } ``` Setting the backoff coefficient to 1 creates a fixed retry interval. The initial interval of 60 seconds sets the polling frequency. Retries do not add events to the Workflow history, keeping it small. ### Periodic sequence (complex polling) For polling that requires multiple Activities or changing parameters between attempts, use Child Workflows with Continue-As-New. The Child Workflow polls in a loop and calls Continue-As-New to prevent unbounded history: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import do_poll @workflow.defn class PollingChildWorkflow: @workflow.run async def run(self, polling_interval_seconds: int) -> str: max_attempts = 10 for _ in range(max_attempts): result = await workflow.execute_activity( do_poll, start_to_close_timeout=timedelta(seconds=10), ) if result == "COMPLETED": return result await workflow.sleep(polling_interval_seconds) # Continue-as-new to prevent unbounded history workflow.continue_as_new(polling_interval_seconds) ``` **Go** ```go // workflow.go func PollingChildWorkflow(ctx workflow.Context, pollingIntervalSeconds int) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) maxAttempts := 10 for i := 0; i < maxAttempts; i++ { var result string err := workflow.ExecuteActivity(ctx, DoPoll).Get(ctx, &result) if err != nil { return "", err } if result == "COMPLETED" { return result, nil } workflow.Sleep(ctx, time.Duration(pollingIntervalSeconds)*time.Second) } // Continue-as-new to prevent unbounded history return "", workflow.NewContinueAsNewError(ctx, PollingChildWorkflow, pollingIntervalSeconds) } ``` **Java** ```java // PeriodicPollingChildWorkflowImpl.java @WorkflowInterface public interface PollingChildWorkflow { @WorkflowMethod String exec(int pollingIntervalInSeconds); } public class PeriodicPollingChildWorkflowImpl implements PollingChildWorkflow { @Override public String exec(int pollingIntervalInSeconds) { ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .build(); PollingActivities activities = Workflow.newActivityStub(PollingActivities.class, options); int maxAttempts = 10; for (int i = 0; i < maxAttempts; i++) { String result = activities.doPoll(); if (result.equals("COMPLETED")) { return result; } Workflow.sleep(Duration.ofSeconds(pollingIntervalInSeconds)); } // Continue-as-new to prevent unbounded history PollingChildWorkflow continueAsNew = Workflow.newContinueAsNewStub(PollingChildWorkflow.class); continueAsNew.exec(pollingIntervalInSeconds); return null; } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities, sleep, continueAsNew } from '@temporalio/workflow'; import type * as activities from './activities'; const { doPoll } = proxyActivities({ startToCloseTimeout: '10s', }); export async function pollingChildWorkflow( pollingIntervalSeconds: number ): Promise { const maxAttempts = 10; for (let i = 0; i < maxAttempts; i++) { const result = await doPoll(); if (result === 'COMPLETED') { return result; } await sleep(`${pollingIntervalSeconds}s`); } // Continue-as-new to prevent unbounded history await continueAsNew(pollingIntervalSeconds); return ''; // unreachable } ``` The Child Workflow polls up to 10 times, sleeping between attempts. After 10 attempts, it calls Continue-As-New to start a fresh execution with the same parameters. The parent Workflow starts the Child Workflow and waits for its result: **Python** ```python # workflows.py from temporalio import workflow @workflow.defn class PeriodicPollingWorkflow: @workflow.run async def run(self) -> str: return await workflow.execute_child_workflow( PollingChildWorkflow.run, 5, id="ChildWorkflowPoll", ) ``` **Go** ```go // workflow.go func PeriodicPollingWorkflow(ctx workflow.Context) (string, error) { cwo := workflow.ChildWorkflowOptions{ WorkflowID: "ChildWorkflowPoll", } ctx = workflow.WithChildOptions(ctx, cwo) var result string err := workflow.ExecuteChildWorkflow(ctx, PollingChildWorkflow, 5).Get(ctx, &result) return result, err } ``` **Java** ```java // PeriodicPollingWorkflowImpl.java public class PeriodicPollingWorkflowImpl implements PollingWorkflow { @Override public String exec() { PollingChildWorkflow childWorkflow = Workflow.newChildWorkflowStub( PollingChildWorkflow.class, ChildWorkflowOptions.newBuilder() .setWorkflowId("ChildWorkflowPoll") .build()); return childWorkflow.exec(5); } } ``` **TypeScript** ```typescript // workflows.ts import { executeChild } from '@temporalio/workflow'; import { pollingChildWorkflow } from './polling-child-workflow'; export async function periodicPollingWorkflow(): Promise { return await executeChild(pollingChildWorkflow, { args: [5], workflowId: 'ChildWorkflowPoll', }); } ``` The parent remains blocked and is unaware of the child's Continue-As-New calls. When the child completes, the parent receives the result. ## When to use ### Frequent polling (1 second or faster) This strategy is a good fit for real-time status checks, high-priority operations requiring fast response, and short-lived external operations (minutes, not hours). It is not a good fit for long-running operations (hours or days), rate-limited APIs, or resource-constrained external services. ### Infrequent polling (1 minute or slower) This strategy is a good fit for batch job completion checks, long-running external processes, rate-limited APIs, and operations that may take hours or days. It is not a good fit for sub-minute polling requirements or operations requiring immediate response. ### Periodic sequence This strategy is a good fit for multi-step polling sequences, changing Activity parameters between polls, and very long-running polls requiring Continue-As-New. It is not a good fit when the frequent or infrequent patterns are sufficient. ## Benefits and trade-offs All three strategies work with any external service that does not support callbacks. Temporal handles retry scheduling and fault tolerance automatically. All timing is based on Workflow time, ensuring deterministic behavior. Frequent polling provides fast response but consumes more resources and requires heartbeating. Infrequent polling is resource-efficient with minimal history growth but has a minimum 1-minute interval. Periodic sequence is the most flexible but adds complexity through Child Workflow management. ## Comparison with alternatives | Strategy | Poll frequency | History impact | Complexity | Best for | | :--- | :--- | :--- | :--- | :--- | | Frequent Polling | 1 second or faster | Medium | Low | Real-time checks | | Infrequent Polling | 1 minute or slower | Minimal | Low | Long operations | | Periodic Sequence | Any | Low (with CAN) | Medium | Complex sequences | | Workflow Timer Loop | Any | High | Medium | Avoid this approach | ## Best practices - **Choose the right strategy.** Match polling frequency to the pattern. - **Set appropriate timeouts.** HeartbeatTimeout must be shorter than StartToCloseTimeout for frequent polling. - **Handle failures gracefully.** Distinguish transient from permanent failures. - **Add exponential backoff.** Use backoff for error cases (not normal polling). - **Implement circuit breakers.** Protect external services from overload. - **Use Continue-As-New.** Prevent unbounded history in periodic sequences. - **Monitor polling metrics.** Track poll attempts, success rates, and durations. - **Respect rate limits.** Adjust polling frequency to API constraints. - **Add jitter.** Prevent thundering herd when many Workflows poll simultaneously. - **Consider webhooks.** If the external service supports callbacks, use async completion instead. ## Common pitfalls - **Wrong pattern choice.** Using frequent polling for hour-long operations wastes resources. - **Missing heartbeats.** Frequent polling without heartbeats causes delayed failure detection. - **Unbounded history.** Not using Continue-As-New in periodic sequences leads to history limit failures. - **Tight polling loops.** Polling too frequently overwhelms external services. - **No timeout.** Polling indefinitely without max attempts or a deadline risks runaway Workflows. - **Ignoring errors.** Not distinguishing between retryable and permanent failures leads to wasted retries. - **Workflow timer loops.** Using Workflow timers instead of proper polling patterns bloats history. ## Related ### Patterns - **[Long-Running Activity](/design-patterns/long-running-activity)**: Reporting progress in long Activities. - **[Continue-As-New](/design-patterns/continue-as-new)**: Managing unbounded Workflow history. ### Sample code ### Java - [Frequent Polling](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/frequent) — Fast polling with heartbeats. - [Infrequent Polling](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/infrequent) — Efficient long-interval polling. - [Periodic Sequence](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/polling/periodicsequence) — Complex polling with Child Workflows. ### TypeScript - [Frequent Polling](https://github.com/temporalio/samples-typescript/tree/main/polling/frequent) — Fast polling with heartbeats. - [Infrequent Polling](https://github.com/temporalio/samples-typescript/tree/main/polling/infrequent) — Efficient long-interval polling. - [Periodic Sequence](https://github.com/temporalio/samples-typescript/tree/main/polling/periodic-sequence) — Complex polling with Child Workflows. ### Python - [Frequent Polling](https://github.com/temporalio/samples-python/tree/main/polling/frequent) — Fast polling with heartbeats. - [Infrequent Polling](https://github.com/temporalio/samples-python/tree/main/polling/infrequent) — Efficient long-interval polling. - [Periodic Sequence](https://github.com/temporalio/samples-python/tree/main/polling/periodic_sequence) — Complex polling with Child Workflows. ### Go - [Frequent Polling](https://github.com/temporalio/samples-go/tree/main/polling/frequent) — Fast polling with heartbeats. - [Infrequent Polling](https://github.com/temporalio/samples-go/tree/main/polling/infrequent) — Efficient long-interval polling. - [Periodic Sequence](https://github.com/temporalio/samples-go/tree/main/polling/periodicsequence) — Complex polling with Child Workflows. --- # Priority Task Queues Source: https://docs.temporal.io/design-patterns/priority-task-queues > Assigns a priority level to Workflows and Activities so that time-sensitive work executes ahead of lower-priority work within a single Task Queue. > **ℹ️ TLDR:** > Assign a `PriorityKey` (1–5) to Workflows and Activities so **high priority work executes ahead of lower priority work** on a shared Task Queue. Use this when a flood of batch or background tasks would otherwise delay high-urgency requests. ## Overview The Priority Task Queues pattern assigns a `PriorityKey` to Workflows, Activities, and Child Workflows so that time-sensitive work executes ahead of lower-priority work within a single Task Queue—without requiring separate queues or routing logic. ## Problem In a shared Task Queue, tasks execute in generally first-in-first-out (FIFO) order. When a large batch of low-priority work—nightly reports, bulk imports, background processing—floods the queue before time-sensitive requests arrive, the higher-priority requests wait behind the entire batch. A single queue with no ordering mechanism gives equal treatment to all tasks regardless of business urgency. ## Solution Temporal's native Priority feature lets you assign a `PriorityKey` (an integer from 1 to 5, where 1 is the highest priority and 5 is the lowest) to any Workflow, Activity, or Child Workflow. The Temporal matching service maintains a sub-queue for each priority level and exhausts all tasks at a given level before dispatching to the next. Tasks default to priority 3 when no key is set. Activities and Child Workflows inherit the parent Workflow's priority unless they set their own. ```mermaid flowchart TD WF1["Workflow\nPriorityKey=1\n(payment)"] --> TQ["my-task-queue"] WF2["Workflow\nPriorityKey=3\n(default)"] --> TQ WF3["Workflow\nPriorityKey=5\n(batch report)"] --> TQ TQ --> P1["Priority 1\nsub-queue"] TQ --> P3["Priority 3\nsub-queue"] TQ --> P5["Priority 5\nsub-queue"] P1 -->|dispatched first| W["Shared Workers"] P3 -->|dispatched second| W P5 -->|dispatched last| W W --> DS["Downstream\nService"] ``` The following describes each step in the diagram: 1. Workflows start with a `PriorityKey` in their start options. Payment workflows use priority 1; routine workflows default to 3; nightly batch reports use priority 5. 2. The Temporal matching service routes each task to the corresponding priority sub-queue inside the single Task Queue. 3. Workers poll the Task Queue and receive tasks in priority order: all priority-1 tasks are dispatched before any priority-2 task, and so on. 4. Activities and Child Workflows inherit the parent Workflow's `PriorityKey` unless they explicitly set their own. ## Implementation Priority is enabled by default in Temporal Cloud and self-hosted Temporal. ### Set Workflow priority at start **Python** ```python from temporalio.common import Priority handle = await client.start_workflow( ChargeCustomer.run, id="charge-customer-wf", task_queue="my-task-queue", priority=Priority(priority_key=1), ) ``` **Go** ```go we, err := c.ExecuteWorkflow( context.Background(), client.StartWorkflowOptions{ ID: "charge-customer-wf", TaskQueue: "my-task-queue", Priority: temporal.Priority{PriorityKey: 1}, }, ChargeCustomer, ) ``` **Java** ```java WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("charge-customer-wf") .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder().setPriorityKey(1).build()) .build(); ChargeCustomer workflow = client.newWorkflowStub(ChargeCustomer.class, options); WorkflowClient.start(workflow::run); ``` ### Set Activity priority Activities inherit the parent Workflow's priority. Override the `PriorityKey` in `ActivityOptions` when an individual Activity should run at a different level than its Workflow. **Python** ```python from temporalio.common import Priority # inside the workflow result = await workflow.execute_activity( process_payment, start_to_close_timeout=timedelta(minutes=1), priority=Priority(priority_key=1), ) ``` **Go** ```go ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{PriorityKey: 1}, } ctx = workflow.WithActivityOptions(ctx, ao) err := workflow.ExecuteActivity(ctx, ProcessPayment).Get(ctx, nil) ``` **Java** ```java ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder().setPriorityKey(1).build()) .build(); PaymentActivities activities = Workflow.newActivityStub(PaymentActivities.class, options); activities.processPayment(); ``` ### Set Child Workflow priority **Python** ```python from temporalio.common import Priority # inside the parent workflow result = await workflow.execute_child_workflow( ProcessOrder.run, id="process-order-child", task_queue="my-task-queue", priority=Priority(priority_key=2), ) ``` **Go** ```go cwo := workflow.ChildWorkflowOptions{ WorkflowID: "process-order-child", TaskQueue: "my-task-queue", Priority: temporal.Priority{PriorityKey: 2}, } ctx = workflow.WithChildOptions(ctx, cwo) err := workflow.ExecuteChildWorkflow(ctx, ProcessOrder).Get(ctx, nil) ``` **Java** ```java ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder() .setWorkflowId("process-order-child") .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder().setPriorityKey(2).build()) .build(); ProcessOrder child = Workflow.newChildWorkflowStub(ProcessOrder.class, options); child.run(); ``` ### Set priority via CLI ```sh temporal workflow start \ --type ChargeCustomer \ --task-queue my-task-queue \ --workflow-id charge-customer-wf \ --input '{"customerId":"12345"}' \ --priority-key 1 ``` ## When to use This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent tasks to proceed even during periods of high load. It also works well when you need to mark urgent tasks that should override normal processing—for example, triggering immediate re-runs of failed critical tasks. It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers (see dedicated queues per tier as a supplementary measure). If your concern is prioritizing work amongst tenants or customers, consider the [Fairness](/design-patterns/fairness) pattern, which distributes capacity proportionally using weighted fairness keys rather than strict ordering. ## Benefits and trade-offs Native priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all priority levels, so idle capacity at low-priority levels is automatically used by higher-priority work without any additional configuration. Lower-priority tasks are blocked until all higher-priority tasks have started. In an environment with a continuously replenished high-priority backlog, low-priority tasks may be significantly delayed. The built-in `PriorityKey` range is 1–5; if more than five distinct levels are needed, the feature cannot accommodate them. ## Comparison with alternatives | Approach | Isolation | Dynamic priority | Complexity | Scales to many priorities | | :--- | :--- | :--- | :--- | :--- | | Temporal PriorityKey (native) | Soft | Yes | Low | Yes (1–5 levels) | | [Fairness](/design-patterns/fairness) | Soft | Yes | Low | Yes (unlimited keys) | | Separate Task Queues per tier | Hard | No | Medium | No (static tiers) | | Single queue (no control) | None | N/A | Lowest | N/A | | External queue (Kafka, SQS) | Hard | Yes | High | Yes | ## Best practices - **Use no more than five priority levels.** The `PriorityKey` range is 1–5. Keep levels coarse—for example, 1 = urgent, 3 = normal, 5 = batch—rather than mapping fine-grained business importance to many values. - **Reserve priority 1 for genuinely urgent work.** If high priority is the fallback when no priority is specified, the highest level fills with routine work and the feature provides no benefit. The default is 3 when no key is set. - **Set `PriorityKey` at Workflow start, not inside Workflow code.** Workflow code cannot change its own priority after it starts. Set the priority in the start options before execution begins. - **Override Activity priority deliberately.** Activities inherit the parent Workflow's priority by default. Override only when a specific Activity must run at a different level than its Workflow. - **Monitor queue depth per priority level.** Sustained backlog growth at a priority level signals that Worker capacity is insufficient for the submitted load at that level. ## Common pitfalls - **Assigning priority 1 to all work by default.** When every caller sets the highest priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level. - **Neglecting low-priority starvation.** Under sustained high load, priority-5 tasks may wait indefinitely. Use `ScheduleToStartTimeout` on low-priority activities to surface starvation as a visible failure. - **Changing priority after scheduling.** `PriorityKey` is evaluated when a task enters the queue and cannot be changed while it waits. To re-prioritize an already-queued task, cancel it and reschedule with the new priority. - **Assuming hard isolation between priority levels.** Priority controls dispatch order, not Worker capacity allocation. A priority-5 task may still consume a Worker slot that is then unavailable for a priority-1 task arriving a moment later. ## Related ### Patterns - **[Fairness](/design-patterns/fairness)**: Distribute capacity proportionally across tenants within a priority level using fairness keys. - **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap absolute throughput to a downstream service regardless of task priority. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. ### Sample code The official Temporal documentation provides SDK code examples for setting priority keys on Workflows, Activities, and Child Workflows across all supported languages: - [Task Queue Priority and Fairness — Temporal docs](/develop/task-queue-priority-fairness#task-queue-priority) --- # QoS & Throughput Patterns Source: https://docs.temporal.io/design-patterns/qos-throughput-patterns > Pattern selection guide for controlling execution rate, protecting downstream services from overload, and ensuring fair capacity distribution across tenants. These patterns control how fast work executes, protect downstream services from overload, and make sure no single caller or tenant monopolizes Worker capacity at the expense of others. ## Patterns in this section - [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting): Caps the Activity execution rate against a downstream service by routing throttled Activities to a dedicated Task Queue whose Workers enforce a throughput limit. - [Priority Task Queues](/design-patterns/priority-task-queues): Assigns a priority level to Workflows and Activities so time-sensitive work runs ahead of lower-priority work on the same Task Queue. - [Fairness](/design-patterns/fairness): Distributes Worker capacity evenly across tenants or users so a burst from one caller does not starve the others. ## Choosing a pattern **A downstream dependency has a fixed rate limit**: use [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting) to cap throughput at the Worker. **Urgent work must not wait behind bulk work**: use [Priority Task Queues](/design-patterns/priority-task-queues). **Multiple tenants share the same Workers**: use [Fairness](/design-patterns/fairness) to keep one tenant's burst from starving others. ## Related sections - [Worker Configuration Patterns](/design-patterns/worker-configuration-patterns) — the Task Queue and Worker setup these patterns route through - [Batch Processing Patterns](/design-patterns/batch-processing-patterns) — rate-control patterns for large record sets - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns) — back off and retry when a rate limit is hit --- # Request-Response via Updates Source: https://docs.temporal.io/design-patterns/request-response-via-updates > Synchronous request-response with validation. Updates modify state and return results directly. ## Overview Workflow Updates enable synchronous request-response interactions where clients receive immediate, typed responses while the Workflow continues processing. Updates modify Workflow state, validate inputs, and return results directly to the caller with strong consistency guarantees. ## Problem In distributed systems, you often need Workflows that provide immediate feedback to clients (validation results, confirmation IDs), require strong consistency guarantees for operations, need typed error handling for validation failures, should validate inputs before accepting work, and allow external systems to modify Workflow state synchronously. Without Updates, clients must use Signals and poll via Queries (complex, eventually consistent), wait for entire Workflow completion (slow), implement complex coordination logic, and handle race conditions between Signals and Queries. ## Solution Temporal's Update API executes an Update handler that can validate inputs, modify state, and return values synchronously. The Update is recorded in Workflow history before returning, providing strong consistency. ```mermaid sequenceDiagram participant Client participant Workflow participant State Client->>+Workflow: Update Request Workflow->>Workflow: Validate Input alt Validation Fails Workflow-->>Client: Error Response else Success Workflow->>State: Modify State State-->>Workflow: Updated Workflow-->>Client: Typed Response end deactivate Workflow ``` The following describes each step in the diagram: 1. The client sends an Update request to the Workflow. 2. The Workflow validates the input. If validation fails, it returns a typed error response. 3. If validation succeeds, the Workflow modifies its state and returns a typed response to the client. ## Implementation The following examples show a task assignment Workflow that accepts tasks via Updates with validation. The Update validator rejects requests when the task limit is reached, and the Update handler assigns the task and returns a result. **Python** ```python # workflows.py import uuid from dataclasses import dataclass from temporalio import workflow MAX_TASKS = 10 @dataclass class AssignmentResult: assignment_id: str task_name: str total_tasks: int @workflow.defn class TaskWorkflow: def __init__(self) -> None: self.tasks: list[str] = [] @workflow.run async def run(self) -> None: await workflow.wait_condition(lambda: False) @workflow.update async def assign_task(self, task_name: str) -> AssignmentResult: assignment_id = str(uuid.uuid4()) self.tasks.append(task_name) return AssignmentResult( assignment_id=assignment_id, task_name=task_name, total_tasks=len(self.tasks), ) @assign_task.validator def validate_assign_task(self, task_name: str) -> None: if len(self.tasks) >= MAX_TASKS: raise ValueError("Task limit reached") @workflow.query def get_tasks(self) -> list[str]: return list(self.tasks) ``` **Go** ```go // workflow.go type TaskWorkflow struct{} const MaxTasks = 10 func (w *TaskWorkflow) Run(ctx workflow.Context) error { tasks := []string{} err := workflow.SetUpdateHandlerWithOptions( ctx, "AssignTask", func(ctx workflow.Context, taskName string) (AssignmentResult, error) { assignmentID := uuid.New().String() tasks = append(tasks, taskName) return AssignmentResult{ AssignmentID: assignmentID, TaskName: taskName, TotalTasks: len(tasks), }, nil }, workflow.UpdateHandlerOptions{ Validator: func(taskName string) error { if len(tasks) >= MaxTasks { return fmt.Errorf("task limit reached") } return nil }, }, ) if err != nil { return err } err = workflow.SetQueryHandler(ctx, "GetTasks", func() ([]string, error) { return tasks, nil }) if err != nil { return err } workflow.GetSignalChannel(ctx, "").Receive(ctx, nil) return nil } ``` **Java** ```java // TaskWorkflow.java @WorkflowInterface public interface TaskWorkflow { @WorkflowMethod void run(); @UpdateMethod AssignmentResult assignTask(String taskName); @QueryMethod List getTasks(); } public class TaskWorkflowImpl implements TaskWorkflow { private static final int MAX_TASKS = 10; private List tasks = new ArrayList<>(); @Override public void run() { Workflow.await(() -> false); } @UpdateValidatorMethod(updateName = "assignTask") protected void validateAssignTask(String taskName) { if (tasks.size() >= MAX_TASKS) { throw new IllegalStateException("Task limit reached"); } } @Override public AssignmentResult assignTask(String taskName) { String assignmentId = UUID.randomUUID().toString(); tasks.add(taskName); return new AssignmentResult(assignmentId, taskName, tasks.size()); } @Override public List getTasks() { return new ArrayList<>(tasks); } } ``` **TypeScript** ```typescript // workflow.ts import * as wf from '@temporalio/workflow'; interface AssignmentResult { assignmentId: string; taskName: string; totalTasks: number; } export const assignTaskUpdate = wf.defineUpdate('assignTask'); export const getTasksQuery = wf.defineQuery('getTasks'); const MAX_TASKS = 10; export async function taskWorkflow(): Promise { const tasks: string[] = []; wf.setHandler( assignTaskUpdate, (taskName: string): AssignmentResult => { const assignmentId = wf.uuid4(); tasks.push(taskName); return { assignmentId, taskName, totalTasks: tasks.length }; }, { validator: (taskName: string): void => { if (tasks.length >= MAX_TASKS) { throw new Error('Task limit reached'); } }, } ); wf.setHandler(getTasksQuery, (): string[] => tasks); await wf.condition(() => false); } ``` In all SDKs, the validator runs before the Update handler. If the validator throws an exception, the Update is rejected and the client receives a typed error. If the validator passes, the Update handler modifies state and returns a typed result. The Update is recorded in Workflow history before the response is returned to the client. ## When to use The Update pattern is a good fit for request-response patterns requiring immediate confirmation, input validation before accepting work, synchronous state modifications with typed responses, operations requiring strong consistency guarantees, and entity Workflows that need external state Updates. It is not a good fit for fire-and-forget operations (use Signals), read-only operations (use Queries), high-throughput scenarios where latency matters (Updates are slower than Signals), or operations that do not need an immediate response. ## Benefits and trade-offs Benefits: - Updates provide a synchronous response — the client receives a typed return value immediately. - Validation failures return as typed exceptions. - The Update is recorded in history before returning, providing strong consistency. - You can modify Workflow state directly from external systems. Trade-offs: - Updates are slower than Signals (they require a history write). - The Update handler blocks Workflow Task execution and consumes Workflow Task execution time. - For fire-and-forget messages that need no response, Updates require more machinery than Signals. - Update arguments and return values are limited by the Workflow history event size (typically 2 MB per event). - Each Update adds events to Workflow history, contributing to the 50K event limit. - There is a maximum of 10 in-flight Updates per Workflow execution and a maximum of 2,000 total Updates in Workflow history. ## Comparison with alternatives | Approach | Use case | Response type | Latency | Consistency | | :--- | :--- | :--- | :--- | :--- | | Update | Request-response | Sync typed value | Higher | Strong | | Signal | Fire-and-forget | None | Lower | Eventual | | Query | Read-only | Sync typed value | Lowest | Eventual | ## Best practices - **Validate early.** Check inputs at the start of the Update handler to fail fast. - **Handle errors.** Throw typed exceptions for validation failures. - **Return quickly.** Do not perform long operations in the Update handler. - **Track Update IDs only across Continue-As-New.** Within a single Workflow Execution, the Server deduplicates retried Updates automatically by Update ID, so a retry does not run the handler twice. You only need to track processed Update IDs in Workflow state when carrying them across a Continue-As-New boundary, because Update ID deduplication is scoped to a single Workflow Run. - **Set timeouts.** Configure appropriate Update timeouts. - **Maintain state consistency.** Ensure state modifications are atomic within the handler. ## Common pitfalls - **Performing long operations in the Update handler.** Update handlers block Workflow Task execution. Offload long-running work to Activities and use `Workflow.await` in the handler to wait for results. - **Exceeding the 2,000 total Updates limit.** Each accepted Update adds events to history. Use Continue-As-New before reaching the limit. The server sets `SuggestContinueAsNew` at 90% of the limit. - **Not setting Update timeouts.** Without a client-side timeout, the caller blocks indefinitely if the Worker is unavailable. Always set a context timeout or deadline. - **Assuming you must set an Update ID for retry safety.** The SDK auto-generates a unique `updateId` when you do not provide one, and retried client calls are deduplicated automatically, so a transient retry will not run the handler twice. Set a stable, business-meaningful `updateId` when you want a retried Update-with-Start to attach to an existing in-flight Update instead of starting duplicate work. - **Using Updates for fire-and-forget.** Updates require a Worker to be online and responsive. For fire-and-forget operations, use Signals instead. ## Related ### Patterns - **Signal**: Fire-and-forget state modifications. - **Query**: Read-only state inspection. - **[Entity Workflow](/design-patterns/entity-workflow)**: Long-running Workflows representing business entities. - **[Early Return](/design-patterns/early-return)**: Returning intermediate results before Workflow completion. ### Sample code - [Safe Message Handlers (Python)](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) — Concurrent Update handling with validation. - [Safe Message Passing (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Update handling with validation. - [Update with Start - Shopping Cart (Go)](https://github.com/temporalio/samples-go/tree/main/shoppingcart) — Update-with-Start for lazy initialization. --- # Resumable Activity (AKA Pause On Failure) Source: https://docs.temporal.io/design-patterns/resumable-activity > Park the Workflow after retries are exhausted and wait for a human to signal a correction, then resume execution from where it left off. > **ℹ️ TLDR:** > After retries are exhausted, **park the Workflow in a waiting state and block on a Signal that notifies the Workflow to proceed or optionally delivers corrected input from a human operator, then re-execute the Activity.** Use this when failures are caused by bad input that can be fixed externally — so the Workflow resumes exactly where it left off instead of being restarted from scratch. ## Overview The Resumable Activity pattern parks a Workflow, durably waiting, after Activity retries are exhausted, waits for a corrective Signal from a human operator, then re-executes the Activity with the corrected input. Use it when failures are caused by bad input data that can be corrected externally — a wrong account number, an invalid reference, a missing record that will be created — and abandoning the Workflow is worse than pausing it. ## Problem When an Activity fails due to bad input, retrying with the same input will never succeed. The standard options are: - **Fail the Workflow immediately.** The client must restart the process from scratch, often re-entering the same data that caused the failure. - **Mark the error as non-retryable.** Same result — the Workflow fails and the client has no way to inject a correction. - **Poll for the correction from inside the Workflow.** Wastes resources for a fix that may never come. What you actually want is for the Workflow to *pause* — consuming zero resources — until an authorized operator provides corrected data, then *resume* exactly where it left off. Temporal's durable execution model makes this possible without any external database, queue, or polling mechanism. ## Solution Use a bounded `RetryPolicy` to allow a few automatic retries (in case the failure is transient), then catch the exhausted `ActivityError` in the Workflow. Transition to an `AWAITING_CORRECTION` state and block on `workflow.wait_condition` (or equivalent). Register a Signal handler that accepts the corrected input and unblocks the condition. When the Signal arrives, re-execute the Activity with the corrected input. A second Signal gates the final approval before completing. ```mermaid sequenceDiagram participant Client participant Admin participant Workflow participant Activity as Transfer Activity Client->>+Workflow: Start transfer(from, invalid-account, $500) Workflow->>Workflow: status = TRANSFERRING loop maxAttempts=3 Workflow->>+Activity: executeTransfer(invalid-account) Activity-->>-Workflow: Failure — account not found end Note over Workflow: Retries exhausted Workflow->>Workflow: status = AWAITING_CORRECTION Note over Workflow: Parked in Temporal — zero cost, no polling Admin->>Workflow: Signal: retryWithCorrection("account-123") Workflow->>Workflow: status = TRANSFERRING Workflow->>+Activity: executeTransfer(account-123) Activity-->>-Workflow: Success Workflow->>Workflow: status = AWAITING_APPROVAL Note over Workflow: Parked again — waiting for client approval Client->>Workflow: Signal: approve(true) Workflow-->>-Client: Transfer completed ``` The following describes each step: 1. The client starts the Workflow with an invalid account number. 2. The Activity fails. Temporal retries automatically up to the configured `maxAttempts`. 3. When retries are exhausted, the Workflow catches the `ActivityError` and sets its status to `AWAITING_CORRECTION`. 4. The Workflow parks itself using `wait_condition` — it consumes no CPU, no polling, no timers. Its state is fully persisted in Temporal. 5. An admin notices the problem (via Temporal UI, an alert, or an operations dashboard) and sends a `retryWithCorrection` Signal with the corrected account number. 6. The Workflow wakes up, applies the correction, and re-executes the Activity — which now succeeds. 7. The Workflow transitions to `AWAITING_APPROVAL` and parks again, waiting for the client to approve the transfer. 8. The client sends an `approve` Signal. The Workflow completes and returns the result. The key insight: **the Workflow never died**. It survived bad input, waited indefinitely without polling, accepted an external correction, and completed cleanly. Its entire state — status, corrected account, approval decision — is durable in Temporal throughout. ## Implementation ### Workflow with correction and approval signals The Workflow maintains state as named fields. Signal handlers set the fields, and `wait_condition` blocks until they are non-null. **Python** ```python # workflows.py from dataclasses import dataclass from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy, SearchAttributeKey from temporalio.exceptions import ActivityError import activities TRANSFER_STATUS_KEY = SearchAttributeKey.for_keyword("TransferStatus") @dataclass class TransferInput: from_account: str to_account: str amount: float @workflow.defn class TransferWorkflow: def __init__(self) -> None: self._status = "PENDING" self._corrected_account: str | None = None self._approval: bool | None = None @workflow.run async def run(self, transfer: TransferInput) -> str: account = transfer.to_account correction_attempts = 0 while True: self._status = "TRANSFERRING" try: result = await workflow.execute_activity( activities.execute_transfer, TransferInput(transfer.from_account, account, transfer.amount), start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy(maximum_attempts=3), ) break # Activity succeeded — exit the correction loop except ActivityError: correction_attempts += 1 if correction_attempts > 5: self._status = "FAILED" workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)]) raise self._status = "AWAITING_CORRECTION" workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)]) workflow.logger.warning( "Transfer failed — waiting for account correction", extra={"to_account": account}, ) # Park until the admin sends a correction signal await workflow.wait_condition( lambda: self._corrected_account is not None ) account = self._corrected_account self._corrected_account = None self._status = "AWAITING_APPROVAL" await workflow.wait_condition(lambda: self._approval is not None) if self._approval: self._status = "COMPLETED" return f"Transfer of {transfer.amount} to {account} completed" self._status = "REJECTED" return "Transfer rejected by client" @workflow.signal def retry_with_correction(self, corrected_account: str) -> None: self._corrected_account = corrected_account @workflow.signal def approve(self, approved: bool) -> None: self._approval = approved @workflow.query def get_status(self) -> str: return self._status ``` **Go** ```go // workflow.go package transfer import ( "fmt" "time" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/workflow" ) type TransferInput struct { FromAccount string ToAccount string Amount float64 } func TransferWorkflow(ctx workflow.Context, input TransferInput) (string, error) { status := "PENDING" if err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) { return status, nil }); err != nil { return "", err } correctionCh := workflow.GetSignalChannel(ctx, "retryWithCorrection") approvalCh := workflow.GetSignalChannel(ctx, "approve") ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 3}, } actCtx := workflow.WithActivityOptions(ctx, ao) account := input.ToAccount correctionCount := 0 for { status = "TRANSFERRING" err := workflow.ExecuteActivity(actCtx, ExecuteTransfer, TransferInput{ FromAccount: input.FromAccount, ToAccount: account, Amount: input.Amount, }).Get(actCtx, nil) if err == nil { break // Activity succeeded — exit the correction loop } correctionCount++ if correctionCount > 5 { // for long correction cycles, consider Continue As New status = "FAILED" _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status}) return "", err } status = "AWAITING_CORRECTION" _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status}) workflow.GetLogger(ctx).Warn("Transfer failed — waiting for account correction", "to_account", account) // Park until the admin sends a correction signal var corrected string _ = workflow.Await(ctx, func() bool { return correctionCh.ReceiveAsync(&corrected) }) account = corrected } status = "AWAITING_APPROVAL" var approved bool _ = workflow.Await(ctx, func() bool { return approvalCh.ReceiveAsync(&approved) }) if approved { status = "COMPLETED" return fmt.Sprintf("Transfer of %.2f to %s completed", input.Amount, account), nil } status = "REJECTED" return "Transfer rejected by client", nil } ``` **Java** ```java // TransferWorkflowImpl.java import io.temporal.activity.ActivityOptions; import io.temporal.common.RetryOptions; import io.temporal.common.SearchAttributeKey; import io.temporal.failure.ActivityFailure; import io.temporal.workflow.SignalMethod; import io.temporal.workflow.QueryMethod; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; import io.temporal.workflow.Workflow; import java.time.Duration; @WorkflowInterface public interface TransferWorkflow { @WorkflowMethod String run(TransferInput input); @SignalMethod void retryWithCorrection(String correctedAccount); @SignalMethod void approve(boolean approved); @QueryMethod String getStatus(); } public class TransferWorkflowImpl implements TransferWorkflow { private String status = "PENDING"; private String correctedAccount; private Boolean approval; private final TransferActivities activities = Workflow.newActivityStub( TransferActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(3) .build()) .build() ); @Override public String run(TransferInput input) { String account = input.getToAccount(); int correctionCount = 0; while (true) { status = "TRANSFERRING"; try { activities.executeTransfer( new TransferInput(input.getFromAccount(), account, input.getAmount()) ); break; // Activity succeeded — exit the correction loop } catch (ActivityFailure e) { correctionCount++; if (correctionCount > 5) { // for long correction cycles, consider Continue As New status = "FAILED"; Workflow.upsertTypedSearchAttributes( SearchAttributeKey.forKeyword("TransferStatus").valueSet(status) ); throw e; } status = "AWAITING_CORRECTION"; Workflow.upsertTypedSearchAttributes( SearchAttributeKey.forKeyword("TransferStatus").valueSet(status) ); Workflow.getLogger(getClass()).warn( "Transfer failed — waiting for account correction: " + account ); // Park until the admin sends a correction signal Workflow.await(() -> correctedAccount != null); account = correctedAccount; correctedAccount = null; } } status = "AWAITING_APPROVAL"; Workflow.await(() -> approval != null); if (approval) { status = "COMPLETED"; return String.format("Transfer of %.2f to %s completed", input.getAmount(), account); } status = "REJECTED"; return "Transfer rejected by client"; } @Override public void retryWithCorrection(String account) { this.correctedAccount = account; } @Override public void approve(boolean decision) { this.approval = decision; } @Override public String getStatus() { return status; } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; export interface TransferInput { fromAccount: string; toAccount: string; amount: number; } export const retryWithCorrectionSignal = wf.defineSignal<[string]>('retryWithCorrection'); export const approveSignal = wf.defineSignal<[boolean]>('approve'); export const getStatusQuery = wf.defineQuery('getStatus'); const { executeTransfer } = wf.proxyActivities({ startToCloseTimeout: '30s', retry: { maximumAttempts: 3 }, }); export async function transferWorkflow(input: TransferInput): Promise { let status = 'PENDING'; let correctedAccount: string | undefined; let approval: boolean | undefined; wf.setHandler(retryWithCorrectionSignal, (account: string) => { correctedAccount = account; }); wf.setHandler(approveSignal, (decision: boolean) => { approval = decision; }); wf.setHandler(getStatusQuery, () => status); let account = input.toAccount; let correctionCount = 0; while (true) { status = 'TRANSFERRING'; try { await executeTransfer({ ...input, toAccount: account }); break; // Activity succeeded — exit the correction loop } catch (err) { correctionCount++; if (correctionCount > 5) { status = 'FAILED'; wf.upsertSearchAttributes({ TransferStatus: [status] }); throw err; } status = 'AWAITING_CORRECTION'; wf.upsertSearchAttributes({ TransferStatus: [status] }); wf.log.warn('Transfer failed — waiting for account correction', { account }); // Park until the admin sends a correction signal await wf.condition(() => correctedAccount !== undefined); account = correctedAccount!; correctedAccount = undefined; } } status = 'AWAITING_APPROVAL'; await wf.condition(() => approval !== undefined); if (approval) { status = 'COMPLETED'; return `Transfer of ${input.amount} to ${account} completed`; } status = 'REJECTED'; return 'Transfer rejected by client'; } ``` ### Send signals An operator sends the correction Signal using the Temporal CLI or any SDK client. The Workflow wakes immediately when the Signal is delivered. ```bash # Correct the account number temporal workflow signal \ --workflow-id transfer-wf-001 \ --name retryWithCorrection \ --input '"account-123"' # Approve the transfer temporal workflow signal \ --workflow-id transfer-wf-001 \ --name approve \ --input 'true' ``` ### Activity implementation The `executeTransfer` Activity must distinguish between permanent failures — such as an invalid account number — and transient failures that Temporal should retry automatically. Throw a non-retryable `ApplicationFailure` for permanent input errors so the Workflow catches the `ActivityError` immediately and transitions to `AWAITING_CORRECTION` instead of exhausting all retry attempts first. Let all other exceptions propagate so the RetryPolicy handles transient failures. **Python** ```python # activities.py from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def execute_transfer(transfer: TransferInput) -> str: # Non-retryable: bad account number requires a human correction, not a retry. if not await account_service.exists(transfer.to_account): raise ApplicationError( f"Account {transfer.to_account} not found", type="AccountNotFoundError", non_retryable=True, ) # Other exceptions propagate as retryable so the RetryPolicy handles them. return await payment_service.transfer( transfer.from_account, transfer.to_account, transfer.amount ) ``` **TypeScript** ```typescript // activities.ts import { ApplicationFailure } from '@temporalio/activity'; import type { TransferInput } from './workflows'; export async function executeTransfer(transfer: TransferInput): Promise { // Non-retryable: bad account number requires a human correction, not a retry. const accountExists = await accountService.exists(transfer.toAccount); if (!accountExists) { throw ApplicationFailure.nonRetryable( `Account ${transfer.toAccount} not found`, 'AccountNotFoundError', ); } // Other exceptions propagate as retryable so the RetryPolicy handles them. return paymentService.transfer(transfer.fromAccount, transfer.toAccount, transfer.amount); } ``` ## State diagram The Workflow transitions through a well-defined set of states. Query `getStatus` at any time to observe the current state. ```mermaid stateDiagram-v2 [*] --> PENDING PENDING --> TRANSFERRING : Workflow starts TRANSFERRING --> AWAITING_CORRECTION : Activity retries exhausted AWAITING_CORRECTION --> TRANSFERRING : retryWithCorrection signal received AWAITING_CORRECTION --> FAILED : 5 correction attempts exceeded TRANSFERRING --> AWAITING_APPROVAL : Activity succeeds AWAITING_APPROVAL --> COMPLETED : approve(true) signal AWAITING_APPROVAL --> REJECTED : approve(false) signal COMPLETED --> [*] REJECTED --> [*] FAILED --> [*] ``` ## Best practices - **Use a bounded `MaximumAttempts` before parking.** Allow a few automatic retries to recover from transient failures. Parking immediately on the first failure forces operators to intervene for problems that would have resolved on their own. - **Manage history growth in long-running correction loops.** Each correction cycle — park, receive signal, re-execute Activity — adds events to the Workflow history (signal received, state transitions, Activity scheduled/completed). For workflows that may receive many corrections over time, use [Continue-As-New](/design-patterns/continue-as-new) to carry the current state into a fresh execution before the history grows too large, rather than relying solely on an arbitrary correction counter. - **Expose status via a Query method.** The `getStatus` Query gives operations tooling visibility into where the Workflow is parked without requiring access to the Workflow history. - **Validate the correction in the Signal handler.** Check that the corrected account is non-empty and matches the expected format before setting the state. An invalid correction parks the Workflow again, but a clear error message helps operators. - **Log and record state at every transition.** The `AWAITING_CORRECTION` and `AWAITING_APPROVAL` states can last hours or days. Structured log lines at each transition make the audit trail clear. For operational visibility, also update a [Search Attribute](/visibility) at each transition (for example, a `Keyword` attribute storing the current status) so operators can filter and query workflows by state directly from the Temporal UI or CLI. - **Notify the operator proactively.** The `AWAITING_CORRECTION` transition is a good point to send an alert — an email, a Slack message, or a ticket — rather than waiting for the operator to notice in the Temporal UI. - **Distinguish this from the Approval pattern.** The [Approval](/design-patterns/approval) pattern gates forward progress on a human decision. This pattern recovers from failure with a human-supplied data correction. Both use Signals and `wait_condition`, but serve different roles in a process. ## Common pitfalls - **Waiting without a timeout.** If operators never send the correction Signal, the Workflow waits indefinitely. Add durable timer if the process must resolve within a time bound. - **Not clearing the correction state before re-entering the loop.** After applying the correction, set `corrected_account = None` (or equivalent) before the next Activity attempt. Otherwise, if the corrected activity also fails, the Workflow immediately re-uses the previous correction instead of waiting for a new one. - **Accepting corrections in the wrong state.** If a Signal arrives while the Activity is running (not parked), the correction should be queued and applied after the current attempt completes. The Signal handler always runs — the condition check (`wait_condition`) determines when the Workflow acts on it. - **Conflating the correction loop with a general retry loop.** This pattern is for correcting *input data*. For retrying the same call against a temporarily unavailable system, use [Fast/Slow Retries](/design-patterns/fast-slow-retries) instead. ## Related ### Patterns - [Approval](/design-patterns/approval): Human-in-the-loop gate for forward progress rather than failure recovery. - [Non-Retryable Errors](/design-patterns/non-retryable-errors): Fail immediately without parking when the error is structural and no correction is expected. - [Fast/Slow Retries](/design-patterns/fast-slow-retries): Infinite patient retries when the downstream system is temporarily unavailable. - [Signal with Start](/design-patterns/signal-with-start): Start the Workflow and send the correction Signal atomically. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. ### Guides - [Recover business processes without restarting](/guides/recover-without-restart): A `recoverableStep` implementation of this pattern in a six-step loan pipeline, with Search Attribute-based routing so operators can find and fix blocked cases. --- # Retry Alerting via Metrics Source: https://docs.temporal.io/design-patterns/retry-metrics > Emit a custom metric from inside the Activity when the attempt count crosses a threshold, surfacing silent persistent failures to on-call teams before an SLA breach. > **ℹ️ TLDR:** > Emit a counter metric from inside the Activity when the attempt number exceeds a threshold, using the SDK's built-in metrics scope. **Use this to surface silent, persistent failures to on-call teams before they breach an SLA** — without changing retry behavior or adding Workflow-level tracking. ## Overview The Retry Alerting via Metrics pattern emits a custom metric counter from inside the Activity whenever the attempt number exceeds a threshold. Use it to surface silent, persistent failures to on-call teams before they breach an SLA — without modifying retry behavior or adding Workflow-level tracking. ## Problem When an Activity retries indefinitely, failures are invisible at the system level until something breaks. The Temporal UI shows the current attempt number, but on-call teams do not watch the UI continuously. Without a metric or alert, a downstream system can be down for hours while the Workflow keeps retrying silently — and the first sign of a problem is an SLA breach or a user complaint. Common gaps: - A payment gateway is down. Workflows are retrying every 5 minutes. No alert fires until the merchant escalates. - An email provider is rejecting requests. Activities are on attempt 50. No metric has been emitted. On-call has no signal. - A third-party API degraded. Retries are accumulating. The engineering team learns about the problem from a customer, not from their own alerting. ## Solution Read the current attempt number from the Activity execution context and emit a counter metric when it exceeds a threshold. The metric is sent through the Temporal SDK's built-in metrics scope — the same pipeline used for SDK-internal metrics — so it flows to whatever metrics backend your Workers are already configured to use, such as Prometheus or StatsD, without additional setup. ```mermaid sequenceDiagram participant Temporal as Temporal Service participant Activity participant Metrics as Metrics Backend loop Each retry attempt Temporal->>+Activity: Execute (attempt N) Activity->>Activity: Check attempt number alt attempt > threshold (e.g. 5) Activity->>Metrics: increment high_activity_error_count end Activity-->>-Temporal: Failure Note over Temporal: Wait backoff interval end Note over Metrics: Alert fires when counter crosses threshold ``` The following describes each step: 1. Temporal executes the Activity, passing the current attempt number in the execution context. 2. The Activity checks whether the attempt number exceeds the threshold. 3. If it does, the Activity increments a counter metric using the SDK's built-in metrics scope. 4. On failure, Temporal waits the backoff interval and retries. 5. The metrics backend accumulates the counter. Your alerting system fires when the counter or rate crosses a configured threshold. ## Implementation ### Emit a counter at high attempt counts Read the attempt number from the Activity info and emit a counter through the SDK metrics scope. Configure the retry policy separately — the metric emission does not change retry behavior. **Python** ```python # activities.py from temporalio import activity from temporalio.exceptions import ApplicationError ALERT_THRESHOLD = 5 @activity.defn async def call_downstream_service(endpoint: str) -> str: info = activity.info() if info.attempt > ALERT_THRESHOLD: meter = activity.metric_meter() meter.create_counter( "high_activity_error_count", "Activity has exceeded the failure attempt threshold", ).add(1) # Attempt the actual work — raises on failure, triggering a retry response = await downstream.call(endpoint) return response.data ``` **Go** ```go // activities.go package downstream import ( "context" "go.temporal.io/sdk/activity" ) const alertThreshold = 5 func CallDownstreamService(ctx context.Context, endpoint string) (string, error) { info := activity.GetInfo(ctx) if info.Attempt > alertThreshold { activity.GetMetricsHandler(ctx). Counter("high_activity_error_count"). Inc(1) } // Attempt the actual work — returns an error on failure, triggering a retry response, err := downstream.Call(endpoint) if err != nil { return "", err } return response.Data, nil } ``` **Java** ```java // CallDownstreamActivityImpl.java import io.temporal.activity.Activity; import io.temporal.activity.ActivityExecutionContext; public class CallDownstreamActivityImpl implements CallDownstreamActivity { private static final int ALERT_THRESHOLD = 5; @Override public String callDownstreamService(String endpoint) { ActivityExecutionContext ctx = Activity.getExecutionContext(); if (ctx.getInfo().getAttempt() > ALERT_THRESHOLD) { ctx.getMetricsScope() .counter("HighActivityErrorCount") .inc(1); } // Attempt the actual work — throws on failure, triggering a retry return downstream.call(endpoint).getData(); } } ``` **TypeScript** ```typescript // activities.ts import { Context } from '@temporalio/activity'; const ALERT_THRESHOLD = 5; export async function callDownstreamService(endpoint: string): Promise { const ctx = Context.current(); if (ctx.info.attempt > ALERT_THRESHOLD) { ctx.metricMeter .createCounter('high_activity_error_count') .add(1); } // Attempt the actual work — throws on failure, triggering a retry const response = await downstream.call(endpoint); return response.data; } ``` ### Workflow configuration Configure the Activity in the Workflow with the desired retry policy. The metric emission inside the Activity is independent of the retry configuration. **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy import activities @workflow.defn class MonitoredRetryWorkflow: @workflow.run async def run(self, endpoint: str) -> str: return await workflow.execute_activity( activities.call_downstream_service, endpoint, start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy( initial_interval=timedelta(seconds=5), backoff_coefficient=2.0, maximum_interval=timedelta(minutes=5), # No maximum_attempts — retries indefinitely until success ), ) ``` **Go** ```go // workflow.go func MonitoredRetryWorkflow(ctx workflow.Context, endpoint string) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{ InitialInterval: 5 * time.Second, BackoffCoefficient: 2.0, MaximumInterval: 5 * time.Minute, // No MaximumAttempts — retries indefinitely until success }, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, CallDownstreamService, endpoint).Get(ctx, &result) return result, err } ``` **Java** ```java // MonitoredRetryWorkflowImpl.java public class MonitoredRetryWorkflowImpl implements MonitoredRetryWorkflow { private final CallDownstreamActivity activities = Workflow.newActivityStub( CallDownstreamActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(5)) .setBackoffCoefficient(2.0) .setMaximumInterval(Duration.ofMinutes(5)) // No setMaximumAttempts — retries indefinitely until success .build()) .build() ); @Override public String run(String endpoint) { return activities.callDownstreamService(endpoint); } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; const { callDownstreamService } = wf.proxyActivities({ startToCloseTimeout: '30s', retry: { initialInterval: '5s', backoffCoefficient: 2, maximumInterval: '5m', // No maximumAttempts — retries indefinitely until success }, }); export async function monitoredRetryWorkflow(endpoint: string): Promise { return await callDownstreamService(endpoint); } ``` ### Add dimension tags to the metric Add tags to the metric to identify which Activity type, endpoint, or Workflow is producing the high attempt counts. This makes the metric actionable in dashboards and alerts. **Python** ```python # activities.py if info.attempt > ALERT_THRESHOLD: meter = activity.metric_meter() meter.create_counter( "high_activity_error_count", "Activity has exceeded the failure attempt threshold", ).add(1, {"activity_type": info.activity_type, "endpoint": endpoint}) ``` **Go** ```go // activities.go if info.Attempt > alertThreshold { activity.GetMetricsHandler(ctx). WithTags(map[string]string{ "activity_type": info.ActivityType.Name, "endpoint": endpoint, }). Counter("high_activity_error_count"). Inc(1) } ``` **Java** ```java // CallDownstreamActivityImpl.java if (ctx.getInfo().getAttempt() > ALERT_THRESHOLD) { ctx.getMetricsScope() .tagged(ImmutableMap.of( "activity_type", ctx.getInfo().getActivityType(), "endpoint", endpoint )) .counter("HighActivityErrorCount") .inc(1); } ``` **TypeScript** ```typescript // activities.ts if (ctx.info.attempt > ALERT_THRESHOLD) { ctx.metricMeter .createCounter('high_activity_error_count') .add(1, { activity_type: ctx.info.activityType, endpoint }); } ``` ## Best practices - **Choose a threshold above normal transient noise.** If your downstream system occasionally has 1–2 retry attempts under normal conditions, set the threshold at 5 or 10 so the metric only fires for genuinely sustained failures. - **Emit on every attempt above the threshold, not only once.** Incrementing the counter on each high-attempt invocation allows alerting systems to detect both the onset and the duration of a problem by watching the counter rate. - **Use the SDK metrics scope, not a third-party library.** The SDK scope integrates with your Worker's existing metrics pipeline and adds default tags such as namespace and task queue automatically. - **Set up rate-based alerts, not count-based.** A count alert requires resetting or remembering the baseline. A rate alert (for example, "more than 3 increments per minute") fires when the problem is active and clears when it resolves. - **Combine with Fast/Slow Retries.** Emit the metric in the slow-phase Activity of a [Fast/Slow Retries](/design-patterns/fast-slow-retries) pattern to alert when the Workflow has been in the slow phase long enough to be a concern. ## Common pitfalls - **Emitting the metric in the Workflow instead of the Activity.** The Workflow does not have access to the Activity's attempt number without passing it explicitly. The Activity context always has the current attempt number — use it there. - **Alerting on the total counter value instead of the rate.** If the counter is cumulative, a single high-attempt event in the past will keep the counter elevated forever. Alert on the increment rate (events per minute) rather than the absolute count. - **Not resetting alerting context on success.** If the Activity eventually succeeds after 50 attempts, the high-attempt metric has already fired. Ensure your alerting system can resolve the alert when the metric rate drops to zero. - **Setting the threshold too low.** A threshold of 1 means the metric fires on the very first retry — which is normal behavior. Calibrate the threshold to your system's expected transient error rate. ## Related ### Patterns - [Fast/Slow Retries](/design-patterns/fast-slow-retries): Combine by emitting this metric inside the slow-phase Activity to alert when patient waiting has gone on too long. - [Fixed Count of Retries](/design-patterns/fixed-count-retries): Cap attempts at a fixed number instead of alerting at a threshold. - [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns): Overview and decision tree for all retry patterns. --- # Saga Pattern Source: https://docs.temporal.io/design-patterns/saga-pattern > Manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail. ## Overview The Saga pattern manages distributed transactions across multiple services by coordinating a sequence of local transactions, each with a compensating action that can undo its effects if subsequent steps fail. ## Problem In distributed systems, you need to maintain data consistency across multiple services or databases without using traditional ACID transactions. When a multi-step business process fails partway through, you must undo the effects of completed steps to maintain system consistency. Traditional two-phase commit does not scale well and creates tight coupling between services. ## Solution You implement each step as a local transaction with a corresponding compensation transaction. If any step fails, you execute compensation transactions in reverse order to undo the effects of all completed steps. You register compensations as each step completes, then automatically trigger them when errors occur to ensure cleanup happens reliably. The following diagram shows the worked example used by the runner: opening a customer account in four steps, where `addBankAccount` simulates a downstream failure to trigger compensation. ```mermaid flowchart TD Start([Start Saga]) --> Step1[Step 1: createAccount] Step1 -->|Success| Step2[Step 2: addAddress] Step1 -->|Failure| End([End: Failed]) Step2 -->|Success| Step3[Step 3: addClient] Step2 -->|Failure| Comp1[clearPostalAddresses] Step3 -->|Success| Step4[Step 4: addBankAccount] Step3 -->|Failure| Comp2[removeClient] Step4 -->|Success| Complete([End: Success]) Step4 -->|Failure| Comp3[disconnectBankAccounts] Comp3 --> Comp2 Comp2 --> Comp1 Comp1 --> End classDef success stroke-width:1px classDef compensation stroke-width:1px classDef complete stroke-width:1px classDef fail stroke-width:1px class Step1,Step2,Step3,Step4 success class Comp1,Comp2,Comp3 compensation class Complete complete class End fail ``` The following describes each step in the diagram: 1. The Saga begins by executing Step 1 (`createAccount`). 2. If Step 1 succeeds, the Workflow proceeds to Step 2 (`addAddress`). If it fails, the Saga ends immediately — no compensations are registered yet. 3. If Step 2 succeeds, the Workflow proceeds to Step 3 (`addClient`). If it fails, the Workflow runs `clearPostalAddresses`. 4. If Step 3 succeeds, the Workflow proceeds to Step 4 (`addBankAccount`). If it fails, the Workflow runs `removeClient`, then `clearPostalAddresses`. 5. If Step 4 succeeds, the Saga completes. If it fails, the Workflow runs all three compensations in reverse: `disconnectBankAccounts`, `removeClient`, `clearPostalAddresses`. Note that `disconnectBankAccounts` is registered before `addBankAccount` runs, so it executes even if `addBankAccount` failed mid-flight — its implementation must be idempotent. ## Implementation The following examples show how each SDK implements the Saga pattern. Each language uses a different mechanism to register and execute compensations, but the core principle is the same: register a compensation before or after each step, and run all compensations in reverse order on failure. **Python** ```python # workflows.py from temporalio import workflow @workflow.defn class OpenAccountWorkflow: @workflow.run async def run(self, req: OpenAccountRequest) -> str: compensations = [] try: # Step 1: createAccount has no compensation — leaving an empty # account stub on later failure is acceptable. await workflow.execute_activity( create_account, req, start_to_close_timeout=timedelta(seconds=10), ) # Register compensation for Step 2 BEFORE execution compensations.append( lambda: workflow.execute_activity( clear_postal_addresses, req, start_to_close_timeout=timedelta(seconds=10), ) ) # Step 2: Add postal address await workflow.execute_activity( add_address, req, start_to_close_timeout=timedelta(seconds=10), ) # Register compensation for Step 3 BEFORE execution compensations.append( lambda: workflow.execute_activity( remove_client, req, start_to_close_timeout=timedelta(seconds=10), ) ) # Step 3: Add client record await workflow.execute_activity( add_client, req, start_to_close_timeout=timedelta(seconds=10), ) # Register compensation for Step 4 BEFORE execution compensations.append( lambda: workflow.execute_activity( disconnect_bank_accounts, req, start_to_close_timeout=timedelta(seconds=10), ) ) # Step 4: Link bank account (this step fails in the demo) await workflow.execute_activity( add_bank_account, req, start_to_close_timeout=timedelta(seconds=10), ) except Exception: # On error, run compensations in reverse order for compensation in reversed(compensations): await compensation() raise ``` **Go** ```go // open_account_workflow.go func OpenAccountWorkflow(ctx workflow.Context, req OpenAccountRequest) error { var compensations []func() runCompensations := func() { for i := len(compensations) - 1; i >= 0; i-- { compensations[i]() } } // Step 1: CreateAccount has no compensation — leaving an empty account // stub on later failure is acceptable. if err := workflow.ExecuteActivity(ctx, CreateAccount, req).Get(ctx, nil); err != nil { return err } // Register compensation for Step 2 BEFORE execution compensations = append(compensations, func() { _ = workflow.ExecuteActivity(ctx, ClearPostalAddresses, req).Get(ctx, nil) }) if err := workflow.ExecuteActivity(ctx, AddAddress, req).Get(ctx, nil); err != nil { runCompensations() return err } // Register compensation for Step 3 BEFORE execution compensations = append(compensations, func() { _ = workflow.ExecuteActivity(ctx, RemoveClient, req).Get(ctx, nil) }) if err := workflow.ExecuteActivity(ctx, AddClient, req).Get(ctx, nil); err != nil { runCompensations() return err } // Register compensation for Step 4 BEFORE execution compensations = append(compensations, func() { _ = workflow.ExecuteActivity(ctx, DisconnectBankAccounts, req).Get(ctx, nil) }) if err := workflow.ExecuteActivity(ctx, AddBankAccount, req).Get(ctx, nil); err != nil { runCompensations() return err } return nil } ``` **Java** ```java // OpenAccountWorkflow.java @WorkflowInterface public interface OpenAccountWorkflow { @WorkflowMethod String openAccount(OpenAccountRequest req); } public class OpenAccountWorkflowImpl implements OpenAccountWorkflow { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .build()); @Override public String openAccount(OpenAccountRequest req) { // Create a Saga instance with compensation options Saga saga = new Saga(new Saga.Options.Builder() .setParallelCompensation(false) // Run compensations sequentially .build()); try { // Step 1: createAccount has no compensation — leaving an empty // account stub on later failure is acceptable. activities.createAccount(req); // Register compensation for Step 2 BEFORE execution saga.addCompensation(activities::clearPostalAddresses, req); activities.addAddress(req); // Register compensation for Step 3 BEFORE execution saga.addCompensation(activities::removeClient, req); activities.addClient(req); // Register compensation for Step 4 BEFORE execution saga.addCompensation(activities::disconnectBankAccounts, req); activities.addBankAccount(req); return "Account " + req.accountId() + " opened"; } catch (Exception e) { // On any error, run all registered compensations in reverse order saga.compensate(); throw e; } } } ``` **TypeScript** ```typescript // workflows.ts type Compensation = () => Promise; export async function openAccount(req: OpenAccountRequest): Promise { const compensations: Compensation[] = []; try { // Step 1: createAccount has no compensation — leaving an empty account // stub on later failure is acceptable. await acts.createAccount(req); // Register compensation for Step 2 BEFORE execution compensations.unshift(() => acts.clearPostalAddresses(req)); await acts.addAddress(req); // Register compensation for Step 3 BEFORE execution compensations.unshift(() => acts.removeClient(req)); await acts.addClient(req); // Register compensation for Step 4 BEFORE execution compensations.unshift(() => acts.disconnectBankAccounts(req)); await acts.addBankAccount(req); return `Account ${req.accountId} opened`; } catch (err) { // On error, run all compensations in reverse order (unshift keeps them in LIFO already) for (const compensate of compensations) { await compensate(); } throw err; } } ``` The key differences between SDKs are: - **Go**: Uses a slice of closures and iterates from the end on error. (Some samples use `defer` instead — both achieve LIFO; the slice form makes the rollback trigger explicit.) - **Python**: Uses a list with `reversed()` to iterate compensations in LIFO order on error. - **TypeScript**: Uses an array with `unshift()` to maintain LIFO order, and manually iterates on error. - **Java**: Uses the SDK's `Saga` helper to track compensations and trigger them with `saga.compensate()`. In all SDKs, compensations are registered before Activity execution and run in reverse order of registration. All compensations must be idempotent and able to handle cases where the forward Activity never executed. ### When to register compensations There are two approaches for when to register compensation Activities: 1. **Register before Activity execution** (recommended for safety): This ensures the compensation runs even if the Activity fails after partial completion. For example, a credit card may be charged but the Activity fails before returning success. The compensation must be idempotent and handle cases where the forward Activity never executed (no-op). This is the safer default when Activities have side effects that may occur before failure. 2. **Register after Activity execution** (appropriate when safe): This only compensates Activities that completed successfully. The compensation logic is simpler because you do not need to check whether the forward action occurred. This approach is appropriate when Activities are truly atomic (all-or-nothing). The risk is partial completion without compensation if the Activity fails mid-execution. The choice depends on your Activity's failure characteristics and whether the compensation can safely handle cases where the forward Activity never executed. When in doubt, register compensations before execution and ensure they are idempotent. ## When to use The Saga pattern is a good fit when you need to maintain consistency across multiple services or databases, traditional distributed transactions (two-phase commit) are too slow or unavailable, you can define compensating actions for each step in your business process, eventual consistency is acceptable for your use case, and you need to handle long-running transactions that may span hours or days. It is not a good fit for operations that require strong ACID consistency, single-service transactions that can use a local database transaction, processes where compensations cannot be defined, or operations that must appear atomic to external observers. ## Benefits and trade-offs The Saga pattern maintains eventual consistency without distributed locks, and each service can use its own database and transaction model. Temporal's durable execution guarantees that compensations will execute even after Worker failures. The pattern scales better than two-phase commit protocols. The trade-offs to consider are that only eventual consistency is provided — intermediate states are visible to other processes. You must design idempotent compensation Activities, and compensation logic must be maintained alongside forward logic. Some operations may not have meaningful compensations. ## Comparison with alternatives | Approach | Consistency | Rollback mechanism | Coupling | Scalability | | :--- | :--- | :--- | :--- | :--- | | Saga (orchestration) | Eventual | Compensating transactions | Loose | High | | Two-phase commit | Strong (ACID) | Distributed lock/rollback | Tight | Low | | Saga (choreography) | Eventual | Event-driven compensations | Very loose | High | | Local transaction | Strong (ACID) | Database rollback | None | Single service | ## Best practices - **Make all compensations idempotent.** Compensations may run even when the forward Activity never executed (if registered before execution) or may run multiple times on retry. Use idempotency keys to ensure safe re-execution. - **Register compensations before Activity execution.** This ensures cleanup runs even if the Activity fails after partial completion. The compensation must handle the case where the forward action never occurred (no-op). - **Use idempotency keys for forward Activities.** Pass a unique identifier (such as a client ID or Workflow ID) to each Activity so retries do not create duplicate side effects. - **Set StartToCloseTimeout on compensation Activities.** Set a `StartToCloseTimeout` but avoid `ScheduleToCloseTimeout` on compensations. Do not set Workflow-level timeouts — let compensations retry until they succeed. - **Use a disconnected context for cancellation compensation.** In Go, use `NewDisconnectedContext` to run compensation Activities after Workflow cancellation, since the original context is already cancelled. - **Keep compensation payloads small.** Pass references (IDs, URLs) instead of full data objects to avoid exceeding the 2 MB payload limit. - **Log compensation failures but continue.** If a compensation fails, log the error and continue executing remaining compensations. In production, alert for manual intervention on persistent compensation failures. - **Re-throw the original error after compensating.** Always re-throw the original exception after running compensations so the Workflow reports the correct failure reason. ## Common pitfalls - **Non-idempotent compensations.** Compensations may run even when the forward Activity never executed (if registered before execution) or may run multiple times on retry. All compensations must be idempotent. - **Forgetting to register a compensation.** If a step succeeds but its compensation was never registered, a later failure leaves that step's effects permanently in place. - **Compensations that can fail permanently.** If a compensation Activity fails with a non-retryable error, the Saga cannot fully roll back. Design compensations with generous retry policies. - **Large payloads in compensation state.** Passing large objects through the compensation chain can exceed the 2 MB payload limit. Use references (IDs, URLs) instead of full data. - **Swallowing the ContinueAsNew exception in TypeScript.** In TypeScript, `continueAsNew` works by throwing a special exception. A `catch` block that does not re-throw it, or a `finally` block that returns a value, silently prevents Continue-As-New. ## Related ### Patterns - **[Error Handling & Retry Patterns](/design-patterns/error-handling-patterns)**: Often combined with the Saga pattern to handle transient failures before compensating. - **[Child Workflows](/design-patterns/child-workflows)**: You can use Child Workflows to organize complex Sagas with multiple sub-Sagas. - **[Long-Running Activity](/design-patterns/long-running-activity)**: Heartbeats work well with long-running compensation Activities. - **[Early Return](/design-patterns/early-return)**: You can combine Early Return with the Saga pattern to return initialization results before compensation runs. ### Guides - [Recover business processes without restarting](/guides/recover-without-restart): Adds LIFO saga compensation to a multi-step loan pipeline, with pre-registered compensations and a recoverable rollback loop for compensations that fail. ### Sample code - [Go Sample](https://github.com/temporalio/samples-go/tree/main/saga) — Saga with `defer`-based compensations. - [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSaga.java) — Saga with the `Saga` API. - [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/saga) — Saga with array-based compensations. - [Python Sample](https://github.com/temporalio/samples-python) — Saga with list-based compensations. - [.NET Sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Saga) — Saga with `try`/`catch` compensations. - [Ruby Sample](https://github.com/temporalio/samples-ruby/tree/main/saga) — Saga with `rescue`-based compensations. - [Python SDK implementation guide](/develop/python/best-practices/error-handling#implement-saga-pattern) — Full walkthrough of the compensation-list technique, including retry and rollback code. --- # Signal with Start Pattern Source: https://docs.temporal.io/design-patterns/signal-with-start > Starts a Workflow when Signaling it if it does not already exist. If already running, it receives the Signal directly. ## Overview Signal with Start is a pattern that lazily creates Workflows when Signaling them. If the Workflow is already running, it receives the Signal; if not, the Workflow starts first and then receives the Signal. This enables entity Workflows that only exist when needed and can receive operations throughout their lifetime. ## Problem In distributed systems, you often need Workflows that represent long-lived entities (accounts, shopping carts, user sessions), consume events from streams (Kafka, SQS) and trigger certain behaviors of an aggregate or entity, receive multiple operations over time, should only exist when there is work to do, and need to handle the first operation without special client logic. Without Signal with Start, clients must check if the Workflow exists before Signaling, start the Workflow if it does not exist and then Signal it, handle race conditions when multiple clients try to start the same Workflow, and write complex coordination logic. ## Solution Temporal's Signal with Start API atomically starts a Workflow (if not running) and delivers a Signal in a single operation. The client does not need to know whether the Workflow exists — the platform handles it automatically. ```mermaid sequenceDiagram participant C as Client participant T as Temporal participant W as Workflow C->>T: SignalWithStart(workflowId, signal, args) T->>T: Check if workflow exists alt Workflow does not exist T->>W: Start Workflow activate W T->>W: Deliver Signal W->>W: Process Signal else Workflow already running T->>W: Deliver Signal only (no start) W->>W: Process Signal end deactivate W ``` The following describes each step in the diagram: 1. The client calls SignalWithStart with a Workflow ID, Signal name, and arguments. 2. Temporal checks whether a Workflow with that ID is already running. 3. If the Workflow does not exist, Temporal starts it and then delivers the Signal. 4. If the Workflow is already running, Temporal delivers the Signal without starting a new instance. ## Implementation ### Basic Signal-With-Start The following examples show a shopping cart entity Workflow that is created lazily when the first item is added. Each SDK uses its own API to atomically start the Workflow (if needed) and deliver the Signal. **Python** ```python # client.py from temporalio.client import Client from workflows import ShoppingCartWorkflow, AddItemSignal async def add_item(client: Client, cart_id: str, item_id: str, product_id: str, quantity: int) -> None: # Atomically start workflow (if needed) and deliver signal await client.start_workflow( ShoppingCartWorkflow.run, id=f"cart-{cart_id}", task_queue="carts", start_signal="add_item", start_signal_args=[AddItemSignal(item_id=item_id, product_id=product_id, quantity=quantity)], ) # workflows.py from dataclasses import dataclass from temporalio import workflow @dataclass class AddItemSignal: item_id: str product_id: str quantity: int @dataclass class CartItem: product_id: str quantity: int @workflow.defn class ShoppingCartWorkflow: def __init__(self) -> None: self.processed_items: set[str] = set() self.items: list[CartItem] = [] @workflow.run async def run(self) -> None: await workflow.wait_condition(lambda: False) # Run forever (entity workflow) @workflow.signal def add_item(self, sig: AddItemSignal) -> None: if sig.item_id in self.processed_items: return # Idempotency: ignore duplicate signals self.processed_items.add(sig.item_id) self.items.append(CartItem(product_id=sig.product_id, quantity=sig.quantity)) ``` **Go** ```go // client.go func AddItem(ctx context.Context, cartID, itemID, productID string, quantity int) error { opts := client.StartWorkflowOptions{ ID: "cart-" + cartID, TaskQueue: "carts", } // Atomically start workflow (if needed) and deliver signal sig := AddItemSignal{ItemID: itemID, ProductID: productID, Quantity: quantity} _, err := c.SignalWithStartWorkflow(ctx, "cart-"+cartID, "addItem", sig, opts, ShoppingCartWorkflow) return err } // workflow.go func ShoppingCartWorkflow(ctx workflow.Context) error { processedItems := make(map[string]bool) var items []CartItem addItemCh := workflow.GetSignalChannel(ctx, "addItem") workflow.Go(ctx, func(ctx workflow.Context) { for { var sig AddItemSignal addItemCh.Receive(ctx, &sig) if processedItems[sig.ItemID] { continue // Idempotency: ignore duplicate signals } processedItems[sig.ItemID] = true items = append(items, CartItem{ProductID: sig.ProductID, Quantity: sig.Quantity}) } }) workflow.Await(ctx, func() bool { return false }) // Run forever (entity workflow) return nil } ``` **Java** ```java // ShoppingCartManager.java public class ShoppingCartManager { public void addItem(String cartId, String itemId, String productId, int quantity) { WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("cart-" + cartId) .setTaskQueue("carts") .build(); ShoppingCartWorkflow workflow = workflowClient.newWorkflowStub(ShoppingCartWorkflow.class, options); // Atomically start workflow (if needed) and deliver signal BatchRequest request = workflowClient.newSignalWithStartRequest(); request.add(workflow::run); request.add(workflow::addItem, itemId, productId, quantity); workflowClient.signalWithStart(request); } } // ShoppingCartWorkflow.java @WorkflowInterface public interface ShoppingCartWorkflow { @WorkflowMethod void run(); @SignalMethod void addItem(String itemId, String productId, int quantity); } public class ShoppingCartWorkflowImpl implements ShoppingCartWorkflow { private Set processedItems = new HashSet<>(); private List items = new ArrayList<>(); @Override public void run() { Workflow.await(() -> false); // Run forever (entity workflow) } @Override public void addItem(String itemId, String productId, int quantity) { if (!processedItems.add(itemId)) { return; // Idempotency: ignore duplicate signals } items.add(new CartItem(productId, quantity)); } } ``` **TypeScript** ```typescript // client.ts export async function addItem( cartId: string, itemId: string, productId: string, quantity: number ) { // Atomically start workflow (if needed) and deliver signal const handle = await client.workflow.signalWithStart(shoppingCartWorkflow, { workflowId: `cart-${cartId}`, taskQueue: 'carts', signal: 'addItem', signalArgs: [itemId, productId, quantity], }); } // workflow.ts export async function shoppingCartWorkflow(): Promise { const processedItems = new Set(); const items: CartItem[] = []; setHandler(addItemSignal, (itemId: string, productId: string, quantity: number) => { if (processedItems.has(itemId)) { return; // Idempotency: ignore duplicate signals } processedItems.add(itemId); items.push({ productId, quantity }); }); await condition(() => false); // Run forever (entity workflow) } export const addItemSignal = defineSignal<[string, string, number]>('addItem'); ``` In all SDKs, the Workflow ID is derived from the business entity (the cart ID), ensuring one Workflow per entity. The Signal handler checks a set of processed item IDs to prevent duplicate processing. The Workflow blocks indefinitely, acting as a long-lived entity that receives operations over its lifetime. ## When to use The Signal with Start pattern is a good fit for entity Workflows (accounts, shopping carts, user sessions, clusters), event-driven architectures (Kafka consumers, message queue processors), Workflows that receive multiple operations over their lifetime, lazy entity creation where you only create when the first operation arrives, and fire-and-forget operations where immediate response is not needed. It is not a good fit for one-time operations (use REJECT_DUPLICATE policy instead), request-response patterns requiring synchronous confirmation (use Update with Start), or operations that need immediate return values. ## Benefits and trade-offs Signal with Start provides an atomic operation — start and Signal happen atomically with no race conditions. Workflows only exist when needed (lazy creation). The client does not need to check if the Workflow exists. The operation is safe to retry because duplicate starts are handled by the Workflow ID. The pattern is a natural fit for long-lived business entities. The trade-offs to consider are that Signals are fire-and-forget with no immediate confirmation that the Signal was processed. You still need to track processed operation IDs in the Workflow for Signal idempotency. Workflows must handle unbounded execution (use Continue-As-New). Signals do not return values — use Queries or Updates for that. Both ALLOW_DUPLICATE and ALLOW_DUPLICATE_FAILED_ONLY work well with Signal with Start: - **ALLOW_DUPLICATE** (default): Allows a new Workflow Execution with the same ID after the previous one has closed (completed, failed, timed out, terminated, or cancelled). Does not affect a currently running Workflow — Signal with Start delivers the Signal to the running execution. - **ALLOW_DUPLICATE_FAILED_ONLY**: Allows restart only if the previous run failed — prevents accidental restarts of running Workflows. - **REJECT_DUPLICATE**: Prevents any duplicate starts — useful for one-time operations, not entity Workflows. - **TERMINATE_IF_RUNNING**: Terminates the running Workflow and starts a new one — use with caution. ## Comparison with alternatives | Approach | Use case | Response type | Idempotency | | :--- | :--- | :--- | :--- | | Signal with Start | Entity Workflows | Fire-and-forget | Signal-level | | Update with Start | Request-response | Sync return value | Update-level | | REJECT_DUPLICATE | One-time operations | Async (Workflow ID) | Workflow-level | ## Best practices - **Derive Workflow ID from entity.** Use stable business identifiers (account ID, user ID). - **Implement Signal idempotency.** Track processed operation IDs to prevent duplicates. - **Use WorkflowInit.** Initialize state before Signals are delivered (Java, .NET, and Python's `__init__`). - **Handle unbounded execution.** Use Continue-As-New for long-running entity Workflows. - **Choose the right Workflow ID policy.** Use ALLOW_DUPLICATE_FAILED_ONLY for entity Workflows. - **Include operation IDs.** Every Signal should include a unique operation or reference ID. - **Return early.** Check for duplicates at the start of Signal handlers. ## Common pitfalls - **Not implementing Signal idempotency.** Signals can be delivered more than once (for example, client retries). Without tracking processed operation IDs, the Workflow processes duplicates. - **Unbounded history growth.** Entity Workflows that receive many Signals without calling Continue-As-New will hit the 50K event or 10K Signal limit. Use `isContinueAsNewSuggested()` to trigger Continue-As-New. - **Losing pending Signals on Continue-As-New.** Drain all pending Signals before calling Continue-As-New, and pass unprocessed ones as input to the new execution. - **Expecting a return value from Signals.** Signals are fire-and-forget. If you need a synchronous response, use Updates or Update-with-Start instead. - **Race between SignalWithStart and Continue-As-New.** Temporal prevents this race — if a Signal arrives while the Workflow is completing via Continue-As-New, the Workflow rewinds to process the Signal first. ## Related ### Patterns - **[Entity Workflow](/design-patterns/entity-workflow)**: Long-running Workflows representing business entities. - **[Continue-As-New](/design-patterns/continue-as-new)**: Managing unbounded Workflow history. - **[Request-Response via Updates](/design-patterns/request-response-via-updates)**: When you need synchronous responses instead of fire-and-forget. - **[Early Return](/design-patterns/early-return)**: Update-with-Start for request-response with lazy initialization. ### Sample code **Python** - [Hello Signal](https://github.com/temporalio/samples-python/tree/main/hello/hello_signal.py) — Basic Signal handling in a Workflow. - [Message Passing](https://github.com/temporalio/samples-python/tree/main/message_passing/introduction) — Introduction to message passing with Signals, Queries, and Updates. **Java** - [Hello Signal](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/hello/HelloSignal.java) — Basic Signal handling in a Workflow. - [Safe Message Passing](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/safemessagepassing) — Concurrent Signal handling with validation. **TypeScript** - [Signals and Queries](https://github.com/temporalio/samples-typescript/tree/main/signals-queries) — Signal and Query usage in a Workflow. **Go** - [Await Signals](https://github.com/temporalio/samples-go/tree/main/await-signals) — Waiting for Signals with timeout. --- # Sliding Window Source: https://docs.temporal.io/design-patterns/sliding-window > Maintains a fixed number of concurrently active Child Workflows, starting a new one each time an existing one completes. > **ℹ️ TLDR:** > Keep exactly `windowSize` child Workflows running at all times — each completion signal triggers the next record to start immediately. Use this when your record set is arbitrarily large, you need **bounded concurrency** to protect downstream systems, and you want higher throughput than a sequential Batch Iterator provides. ## Overview The Sliding Window pattern maintains a fixed-size pool of concurrently running child Workflows. As each child completes it signals the parent, which immediately starts a replacement — keeping the concurrency level constant and progressing at the rate of the fastest processor. Continue-as-New prevents the parent's history from growing without bound. ## Problem The [Batch Iterator](/design-patterns/batch-iterator) processes records sequentially — the overall throughput is limited by the slowest record in each page. The [Fan-Out](/design-patterns/fanout-child-workflows) pattern starts all children at once, which can overwhelm downstream systems when the record set is large. You need a way to process an arbitrarily large record set with bounded concurrency, maximum throughput within that bound, and protection against history bloat. ## Solution The parent Workflow keeps a live count of in-flight children (`active`) and runs a single loop that starts a child whenever a slot is free. The first `windowSize` slots are free, so those children start immediately; after that, a backpressure condition blocks each start until an in-flight child signals completion and frees a slot. Each child processes one record and, when finished, signals the parent, which decrements `active` and starts the next record's child. Continue-as-New is called after the parent has started `windowSize` children. Because child Workflows have stable Workflow IDs and Continue-as-New preserves the parent's Workflow ID, children started by a previous run can still signal the current run. The parent carries `active` into the next run so it knows how many carried-over children will still signal it. ```mermaid flowchart TD Records["📋 Record IDs\n[r0, r1, r2, ...]"] Parent["Parent Workflow\n(window size = W)"] C1["Child r0\n✅ done"] C2["Child r1\n⏳ running"] C3["Child r2\n⏳ running"] C4["Child r3\n🆕 started"] CAN["continueAsNew\n(startIndex + W)"] Records --> Parent Parent <-->|"start W children /
signal: complete"| C1 Parent --> C2 Parent --> C3 Parent -->|"slot free → start next"| C4 Parent -->|"after W children started"| CAN ``` The following describes each step in the diagram: 1. The parent Workflow starts with a list of record IDs and a configured `windowSize`. 2. It starts the first `windowSize` children concurrently, one per record. Each child reads the parent's Workflow ID from its own Workflow metadata, so it knows where to signal. 3. As each child completes, it sends a completion signal to the parent. 4. The parent receives the signal, decrements its in-flight counter (`active`), and starts the next child (the next record in the list). 5. After starting `windowSize` children in total, the parent calls `continueAsNew` with the updated start index. The window slides forward without gaps because the parent's Workflow ID is preserved across runs. 6. Children from previous runs that have not yet signalled will find the new run when they send the signal, because the parent Workflow ID remains the same. ## Implementation The following examples show how each SDK implements the Sliding Window pattern. **TypeScript** ```typescript // workflows.ts import { ApplicationFailure, ParentClosePolicy, condition, continueAsNew, defineSignal, getExternalWorkflowHandle, proxyActivities, setHandler, startChild, workflowInfo, } from "@temporalio/workflow"; import type * as activities from "./activities"; import { COMPLETION_SIGNAL, TASK_QUEUE, WINDOW_SIZE, type SlidingWindowInput } from "./shared"; const { processRecord } = proxyActivities({ startToCloseTimeout: "30 seconds", }); export const completionSignal = defineSignal<[string]>(COMPLETION_SIGNAL); // Child Workflow: processes one record and signals the parent on completion. export async function recordProcessorWorkflow(recordId: string): Promise { await processRecord(recordId); // Read the parent's Workflow ID from context. It is stable across the parent's // Continue-as-New runs, so signaling by Workflow ID (no run ID) always reaches // the current run. Ignore NOT_FOUND — the parent's final run may have completed. try { const parent = getExternalWorkflowHandle(workflowInfo().parent!.workflowId); await parent.signal(completionSignal, recordId); } catch (err) { if (!(err instanceof ApplicationFailure && err.type === "NOT_FOUND")) throw err; } } // Parent Workflow: maintains a fixed window of concurrent Child Workflows and calls // Continue-as-New after dispatching windowSize children so history stays bounded. export async function slidingWindowWorkflow(input: SlidingWindowInput): Promise { const { recordIds, windowSize = WINDOW_SIZE, startIndex = 0 } = input; const parentId = workflowInfo().workflowId; // Total records completed across all runs, carried over via Continue-as-New. let totalProcessed = input.totalProcessed ?? 0; // Children started in this run; triggers Continue-as-New once it hits windowSize. let dispatched = 0; // Live in-flight count: +1 per start, -1 per completion signal, carried across runs. let active = input.active ?? 0; setHandler(completionSignal, () => { active--; totalProcessed++; }); // Slide the window: keep it full, starting one child per free slot. The first // (windowSize - active) slots are already free, so those children start without // waiting; after that, each start waits for an in-flight child to free a slot. let nextIndex = startIndex; while (nextIndex < recordIds.length) { // Backpressure: block until the window has a free slot. await condition(() => active < windowSize); await startChild(recordProcessorWorkflow, { args: [recordIds[nextIndex]], workflowId: `${parentId}/record-${recordIds[nextIndex]}`, taskQueue: TASK_QUEUE, parentClosePolicy: ParentClosePolicy.ABANDON, }); nextIndex++; dispatched++; active++; // Once this run has filled the window with fresh children, Continue-as-New so // history stays bounded. Carry active so the next run knows how many children // will still signal it. if (dispatched >= windowSize) { await continueAsNew({ recordIds, windowSize, startIndex: nextIndex, totalProcessed, active }); } } // Wait for all remaining in-flight children to complete. await condition(() => active === 0); return totalProcessed; } ``` **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow from temporalio.exceptions import ApplicationError from temporalio.workflow import ParentClosePolicy, continue_as_new from activities import process_record from shared import COMPLETION_SIGNAL, TASK_QUEUE, WINDOW_SIZE, SlidingWindowInput @workflow.defn class RecordProcessorWorkflow: """Child Workflow: processes one record and signals the parent on completion.""" @workflow.run async def run(self, record_id: str) -> None: await workflow.execute_activity( process_record, record_id, start_to_close_timeout=timedelta(seconds=30), ) # Read the parent's Workflow ID from context. It is stable across the parent's # continue_as_new runs, so signaling by Workflow ID always reaches the current # run. Ignore NOT_FOUND — the parent's final run may have already completed. parent = workflow.get_external_workflow_handle(workflow.info().parent.workflow_id) try: await parent.signal(COMPLETION_SIGNAL, record_id) except ApplicationError as e: if "not found" not in str(e).lower(): raise @workflow.defn class SlidingWindowWorkflow: """Parent Workflow: maintains a fixed window of concurrent Child Workflows.""" def __init__(self) -> None: # Live in-flight count: +1 per start, -1 per completion signal, carried across # runs. An instance field (not a run() local) because the signal handler is a # separate method and completions can signal before run() starts. self._active = 0 # Total records completed across all runs, carried over via continue_as_new. self._total_processed = 0 @workflow.signal(name=COMPLETION_SIGNAL) def record_completed(self, record_id: str) -> None: self._active -= 1 self._total_processed += 1 @workflow.run async def run(self, input: SlidingWindowInput) -> int: # Use += so completions that signal before run() starts are preserved. self._total_processed += input.total_processed self._active += input.active record_ids = input.record_ids window_size = input.window_size parent_id = workflow.info().workflow_id next_index = input.start_index # Children started in this run; triggers continue_as_new once it hits window_size. dispatched = 0 # Slide the window: keep it full, starting one child per free slot. The first # (window_size - active) slots are already free, so those children start without # waiting; after that, each start waits for an in-flight child to free a slot. while next_index < len(record_ids): # Backpressure: block until the window has a free slot. await workflow.wait_condition(lambda: self._active < window_size) await workflow.start_child_workflow( RecordProcessorWorkflow.run, record_ids[next_index], id=f"{parent_id}/record-{record_ids[next_index]}", task_queue=TASK_QUEUE, parent_close_policy=ParentClosePolicy.ABANDON, ) next_index += 1 dispatched += 1 self._active += 1 # Once this run has filled the window with fresh children, continue_as_new so # history stays bounded. Carry _active so the next run knows how many children # will still signal it. if dispatched >= window_size: continue_as_new(args=[SlidingWindowInput( record_ids=record_ids, window_size=window_size, start_index=next_index, total_processed=self._total_processed, active=self._active, )]) # Wait for all remaining in-flight children to complete. await workflow.wait_condition(lambda: self._active == 0) return self._total_processed ``` **Go** ```go // workflows.go package main import ( "fmt" "strings" "time" enums "go.temporal.io/api/enums/v1" "go.temporal.io/sdk/workflow" ) const CompletionSignal = "recordCompleted" // RecordProcessorWorkflow is the child Workflow: it processes one record and // signals the parent on completion. func RecordProcessorWorkflow(ctx workflow.Context, recordID string) error { ao := workflow.ActivityOptions{StartToCloseTimeout: 30 * time.Second} ctx = workflow.WithActivityOptions(ctx, ao) if err := workflow.ExecuteActivity(ctx, ProcessRecord, recordID).Get(ctx, nil); err != nil { return err } // Read the parent's Workflow ID from context. It is stable across the parent's // ContinueAsNew runs, so signaling by Workflow ID (empty run ID) always reaches // the current run. Ignore not-found — the parent's final run may have completed. parentID := workflow.GetInfo(ctx).ParentWorkflowExecution.ID err := workflow.SignalExternalWorkflow(ctx, parentID, "", CompletionSignal, recordID).Get(ctx, nil) if err != nil && strings.Contains(err.Error(), "not found") { return nil } return err } // SlidingWindowWorkflow is the parent Workflow: it maintains a fixed window of // concurrent Child Workflows and calls ContinueAsNew after dispatching windowSize // children so history stays bounded. func SlidingWindowWorkflow(ctx workflow.Context, input SlidingWindowInput) (int, error) { windowSize := input.WindowSize if windowSize <= 0 { windowSize = WindowSize } recordIDs := input.RecordIDs parentID := workflow.GetInfo(ctx).WorkflowExecution.ID completedCh := workflow.GetSignalChannel(ctx, CompletionSignal) nextIndex := input.StartIndex // Total records completed across all runs, carried over via ContinueAsNew. totalProcessed := input.TotalProcessed // Children started in this run; triggers ContinueAsNew once it hits windowSize. dispatched := 0 // Live in-flight count: +1 per start, -1 per completion signal, carried across runs. active := input.Active startChild := func(recordID string) error { cwo := workflow.ChildWorkflowOptions{ WorkflowID: fmt.Sprintf("%s/record-%s", parentID, recordID), TaskQueue: TaskQueue, ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON, } future := workflow.ExecuteChildWorkflow(workflow.WithChildOptions(ctx, cwo), RecordProcessorWorkflow, recordID) // Wait for the child to start so the command commits before any ContinueAsNew. return future.GetChildWorkflowExecution().Get(ctx, nil) } // Slide the window: keep it full, starting one child per free slot. for nextIndex < len(recordIDs) { // Backpressure: if the window is full, block on the completion channel until an // in-flight child signals, freeing a slot. Otherwise start without waiting. if active >= windowSize { completedCh.Receive(ctx, nil) totalProcessed++ active-- } if err := startChild(recordIDs[nextIndex]); err != nil { return 0, err } nextIndex++ dispatched++ active++ // Once this run has filled the window with fresh children, ContinueAsNew so // history stays bounded. Carry active so the next run knows how many children // will still signal it. if dispatched >= windowSize { return 0, workflow.NewContinueAsNewError(ctx, SlidingWindowWorkflow, SlidingWindowInput{ RecordIDs: recordIDs, WindowSize: windowSize, StartIndex: nextIndex, TotalProcessed: totalProcessed, Active: active, }) } } // Drain all remaining in-flight children. for active > 0 { completedCh.Receive(ctx, nil) totalProcessed++ active-- } return totalProcessed, nil } ``` **Java** ```java // SlidingWindowWorkflow.java import io.temporal.activity.ActivityOptions; import io.temporal.api.enums.v1.ParentClosePolicy; import io.temporal.workflow.*; import java.time.Duration; import java.util.List; public interface SlidingWindowWorkflow { /** Parent Workflow: maintains a fixed window of concurrent Child Workflows. */ @WorkflowInterface interface Parent { @WorkflowMethod int run(Shared.SlidingWindowInput input); @SignalMethod void recordCompleted(String recordId); } /** Child Workflow: processes one record and signals the parent on completion. */ @WorkflowInterface interface Child { @WorkflowMethod void run(String recordId); } final class ParentImpl implements Parent { // Live in-flight count: +1 per start, -1 per completion signal, carried across // runs. An instance field (not a run() local) because the signal handler is a // separate method and completions can signal before run() starts. private int active = 0; // Total records completed across all runs, carried over via Continue-as-New. private int totalProcessed = 0; @Override public void recordCompleted(String recordId) { active--; totalProcessed++; } @Override public int run(Shared.SlidingWindowInput input) { // Use += so completions that signal before run() starts are preserved. this.totalProcessed += input.totalProcessed; this.active += input.active; int windowSize = input.windowSize > 0 ? input.windowSize : Shared.WINDOW_SIZE; List recordIds = input.recordIds; String parentId = Workflow.getInfo().getWorkflowId(); int nextIndex = input.startIndex; // Children started in this run; triggers Continue-as-New once it hits windowSize. int dispatched = 0; // Slide the window: keep it full, starting one child per free slot. The first // (windowSize - active) slots are already free, so those children start without // waiting; after that, each start waits for an in-flight child to free a slot. while (nextIndex < recordIds.size()) { // Backpressure: block until the window has a free slot. Workflow.await(() -> active < windowSize); String recordId = recordIds.get(nextIndex); ChildWorkflowOptions opts = ChildWorkflowOptions.newBuilder() .setWorkflowId(parentId + "/record-" + recordId) .setTaskQueue(Shared.TASK_QUEUE) .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON) .build(); Child child = Workflow.newChildWorkflowStub(Child.class, opts); Async.procedure(child::run, recordId); // Wait until the child has started before counting it (and before any // Continue-as-New, which would otherwise race child startup). Workflow.getWorkflowExecution(child).get(); nextIndex++; dispatched++; active++; // Once this run has filled the window with fresh children, Continue-as-New // so history stays bounded. Carry active so the next run knows how many // children will still signal it. if (dispatched >= windowSize) { Workflow.newContinueAsNewStub(Parent.class) .run(new Shared.SlidingWindowInput( recordIds, windowSize, nextIndex, this.totalProcessed, active)); return 0; // unreachable; Continue-as-New throws } } // Wait for all remaining in-flight children to complete. Workflow.await(() -> active == 0); return this.totalProcessed; } } final class ChildImpl implements Child { private final Activities activities = Workflow.newActivityStub( Activities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); @Override public void run(String recordId) { activities.processRecord(recordId); // Read the parent's Workflow ID from context. It is stable across the parent's // Continue-as-New runs, so signaling by Workflow ID always reaches the current // run. Ignore not-found — the parent's final run may have already completed. String parentWorkflowId = Workflow.getInfo().getParentWorkflowId().orElseThrow(); ExternalWorkflowStub parent = Workflow.newUntypedExternalWorkflowStub(parentWorkflowId); try { parent.signal(Shared.COMPLETION_SIGNAL, recordId); } catch (Exception e) { String msg = e.getMessage() != null ? e.getMessage() : ""; if (!msg.toLowerCase().contains("not found")) { throw e; } } } } } ``` ## Best practices - **Preserve the parent Workflow ID across Continue-as-New.** The parent's Workflow ID is stable across `continueAsNew` runs — do not generate a new one. Children read the parent's Workflow ID from their own Workflow metadata (`workflowInfo().parent` in TypeScript, `workflow.info().parent` in Python, `workflow.GetInfo(ctx).ParentWorkflowExecution` in Go, `Workflow.getInfo().getParentWorkflowId()` in Java) rather than receiving it as an argument, then signal by Workflow ID (with no run ID) so they always reach the current run. - **Use `PARENT_CLOSE_POLICY_ABANDON` on child Workflows.** This lets children that were started by a previous run complete normally even after the parent has continued as new. - **Size the window conservatively at first.** Each in-flight child counts toward the 2,000 unfinished-actions limit for the parent. A window of 50–200 is a reasonable starting point depending on child duration and downstream capacity. - **Pass only IDs (not full records) to child Workflows.** Workflow inputs are stored in event history. Keep them small. - **Carry minimal state into `continueAsNew`.** Pass `windowSize`, `startIndex`, the live in-flight count (`active`), a running `totalProcessed`, and the record ID list (or a reference to it). Do not accumulate results in the parent — collect them out-of-band if needed. ## Common pitfalls - **Losing signals across Continue-as-New.** If a child signals before the parent's new run has registered the signal handler, the signal can be buffered and delivered correctly — Temporal buffers signals for existing Workflow IDs. However, ensure the signal handler is registered before any await, not conditionally. - **Race between Continue-as-New and remaining signal draining.** After `continueAsNew`, the new run must handle signals from children started by the previous run. Pass `startIndex` (the next *unstarted* record) and `active` (the live in-flight count at the moment of CAN) to the new run so it knows how many carried-over children to expect signals from, without re-starting them. The new run folds `active` in with `+=`, so a completion that arrives before `run()` executes is still counted correctly. - **Thundering herd on startup.** Starting hundreds of children simultaneously causes a burst of Activity polls. Ramp up the window gradually or use the [Batch Iterator](/design-patterns/batch-iterator) if rate limiting is more important than throughput. ## Related ### Patterns - [Continue-as-New pattern](/design-patterns/continue-as-new) — history management fundamentals - [Batch Iterator](/design-patterns/batch-iterator) — sequential alternative when ordered, one-at-a-time processing is acceptable - [MapReduce Tree](/design-patterns/mapreduce-tree) — fully parallel alternative when rate limiting is not needed - [Temporal limits reference](/cloud/limits) - [Sliding window sample (Java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch/slidingwindow) --- # Task Orchestration Patterns Source: https://docs.temporal.io/design-patterns/task-orchestration-patterns > Pattern selection guide for composing and coordinating multiple units of work within a Workflow. These patterns compose and coordinate multiple units of work within a Workflow — decomposing large processes into reusable pieces, running work concurrently, and racing alternatives against each other. ## Patterns in this section - [Child Workflows](/design-patterns/child-workflows): Decomposes a complex Workflow into smaller, reusable units. Each child has its own Workflow ID, history, and lifecycle. - [Parallel Execution](/design-patterns/parallel-execution): Runs multiple Activities concurrently for higher throughput, with error handling and a bound on how many run at once. - [Pick First (Race)](/design-patterns/pick-first): Starts multiple Activities in parallel, takes the first result to arrive, and cancels the rest. ## Choosing a pattern **A process is large or reused across Workflows**: break it into [Child Workflows](/design-patterns/child-workflows) with independent histories. **Independent work can run at the same time**: use [Parallel Execution](/design-patterns/parallel-execution) with a concurrency bound. **Several approaches compete and you want the first to finish**: use [Pick First (Race)](/design-patterns/pick-first) and cancel the losers. ## Related sections - [Batch Processing Patterns](/design-patterns/batch-processing-patterns) — fan-out orchestration scaled to large record sets - [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — reduce the latency of the work each task performs --- # Updatable / Debounced Timer Pattern Source: https://docs.temporal.io/design-patterns/updatable-timer > Dynamically adjustable timers that respond to Signals or Updates. Extend, shorten, or cancel timers based on external events. ## Overview The Updatable / Debounced Timer pattern implements a sleep operation that can be interrupted and dynamically adjusted via Signals. It enables Workflows to wait for deadlines that can be extended or shortened based on external events, making it suitable for approval processes, SLA management, and time-sensitive business operations. ## Problem In business processes, you often need Workflows that wait for a deadline (approval timeout, SLA expiration, grace period), allow the deadline to be extended or shortened dynamically, react immediately when the deadline changes, and continue waiting with the new deadline without restarting. Without an updatable timer, you must use fixed timeouts that cannot be adjusted, cancel and restart Workflows to change deadlines, poll frequently to check for deadline changes, or implement complex state machines to handle timing updates. ## Solution The Updatable / Debounced Timer uses a blocking wait with both a time limit and an update condition. When a Signal updates the wake-up time, the condition becomes true, the Workflow recalculates the sleep duration, and blocks again with the new deadline. Each SDK provides a different mechanism for this: - **Java**: `Workflow.await(Duration, condition)` returns `false` when the duration expires, or `true` when the condition is met. - **TypeScript**: `wf.condition(fn, timeout)` returns `false` when the timeout expires, or `true` when the function returns `true`. - **Python**: `workflow.wait_condition(fn, timeout=duration)` returns normally when the condition is met, or raises `asyncio.TimeoutError` on timeout. - **Go**: `workflow.NewTimer()` combined with `workflow.NewSelector()` to race a timer against a Signal channel. ```mermaid sequenceDiagram participant Client participant Workflow participant Timer Client->>Workflow: Start with deadline activate Workflow Workflow->>Timer: sleepUntil(deadline) activate Timer Note over Timer: Waiting... Client->>Workflow: Signal: extendDeadline(newTime) Workflow->>Timer: Update wakeUpTime Timer->>Timer: Recalculate duration Note over Timer: Waiting with new deadline... Timer-->>Workflow: Timer expired deactivate Timer Workflow-->>Client: Complete deactivate Workflow ``` The following describes each step in the diagram: 1. The client starts the Workflow with an initial deadline. 2. The Workflow calls `sleepUntil(deadline)`, which blocks until the deadline. 3. The client sends a Signal to extend the deadline. 4. The timer recalculates the remaining duration based on the new deadline and continues waiting. 5. When the timer expires, the Workflow completes. The core of the pattern is a reusable timer helper that loops on a blocking wait, recalculating the sleep duration each time the wake-up time is updated: **Python** ```python # updatable_timer.py import asyncio from datetime import timedelta from temporalio import workflow class UpdatableTimer: def __init__(self, wake_up_time: float) -> None: self._wake_up_time = wake_up_time self._wake_up_time_updated = False async def sleep_until(self, wake_up_time: float) -> None: self._wake_up_time = wake_up_time while True: self._wake_up_time_updated = False sleep_secs = self._wake_up_time - workflow.time() try: await workflow.wait_condition( lambda: self._wake_up_time_updated, timeout=timedelta(seconds=max(sleep_secs, 0)), ) # Condition met: wake-up time was updated, loop to recalculate except asyncio.TimeoutError: break # Timer expired def update_wake_up_time(self, wake_up_time: float) -> None: self._wake_up_time = wake_up_time self._wake_up_time_updated = True # Unblocks wait_condition @property def wake_up_time(self) -> float: return self._wake_up_time ``` **Go** ```go // updatable_timer.go func sleepUntil(ctx workflow.Context, wakeUpTime time.Time, wakeUpChannel workflow.ReceiveChannel) error { for { timerCtx, cancelTimer := workflow.WithCancel(ctx) duration := wakeUpTime.Sub(workflow.Now(ctx)) if duration <= 0 { cancelTimer() break } timer := workflow.NewTimer(timerCtx, duration) selector := workflow.NewSelector(ctx) timerFired := false selector.AddFuture(timer, func(f workflow.Future) { timerFired = true }) selector.AddReceive(wakeUpChannel, func(c workflow.ReceiveChannel, more bool) { c.Receive(ctx, &wakeUpTime) // Cancel the current timer so it can be recreated with the new deadline cancelTimer() }) selector.Select(ctx) if timerFired { break // Timer expired } // Signal received with new wakeUpTime, loop to recalculate } return nil } ``` **Java** ```java // UpdatableTimer.java public class UpdatableTimer { private long wakeUpTime; private boolean wakeUpTimeUpdated; public void sleepUntil(long wakeUpTime) { this.wakeUpTime = wakeUpTime; while (true) { wakeUpTimeUpdated = false; Duration sleepInterval = Duration.ofMillis(this.wakeUpTime - Workflow.currentTimeMillis()); if (!Workflow.await(sleepInterval, () -> wakeUpTimeUpdated)) { break; // Timer expired } // Timer was updated, loop to recalculate } } public void updateWakeUpTime(long wakeUpTime) { this.wakeUpTime = wakeUpTime; this.wakeUpTimeUpdated = true; // Unblocks await } } ``` **TypeScript** ```typescript // updatable-timer.ts import * as wf from '@temporalio/workflow'; export class UpdatableTimer implements PromiseLike { deadlineUpdated = false; #deadline: number; constructor(deadline: number) { this.#deadline = deadline; } private async run(): Promise { while (true) { this.deadlineUpdated = false; if ( !(await wf.condition( () => this.deadlineUpdated, this.#deadline - Date.now(), )) ) { break; // Timer expired } // Timer was updated, loop to recalculate } } then( onfulfilled?: (value: void) => TResult1 | PromiseLike, onrejected?: (reason: any) => TResult2 | PromiseLike, ): PromiseLike { return this.run().then(onfulfilled, onrejected); } set deadline(value: number) { this.#deadline = value; this.deadlineUpdated = true; } get deadline(): number { return this.#deadline; } } ``` In Java and TypeScript, the `sleepUntil` method calculates the sleep interval and calls a blocking wait with both a duration and a condition. If the duration expires first, the wait returns `false` (Java/TypeScript) or raises `asyncio.TimeoutError` (Python), and the timer completes. If the update flag is set via a Signal, the condition becomes true, the wait unblocks, and the loop recalculates the interval with the new deadline. In Go, a `Selector` races a `Timer` against a Signal channel; when the Signal arrives, the current timer is cancelled and a new one is created with the updated deadline. ## Implementation ### Basic approval Workflow The following implementation combines the updatable timer with an approval flag. The Workflow waits for either an approval Signal or the deadline to expire: **Python** ```python # workflows.py import asyncio from datetime import timedelta from temporalio import workflow @workflow.defn class ApprovalWorkflow: def __init__(self) -> None: self._approved = False self._status = "PENDING" @workflow.run async def run(self, approval_deadline: float) -> None: timeout_secs = approval_deadline - workflow.time() try: await workflow.wait_condition( lambda: self._approved, timeout=timedelta(seconds=max(timeout_secs, 0)), ) self._status = "APPROVED" except asyncio.TimeoutError: self._status = "REJECTED" @workflow.signal def approve(self) -> None: self._approved = True @workflow.query def get_status(self) -> str: return self._status ``` **Go** ```go // workflow.go func ApprovalWorkflow(ctx workflow.Context, approvalDeadline time.Time) (string, error) { logger := workflow.GetLogger(ctx) status := "PENDING" approved := false // Listen for the approve signal in a goroutine workflow.Go(ctx, func(ctx workflow.Context) { ch := workflow.GetSignalChannel(ctx, "approve") ch.Receive(ctx, nil) approved = true }) // Wait for approval or timeout duration := approvalDeadline.Sub(workflow.Now(ctx)) ok, _ := workflow.AwaitWithTimeout(ctx, duration, func() bool { return approved }) if ok { status = "APPROVED" } else { status = "REJECTED" } logger.Info("Approval workflow completed", "status", status) return status, nil } ``` **Java** ```java // ApprovalWorkflowImpl.java @WorkflowInterface public interface ApprovalWorkflow { @WorkflowMethod void execute(long approvalDeadline); @SignalMethod void extendDeadline(long newDeadline); @SignalMethod void approve(); @QueryMethod String getStatus(); } public class ApprovalWorkflowImpl implements ApprovalWorkflow { private UpdatableTimer timer = new UpdatableTimer(); private boolean approved = false; private String status = "PENDING"; @Override public void execute(long approvalDeadline) { Workflow.await( Duration.ofMillis(approvalDeadline - Workflow.currentTimeMillis()), () -> approved); if (approved) { status = "APPROVED"; } else { status = "REJECTED"; } } @Override public void extendDeadline(long newDeadline) { timer.updateWakeUpTime(newDeadline); } @Override public void approve() { approved = true; } @Override public String getStatus() { return status; } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; export const extendDeadlineSignal = wf.defineSignal<[number]>('extendDeadline'); export const approveSignal = wf.defineSignal('approve'); export const getStatusQuery = wf.defineQuery('getStatus'); export async function approvalWorkflow(approvalDeadline: number): Promise { let approved = false; let status = 'PENDING'; wf.setHandler(approveSignal, () => { approved = true; }); wf.setHandler(getStatusQuery, () => status); // Wait for approval or deadline expiration const approvedBeforeDeadline = await wf.condition( () => approved, approvalDeadline - Date.now(), ); status = approvedBeforeDeadline ? 'APPROVED' : 'REJECTED'; } ``` The Workflow waits with both a deadline duration and a condition that checks the `approved` flag. If the `approve` Signal arrives before the deadline, the condition becomes true and the Workflow sets the status to APPROVED. If the deadline expires first, the Workflow sets the status to REJECTED. ### Multiple deadline extensions The following implementation uses the `UpdatableTimer` directly to support multiple deadline extensions. The Workflow blocks on the timer helper and checks the approval flag after the timer completes: **Python** ```python # workflows.py from temporalio import workflow from .updatable_timer import UpdatableTimer @workflow.defn class MultiExtensionApprovalWorkflow: def __init__(self) -> None: self._timer = UpdatableTimer(0) self._approved = False self._rejected = False @workflow.run async def run(self, initial_deadline: float) -> None: await self._timer.sleep_until(initial_deadline) if not self._approved: self._rejected = True @workflow.signal def extend_deadline(self, new_deadline: float) -> None: if not self._approved and not self._rejected: self._timer.update_wake_up_time(new_deadline) @workflow.signal def approve(self) -> None: self._approved = True ``` **Go** ```go // workflow.go func MultiExtensionApprovalWorkflow(ctx workflow.Context, initialDeadline time.Time) (string, error) { approved := false rejected := false wakeUpTime := initialDeadline wakeUpChannel := workflow.NewChannel(ctx) // Listen for approval signal workflow.Go(ctx, func(ctx workflow.Context) { ch := workflow.GetSignalChannel(ctx, "approve") ch.Receive(ctx, nil) approved = true }) // Listen for deadline extension signals workflow.Go(ctx, func(ctx workflow.Context) { ch := workflow.GetSignalChannel(ctx, "extendDeadline") for { var newDeadline time.Time ch.Receive(ctx, &newDeadline) if !approved && !rejected { wakeUpChannel.Send(ctx, newDeadline) } } }) // Block on the updatable timer _ = sleepUntil(ctx, wakeUpTime, wakeUpChannel) if !approved { rejected = true } if rejected { return "REJECTED", nil } return "APPROVED", nil } ``` **Java** ```java // MultiExtensionApprovalWorkflowImpl.java public class MultiExtensionApprovalWorkflowImpl implements ApprovalWorkflow { private UpdatableTimer timer = new UpdatableTimer(); private boolean approved = false; private boolean rejected = false; @Override public void execute(long initialDeadline) { timer.sleepUntil(initialDeadline); if (!approved) { rejected = true; } } @Override public void extendDeadline(long newDeadline) { if (!approved && !rejected) { timer.updateWakeUpTime(newDeadline); } } @Override public void approve() { approved = true; } } ``` **TypeScript** ```typescript // workflows.ts import * as wf from '@temporalio/workflow'; import { UpdatableTimer } from './updatable-timer'; export const extendDeadlineSignal = wf.defineSignal<[number]>('extendDeadline'); export const approveSignal = wf.defineSignal('approve'); export async function multiExtensionApprovalWorkflow( initialDeadline: number, ): Promise { let approved = false; let rejected = false; const timer = new UpdatableTimer(initialDeadline); wf.setHandler(extendDeadlineSignal, (newDeadline: number) => { if (!approved && !rejected) { timer.deadline = newDeadline; } }); wf.setHandler(approveSignal, () => { approved = true; }); await timer; // Blocks until the timer expires if (!approved) { rejected = true; } } ``` The `extendDeadline` Signal handler checks that the Workflow has not already been approved or rejected before updating the timer. Each update unblocks the timer loop, which recalculates the remaining duration and blocks again. ## When to use The Updatable Timer pattern is a good fit for approval Workflows with deadline extensions, SLA management with grace periods, time-based escalations that can be postponed, auction bidding with extended closing times, and payment grace periods that can be adjusted. It is not a good fit for fixed timeouts that never change (use a fixed sleep), immediate cancellation (use cancellation scopes), or complex scheduling (use Temporal Schedules). ## Benefits and trade-offs The pattern allows you to adjust deadlines without restarting Workflows. Changes take effect instantly. The timer helper is reusable across multiple Workflows. All timing is based on Workflow time, ensuring replay consistency. You can Query the current deadline at any time. The trade-offs to consider are that the pattern requires an external process to send update Signals. Each timer instance manages one deadline. Previous deadlines are not tracked (add tracking if needed). You must calculate absolute timestamps rather than relative durations. ## Comparison with alternatives | Approach | Dynamic updates | Complexity | Use case | | :--- | :--- | :--- | :--- | | Updatable / Debounced Timer | Yes | Medium | Adjustable deadlines | | Fixed sleep | No | Low | Fixed delays | | Cancellation Scope | Yes (cancel only) | Medium | Abort operations | | Polling Loop | Yes | High | Frequent checks | ## Best practices - **Use absolute timestamps.** Store wake-up time as an absolute value (epoch millis in Java/TypeScript, epoch seconds in Python, `time.Time` in Go), not relative durations. - **Validate updates.** Ensure new deadlines are in the future. - **Add Queries.** Expose the current deadline via Query methods. - **Handle edge cases.** Check if the timer already expired before updating. - **Consider max extensions.** Limit how many times or how far deadlines can be extended. - **Log changes.** Log each deadline update for observability. - **Reuse the timer helper.** Extract to a helper class or function for use across Workflows. - **Combine with conditions.** Use a blocking wait with both time and business conditions. ## Common pitfalls - **Using time-based conditions without a duration.** A wait without a timeout does not create a timer. The condition is only re-evaluated on state changes (Signals, Activity completions). Always provide a timeout for time-based waits. - **Expecting the wait to re-evaluate its duration.** The timer duration is set once when the wait is called. Changing the duration variable afterward has no effect. This is why the timer helper loops and recalculates. - **Not validating new deadlines.** Accepting a deadline in the past causes the timer to expire immediately. Always check that the new deadline is in the future before updating. - **Accumulating uncancelled timers in Java.** In the Java SDK, `Workflow.await(Duration, condition)` does not automatically cancel its internal timer when the condition is met. Repeated calls in a loop accumulate timers. Wrap in a `CancellationScope` if this is a concern. - **Not cancelling timers in Go.** In the Go SDK, always cancel the previous timer (via `workflow.WithCancel`) before creating a new one. Uncancelled timers wake up the Workflow unnecessarily, creating extra Worker load. ## Related ### Patterns - **[Signal with Start](/design-patterns/signal-with-start)**: Receiving external events to modify behavior. - **[Approval Pattern](/design-patterns/approval)**: Approval Workflows with adjustable deadlines. ### Sample code - [Java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/updatabletimer) -- Complete implementation with starter and updater. - [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/timer-examples) -- Updatable timer with `condition` and `UpdatableTimer` class. - [Python](https://github.com/temporalio/samples-python/tree/main/updatable_timer) -- Updatable timer with `wait_condition` and helper class. - [Go](https://github.com/temporalio/samples-go/tree/main/updatabletimer) -- Timer cancellation with Selector, Signal channel, and `WithCancel`. --- # Worker Configuration Patterns Source: https://docs.temporal.io/design-patterns/worker-configuration-patterns > Pattern selection guide for configuring how Workers are set up, how work is routed, and how Activities access external dependencies. These patterns cover how you set up Workers, route work to them, and give Activities the external dependencies they need — while keeping Workflow code deterministic and testable. ## Patterns in this section - [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue): Routes Activities to a specific Worker using a unique Task Queue, for Worker affinity and host-specific processing. - [Activity Dependency Injection](/design-patterns/activity-dependency-injection): Injects external dependencies — clients, connections, configuration — into Activities at Worker startup, keeping Workflow code deterministic and Activities testable. ## Choosing a pattern **A sequence of Activities must run on the same Worker host**: use [Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue) to pin them to one Worker. **Activities depend on external resources**: use [Activity Dependency Injection](/design-patterns/activity-dependency-injection) to supply them at startup rather than constructing them inside each Activity. ## Related sections - [QoS & Throughput Patterns](/design-patterns/qos-throughput-patterns) — Task Queue routing for rate control and fairness - [Performance & Latency Patterns](/design-patterns/performance-latency-patterns) — run Activities in-process with Local Activities --- # Worker-Specific Task Queues Pattern Source: https://docs.temporal.io/design-patterns/worker-specific-taskqueue > Routes Activities to specific Workers using unique Task Queues for Worker affinity and host-specific processing. ## Overview The Worker-Specific Task Queues pattern enables routing Activities to specific Worker hosts when Activities must execute on the same machine. This is essential for Workflows where subsequent Activities depend on local state, files, or resources created by previous Activities on a particular host. ## Problem In distributed systems, you often need Workflows that download a file to a Worker's local disk and then process and upload it from the same location, establish a connection or session that subsequent Activities must reuse, create temporary resources on one host that later Activities need to access, or maintain affinity to a specific Worker for performance or data locality. Without Worker-specific routing, Activities execute on different hosts and cannot access local files or state. You must set up complex distributed file systems or shared storage, handle race conditions when multiple Workers access the same resources, and accept that you cannot guarantee Activity colocation. ## Solution You use a two-tier Task Queue architecture: a default shared Task Queue for initial Activities, and dynamically-named host-specific Task Queues for Activities that must run on the same Worker. The first Activity returns its host-specific Task Queue name, and subsequent Activities use that queue. ```mermaid sequenceDiagram participant Workflow participant Worker1 participant Worker2 participant Worker3 Note over Workflow: Default Task Queue: "FileProcessing" Workflow->>Worker2: download() - any worker activate Worker2 Worker2-->>Workflow: {hostQueue: "FileProcessing-host2", file: "/tmp/data"} deactivate Worker2 Note over Workflow: Switch to host-specific queue Note over Workflow: Task Queue: "FileProcessing-host2" Workflow->>Worker2: process(file) - MUST be Worker2 activate Worker2 Worker2-->>Workflow: processed file deactivate Worker2 Workflow->>Worker2: upload(file) - MUST be Worker2 activate Worker2 Worker2-->>Workflow: done deactivate Worker2 Note over Worker1,Worker3: Workers 1 & 3 never see
host-specific activities ``` The following describes each step in the diagram: 1. The Workflow dispatches the download Activity on the default Task Queue. Any available Worker picks it up. 2. Worker 2 downloads the file and returns both the local file path and its host-specific Task Queue name. 3. The Workflow creates new Activity options targeting Worker 2's host-specific Task Queue. 4. The process and upload Activities execute on Worker 2, where the file is already on disk. 5. Workers 1 and 3 never see the host-specific Activities. The following snippet shows how the Workflow switches from the default Task Queue to the host-specific queue: **Python** ```python # workflows.py downloaded = await workflow.execute_activity( download, source, start_to_close_timeout=timedelta(seconds=20), ) processed = await workflow.execute_activity( process, downloaded.file_name, task_queue=downloaded.host_task_queue, schedule_to_start_timeout=timedelta(seconds=10), start_to_close_timeout=timedelta(seconds=20), ) await workflow.execute_activity( upload, args=[processed, destination], task_queue=downloaded.host_task_queue, schedule_to_start_timeout=timedelta(seconds=10), start_to_close_timeout=timedelta(seconds=20), ) ``` **Go** ```go // workflow.go var defaultActivities *StoreActivities downloaded, err := defaultActivities.Download(ctx, source) if err != nil { return err } hostOptions := workflow.ActivityOptions{ TaskQueue: downloaded.HostTaskQueue, ScheduleToStartTimeout: 10 * time.Second, StartToCloseTimeout: 20 * time.Second, } hostCtx := workflow.WithActivityOptions(ctx, hostOptions) var hostActivities *StoreActivities processed, err := hostActivities.Process(hostCtx, downloaded.FileName) if err != nil { return err } err = hostActivities.Upload(hostCtx, processed, destination) ``` **Java** ```java // FileProcessingWorkflowImpl.java TaskQueueFileNamePair downloaded = defaultTaskQueueActivities.download(source); ActivityOptions hostOptions = ActivityOptions.newBuilder() .setTaskQueue(downloaded.getHostTaskQueue()) .setScheduleToStartTimeout(Duration.ofSeconds(10)) .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); StoreActivities hostSpecificActivities = Workflow.newActivityStub(StoreActivities.class, hostOptions); String processed = hostSpecificActivities.process(downloaded.getFileName()); hostSpecificActivities.upload(processed, destination); ``` **TypeScript** ```typescript // workflows.ts const { download } = proxyActivities({ startToCloseTimeout: '20s', }); const downloaded = await download(source); const hostSpecificActivities = proxyActivities({ taskQueue: downloaded.hostTaskQueue, scheduleToStartTimeout: '10s', startToCloseTimeout: '20s', }); const processed = await hostSpecificActivities.process(downloaded.fileName); await hostSpecificActivities.upload(processed, destination); ``` The `taskQueue` option (or `setTaskQueue()` in Java) routes subsequent Activities to the specific Worker that downloaded the file. The `scheduleToStartTimeout` (or `setScheduleToStartTimeout()` in Java) is critical — if the specific Worker is unavailable, the Activity Task cannot sit in the host-specific queue indefinitely. This timeout is non-retryable: when it expires, the Activity fails rather than retrying, because a retry would only place the Task back on the same queue. The Workflow catches that failure and retries the entire sequence so the work can restart on a different host. ## Implementation ### Activity Definition with host-specific return The download Activity returns both the file path and the host-specific Task Queue name: **Python** ```python # activities.py from dataclasses import dataclass from temporalio import activity @dataclass class TaskQueueFileNamePair: host_task_queue: str file_name: str @activity.defn async def download(source: str) -> TaskQueueFileNamePair: local_file = await download_to_local_disk(source) return TaskQueueFileNamePair( host_task_queue=host_specific_task_queue, file_name=local_file, ) @activity.defn async def process(file_name: str) -> str: return await process_local_file(file_name) @activity.defn async def upload(file_name: str, destination: str) -> None: await upload_from_local_disk(file_name, destination) ``` **Go** ```go // activities.go type TaskQueueFileNamePair struct { HostTaskQueue string FileName string } type StoreActivities struct { HostSpecificTaskQueue string } func (a *StoreActivities) Download(ctx context.Context, source string) (*TaskQueueFileNamePair, error) { localFile, err := downloadToLocalDisk(source) if err != nil { return nil, err } return &TaskQueueFileNamePair{ HostTaskQueue: a.HostSpecificTaskQueue, FileName: localFile, }, nil } func (a *StoreActivities) Process(ctx context.Context, fileName string) (string, error) { return processLocalFile(fileName) } func (a *StoreActivities) Upload(ctx context.Context, fileName string, destination string) error { return uploadFromLocalDisk(fileName, destination) } ``` **Java** ```java // StoreActivities.java public interface StoreActivities { class TaskQueueFileNamePair { private final String hostTaskQueue; private final String fileName; public TaskQueueFileNamePair(String hostTaskQueue, String fileName) { this.hostTaskQueue = hostTaskQueue; this.fileName = fileName; } public String getHostTaskQueue() { return hostTaskQueue; } public String getFileName() { return fileName; } } TaskQueueFileNamePair download(URL source); String process(String fileName); void upload(String fileName, URL destination); } ``` **TypeScript** ```typescript // activities.ts export interface TaskQueueFileNamePair { hostTaskQueue: string; fileName: string; } export async function download(source: string): Promise { const localFile = await downloadToLocalDisk(source); return { hostTaskQueue: getHostSpecificTaskQueue(), fileName: localFile, }; } export async function process(fileName: string): Promise { return await processLocalFile(fileName); } export async function upload(fileName: string, destination: string): Promise { await uploadFromLocalDisk(fileName, destination); } ``` The download Activity bundles the local file path with the Task Queue name so the Workflow knows where to route subsequent Activities. ### Activity implementation The Activity implementation receives the host-specific Task Queue name at construction time and includes it in the download result: **Python** ```python # activities.py # In Python, the host-specific Task Queue name is injected at Worker # startup and captured by the activity closure or class instance. host_specific_task_queue: str = "" @activity.defn async def download(source: str) -> TaskQueueFileNamePair: local_file = await download_to_local_disk(source) return TaskQueueFileNamePair( host_task_queue=host_specific_task_queue, file_name=local_file, ) @activity.defn async def process(file_name: str) -> str: processed = await process_local_file(file_name) return processed @activity.defn async def upload(file_name: str, destination: str) -> None: await upload_from_local_disk(file_name, destination) ``` **Go** ```go // activities.go // In Go, the host-specific Task Queue name is set on the struct // at Worker startup and returned by the Download method. func (a *StoreActivities) Download(ctx context.Context, source string) (*TaskQueueFileNamePair, error) { localFile, err := downloadToLocalDisk(source) if err != nil { return nil, err } return &TaskQueueFileNamePair{ HostTaskQueue: a.HostSpecificTaskQueue, FileName: localFile, }, nil } func (a *StoreActivities) Process(ctx context.Context, fileName string) (string, error) { return processLocalFile(fileName) } func (a *StoreActivities) Upload(ctx context.Context, fileName string, destination string) error { return uploadFromLocalDisk(fileName, destination) } ``` **Java** ```java // StoreActivitiesImpl.java public class StoreActivitiesImpl implements StoreActivities { private final String hostSpecificTaskQueue; public StoreActivitiesImpl(String hostSpecificTaskQueue) { this.hostSpecificTaskQueue = hostSpecificTaskQueue; } @Override public TaskQueueFileNamePair download(URL source) { File localFile = downloadToLocalDisk(source); return new TaskQueueFileNamePair( hostSpecificTaskQueue, localFile.getAbsolutePath()); } @Override public String process(String fileName) { File processed = processLocalFile(new File(fileName)); return processed.getAbsolutePath(); } @Override public void upload(String fileName, URL destination) { uploadFromLocalDisk(new File(fileName), destination); } } ``` **TypeScript** ```typescript // activities.ts // In TypeScript, the host-specific Task Queue name is captured via // closure when defining the activity functions. A common approach is // to initialize it at Worker startup and reference it from activities. let hostSpecificTaskQueue: string; export function initActivities(taskQueue: string) { hostSpecificTaskQueue = taskQueue; } function getHostSpecificTaskQueue(): string { return hostSpecificTaskQueue; } export async function download(source: string): Promise { const localFile = await downloadToLocalDisk(source); return { hostTaskQueue: getHostSpecificTaskQueue(), fileName: localFile, }; } export async function process(fileName: string): Promise { return await processLocalFile(fileName); } export async function upload(fileName: string, destination: string): Promise { await uploadFromLocalDisk(fileName, destination); } ``` The `download` method returns the host-specific Task Queue name alongside the file path. The `process` and `upload` methods operate on local files, which are guaranteed to exist because they run on the same host. ### Workflow implementation The Workflow uses the default Task Queue for the initial download and switches to the host-specific queue for subsequent Activities: **Python** ```python # workflows.py from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import download, process, upload @workflow.defn class FileProcessingWorkflow: @workflow.run async def run(self, source: str, destination: str) -> None: downloaded = await workflow.execute_activity( download, source, start_to_close_timeout=timedelta(seconds=20), ) processed = await workflow.execute_activity( process, downloaded.file_name, task_queue=downloaded.host_task_queue, schedule_to_start_timeout=timedelta(seconds=10), start_to_close_timeout=timedelta(seconds=20), ) await workflow.execute_activity( upload, args=[processed, destination], task_queue=downloaded.host_task_queue, schedule_to_start_timeout=timedelta(seconds=10), start_to_close_timeout=timedelta(seconds=20), ) ``` **Go** ```go // workflow.go func FileProcessingWorkflow(ctx workflow.Context, source string, destination string) error { defaultOptions := workflow.ActivityOptions{ StartToCloseTimeout: 20 * time.Second, } defaultCtx := workflow.WithActivityOptions(ctx, defaultOptions) var activities *StoreActivities var downloaded TaskQueueFileNamePair err := workflow.ExecuteActivity(defaultCtx, activities.Download, source).Get(ctx, &downloaded) if err != nil { return err } hostOptions := workflow.ActivityOptions{ TaskQueue: downloaded.HostTaskQueue, ScheduleToStartTimeout: 10 * time.Second, StartToCloseTimeout: 20 * time.Second, } hostCtx := workflow.WithActivityOptions(ctx, hostOptions) var processed string err = workflow.ExecuteActivity(hostCtx, activities.Process, downloaded.FileName).Get(ctx, &processed) if err != nil { return err } return workflow.ExecuteActivity(hostCtx, activities.Upload, processed, destination).Get(ctx, nil) } ``` **Java** ```java // FileProcessingWorkflowImpl.java public class FileProcessingWorkflowImpl implements FileProcessingWorkflow { private final StoreActivities defaultTaskQueueActivities; public FileProcessingWorkflowImpl() { ActivityOptions defaultOptions = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); this.defaultTaskQueueActivities = Workflow.newActivityStub(StoreActivities.class, defaultOptions); } @Override public void processFile(URL source, URL destination) { TaskQueueFileNamePair downloaded = defaultTaskQueueActivities.download(source); ActivityOptions hostOptions = ActivityOptions.newBuilder() .setTaskQueue(downloaded.getHostTaskQueue()) .setScheduleToStartTimeout(Duration.ofSeconds(10)) .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); StoreActivities hostSpecificActivities = Workflow.newActivityStub(StoreActivities.class, hostOptions); String processed = hostSpecificActivities.process(downloaded.getFileName()); hostSpecificActivities.upload(processed, destination); } } ``` **TypeScript** ```typescript // workflows.ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { download } = proxyActivities({ startToCloseTimeout: '20s', }); export async function fileProcessingWorkflow( source: string, destination: string, ): Promise { const downloaded = await download(source); const hostSpecificActivities = proxyActivities({ taskQueue: downloaded.hostTaskQueue, scheduleToStartTimeout: '10s', startToCloseTimeout: '20s', }); const processed = await hostSpecificActivities.process(downloaded.fileName); await hostSpecificActivities.upload(processed, destination); } ``` The Workflow creates two sets of Activity options: one for the default Task Queue and one for the host-specific queue returned by the download Activity. ### Worker setup Each Worker registers with both the default Task Queue and its own host-specific Task Queue: **Python** ```python # worker.py import asyncio import uuid import socket from temporalio.client import Client from temporalio.worker import Worker from workflows import FileProcessingWorkflow from activities import download, process, upload, host_specific_task_queue import activities as act_module async def main(): client = await Client.connect("localhost:7233") default_task_queue = "FileProcessing" host_task_queue = f"FileProcessing-{socket.gethostname()}-{uuid.uuid4()}" act_module.host_specific_task_queue = host_task_queue default_worker = Worker( client, task_queue=default_task_queue, workflows=[FileProcessingWorkflow], activities=[download, process, upload], ) host_worker = Worker( client, task_queue=host_task_queue, activities=[download, process, upload], ) await asyncio.gather(default_worker.run(), host_worker.run()) if __name__ == "__main__": asyncio.run(main()) ``` **Go** ```go // worker/main.go func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() defaultTaskQueue := "FileProcessing" hostTaskQueue := fmt.Sprintf("FileProcessing-%s-%s", getHostName(), uuid.New().String()) activities := &StoreActivities{HostSpecificTaskQueue: hostTaskQueue} defaultWorker := worker.New(c, defaultTaskQueue, worker.Options{}) defaultWorker.RegisterWorkflow(FileProcessingWorkflow) defaultWorker.RegisterActivity(activities) hostWorker := worker.New(c, hostTaskQueue, worker.Options{}) hostWorker.RegisterActivity(activities) err = defaultWorker.Start() if err != nil { log.Fatalln("Unable to start default worker", err) } err = hostWorker.Start() if err != nil { log.Fatalln("Unable to start host worker", err) } // Block until interrupted select {} } ``` **Java** ```java // FileProcessingWorker.java public class FileProcessingWorker { public static void main(String[] args) { WorkflowClient client = WorkflowClient.newInstance(service); String defaultTaskQueue = "FileProcessing"; String hostTaskQueue = "FileProcessing-" + getHostName(); WorkerFactory factory = WorkerFactory.newInstance(client); Worker defaultWorker = factory.newWorker(defaultTaskQueue); defaultWorker.registerWorkflowImplementationTypes( FileProcessingWorkflowImpl.class); defaultWorker.registerActivitiesImplementations( new StoreActivitiesImpl(hostTaskQueue)); Worker hostWorker = factory.newWorker(hostTaskQueue); hostWorker.registerActivitiesImplementations( new StoreActivitiesImpl(hostTaskQueue)); factory.start(); } } ``` **TypeScript** ```typescript // worker.ts import { Worker, NativeConnection } from '@temporalio/worker'; import * as activities from './activities'; import { v4 as uuid } from 'uuid'; import os from 'os'; async function run() { const defaultTaskQueue = 'FileProcessing'; const hostTaskQueue = `FileProcessing-${os.hostname()}-${uuid()}`; activities.initActivities(hostTaskQueue); const defaultWorker = await Worker.create({ workflowsPath: require.resolve('./workflows'), activities, taskQueue: defaultTaskQueue, }); const hostWorker = await Worker.create({ activities, taskQueue: hostTaskQueue, }); await Promise.all([defaultWorker.run(), hostWorker.run()]); } run().catch((err) => { console.error(err); process.exit(1); }); ``` The default Worker handles Workflows and initial Activities. The host-specific Worker handles only Activities that require Worker affinity. Both Workers receive the same Activity implementation, but only the host-specific Worker receives Activities routed to its queue. ## When to use The Worker-Specific Task Queues pattern is a good fit for file processing Workflows (download, process, upload on the same host), database connection pooling (maintain a connection across Activities), GPU-bound operations (route to Workers with specific hardware), session-based external API calls, and temporary resource management (cache, temp files, locks). It is not a good fit for stateless Activities that can run anywhere, Activities that use shared storage (S3, databases), high-availability requirements (host failure blocks the Workflow), or Workflows without local state dependencies. ## Benefits and trade-offs Activities access local files and state without network overhead. You do not need distributed file systems or state management. Data transfer between Workers is eliminated. The first Activity can run on any Worker; only subsequent ones are pinned. Task Queue routing is recorded in Workflow history, ensuring deterministic behavior. The trade-offs to consider are that if the specific Worker crashes, Activities cannot proceed until the ScheduleToStartTimeout expires. Host-specific queues may have uneven load distribution. You must manage multiple Task Queues per Worker. You must set ScheduleToStartTimeout to handle Worker unavailability. You need to handle cleanup if the Workflow fails mid-process. ## Comparison with alternatives | Approach | Locality | Complexity | Availability | | :--- | :--- | :--- | :--- | | Worker-Specific Queues | Guaranteed | Medium | Lower | | Shared Storage (S3) | None | Low | Higher | | Session Framework (Go) | Guaranteed | Low | Lower | ## Best practices - **Set ScheduleToStartTimeout.** Always configure this for host-specific queues to handle Worker failures. - **Implement cleanup.** Use try-finally or cancellation scopes to clean up local resources. - **Use unique queue names.** Use hostname, IP, or UUID to ensure unique Task Queue names. - **Monitor queue depth.** Alert on growing host-specific queue backlogs. - **Drain gracefully.** Drain host-specific queues before stopping Workers. - **Retry the entire sequence.** Wrap the sequence in retry logic to restart on a different host if needed. - **Limit concurrent Workflows.** Limit concurrent Workflows per Worker to prevent resource exhaustion. - **Add health checks.** Verify Worker health before accepting work on host-specific queues. ## Common pitfalls - **Missing ScheduleToStartTimeout on host-specific queues.** Without this timeout, if the target Worker is down, the Activity waits indefinitely. Always set `ScheduleToStartTimeout` so the Workflow can detect unavailability and retry on a different host. - **Not registering the Worker on both queues.** Each Worker must listen on both the default shared Task Queue (for Workflows and initial Activities) and its own host-specific queue. Forgetting the host-specific queue means routed Activities are never picked up. - **Assuming the host-specific Worker is always available.** The pinned Worker can crash or be restarted. Design the Workflow to retry the entire sequence on a different host when the `ScheduleToStartTimeout` expires. - **Leaking temporary files on failure.** If the Workflow fails after downloading but before uploading, temporary files remain on disk. Use cleanup logic (defer, try-finally, or cancellation scopes) to remove local resources. - **Using host-specific queues when shared storage suffices.** If all Workers can access the same storage (S3, NFS), Worker-specific routing adds unnecessary complexity and reduces availability. ## Related ### Patterns - **[Long-Running Activity](/design-patterns/long-running-activity)**: For tracking progress and handling cancellation when the colocated Activities run for minutes or hours. ### Guides - [Ensure Activity execution on the same Worker](/guides/worker-execution-affinity): A Python walkthrough of the shared-queue-discovers-unique-queue technique, with heartbeat and Schedule-to-Start timeouts for detecting a crashed Worker. - [Route specialized workloads](/guides/route-specialized-workloads): Routes Activities to dedicated GPU, high-memory, and CPU Worker pools by resource requirement — a related Task Queue routing technique, but by hardware capability rather than by Worker affinity. ### Sample code - [Java Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/fileprocessing) — Complete file processing implementation. - [TypeScript Sample](https://github.com/temporalio/samples-typescript/tree/main/worker-specific-task-queues) — Worker-specific Task Queues with file processing. - [Python Sample](https://github.com/temporalio/samples-python/tree/main/worker_specific_task_queues) — Worker-specific Task Queues with file processing. - [Go Sample](https://github.com/temporalio/samples-go/tree/main/worker-specific-task-queues) — Worker-specific Task Queues with file processing. --- # Workflow Messaging Patterns Source: https://docs.temporal.io/design-patterns/workflow-messaging-patterns > Pattern selection guide for sending data into running Workflows and receiving responses or triggering behavior changes. These patterns cover how external callers communicate with running Workflows — starting them on demand, sending data in, reading results back, and collecting streams of events. They build on Temporal's Signals and Updates. ## Patterns in this section - [Signal with Start](/design-patterns/signal-with-start): Starts a Workflow and delivers a Signal in a single atomic operation. If the Workflow already runs, it receives the Signal directly. - [Request-Response via Updates](/design-patterns/request-response-via-updates): Sends a request into a running Workflow and receives a validated result on the same call, using an Update handler. - [Event Accumulator](/design-patterns/event-accumulator): Collects a stream of incoming Signals into a buffer and processes them together as a batch, rather than one at a time. ## Choosing a pattern **You want to send a message without checking whether the Workflow is running**: use [Signal with Start](/design-patterns/signal-with-start). **You need a result back from the Workflow, with validation**: use [Request-Response via Updates](/design-patterns/request-response-via-updates). **You receive many events and want to process them in batches**: use the [Event Accumulator](/design-patterns/event-accumulator) to buffer and flush. ## Related sections - [Entity & Lifecycle Patterns](/design-patterns/entity-lifecycle-patterns) — long-lived Workflows that consume these messages over time - [Distributed Transaction Patterns](/design-patterns/distributed-transaction-patterns) — Early Return builds on the Update-with-Start mechanism --- # Develop durable applications with Temporal SDKs Source: https://docs.temporal.io/develop > Discover comprehensive Temporal SDK feature guides and API references. Enhance your Temporal Application development with .NET, Go, Java, PHP, Python, Ruby, Rust, and TypeScript. The Temporal SDK developer guides provide a comprehensive overview of the structures, primitives, and features used in [Temporal Application](/temporal#temporal-application) development. - **Go** (v1.48.0): [Developer guide](/develop/go) · [API reference](http://t.mp/go-api) - **Java** (v1.38.0): [Developer guide](/develop/java) · [API reference](http://t.mp/java-api) - **.NET** (v1.18.0): [Developer guide](/develop/dotnet) · [API reference](https://dotnet.temporal.io/) - **PHP** (v2.17.1): [Developer guide](/develop/php) · [API reference](https://php.temporal.io/namespaces/temporal.html) - **Python** (v1.32.0): [Developer guide](/develop/python) · [API reference](https://python.temporal.io) - **Ruby** (v1.7.0): [Developer guide](/develop/ruby) · [API reference](https://ruby.temporal.io/) - **Rust** (v1.0.0): [Developer guide](/develop/rust) · [API reference](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/) - **TypeScript** (v1.23.0): [Developer guide](/develop/typescript) · [API reference](https://typescript.temporal.io) --- # .Net SDK developer guide Source: https://docs.temporal.io/develop/dotnet ![.NET](/img/assets/banner-dotnet-temporal.png) ## Install and get started You can find detailed installation instructions for the .NET SDK in the [Quickstart](/develop/dotnet/set-up-your-local-dotnet). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Workflow basics](/develop/dotnet/workflows/basics) - [Activity basics](/develop/dotnet/activities/basics) - [Start an Activity execution](/develop/dotnet/activities/execution) - [Run Worker processes](/develop/dotnet/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/dotnet/workflows) - [Workflow basics](/develop/dotnet/workflows/basics) - [Child Workflows](/develop/dotnet/workflows/child-workflows) - [Continue-As-New](/develop/dotnet/workflows/continue-as-new) - [Cancellation](/develop/dotnet/workflows/cancellation) - [Timeouts](/develop/dotnet/workflows/timeouts) - [Message passing](/develop/dotnet/workflows/message-passing) - [Schedules](/develop/dotnet/workflows/schedules) - [Timers](/develop/dotnet/workflows/timers) - [Dynamic Workflow](/develop/dotnet/workflows/dynamic-workflow) - [Versioning](/develop/dotnet/workflows/versioning) ## [Activities](/develop/dotnet/activities) - [Activity basics](/develop/dotnet/activities/basics) - [Activity execution](/develop/dotnet/activities/execution) - [Standalone Activities](/develop/dotnet/activities/standalone-activities-quickstart) - [Timeouts](/develop/dotnet/activities/timeouts) - [Asynchronous Activity completion](/develop/dotnet/activities/asynchronous-activity) - [Dynamic Activity](/develop/dotnet/activities/dynamic-activity) - [Benign exceptions](/develop/dotnet/activities/benign-exceptions) ## [Workers](/develop/dotnet/workers) - [Worker processes](/develop/dotnet/workers/run-worker-process) - [Interceptors](/develop/dotnet/workers/interceptors) ## [Temporal Client](/develop/dotnet/client) - [Temporal Client](/develop/dotnet/client/temporal-client) ## [Temporal Nexus](/develop/dotnet/nexus) - [Quickstart](/develop/dotnet/nexus/quickstart) - [Feature guide](/develop/dotnet/nexus/feature-guide) ## [Platform](/develop/dotnet/platform) - [Observability](/develop/dotnet/platform/observability) - [Enriching the UI](/develop/dotnet/platform/enriching-ui) ## [Best practices](/develop/dotnet/best-practices) - [Error handling](/develop/dotnet/best-practices/error-handling) - [Testing](/develop/dotnet/best-practices/testing-suite) - [Debugging](/develop/dotnet/best-practices/debugging) - [Converters and encryption](/develop/dotnet/best-practices/data-handling) ## Temporal .NET technical resources - [.NET Quickstart](/develop/dotnet/set-up-your-local-dotnet) - [.NET API Documentation](https://dotnet.temporal.io/api/) - [.NET SDK Code Samples](https://github.com/temporalio/samples-dotnet) - [.NET SDK GitHub](https://github.com/temporalio/sdk-dotnet) - [Temporal 101 in .NET Free Course](https://learn.temporal.io/courses/temporal_101/dotnet/) Get Connected with the Temporal .NET Community - [Temporal .NET Community Slack](https://temporalio.slack.com/archives/C012SHMPDDZ) - [.NET SDK Forum](https://community.temporal.io/tag/dotnet-sdk) --- # Activities - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities > This section explains how to implement Activities with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Activities - [Activity basics](/develop/dotnet/activities/basics) - [Activity execution](/develop/dotnet/activities/execution) - [Standalone Activities Quickstart](/develop/dotnet/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/dotnet/activities/standalone-activities) - [Timeouts](/develop/dotnet/activities/timeouts) - [Asynchronous Activity completion](/develop/dotnet/activities/asynchronous-activity) - [Dynamic Activity](/develop/dotnet/activities/dynamic-activity) - [Benign exceptions](/develop/dotnet/activities/benign-exceptions) --- # Asynchronous Activity completion - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/asynchronous-activity > Asynchronously complete an Activity in Temporal. Follow simple steps to allow an Activity Function to return without the Activity Execution completing. This page describes how to asynchronously complete an Activity. [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To mark an Activity as completing asynchronously, do the following inside the Activity. ```csharp // Capture token for later completion capturedToken = ActivityExecutionContext.Current.Info.TaskToken; // Throw special exception that says an activity will be completed somewhere else throw new CompleteAsyncException(); ``` To update an Activity outside the Activity, use the [GetAsyncActivityHandle()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_GetAsyncActivityHandle_System_Byte___) method to get the handle of the Activity. ```csharp var handle = myClient.GetAsyncActivityHandle(capturedToken); ``` Then, on that handle, you can call the results of the Activity, `HeartbeatAsync`, `CompleteAsync`, `FailAsync`, or `ReportCancellationAsync` method to update the Activity. ```csharp await handle.CompleteAsync("Completion value."); ``` --- # Activity basics - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/basics > This section explains Activity basics with the .NET SDK ## Develop an Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/dotnet/activities/standalone-activities-quickstart). You can develop an Activity Definition by using the `[Activity]` attribute from the `Temporalio.Activities` namespace on the method. To register a method as an Activity with a custom name, use an attribute parameter, for example `[Activity("your-activity")]`. Otherwise, the activity name is the unqualified method name (sans an "Async" suffix if the method is async). Activities can be asynchronous or synchronous. ```csharp using Temporalio.Activities; public class MyActivities { // Activities can be async and/or static too. We just demonstrate instance methods since many // use them that way. [Activity] public string MyActivity(MyActivityParams input) => $"{input.Greeting}, {input.Name}!"; } ``` There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a method signature. Activity parameters are the method parameters of the method with the `[Activity]` attribute. These can be any data type Temporal can convert, including records. Technically this can be multiple parameters, but Temporal strongly encourages a single parameter containing all input fields. --- # Benign exceptions - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. **How to mark an Activity error as benign using the Temporal .NET SDK** When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the `category` parameter to `ApplicationErrorCategory.Benign` when throwing an [`ApplicationFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.ApplicationFailureException.html). Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```csharp using Temporalio.Activities; using Temporalio.Api.Enums.V1; using Temporalio.Exceptions; public class MyActivities { [Activity] public async Task MyActivityAsync() { try { return await CallExternalServiceAsync(); } catch (Exception e) { // Mark this error as benign since it's expected throw new ApplicationFailureException( "Service is down", inner: e, category: ApplicationErrorCategory.Benign); } } } ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Dynamic Activity - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/dynamic-activity > This section explains Dynamic Activities with the .NET SDK ## Set a Dynamic Activity **How to set a Dynamic Activity using the Temporal .NET SDK** A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered. An Activity can be made dynamic by setting `Dynamic` as `true` on the `[Activity]` attribute. You must register the Activity with the Worker before it can be invoked. Only one Dynamic Activity can be present on a Worker. The Activity Definition must then accept a single argument of type `Temporalio.Converters.IRawValue[]`. The [PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Activities.ActivityExecutionContext.html#Temporalio_Activities_ActivityExecutionContext_PayloadConverter) property on the `ActivityExecutionContext` is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` namespace. ```csharp public class MyActivities { [Activity(Dynamic = true)] public string DynamicActivity(IRawValue[] args) { var input = ActivityExecutionContext.Current.PayloadConverter.ToValue(args.Single()); return $"{input.Greeting}, {input.Name}!"; } } ``` --- # Activity execution - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/execution > Shows how to perform Activity execution with the .NET SDK ## Start Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History. A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity implementation code should be _idempotent_. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. To spawn an Activity Execution, use the `ExecuteActivityAsync` operation from within your Workflow Definition. ```csharp using Temporalio.Workflows; [Workflow] public class MyWorkflow { public async Task RunAsync(string name) { var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } ``` Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. ### Get Activity Execution results The Activity result is returned in the Task from the `ExecuteActivityAsync` call. --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/dotnet/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal .NET SDK. > **Public Preview** Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/dotnet/activities/standalone-activities-quickstart). > This page covers the following: - [Start a Standalone Activity without waiting for the result](#start-activity) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [Wait for the result of a Standalone Activity](#get-activity-result) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the [StandaloneActivity](https://github.com/temporalio/samples-dotnet/tree/main/src/StandaloneActivity) sample project. > ## Start a Standalone Activity without waiting for the result Use [`client.StartActivityAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html) to start a Standalone Activity and get a handle without waiting for the result: [src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs) ```csharp using Temporalio.Client; using Temporalio.Common.EnvConfig; using TemporalioSamples.StandaloneActivity; var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); connectOptions.TargetHost ??= "localhost:7233"; var client = await TemporalClient.ConnectAsync(connectOptions); var handle = await client.StartActivityAsync( () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")), new("standalone-activity-id", "standalone-activity-sample") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); Console.WriteLine($"Started activity: {handle.Id}"); // Wait for the result later var result = await handle.GetResultAsync(); Console.WriteLine($"Activity result: {result}"); ``` With the Temporal Server and Worker running, open a new terminal in the `samples-dotnet` directory and run: ``` dotnet run --project src/StandaloneActivity start-activity ``` Or use the Temporal CLI: ```bash temporal activity start \ --type ComposeGreeting \ --activity-id standalone-activity-id \ --task-queue standalone-activity-sample \ --schedule-to-close-timeout 10s \ --input '{"Greeting": "Hello", "Name": "World"}' ``` ## Get a handle to an existing Standalone Activity Use `client.GetActivityHandle()` to create a handle to a previously started Standalone Activity: ```csharp // Without a known result type var handle = client.GetActivityHandle("my-activity-id", runId: "the-run-id"); // With a known result type var typedHandle = client.GetActivityHandle("my-activity-id", runId: "the-run-id"); ``` You can use the handle to wait for the result, describe, cancel, or terminate the Activity. ## Wait for the result of a Standalone Activity Under the hood, calling `client.ExecuteActivityAsync()` is the same as calling `client.StartActivityAsync()` to durably enqueue the Standalone Activity, and then calling `await handle.GetResultAsync()` to wait for the Activity to be executed and return the result: ```csharp var result = await handle.GetResultAsync(); ``` Or use the Temporal CLI to wait for a result by Activity ID: ```bash temporal activity result --activity-id my-standalone-activity-id ``` ## List Standalone Activities Use [`client.ListActivitiesAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html) to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an `IAsyncEnumerable` that yields `ActivityExecution` entries. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. [src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs) ```csharp using Temporalio.Client; using Temporalio.Common.EnvConfig; var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); connectOptions.TargetHost ??= "localhost:7233"; var client = await TemporalClient.ConnectAsync(connectOptions); await foreach (var info in client.ListActivitiesAsync( "TaskQueue = 'standalone-activity-sample'")) { Console.WriteLine( $"ActivityID: {info.ActivityId}, Type: {info.ActivityType}, Status: {info.Status}"); } ``` Run it: ``` dotnet run --project src/StandaloneActivity list-activities ``` Or use the Temporal CLI: ```bash temporal activity list ``` The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow Visibility](/visibility). For example, `"ActivityType = 'ComposeGreeting' AND Status = 'Running'"`. ## Count Standalone Activities Use [`client.CountActivitiesAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html) to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) - not the number of queued tasks. It works the same way as counting Workflow Executions. [src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs) ```csharp using Temporalio.Client; using Temporalio.Common.EnvConfig; var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); connectOptions.TargetHost ??= "localhost:7233"; var client = await TemporalClient.ConnectAsync(connectOptions); var resp = await client.CountActivitiesAsync( "TaskQueue = 'standalone-activity-sample'"); Console.WriteLine($"Total activities: {resp.Count}"); ``` Run it: ``` dotnet run --project src/StandaloneActivity count-activities ``` Or use the Temporal CLI: ```bash temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The code samples on this page use `ClientEnvConfig.LoadClientConnectOptions()`, so the same code works against Temporal Cloud - just configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/dotnet/activities/standalone-activities-quickstart). --- # Standalone Activities .NET Quickstart Source: https://docs.temporal.io/develop/dotnet/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal .NET SDK without writing a Workflow. # Quickstart Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/dotnet/activities/basics#develop-activity). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **📝 Note:** > > This documentation uses source code from the [StandaloneActivity](https://github.com/temporalio/samples-dotnet/tree/main/src/StandaloneActivity) sample project. > ## Get started with Standalone Activities Prerequisites: - **[.NET](https://dotnet.microsoft.com/download)** 8.0+ - **Temporal .NET SDK** (v1.12.0 or higher). See the [.NET Quickstart](/develop/dotnet/set-up-your-local-dotnet) for install instructions. - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server will now be available for client connections on `localhost:7233`, and the Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-dotnet](https://github.com/temporalio/samples-dotnet) repository to follow along: ``` git clone https://github.com/temporalio/samples-dotnet.git cd samples-dotnet ``` The sample project is structured as follows: ``` src/StandaloneActivity/ ├── MyActivities.cs ├── Program.cs ├── README.md └── TemporalioSamples.StandaloneActivity.csproj ``` ## Define your Activity An Activity in the Temporal .NET SDK is a method decorated with the `[Activity]` attribute. The way you write a Standalone Activity is identical to how you write an Activity orchestrated by a Workflow. In fact, the same Activity can be executed both as a Standalone Activity and as a Workflow Activity. [src/StandaloneActivity/MyActivities.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/MyActivities.cs) ```csharp namespace TemporalioSamples.StandaloneActivity; using Temporalio.Activities; public static class MyActivities { [Activity] public static Task ComposeGreetingAsync(ComposeGreetingInput input) => Task.FromResult($"{input.Greeting}, {input.Name}!"); } public record ComposeGreetingInput(string Greeting, string Name); ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities — you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to develop a Worker](/develop/dotnet/workers/run-worker-process) for more details on Worker setup and configuration options. [src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs) Open a new terminal, navigate to the `samples-dotnet` directory, and run the Worker. Leave this terminal running - the Worker needs to stay up to process activities. ```csharp using Microsoft.Extensions.Logging; using Temporalio.Client; using Temporalio.Common.EnvConfig; using Temporalio.Worker; using TemporalioSamples.StandaloneActivity; var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); connectOptions.TargetHost ??= "localhost:7233"; connectOptions.LoggerFactory = LoggerFactory.Create(builder => builder. AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] "). SetMinimumLevel(LogLevel.Information)); var client = await TemporalClient.ConnectAsync(connectOptions); const string taskQueue = "standalone-activity-sample"; using var tokenSource = new CancellationTokenSource(); Console.CancelKeyPress += (_, eventArgs) => { tokenSource.Cancel(); eventArgs.Cancel = true; }; using var worker = new TemporalWorker( client, new TemporalWorkerOptions(taskQueue). AddActivity(MyActivities.ComposeGreetingAsync)); await worker.ExecuteAsync(tokenSource.Token); ``` ```bash dotnet run --project src/StandaloneActivity worker ``` ## Execute a Standalone Activity Use [`client.ExecuteActivityAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClientExtensions.html) to execute a Standalone Activity and wait for the result. Call this from your application code, not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to be executed on your Worker, and then returns the result. [src/StandaloneActivity/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/StandaloneActivity/Program.cs) You can pass the Activity as either a lambda expression or a string Activity type name. `StartActivityOptions` requires `Id`, `TaskQueue`, and at least one of `ScheduleToCloseTimeout` or `StartToCloseTimeout`. See [`StartActivityOptions`](https://dotnet.temporal.io/api/Temporalio.Client.StartActivityOptions.html) in the API reference for the full set of options. To run it: 1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above). 2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above). 3. Open a new terminal, navigate to the `samples-dotnet` directory, and run `dotnet run --project src/StandaloneActivity execute-activity`. Or use the Temporal CLI. ```csharp using Temporalio.Client; using Temporalio.Common.EnvConfig; using TemporalioSamples.StandaloneActivity; var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); connectOptions.TargetHost ??= "localhost:7233"; var client = await TemporalClient.ConnectAsync(connectOptions); var result = await client.ExecuteActivityAsync( () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")), new("standalone-activity-id", "standalone-activity-sample") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); Console.WriteLine($"Activity result: {result}"); ``` ```csharp // Using a lambda expression (type-safe) var result = await client.ExecuteActivityAsync( () => MyActivities.ComposeGreetingAsync(new ComposeGreetingInput("Hello", "World")), new("standalone-activity-id", "standalone-activity-sample") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); // Using a string type name var result = await client.ExecuteActivityAsync( "ComposeGreeting", new object?[] { new ComposeGreetingInput("Hello", "World") }, new("standalone-activity-id", "standalone-activity-sample") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); ``` ```bash dotnet run --project src/StandaloneActivity execute-activity ``` ```bash temporal activity execute \\ --type ComposeGreeting \\ --activity-id standalone-activity-id \\ --task-queue standalone-activity-sample \\ --schedule-to-close-timeout 10s \\ --input '{"Greeting": "Hello", "Name": "World"}' ``` ## Run with Temporal Cloud All code samples on this page use [`ClientEnvConfig.LoadClientConnectOptions()`](https://dotnet.temporal.io/api/Temporalio.Common.EnvConfig.ClientEnvConfig.html) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/dotnet/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/dotnet/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/dotnet/activities/basics)**: How to write and register Activities with the .NET SDK. --- # Activity Timeouts - .NET SDK Source: https://docs.temporal.io/develop/dotnet/activities/timeouts > Optimize Workflow Execution with Temporal .NET SDK - Set Activity Timeouts and Retry Policies efficiently. ## Activity Timeouts Each Activity Timeout controls the maximum duration of a different aspect of an Activity Execution. The following Timeouts are available in the Activity Options. - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. These values can be set in the `ActivityOptions` when calling `ExecuteActivityAsync`. Available timeouts are: - ScheduleToCloseTimeout - StartToCloseTimeout - ScheduleToStartTimeout ```csharp return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); ``` ### Set an Activity Retry Policy A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To create an Activity Retry Policy in .NET, set the `RetryPolicy` on the `ActivityOptions` when calling `ExecuteActivityAsync`. ```csharp return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5), RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) }, }); ``` ### Override the Retry interval with `nextRetryDelay` When you throw an [Application Failure](/references/failures#application-failure) and assign the `nextRetryDelay` field, its value replaces and overrides the Retry interval defined in the active Retry Policy. For example, you might scale the next Retry delay interval based on the current number of attempts. Here's how you'd do that in an Activity. In the following sample, the `attempt` count is retrieved from the Activity Execution context and used to set the number of seconds for the next Retry delay: ```csharp var attempt = ActivityExecutionContext.Current.Info.Attempt; throw new ApplicationFailureException( $"Something bad happened on attempt {attempt}", errorType: "my_failure_type", nextRetryDelay: TimeSpan.FromSeconds(3 * attempt)); ``` ## Heartbeat an Activity An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. Heartbeats can contain a `Details` field describing the Activity's current progress. If an Activity gets retried, the Activity can access the `Details` from the last Heartbeat that was sent to the Temporal Service. To Heartbeat an Activity Execution in .NET, use the [`Heartbeat()`](https://dotnet.temporal.io/api/Temporalio.Activities.ActivityExecutionContext.html#Temporalio_Activities_ActivityExecutionContext_Heartbeat_System_Object___) method on the `ActivityExecutionContext`. ```csharp [Activity] public async Task MyActivityAsync() { while (true) { // Send heartbeat ActivityExecutionContext.Current.Heartbeat(); // Do some work, passing the cancellation token await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken); } } ``` In addition to obtaining cancellation information, Heartbeats also support detail data that persists on the server for retrieval during Activity retry. If an Activity calls `Heartbeat(123, 456)` and then fails and is retried, `HeartbeatDetails` on the `ActivityInfo` returns an collection containing `123` and `456` on the next Run. ### Set a Heartbeat Timeout A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). `HeartbeatTimeout` is a property on `ActivityOptions` for `ExecuteActivityAsync` used to set the maximum time between Activity Heartbeats. ```csharp await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5), HeartbeatTimeout = TimeSpan.FromSeconds(30), }); ``` --- # Best practices - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices > This section explains how to implement best practices with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Best practices - [Error handling](/develop/dotnet/best-practices/error-handling) - [Testing](/develop/dotnet/best-practices/testing-suite) - [Debugging](/develop/dotnet/best-practices/debugging) - [Converters and encryption](/develop/dotnet/best-practices/data-handling) --- # Data handling - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/dotnet/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/dotnet/best-practices/data-handling/data-encryption) | | ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | | **Default** | JSON serialization | None (passthrough) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the .NET SDK. ## Payload conversion Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. ### Conversion sequence The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this: ![Temporal converter architecture](/img/info/converter-architecture.png) ## Custom Payload Converter Data converters are used to convert raw Temporal payloads to/from actual .NET types. A custom data converter can be set via the `DataConverter` option when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. Payload converters convert .NET values to/from serialized bytes. Payload codecs convert bytes to bytes (for example, for compression or encryption). Failure converters convert exceptions to/from serialized failures. Data converters are in the `Temporalio.Converters` namespace. The default data converter uses a default payload converter, which supports the following types: - `null` - `byte[]` - `Google.Protobuf.IMessage` instances - Anything that `System.Text.Json` supports - `IRawValue` as unconverted raw payloads Custom converters can be created for all uses. For example, to create client with a data converter that converts all C# property names to camel case, you would: ```csharp using System.Text.Json; using Temporalio.Client; using Temporalio.Converters; public class CamelCasePayloadConverter : DefaultPayloadConverter { public CamelCasePayloadConverter() : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }) { } } var client = await TemporalClient.ConnectAsync(new() { TargetHost = "localhost:7233", Namespace = "my-namespace", DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() }, }); ``` --- # Payload encryption - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the .NET SDK. Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. ## Custom Payload Codec Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`. The `IPayloadCodec` needs to implement `EncodeAsync()` and `DecodeAsync()` methods. These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field. Do not mutate the existing payloads. Here is an example of an encryption codec that just uses base64 in each direction: ```csharp public class EncryptionCodec : IPayloadCodec { public Task> EncodeAsync(IReadOnlyCollection payloads) => Task.FromResult>(payloads.Select(p => { return new Payload() { // Set our specific encoding. We may also want to add a key ID in here for use by // the decode side Metadata = { ["encoding"] = "binary/my-payload-encoding" }, Data = ByteString.CopyFrom(Encrypt(p.ToByteArray())), }; }).ToList()); public Task> DecodeAsync(IReadOnlyCollection payloads) => Task.FromResult>(payloads.Select(p => { // Ignore if it doesn't have our expected encoding if (p.Metadata.GetValueOrDefault("encoding") != "binary/my-payload-encoding") { return p; } // Decrypt return Payload.Parser.ParseFrom(Decrypt(p.Data.ToByteArray())); }).ToList()); private byte[] Encrypt(byte[] data) => Encoding.ASCII.GetBytes(Convert.ToBase64String(data)); private byte[] Decrypt(byte[] data) => Convert.FromBase64String(Encoding.ASCII.GetString(data)); } ``` ### Set Data Converter to use custom Payload Codec When creating a client, the default `DataConverter` can be updated with the payload codec like so: ```csharp var myClient = await TemporalClient.ConnectAsync(new("localhost:7233") { DataConverter = DataConverter.Default with { PayloadCodec = new EncryptionCodec() }, }); ``` - Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. - Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. For reference, see the [Encryption](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) sample. ## Using a Codec Server A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. --- # Debugging - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/debugging > Debug Workflows in development and production environments using Temporal .NET SDK. Use logging, debugger, Web UI, CLI, replay, tracing, and more for efficient troubleshooting. ## Debugging This page shows how to do the following: - [Debug in a development environment](#debug-in-a-development-environment) - [Debug in a production environment](#debug-in-a-development-production) ### Debug in a development environment In developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow. In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). The Web UI provides insight into your Workflows, making it easier to identify issues and monitor the state of your Workflows in real time. ### Debug in a production environment **How to debug in a production environment using the Temporal .NET SDK** You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/dotnet/best-practices/testing-suite#replay) - [Tracing](/develop/dotnet/platform/observability#tracing) - [Logging](/develop/dotnet/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). For more information, see [Observability ▶️ Metrics](/develop/dotnet/platform/observability#metrics) for setting up SDK metrics. Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). --- # Error handling - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/error-handling > Handle errors with Temporal .Net SDK ## Raise and Handle Exceptions In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language. Temporal uses several different error classes internally — for example, [`CancelledFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.CanceledFailureException.html) in the .NET SDK, to handle a Workflow cancellation. You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic. The one Temporal error class that you will typically raise deliberately is [`ApplicationFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.ApplicationFailureException.html). In fact, *any* other exceptions that are raised from your C# code in a Temporal Activity will be converted to an `ApplicationFailureException` internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries. In other words, these two code samples do the same thing: ```csharp [Serializable] public class InvalidDepartmentException : Exception { public InvalidDepartmentException() : base() { } public InvalidDepartmentException(string message) : base(message) { } public InvalidDepartmentException(string message, Exception inner) : base(message, inner) { } } [Activity] public Task SendBillAsync(Bill bill) { throw new InvalidDepartmentException("Invalid department"); } ``` ```csharp [Activity] public Task SendBillAsync(Bill bill) { throw new ApplicationFailureException("Invalid department", errorType: "InvalidDepartmentException"); } ``` Depending on your implementation, you may decide to use either method. One reason to use the Temporal `ApplicationFailureException` class is because it allows you to set an additional `non_retryable` parameter. This way, you can decide whether an error should not be retried automatically by Temporal. This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse: ```csharp [Activity] public Task SendBillAsync(Bill bill) { throw new ApplicationFailureException("Invalid department", nonRetryable: true); } ``` You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/dotnet/activities/timeouts#activity-retries). ## Failing Workflows One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately. The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold. Activities will not actually *return* a failure to your Workflow until this condition or another non-retryable condition is met. At this point, you can decide how to handle an error returned by your Activity the way you would in any other program. For example, you could implement a [Saga Pattern](/design-patterns/saga-pattern) — see this [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Saga) — that uses `try`/`catch` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure. **You will only fail a Workflow by manually raising an `ApplicationFailureException` from the Workflow code.** You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue: ```csharp try { await Workflow.ExecuteActivityAsync( (Activities act) => act.ValidateCreditCardAsync(order.Customer.CreditCardNumber), options); } catch (ActivityFailureException err) { logger.LogError("Unable to process credit card: {Message}", err.Message); throw new ApplicationFailureException(message: "Invalid credit card number error"); } ``` This works differently in a Workflow than raising exceptions from Activities. In an Activity, any C# exceptions or custom exceptions are converted to a Temporal `ApplicationError`. In a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationError` will only fail that particular [Workflow Task](/tasks#workflow-task-execution) and be retried. This includes any typical C# `RuntimeError`s that are raised automatically. These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly. --- # Testing - .NET SDK Source: https://docs.temporal.io/develop/dotnet/best-practices/testing-suite > The .NET test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution. The .NET test-suite feature guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks **Compatible testing frameworks** The .NET SDK is compatible with any testing framework and does not have a specific recommendation. Most .NET SDK samples use [xUnit](https://xunit.net/). ## Testing Workflows **How to test Workflow Definitions using the Temporal .NET SDK** Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code. Using the time-skipping Workflow test environment can help there. ### Testing Workflows with standard server A non-time-skipping `Temporalio.Testing.WorkflowEnvironment` can be started via `StartLocalAsync` which supports all standard Temporal features. It is actually the real Temporal dev server packaged in the Temporal CLI, lazily downloaded on first use, and run as a sub-process in the background. Assuming tests properly use separate Task Queues, the same server can and should be reused across tests. Here's a simple example of a Workflow: ```csharp [Workflow] public class SayHelloWorkflow { [WorkflowRun] public async Task RunAsync(string name) { return $"Hello, {name}!"; } } ``` Here's how a test of that Workflow may appear in xUnit: ```csharp using Temporalio.Testing; using Temporalio.Worker; [Fact] public async Task SayHelloWorkflow_SimpleRun_Succeeds() { // Start local dev server await using var env = await WorkflowEnvironment.StartLocalAsync(); // Create a worker using var worker = new TemporalWorker( env.Client, new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}"). AddWorkflow()); // Run the worker only for the life of the code within await worker.ExecuteAsync(async () => { // Execute the workflow and confirm the result var result = await env.Client.ExecuteWorkflowAsync( (SayHelloWorkflow wf) => wf.RunAsync("Temporal"), new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!)); Assert.Equal("Hello, Temporal!", result); }); } ``` While this is just a demonstration, a local server is often used as a fixture across many tests. ### Testing Workflows with time skipping Sometimes there is a need to test Workflows that run a long time or to test that timeouts occur. A time-skipping `Temporalio.Testing.WorkflowEnvironment` can be started via `StartTimeSkippingAsync` which is a reimplementation of the Temporal server with special time skipping capabilities. Like `StartLocalAsync`, this also lazily downloads the process to run when first called. Note, unlike `StartLocalAsync`, this class is not thread safe nor safe for use with independent tests. It can be technically be reused, but only for one test at a time because time skipping is locked/unlocked at the environment level. Developers are encouraged to run it per test needed. #### Automatic time skipping Here's a simple example of a Workflow that waits a day: ```csharp [Workflow] public class WaitADayWorkflow { [WorkflowRun] public async Task RunAsync() { await Workflow.DelayAsync(TimeSpan.FromDays(1)); return "all done"; } } ``` A regular integration test of this Workflow on a normal server would be way too slow. However, the time-skipping server automatically skips to the next event when we wait on the result. Here's a test for that Workflow in xUnit: ```csharp using Temporalio.Testing; using Temporalio.Worker; [Fact] public async Task WaitADayWorkflow_SimpleRun_Succeeds() { // Start time-skipping test server await using var env = await WorkflowEnvironment.StartTimeSkippingAsync(); // Create a worker using var worker = new TemporalWorker( env.Client, new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}"). AddWorkflow()); // Run the worker only for the life of the code within await worker.ExecuteAsync(async () => { // Execute the workflow and confirm the result var result = await env.Client.ExecuteWorkflowAsync( (WaitADayWorkflow wf) => wf.RunAsync(), new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!)); Assert.Equal("all done", result); }); } ``` This test will run almost instantly. This is because by calling `ExecuteWorkflowAsync` on our client, we are actually calling `StartWorkflowAsync` + `GetResultAsync`, and `GetResultAsync` automatically skips time as much as it can (basically until the end of the Workflow or until an Activity is run). To disable automatic time-skipping while waiting for a workflow result, run code as a lambda passed to `env.WithAutoTimeSkippingDisabled` or `env.WithAutoTimeSkippingDisabledAsync`. #### Manual time skipping Until a Workflow is waited on, all time skipping in the time-skipping environment is done manually via `WorkflowEnvironment.DelayAsync`. Here's a Workflow that waits for a Signal or times out: ```csharp [Workflow] public class SignalWorkflow { private bool signalReceived = false; [WorkflowRun] public async Task RunAsync() { // Wait for signal or timeout in 45 seconds if (Workflow.WaitConditionAsync(() => signalReceived, TimeSpan.FromSeconds(45))) { return "got signal"; } return "got timeout"; } [WorkflowSignal] public async Task SomeSignalAsync() => signalReceived = true; } ``` To test a normal Signal in xUnit, you might: ```csharp using Temporalio.Testing; using Temporalio.Worker; [Fact] public async Task SignalWorkflow_SendSignal_HasExpectedResult() { await using var env = await WorkflowEnvironment.StartTimeSkippingAsync(); using var worker = new TemporalWorker( env.Client, new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}"). AddWorkflow()); await worker.ExecuteAsync(async () => { var handle = await env.Client.StartWorkflowAsync( (SignalWorkflow wf) => wf.RunAsync(), new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!)); await handle.SignalAsync(wf => wf.SomeSignalAsync()); Assert.Equal("got signal", await handle.GetResultAsync()); }); } ``` But how would you test the timeout part? Like so: ```csharp using Temporalio.Testing; using Temporalio.Worker; [Fact] public async Task SignalWorkflow_SignalTimeout_HasExpectedResult() { await using var env = await WorkflowEnvironment.StartTimeSkippingAsync(); using var worker = new TemporalWorker( env.Client, new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}"). AddWorkflow()); await worker.ExecuteAsync(async () => { var handle = await env.Client.StartWorkflowAsync( (SignalWorkflow wf) => wf.RunAsync(), new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!)); await env.DelayAsync(TimeSpan.FromSeconds(50)); Assert.Equal("got timeout", await handle.GetResultAsync()); }); } ``` ### Mocking Activities When testing Workflows, often you don't want to actually run the Activities. Activities are just methods with the `[Activity]` attribute. Simply write different/empty/fake/asserting ones and pass those to the Worker to have different activities called during the test. ## Testing Activities **How to test Activity Definitions using the Temporal .NET SDK** Unit testing an Activity or any code that could run in an Activity is done via the `Temporalio.Testing.ActivityEnvironment` class. Simply instantiate the class, and any code inside `RunAsync` will be invoked inside the activity context. The following important members are available on the environment to affect the activity context: - `Info` - Activity info, defaulted to a basic set of values. - `Logger` - Activity logger, defaulted to a null logger. - `Cancel(CancelReason)` - Helper to set the reason and cancel the source. - `CancelReason` - Cancel reason. - `CancellationTokenSource` - Token source for issuing cancellation. - `Heartbeater` - Callback invoked each heartbeat. - `WorkerShutdownTokenSource` - Token source for issuing Worker shutdown. - `PayloadConverter` - Defaulted to default payload converter. ## Replay test **How to do a Replay test using the Temporal .NET SDK** Given a Workflow's history, it can be replayed locally to check for things like non-determinism errors. For example, assuming the `history` parameter below is given a JSON string of history exported from the CLI or web UI, the following method will replay it: ```csharp using Temporalio; using Temporalio.Worker; public static async Task ReplayFromJsonAsync(string historyJson) { var replayer = new WorkflowReplayer( new WorkflowReplayerOptions().AddWorkflow()); await replayer.ReplayWorkflowAsync(WorkflowHistory.FromJson("my-workflow-id", historyJson)); } ``` If there is a non-determinism, this will throw an exception. Event history can be loaded from more than just JSON. It can be fetched individually from a Workflow handle, or even in a list. For example, the following code will check that all Workflow histories for a certain Workflow type (that is, workflow class) are safe with the current Workflow code. ```csharp using Temporalio; using Temporalio.Client; using Temporalio.Worker; public static async Task CheckPastHistoriesAsync(ITemporalClient client) { var replayer = new WorkflowReplayer( new WorkflowReplayerOptions().AddWorkflow()); var listIter = client.ListWorkflowHistoriesAsync("WorkflowType = 'SayHello'"); await foreach (var result in replayer.ReplayWorkflowsAsync(listIter)) { if (result.ReplayFailure != null) { ExceptionDispatchInfo.Throw(result.ReplayFailure); } } } ``` --- # Client - .NET SDK Source: https://docs.temporal.io/develop/dotnet/client > This section explains how to implement the Temporal Client with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Temporal Client - [Temporal Client](/develop/dotnet/client/temporal-client) --- # Temporal Client - .NET SDK Source: https://docs.temporal.io/develop/dotnet/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service. Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and Queries to Workflow Executions, getting Workflow results, and more. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the .NET SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow) - [Get Workflow results](#get-workflow-results) A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service. ## Connect to development Temporal Service Use [`TemporalClient.ConnectAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_ConnectAsync_Temporalio_Client_TemporalClientConnectOptions_) to create a client. Connection options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide these options directly in code, or load them from **environment variables** and/or a **TOML configuration file** using the `Temporalio.Client.EnvConfig` helpers. We recommend environment variables or a configuration file for secure, repeatable configuration. When you’re running a Temporal Service locally (such as with the [Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to [Environment Configuration](/references/client-environment-configuration) for more details about configuration files and profiles. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml title="config.toml" # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS auto-enables when TLS config or an API key is present # disabled = false client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a profile from the configuration file as follows. In this example, you load the `default` profile for local development: ```csharp title="LoadFromFile.cs" {27-30} using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading the default environment configuration profile /// from a TOML file. /// public static class LoadFromFile { public static async Task RunAsync() { Console.WriteLine("--- Loading default profile from config.toml ---"); try { // For this sample to be self-contained, we explicitly provide the path to // the config.toml file included in this directory. // By default though, the config.toml file will be loaded from // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml"); // LoadClientConnectOptions is a helper that loads a profile and prepares // the config for TemporalClient.ConnectAsync. By default, it loads the // "default" profile. var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions { ConfigSource = DataSource.FromPath(configFile), }); Console.WriteLine($"Loaded 'default' profile from {configFile}."); Console.WriteLine($" Address: {connectOptions.TargetHost}"); Console.WriteLine($" Namespace: {connectOptions.Namespace}"); if (connectOptions.RpcMetadata?.Count > 0) { Console.WriteLine($" gRPC Metadata: {string.Join(", ", connectOptions.RpcMetadata.Select(kv => $"{kv.Key}={kv.Value}"))}"); } Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); // Test the connection by checking the service var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new()); Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **Environment Variables** Use the `EnvConfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. Set the following environment variables before running your .NET application. Replace the placeholder values with your actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the `default` Namespace. You may omit these variables entirely since they're the defaults. ```bash export TEMPORAL_NAMESPACE="default" export TEMPORAL_ADDRESS="localhost:7233" ``` After setting the environment variables, use the following code to create the Temporal Client: ```csharp using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading the default environment configuration profile /// from a TOML file. /// public static class LoadFromFile { public static async Task RunAsync() { try { var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. ```csharp using System; using System.Threading.Tasks; using Temporalio.Client; namespace TemporalioSamples.Manual { public static class ManualConnect { public static async Task RunAsync() { Console.WriteLine("--- Connecting manually to Temporal ---"); var client = await TemporalClient.ConnectAsync(new TemporalClientConnectOptions { TargetHost = "localhost:7233", Namespace = "default", }); Console.WriteLine("✅ Connected to local Temporal service!"); } } } ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace and Account ID_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the `ClientEnvConfig.LoadClientConnectOptions` method to create a Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. ```csharp title="LoadProfile.cs" {25. 41} using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading a named environment configuration profile and /// programmatically overriding its values. /// public static class LoadProfile { public static async Task RunAsync() { Console.WriteLine("--- Loading 'staging' profile with programmatic overrides ---"); try { var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml"); var profileName = "staging"; Console.WriteLine("The 'staging' profile in config.toml has an incorrect address (localhost:9999)."); Console.WriteLine("We'll programmatically override it to the correct address."); // Load the 'staging' profile var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions { Profile = profileName, ConfigSource = DataSource.FromPath(configFile), }); // Override the target host to the correct address. // This is the recommended way to override configuration values. connectOptions.TargetHost = "localhost:7233"; Console.WriteLine($"\nLoaded '{profileName}' profile from {configFile} with overrides."); Console.WriteLine($" Address: {connectOptions.TargetHost} (overridden from localhost:9999)"); Console.WriteLine($" Namespace: {connectOptions.Namespace}"); Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); // Test the connection by checking the service var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new()); Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your .NET application. Import the `Temporalio.Client.EnvConfig` namespace to set connection options for the Temporal Client using environment variables. The `ClientEnvConfig.LoadClientConnectOptions` method will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. ```csharp {16,20} using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading the default environment configuration profile /// from a TOML file. /// public static class LoadFromFile { public static async Task RunAsync() { try { var connectOptions = ClientEnvConfig.LoadClientConnectOptions(); Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **Code** You can also provide connections options in your .NET code directly. To create an initial connection, provide the Namespace and API key values to the ` TemporalClient.ConnectAsync` method. ```csharp var myClient = TemporalClient.ConnectAsync(new() { Namespace = ".", ApiKey = "", Tls = new(), }); ``` To update an API key, update the value of `ApiKey` on the existing client connection: ```csharp myClient.Connection.ApiKey = myKeyUpdated; ``` To connect using mTLS instead of an API key, provide the client certificate and private key on `Tls`: ```csharp var myClient = TemporalClient.ConnectAsync(new() { Namespace = ".", Tls = new() { ClientCert = File.ReadAllBytes("client-cert.pem"), ClientPrivateKey = File.ReadAllBytes("client-private-key.pem"), }, }); ``` Unlike `ApiKey`, `TlsOptions` is not mutable on an existing connection — the certificate bytes are fixed once the connection is established. To rotate an mTLS client certificate without restarting your Worker, connect a new client with the new certificate and assign it to the running `TemporalWorker`'s `Client` property: ```csharp var newClient = await TemporalClient.ConnectAsync(new() { Namespace = ".", Tls = new() { ClientCert = File.ReadAllBytes("client-cert-new.pem"), ClientPrivateKey = File.ReadAllBytes("client-private-key-new.pem"), }, }); // worker is the TemporalWorker instance already running against the old client worker.Client = newClient; ``` Setting `Client` replaces the connection the Worker uses for subsequent calls to the Temporal Service (Workflow Task completion, Activity Heartbeats, and so on); calls already in flight on the old client are not interrupted. ## Start a Workflow **How to start a Workflow using the Temporal .NET SDK** [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. To start a Workflow Execution in .NET, use either the `StartWorkflowAsync()` or `ExecuteWorkflowAsync()` methods in the Client. You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) and [Task Queue](/task-queue) in the `WorkflowOptions` given to the method. ```csharp var result = await client.ExecuteWorkflowAsync( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue"); Console.WriteLine("Result: {0}", result); ``` ## Get Workflow results **How to get the results of a Workflow Execution using the Temporal .NET SDK** If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. Use `StartWorkflowAsync()` or `GetWorkflowHandle()` to return a Workflow handle. Then use the `GetResultAsync()` method to await on the result of the Workflow. To get a handle for an existing Workflow by its Id, you can use `GetWorkflowHandle()`. Then use [`DescribeAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_DescribeAsync_Temporalio_Client_WorkflowDescribeOptions_) to get the current status of the Workflow. If the Workflow does not exist, this call fails. ```csharp var handle = client.GetWorkflowHandle("my-workflow-id"); var result = await handle.GetResultAsync(); Console.WriteLine("Result: {0}", result); ``` --- # Nexus - .NET SDK Source: https://docs.temporal.io/develop/dotnet/nexus > This section explains how to use Temporal Nexus with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Temporal Nexus - [Quickstart](/develop/dotnet/nexus/quickstart) - [Feature guide](/develop/dotnet/nexus/feature-guide) - [Standalone Operations](/develop/dotnet/nexus/standalone-operations) --- # Temporal Nexus - .NET SDK feature guide Source: https://docs.temporal.io/develop/dotnet/nexus/feature-guide > Use Temporal Nexus within the .NET SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Use [Temporal Nexus](/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. > **💡 Tip:** > > New to Nexus? Start with the [Nexus .NET Quickstart](/develop/dotnet/nexus/quickstart). > This page shows how to do the following: - [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server) - [Create caller and handler Namespaces](#create-caller-handler-namespaces) - [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint) - [Define the Nexus Service contract](#define-nexus-service-contract) - [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers) - [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service) - [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server) - [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud) > **📝 Note:** > > This documentation uses source code derived from the [.NET Nexus sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusSimple). > ## Run the Temporal Development Server with Nexus enabled Prerequisites: - [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/dotnet/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) (v1.3.0 or higher recommended) - [Install the latest Temporal .NET SDK](https://learn.temporal.io/getting_started/dotnet/dev_environment/#install-the-temporal-net-sdk) (v1.9.0 or higher) The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled. ``` temporal server start-dev ``` This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`. ## Create caller and handler Namespaces Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. ``` temporal operator namespace create --namespace nexus-simple-handler-namespace temporal operator namespace create --namespace nexus-simple-caller-namespace ``` `nexus-simple-handler-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `nexus-simple-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. ## Create a Nexus Endpoint to route requests from caller to handler After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. ``` temporal operator nexus endpoint create \ --name nexus-simple-endpoint \ --target-namespace nexus-simple-handler-namespace \ --target-task-queue nexus-simple-handler-sample ``` You can also use the Web UI to create the Namespaces and Nexus endpoint. ## Define the Nexus Service contract Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](/dataconversion). The default data converter encodes payloads in the following order: Null, Byte array, Protobuf JSON, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, Protobuf and JSON are common choices. This example uses .NET classes serialized into JSON. [NexusSimple/IHelloService.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/IHelloService.cs) ```csharp using NexusRpc; [NexusService] public interface IHelloService { static readonly string EndpointName = "nexus-simple-endpoint"; [NexusOperation] EchoOutput Echo(EchoInput input); [NexusOperation] HelloOutput SayHello(HelloInput input); public record EchoInput(string Message); public record EchoOutput(string Message); public record HelloInput(string Name, HelloLanguage Language); public record HelloOutput(string Message); public enum HelloLanguage { En, Fr, De, Es, Tr, } } ``` ## Develop a Nexus Service and Operation handlers Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/cloud/limits#nexus-operation-request-timeout). Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database. Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint. The `Temporalio.Nexus` namespace has utilities to help create Nexus Operations: - `NexusOperationExecutionContext.Current.TemporalClient` \- Get the Temporal Client that the Worker was initialized with for synchronous handlers backed by Temporal primitives such as Signals and Queries - `WorkflowRunOperationHandler.FromHandleFactory` \- Run a Workflow as an asynchronous Nexus Operation This example starts with a sync Operation handler example using the `OperationHandler.Sync` method, and then shows how to create an async Operation handler that uses `WorkflowRunOperationHandler.FromHandleFactory` to start a handler Workflow from a Nexus Operation. ### Develop a Synchronous Nexus Operation handler The `OperationHandler.Sync` method is for exposing simple RPC handlers. Use `NexusOperationExecutionContext.Current.TemporalClient` to get the Temporal Client for signaling, querying, and listing Workflows. Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking). [NexusSimple/Handler/HelloService.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Handler/HelloService.cs) ```csharp using NexusRpc.Handlers; [NexusServiceHandler(typeof(IHelloService))] public class HelloService { [NexusOperationHandler] public IOperationHandler Echo() => // This Nexus service operation is a simple sync handler OperationHandler.Sync( (ctx, input) => new(input.Message)); // ... } ``` ### Use the Temporal Client for Signals, Queries, and Updates A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](/cloud/limits#nexus-operation-request-timeout). Updates should be short-lived to stay within this deadline. The [nexus_messaging](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging) sample shows how to create a Nexus Service that uses synchronous operations to send Updates and Queries: Use `NexusOperationExecutionContext`, like below, to get the Client that the Worker was initialized with. In this example, the Workflow Id is derived from the client Id using the `WorkflowIdForUser` method. This converts a given client Id (in this case, the client is passing in a user Id) to generate a Workflow Id from it. This way the client only needs the identifier it cares about. [NexusMessaging/CallerPattern/Handler/NexusGreetingService.cs](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging/CallerPattern/Handler/NexusGreetingService.cs) ```csharp private static string WorkflowIdForUser(string userId) => $"GreetingWorkflow_for_{userId}"; [NexusOperationHandler] public IOperationHandler GetLanguages() => OperationHandler.Sync( async (ctx, input) => { // Access the Temporal client from the Nexus operation context var client = NexusOperationExecutionContext.Current.TemporalClient; var handle = client.GetWorkflowHandle(WorkflowIdForUser(input.UserId)); return await handle.QueryAsync(wf => wf.QueryLanguages(input.IncludeUnsupported)); }); ... ``` There are two examples of messaging through Nexus in the sample code: the [caller pattern](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging/CallerPattern) and the [on-demand pattern](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusMessaging/OnDemandPattern). The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it. ### Develop an Asynchronous Nexus Operation handler to start a Workflow Use the `WorkflowRunOperationHandler.FromHandleFactory` method, which is the easiest way to expose a Workflow as an operation. [NexusSimple/Handler/HelloService.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Handler/HelloService.cs) ```csharp using NexusRpc.Handlers; using Temporalio.Nexus; [NexusServiceHandler(typeof(IHelloService))] public class HelloService { // ... [NexusOperationHandler] public IOperationHandler SayHello() => // This Nexus service operation is backed by a workflow run WorkflowRunOperationHandler.FromHandleFactory( (WorkflowRunOperationContext context, IHelloService.HelloInput input) => context.StartWorkflowAsync( (HelloHandlerWorkflow wf) => wf.RunAsync(input), // Workflow IDs should typically be business meaningful IDs and are used to // dedupe workflow starts. For this example, we're using the request ID // allocated by Temporal when the caller workflow schedules the operation, // this ID is guaranteed to be stable across retries of this operation. new() { Id = context.HandlerContext.RequestId })); } ``` Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. In general, the ID should be passed in the Operation input as part of the Nexus Service contract. > **💡 Tip:** > RESOURCES > > [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing. > #### Map a Nexus Operation input to multiple Workflow arguments A Nexus Operation can only take one input parameter. If you want a Nexus Operation to start a Workflow that takes multiple arguments, simply pass in different arguments using `RunAsync`. [NexusMultiArg/Handler/HelloService.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusMultiArg/Handler/HelloService.cs) ```csharp [NexusServiceHandler(typeof(IHelloService))] public class HelloService { [NexusOperationHandler] public IOperationHandler SayHello() => // This Nexus service operation is backed by a workflow run. For this sample, we are // altering the parameters to the workflow (in this case expanding to two parameters). WorkflowRunOperationHandler.FromHandleFactory( (WorkflowRunOperationContext context, IHelloService.HelloInput input) => context.StartWorkflowAsync( (HelloHandlerWorkflow wf) => wf.RunAsync(input.Language, input.Name), // Workflow IDs should typically be business meaningful IDs and are used to // dedupe workflow starts. For this example, we're using the request ID // allocated by Temporal when the caller workflow schedules the operation, // this ID is guaranteed to be stable across retries of this operation. new() { Id = context.HandlerContext.RequestId })); } ``` ### Register a Nexus Service in a Worker After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus Service in a Worker. [NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs) ```csharp async Task RunHandlerWorkerAsync() { // Run worker until cancelled logger.LogInformation("Running handler worker"); using var worker = new TemporalWorker( await ConnectClientAsync("nexus-simple-handler-namespace"), new TemporalWorkerOptions(taskQueue: "nexus-simple-handler-sample"). AddNexusService(new HelloService()). AddWorkflow()); try { await worker.ExecuteAsync(tokenSource.Token); } catch (OperationCanceledException) { logger.LogInformation("Handler worker cancelled"); } } ``` ### Use dependency injection with a Nexus Service handler Nexus Service handlers support dependency injection through the [Temporalio.Extensions.Hosting](https://github.com/temporalio/sdk-dotnet/tree/main/src/Temporalio.Extensions.Hosting) generic-host Worker. Register the handler on the Worker with `AddScopedNexusService`, and the container injects the handler's constructor dependencies. Use `AddSingletonNexusService` or `AddTransientNexusService` for singleton or transient lifetimes instead, mirroring `AddScopedActivities` / `AddSingletonActivities` / `AddTransientActivities`. For a complete, runnable example, see the [NexusDependencyInjection sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusDependencyInjection). [NexusDependencyInjection/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusDependencyInjection/Program.cs) ```csharp IHost host = Host.CreateDefaultBuilder(args) .ConfigureServices(ctx => ctx. // Add the dependency that will be injected into the Nexus Service handler AddScoped(). // Add the worker AddHostedTemporalWorker(handlerTaskQueue). ConfigureOptions(options => options.ClientOptions = LoadConnectOptions()). // Add the Nexus Service handler at the scoped level AddScopedNexusService()) .Build(); await host.RunAsync(); ``` The handler receives its dependencies through its constructor. The container creates a new scoped handler instance and its scoped dependencies for each Operation invocation; it does not cache them between invocations: [NexusDependencyInjection/Handler/GreetingServiceHandler.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusDependencyInjection/Handler/GreetingServiceHandler.cs) ```csharp [NexusServiceHandler(typeof(IGreetingService))] public class GreetingServiceHandler { private readonly IGreetingClient greetingClient; // The dependency is injected by the container public GreetingServiceHandler(IGreetingClient greetingClient) => this.greetingClient = greetingClient; [NexusOperationHandler] public IOperationHandler SayHello() => OperationHandler.Sync( (ctx, input) => greetingClient.GetGreetingAsync(input.Name)); } ``` ## Develop a caller Workflow that uses the Nexus Service Import the Service API package that has the necessary service and operation names and input/output types to execute a Nexus Operation from the caller Workflow: [NexusSimple/Caller/EchoCallerWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Caller/EchoCallerWorkflow.workflow.cs) ```csharp using Temporalio.Workflows; [Workflow] public class EchoCallerWorkflow { [WorkflowRun] public async Task RunAsync(string message) { var output = await Workflow.CreateNexusWorkflowClient(IHelloService.EndpointName). ExecuteNexusOperationAsync(svc => svc.Echo(new(message))); return output.Message; } } ``` [NexusSimple/Caller/HelloCallerWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Caller/HelloCallerWorkflow.workflow.cs) ```csharp using Temporalio.Workflows; [Workflow] public class HelloCallerWorkflow { [WorkflowRun] public async Task RunAsync(string name, IHelloService.HelloLanguage language) { var output = await Workflow.CreateNexusWorkflowClient(IHelloService.EndpointName). ExecuteNexusOperationAsync(svc => svc.SayHello(new(name, language))); return output.Message; } } ``` ### Set Nexus Operation timeouts Nexus Operations support [three types of timeouts](/nexus/operations#timeouts) that control how long the caller is willing to wait at different stages of the Operation lifecycle. Set these timeouts in `NexusWorkflowOperationOptions` when calling `ExecuteNexusOperationAsync`. #### Schedule-to-Close timeout The [Schedule-to-Close timeout](/nexus/operations#schedule-to-close-timeout) limits the total duration of the Operation from when it is scheduled to when it completes. The Nexus Machinery automatically retries failed requests until this timeout is exceeded. ```csharp var output = await Workflow.CreateNexusWorkflowClient(IHelloService.EndpointName). ExecuteNexusOperationAsync(svc => svc.SayHello(new(name, language)), new NexusWorkflowOperationOptions { ScheduleToCloseTimeout = TimeSpan.FromMinutes(10), }); ``` #### Schedule-to-Start timeout The [Schedule-to-Start timeout](/nexus/operations#schedule-to-start-timeout) limits how long the caller will wait for the Operation to be started by the handler. If not set, no Schedule-to-Start timeout is enforced. ```csharp var output = await Workflow.CreateNexusWorkflowClient(IHelloService.EndpointName). ExecuteNexusOperationAsync(svc => svc.SayHello(new(name, language)), new NexusWorkflowOperationOptions { ScheduleToStartTimeout = TimeSpan.FromMinutes(2), }); ``` #### Start-to-Close timeout The [Start-to-Close timeout](/nexus/operations#start-to-close-timeout) limits how long the caller will wait for an asynchronous Operation to complete after it has been started. This timeout only applies to asynchronous Operations. If not set, no Start-to-Close timeout is enforced. ```csharp var output = await Workflow.CreateNexusWorkflowClient(IHelloService.EndpointName). ExecuteNexusOperationAsync(svc => svc.SayHello(new(name, language)), new NexusWorkflowOperationOptions { StartToCloseTimeout = TimeSpan.FromMinutes(5), }); ``` ### Register the caller Workflow in a Worker After developing the caller Workflow, the next step is to register it with a Worker. [NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs) ```csharp async Task RunCallerWorkerAsync() { // Run worker until cancelled logger.LogInformation("Running caller worker"); using var worker = new TemporalWorker( await ConnectClientAsync("nexus-simple-caller-namespace"), new TemporalWorkerOptions(taskQueue: "nexus-simple-caller-sample"). AddWorkflow(). AddWorkflow()); try { await worker.ExecuteAsync(tokenSource.Token); } catch (OperationCanceledException) { logger.LogInformation("Caller worker cancelled"); } } ``` ### Develop a starter to start the caller Workflow To initiate the caller Workflow, a starter program is used. [NexusSimple/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/NexusSimple/Program.cs) ```csharp async Task ExecuteCallerWorkflowAsync() { logger.LogInformation("Executing caller echo workflow"); var client = await ConnectClientAsync("nexus-simple-caller-namespace"); var result1 = await client.ExecuteWorkflowAsync( (EchoCallerWorkflow wf) => wf.RunAsync("Nexus Echo 👋"), new(id: "nexus-simple-echo-id", taskQueue: "nexus-simple-caller-sample")); logger.LogInformation("Workflow result: {Result}", result1); logger.LogInformation("Executing caller hello workflow"); var result2 = await client.ExecuteWorkflowAsync( (HelloCallerWorkflow wf) => wf.RunAsync("Temporal", IHelloService.HelloLanguage.Es), new(id: "nexus-simple-hello-id", taskQueue: "nexus-simple-caller-sample")); logger.LogInformation("Workflow result: {Result}", result2); } ``` ## Make Nexus calls across Namespaces with a development Server Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app. ### Run Workers connected to a local development server Run the Nexus handler Worker: ```bash dotnet run handler-worker ``` In another terminal window, run the Nexus caller Worker: ```bash dotnet run caller-worker ``` ### Start a caller Workflow With the Workers running, the final step in the local development process is to start a caller Workflow. Run the starter: ```bash dotnet run caller-workflow ``` This will show the two workflows started and their results. ### Canceling a Nexus Operation To cancel a Nexus Operation from within a Workflow, cancel the cancellation token passed to the operation call. Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancellation request. If ignored, the operation may enter a terminal state. When a Nexus operation is started, the caller can specify different cancellation types that control how the caller reacts to cancellation: - `Abandon` - Do not request cancellation of the operation. - `TryCancel` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is done. - `WaitCancellationRequested` - Request cancellation of the operation and wait for confirmation that the request was received. Doesn't wait for actual cancellation. - `WaitCancellationCompleted` - Wait for operation completion. Operation may or may not complete as cancelled. The default is `WaitCancellationCompleted`. Users can set a different option for `CancellationType` in `NexusWorkflowOperationOptions` when starting an operation. Once the caller Workflow completes, the caller's Nexus Machinery will not make any further attempts to cancel operations that are still running. It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending operations to finish before exiting the Workflow. See the [Nexus cancellation sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusCancellation) for reference. ## Make Nexus calls across Namespaces in Temporal Cloud This section assumes you are already familiar with how to connect a Worker to Temporal Cloud. The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install `tcld` and generate certificates Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command (on macOS): ``` brew install temporalio/brew/tcld ``` If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: ``` tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key ``` These certificates will be valid for one year. ### Create caller and handler Namespaces Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. **Temporal CLI** ``` temporal cloud login temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` **tcld** ``` tcld login tcld namespace create \ --namespace \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 tcld namespace create \ --namespace \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/Namespaces](https://cloud.temporal.io/Namespaces). ### Create a Nexus Endpoint to route requests from caller to handler To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. **Temporal CLI** ``` temporal cloud nexus endpoint create \ --name nexus-simple-endpoint \ --target-task-queue nexus-simple-handler-sample \ --target-namespace \ --allow-namespace \ --description-file endpoint_description.md ``` **tcld** ``` tcld nexus endpoint create \ --name nexus-simple-endpoint \ --target-task-queue nexus-simple-handler-sample \ --target-namespace \ --allow-namespace \ --description-file endpoint_description.md ``` The `--allow-namespace` is used to build an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in Runtime Access Control. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus). ## Observability ### Web UI A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history: ![Observability Sync](/img/cloud/nexus/go-sdk-observability-sync.png) An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history: ![Observability Async](/img/cloud/nexus/go-sdk-observability-async.png) ### Temporal CLI Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: ``` temporal workflow describe -w ``` Nexus events are included in the caller's Event history: ``` temporal workflow show -w ``` For **asynchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationStarted` - `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationCompleted` > **📝 Note:** > > `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. > ## Learn more - Read the high-level description of the [Temporal Nexus feature](/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082). - Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus). - Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus). --- # Nexus .NET Quickstart Source: https://docs.temporal.io/develop/dotnet/nexus/quickstart > Build a Nexus Service that wraps an existing Temporal Workflow using the .NET SDK [Temporal Nexus](/evaluate/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow. > **ℹ️ Info:** > To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus). **Prerequisites:** Complete the [.NET SDK Quickstart](/develop/dotnet/set-up-your-local-dotnet) first. You should have `SayHelloWorkflow`, `MyActivities`, and a `Worker` project from that guide. ## What you'll build You have `SayHelloWorkflow` running in the `default` Namespace. By the end of this guide: 1. A Nexus Service will expose `SayHelloWorkflow` as an Operation. 2. A second Namespace will contain a Workflow that calls that Operation. 3. The caller Workflow will get back `"Hello, Temporal!"` — the same result, but across Namespaces. ## 1. Define the Nexus Service Create a file called `ISayHelloNexusService.cs` in the `Workflow` project. The `[NexusService]` attribute on an interface defines the Nexus Service contract. `[NexusOperation]` marks each method that callers can invoke. The `EndpointName` static field is shared between the handler and caller to keep the endpoint name in one place. `SayHelloWorkflow` returns `string`, so the operation output type is `string`. The input is a `record` carrying the workflow argument. ```csharp namespace MyNamespace; using NexusRpc; [NexusService] public interface ISayHelloNexusService { public static readonly string EndpointName = "my-nexus-endpoint-name"; [NexusOperation] string SayHello(MyInput input); public record MyInput(string Name); } ``` ## 2. Define the Nexus Operation handler Create a file called `SayHelloNexusServiceHandler.cs` in the `Workflow` project. `[NexusServiceHandler]` links this class to the `ISayHelloNexusService` contract. Each `[NexusOperationHandler]` method returns an `IOperationHandler` that describes how the operation runs. `WorkflowRunOperationHandler.FromHandleFactory` creates an asynchronous operation backed by a Workflow run. The `input.Name` bridges the Nexus `MyInput` record to `SayHelloWorkflow`'s `string` parameter. Using `context.HandlerContext.RequestId` as the Workflow ID ensures that retried Nexus operation requests are deduplicated. ```csharp namespace MyNamespace; using NexusRpc.Handlers; using Temporalio.Nexus; [NexusServiceHandler(typeof(ISayHelloNexusService))] public class SayHelloNexusServiceHandler { [NexusOperationHandler] public IOperationHandler SayHello() => WorkflowRunOperationHandler.FromHandleFactory( (WorkflowRunOperationContext context, ISayHelloNexusService.MyInput input) => context.StartWorkflowAsync( (SayHelloWorkflow wf) => wf.RunAsync(input.Name), new() { Id = context.HandlerContext.RequestId })); } ``` ## 3. Register the Nexus Service handler in a Worker Update `Worker/Program.cs` to register the Nexus Service handler alongside the existing Workflow and Activity registrations. A Worker will only handle incoming Nexus requests if the Nexus Service handlers are registered. Like `.AddActivity()`, `.AddNexusService()` takes an instance — both register concrete objects that the Worker dispatches work to. ```csharp // Worker/Program.cs var activities = new MyActivities(); using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") .AddActivity(activities.SayHello) .AddWorkflow() .AddNexusService(new SayHelloNexusServiceHandler())); ``` ## 4. Develop the caller Workflow Create a file called `CallerWorkflow.cs` in the `Workflow` project. The caller Workflow uses `Workflow.CreateNexusWorkflowClient()` to get a typed client bound to the Nexus Endpoint. `ExecuteNexusOperationAsync` starts the operation and waits for the result. The caller only depends on the Service contract (`ISayHelloNexusService`), not the handler implementation. This decoupling is what allows the caller and handler to live in separate Namespaces or even separate codebases. ```csharp namespace MyNamespace; using Temporalio.Workflows; [Workflow] public class CallerWorkflow { public static readonly string CallerTaskQueue = "my-caller-task-queue"; [WorkflowRun] public async Task RunAsync(string name) { return await Workflow .CreateNexusWorkflowClient( ISayHelloNexusService.EndpointName) .ExecuteNexusOperationAsync(svc => svc.SayHello(new(name))); } } ``` ## 5. Create the caller Namespace and Nexus Endpoint Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server. Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. The endpoint name must match the `EndpointName` constant defined in Step 1. Make sure your local Temporal dev server is running (`temporal server start-dev`). ```bash temporal operator namespace create --namespace my-caller-namespace ``` ```bash temporal operator nexus endpoint create \\ --name my-nexus-endpoint-name \\ --target-namespace default \\ --target-task-queue my-task-queue ``` ## 6. Add the caller project Create a `CallerStarter` console project that starts a caller Worker and executes the Workflow. The commands on the right create a new console project, add it to the solution, reference the Workflow project, and add the Temporalio package. ```bash dotnet new console -o CallerStarter dotnet sln TemporalioHelloWorld.sln \\ add CallerStarter/CallerStarter.csproj dotnet add CallerStarter/CallerStarter.csproj \\ reference Workflow/Workflow.csproj dotnet add CallerStarter/CallerStarter.csproj package Temporalio ``` ## 7. Create the caller starter Create the `CallerStarter/Program.cs` file with the code on the right. This brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `SayHello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller. ```csharp using MyNamespace; using Temporalio.Client; using Temporalio.Worker; var client = await TemporalClient.ConnectAsync( new("localhost:7233") { Namespace = "my-caller-namespace" }); using var tokenSource = new CancellationTokenSource(); Console.CancelKeyPress += (_, eventArgs) => { tokenSource.Cancel(); eventArgs.Cancel = true; }; using var worker = new TemporalWorker( client, new TemporalWorkerOptions(CallerWorkflow.CallerTaskQueue) .AddWorkflow()); Console.WriteLine("Running caller worker"); var workerTask = worker.ExecuteAsync(tokenSource.Token); var result = await client.ExecuteWorkflowAsync( (CallerWorkflow wf) => wf.RunAsync("Temporal"), new(id: $"caller-workflow-{Guid.NewGuid()}", taskQueue: CallerWorkflow.CallerTaskQueue)); Console.WriteLine("Workflow result: {0}", result); tokenSource.Cancel(); try { await workerTask; } catch (OperationCanceledException) { } ``` ## 8. Run and verify Run the application using the commands on the right. You should see: ``` Workflow result: Hello, Temporal! ``` Open the [Temporal Web UI](http://localhost:8233) and find the `CallerWorkflow` execution in the `my-caller-namespace` Namespace. You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history. ```bash dotnet run --project Worker/Worker.csproj ``` ```bash dotnet run --project CallerStarter/CallerStarter.csproj ``` ## Next Steps Now that you have a working Nexus Service, here are some resources to deepen your understanding: - **[.NET Nexus Feature Guide](/develop/dotnet/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls. - **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics. - **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker. - **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns. - **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries. - **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls. - **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests. - **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity. --- # Standalone Nexus Operations - .NET SDK Source: https://docs.temporal.io/develop/dotnet/nexus/standalone-operations > Execute Nexus Operations independently without a Workflow using the Temporal .NET SDK. > **Pre-release** > Requires .NET SDK `1.16.0` or above. All APIs are experimental and may be subject to backwards-incompatible changes. [Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using `Workflow.CreateNexusWorkflowClient()`, you execute a Standalone Nexus Operation directly from a Nexus Client created using `ITemporalClient.CreateNexusClient()`. Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/dotnet/nexus/feature-guide) for details on [defining a Service contract](/develop/dotnet/nexus/feature-guide#define-nexus-service-contract), [developing Operation handlers](/develop/dotnet/nexus/feature-guide#develop-nexus-service-operation-handlers), and [registering a Service in a Worker](/develop/dotnet/nexus/feature-guide#register-a-nexus-service-in-a-worker). This page focuses on the client-side APIs that are unique to Standalone Nexus Operations: - [Execute a Standalone Nexus Operation](#execute-operation) - [Get the result of a Standalone Nexus Operation](#get-operation-result) - [List Standalone Nexus Operations](#list-operations) - [Count Standalone Nexus Operations](#count-operations) - [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud) > **📝 Note:** > This documentation uses source code from the > [.NET Nexus Standalone sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusStandaloneOperations). > ## Prerequisites Standalone Nexus Operations are at Pre-release and require a special Temporal CLI build. ### 1. Install and verify the Pre-release Temporal CLI The `temporal nexus operation` commands require a Pre-release build of the Temporal CLI. See [Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for the platform downloads, then verify: ```bash ./temporal --version # temporal version 1.7.4-standalone-nexus-operations ``` Run it as `./temporal` from the directory where you extracted it. The standard `brew install temporal` build does not include Standalone Nexus Operation support during Pre-release. ### 2. Start a local dev server The Pre-release dev server enables Standalone Nexus Operations by default — no dynamic config is required. Start it with the caller and handler Namespaces pre-created: ```bash ./temporal server start-dev \ --namespace my-caller-namespace \ --namespace my-handler-namespace ``` The starter and Worker connect to two different Namespaces (a caller Namespace and a handler Namespace), mirroring how Nexus crosses Namespace boundaries. To run the examples on this page against the [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/NexusStandaloneOperations), create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue: ```bash ./temporal operator nexus endpoint create \ --name my-nexus-endpoint \ --target-namespace my-handler-namespace \ --target-task-queue nexus-handler-queue ``` Start the sample Worker in the handler Namespace: ```bash TEMPORAL_NAMESPACE=my-handler-namespace dotnet run worker ``` Run the starter in the caller Namespace (from a separate terminal): ```bash TEMPORAL_NAMESPACE=my-caller-namespace dotnet run starter ``` ## Execute a Standalone Nexus Operation To execute a Standalone Nexus Operation, first create a [`NexusClient`](https://dotnet.temporal.io/api/Temporalio.Client.NexusClient.html) using `ITemporalClient.CreateNexusClient()`, bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `ExecuteNexusOperationAsync()` from application code (for example, a starter program), not from inside a Workflow Definition. `ExecuteNexusOperationAsync` is a shortcut that starts the Operation and waits for the result. If you need a handle to the Operation while it runs, call `StartNexusOperationAsync` instead — it returns a [`NexusOperationHandle`](https://dotnet.temporal.io/api/Temporalio.Client.NexusOperationHandle.html) that you can use to get the result, describe, cancel, or terminate the Operation. `Id` is required on [`NexusOperationOptions`](https://dotnet.temporal.io/api/Temporalio.Client.NexusOperationOptions.html); `ScheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server. ```csharp var nexusClient = client.CreateNexusClient("my-nexus-endpoint"); var result = await nexusClient.ExecuteNexusOperationAsync( svc => svc.Echo(new("Nexus Echo 👋")), new("unique-operation-id") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); ``` You can also use the untyped overload that takes the Operation name as a string: ```csharp var nexusClient = client.CreateNexusClient("my-nexus-endpoint", "HelloService"); var handle = await nexusClient.StartNexusOperationAsync( "Echo", new IHelloService.EchoInput("Nexus Echo 👋"), new("unique-operation-id") { ScheduleToCloseTimeout = TimeSpan.FromSeconds(10), }); ``` Or use the Temporal CLI to execute a Standalone Nexus Operation: ```bash ./temporal nexus operation execute \ --namespace my-caller-namespace \ --endpoint my-nexus-endpoint \ --service HelloService \ --operation Echo \ --operation-id my-echo-op \ --input '{"Message":"hello"}' ``` ## Get the result of a Standalone Nexus Operation Use `NexusOperationHandle.GetResultAsync()` to await the Operation's completion and retrieve its result. This works for both synchronous and asynchronous (Workflow-backed) Operations. ```csharp var output = await handle.GetResultAsync(); logger.LogInformation("Operation result: {Message}", output.Message); ``` If the Operation completed successfully, the result is deserialized into the handle's result type. If the Operation failed, a `NexusOperationFailedException` is thrown. You can also recover a handle for an already-started Operation using `GetNexusOperationHandle()` on the Temporal Client: ```csharp var handle = client.GetNexusOperationHandle("unique-operation-id"); var output = await handle.GetResultAsync(); ``` Or use the Temporal CLI to wait for a result by Operation ID: ```bash ./temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op ``` ## List Standalone Nexus Operations Use [`ITemporalClient.ListNexusOperationsAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListNexusOperationsAsync_System_String_Temporalio_Client_NexusOperationListOptions_) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The call returns an `IAsyncEnumerable` that you can iterate with `await foreach`. Note that `ListNexusOperationsAsync` is called on the base `ITemporalClient`, not on the `NexusClient`. ```csharp await foreach (var execution in client.ListNexusOperationsAsync( "Endpoint = 'my-nexus-endpoint'")) { logger.LogInformation( "OperationID: {Id}, Operation: {Operation}, Status: {Status}", execution.OperationId, execution.Operation, execution.Status); } ``` The query string accepts [List Filter](/list-filter) syntax. For example, `"Endpoint = 'my-endpoint' AND Status = 'Running'"`. Or use the Temporal CLI: ```bash ./temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Count Standalone Nexus Operations Use [`ITemporalClient.CountNexusOperationsAsync()`](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_CountNexusOperationsAsync_System_String_Temporalio_Client_NexusOperationCountOptions_) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. Note that `CountNexusOperationsAsync` is called on the base `ITemporalClient`, not on the `NexusClient`. ```csharp var count = await client.CountNexusOperationsAsync( "Endpoint = 'my-nexus-endpoint'"); logger.LogInformation("Total Nexus operations: {Count}", count.Count); ``` Or use the Temporal CLI: ```bash ./temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Run Standalone Nexus Operations with Temporal Cloud Standalone Nexus Operations work against Temporal Cloud with the same code — only the client connection options change. For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate generation, and authentication options, see [Make Nexus calls across Namespaces in Temporal Cloud](/develop/dotnet/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) and [Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud). --- # Platform - .NET SDK Source: https://docs.temporal.io/develop/dotnet/platform > This section explains how to implement platform with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Platform - [Observability](/develop/dotnet/platform/observability) - [Enriching the UI](/develop/dotnet/platform/enriching-ui) --- # Enriching the user interface - .NET SDK Source: https://docs.temporal.io/develop/dotnet/platform/enriching-ui > Add contextual information to Workflows and events in the Temporal UI using the .NET SDK. Temporal supports adding context to Workflows and events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```csharp using Temporalio.Client; // Create client var client = await TemporalClient.ConnectAsync(new("localhost:7233")); // Start a Workflow with static summary and details var handle = await client.StartWorkflowAsync( (YourWorkflow wf) => wf.RunAsync("Workflow input"), new WorkflowOptions { Id = "your-Workflow-id", TaskQueue = "your-task-queue", StaticSummary = "Order processing for customer #12345", StaticDetails = "Processing premium order with expedited shipping" }); ``` `StaticSummary` is a single-line description that appears in the Workflow list view, limited to 200 bytes. `StaticDetails` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use the `ExecuteWorkflowAsync` method with the same parameters: ```csharp var result = await client.ExecuteWorkflowAsync( (YourWorkflow wf) => wf.RunAsync("Workflow input"), new WorkflowOptions { Id = "your-Workflow-id", TaskQueue = "your-task-queue", StaticSummary = "Order processing for customer #12345", StaticDetails = "Processing premium order with expedited shipping" }); ``` ### Inside the Workflow Within a Workflow, you can get and set the _current Workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```csharp using Temporalio.Workflows; [Workflow] public class YourWorkflow { [WorkflowRun] public async Task RunAsync(string input) { // Get the current details var currentDetails = Workflow.CurrentDetails; Workflow.Logger.LogInformation($"Current details: {currentDetails}"); // Set/update the current details Workflow.CurrentDetails = "Updated Workflow details with new status"; return "Workflow completed"; } } ``` ### Adding Summary to Activities and Timers You can attach a metadata parameter `Summary` to Activities when starting them from within a Workflow: ```csharp using Temporalio.Activities; using Temporalio.Workflows; [Workflow] public class YourWorkflow { [WorkflowRun] public async Task RunAsync(string input) { // Execute an activity with a summary var result = await Workflow.ExecuteActivityAsync( (YourActivities act) => act.YourActivityAsync(input), new ActivityOptions { StartToCloseTimeout = TimeSpan.FromSeconds(10), Summary = "Processing user data" }); return result; } } ``` Similarly, you can attach a `Summary` to timers within a Workflow: ```csharp using Temporalio.Workflows; [Workflow] public class YourWorkflow { [WorkflowRun] public async Task RunAsync(string input) { // Create a timer with a summary await Workflow.DelayWithOptionsAsync(new DelayOptions(TimeSpan.FromMinutes(5)) { Summary = "Waiting for payment confirmation" }); return "Timer completed"; } } ``` The input format for `Summary` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the Workflow details page, you'll find the Workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the Workflow - **Current Details** - Displays the dynamic details that can be updated during Workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. --- # Observability - .NET SDK Source: https://docs.temporal.io/develop/dotnet/platform/observability > Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more. This page covers features related to viewing the state of the application, including: - [Metrics](#metrics) - [Tracing](#tracing) - [Logging](#logging) - [Visibility](#visibility) The observability feature guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application). This includes the ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. ## Emit metrics Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process. For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics). - For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide. - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). - For an end-to-end example that exposes metrics with the .NET SDK, refer to the [samples-dotnet](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry) repo. Metrics in .NET are configured on the `Metrics` property of the `Telemetry` property on the `TemporalRuntime`. That object should be created globally and should be used for all clients; therefore, you should configure this before any other Temporal code. ### Set a Prometheus endpoint The following example exposes a Prometheus endpoint on port `9000`. ```csharp using Temporalio.Client; using Temporalio.Runtime; var runtime = new TemporalRuntime(new() { Telemetry = new() { Metrics = new() { Prometheus = new("0.0.0.0:9000") } }, }); var client = await Temporalio.ConnectAsync(new("localhost:7233") { Runtime = runtime }); ``` ### Set a custom metric meter A custom metric meter can be set on the telemetry options to handle metrics programmatically. The [Temporalio.Extensions.DiagnosticSource](https://github.com/temporalio/sdk-dotnet/tree/main/src/Temporalio.Extensions.DiagnosticSource) extension provides a custom metric meter implementation that sends all metrics to a [System.Diagnostics.Metrics.Meter](https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.metrics.meter) instance. ```csharp using System.Diagnostics.Metrics; using Temporalio.Client; using Temporalio.Extensions.DiagnosticSource; using Temporalio.Runtime; // Create .NET meter using var meter = new Meter("My.Meter"); // Can create MeterListener or OTel meter provider here... // Create Temporal runtime with a custom metric meter for that meter var runtime = new TemporalRuntime(new() { Telemetry = new() { Metrics = new() { CustomMetricMeter = new CustomMetricMeter(meter) }, }, }); var client = await Temporalio.ConnectAsync(new("localhost:7233") { Runtime = runtime }); ``` ## Setup Tracing Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows. To configure OpenTelemetry tracing in .NET, use the [Temporalio.Extensions.OpenTelemetry](https://github.com/temporalio/sdk-dotnet/tree/main/src/Temporalio.Extensions.OpenTelemetry) extension. The [`Temporalio.Extensions.OpenTelemetry.TracingInterceptor`](https://dotnet.temporal.io/api/Temporalio.Extensions.OpenTelemetry.TracingInterceptor.html) class can be set as an interceptor in the client options, or provided through a [Plugin](/develop/plugins-guide#interceptors) if you're building a reusable library. When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution. ## Log from a Workflow Logging enables you to record critical information during code execution. Loggers create an audit trail and capture information about your Workflow's operation. An appropriate logging level depends on your specific needs. During development or troubleshooting, you might use debug or even trace. In production, you might use info or warn to avoid excessive log volume. Logging uses the .NET standard logging APIs. You can find the log levels supported in [their official documentation](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.loglevel?view=net-11.0-pp). The Temporal SDK core normally uses `WARN` as its default logging level. The `LoggerFactory` can be set in the client. The following example shows logging on the console and sets the level to `Information`. ```csharp var client = await TemporalClient.ConnectAsync(new("localhost:7233") { LoggerFactory = LoggerFactory.Create(builder => builder. AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] "). SetMinimumLevel(LogLevel.Information)), }); ``` You can log from a Workflow using `Workflow.Logger` which is an instance of .NET's `ILogger`. ```csharp Workflow.Logger.LogInformation("Given name: {Name}", name); ``` ## Use Visibility APIs The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service. ### Use Search Attributes The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service in the CLI or Web UI. - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text` - Replace `CustomKeywordField` with the name of your Search Attribute. - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList` - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an option when starting the Execution. - In the Workflow by calling `UpsertTypedSearchAttributes`. - Read the value of the Search Attribute: - On the Client by calling `Describe` on a `WorkflowHandle`. - In the Workflow by looking at `WorkflowInfo`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/operator#list-2) - In code by calling `ListWorkflowsAsync`. ### List Workflow Executions Use the [ListWorkflowsAsync()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListWorkflowsAsync_System_String_Temporalio_Client_WorkflowListOptions_) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows. The result is an async enumerable. ```csharp await foreach (var wf in client.ListWorkflowsAsync("WorkflowType='GreetingWorkflow'")) { Console.WriteLine("Workflow: {0}", wf.Id); } ``` ### Set Custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create`or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. To set custom Search Attributes, use the `TypedSearchAttributes` property on `WorkflowOptions` for `StartWorkflowAsync` or `ExecuteWorkflowAsync`. Typed search attributes are a `SearchAttributeCollection` created with a builder. ```csharp // This only needs to be created once, so it is common to make it a static readonly even though we // create inline here for demonstration var myKeywordAttributeKey = SearchAttributeKey.CreateKeyword("MyKeywordAttribute"); // Start workflow with the search attribute collection var handle = await client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue") { TypedSearchAttributes = new SearchAttributeCollection.Builder(). Set(myKeywordAttributeKey, "SomeKeywordValue"). ToSearchAttributeCollection(), }); ``` ### Upsert Search Attributes You can upsert Search Attributes to add, update, or remove Search Attributes from within Workflow code. To upsert custom Search Attributes, use the [`UpsertTypedSearchAttributes()`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_UpsertTypedSearchAttributes_Temporalio_Workflows_SearchAttributeUpdate___) method with a set of updates. Keys can be predefined for reuse. ```csharp // These only need to be created once, so it is common to make them static readonly even though we // create inline here for demonstration var myKeywordAttributeKey = SearchAttributeKey.CreateKeyword("MyKeywordAttribute"); var myTextAttributeKey = SearchAttributeKey.CreateText("MyTextAttribute"); // Add/Update the keyword one and remove the text one Workflow.UpsertTypedSearchAttributes( myKeywordAttributeKey.ValueSet("SomeKeywordValue"), myTextAttributeKey.ValueUnset()); ``` --- # Set up your local with the .NET SDK Source: https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet > Configure your local development environment to get started developing with Temporal # Quickstart Configure your local development environment to get started developing with Temporal. ## Install .NET The .NET SDK requires .NET 6.0 or later. Install the latest version of .NET by following the [official .NET instructions](https://dotnet.microsoft.com/download). ## Install the Temporal .NET SDK Create a solution and the three projects used in this guide: `Workflow` (class library), `Worker` (console), and `Client` (console). Add them to the solution. Tip: You can also centralize the `Temporalio` package for all projects using `Directory.Packages.props` and `Directory.Build.props` at the solution root. ```bash # Create solution and projects mkdir TemporalioHelloWorld cd TemporalioHelloWorld dotnet new sln -n TemporalioHelloWorld dotnet new classlib -o Workflow dotnet new console -o Worker dotnet new console -o Client # Add projects to the solution dotnet sln TemporalioHelloWorld.sln add Workflow/Workflow.csproj Worker/Worker.csproj Client/Client.csproj # Add project references dotnet add Worker/Worker.csproj reference Workflow/Workflow.csproj dotnet add Client/Client.csproj reference Workflow/Workflow.csproj # Install Temporal SDK in each project dotnet add Workflow/Workflow.csproj package Temporalio dotnet add Worker/Worker.csproj package Temporalio dotnet add Client/Client.csproj package Temporalio ``` Build the solution: ```bash dotnet build ``` ## Install Temporal CLI and start the development server The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI: **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - Your .NET SDK installation is working - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly #### Tip: Example Directory Structure ```text TemporalioHelloWorld/ ├── Client/ │ ├── Client.csproj │ └── Program.cs # Starts a workflow ├── Worker/ │ ├── Worker.csproj │ └── Program.cs # Runs a worker ├── Workflow/ │ ├── Workflow.csproj │ ├── MyActivities.cs # Activity definition │ └── SayHelloWorkflow.cs # Workflow definition └── TemporalioHelloWorld.sln ``` ### 1. Create the Activity and Workflow #### Create an Activity file (MyActivities.cs) in the Workflow project: ```csharp namespace MyNamespace; using Temporalio.Activities; public class MyActivities { // Activities can be async and/or static too! We just demonstrate instance // methods since many will use them that way. [Activity] public string SayHello(string name) => $"Hello, {name}!"; } ``` An Activity is a normal function or method that executes a single, well-defined action (either short or long running), which often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration. #### Create a Workflow file (SayHelloWorkflow.cs) in the Workflow project: ```csharp namespace MyNamespace; using Temporalio.Workflows; [Workflow] public class SayHelloWorkflow { [WorkflowRun] public async Task RunAsync(string name) { // This workflow just runs a simple activity to completion. // StartActivityAsync could be used to just start and there are many // other things that you can do inside a workflow. return await Workflow.ExecuteActivityAsync( // This is a lambda expression where the instance is typed. If this // were static, you wouldn't need a parameter. (MyActivities act) => act.SayHello(name), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) } ); } } ``` Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. ### 2. Create the Worker With your Activity and Workflow defined, you need a Worker to execute them. #### Create a Worker file (Program.cs) in the Worker project: ```csharp using MyNamespace; using Temporalio.Client; using Temporalio.Worker; // Create a client to localhost on "default" namespace var client = await TemporalClient.ConnectAsync(new("localhost:7233")); // Cancellation token to shutdown worker on ctrl+c using var tokenSource = new CancellationTokenSource(); Console.CancelKeyPress += (_, eventArgs) => { tokenSource.Cancel(); eventArgs.Cancel = true; }; // Create an activity instance since we have instance activities. If we had // all static activities, we could just reference those directly. var activities = new MyActivities(); // Create worker with the activity and workflow registered using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") .AddActivity(activities.SayHello) .AddWorkflow() ); // Run worker until cancelled Console.WriteLine("Running worker"); try { await worker.ExecuteAsync(tokenSource.Token); } catch (OperationCanceledException) { Console.WriteLine("Worker cancelled"); } ``` Run the Worker: ```bash dotnet run --project Worker/Worker.csproj ``` Keep this terminal running - you should see `Running worker` displayed. A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues the Workflow or Activity task from the Task Queue, it then executes that task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). ### 3. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. This final step will validate that everything is working correctly. #### Create a Client file (Program.cs) in the Client project: ```csharp using MyNamespace; using Temporalio.Client; // Create a client to localhost on "default" namespace var client = await TemporalClient.ConnectAsync(new("localhost:7233")); // Run workflow var result = await client.ExecuteWorkflowAsync( (SayHelloWorkflow wf) => wf.RunAsync("Temporal"), new(id: $"my-workflow-id-{Guid.NewGuid()}", taskQueue: "my-task-queue") ); Console.WriteLine("Workflow result: {0}", result); ``` While the Worker is still running, run the Workflow: ```bash dotnet run --project Client/Client.csproj ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the workflow and activity - Output: `Workflow result: Hello Temporal` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/dotnet/first_program_in_dotnet/): Create a basic Workflow and run it with the Temporal .NET SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Workers - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers > This section explains how to implement Workers with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Workers - [Worker processes](/develop/dotnet/workers/run-worker-process) - [Interceptors](/develop/dotnet/workers/interceptors) --- # Interceptors - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers/interceptors Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to apply shared behavior across many calls, such as tracing and authorization, before calls reach the application code and after they return. This is similar to middleware in other frameworks, like [ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware). There are two main types of interceptors: inbound and outbound. * Outbound interceptors wrap network calls, running before they reach the network and after they return. * Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns. Concretely, there are five categories of inbound and outbound calls that you can modify in this way: | | [Outbound Client](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.ClientOutboundInterceptor.html) | [Inbound Workflow](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.WorkflowInboundInterceptor.html) | [Outbound Workflow](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.WorkflowOutboundInterceptor.html) | [Inbound Activity](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.ActivityInboundInterceptor.html) | [Outbound Activity](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.ActivityOutboundInterceptor.html) | | --- | --- | --- | --- | --- | --- | | **Description** | Wraps calls from your application to the Temporal Client to start a Workflow or send [Messages](/encyclopedia/workflow-message-passing/) to it | Wraps calls arriving into a [Workflow Execution](/workflow-execution), such as executing the Workflow, handling [Messages](/encyclopedia/workflow-message-passing/) | Wraps calls a [Workflow](/workflow-definition) makes to the SDK, such as scheduling [Activities](/activities), starting [Child Workflows](/child-workflows), and invoking [Nexus Operations](/nexus) | Wraps calls arriving into an [Activity Execution](/activity-execution) | Wraps calls an [Activity](/activities) makes to the SDK, such as sending [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) and reading Activity info | | **Runs on** | Client | Worker (Workflow sandbox) | Worker (Workflow sandbox) | Worker (Activity context) | Worker (Activity context) | | **Example methods** | `StartWorkflowAsync()`, `WorkflowHandle.SignalAsync()`, `ListWorkflowsAsync()` | `ExecuteWorkflowAsync()`, `WorkflowHandle.QueryAsync()`, `WorkflowHandle.SignalAsync ()`, `WorkflowHandle.ExecuteUpdateAsync()` | `StartActivityAsync()`, `StartChildWorkflowAsync()`, `ChildWorkflowHandle.SignalAsync()`, `StartNexusOperationAsync()` | `ExecuteActivityAsync()` | `Info()`, `Heartbeat()` | > **⚠️ Warning:** > Workflow interceptors and replay > > Workflow inbound and outbound interceptor methods also execute during [replay](/develop/dotnet/best-practices/testing-suite#replay). Use replay-safe APIs for logging, randomness, and time in these interceptors. > See [Develop Workflow logic](/develop/dotnet/workflows/basics#workflow-logic-requirements) for details. > > If you want to write generic code shared by all inbound Workflow call handlers but want to skip read-only operations, check [`Workflow.Unsafe.IsReplaying`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplaying). > > Activity and Client interceptors are not affected by replay. > ## Register an Interceptor Registering an interceptor means supplying an interceptor instance to the SDK so Temporal can invoke it when matching Client or Worker calls occur. Once registered, the interceptor runs as part of the call path and can observe or modify request and response data. ### Register on the Client Pass interceptors in the `Interceptors` property of [`TemporalClientConnectOptions`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClientConnectOptions.html). Client interceptors modify outbound calls such as starting and signaling Workflows. One example is [setting up tracing](/develop/dotnet/platform/observability) to see your call graph of a Workflow. ```csharp using Temporalio.Extensions.OpenTelemetry; var interceptor = new TracingInterceptor(); var client = await TemporalClient.ConnectAsync(new() { TargetHost = "localhost:7233", Interceptors = [interceptor], }); ``` The `Interceptors` list can contain multiple interceptors. The default behavior for interceptors is to form a chain. A method implemented on an interceptor instance in the list can perform side effects, and modify the data, before passing it on to the corresponding method on the next interceptor in the list. ### Register via a Plugin If you're building a reusable library or want to bundle interceptors with other primitives, you can register them through a [Plugin](/develop/plugins-guide#interceptors). ### Register on the Worker only If your interceptor doesn't affect the Client, you can pass interceptors in the `Interceptors` argument of `TemporalWorkerOptions`. Worker interceptors modify inbound and outbound Workflow and Activity calls. ```csharp using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") { Interceptors = [interceptor] } .AddActivity(activities.SayHello) .AddWorkflow() ); ``` ## How to implement Interceptors Interceptors run as a chain. Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage. ### Implementing Client call Interceptors To modify outbound Client calls, define a class implementing [`IClientInterceptor`](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.IClientInterceptor.html). Implement `InterceptClient()` to return a [`ClientOutboundInterceptor`](https://dotnet.temporal.io/api/Temporalio.Client.Interceptors.ClientOutboundInterceptor.html), overriding the outbound Client calls you want to modify. `IClientInterceptor.InterceptClient` receives the next `ClientOutboundInterceptor` in the chain and returns the created interceptor. This example implements an Interceptor on outbound Client calls that sets a certain key in the outbound `headers` field. A User ID is context-propagated by being sent in a header field with outbound requests: ```csharp using Google.Protobuf; using Temporalio.Api.Common.V1; using Temporalio.Client; using Temporalio.Client.Interceptors; public static class UserContext { private static readonly AsyncLocal CurrentUser = new(); public static string? UserId { get => CurrentUser.Value; set => CurrentUser.Value = value; } } public class ContextPropagationInterceptor : IClientInterceptor { public ClientOutboundInterceptor InterceptClient( ClientOutboundInterceptor nextInterceptor) => new ContextPropagationClientOutboundInterceptor(nextInterceptor); } public class ContextPropagationClientOutboundInterceptor( ClientOutboundInterceptor next) : ClientOutboundInterceptor(next) { public override Task> StartWorkflowAsync(StartWorkflowInput input) { var headers = input.Headers ?? new Dictionary(); headers["user-id"] = new Payload { Metadata = { ["encoding"] = ByteString.CopyFromUtf8("plain/text") }, Data = ByteString.CopyFromUtf8(UserContext.UserId), }; return base.StartWorkflowAsync(input with { Headers = headers }); } } ``` You can then [register](#register) this interceptor in your client/starter code. Your interceptor classes don't need to implement every method. The default implementation is always to pass the data on to the next method in the interceptor chain. During execution, when the SDK encounters an Inbound Activity call, it will look to the first Interceptor instance, get hold of the appropriate intercepted method, and call it. The intercepted method will perform its function then call the same method on the next Interceptor in the chain. At the end of the chain the SDK will call the "real" SDK method. ### Implementing Worker call Interceptors To modify inbound Workflow and Activity calls, define a class implementing [`IWorkerInterceptor`](https://dotnet.temporal.io/api/Temporalio.Worker.Interceptors.IWorkerInterceptor.html). It provides `InterceptActivity()`, `InterceptWorkflow()`, and `InterceptNexusOperation()` methods for Activity, Workflow, and Nexus interception. This example demonstrates using an interceptor to measure [Schedule-To-Start](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) and Schedule-To-Close latency. Notice how the interceptor wraps the call. It records Schedule-To-Start before `ExecuteActivityAsync`, then records Schedule-To-Close after it completes: ```csharp using Temporalio.Activities; using Temporalio.Worker; using Temporalio.Worker.Interceptors; public class SimpleWorkerInterceptor : IWorkerInterceptor { public ActivityInboundInterceptor InterceptActivity( ActivityInboundInterceptor nextInterceptor) => new ActivityMetricsInterceptor(nextInterceptor); public WorkflowInboundInterceptor InterceptWorkflow( WorkflowInboundInterceptor nextInterceptor) => nextInterceptor; public NexusOperationInboundInterceptor InterceptNexusOperation( NexusOperationInboundInterceptor nextInterceptor) => nextInterceptor; } public class ActivityMetricsInterceptor(ActivityInboundInterceptor next) : ActivityInboundInterceptor(next) { public override async Task ExecuteActivityAsync( ExecuteActivityInput input) { var info = ActivityExecutionContext.Current.Info; var started = DateTimeOffset.UtcNow; // Before the activity executes var scheduleToStart = started - info.CurrentAttemptScheduledTime; Console.WriteLine( $"Schedule-To-Start latency: {scheduleToStart}"); // Execute the activity var result = await base.ExecuteActivityAsync(input); // After the activity completes var scheduleToClose = DateTimeOffset.UtcNow - info.CurrentAttemptScheduledTime; Console.WriteLine( $"Schedule-To-Close latency: {scheduleToClose}"); return result; } } ``` Register it on the Worker: ```csharp using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") { Interceptors = new IWorkerInterceptor[] { new SimpleWorkerInterceptor(), }, } .AddActivity(activities.SayHello) .AddWorkflow()); await worker.ExecuteAsync(); ``` --- # Run a Worker - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers/run-worker-process > Create and run a Temporal Worker using the .NET SDK. This page covers long-lived Workers that you host and run as persistent processes. For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/dotnet/workers/serverless-workers). ## Create and run a Worker Create a `TemporalWorker` with a [Temporal Client](/develop/dotnet/client/temporal-client) and a `TemporalWorkerOptions` that names the Task Queue to poll. Add the Workflows and Activities the Worker can execute, then call `ExecuteAsync()` to start polling. [features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs) ```cs var options = new TemporalWorkerOptions("my-task-queue"); options.AddWorkflow(); options.AddAllActivities(typeof(GreetingActivities), null); using var worker = new TemporalWorker(client, options); await worker.ExecuteAsync(CancellationToken.None); ``` `ExecuteAsync()` takes a `CancellationToken` and polls until that token is cancelled. The snippet passes `CancellationToken.None`, which never cancels, so the Worker runs until the process exits. To stop a Worker on demand, pass a token you control instead. See [Shut down a Worker](#shut-down-a-worker). `TemporalWorker` implements `IDisposable`, so declare it with `using` to release its resources when the process exits. ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Add Workflows with `AddWorkflow()` and Activities with `AddActivity()` or `AddAllActivities()`: ```csharp var options = new TemporalWorkerOptions("my-task-queue"); options.AddWorkflow(); options.AddWorkflow(); options.AddAllActivities(new MyActivities(databaseClient)); ``` `AddAllActivities()` registers every method marked with `[Activity]`. Pass an instance to register instance methods, which lets Activities share state such as a database client. For a class of static Activity methods, pass the type and `null` instead. ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/dotnet/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options `TemporalWorkerOptions` controls concurrency limits, pollers, timeouts, and caching, including `MaxConcurrentActivities`, `MaxConcurrentWorkflowTasks`, and `MaxCachedWorkflows`. The defaults work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `DeploymentOptions`, then set a versioning behavior on each Workflow. [features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs) ```cs var options = new TemporalWorkerOptions("my-task-queue") { DeploymentOptions = new WorkerDeploymentOptions( new WorkerDeploymentVersion("my-app", "1.0"), useWorkerVersioning: true), }; options.AddWorkflow(); options.AddAllActivities(typeof(GreetingActivities), null); using var worker = new TemporalWorker(client, options); ``` Set the behavior per Workflow with `[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]`, or set a default for the whole Worker with `DefaultVersioningBehavior` on `WorkerDeploymentOptions`. `VersioningBehavior` comes from the `Temporalio.Common` namespace. A versioning behavior applies only to a Worker that has versioning enabled. If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker To stop a Worker on demand, start it with a token you can cancel yourself instead of `CancellationToken.None`. Create a `CancellationTokenSource`, pass its `Token` to `ExecuteAsync()`, then cancel the source when the Worker should stop, such as from a `Console.CancelKeyPress` handler. The Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to `GracefulShutdownTimeout`. [features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs) ```cs using var tokenSource = new CancellationTokenSource(); Console.CancelKeyPress += (_, eventArgs) => { tokenSource.Cancel(); eventArgs.Cancel = true; }; var options = new TemporalWorkerOptions("my-task-queue") { GracefulShutdownTimeout = TimeSpan.FromSeconds(30), }; options.AddWorkflow(); using var worker = new TemporalWorker(client, options); await worker.ExecuteAsync(tokenSource.Token); ``` `ExecuteAsync()` throws `OperationCanceledException` once the Worker stops, so catch it where you want the process to exit cleanly. See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Serverless Workers - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the .NET SDK. > **Public Preview** > AWS Lambda support is in Public Preview. GCP Cloud Run support is in Pre-release, and its APIs may change in > backwards-incompatible ways. To request Cloud Run access, create a [support ticket](/cloud/support#support-ticket) or > contact your account team, and [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear > when Cloud Run reaches Public Preview. Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes. Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**AWS Lambda**](/develop/dotnet/workers/serverless-workers/aws-lambda) - Use the `Temporalio.Extensions.Aws.Lambda` NuGet package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle. - [**GCP Cloud Run**](/develop/dotnet/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, container packaging, and handling scale-in. --- # Serverless Workers on AWS Lambda - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers/aws-lambda > Write a Temporal Worker that runs on AWS Lambda using the .NET SDK Temporalio.Extensions.Aws.Lambda package. > **Public Preview** The `Temporalio.Extensions.Aws.Lambda` NuGet package lets you run a Temporal Serverless Worker on AWS Lambda. Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline. You register Workflows and Activities the same way you would with a standard Worker. For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda). ## Create and run a Worker in Lambda Add the `Temporalio.Extensions.Aws.Lambda` NuGet package: ```bash dotnet add package Temporalio.Extensions.Aws.Lambda ``` Use `TemporalLambdaWorker.CreateHandler` to create a Lambda handler that runs a Temporal Worker. Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities. Assign the result to a static field so the handler is created once during Lambda cold start and reused across invocations. ```csharp {10-17} namespace MyApp; using Amazon.Lambda.Core; using Temporalio.Common; using Temporalio.Extensions.Aws.Lambda; public class LambdaFunction { private static readonly Func WorkerHandler = TemporalLambdaWorker.CreateHandler( new WorkerDeploymentVersion("my-app", "build-1"), config => { config.WorkerOptions.TaskQueue = "my-task-queue"; config.WorkerOptions.AddWorkflow(); config.WorkerOptions.AddActivity(Activities.HelloActivity); }); public Task HandlerAsync(Stream input, ILambdaContext context) => WorkerHandler(input, context); } ``` For a complete project, see the [Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/main/src/LambdaWorker). The `WorkerDeploymentVersion` is required. Worker Deployment Versioning is always enabled for Serverless Workers. Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`. Set it per-Workflow with the `[Workflow]` attribute, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. The default versioning behavior is `AutoUpgrade`. [src/LambdaWorker/Worker/SampleWorkflow.workflow.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/Worker/SampleWorkflow.workflow.cs) ```csharp {7} namespace TemporalioSamples.LambdaWorker.Worker; using Microsoft.Extensions.Logging; using Temporalio.Common; using Temporalio.Workflows; [Workflow(VersioningBehavior = VersioningBehavior.Pinned)] public class SampleWorkflow { [WorkflowRun] public async Task RunAsync(string name) { Workflow.Logger.LogInformation("SampleWorkflow started with name: {Name}", name); var result = await Workflow.ExecuteActivityAsync( () => Activities.HelloActivity(name), new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) }); Workflow.Logger.LogInformation("SampleWorkflow completed with result: {Result}", result); return result; } } ``` ## Configure the Temporal connection The `Temporalio.Extensions.Aws.Lambda` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details. Compared with long-lived Workers, the location of the config file is resolved differently, in the following order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. The file is optional. If absent, only environment variables are used. Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options. ### TLS/CA loading on Lambda Some AWS Lambda .NET images override the `SSL_CERT_FILE` environment variable in a way that prevents the SDK's Rust-based runtime from loading system root CAs. If you encounter TLS certificate errors on Lambda, see the [AWS Lambda .NET CA loading workaround](https://github.com/temporalio/sdk-dotnet#aws-lambda-net-ca-loading-issues) in the SDK README. ## Adjust Worker defaults for Lambda The `Temporalio.Extensions.Aws.Lambda` package applies conservative defaults suited to short-lived Lambda invocations. These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment. | Setting | Lambda default | |---|---| | `MaxConcurrentActivities` | 2 | | `MaxConcurrentWorkflowTasks` | 10 | | `MaxConcurrentLocalActivities` | 2 | | `MaxConcurrentNexusTasks` | 5 | | `MaxConcurrentWorkflowTaskPolls` | 2 | | `MaxConcurrentActivityTaskPolls` | 1 | | `MaxConcurrentNexusTaskPolls` | 1 | | `MaxCachedWorkflows` | 30 | | `GracefulShutdownTimeout` | 5 seconds | | `DisableEagerActivityExecution` | Always `true` | | `ShutdownDeadlineBuffer` | 7 seconds | `DisableEagerActivityExecution` is always `true` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. `ShutdownDeadlineBuffer` is specific to the `Temporalio.Extensions.Aws.Lambda` package. It controls the time reserved after the worker run budget for worker shutdown and hooks. The default is 7 seconds. If your Worker handles long-running Activities, increase `GracefulShutdownTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities). ## Add observability with OpenTelemetry The `Temporalio.Extensions.Aws.Lambda.OpenTelemetry` NuGet package provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. The underlying metrics and traces are the same ones the .NET SDK emits in any environment. For general observability concepts and the full list of available metrics, see the [SDK metrics reference](/references/sdk-metrics). Add the OpenTelemetry extension package: ```bash dotnet add package Temporalio.Extensions.Aws.Lambda.OpenTelemetry ``` Call `ApplyOpenTelemetryDefaults` on the options in the configure callback: [src/LambdaWorker/Worker/Function.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/Worker/Function.cs) ```csharp {17} namespace TemporalioSamples.LambdaWorker.Worker; using Amazon.Lambda.Core; using Temporalio.Common; using Temporalio.Extensions.Aws.Lambda; using Temporalio.Extensions.Aws.Lambda.OpenTelemetry; public class LambdaFunction { private static readonly Func WorkerHandler = TemporalLambdaWorker.CreateHandler( new WorkerDeploymentVersion( LambdaWorkerSample.DeploymentName, LambdaWorkerSample.BuildId), config => { config.ApplyOpenTelemetryDefaults(); LambdaWorkerSample.ConfigureWorkerOptions(config.WorkerOptions); }); public Task HandlerAsync(Stream input, ILambdaContext context) => WorkerHandler(input, context); } ``` `ApplyOpenTelemetryDefaults` configures Temporal tracing with `TracingInterceptor`, creates an OTLP trace exporter and tracer provider, configures Core SDK OTLP metrics, uses AWS X-Ray-compatible trace IDs, and registers a per-invocation shutdown hook that force-flushes traces. By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. You can customize the defaults by passing a `LambdaWorkerOpenTelemetryOptions` object: ```csharp config.ApplyOpenTelemetryDefaults( new LambdaWorkerOpenTelemetryOptions { CollectorEndpoint = "http://localhost:4317", ServiceName = "my-worker", MetricsExportInterval = TimeSpan.FromSeconds(5), }); ``` Core SDK metrics export every 10 seconds by default. Set `MetricsExportInterval` shorter than your Lambda timeout to increase the chance that at least one metrics export happens during each invocation. To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function. .NET does not need a language-specific ADOT layer because the OTel SDK is included as a dependency of the package. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle the following `otel-collector-config.yaml` in your Lambda deployment package: [src/LambdaWorker/otel-collector-config.template.yaml](https://github.com/temporalio/samples-dotnet/blob/main/src/LambdaWorker/otel-collector-config.template.yaml) ```yaml receivers: otlp: protocols: grpc: endpoint: "localhost:4317" http: endpoint: "localhost:4318" exporters: awsxray: region: ${env:AWS_REGION} awsemf: namespace: TemporalWorkerMetrics log_group_name: /aws/lambda/${env:AWS_LAMBDA_FUNCTION_NAME} region: ${env:AWS_REGION} dimension_rollup_option: NoDimensionRollup resource_to_telemetry_conversion: enabled: true service: pipelines: traces: receivers: [otlp] exporters: [awsxray] metrics: receivers: [otlp] exporters: [awsemf] telemetry: logs: level: info metrics: address: localhost:8888 ``` Set the following environment variable on the Lambda function to point the Collector at the bundled config: - `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` Enable X-Ray active tracing on the Lambda function: ```bash aws lambda update-function-configuration \ --function-name \ --tracing-config Mode=Active ``` The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role. Without these permissions, the Collector fails silently and no telemetry appears. You can also configure tracing and metrics manually using `TracingInterceptor` and `TemporalRuntime`: ```csharp using Temporalio.Extensions.OpenTelemetry; config.ClientOptions.Interceptors = new[] { new TracingInterceptor() }; config.ClientOptions.Runtime = new TemporalRuntime(new TemporalRuntimeOptions { Telemetry = new TelemetryOptions { Metrics = new MetricsOptions(new OpenTelemetryOptions("http://collector:4317")), }, }); ``` --- # Serverless Workers on GCP Cloud Run - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the .NET SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other .NET Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific package. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running .NET Worker, then set `DeploymentOptions` on `TemporalWorkerOptions` to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```csharp using Temporalio.Client; using Temporalio.Common; using Temporalio.Worker; var client = await TemporalClient.ConnectAsync( new(Environment.GetEnvironmentVariable("TEMPORAL_ADDRESS")!) { Namespace = Environment.GetEnvironmentVariable("TEMPORAL_NAMESPACE")!, ApiKey = Environment.GetEnvironmentVariable("TEMPORAL_API_KEY"), Tls = new(), }); var options = new TemporalWorkerOptions( Environment.GetEnvironmentVariable("TEMPORAL_TASK_QUEUE")!) { DeploymentOptions = new(new("my-app", "build-1"), useWorkerVersioning: true) { DefaultVersioningBehavior = VersioningBehavior.Pinned, }, }; options.AddWorkflow(); options.AddAllActivities(typeof(GreetingActivities), null); using var worker = new TemporalWorker(client, options); await worker.ExecuteAsync(CancellationToken.None); ``` The two arguments to `WorkerDeploymentVersion` are the deployment name and the build ID, and together they identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `Pinned` or `AutoUpgrade`. Setting `DefaultVersioningBehavior` as shown above covers every Workflow on the Worker. To set the behavior per Workflow instead, set `VersioningBehavior` on the `Workflow` attribute: ```csharp using Temporalio.Common; using Temporalio.Workflows; [Workflow(VersioningBehavior = VersioningBehavior.Pinned)] public class GreetingWorkflow { [WorkflowRun] public async Task RunAsync(string name) => // ... } ``` For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/dotnet/workers/run-worker-process). ## Configure the Temporal connection Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext. The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace. To load those values through the shared configuration format instead of reading them yourself, use `ClientEnvConfig.LoadClientConnectOptions()` from the `Temporalio.Common.EnvConfig` namespace. For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration). ## Package the Worker image Publish the Worker and run it on a .NET runtime image: ```dockerfile FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build WORKDIR /src COPY *.csproj ./ RUN dotnet restore COPY . . RUN dotnet publish -c Release -o /out FROM mcr.microsoft.com/dotnet/runtime:9.0 WORKDIR /app COPY --from=build /out ./ CMD ["dotnet", "MyWorker.dll"] ``` The Worker runs on a Rust core that reads TLS roots from the operating system's certificate store, so the runtime image must include one. The Debian-based `mcr.microsoft.com/dotnet/runtime` images do. ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/dotnet/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over: ```csharp [Activity] public static string Process(IReadOnlyList items) { for (var i = 0; i < items.Count; i++) { ActivityExecutionContext.Current.Heartbeat(i); // ... process items[i] } return "done"; } ``` For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - .NET SDK](/develop/dotnet/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). --- # Workflows - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows > This section explains how to implement Workflows with the .NET SDK ![.NET SDK Banner](/img/assets/banner-dotnet-temporal.png) ## Workflows - [Workflow basics](/develop/dotnet/workflows/basics) - [Child Workflows](/develop/dotnet/workflows/child-workflows) - [Continue-As-New](/develop/dotnet/workflows/continue-as-new) - [Cancellation](/develop/dotnet/workflows/cancellation) - [Timeouts](/develop/dotnet/workflows/timeouts) - [Message passing](/develop/dotnet/workflows/message-passing) - [Schedules](/develop/dotnet/workflows/schedules) - [Timers](/develop/dotnet/workflows/timers) - [Dynamic Workflow](/develop/dotnet/workflows/dynamic-workflow) - [Versioning](/develop/dotnet/workflows/versioning) --- # Workflow basics - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/basics > This section explains Workflow basics with the .NET SDK ## Develop a Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal .NET SDK programming model, Workflows are defined as classes. Specify the `[Workflow]` attribute from the `Temporalio.Workflows` namespace on the Workflow class to identify a Workflow. Use the `[WorkflowRun]` attribute to mark the entry point method to be invoked. This must be set on one asynchronous method defined on the same class as `[Workflow]`. ```csharp using Temporalio.Workflows; [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync(string name) { var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } ``` Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. ### Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. The `[WorkflowInit]` attribute gives message handlers access to Workflow input. When you use the `[WorkflowInit]` attribute on your constructor, you give the constructor the same Workflow parameters as your `[WorkflowRun]` method. The SDK will then ensure that your constructor receives the Workflow input arguments that the Client sent. The Workflow input arguments are also passed to your `[WorkflowRun]` method. That always happens, whether or not you use the `[WorkflowInit]` attribute. Here's an example. The constructor and `RunAsync` must have the same parameters with the same types: ```csharp [Workflow] public class WorkflowInitWorkflow { public record Input(string Name); private readonly string nameWithTitle; private bool titleHasBeenChecked; [WorkflowInit] public WorkflowInitWorkflow(Input input) => nameWithTitle = $"Knight {input.Name}"; [WorkflowRun] public async Task RunAsync(Input ignored) { await Workflow.WaitConditionAsync(() => titleHasBeenChecked); return $"Hello, {nameWithTitle}"; } } ``` ## Workflow logic requirements Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow. This means there are several things Workflows shouldn't do such as: - Perform IO (network, disk, stdio, etc) - Access/alter external mutable state - Do any threading - Do anything using the system clock (for example, `DateTime.Now`) - This includes .NET timers (for example, `Task.Delay` or `Thread.Sleep`) - Make any random calls - Make any not-guaranteed-deterministic calls (for example, iterating over a dictionary) The SDK provides replay-safe alternatives for common needs. ### Logging Use [`Workflow.Logger`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Logger) instead of `Console.WriteLine` or a logger you resolve yourself. The SDK logger appends Workflow details to every log and skips logging during replay: ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync(string name) { Workflow.Logger.LogInformation("Starting workflow for {Name}", name); // ... } } ``` For logger configuration, see [Observability: Log from a Workflow](/develop/dotnet/platform/observability#logging). ### Random numbers and GUIDs Use [`Workflow.Random`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Random) to get a deterministic random instance seeded per Workflow Execution, and [`Workflow.NewGuid()`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_NewGuid) instead of `Guid.NewGuid()`: ```csharp var value = Workflow.Random.Next(1, 100); var uniqueId = Workflow.NewGuid(); ``` ### Current time Use [`Workflow.UtcNow`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_UtcNow) instead of `DateTime.Now` or `DateTime.UtcNow`. It returns the time of the last Workflow Task, which is consistent across replays: ```csharp var currentTime = Workflow.UtcNow; ``` To wait, use `Workflow.DelayAsync` instead of `Task.Delay` or `Thread.Sleep`. ### Detecting replay (advanced) Use [`Workflow.Unsafe.IsReplaying`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplaying) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an [Interceptor](/develop/dotnet/workers/interceptors). > **⚠️ Caution:** > > Never use this to affect Workflow business logic. Branching on replay status breaks determinism. > ```csharp if (!Workflow.Unsafe.IsReplaying) { EmitMetric("workflow_started", 1); } ``` If your goal is to always take action when something new is happening, check that [`Workflow.Unsafe.IsReplayingHistoryEvents`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.Unsafe.html#Temporalio_Workflows_Workflow_Unsafe_IsReplayingHistoryEvents) is false instead. That is false during read-only operations like Queries and Update validators. This is what the SDK's built-in logger and tracing interceptors use internally. ### .NET Task Determinism Some calls in .NET do unsuspecting non-deterministic things and are easy to accidentally use. This is especially true with `Task`s. Temporal requires that the deterministic `TaskScheduler.Current` is used, but many .NET async calls will use `TaskScheduler.Default` implicitly (and some analyzers even encourage this). Here are some known gotchas to avoid with .NET tasks inside of Workflows: - Use `Workflow.RunTaskAsync` instead of `Task.Run`. `Task.Run` uses the default scheduler and puts work on the thread pool. - You can also use `Task.Factory.StartNew` with current scheduler or instantiate the `Task` and run `Task.Start` on it. - If you need to use `Task.ConfigureAwait`, use `Task.ConfigureAwait(true)`. `Task.ConfigureAwait(false)` won't use the current context. - There is no significant performance benefit to `Task.ConfigureAwait` in workflows because of how the scheduler works. - Avoid anything that defaults to the default task scheduler. - Use `Workflow.DelayAsync`, `Workflow.WaitConditionAsync`, or non-timeout-based cancellation token sources instead of `Task.Delay`, `Task.Wait`, timeout-based `CancellationTokenSource`, or anything that uses .NET built-in timers. - Use `Workflow.WhenAnyAsync` instead of `Task.WhenAny`. - Technically this only applies to an enumerable set of tasks with results or more than 2 tasks with results. Other uses are safe. See [this issue](https://github.com/dotnet/runtime/issues/87481). - Use `Workflow.WhenAllAsync` instead of `Task.WhenAll`. - Technically `Task.WhenAll` is currently deterministic in .NET and safe, but it is better to use the wrapper to be sure. - Use `CancellationTokenSource.Cancel` instead of `CancellationTokenSource.CancelAsync`. - Use `Temporalio.Workflows.Semaphore` or `Temporalio.Workflows.Mutex` instead of `System.Threading.Semaphore`, `System.Threading.SemaphoreSlim`, or `System.Threading.Mutex`. - _Technically_ `SemaphoreSlim` does work if only the async form of `WaitAsync` is used without no timeouts and `Release` is used. But anything else can deadlock the workflow and its use is cumbersome since it must be disposed. - Be wary of additional libraries' implicit use of the default scheduler. - For example, while there are articles for `Dataflow` about [using a specific scheduler](https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-specify-a-task-scheduler-in-a-dataflow-block), there are hidden implicit uses of `TaskScheduler.Default`. For example, see [this bug](https://github.com/dotnet/runtime/issues/83159). In order to help catch wrong scheduler use, by default the Temporal .NET SDK adds an event source listener for info-level task events. While this technically receives events from all uses of tasks in the process, we make sure to ignore anything that is not running in a Workflow in a high performant way (basically one thread local check). For code that does run in a Workflow and accidentally starts a task in another scheduler, an `InvalidWorkflowOperationException` will be thrown which "pauses" the Workflow (fails the Workflow Task which continually retries until the code is fixed). This is unfortunately a runtime-only check, but can help catch mistakes early. If this needs to be turned off for any reason, set `DisableWorkflowTracingEventListener` to `true` in Worker options. In the near future for modern .NET versions we hope to use the [new `TimeProvider` API](https://github.com/dotnet/runtime/issues/36617) which will allow us to control current time and timers. ### Workflow .editorconfig Since Workflow code follows some different logic rules than regular C# code, there are some common analyzer rules that developers may want to disable. To ensure these are only disabled for Workflows, current recommendation is to use the `.workflow.cs` extension for files containing Workflows. Here are the rules to disable: - [CA1024](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1024) - This encourages properties instead of methods that look like getters. However for reflection reasons we cannot use property getters for queries, so it is very normal to have ```csharp [WorkflowQuery] public string GetSomeThing() => someThing; ``` - [CA1822](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1822) - This encourages static methods when methods don't access instance state. Workflows however use instance methods for run, Signals, Queries, or Updates even if they could be static. - [CA2007](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2007) - This encourages users to use `ConfigureAwait` instead of directly waiting on a task. But in Workflows, there is no benefit to this and it just adds noise (and if used, needs to be `ConfigureAwait(true)` not `ConfigureAwait(false)`). - [CA2008](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2008) - This encourages users to always apply an explicit task scheduler because the default of `TaskScheduler.Current` is bad. But for Workflows, the default of `TaskScheduler.Current` is good/required. - [CA5394](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca5394) - This discourages use of non-crypto random. But deterministic Workflows, via `Workflow.Random` intentionally provide a deterministic non-crypto random instance. - `CS1998` - This discourages use of `async` on async methods that don't `await`. But Workflows handlers like Signals are often easier to write in one-line form this way, for example `public async Task SignalSomethingAsync(string value) => this.value = value;`. - [VSTHRD105](https://github.com/microsoft/vs-threading/blob/main/doc/analyzers/VSTHRD105.md) - This is similar to `CA2008` above in that use of implicit current scheduler is discouraged. That does not apply to Workflows where it is encouraged/required. Here is the `.editorconfig` snippet for the above which may frequently change as more analyzers need to be adjusted: ```ini ##### Configuration specific for Temporal workflows ##### [*.workflow.cs] # We use getters for queries, they cannot be properties dotnet_diagnostic.CA1024.severity = none # Don't force workflows to have static methods dotnet_diagnostic.CA1822.severity = none # Do not need ConfigureAwait for workflows dotnet_diagnostic.CA2007.severity = none # Do not need task scheduler for workflows dotnet_diagnostic.CA2008.severity = none # Workflow randomness is intentionally deterministic dotnet_diagnostic.CA5394.severity = none # Allow async methods to not have await in them dotnet_diagnostic.CS1998.severity = none # Don't avoid, but rather encourage things using TaskScheduler.Current in workflows dotnet_diagnostic.VSTHRD105.severity = none ``` ### Customize Workflow Type Workflows have a Type that are referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. You can customize the Workflow name with a custom name in the attribute. For example, `[Workflow("my-workflow-name")]`. If the name parameter is not specified, the Workflow name defaults to the unqualified class name. ```csharp using Temporalio.Workflows; [Workflow("MyDifferentWorkflowName")] public class MyWorkflow { public async Task RunAsync(string name) { var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } ``` --- # Cancellation - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/cancellation This page shows how to interrupt a Workflow Execution. You can interrupt a Workflow Execution in one of the following ways: - [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution. - [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. ## Cancellation To give a Workflow and its Activities the ability to be cancelled, do the following: - Handle a Cancellation request within a Workflow. - Set Activity Heartbeat Timeouts. - Listen for and handle a Cancellation request within an Activity. - Send a Cancellation request from a Temporal Client. ### Handle Cancellation in Workflow Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on Cancellation to perform cleanup. Cancellation Requests on Workflows cancel the `Workflow.CancellationToken`. This is the token that is implicitly used for all calls within the workflow as well (for example, Timers, Activities, etc) and therefore cancellation is propagated to them to be handled and bubble out. ```csharp [WorkflowRun] public async Task RunAsync() { try { // Whether this workflow waits on the activity to handle the cancellation or not is // dependent upon the CancellationType option. We leave the default here which sends the // cancellation but does not wait on it to be handled. await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyNormalActivity(), new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5) }); } catch (Exception e) when (TemporalException.IsCanceledException(e)) { // The "when" clause above is because we only want to apply the logic to cancellation, but // this kind of cleanup could be done on any/all exceptions too. Workflow.Logger.LogError(e, "Cancellation occurred, performing cleanup"); // Call cleanup activity. If this throws, it will swallow the original exception which we // are ok with here. This could be changed to just log a failure and let the original // cancellation continue. // The default token on Workflow.CancellationToken is now marked // cancelled, so we pass a different one. We use CancellationToken.None here because the // cleanup activity itself doesn't need to be cancellable; if it did (e.g. you want to // cancel cleanup from a timeout or another signal), create a new detached // CancellationTokenSource and pass its Token instead. await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyCancellationCleanupActivity(), new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5), CancellationToken = CancellationToken.None, }); // Rethrow the cancellation throw; } } ``` ### Handle Cancellation in an Activity Ensure that the Activity is [Heartbeating](/develop/dotnet/activities/timeouts#activity-heartbeats) to receive the Cancellation request and stop execution. Also make sure that the [Heartbeat Timeout](/develop/dotnet/activities/timeouts#heartbeat-timeout) is set on the Activity Options when calling from the Workflow. An Activity Cancellation Request cancels the `CancellationToken` on the `ActivityExecutionContext`. ```csharp [Activity] public async Task MyActivityAsync() { // This is a naive loop simulating work, but similar heartbeat/cancellation logic applies to // other scenarios as well while (true) { // Send heartbeat ActivityExecutionContext.Current.Heartbeat(); // Do some work, passing the cancellation token await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken); } } ``` ### Request Cancellation Use `CancelAsync` on the `WorkflowHandle` to cancel a Workflow Execution. ```csharp // Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID. // This could also just be a handle that is returned from StartWorkflowAsync instead. var handle = myClient.GetWorkflowHandle("my-workflow-id"); // Send cancellation. This returns when cancellation is received by the server. Wait on the handle's // result to wait for cancellation to be applied. await handle.CancelAsync(); ``` #### How to request Cancellation of an Activity By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation token is used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created. ```csharp [WorkflowRun] public async Task RunAsync() { // Create a source linked to workflow cancellation. A new source could be created instead if we // didn't want it associated with workflow cancellation. using var cancelActivitySource = CancellationTokenSource.CreateLinkedTokenSource( Workflow.CancellationToken); // Start the activity. Whether this workflow waits on the activity to handle the cancellation // or not is dependent upon the CancellationType option. We leave the default here which sends // the cancellation but does not wait on it to be handled. var activityTask = Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyNormalActivity(), new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5), CancellationToken = cancelActivitySource.Token; }); activityTask.Start(); // Wait 5 minutes, then cancel it await Workflow.DelayAsync(TimeSpan.FromMinutes(5)); cancelActivitySource.Cancel(); // Wait on the activity which will throw cancellation which will fail the workflow await activityTask; } ``` ## Termination To Terminate a Workflow Execution in .NET, use the [TerminateAsync()](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_TerminateAsync_System_String_Temporalio_Client_WorkflowTerminateOptions_) method on the Workflow handle. ```csharp // Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID. // This could also just be a handle that is returned from StartWorkflowAsync instead. var handle = myClient.GetWorkflowHandle("my-workflow-id"); // Terminate await handle.TerminateAsync(); ``` Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the UI when that happens. ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/child-workflows > Start a Child Workflow Execution and set a Parent Close Policy using Temporal .NET SDK. Discover methods like ExecuteChildWorkflowAsync and manage Workflow behaviors. This page shows how to do the following: - [Start a Child Workflow Execution](#child-workflows) - [Set a Parent Close Policy](#parent-close-policy) ## Start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In .NET, awaiting `StartChildWorkflowAsync()` or `ExecuteChildWorkflowAsync()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves. If you start a Child Workflow from a non-main context (for example, a Signal or Update handler), make sure the Parent Workflow doesn't complete before that call resolves. To spawn a Child Workflow Execution in .NET, use the `ExecuteChildWorkflowAsync()` method which starts the Child Workflow and waits for completion or use the `StartChildWorkflowAsync()` method to start a Child Workflow and return its handle. This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running. > **📝 Note:** > > `ExecuteChildWorkflowAsync()` is a helper method for `StartChildWorkflowAsync()` plus `await handle.GetResultAsync()`. > ```csharp await Workflow.ExecuteChildWorkflowAsync((MyChildWorkflow wf) => wf.RunAsync()); ``` ## Set a Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. Set the `ParentClosePolicy` property inside the [`ChildWorkflowOptions`](https://dotnet.temporal.io/api/Temporalio.Workflows.ChildWorkflowOptions.html) for `ExecuteChildWorkflowAsync` or `StartChildWorkflowAsync` to specify the behavior of the Child Workflow when the Parent Workflow closes. ```csharp await Workflow.ExecuteChildWorkflowAsync( (MyChildWorkflow wf) => wf.RunAsync(), new() { ParentClosePolicy = ParentClosePolicy.Abandon }); ``` --- # Continue-As-New - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/continue-as-new > Use Temporal's Continue-As-New in .NET to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for .NET developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) - [How to test Continue-as-New?](#how-to-test) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the .NET SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically set to `None` for the original caller of the Workflow. [View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code. ```csharp public record Input { public State State { get; init; } = new(); public bool TestContinueAsNew { get; init; } } [WorkflowInit] public ClusterManagerWorkflow(Input input) ```` The test hook in the above snippet is covered [below](#how-to-test). Inside your Workflow, throw a [`CreateContinueAsNewException`](https://dotnet.temporal.io/api/Temporalio.Workflows.ContinueAsNewException.html) exception. This stops the Workflow right away and starts a new one. [View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code. ```csharp throw Workflow.CreateContinueAsNewException((ClusterManagerWorkflow wf) => wf.RunAsync(new() { State = CurrentState, TestContinueAsNew = input.TestContinueAsNew, })); ```` ### Considerations for Workflows with Message Handlers If you use Updates or Signals, don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish in your main Workflow before you throw `CreateContinueAsNewException`. See the [`AllHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance. ## When is it right to Continue-as-New using the .NET SDK? Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `Workflow.ContinueAsNewSuggested` to check if it's time. ## How to test Continue-as-New using the .NET SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, when `TestContinueAsNew == true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value. A helper variable in the Workflow checks it each time it considers using Continue-as-New: [View the source code](https://github.com/temporalio/samples-dotnet/blob/main/src/SafeMessageHandlers/ClusterManagerWorkflow.workflow.cs) in the context of the rest of the application code. ```csharp private bool ShouldContinueAsNew => // Don't continue as new while update running Workflow.AllHandlersFinished && // Continue if suggested or, for ease of testing, max history reached (Workflow.ContinueAsNewSuggested || Workflow.CurrentHistoryLength > maxHistoryLength); ``` --- # Dynamic Workflow - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/dynamic-workflow > This section explains Dynamic Workflows with the .NET SDK ## Set a Dynamic Workflow **How to set a Dynamic Workflow using the Temporal .NET SDK** A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. A Workflow can be made dynamic by setting `Dynamic` as `true` on the `[Workflow]` attribute. You must register the Workflow with the Worker before it can be invoked. Only one Dynamic Workflow can be present on a Worker. The Workflow Definition must then accept a single argument of type `Temporalio.Converters.IRawValue[]`. The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` namespace. ```csharp [Workflow(Dynamic = true)] public class DynamicWorkflow { [WorkflowRun] public async Task RunAsync(IRawValue[] args) { var name = Workflow.PayloadConverter.ToValue(args.Single()); var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } ``` --- # Message passing - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/message-passing > Develop with Queries, Signals, and Updates with the Temporal .NET SDK. A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates. The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values. Temporal Clients use messages to read Workflow state and control execution. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal .NET SDK. ## Write message handlers > **ℹ️ Info:** > The code that follows is part of a [working solution](https://github.com/temporalio/samples-dotnet/tree/main/src/SafeMessageHandlers). Follow these guidelines when writing your message handlers: - Message handlers are defined as methods on the Workflow class, using one of the three attributes: [`WorkflowQueryAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowQueryAttribute.html), [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html), and [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html). - The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion). - Prefer data classes to multiple input parameters. Data class parameters allow you to add fields without changing the calling signature. Keep in mind that serialization and deserialization can fail with the default data converter if the new field does not have a default value. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution. Define as a method: ```csharp [Workflow] public class GreetingWorkflow { public enum Language { Chinese, English, French, Spanish, Portuguese, } public record GetLanguagesInput(bool IncludeUnsupported); // ... [WorkflowQuery] public IList GetLanguages(GetLanguagesInput input) => Enum.GetValues(). Where(language => input.IncludeUnsupported || Greetings.ContainsKey(language)). ToList(); // ... ``` Or as a property getter: ```csharp [Workflow] public class GreetingWorkflow { public enum Language { Chinese, English, French, Spanish, Portuguese, } // ... [WorkflowQuery] public Language CurrentLanguage { get; private set; } = Language.English; // ... ``` - The Query attribute can accept arguments. See the API reference docs: [`WorkflowQueryAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowQueryAttribute.html). - A Query handler must not modify Workflow state. - You can't perform async blocking operations such as executing an Activity in a Query handler. ### Signal handlers A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow: ```csharp [Workflow] public class GreetingWorkflow { public record ApproveInput(string Name); // ... [WorkflowSignal] public async Task ApproveAsync(ApproveInput input) { approvedForRelease = true; approverName = input.Name; } // ... ``` - The Signal attribute can accept arguments. Refer to the API docs: [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html). - The handler should not return a value. The response is sent immediately from the server, without waiting for the Workflow to process the Signal. - Signal (and Update) handlers can be asynchronous and blocking. This allows you to use Activities, Child Workflows, durable [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers. ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```csharp [Workflow] public class GreetingWorkflow { public enum Language { Chinese, English, French, Spanish, Portuguese, } // ... [WorkflowUpdateValidator(nameof(SetCurrentLanguageAsync))] public void ValidateLanguage(Language language) { if (!Greetings.ContainsKey(language)) { throw new ApplicationFailureException($"{language} is not supported"); } } [WorkflowUpdate] public async Task SetCurrentLanguageAsync(Language language) { var previousLanguage = CurrentLanguage; CurrentLanguage = language; return previousLanguage; } // ... ``` - The Update attribute can take arguments (like, `Name`, `Dynamic` and `UnfinishedPolicy`) as described in the API reference docs for [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html). - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them. - Define an Update validator with the [`WorkflowUpdateValidatorAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateValidatorAttribute.html) attribute. Use the Name argument when declaring the validator to connect it to its Update. The validator must be a `void` type and accept the same argument types as the handler. - Accepting and rejecting Updates with validators: - To reject an Update, raise an exception of any type in the validator. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic. - When a Validator raises an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History. The caller receives an "Update failed" error. - Use [`Workflow.CurrentUpdateInfo`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_CurrentUpdateInfo) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - Update (and Signal) handlers can be asynchronous and blocking. This allows you to use Activities, Child Workflows, durable [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Update and Signal handlers. ## Send messages To send Queries, Signals, or Updates you call methods on a [`WorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html) object. To obtain the WorkflowStub, you can: - Use [`TemporalClient.StartWorkflowAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_StartWorkflowAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowOptions_) to start a Workflow and return its handle. - Use the [`TemporalClient.GetWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_GetWorkflowHandle_System_String_System_String_System_String_) method to retrieve a Workflow handle by its Workflow Id. For example: ```csharp var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var workflowHandle = await client.StartWorkflowAsync( (GreetingWorkflow wf) => wf.RunAsync(), new(id: "message-passing-workflow-id", taskQueue: "message-passing-sample")); ``` To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition. > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution. > ### Send a Query Call a Query method with [`WorkflowHandle.QueryAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_QueryAsync__1_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowQueryOptions_): ```csharp var supportedLanguages = await workflowHandle.QueryAsync(wf => wf.GetLanguages(new(false))); ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### Send a Signal from a Client Use [`WorkflowHandle.SignalAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_SignalAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowSignalOptions_) from Client code to send a Signal: ```csharp await workflowHandle.SignalAsync(wf => wf.ApproveAsync(new("MyUser"))); ``` - The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution. - The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History. #### Send a Signal from a Workflow A Workflow can send a Signal to another Workflow, known as an _External Signal_. In this case you need to obtain a Workflow handle for the external Workflow. Use `Workflow.GetExternalWorkflowHandle`, passing a running Workflow Id, to retrieve a typed Workflow handle: ```csharp // ... [Workflow] public class WorkflowB { [WorkflowRun] public async Task RunAsync() { var handle = Workflow.GetExternalWorkflowHandle("workflow-a"); await handle.SignalAsync(wf => wf.YourSignalAsync("signal argument")); } // ... ``` When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. #### Signal-With-Start Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running. If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled. To use Signal-With-Start, call `SignalWithStart` with a lambda expression invoking it: ```csharp var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var options = new WorkflowOptions(id: "your-signal-with-start-workflow", taskQueue: "signal-tq"); options.SignalWithStart((GreetingWorkflow wf) => wf.SubmitGreetingAsync("User Signal with Start")); await client.StartWorkflowAsync((GreetingWorkflow wf) => wf.RunAsync(), options); ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A Client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. To send an Update to a Workflow Execution, you can: - Call the Update method with `ExecuteUpdateAsync` from the Client and wait for the Update to complete. This code fetches an Update result: ```csharp var previousLanguage = await workflowHandle.ExecuteUpdateAsync( wf => wf.SetCurrentLanguageAsync(GreetingWorkflow.Language.Chinese)); ``` 2. Use `StartUpdateAsync` to receive a handle as soon as the Update is accepted. It returns an `UpdateHandle` - Use this `UpdateHandle` later to fetch your results. - Asynchronous Update handlers normally perform long-running async Activities. - `StartUpdateAsync` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete. For example: ```csharp // Wait until the update is accepted var updateHandle = await workflowHandle.StartUpdateAsync( wf => wf.SetGreetingAsync(new HelloWorldInput("World")), new(waitForStage: WorkflowUpdateStage.Accepted)); // Wait until the update is completed var updateResult = await updateHandle.GetResultAsync(); ``` For more details, see the "Async handlers" section. #### Update-with-Start > **💡 Tip:** > > For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended. > [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/dotnet/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. Use `ExecuteUpdateWithStartAsync` to start the Update and wait for the result in one go. Alternatively, use `StartUpdateWithStartAsync` to start the Update and receive a `WorkflowUpdateHandle`, and then use `await updateHandle.GetResultAsync()` to retrieve the result from the Update. These calls return once the requested Update wait stage has been reached, or when the request times out. - You will need to provide a `WithStartWorkflowOperation` to define the Workflow that will be started if necessary, and its arguments. - You must specify an [IdConflictPolicy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`. Note that a `WithStartWorkflowOperation` can only be used once. Here's an example taken from the [UpdateWithStartLazyInit](https://github.com/temporalio/samples-dotnet/blob/main/src/UpdateWithStartLazyInit/Program.cs) sample: ```csharp async Task AddCartItemAsync(string sessionId, ShoppingCartItem item) { // Issue an update-with-start that will create the workflow if it does not // exist before attempting the update // Create the start operation var startOperation = WithStartWorkflowOperation.Create( (ShoppingCartWorkflow wf) => wf.RunAsync(), new(id: $"cart-{sessionId}", taskQueue: TaskQueue) { IdConflictPolicy = Temporalio.Api.Enums.V1.WorkflowIdConflictPolicy.UseExisting, }); // Issue the update-with-start, swallowing item-unavailable failure decimal? subtotal; try { subtotal = await client.ExecuteUpdateWithStartWorkflowAsync( (ShoppingCartWorkflow wf) => wf.AddItemAsync(item), new(startOperation)); } catch (WorkflowUpdateFailedException e) when ( e.InnerException is ApplicationFailureException appErr && appErr.ErrorType == "ItemUnavailable") { // Set subtotal to null if item was not found subtotal = null; } return new(await startOperation.GetHandleAsync(), subtotal); } ``` > **ℹ️ Info:** > NON-TYPE SAFE API CALLS > > In real-world development, sometimes you may be unable to import Workflow Definition method signatures. > When you don't have access to the Workflow Definition or it isn't written in .NET, you can still use non-type safe APIs and dynamic method invocation. > Pass method names instead of method objects to: > > - [`TemporalClient.StartWorkflowAsync`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_StartWorkflowAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowOptions_) > - [`WorkflowHandle.QueryAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_QueryAsync__1_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowQueryOptions_) > - [`WorkflowHandle.SignalAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_SignalAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowSignalOptions_) > - [`WorkflowHandle.ExecuteUpdateAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_ExecuteUpdateAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowUpdateOptions_) > - [`WorkflowHandle.StartUpdateAsync`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_StartUpdateAsync_System_String_System_Collections_Generic_IReadOnlyCollection_System_Object__Temporalio_Client_WorkflowUpdateStartOptions_) > > Use non-type safe overloads of these APIs: > > - [`TemporalClient.GetWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Client.TemporalClient.html#Temporalio_Client_TemporalClient_GetWorkflowHandle_System_String_System_String_System_String_) > - [`Workflow.GetExternalWorkflowHandle`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_GetExternalWorkflowHandle_System_String_System_String_) > ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow) and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). > ### Add async handlers to use `await` Signal and Update handlers can be asynchronous as well as blocking. Using asynchronous calls allows you to `await` Activities, Child Workflows, [`Workflow.DelayAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_DelayAsync_System_Int32_System_Nullable_System_Threading_CancellationToken__) Timers, [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) wait conditions, etc. This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls. It's essential to understand the things that could go wrong in order to use asynchronous handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below. The following code executes an Activity that simulates a network call to a remote service: ```csharp public class MyActivities { private static readonly Dictionary Greetings = new() { [Language.Arabic] = "مرحبا بالعالم", [Language.Chinese] = "你好，世界", [Language.English] = "Hello, world", [Language.French] = "Bonjour, monde", [Language.Hindi] = "नमस्ते दुनिया", [Language.Spanish] = "Hola mundo", }; [Activity] public async Task CallGreetingServiceAsync(Language language) { // Pretend that we are calling a remove service await Task.Delay(200); return Greetings.TryGetValue(language, out var value) ? value : null; } } ``` The following code modifies a `WorkflowUpdate` for asynchronous use of the preceding Activity: ```csharp [Workflow] public class GreetingWorkflow { private readonly Mutex mutex = new(); // ... [WorkflowUpdate] public async Task SetLanguageAsync(Language language) { // 👉 Use a mutex here to ensure that multiple calls to SetLanguageAsync are processed in order. await mutex.WaitOneAsync(); try { if (!greetings.ContainsKey(language)) { var greeting = Workflow.ExecuteActivityAsync( (MyActivities acts) => acts.CallGreetingServiceAsync(language), new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) }); if (greeting == null) { // 👉 An update validator cannot be async, so cannot be used to check that the remote // CallGreetingServiceAsync supports the requested language. Throwing ApplicationFailureException // will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be // added to history. throw new ApplicationFailureException( $"Greeting service does not support {language}"); } greetings[language] = greeting; } var previousLanguage = CurrentLanguage; CurrentLanguage = language; return previousLanguage; } finally { mutex.ReleaseMutex(); } } } ``` After updating the code for asynchronous calls, your Update handler can schedule an Activity and await the result. Although an async Signal handler can initiate similar network tasks, using an Update handler allows the Client to receive a result or error once the Activity completes. This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, etc. ### Add wait conditions to block Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. Using a wait condition with [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) sets a function that prevents the code from proceeding until the condition returns `true`. This is an important feature that helps you control your handler logic. Here are two important use cases for `Workflow.WaitConditionAsync`: - Waiting in a handler until it is appropriate to continue. - Waiting in the main Workflow until all active handlers have finished. The condition state you're waiting for can be updated by and reflect any part of the Workflow code. This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method, and so forth. #### Use wait conditions in handlers Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. Using a wait condition with [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) sets a function that prevents the code from proceeding until the condition returns `true`. This is an important feature that helps you control your handler logic. Consider a `ReadyForUpdateToExecute` method that runs before your Update handler executes. The [`Workflow.WaitConditionAsync`](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html?#Temporalio_Workflows_Workflow_WaitConditionAsync_System_Func_System_Boolean__System_Int32_System_Nullable_System_Threading_CancellationToken__) call waits until your condition is met: ```csharp [WorkflowUpdate] public async Task MyUpdateAsync(UpdateInput updateInput) { await Workflow.WaitConditionAsync(() => ReadyForUpdateToExecute(updateInput)); // ... } ``` Remember: Handlers can execute before the main Workflow method starts. #### Ensure your handlers finish before the Workflow completes Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result. The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying retrieve Update results. Use `Workflow.AllHandlersFinished` to address this problem and allow your Workflow to end smoothly: ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { // ... await Workflow.WaitConditionAsync(() => Workflow.AllHandlersFinished); return "workflow-result"; } // ... ``` By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions. You can silence these warnings on a per-handler basis by passing the `UnfinishedPolicy` argument to the [`WorkflowSignalAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowSignalAttribute.html) / [`WorkflowUpdateAttribute`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowUpdateAttribute.html) decorator: ```csharp [WorkflowUpdate(UnfinishedPolicy = HandlerUnfinishedPolicy.Abandon)] public async Task MyUpdateAsync() { // ... ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. ### Use `[WorkflowInit]` to operate on Workflow input before any handler executes The `[WorkflowInit]` attribute gives message handlers access to [Workflow input](/handling-messages#workflow-initializers). When you use the `[WorkflowInit]` attribute on your constructor, you give the constructor the same Workflow parameters as your `[WorkflowRun]` method. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/dotnet/client/temporal-client#start-workflow). The Workflow input arguments are also passed to your `[WorkflowRun]` method -- that always happens, whether or not you use the `[WorkflowInit]` attribute. Here's an example. The constructor and `RunAsync` must have the same parameters with the same types: ```csharp [Workflow] public class WorkflowInitWorkflow { public record Input(string Name); private readonly string nameWithTitle; private bool titleHasBeenChecked; [WorkflowInit] public WorkflowInitWorkflow(Input input) => nameWithTitle = $"Sir {input.Name}"; [WorkflowRun] public async Task RunAsync(Input ignored) { await Workflow.WaitConditionAsync(() => titleHasBeenChecked); return $"Hello, {nameWithTitle}"; } [WorkflowUpdate] public async Task CheckTitleValidityAsync() { // The handler is now guaranteed to see the workflow input after it has // been processed by the constructor. var valid = await Workflow.ExecuteActivityAsync( (MyActivities acts) -> acts.CheckTitleValidityAsync(nameWithTitle), new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) }); titleHasBeenChecked = true; return valid; } } ``` ### Use locks to prevent concurrent handler execution Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```csharp [Workflow] public class MyWorkflow { // ... [WorkflowSignal] public async Task BadHandlerAsync() { var data = await Workflow.ExecuteActivityAsync( (MyActivities acts) => acts.FetchDataAsync(), new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) }); this.x = data.X; // 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then // there may be times when the Workflow has this.x from one Activity execution and this.y from another. await Workflow.DelayAsync(1000); this.y = data.Y; } } ``` Coordinating access with [`Workflows.Mutex`](https://dotnet.temporal.io/api/Temporalio.Workflows.Mutex.html), a mutual exclusion lock, corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```csharp [Workflow] public class MyWorkflow { private readonly Mutex mutex = new(); // ... [WorkflowSignal] public async Task SafeHandlerAsync() { await mutex.WaitOneAsync(); try { var data = await Workflow.ExecuteActivityAsync( (MyActivities acts) => acts.FetchDataAsync(), new() { StartToCloseTimeout = TimeSpan.FromSeconds(10) }); this.x = data.X; // ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow // method, but no other execution of this handler can run until this execution finishes. await Workflow.DelayAsync(1000); this.y = data.Y; } finally { mutex.ReleaseMutex(); } } } ``` For additional concurrency options, you can use [`Workflows.Semaphore`](https://dotnet.temporal.io/api/Temporalio.Workflows.Semaphore.html). Semaphores manage access to shared resources and coordinate the order in which threads or processes execute. ## Message handler troubleshooting When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The Client can't contact the server**: You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception whose `Code` property is [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `Unavailable` (after some retries). - **The Workflow does not exist**: You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception whose `Code` property is [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `NotFound`. See [Exceptions in message handlers](/handling-messages#exceptions) for a non–.NET-specific discussion of this topic. ### Problems when sending a Signal When using Signal, the only exception that will result from your requests during its execution is `RpcException`. All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle. For Queries and Updates, the Client waits for a response from the Worker. If an issue occurs during the handler Execution by the Worker, the Client may receive an exception. ### Problems when sending an Update When working with Updates, you may encounter these errors: - **No Workflow Workers are polling the Task Queue**: Your request will be retried by the SDK Client indefinitely. Use a `CancellationToken` in your [RPC options](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowUpdateOptions.html#Temporalio_Client_WorkflowUpdateOptions_Rpc) to cancel the Update. This raises a [Temporalio.Exceptions.WorkflowUpdateRpcTimeoutOrCanceledException](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowUpdateRpcTimeoutOrCanceledException.html) exception. - **Update failed**: You'll receive a [`Temporalio.Exceptions.WorkflowUpdateFailedException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowUpdateFailedException.html) exception. There are two ways this can happen: - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler. - The Update failed after having been accepted. Update failures are like [Workflow failures](/references/failures). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity (if the Activity retries have been set to a finite number) - The Workflow author raising `ApplicationFailure` - Any error listed in [`TemporalWorkerOptions.WorkflowFailureExceptionTypes`](https://dotnet.temporal.io/api/Temporalio.Worker.TemporalWorkerOptions.html#Temporalio_Worker_TemporalWorkerOptions_WorkflowFailureExceptionTypes) on the Worker or [`WorkflowAttribute.FailureExceptionTypes`](https://dotnet.temporal.io/api/Temporalio.Workflows.WorkflowAttribute.html#Temporalio_Workflows_WorkflowAttribute_FailureExceptionTypes) on the Workflow (empty by default) - **The handler caused the Workflow Task to fail**: A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) exception. - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowUpdateHandle.html) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) "workflow execution already completed". This will happen if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Problems when sending a Query When working with Queries, you may encounter these errors: - **There is no Workflow Worker polling the Task Queue**: You'll receive a [`Temporalio.Exceptions.RpcException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.html) on which the `Code` is a [`RpcException.StatusCode`](https://dotnet.temporal.io/api/Temporalio.Exceptions.RpcException.StatusCode.html) with a status of `FailedPrecondition`. - **Query failed**: You'll receive a [`Temporalio.Exceptions.WorkflowQueryFailedException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.WorkflowQueryFailedException.html) exception if something goes wrong during a Query. Any exception in a Query handler will trigger this error. This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. ## Dynamic Handler Temporal supports Dynamic Queries, Signals, Updates, Workflows, and Activities. These are unnamed handlers that are invoked if no other statically defined handler with the given name exists. Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, Updates, Workflows, or Activities, aren't known at run time. > **⚠️ Caution:** > > Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach. > Overusing them can lead to maintainability and debugging issues down the line. > > Instead, Signals, Queries, Workflows, or Activities should be defined statically whenever possible, with clear names that indicate their purpose. > Use static definitions as the primary way of structuring your Workflows. > > Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime. > They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic. > ### Set a Dynamic Query A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered. A Query can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowQuery]` attribute. Only one Dynamic Query can be present on a Workflow. The Query Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments. The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace. ```csharp [WorkflowQuery(Dynamic = true)] public string DynamicQueryAsync(string queryName, IRawValue[] args) { var input = Workflow.PayloadConverter.ToValue(args.Single()); return statuses[input.Type]; } ``` ### Set a Dynamic Signal A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered. A Signal can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowSignal]` attribute. Only one Dynamic Signal can be present on a Workflow. The Signal Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments. The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace. ```csharp [WorkflowSignal(Dynamic = true)] public async Task DynamicSignalAsync(string signalName, IRawValue[] args) { var input = Workflow.PayloadConverter.ToValue(args.Single()); pendingThings.Add(input); } ``` ### Set a Dynamic Update A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered. An Update can be made dynamic by setting `Dynamic` to `true` on the `[WorkflowUpdate]` attribute. Only one Dynamic Update can be present on a Workflow. The Update Handler parameters must accept a `string` name and `Temporalio.Converters.IRawValue[]` for the arguments. The [Workflow.PayloadConverter](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_PayloadConverter) property is used to convert an `IRawValue` object to the desired type using extension methods in the `Temporalio.Converters` Namespace. ```csharp [WorkflowUpdate(Dynamic = true)] public async Task DynamicUpdateAsync(string updateName, IRawValue[] args) { var input = Workflow.PayloadConverter.ToValue(args.Single()); pendingThings.Add(input); return statuses[input.Type]; } ``` --- # Schedules - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/schedules > Manage and optimize Scheduled Workflows using the Temporal .NET SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and use Start Delay options. This page shows how to do the following: - [Schedule a Workflow](#schedule-a-workflow) - [Create a Scheduled Workflow](#create-a-workflow) - [Backfill a Scheduled Workflow](#backfill-a-scheduled-workflow) - [Delete a Scheduled Workflow](#delete-a-scheduled-workflow) - [Describe a Scheduled Workflow](#describe-a-scheduled-workflow) - [List a Scheduled Workflow](#list-a-scheduled-workflow) - [Pause a Scheduled Workflow](#pause-a-scheduled-workflow) - [Trigger a Scheduled Workflow](#trigger-a-scheduled-workflow) - [Update a Scheduled Workflow](#update-a-scheduled-workflow) - [Use Start Delay](#start-delay) ## Schedule a Workflow Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes. Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process. Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy). If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions. ### Create a Scheduled Workflow The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands. To create a Scheduled Workflow Execution in .NET, use the [CreateScheduleAsync](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_CreateScheduleAsync_System_String_Temporalio_Client_Schedules_Schedule_Temporalio_Client_Schedules_ScheduleOptions_) method on the Client. Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution. Set the Schedule's `Action` property to an instance of `ScheduleActionStartWorkflow` to schedule a Workflow Execution. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = await client.CreateScheduleAsync( "my-schedule-id", new( Action: ScheduleActionStartWorkflow.Create( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue")), Spec: new() { Intervals = new List { new(Every: TimeSpan.FromDays(5)) }, })); ``` > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > ### Backfill a Scheduled Workflow The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time. To backfill a Scheduled Workflow Execution in .NET, use the [BackfillAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_BackfillAsync_System_Collections_Generic_IReadOnlyCollection_Temporalio_Client_Schedules_ScheduleBackfill__Temporalio_Client_RpcOptions_) method on the Schedule Handle. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); var now = DateTime.Now; await handle.BackfillAsync(new List { new( StartAt: now - TimeSpan.FromDays(30), EndAt: now - TimeSpan.FromDays(20), Overlap: ScheduleOverlapPolicy.AllowAll), }); ``` ### Delete a Scheduled Workflow The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule. To delete a Scheduled Workflow Execution in .NET, use the [DeleteAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_DeleteAsync_Temporalio_Client_RpcOptions_) method on the Schedule Handle. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); await handle.DeleteAsync(); ``` ### Describe a Scheduled Workflow The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs. To describe a Scheduled Workflow Execution in .NET, use the [DescribeAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_DescribeAsync_Temporalio_Client_RpcOptions_) method on the Schedule Handle. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); var desc = await handle.DescribeAsync(); Console.WriteLine("Schedule info: {0}", desc.Info); ``` ### List a Scheduled Workflow The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs. To list all schedules, use the [ListSchedulesAsync()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_ListSchedulesAsync_Temporalio_Client_Schedules_ScheduleListOptions_) asynchronous method on the Client. This returns an async enumerable. If a schedule is added or deleted, it may not be available in the list immediately. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); await foreach (var desc in client.ListSchedulesAsync()) { Console.WriteLine("Schedule info: {0}", desc.Info); } ``` ### Pause a Scheduled Workflow The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason. To pause a Scheduled Workflow Execution in .NET, use the [PauseAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_PauseAsync_System_String_Temporalio_Client_RpcOptions_) method on the Schedule Handle. You can pass a note to the `PauseAsync()` method to provide a reason for pausing the schedule. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); await handle.PauseAsync("Pausing the schedule for now"); ``` ### Trigger a Scheduled Workflow The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time. To trigger a Scheduled Workflow Execution in .NET, use the [TriggerAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_TriggerAsync_Temporalio_Client_Schedules_ScheduleTriggerOptions_) method on the Schedule Handle. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); await handle.TriggerAsync(); ``` ### Update a Scheduled Workflow The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval. To update a Scheduled Workflow Execution in .NET, use the [UpdateAsync()](https://dotnet.temporal.io/api/Temporalio.Client.Schedules.ScheduleHandle.html#Temporalio_Client_Schedules_ScheduleHandle_UpdateAsync_System_Func_Temporalio_Client_Schedules_ScheduleUpdateInput_Temporalio_Client_Schedules_ScheduleUpdate__Temporalio_Client_RpcOptions_) method on the Schedule Handle. This method accepts a callback that provides input with the current schedule. A new schedule can be created and returned from that callback to perform the update. ```csharp using Temporalio.Client; using Temporalio.Client.Schedules; var client = await TemporalClient.ConnectAsync(new("localhost:7233")); var handle = client.GetScheduleHandle("my-schedule-id"); await handle.UpdateAsync(input => { var newAction = ScheduleActionStartWorkflow.Create( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue")); return new(input.Description.Schedule with { Action = newAction }); }); ``` ## Use Start Delay Use the `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule. Use the `StartDelay` option on `WorkflowOptions` in either the `StartWorkflowAsync()` or `ExecuteWorkflowAsync()` methods in the Client. ```csharp var handle = await client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue") { StartDelay = TimeSpan.FromHours(3), }); ``` --- # Workflow Timeouts - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/timeouts > Optimize Workflow Execution with Temporal .NET SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts). - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task. These values can be set in the `WorkflowOptions` when calling `StartWorkflowAsync` or `ExecuteWorkflowAsync`. Available timeouts are: - ExecutionTimeout - RunTimeout - TaskTimeout ```csharp var result = await client.ExecuteWorkflowAsync( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue") { WorkflowExecutionTimeout = TimeSpan.FromMinutes(5), }); ``` ### Set Workflow retries A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. The `RetryPolicy` can be set in the `WorkflowOptions` when calling `StartWorkflowAsync` or `ExecuteWorkflowAsync`. ```csharp var result = await client.ExecuteWorkflowAsync( (MyWorkflow wf) => wf.RunAsync(), new(id: "my-workflow-id", taskQueue: "my-task-queue") { RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) }, }); ``` --- # Timers - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/timers > Set a Durable Timer using the Temporal .NET SDK. Pause Workflow execution for days or months. Timers are persisted and highly resource-efficient using Workflow.DelayAsync. This page describes how to set a Durable Timer using the Temporal .NET SDK. A [Durable Timer](/workflow-execution/timers-delays) is used to pause the execution of a Workflow for a specified duration. A Workflow can sleep for days or even months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the Durable Timer call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To add a Timer in a Workflow, use `Workflow.DelayAsync`. This is like a deterministic form of `Task.Delay`. ```csharp // Sleep for 3 days await Workflow.DelayAsync(TimeSpan.FromDays(3)); ``` --- # Versioning - .NET SDK Source: https://docs.temporal.io/develop/dotnet/workflows/versioning > Use the .NET SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Adding a patch A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Suppose you have an initial Workflow version called `PrePatchActivity`: ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PrePatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); // ... } } ``` Now, you want to update your code to run `PostPatchActivity` instead. This represents your desired end state. ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PostPatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); // ... } } ``` The problem is that you cannot deploy `PostPatchActivity` directly until you're certain there are no more running Workflows created using the `PrePatchActivity` code, otherwise you are likely to cause a nondeterminism error. Instead, you'll need to deploy `PostPatchActivity` and use the [Patched](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Patched_System_String_) method to determine which version of the code to execute. Patching is a three step process: 1. Use [Patched](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_Patched_System_String_) to patch in new code and run it alongside the old code. 2. Remove the old code and apply [DeprecatePatch](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_DeprecatePatch_System_String_). 3. Once all old Workflows have left retention, remove `DeprecatePatch`. ### Patching in new code Using `Patched` inserts a marker into the Event History. During replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case, `my-patch`). This ensures you can safely deploy code from `PostPatchActivity` as a "feature flag" alongside the original version (`PrePatchActivity`). ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { if (Workflow.Patched("my-patch")) { this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PostPatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } else { this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PrePatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } // ... } } ``` ### Deprecating patches After all Workflows started with `PrePatchActivity` code have left retention, you can [deprecate the patch](https://dotnet.temporal.io/api/Temporalio.Workflows.Workflow.html#Temporalio_Workflows_Workflow_DeprecatePatch_System_String_). Deprecated patches serve as a bridge between the final stage of the patching process and the final state that no longer has patches. They function similarly to regular patches by adding a marker to the Event History. However, this marker won't cause a replay failure when the Workflow code doesn't produce it. If, during the deployment of `PostPatchActivity`, there are still live Workers running `PrePatchActivity` code and these Workers pick up Workflow histories generated by `PostPatchActivity`, they will safely use the patched branch. ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { Workflow.DeprecatePatch("my-patch") this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PostPatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); // ... } } ``` ### Removing a patch You can safely deploy `PostPatchActivity` once all Workflows labeled my-patch or earlier have left retention, based on the previously mentioned assertion. ```csharp [Workflow] public class MyWorkflow { [WorkflowRun] public async Task RunAsync() { this.result = await Workflow.ExecuteActivityAsync( (MyActivities a) => a.PostPatchActivity(), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); // ... } } ``` Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Detailed Overview of the Patched Function This video provides an overview of how the `patched()` function works:
For a more in-depth explanation, refer to the [Patching](/patching) Encyclopedia entry. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `SayHelloWorkflow` as `SayHelloWorkflowV2`: ```csharp [Workflow] public class SayHelloWorkflow { [WorkflowRun] # this function contains the original code } [Workflow] public class SayHelloWorkflowV2 { [WorkflowRun] # this function contains the updated code } ``` You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types: ```csharp using var worker = new TemporalWorker( client, new TemporalWorkerOptions("greeting-tasks") .AddWorkflow() .AddWorkflow()); ``` The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ### Testing a Workflow for replay safety To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/dotnet/best-practices/testing-suite#replay). --- # Environment configuration Source: https://docs.temporal.io/develop/environment-configuration > Configure Temporal Clients using environment variables and TOML configuration files Temporal CLI and SDKs support configuring a Temporal Client using environment variables and TOML configuration files, rather than setting connection options programmatically in your code. This decouples connection settings from application logic, making it easier to manage different environments such as development, staging, and production without code changes. For a list of all available configuration settings, their corresponding environment variables, and TOML file paths, refer to [Temporal Client Environment Configuration Reference](../references/client-environment-configuration). ## Configuration methods You can configure your client using a TOML file, environment variables, or a combination of both. The configuration is loaded with a specific order of precedence: 1. Environment variables: These have the highest precedence. If an environment variable defines a setting, it will always override any value set in a configuration file. This makes it easy to provide secrets in dynamic environments. 2. TOML configuration file: A TOML file can be used to define one or more configuration profiles. This file is located by checking the following sources in order: 1. The path specified by the `TEMPORAL_CONFIG_FILE` environment variable. 2. The default configuration path for your operating system: - Linux: `~/.config/temporalio/temporal.toml` - macOS: `$HOME/Library/Application Support/temporalio/temporal.toml` - Windows: `%AppData%\temporalio\temporal.toml` ## TOML file configuration You can use configuration profiles to maintain separate configurations within a single file for different environments. The Temporal client uses the `default` profile unless you specify another via the `TEMPORAL_PROFILE` environment variable or in the SDK's load options. If a requested profile doesn't exist, the application will return an error. Here is an example `temporal.toml` file that defines two profiles: `default` for local development and `prod` for production. ```toml # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly # disabled = false # Use certificate files for mTLS client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" # Staging profile with inline certificate data [profile.staging] address = "staging.temporal.example.com:7233" namespace = "staging" [profile.staging.tls] # Example of providing certificate data directly (base64 or PEM format) client_cert_data = """-----BEGIN CERTIFICATE----- MIICertificateDataHere... -----END CERTIFICATE-----""" client_key_data = """-----BEGIN PRIVATE KEY----- MIIPrivateKeyDataHere... -----END PRIVATE KEY-----""" ``` Select a concept to highlight the matching connection settings in a Cloud profile: annotations={[ { label: 'Address', description: 'Temporal Service host and port. For Temporal Cloud, use your Namespace endpoint (for example, your-namespace.account.tmprl.cloud:7233).', lines: [2], }, { label: 'Namespace', description: 'Namespace this Client connects to. On Temporal Cloud, include the account suffix when your tooling expects it.', lines: [3], }, { label: 'API key', description: 'Authenticates the Client to Temporal Cloud. Prefer environment variables for secrets in real deployments; TOML is fine for local profiles.', lines: [4], }, { label: 'TLS', description: 'Optional mTLS settings. TLS is often auto-enabled when an API key or TLS block is present; set certificate paths for mutual TLS.', lines: [6, 7, 8], }, ]} > ```toml [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" [profile.prod.tls] client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" ``` The [`temporal cloud login`](/cli/cloud#interactive-login) command also writes to this file. When you run `temporal cloud login --profile prod`, the OAuth token is stored in the specified profile automatically. Subsequent commands that use that profile read the token from the TOML file to authenticate with Temporal Cloud. ## CLI integration The Temporal CLI tool includes `temporal config` commands that allow you to read and write to the TOML configuration file. This provides a convenient way to manage your connection profiles without manually editing the file. Refer to [Temporal CLI Reference - `temporal config`](../cli/command-reference/config.mdx) for more details. - `temporal config get `: Reads a specific value from the current profile. - `temporal config set `: Sets a property in the current profile. - `temporal config delete `: Deletes a property from the current profile. - `temporal config list`: Lists all available profiles in the config file. These CLI commands directly manipulate the `temporal.toml` file. This differs from the SDKs, which only _read_ from the file and environment at runtime to establish a client connection. You can select a profile for the CLI to use with the `--profile` flag. For example, `temporal --profile prod ...`. Connection flags on other `temporal` commands (for example, `--address`, `--namespace`, `--api-key`) sit above both environment variables and the TOML file in precedence. The full order for the CLI is: 1. Command-line flags 2. Environment variables 3. TOML configuration file (active profile) So `temporal workflow list --address localhost:7233` uses `localhost:7233` even if `TEMPORAL_ADDRESS` or the active profile's `address` sets something else. The following code blocks provide copy-paste-friendly examples for setting up CLI profiles for both local development and Temporal Cloud. **Local + Prod with Cloud API key** This example shows how to set up a default profile for local development and a `prod` profile for Temporal Cloud using an API key. ```bash # (Optional) initialize the default profile for local development temporal config set --prop address --value "localhost:7233" temporal config set --prop namespace --value "default" # Configure a Temporal Cloud profile that authenticates with an API key temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" temporal --profile prod config set --prop namespace --value "." temporal --profile prod config set --prop api_key --value "" ``` **API key + advanced options** This example shows how to set up a more advanced Temporal Cloud profile with TLS overrides and custom gRPC metadata. ```bash # Base API key properties (replace the placeholders) temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" temporal --profile prod config set --prop namespace --value "." temporal --profile prod config set --prop api_key --value "" # Optional TLS overrides (only needed when you must pin certs or tweak SNI) temporal --profile prod config set --prop tls.server_name --value "." temporal --profile prod config set --prop tls.ca_cert_path --value "/path/to/ca.pem" # Optional gRPC metadata for observability or routing temporal --profile prod config set --prop grpc_meta.environment --value "production" temporal --profile prod config set --prop grpc_meta.service-version --value "v1.2.3" ``` ## Load configuration profile and environment variables If you don't specify a profile, the SDKs load the `default` profile and the environment variables. If you haven't set `TEMPORAL_CONFIG_FILE`, the SDKs will look for the configuration file in the default location. Refer to [Configuration methods](#configuration-methods) for the default locations for your operating system. No matter what profile you choose to load, environment variables are always loaded when you use the APIs in the environment configuration package to load Temporal Client connection options. They always take precedence over TOML file settings in the profiles. **Python** To load the `default` profile along with any environment variables in Python, use the `ClientConfigProfile.load()` method from the `temporalio.envconfig` package. ```python {7-8} import asyncio from temporalio.client import Client from temporalio.envconfig import ClientConfigProfile async def main(): # Load the "default" profile from default locations and environment variables. default_profile = ClientConfigProfile.load() connect_config = default_profile.to_client_connect_config() # Connect to the client using the loaded configuration. client = await Client.connect(**connect_config) print(f"✅ Client connected to {client.service_client.config.target_host} in namespace '{client.namespace}'") if __name__ == "__main__": asyncio.run(main()) ``` **Go** To load the `default` profile along with any environment variables in Go, use the `envconfig.MustLoadDefaultClientOptions()` function from the `temporalio.envconfig` package. ```go {13} package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Loads the "default" profile from the standard location and environment variables. c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalf("Failed to create client: %v", err) } defer c.Close() fmt.Printf("✅ Connected to Temporal Service") } ``` **Ruby** To load the `default` profile along with any environment variables in Ruby, use the `EnvConfig::ClientConfig.load_client_connect_options()` method from the `temporalio.env_config` package. ```Ruby {16-18} require 'temporalio/client' require 'temporalio/env_config' def main puts '--- Loading default profile from config.toml ---' # For this sample to be self-contained, we explicitly provide the path to # the config.toml file included in this directory. # By default though, the config.toml file will be loaded from # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). config_file = File.join(__dir__, 'config.toml') # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( config_source: Pathname.new(config_file) ) puts "Loaded 'default' profile from #{config_file}." puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end ``` **.NET** To load the `default` profile along with any environment variables in .NET C#, use the `ClientEnvConfig.LoadClientConnectOptions()` method from the `Temporalio.Client.EnvConfig` package. ```csharp {22,27-30} using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading the default environment configuration profile /// from a TOML file. /// public static class LoadFromFile { public static async Task RunAsync() { Console.WriteLine("--- Loading default profile from config.toml ---"); try { // For this sample to be self-contained, we explicitly provide the path to // the config.toml file included in this directory. // By default though, the config.toml file will be loaded from // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml"); // LoadClientConnectOptions is a helper that loads a profile and prepares // the config for TemporalClient.ConnectAsync. By default, it loads the // "default" profile. var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions { ConfigSource = DataSource.FromPath(configFile), }); Console.WriteLine($"Loaded 'default' profile from {configFile}."); Console.WriteLine($" Address: {connectOptions.TargetHost}"); Console.WriteLine($" Namespace: {connectOptions.Namespace}"); if (connectOptions.RpcMetadata?.Count > 0) { Console.WriteLine($" gRPC Metadata: {string.Join(", ", connectOptions.RpcMetadata.Select(kv => $"{kv.Key}={kv.Value}"))}"); } Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); // Test the connection by checking the service var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new()); Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **TypeScript** To load the `default` profile along with any environment variables in TypeScript, use the `loadClientConnectConfig` helper from `@temporalio/envconfig` package. [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { console.log('--- Loading default profile from config.toml ---'); // For this sample to be self-contained, we explicitly provide the path to // the config.toml file included in this directory. // By default though, the config.toml file will be loaded from // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). const configFile = resolve(__dirname, '../config.toml'); // loadClientConnectConfig is a helper that loads a profile and prepares // the configuration for Connection.connect and Client. By default, it loads the // "default" profile. const config = loadClientConnectConfig({ configSource: { path: configFile }, }); console.log(`Loaded 'default' profile from ${configFile}.`); console.log(` Address: ${config.connectionOptions.address}`); console.log(` Namespace: ${config.namespace}`); console.log(` gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Java** To load the `default` profile along with any environment variables in Java, use the `ClientConfigProfile.load` method from the `envconfig` package. This method will load the `default` profile from the default location and any environment variables. Environment variables take precedence over the configuration file settings. Then use `profile.toWorkflowServiceStubsOptions` and `profile.toWorkflowClientOptions` to convert the profile to `WorkflowServiceStubsOptions` and `WorkflowClientOptions` respectively. Then use `WorkflowClient.newInstance` to create a Temporal Client. ```java import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadFromFile { private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class); public static void main(String[] args) { try { ClientConfigProfile profile = ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build()); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` ## Load configuration from a custom path To load configuration from a non-standard file location without relying on the `TEMPORAL_CONFIG_FILE` environment variable, you can use a function from the `temporalio.envconfig` package. The specific method you need to call depends on the SDK you are using. This is useful if you store application-specific configurations separately. Loading connection options using this method will still respect environment variables, which take precedence over the file settings. **Python** To load a specific profile from a custom path in Python, use the `ClientConfig.load_client_connect_config()` method with the `config_file` parameter. In this example, we construct the path to a `config.toml` file located in the same directory as the script. After loading the connection options, you can override specific settings programmatically before passing them to `Client.connect()`. ```py {12-13,21-23} import asyncio from pathlib import Path from temporalio.client import Client from temporalio.envconfig import ClientConfig async def main(): """ Demonstrates loading a named profile and overriding values programmatically. """ print("--- Loading 'staging' profile with programmatic overrides ---") config_file = Path(__file__).parent / "config.toml" profile_name = "staging" print( "The 'staging' profile in config.toml has an incorrect address (localhost:9999)." ) print("We'll programmatically override it to the correct address.") # Load the 'staging' profile. connect_config = ClientConfig.load_client_connect_config( profile=profile_name, config_file=str(config_file), ) # Override the target host to the correct address. # This is the recommended way to override configuration values. connect_config["target_host"] = "localhost:7233" print(f"\nLoaded '{profile_name}' profile from {config_file} with overrides.") print( f" Address: {connect_config.get('target_host')} (overridden from localhost:9999)" ) print(f" Namespace: {connect_config.get('namespace')}") print("\nAttempting to connect to client...") try: await Client.connect(**connect_config) # type: ignore print("✅ Client connected successfully!") except Exception as e: print(f"❌ Failed to connect: {e}") if __name__ == "__main__": asyncio.run(main()) ``` **Go** To load a specific profile from a custom filepath in Go, use the `envconfig.LoadClientOptions()` function with the `ConfigFilePath` field set in the `LoadClientOptionsRequest` struct. Use the `ConfigFileProfile` field to specify the profile name. After loading the connection options, you can override specific settings programmatically before passing them to `client.Dial()`. Refer to the [GO SDK API documentation](https://pkg.go.dev/go.temporal.io/sdk/contrib/envconfig) for all available options. ```go {14-16} package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Load a specific profile from the TOML config file. // This requires a [profile.prod] section in your config. opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{ ConfigFileProfile: "prod", ConfigFilePath: "/Users/yourname/.config/my-app/temporal.toml", }) if err != nil { log.Fatalf("Failed to load 'prod' profile: %v", err) } // Programmatically override the Namespace value. opts.Namespace = "new-namespace" c, err := client.Dial(opts) if err != nil { log.Fatalf("Failed to connect using 'prod' profile: %v", err) } defer c.Close() fmt.Printf("✅ Connected to Temporal namespace %q on %s using 'prod' profile\n", c.Options().Namespace, c.Options().HostPort) } ``` **Ruby** To load a specific profile from a custom path in Ruby, use the `EnvConfig::ClientConfig.load_client_connect_options()` method with the `config_source` parameter. In this example, we construct the path to a `config.toml` file located in the same directory as the script. Use the `profile` parameter to specify the profile name. After loading the connection options, you can override specific settings programmatically before passing them to `Client.connect()`. Refer to the [Ruby SDK API documentation](https://ruby.temporal.io/Temporalio/EnvConfig.html) for all available options. ```Ruby {7-8,14-16} require 'temporalio/client' require 'temporalio/env_config' def main puts "--- Loading 'staging' profile with programmatic overrides ---" config_file = File.join(__dir__, 'config.toml') profile_name = 'staging' puts "The 'staging' profile in config.toml has an incorrect address (localhost:9999)." puts "We'll programmatically override it to the correct address." # Load the 'staging' profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( profile: profile_name, config_source: Pathname.new(config_file) ) # Override the target host to the correct address. # This is the recommended way to override configuration values. args[0] = 'localhost:7233' puts "\nLoaded '#{profile_name}' profile from #{config_file} with overrides." puts " Address: #{args[0]} (overridden from localhost:9999)" puts " Namespace: #{args[1]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end main if $PROGRAM_NAME == __FILE__ ``` **.NET** To load a specific profile from a custom path in .NET C#, use the `ClientEnvConfig.LoadClientConnectOptions()` method with the `ProfileLoadOptions` parameter. Use the `Profile` property to specify the profile name and the `ConfigSource` property to specify the file path. After loading the connection options, you can override specific settings programmatically before passing them to `TemporalClient.ConnectAsync()`. Refer to the [C# SDK API documentation](https://dotnet.temporal.io/api/Temporalio.Common.EnvConfig.html) for all available options. ```csharp {18-19,25-28} using Temporalio.Client; using Temporalio.Client.EnvConfig; namespace TemporalioSamples.EnvConfig; /// /// Sample demonstrating loading a named environment configuration profile and /// programmatically overriding its values. /// public static class LoadProfile { public static async Task RunAsync() { Console.WriteLine("--- Loading 'staging' profile with programmatic overrides ---"); try { var configFile = Path.Combine(Directory.GetCurrentDirectory(), "config.toml"); var profileName = "staging"; Console.WriteLine("The 'staging' profile in config.toml has an incorrect address (localhost:9999)."); Console.WriteLine("We'll programmatically override it to the correct address."); // Load the 'staging' profile var connectOptions = ClientEnvConfig.LoadClientConnectOptions(new ClientEnvConfig.ProfileLoadOptions { Profile = profileName, ConfigSource = DataSource.FromPath(configFile), }); // Override the target host to the correct address. // This is the recommended way to override configuration values. connectOptions.TargetHost = "localhost:7233"; Console.WriteLine($"\nLoaded '{profileName}' profile from {configFile} with overrides."); Console.WriteLine($" Address: {connectOptions.TargetHost} (overridden from localhost:9999)"); Console.WriteLine($" Namespace: {connectOptions.Namespace}"); Console.WriteLine("\nAttempting to connect to client..."); var client = await TemporalClient.ConnectAsync(connectOptions); Console.WriteLine("✅ Client connected successfully!"); // Test the connection by checking the service var sysInfo = await client.Connection.WorkflowService.GetSystemInfoAsync(new()); Console.WriteLine("✅ Successfully verified connection to Temporal server!\n{0}", sysInfo); } catch (Exception ex) when (ex is not OperationCanceledException) { Console.WriteLine($"❌ Failed to connect: {ex.Message}"); } } } ``` **TypeScript** To load a specific profile from a custom path in TypeScript, use the `loadClientConnectConfig` helper from `@temporalio/envconfig` package with the `profile` and `configFile` options. [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { console.log('--- Loading default profile from config.toml ---'); // For this sample to be self-contained, we explicitly provide the path to // the config.toml file included in this directory. // By default though, the config.toml file will be loaded from // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). const configFile = resolve(__dirname, '../config.toml'); // loadClientConnectConfig is a helper that loads a profile and prepares // the configuration for Connection.connect and Client. By default, it loads the // "default" profile. const config = loadClientConnectConfig({ configSource: { path: configFile }, }); console.log(`Loaded 'default' profile from ${configFile}.`); console.log(` Address: ${config.connectionOptions.address}`); console.log(` Namespace: ${config.namespace}`); console.log(` gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Java** To load a profile configuration file from a custom path in Java, use the `ClientConfigProfile.load` method from the `envconfig` package with the `ConfigFilePath` parameter. This method will load the profile from the custom path and any environment variables. Environment variables take precedence over the configuration file settings. ```java {21-25} import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadFromFile { private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class); public static void main(String[] args) { try { String configFilePath = Paths.get(LoadFromFile.class.getResource("/config.toml").toURI()).toString(); ClientConfigProfile profile = ClientConfigProfile.load( LoadClientConfigProfileOptions.newBuilder() .setConfigFilePath(configFilePath) .build()); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` --- # Go SDK developer guide Source: https://docs.temporal.io/develop/go ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Install and get started You can find detailed installation instructions for the Go SDK in the [Quickstart](/develop/go/set-up-your-local-go). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Workflow basics](/develop/go/workflows/basics) - [Activity basics](/develop/go/activities/basics) - [Start an Activity execution](/develop/go/activities/execution) - [Run Worker processes](/develop/go/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/go/workflows) - [Workflow basics](/develop/go/workflows/basics) - [Child Workflows](/develop/go/workflows/child-workflows) - [Continue-As-New](/develop/go/workflows/continue-as-new) - [Cancellation](/develop/go/workflows/cancellation) - [Timeouts](/develop/go/workflows/timeouts) - [Message passing](/develop/go/workflows/message-passing) - [Selectors](/develop/go/workflows/selectors) - [Side effects](/develop/go/workflows/side-effects) - [Schedules](/develop/go/workflows/schedules) - [Timers](/develop/go/workflows/timers) - [Dynamic Workflow](/develop/go/workflows/dynamic-workflow) - [Versioning](/develop/go/workflows/versioning) - [Workflow Streams](/develop/go/workflows/workflow-streams) ## [Activities](/develop/go/activities) - [Activity basics](/develop/go/activities/basics) - [Activity execution](/develop/go/activities/execution) - [Standalone Activities](/develop/go/activities/standalone-activities-quickstart) - [Timeouts](/develop/go/activities/timeouts) - [Asynchronous Activity completion](/develop/go/activities/asynchronous-activity) - [Dynamic Activity](/develop/go/activities/dynamic-activity) - [Benign exceptions](/develop/go/activities/benign-exceptions) ## [Workers](/develop/go/workers) - [Run a Worker](/develop/go/workers/run-worker-process) - [Sessions](/develop/go/workers/sessions) - [Serverless Workers](/develop/go/workers/serverless-workers) ## [Temporal Client](/develop/go/client) - [Temporal Client](/develop/go/client/temporal-client) - [Namespaces](/develop/go/client/namespaces) ## [Temporal Nexus](/develop/go/nexus) - [Quickstart](/develop/go/nexus/quickstart) - [Feature guide](/develop/go/nexus/feature-guide) - [Standalone Operations](/develop/go/nexus/standalone-operations) ## [Platform](/develop/go/platform) - [Observability](/develop/go/platform/observability) - [Enriching the UI](/develop/go/platform/enriching-ui) ## [Best practices](/develop/go/best-practices) - [Multithreading](/develop/go/best-practices/multithreading) - [Context propagation](/develop/go/best-practices/context-propagation) - [Error handling](/develop/go/best-practices/error-handling) - [Debugging](/develop/go/best-practices/debugging) - [Testing](/develop/go/best-practices/testing-suite) - [Data handling](/develop/go/data-handling) ## [Integrations](/develop/go/integrations) - [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#go) - [Google ADK integration](/develop/go/integrations/google-adk) - [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2) ## Temporal Go technical resources - [Go SDK Quickstart - Setup Guide](/develop/go/set-up-your-local-go) - [Go API Documentation](https://pkg.go.dev/go.temporal.io/sdk) - [Go SDK Code Samples](https://github.com/temporalio/samples-go) - [Go SDK GitHub](https://github.com/temporalio/sdk-go) - [Temporal 101 in Go Free Course](https://learn.temporal.io/courses/temporal_101/go/) ### Where are SDK-specific code examples? - [Background Check application](https://github.com/temporalio/background-checks): Provides a non-trivial Temporal Application implementation in conjunction with [application documentation](https://learn.temporal.io/examples/go/background-checks/). - [Hello world application template in Go](https://github.com/temporalio/hello-world-project-template-go): Provides a quick-start development app for users. This sample works in conjunction with the ["Hello World!" from scratch tutorial in Go](https://learn.temporal.io/getting_started/go/hello_world_in_go/). - [Money transfer application template in Go](https://github.com/temporalio/money-transfer-project-template-go): Provides a quick-start development app for users. It demonstrates a basic "money transfer" Workflow Definition and works in conjunction with the [Run your first app tutorial in Go](https://learn.temporal.io/getting_started/go/first_program_in_go/). - [Subscription-style Workflow Definition in Go](https://github.com/temporalio/subscription-workflow-project-template-go): Demonstrates some of the patterns that could be implemented for a subscription-style business process. - [eCommerce application example in Go](https://github.com/temporalio/temporal-ecommerce): Showcases a per-user shopping cart–style Workflow Definition that includes an API for adding and removing items from the cart as well as a web UI. This application sample works in conjunction with the [eCommerce in Go tutorial](https://learn.temporal.io/tutorials/go/build-an-ecommerce-app). ## Get connected with the Temporal Go community - [Temporal Go Community Slack](https://temporalio.slack.com/archives/CTDTU3J4T) - [Go SDK Forum](https://community.temporal.io/tag/go-sdk) --- # Activities - Go SDK Source: https://docs.temporal.io/develop/go/activities > This section explains how to implement Activities with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Activities - [Activity basics](/develop/go/activities/basics) - [Activity execution](/develop/go/activities/execution) - [Standalone Activities Quickstart](/develop/go/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/go/activities/standalone-activities) - [Timeouts](/develop/go/activities/timeouts) - [Asynchronous Activity completion](/develop/go/activities/asynchronous-activity) - [Dynamic Activity](/develop/go/activities/dynamic-activity) - [Benign exceptions](/develop/go/activities/benign-exceptions) --- # Asynchronous Activity completion - Go SDK Source: https://docs.temporal.io/develop/go/activities/asynchronous-activity > Asynchronous Activity Completion lets the Activity Function return without finishing Activity Execution. Use Task Tokens and Temporal Client to complete the Activity externally. [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. **Step 1: Provide the external system with a Task Token to complete the Activity Execution.** To do this, use the `GetInfo()` API from the `go.temporal.io/sdk/activity` package. ```go // Retrieve the Activity information needed to asynchronously complete the Activity. activityInfo := activity.GetInfo(ctx) taskToken := activityInfo.TaskToken // Send the taskToken to the external service that will complete the Activity. ``` **Step 2: Return an `activity.ErrResultPending` error to indicate that the Activity is completing asynchronously.** ```go return "", activity.ErrResultPending ``` **Step 3: Use the Temporal Client to complete the Activity using the Task Token.** ```go // Instantiate a Temporal service client. // The same client can be used to complete or fail any number of Activities. // The client is a heavyweight object that should be created once per process. temporalClient, err := client.Dial(client.Options{}) // Complete the Activity. temporalClient.CompleteActivity(context.Background(), taskToken, result, nil) ``` The following are the parameters of the `CompleteActivity` function: - `taskToken`: The value of the binary `TaskToken` field of the `ActivityInfo` struct retrieved inside the Activity. - `result`: The return value to record for the Activity. The type of this value must match the type of the return value declared by the Activity function. - `err`: The error code to return if the Activity terminates with an error. If `err` is not null, the value of the `result` field is ignored. To fail the Activity, you would do the following: ```go // Fail the Activity. client.CompleteActivity(context.Background(), taskToken, nil, err) ``` --- # Activity basics - Go SDK Source: https://docs.temporal.io/develop/go/activities/basics > This section explains Activity basics with the Go SDK ## How to develop an Activity Definition in Go In the Temporal Go SDK programming model, an Activity Definition is an exportable function or a `struct` method. Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The Activity definition and Worker registration are identical to regular Activities, and only the execution path differs. See [Standalone Activities](/develop/go/activities/standalone-activities-quickstart). Below is an example of both a basic Activity Definition and of an Activity defined as a Struct method. An _Activity struct_ can have more than one method, with each method acting as a separate Activity Type. Activities written as struct methods can use shared struct variables, such as: - an application level DB pool - client connection to another service - reusable utilities - any other expensive resources that you only want to initialize once per process Because this is such a common need, the rest of this guide shows Activities written as `struct` methods. > **📝 Note:** > > While it is possible to register struct methods as Workflows, this is strongly discouraged. > In some cases, struct methods as Workflows may cause non-deterministic errors. We recommend > only using struct methods for Activities. > ```go package yourapp import ( "context" "go.temporal.io/sdk/activity" ) func YourSimpleActivityDefinition(ctx context.Context) error { return nil } type YourActivityObject struct { Message *string Number *int } func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) { // Use Activities for calling external APIs. // This is just an example of using the logger to print "Hello World!" logger := activity.GetLogger(ctx) logger.Info("The message is:", param.ActivityParamX) logger.Info("The number is:", param.ActivityParamY) // Return data using a Struct so that the function signature is backward compatible. result := &YourActivityResultObject{ ResultFieldX: "Success", ResultFieldY: 1, } // Return the results back to the Workflow Execution. // The results persist within the Event History of the Workflow Execution. return result, nil } ``` ### How to develop Activity Parameters There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. The first parameter of an Activity Definition is `context.Context`. This parameter is optional for an Activity Definition, though it is recommended, especially if the Activity is expected to use other Go SDK APIs. An Activity Definition can support as many other custom parameters as needed. However, all parameters must be serializable (parameters can't be channels, functions, variadic, or unsafe pointers), and it is recommended to pass a single struct that can be updated later. ```go {6-9,11} type YourActivityParam struct { ActivityParamX string ActivityParamY int } type YourActivityObject struct { Message *string Number *int } func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) { // Use Activities for calling external APIs. // This is just an example of using the logger to print "Hello World!" logger := activity.GetLogger(ctx) logger.Info("The message is:", param.ActivityParamX) logger.Info("The number is:", param.ActivityParamY) // Return data using a Struct so that the function signature is backward compatible. result := &YourActivityResultObject{ ResultFieldX: "Success", ResultFieldY: 1, } // Return the results back to the Workflow Execution. // The results persist within the Event History of the Workflow Execution. return result, nil } ``` ### How to define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). A Go-based Activity Definition can return either just an `error` or a `customValue, error` combination (same as a Workflow Definition). You may wish to use a `struct` type to hold all custom values, just keep in mind they must all be serializable. ```go {2-5,14-17} // ... type YourActivityResultObject struct { ResultFieldX string ResultFieldY int } func (a *YourActivityObject) YourActivityDefinition(ctx context.Context, param YourActivityParam) (*YourActivityResultObject, error) { // Use Activities for calling external APIs. // This is just an example of using the logger to print "Hello World!" logger := activity.GetLogger(ctx) logger.Info("The message is:", param.ActivityParamX) logger.Info("The number is:", param.ActivityParamY) // Return data using a Struct so that the function signature is backward compatible. result := &YourActivityResultObject{ ResultFieldX: "Success", ResultFieldY: 1, } // Return the results back to the Workflow Execution. // The results persist within the Event History of the Workflow Execution. return result, nil } ``` ### How to customize Activity Type in Go To customize the Activity Type, set the `Name` parameter with `RegisterOptions` when registering your Activity with a Worker. ```go {8,22-25} func main() { temporalClient, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer temporalClient.Close() yourWorker := worker.New(temporalClient, "your-custom-task-queue-name", worker.Options{}) yourWorker.RegisterWorkflow(yourapp.YourWorkflowDefinition) registerWFOptions := workflow.RegisterOptions{ Name: "JustAnotherWorkflow", } yourWorker.RegisterWorkflowWithOptions(yourapp.YourSimpleWorkflowDefinition, registerWFOptions) message := "This could be a connection string or endpoint details" number := 100 activities := &yourapp.YourActivityObject{ Message: &message, Number: &number, } registerAOptions := activity.RegisterOptions{ Name: "JustAnotherActivity", } yourWorker.RegisterActivityWithOptions(yourapp.YourSimpleActivityDefinition, registerAOptions) err = yourWorker.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start Worker", err) } } ``` --- # Benign exceptions - Go SDK Source: https://docs.temporal.io/develop/go/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. When Activities return errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, use [`temporal.NewApplicationErrorWithOptions`](https://pkg.go.dev/go.temporal.io/sdk/temporal#NewApplicationErrorWithOptions) and set the `Category` field to `temporal.ApplicationErrorCategoryBenign` in the `ApplicationErrorOptions`. Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```go import ( "go.temporal.io/sdk/activity" "go.temporal.io/sdk/temporal" ) func MyActivity(ctx context.Context) (string, error) { result, err := callExternalService() if err != nil { // Mark this error as benign since it's expected return "", temporal.NewApplicationErrorWithOptions( err.Error(), "", temporal.ApplicationErrorOptions{ Category: temporal.ApplicationErrorCategoryBenign, }, ) } return result, nil } ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Dynamic Activity - Go SDK Source: https://docs.temporal.io/develop/go/activities/dynamic-activity > This section explains Dynamic Activities with the Go SDK ## Set a Dynamic Activity A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered. An Activity can be registered as dynamic by using `worker.RegisterDynamicActivity()`. You must register the Activity with the Worker before it can be invoked. Only one Dynamic Activity can be present on a Worker. The Activity Definition must then accept a single argument of type `converter.EncodedValues`. This code snippet is taken from the [Dynamic Workflow example from samples-go](https://github.com/temporalio/samples-go/tree/main/dynamic-workflows). ```go func DynamicActivity(ctx context.Context, args converter.EncodedValues) (string, error) { var arg1, arg2 string err := args.Get(&arg1, &arg2) if err != nil { return "", fmt.Errorf("failed to decode arguments: %w", err) } info := activity.GetInfo(ctx) result := fmt.Sprintf("%s - %s - %s", info.WorkflowType.Name, arg1, arg2) return result, nil } ``` --- # Activity execution - Go SDK Source: https://docs.temporal.io/develop/go/activities/execution > Shows how to perform Activity execution with the Go SDK ## How to start an Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History. A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity implementation code should be _idempotent_. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal Service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. To spawn an [Activity Execution](/activity-execution), call [`ExecuteActivity()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ExecuteActivity) inside your Workflow Definition. The API is available from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package. The `ExecuteActivity()` API call requires an instance of `workflow.Context`, the Activity function name, and any variables to be passed to the Activity Execution. The Activity function name can be provided as a variable object (no quotations) or as a string. The benefit of passing the actual function object is that the framework can validate the parameters against the Activity Definition. The `ExecuteActivity` call returns a Future, which can be used to get the result of the Activity Execution. ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) { activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) activityParam := YourActivityParam{ ActivityParamX: param.WorkflowParamX, ActivityParamY: param.WorkflowParamY, } var a *YourActivityObject var activityResult YourActivityResultObject err := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam).Get(ctx, &activityResult) if err != nil { return nil, err } } ``` ### How to set the required Activity Timeouts Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. To set an Activity Timeout in Go, create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the Activity Timeout field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. Available timeouts are: - `StartToCloseTimeout` - `ScheduleToClose` - `ScheduleToStartTimeout` ```go activityOptions := workflow.ActivityOptions{ // Set Activity Timeout duration ScheduleToCloseTimeout: 10 * time.Second, // StartToCloseTimeout: 10 * time.Second, // ScheduleToStartTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` ### Go ActivityOptions reference Create an instance of [`ActivityOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ActivityOptions) from the `go.temporal.io/sdk/workflow` package and use [`WithActivityOptions()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#WithActivityOptions) to apply it to the instance of `workflow.Context`. The instance of `workflow.Context` is then passed to the `ExecuteActivity()` call. | Field | Required | Type | | --------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------- | | [`ActivityID`](#activityid) | No | `string` | | [`TaskQueueName`](#taskqueuename) | No | `string` | | [`ScheduleToCloseTimeout`](#scheduletoclosetimeout) | Yes (or `StartToCloseTimeout`) | `time.Duration` | | [`ScheduleToStartTimeout`](#scheduletostarttimeout) | No | `time.Duration` | | [`StartToCloseTimeout`](#scheduletoclosetimeout) | Yes (or `ScheduleToCloseTimeout`) | `time.Duration` | | [`HeartbeatTimeout`](#heartbeattimeout) | No | `time.Duration` | | [`WaitForCancellation`](#waitforcancellation) | No | `bool` | | [`OriginalTaskQueueName`](#originaltaskqueuename) | No | `string` | | [`RetryPolicy`](#retrypolicy) | No | [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) | #### ActivityID - Type: `string` - Default: None ```go activityOptions := workflow.ActivityOptions{ ActivityID: "your-activity-id", } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` - [What is an Activity Id](/activity-execution#activity-id) #### TaskQueueName - Type: `string` - Default: Inherits the TaskQueue name from the Workflow. ```go activityOptions := workflow.ActivityOptions{ TaskQueueName: "your-task-queue-name", } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` - [What is a Task Queue](/task-queue) #### ScheduleToCloseTimeout To set a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `ScheduleToCloseTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. This or `StartToCloseTimeout` must be set. - Type: `time.Duration` - Default: ∞ (infinity - no limit) ```go activityOptions := workflow.ActivityOptions{ ScheduleToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### ScheduleToStartTimeout To set a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `ScheduleToStartTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. - Type: `time.Duration` - Default: ∞ (infinity - no limit) ```go activityOptions := workflow.ActivityOptions{ ScheduleToStartTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### StartToCloseTimeout To set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `StartToCloseTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. This or `ScheduleToCloseTimeout` must be set. - Type: `time.Duration` - Default: Same as the `ScheduleToCloseTimeout` ```go activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### HeartbeatTimeout To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `HeartbeatTimeout` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. ```go activityOptions := workflow.ActivityOptions{ HeartbeatTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### WaitForCancellation If `true` the Activity Execution will finish executing should there be a Cancellation request. - Type: `bool` - Default: `false` ```go activityOptions := workflow.ActivityOptions{ WaitForCancellation: false, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### OriginalTaskQueueName ```go activityOptions := workflow.ActivityOptions{ OriginalTaskQueueName: "your-original-task-queue-name", } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` #### RetryPolicy To set a [RetryPolicy](/encyclopedia/retry-policies), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `RetryPolicy` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. - Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) - Default: ```go retryPolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, // 100 * InitialInterval MaximumAttempts: 0, // Unlimited NonRetryableErrorTypes: []string, // empty } ``` Providing a Retry Policy here is a customization that overwrites individual Field defaults. ```go retryPolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, } activityOptions := workflow.ActivityOptions{ RetryPolicy: retryPolicy, } ctx = workflow.WithActivityOptions(ctx, activityOptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` ### How to get the results of an Activity Execution The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable. Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available. The `ExecuteActivity` API call returns an instance of [`workflow.Future`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Futures) which has the following two methods: - `Get()`: Takes an instance of the `workflow.Context`, that was passed to the Activity Execution, and a pointer as parameters. The variable associated with the pointer is populated with the Activity Execution result. This call blocks until the results are available. - `IsReady()`: Returns `true` when the result of the Activity Execution is ready. Call the `Get()` method on the instance of `workflow.Future` to get the result of the Activity Execution. The type of the result parameter must match the type of the return value declared by the Activity function. ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (YourWorkflowResponse, error) { // ... future := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam) var yourActivityResult YourActivityResult if err := future.Get(ctx, &yourActivityResult); err != nil { // ... } // ... } ``` Use the `IsReady()` method first to make sure the `Get()` call doesn't cause the Workflow Execution to wait on the result. ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (YourWorkflowResponse, error) { // ... future := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam) // ... if(future.IsReady()) { var yourActivityResult YourActivityResult if err := future.Get(ctx, &yourActivityResult); err != nil { // ... } } // ... } ``` It is idiomatic to invoke multiple Activity Executions from within a Workflow. Therefore, it is also idiomatic to either block on the results of the Activity Executions or continue on to execute additional logic, checking for the Activity Execution results at a later time. --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/go/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal Go SDK. > **Public Preview** Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition using `workflow.ExecuteActivity()`, you start a Standalone Activity directly from a Temporal Client using `client.ExecuteActivity()`. The Activity definition and Worker registration are identical to regular Activities, and only the execution path differs. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/go/activities/standalone-activities-quickstart). > This page covers the following: - [Get the result of a Standalone Activity](#get-activity-result) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the > [standalone-activity/helloworld](https://github.com/temporalio/samples-go/tree/main/standalone-activity/helloworld). > ## Get the result of a Standalone Activity Use `ActivityHandle.Get()` to block until the Activity completes and retrieve its result. This is analogous to calling `Get()` on a `WorkflowRun`. ```go var result string err = handle.Get(context.Background(), &result) if err != nil { log.Fatalln("Activity failed", err) } log.Println("Activity result:", result) ``` If the Activity completed successfully, the result is deserialized into the provided pointer. If the Activity failed, the failure is returned as an error. Or use the Temporal CLI to wait for a result by Activity ID: ```bash temporal activity result --activity-id standalone_activity_helloworld_ActivityID ``` ## Get a handle to an existing Standalone Activity Use `client.GetActivityHandle()` to create a handle to a previously started Standalone Activity. This is analogous to `client.GetWorkflow()` for Workflow Executions. Both `ActivityID` and `RunID` are required. ```go handle := c.GetActivityHandle(client.GetActivityHandleOptions{ ActivityID: "standalone_activity_helloworld_ActivityID", RunID: "the-run-id", }) // Use the handle to get the result, describe, cancel, or terminate var result string err := handle.Get(context.Background(), &result) if err != nil { log.Fatalln("Unable to get activity result", err) } ``` ## List Standalone Activities Use [`client.ListActivities()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields [`ActivityExecutionInfo`](https://pkg.go.dev/go.temporal.io/sdk/client#ActivityExecutionInfo) entries. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. ```go resp, err := c.ListActivities(context.Background(), client.ListActivitiesOptions{ Query: "TaskQueue = 'standalone-activity-helloworld'", }) if err != nil { log.Fatalln("Unable to list activities", err) } for info, err := range resp.Results { if err != nil { log.Fatalln("Error iterating activities", err) } log.Printf("ActivityID: %s, Type: %s, Status: %v\n", info.ActivityID, info.ActivityType, info.Status) } ``` Or use the Temporal CLI: ```bash temporal activity list ``` The `Query` field accepts the same [List Filter](/list-filter) syntax used for Workflow Visibility. For example, `"ActivityType = 'Activity' AND Status = 'Running'"`. ## Count Standalone Activities Use [`client.CountActivities()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) - not the number of queued tasks. It works the same way as counting Workflow Executions. ```go resp, err := c.CountActivities(context.Background(), client.CountActivitiesOptions{ Query: "TaskQueue = 'standalone-activity-helloworld'", }) if err != nil { log.Fatalln("Unable to count activities", err) } log.Println("Total activities:", resp.Count) ``` Or use the Temporal CLI: ```bash temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The Worker and Client code in the [Standalone Activities Quickstart](/develop/go/activities/standalone-activities-quickstart) use `envconfig.MustLoadDefaultClientOptions()`, so the same code works against Temporal Cloud - configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/go/activities/standalone-activities-quickstart). --- # Standalone Activities Go Quickstart Source: https://docs.temporal.io/develop/go/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal Go SDK without writing a Workflow. # Quickstart Standalone Activities are Activity Executions that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition using `workflow.ExecuteActivity()`, you start a Standalone Activity directly from a Temporal Client using `client.ExecuteActivity()`. The Activity definition and Worker registration are identical to regular Activities, and only the execution path differs. > **📝 Note:** > > This documentation uses source code from the > [standalone-activity/helloworld](https://github.com/temporalio/samples-go/tree/main/standalone-activity/helloworld). > ## Get started with Standalone Activities Prerequisites: - **[Go](https://go.dev/dl/)** 1.22+ - **[Temporal Go SDK](/develop/go/set-up-your-local-go#install-the-temporal-go-sdk)** (v1.41.0 or higher) - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server should now be available for client connections on `localhost:7233`, and the Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-go](https://github.com/temporalio/samples-go) repository to follow along: ``` git clone https://github.com/temporalio/samples-go.git cd samples-go ``` The sample project is structured as follows: ``` standalone-activity/helloworld/ ├── activity.go ├── worker/ │ └── main.go └── starter/ └── main.go ``` ## Define your Activity Define your Activity in a shared file so that both the Worker and starter can reference it. [standalone-activity/helloworld/activity.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/activity.go) ```go package helloworld import ( "context" "go.temporal.io/sdk/activity" ) func Activity(ctx context.Context, name string) (string, error) { logger := activity.GetLogger(ctx) logger.Info("Activity", "name", name) return "Hello " + name + "!", nil } ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow-driven Activities — you create a Worker, register the Activity, and call `Run()`. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to develop a Worker in Go](/develop/go/workers/run-worker-process#develop-worker) for more details on Worker setup and configuration options. [standalone-activity/helloworld/worker/main.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/worker/main.go) Open a new terminal, navigate to the `samples-go` directory, and run the Worker. Leave this terminal running - the Worker needs to stay up to process activities. ```go package main import ( "github.com/temporalio/samples-go/standalone-activity/helloworld" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" "go.temporal.io/sdk/worker" "log" ) func main() { c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "standalone-activity-helloworld", worker.Options{}) w.RegisterActivity(helloworld.Activity) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` ```bash go run standalone-activity/helloworld/worker/main.go ``` ## Execute a Standalone Activity Use [`client.ExecuteActivity()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to start a Standalone Activity Execution. This is called from application code (for example, a starter program), not from inside a Workflow Definition. `ExecuteActivity` returns an [`ActivityHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#ActivityHandle) that you can use to get the result, describe, cancel, or terminate the Activity. The following starter program demonstrates how to execute a Standalone Activity, get its result, list activities, and count activities: Create [standalone-activity/helloworld/starter/main.go](https://github.com/temporalio/samples-go/blob/main/standalone-activity/helloworld/starter/main.go). You can pass the Activity as either a function reference or a string Activity type name: ```go handle, err := c.ExecuteActivity(ctx, options, helloworld.Activity, "arg1") // Using a string type name handle, err := c.ExecuteActivity(ctx, options, "Activity", "arg1") ``` `client.StartActivityOptions` requires `ID`, `TaskQueue`, and at least one of `ScheduleToCloseTimeout` or `StartToCloseTimeout`. See [`StartActivityOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartActivityOptions) in the API reference for the full set of options. To run the starter: 1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above). 2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above). 3. Open a new terminal, navigate to the `samples-go` directory, and run `go run standalone-activity/helloworld/starter/main.go`. Or use the Temporal CLI to execute a Standalone Activity. ```go package main import ( "context" "github.com/temporalio/samples-go/standalone-activity/helloworld" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" "log" "time" ) func main() { c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() activityOptions := client.StartActivityOptions{ ID: "standalone_activity_helloworld_ActivityID", TaskQueue: "standalone-activity-helloworld", ScheduleToCloseTimeout: 10 * time.Second, } handle, err := c.ExecuteActivity(context.Background(), activityOptions, helloworld.Activity, "Temporal") if err != nil { log.Fatalln("Unable to execute activity", err) } log.Println("Started standalone activity", "ActivityID", handle.GetID(), "RunID", handle.GetRunID()) var result string err = handle.Get(context.Background(), &result) if err != nil { log.Fatalln("Unable get standalone activity result", err) } log.Println("Activity result:", result) resp, err := c.ListActivities(context.Background(), client.ListActivitiesOptions{ Query: "TaskQueue = 'standalone-activity-helloworld'", }) if err != nil { log.Fatalln("Unable to list activities", err) } log.Println("ListActivity results") for info, err := range resp.Results { if err != nil { log.Fatalln("Error iterating activities", err) } log.Printf("\tActivityID: %s, Type: %s, Status: %v\n", info.ActivityID, info.ActivityType, info.Status) } resp1, err := c.CountActivities(context.Background(), client.CountActivitiesOptions{ Query: "TaskQueue = 'standalone-activity-helloworld'", }) if err != nil { log.Fatalln("Unable to count activities", err) } log.Println("Total activities:", resp1.Count) } ``` ```bash go run standalone-activity/helloworld/starter/main.go ``` ```bash temporal activity execute \\ --type Activity \\ --activity-id standalone_activity_helloworld_ActivityID \\ --task-queue standalone-activity-helloworld \\ --schedule-to-close-timeout 10s \\ --input '"Temporal"' ``` ## Run with Temporal Cloud All code samples on this page use [`envconfig.MustLoadDefaultClientOptions()`](https://pkg.go.dev/go.temporal.io/sdk/contrib/envconfig) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/go/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/go/activities/standalone-activities)**: Get results, Activity handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/go/activities/basics)**: How to write and register Activities with the Go SDK. --- # Activity Timeouts - Go SDK Source: https://docs.temporal.io/develop/go/activities/timeouts > Optimize Workflow Execution with Temporal Go SDK - Set Activity Timeouts and Retry Policies efficiently. ## How to set Activity timeouts Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution. The following timeouts are available in the Activity Options. - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. To set an Activity Timeout in Go, create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the Activity Timeout field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. Available timeouts are: - `StartToCloseTimeout` - `ScheduleToClose` - `ScheduleToStartTimeout` ```go activityoptions := workflow.ActivityOptions{ // Set Activity Timeout duration ScheduleToCloseTimeout: 10 * time.Second, // StartToCloseTimeout: 10 * time.Second, // ScheduleToStartTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityoptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` ### Set a custom Activity Retry Policy A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To set a [RetryPolicy](/encyclopedia/retry-policies), create an instance of `ActivityOptions` from the `go.temporal.io/sdk/workflow` package, set the `RetryPolicy` field, and then use the `WithActivityOptions()` API to apply the options to the instance of `workflow.Context`. - Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) - Default: ```go retrypolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, // 100 * InitialInterval MaximumAttempts: 0, // Unlimited NonRetryableErrorTypes: []string, // empty } ``` Providing a Retry Policy here is a customization, and overwrites individual Field defaults. ```go retrypolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, } activityoptions := workflow.ActivityOptions{ RetryPolicy: retrypolicy, } ctx = workflow.WithActivityOptions(ctx, activityoptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) if err != nil { // ... } ``` ### Overriding the retry interval with Next Retry Delay You may return an [Application Failure](/references/failures#application-failure) with the `NextRetryDelay` field set. This value will replace and override whatever the Retry interval would be on the Retry Policy. For example, if in an Activity, you want to base the interval on the number of attempts: ```go attempt := activity.GetInfo(ctx).Attempt; return temporal.NewApplicationErrorWithOptions(fmt.Sprintf("Something bad happened on attempt %d", attempt), "NextDelay", temporal.ApplicationErrorOptions{ NextRetryDelay: 3 * time.Second * delay, }) ``` ## Activity Heartbeats An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. Heartbeats can contain a `details` field describing the Activity's current progress. If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service. To [Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) in an Activity in Go, use the `RecordHeartbeat` API. ```go import ( // ... "go.temporal.io/sdk/workflow" // ... ) func YourActivityDefinition(ctx, YourActivityDefinitionParam) (YourActivityDefinitionResult, error) { // ... activity.RecordHeartbeat(ctx, details) // ... } ``` When an Activity Task Execution times out due to a missed Heartbeat, the last value of the `details` variable above is returned to the calling Workflow in the `details` field of `TimeoutError` with `TimeoutType` set to `Heartbeat`. You can also Heartbeat an Activity from an external source: ```go // The client is a heavyweight object that should be created once per process. temporalClient, err := client.Dial(client.Options{}) // Record heartbeat. err := temporalClient.RecordActivityHeartbeat(ctx, taskToken, details) ``` The parameters of the `RecordActivityHeartbeat` function are: - `taskToken`: The value of the binary `TaskToken` field of the `ActivityInfo` struct retrieved inside the Activity. - `details`: The serializable payload containing progress information. If an Activity Execution Heartbeats its progress before it failed, the retry attempt will have access to the progress information, so that the Activity Execution can resume from the failed state. Here's an example of how this can be implemented: ```go func SampleActivity(ctx context.Context, inputArg InputParams) error { startIdx := inputArg.StartIndex if activity.HasHeartbeatDetails(ctx) { // Recover from finished progress. var finishedIndex int if err := activity.GetHeartbeatDetails(ctx, &finishedIndex); err == nil { startIdx = finishedIndex + 1 // Start from next one. } } // Normal Activity logic... for i:=startIdx; i This section explains how to implement best practices with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Best practices - [Multithreading](/develop/go/best-practices/multithreading) - [Error handling](/develop/go/best-practices/error-handling) - [Debugging](/develop/go/best-practices/debugging) - [Testing](/develop/go/best-practices/testing-suite) - [Data handling](/develop/go/data-handling) --- # Context Propagation - Go SDK Source: https://docs.temporal.io/develop/go/best-practices/context-propagation > How to propagate custom key-value data across Workflow, Activity, and Child Workflow boundaries using the Temporal Go SDK. Context propagation lets you pass custom key-value data from a Client to Workflows, and from Workflows to Activities and Child Workflows, without threading it through every function signature. Common use cases include propagating tenant identifiers, auth tokens, or other request-scoped metadata. > **💡 Tip:** > > If you want to propagate tracing context, the Go SDK provides tracing integrations that handle propagation for you. > [Choose a tracing integration](/develop/go/platform/observability#tracing) before implementing a custom context > propagator. > ## How it works 1. **Register** a context propagator on the Client via `ContextPropagators` in [ClientOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions) 2. **Inject** - On outbound calls, the SDK calls `Inject` (from `context.Context`) or `InjectFromWorkflow` (from `workflow.Context`) to serialize values into Temporal headers 3. **Extract** - On inbound calls, the SDK calls `Extract` (into `context.Context`) or `ExtractToWorkflow` (into `workflow.Context`) to deserialize headers back into the context 4. **Access** - Your Workflow and Activity code reads values from the context as usual ## Implement a context propagator A context propagator implements the [`ContextPropagator`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ContextPropagator) interface: ```go type ContextPropagator interface { // Inject writes values from a Go context.Context into headers (Client/Activity side) Inject(context.Context, HeaderWriter) error // Extract reads headers into a Go context.Context (Client/Activity side) Extract(context.Context, HeaderReader) (context.Context, error) // InjectFromWorkflow writes values from a workflow.Context into headers InjectFromWorkflow(Context, HeaderWriter) error // ExtractToWorkflow reads headers into a workflow.Context ExtractToWorkflow(Context, HeaderReader) (Context, error) } ``` There are two pairs of methods because Go uses `context.Context` in non-Workflow code (Client, Activities) and `workflow.Context` inside Workflows. You must implement all four methods for values to propagate across every boundary (Client → Workflow → Activity/Child Workflow). Here is a propagator that carries a custom key-value pair from the Client to Workflows and Activities (from the [context propagation sample](https://github.com/temporalio/samples-go/tree/main/ctxpropagation)): [ctxpropagation/propagator.go](https://github.com/temporalio/samples-go/blob/main/ctxpropagation/propagator.go) ```go type ( // contextKey is an unexported type used as key for items stored in the // Context object contextKey struct{} // propagator implements the custom context propagator propagator struct{} // Values is a struct holding values Values struct { Key string `json:"key"` Value string `json:"value"` } ) // PropagateKey is the key used to store the value in the Context object var PropagateKey = contextKey{} // HeaderKey is the key used by the propagator to pass values through the // Temporal server headers const HeaderKey = "custom-header" // NewContextPropagator returns a context propagator that propagates a set of // string key-value pairs across a workflow func NewContextPropagator() workflow.ContextPropagator { return &propagator{} } // Inject injects values from context into headers for propagation func (s *propagator) Inject(ctx context.Context, writer workflow.HeaderWriter) error { value := ctx.Value(PropagateKey) payload, err := converter.GetDefaultDataConverter().ToPayload(value) if err != nil { return err } writer.Set(HeaderKey, payload) return nil } // InjectFromWorkflow injects values from context into headers for propagation func (s *propagator) InjectFromWorkflow(ctx workflow.Context, writer workflow.HeaderWriter) error { value := ctx.Value(PropagateKey) payload, err := converter.GetDefaultDataConverter().ToPayload(value) if err != nil { return err } writer.Set(HeaderKey, payload) return nil } // Extract extracts values from headers and puts them into context func (s *propagator) Extract(ctx context.Context, reader workflow.HeaderReader) (context.Context, error) { if value, ok := reader.Get(HeaderKey); ok { var values Values if err := converter.GetDefaultDataConverter().FromPayload(value, &values); err != nil { return ctx, nil } ctx = context.WithValue(ctx, PropagateKey, values) } return ctx, nil } ``` ## Register the propagator and set context values Register the propagator on the Client. Then set context values before starting a Workflow: [ctxpropagation/starter/main.go](https://github.com/temporalio/samples-go/blob/main/ctxpropagation/starter/main.go) ```go // The client is a heavyweight object that should be created once per process. c, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, Interceptors: []interceptor.ClientInterceptor{tracingInterceptor}, ContextPropagators: []workflow.ContextPropagator{ctxpropagation.NewContextPropagator()}, }) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() workflowID := "ctx-propagation_" + uuid.New() workflowOptions := client.StartWorkflowOptions{ ID: workflowID, TaskQueue: "ctx-propagation", } ctx := context.Background() ctx = context.WithValue(ctx, ctxpropagation.PropagateKey, &ctxpropagation.Values{Key: "test", Value: "tested"}) we, err := c.ExecuteWorkflow(ctx, workflowOptions, ctxpropagation.CtxPropWorkflow) ``` You can also register context propagators through a [Plugin](/develop/plugins-guide) if you are building a reusable library. ## Access propagated values In your Workflow, the propagated values are available on the `workflow.Context`. When the Workflow starts an Activity, the SDK automatically propagates the same values: [ctxpropagation/workflow.go](https://github.com/temporalio/samples-go/blob/main/ctxpropagation/workflow.go) ```go // CtxPropWorkflow workflow definition func CtxPropWorkflow(ctx workflow.Context) (err error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: 2 * time.Second, // such a short timeout to make sample fail over very fast } ctx = workflow.WithActivityOptions(ctx, ao) if val := ctx.Value(PropagateKey); val != nil { vals := val.(Values) workflow.GetLogger(ctx).Info("custom context propagated to workflow", vals.Key, vals.Value) } var values Values if err = workflow.ExecuteActivity(ctx, SampleActivity).Get(ctx, &values); err != nil { workflow.GetLogger(ctx).Error("Workflow failed.", "Error", err) return err } workflow.GetLogger(ctx).Info("context propagated to activity", values.Key, values.Value) workflow.GetLogger(ctx).Info("Workflow completed.") return nil } ``` [ctxpropagation/activities.go](https://github.com/temporalio/samples-go/blob/main/ctxpropagation/activities.go) ```go func SampleActivity(ctx context.Context) (*Values, error) { if val := ctx.Value(PropagateKey); val != nil { vals := val.(Values) return &vals, nil } return nil, nil } ``` You can configure multiple context propagators on a single Client, each responsible for its own set of keys. ## Context propagation over Nexus Nexus does not use the `ContextPropagator` interface. It relies on a Temporal-agnostic protocol with its own header format (`nexus.Header`, a wrapper around `map[string]string`). To propagate context over Nexus Operation calls, use interceptors to explicitly serialize and deserialize context into the Nexus header. See the [Nexus Context Propagation sample](https://github.com/temporalio/samples-go/tree/main/nexus-context-propagation). ## Further reading - [Passing Context with Temporal](https://spiralscout.com/blog/passing-context-with-temporal) - A conceptual guide to middleware and walkthrough of building a context propagator in Go --- # Debugging - Go SDK Source: https://docs.temporal.io/develop/go/best-practices/debugging > Use debugger tools and set TEMPORAL_DEBUG to true for debugging Workflow Definitions with the Temporal Go SDK, and debug production Workflows via Web UI, CLI, or tracing. You can use a debugger tool provided by your favorite IDE to debug your Workflow Definitions prior to testing or executing them. The Temporal Go SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control. Because of this you can often encounter a `PanicError: Potential deadlock detected` while stepping through Workflow Definitions during debugging. To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to `true` before debugging your Workflow Definition. > **📝 Note:** > > Make sure to set `TEMPORAL_DEBUG` to true only during debugging. > ## How to debug in a development environment In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). ## How to debug in a production environment You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/go/best-practices/testing-suite#replay) - [Tracing](/develop/go/platform/observability#tracing) - [Logging](/develop/go/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). For more information, see [Metrics](/develop/go/platform/observability#metrics) for setting up SDK metrics. Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). ## How to test Workflow Definitions in Go The Temporal Go SDK provides a test framework to facilitate testing Workflow implementations. This framework is suited for implementing unit tests as well as functional tests of the Workflow logic. The following code implements unit tests for the `SimpleWorkflow` sample: ```go package sample import ( "context" "errors" "testing" "github.com/stretchr/testify/mock" "github.com/stretchr/testify/suite" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/testsuite" ) type UnitTestSuite struct { suite.Suite testsuite.WorkflowTestSuite env *testsuite.TestWorkflowEnvironment } func (s *UnitTestSuite) SetupTest() { s.env = s.NewTestWorkflowEnvironment() } func (s *UnitTestSuite) AfterTest(suiteName, testName string) { s.env.AssertExpectations(s.T()) } func (s *UnitTestSuite) Test_SimpleWorkflow_Success() { s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( func(ctx context.Context, value string) (string, error) { s.Equal("test_success", value) return value, nil }) s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure") s.True(s.env.IsWorkflowCompleted()) err := s.env.GetWorkflowError() s.Error(err) var applicationErr *temporal.ApplicationError s.True(errors.As(err, &applicationErr)) s.Equal("SimpleActivityFailure", applicationErr.Error()) } func TestUnitTestSuite(t *testing.T) { suite.Run(t, new(UnitTestSuite)) } ``` #### Setup To run unit tests, we first define a test suite struct that absorbs both the basic suite functionality from [testify](https://pkg.go.dev/github.com/stretchr/testify/suite) via `suite.Suite` and the suite functionality from the Temporal test framework via `testsuite.WorkflowTestSuite`. Because every test in this test suite will test our Workflow, we add a property to our struct to hold an instance of the test environment. This allows us to initialize the test environment in a setup method. For testing Workflows, we use a `testsuite.TestWorkflowEnvironment`. Next, we implement a `SetupTest` method to set up a new test environment before each test. Doing so ensures that each test runs in its own isolated sandbox. We also implement an `AfterTest` function where we assert that all the mocks we set up were indeed called by invoking `s.env.AssertExpectations(s.T())`. Timeout for the entire test can be set using `SetTestTimeout` in the Workflow or Activity environment. Finally, we create a regular test function recognized by the `go test` command, and pass the struct to `suite.Run`. #### A Simple Test The simplest test case we can write is to have the test environment execute the Workflow and then evaluate the results. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_Success() { s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` Calling `s.env.ExecuteWorkflow(...)` executes the Workflow logic and any invoked Activities inside the test process. The first parameter of `s.env.ExecuteWorkflow(...)` contains the Workflow functions, and any subsequent parameters contain values for custom input parameters declared by the Workflow function. > Note that unless the Activity invocations are mocked or Activity implementation > replaced (see [Activity mocking and overriding](#activity-mocking-and-overriding)), the test environment > will execute the actual Activity code including any calls to outside services. After executing the Workflow in the above example, we assert that the Workflow ran through completion via the call to `s.env.IsWorkflowComplete()`. We also assert that no errors were returned by asserting on the return value of `s.env.GetWorkflowError()`. If our Workflow returned a value, we could have retrieved that value via a call to `s.env.GetWorkflowResult(&value)` and had additional asserts on that value. #### Activity mocking and overriding When running unit tests on Workflows, we want to test the Workflow logic in isolation. Additionally, we want to inject Activity errors during our test runs. The test framework provides two mechanisms that support these scenarios: Activity mocking and Activity overriding. Both of these mechanisms allow you to change the behavior of Activities invoked by your Workflow without the need to modify the actual Workflow code. Let's take a look at a test that simulates a test that fails via the "Activity mocking" mechanism. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure") s.True(s.env.IsWorkflowCompleted()) err := s.env.GetWorkflowError() s.Error(err) var applicationErr *temporal.ApplicationError s.True(errors.As(err, &applicationErr)) s.Equal("SimpleActivityFailure", applicationErr.Error()) } ``` This test simulates the execution of the Activity `SimpleActivity` that is invoked by our Workflow `SimpleWorkflow` returning an error. We accomplish this by setting up a mock on the test environment for the `SimpleActivity` that returns an error. ```go s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) ``` With the mock set up we can now execute the Workflow via the `s.env.ExecuteWorkflow(...)` method and assert that the Workflow completed successfully and returned the expected error. Simply mocking the execution to return a desired value or error is a pretty powerful mechanism to isolate Workflow logic. However, sometimes we want to replace the Activity with an alternate implementation to support a more complex test scenario. Let's assume we want to validate that the Activity gets called with the correct parameters. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( func(ctx context.Context, value string) (string, error) { s.Equal("test_success", value) return value, nil }) s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` In this example, we provide a function implementation as the parameter to `Return`. This allows us to provide an alternate implementation for the Activity `SimpleActivity`. The framework will execute this function whenever the Activity is invoked and pass on the return value from the function as the result of the Activity invocation. Additionally, the framework will validate that the signature of the "mock" function matches the signature of the original Activity function. Since this can be an entire function, there is no limitation as to what we can do here. In this example, we assert that the `value` param has the same content as the value param we passed to the Workflow. #### Queries `TestWorkflowEnvironment` instances have a [`QueryWorkflow()` method](https://pkg.go.dev/go.temporal.io/temporal/internal#TestWorkflowEnvironment.QueryWorkflow) that lets you query the state of the currently running Workflow. For example, suppose you have a Workflow that lets you query the progress of a long running task as shown below. ```go func ProgressWorkflow(ctx workflow.Context, percent int) error { logger := workflow.GetLogger(ctx) err := workflow.SetQueryHandler(ctx, "getProgress", func(input []byte) (int, error) { return percent, nil }) if err != nil { logger.Info("SetQueryHandler failed.", "Error", err) return err } for percent = 0; percent<100; percent++ { // Important! Use `workflow.Sleep()`, not `time.Sleep()`, because Temporal's // test environment doesn't stub out `time.Sleep()`. workflow.Sleep(ctx, time.Second*1) } return nil } ``` This Workflow tracks the current progress of a task in percentage terms, and increments the percentage by 1 every second. Below is how you would write a test case that queries this Workflow. Note that you should always query the Workflow either after `ExecuteWorkflow()` is done or in a `RegisterDelayedCallback()` callback, otherwise you'll get a `runtime error` panic. ```go func (s *UnitTestSuite) Test_ProgressWorkflow() { value := 0 // After 10 seconds plus padding, progress should be 10. // Note that `RegisterDelayedCallback()` doesn't actually make your test wait for 10 seconds! // Temporal's test framework advances time internally, so this test should take < 1 second. s.env.RegisterDelayedCallback(func() { res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(10, value) }, time.Second*10+time.Millisecond*1) s.env.ExecuteWorkflow(ProgressWorkflow, 0) s.True(s.env.IsWorkflowCompleted()) // Once the workflow is completed, progress should always be 100 res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(value, 100) } ``` > **📝 Note:** > > `RegisterDelayedCallback` can also be used to send [Signals](/sending-messages#sending-signals). > When using "Signal-With-Start", set the delay to `0`. #### Debugging You can use a debugger tool provided by your favorite IDE to debug your Workflow Definitions prior to testing or executing them. The Temporal Go SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control. Because of this you can often encounter a `PanicError: Potential deadlock detected` while stepping through Workflow Definitions during debugging. To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to `true` before debugging your Workflow Definition. > **📝 Note:** > > Make sure to set `TEMPORAL_DEBUG` to true only during debugging. > --- # Error handling - Go SDK Source: https://docs.temporal.io/develop/go/best-practices/error-handling > How to handle errors using the Temporal Go SDK. ## Error handling Within a Workflow, an Activity or Child Workflow execution might fail. You can handle errors differently based on the error type. If the Activity returns an error as `errors.New()` or `fmt.Errorf()`, that error is converted into `*temporal.ApplicationError`. If the Activity returns an error as `temporal.NewNonRetryableApplicationError("error message", details)`, that error is returned as `*temporal.ApplicationError`. There are other types of errors such as `*temporal.TimeoutError`, `*temporal.CanceledError` and `*temporal.PanicError`. Here's an example of handling Activity errors within Workflow code that differentiates between different error types. ```go err := workflow.ExecuteActivity(ctx, YourActivity, ...).Get(ctx, nil) if err != nil { var applicationErr *ApplicationError if errors.As(err, &applicationErr) { // retrieve error message workflow.GetLogger(ctx).Info("Application error", "error", applicationErr.Error()) // handle Activity errors (created via NewApplicationError() API) var detailMsg string // assuming Activity return error by NewApplicationError("message", true, "string details") applicationErr.Details(&detailMsg) // extract strong typed details // handle Activity errors (errors created other than using NewApplicationError() API) switch applicationErr.Type() { case "CustomErrTypeA": // handle CustomErrTypeA case CustomErrTypeB: // handle CustomErrTypeB default: // newer version of Activity could return new errors that Workflow was not aware of. } } var canceledErr *CanceledError if errors.As(err, &canceledErr) { // handle cancellation } var timeoutErr *TimeoutError if errors.As(err, &timeoutErr) { // handle timeout, could check timeout type by timeoutErr.TimeoutType() switch err.TimeoutType() { case commonpb.ScheduleToStart: // Handle ScheduleToStart timeout. case commonpb.StartToClose: // Handle StartToClose timeout. case commonpb.Heartbeat: // Handle heartbeat timeout. default: } } var panicErr *PanicError if errors.As(err, &panicErr) { // handle panic, message and call stack are available by panicErr.Error() and panicErr.StackTrace() } } ``` ### Panics and deferred functions In Go, [`defer` schedules cleanup functions and `recover()` catches panics](https://go.dev/blog/defer-panic-and-recover) to prevent them from crashing the program. This doesn't work the same way in Temporal Workflow code — you cannot `recover()` from a panic inside a `defer`. Deferred functions that try to interact with the Temporal SDK during panic unwinding will re-panic immediately. Use `defer` only for local cleanup. Handle Temporal API cleanup through explicit error checks instead. --- # Temporal Go SDK multithreading Source: https://docs.temporal.io/develop/go/best-practices/multithreading > The Temporal Go SDK ensures deterministic multithreading in Workflows using workflow.Go(), avoiding race conditions and eliminating the need for mutexes. The Temporal Go SDK allows you to create additional goroutines (threads) in your Workflows by calling `workflow.Go()`. Native Go threading is never allowed in Workflow code, as it would create determinism errors. You might sometimes need to execute multiple Activities or Child Workflows in parallel and then await the result of all of them. Normally, this would require a lock or [mutex](https://en.wikipedia.org/wiki/Lock_(computer_science)) around some shared data structure to avoid race conditions that could occur when multiple asynchronous operations try to modify the data structure. Although Temporal Workflows run Asynchronously in Go, there is a control in place that ensures only one thread can access at a time. ## How multithreading works Temporal's Go SDK contains a deterministic runner to control the thread execution. This deterministic runner will decide which Workflow thread to run in the right order, and one at a time. Each task will execute in a loop until all threads are blocked. `workflow.Go()` creates a new thread and adds it to this runner. This significantly minimizes the likelihood of race conditions, and eliminates the need to use a mutex. For a complex example, refer to the [Go Particle Swarm Operation Sample](https://github.com/temporalio/samples-go/tree/main/pso). For an example using Signals, refer to the [Go Await Signal Sample](https://github.com/temporalio/samples-go/tree/main/await-signals) ## Static analysis with the workflowcheck tool The Temporal Go SDK also provides a command line tool called [`workflowcheck`](https://github.com/temporalio/sdk-go/blob/main/contrib/tools/workflowcheck/README.md) to statically analyze Workflow Definitions. This can help eliminate potential instances of non-determinism. --- # Testing - Go SDK Source: https://docs.temporal.io/develop/go/best-practices/testing-suite > The Testing section of the Temporal Application development guide details frameworks for Workflow and integration testing. Create end-to-end, integration, unit tests, and more for Workflows and Activities. Each test runs in an isolated environment, ensuring accurate and reliable testing. Discover how to mock and override Activities, test The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks The Temporal Go SDK provides a test framework to facilitate testing Workflow implementations. This framework is suited for implementing unit tests as well as functional tests of the Workflow logic. ## Test setup To run unit tests, we first define a test suite struct that absorbs both the basic suite functionality from [testify](https://pkg.go.dev/github.com/stretchr/testify/suite) via `suite.Suite` and the suite functionality from the Temporal test framework via `testsuite.WorkflowTestSuite`. Because every test in this test suite will test our Workflow, we add a property to our struct to hold an instance of the test environment. This allows us to initialize the test environment in a setup method. For testing Workflows, we use a `testsuite.TestWorkflowEnvironment`. ```go type UnitTestSuite struct { suite.Suite testsuite.WorkflowTestSuite env *testsuite.TestWorkflowEnvironment } ``` Next, we implement a `SetupTest` method to set up a new test environment before each test. Doing so ensures that each test runs in its own isolated sandbox. ```go func (s *UnitTestSuite) SetupTest() { s.env = s.NewTestWorkflowEnvironment() } ``` We also implement an `AfterTest` function where we assert that all the mocks we set up were indeed called by invoking `s.env.AssertExpectations(s.T())`. Timeout for the entire test can be set using `SetTestTimeout` in the Workflow or Activity environment. ```go func (s *UnitTestSuite) AfterTest(suiteName, testName string) { s.env.AssertExpectations(s.T()) } ``` Finally, we create a regular test function recognized by the `go test` command, and pass the struct to `suite.Run`. ```go func TestUnitTestSuite(t *testing.T) { suite.Run(t, new(UnitTestSuite)) } ``` ## Testing Activities An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. Here's an example of a simple Activity that will be referenced in the sections below: ```go package greeting import ( "context" "fmt" ) func Greet(ctx context.Context, name string) (string, error) { return fmt.Sprintf("Hello %s", name), nil } ``` ### Run an Activity If an Activity references its context, you need to mock that context when testing in isolation. You can do that by using [`NewTestActivityEnvironment`](https://pkg.go.dev/go.temporal.io/sdk/testsuite#TestActivityEnvironment) from the [`WorkflowTestSuite`](https://pkg.go.dev/go.temporal.io/sdk/testsuite#WorkflowTestSuite). ```go package greeting import ( "testing" "github.com/stretchr/testify/suite" "go.temporal.io/sdk/testsuite" ) type ActivityTestSuite struct { suite.Suite testsuite.WorkflowTestSuite } func TestActivityTestSuite(t *testing.T) { suite.Run(t, new(ActivityTestSuite)) } func (s *ActivityTestSuite) TestGreet() { env := s.NewTestActivityEnvironment() env.RegisterActivity(Greet) result, err := env.ExecuteActivity(Greet, "Temporal") s.NoError(err) var greeting string s.NoError(result.Get(&greeting)) s.Equal("Hello Temporal!", greeting) } ``` ### Listen to Heartbeats When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them. ```go func (s *ActivityTestSuite) Test_HeatbeatActivity() { testSuite := &testsuite.WorkflowTestSuite{} env := testSuite.NewTestActivityEnvironment() env.RegisterActivity(Greet) var heartBeatCount int env.SetOnActivityHeartbeatListener(func(activityInfo *activity.Info, details converter.EncodedValues) { var val string err := details.Get(&val) s.NoError(err) s.Equal("status-report-to-workflow", val) heartBeatCount++ }) _, err := env.ExecuteActivity(Greet, "Temporal") s.NoError(err) s.Equal(1, heartBeatCount) } ``` ### Cancel an Activity If an Activity is supposed to react to a Cancellation, you can test whether it reacts correctly by canceling it. ```go func (s *UnitTestSuite) Test_CancelActivity() { testSuite := &testsuite.WorkflowTestSuite{} env := testSuite.NewTestActivityEnvironment() ctx, cancel := context.WithCancel(context.Background()) env.SetWorkerOptions(worker.Options{ BackgroundActivityContext: ctx, }) env.RegisterActivity(Greet) done := make(chan struct{}) go func() { defer close(done) // Cancel the activity after 5s time.Sleep(5 * time.Second) cancel() }() _, err := env.ExecuteActivity(Greet, "Temporal") <-done // Expect the activity to return a cancled error s.Error(err) } ``` ## Testing Workflows When running unit tests on Workflows, we want to test the Workflow logic in isolation. The simplest test case we can write is to have the test environment execute the Workflow and then evaluate the results. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_Success() { s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` Calling `s.env.ExecuteWorkflow(...)` executes the Workflow logic and any invoked Activities inside the test process. The first parameter of `s.env.ExecuteWorkflow(...)` contains the Workflow functions, and any subsequent parameters contain values for custom input parameters declared by the Workflow. > Note that unless the Activity invocations are mocked or Activity implementation replaced (see [Activity mocking and overriding](#mock-activities)), the test environment will execute the actual Activity code including any calls to outside services. After executing the Workflow in the above example, we assert that the Workflow ran through completion via the call to `s.env.IsWorkflowCompleted()`. We also assert that no errors were returned by asserting on the return value of `s.env.GetWorkflowError()`. If our Workflow returned a value, we could have retrieved that value via a call to `s.env.GetWorkflowResult(&value)` and had additional asserts on that value. ### Query tests `TestWorkflowEnvironment` instances have a [`QueryWorkflow()` method](https://pkg.go.dev/go.temporal.io/temporal/internal#TestWorkflowEnvironment.QueryWorkflow) that lets you query the state of the currently running Workflow. For example, suppose you have a Workflow that lets you query the progress of a long running task as shown below. ```go func ProgressWorkflow(ctx workflow.Context, percent int) error { logger := workflow.GetLogger(ctx) err := workflow.SetQueryHandler(ctx, "getProgress", func(input []byte) (int, error) { return percent, nil }) if err != nil { logger.Info("SetQueryHandler failed.", "Error", err) return err } for percent = 0; percent<100; percent++ { // Important! Use `workflow.Sleep()`, not `time.Sleep()`, because Temporal's // test environment doesn't stub out `time.Sleep()`. workflow.Sleep(ctx, time.Second*1) } return nil } ``` This Workflow tracks the current progress of a task in percentage terms, and increments the percentage by 1 every second. Below is how you would write a test case that queries this Workflow. Note that you should always query the Workflow either after `ExecuteWorkflow()` is done or in a `RegisterDelayedCallback()` callback, otherwise you'll get a `runtime error` panic. ```go func (s *UnitTestSuite) Test_ProgressWorkflow() { value := 0 // After 10 seconds plus padding, progress should be 10. // Note that `RegisterDelayedCallback()` doesn't actually make your test wait for 10 seconds! // Temporal's test framework advances time internally, so this test should take < 1 second. s.env.RegisterDelayedCallback(func() { res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(10, value) }, time.Second*10+time.Millisecond*1) s.env.ExecuteWorkflow(ProgressWorkflow, 0) s.True(s.env.IsWorkflowCompleted()) // Once the workflow is completed, progress should always be 100 res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(value, 100) } ``` > **📝 Note:** > > `RegisterDelayedCallback` can also be used to send [Signals](/sending-messages#sending-signals). > When using "Signal-With-Start", set the delay to `0`. ### How to mock Activities When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker to test the Workflow logic in isolation. Additionally, you can inject Activity errors during your test runs. The test framework provides two mechanisms that support these scenarios: Activity mocking and Activity overriding. Both of these mechanisms allow you to change the behavior of Activities invoked by your Workflow without the need to modify the actual Workflow code. Let's take a look at a test that simulates a test that fails via the Activity mocking. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure") s.True(s.env.IsWorkflowCompleted()) err := s.env.GetWorkflowError() s.Error(err) var applicationErr *temporal.ApplicationError s.True(errors.As(err, &applicationErr)) s.Equal("SimpleActivityFailure", applicationErr.Error()) } ``` This test simulates the execution of the Activity `SimpleActivity` that is invoked by our Workflow `SimpleWorkflow` returning an error. We accomplish this by setting up a mock on the test environment for the `SimpleActivity` that returns an error. ```go s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) ``` With the mock set up we can now execute the Workflow via the `s.env.ExecuteWorkflow(...)` method and assert that the Workflow completed successfully and returned the expected error. Simply mocking the execution to return a desired value or error is a pretty powerful mechanism to isolate Workflow logic. However, sometimes we want to replace the Activity with an alternate implementation to support a more complex test scenario. Let's assume we want to validate that the Activity gets called with the correct parameters. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( func(ctx context.Context, value string) (string, error) { s.Equal("test_success", value) return value, nil }) s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` In this example, we provide a function implementation as the parameter to `Return`. This allows us to provide an alternate implementation for the Activity `SimpleActivity`. The framework will execute this function whenever the Activity is invoked and pass on the return value from the function as the result of the Activity invocation. Additionally, the framework will validate that the signature of the "mock" function matches the signature of the original Activity function. Since this can be an entire function, there is no limitation as to what we can do here. In this example, we assert that the `value` param has the same content as the value param we passed to the Workflow. ### How to mock Nexus operations Mocking Nexus operations lets you test a Workflow that executes Nexus operations without needing a Nexus handler to run the actual Nexus operation. You can mock the Nexus operation or override its implementation with the test Workflow environment. Consider a test that simulates a Nexus operation call. In this example, the Nexus operation is called `sample-operation`, the input type is `SampleInput`, the output type is `SampleOutput`, and it belongs to the Nexus service `sample-service`. The example below mocks a call to a Nexus synchronous operation, indicated by the returned value type `*nexus.HandlerStartOperationResultSync[T]`. Since `OnNexusOperation` needs to know the operation's name, input type and output type, and you might not have access to the Nexus operation on the handler side, you can use `nexus.NewOperationReference` to create a Nexus operation reference that represents the operation without its implementation (basically, it represents the signature of the Nexus operation). You may also use the operation itself instead of creating the operation reference if you have it available. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_NexusSyncOperation() { s.env.OnNexusOperation( "sample-service", nexus.NewOperationReference[SampleInput, SampleOutput]("sample-operation"), SampleInput{}, workflow.NexusOperationOptions{}, ).Return( &nexus.HandlerStartOperationResultSync[SampleOutput]{ Value: SampleOutput{}, }, nil, // error if you want to simulate an error in the ExecuteOperation call ) // You can also add a delay to return the mock values by calling After(). // Eg: s.env.OnNexusOperation(...).Return(...).After(1*time.Second) s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` Besides the synchronous operations, you can also mock asynchronous operations. The following example demonstrates how to test a Workflow executing a Nexus asynchronous operation. The returned value type in this case must be `*nexus.HandlerStartOperationResultAsync` with an `OperationToken`, which can be any string of your choice. Furthermore, you must call `RegisterNexusAsyncOperationCompletion` to register the result of the asynchronous operation identified by the tuple service name, operation name, and operation token. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_NexusAsyncOperation() { s.env.OnNexusOperation( "sample-service", nexus.NewOperationReference[SampleInput, SampleOutput]("sample-operation"), SampleInput{}, workflow.NexusOperationOptions{}, ).Return( &nexus.HandlerStartOperationResultAsync{ OperationToken: "sample-operation-token", }, nil, // error if you want to simulate an error in the ExecuteOperation call ) err := env.RegisterNexusAsyncOperationCompletion( "sample-service", "sample-operation", "sample-operation-token", // must match the OperationToken above SampleOutput{}, nil, // error if you want to simulate an error in the operation 2*time.Second, // delay to simulate how long the operation takes after it starts ) s.NoError(err) s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` If your Workflow executes multiple Nexus asynchronous operations, you can mock each of them with different operation tokens, and register the completion results using the corresponding operation token. If mocking Nexus operations is not enough, and you need to run some custom logic when the Nexus operation is executed, you can override it as follows. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_NexusSyncOperation() { var SampleOperation = nexus.NewSyncOperation( "sample-operation", func(ctx context.Context, input SampleInput, options nexus.StartOperationOptions) (SampleOutput, error) { // Custom logic here. return SampleOutput{}, nil }, ) service := nexus.NewService("sample-service") s.NoError(service.Register(SampleOperation)) env.RegisterNexusService(service) s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` The following example shows how to override a Nexus asynchronous operation. ```go func (s *UnitTestSuite) Test_SimpleWorkflow_NexusSyncOperation() { SampleHandlerWorkflow := func(_ workflow.Context, input SampleInput) (SampleOutput, error) { // Custom logic here. return SampleOutput{}, nil } SampleOperation := nexus.NewWorkflowRunOperation( "sample-operation", SampleHandlerWorkflow, func(ctx context.Context, input SampleInput, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) { // Custom logic to build client.StartWorkflowOptions. return client.StartWorkflowOptions{}, nil }, ) service := nexus.NewService("sample-service") s.NoError(service.Register(SampleOperation)) env.RegisterNexusService(service) s.env.ExecuteWorkflow(SimpleWorkflow, "test_nexus_operation") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` ### How to skip time Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Skip time automatically The Go SDK test environment automatically skips time when possible. When using the `testsuite.TestWorkflowEnvironment`, time advances automatically whenever there are no Activities running. This means: - Workflow timers (like `workflow.Sleep`) are fast-forwarded - Time doesn't skip while Activities are executing Example: ```go import ( "testing" "go.temporal.io/sdk/testsuite" ) func TestWorkflow_AutoTimeSkipping(t *testing.T) { var ts testsuite.WorkflowTestSuite env := ts.NewTestWorkflowEnvironment() env.ExecuteWorkflow(MyWorkflow) if !env.IsWorkflowCompleted() { t.Fatal("workflow did not complete") } if err := env.GetWorkflowError(); err != nil { t.Fatal(err) } } ``` #### Skip time manually By default, this test suite uses a mock Workflow clock which automatically moves forward to fire the next Timer when the Workflow is blocked. In addition to automatic skipping, you can manually control time progression in the Go test environment. Use `env.RegisterDelayedCallback()` to create a new Timer with a specified `delayDuration` using the mock Workflow clock. When the Timer fires, the callback will be called. This is useful when you want fine-grained control over when Timers fire. Example: ```go package sleepfordays import ( "testing" "time" "github.com/stretchr/testify/mock" "github.com/stretchr/testify/require" "go.temporal.io/sdk/testsuite" ) func TestSleepForDaysWorkflow(t *testing.T) { testSuite := &testsuite.WorkflowTestSuite{} env := testSuite.NewTestWorkflowEnvironment() numActivityCalls := 0 env.RegisterActivity(SendEmailActivity) env.OnActivity(SendEmailActivity, mock.Anything, mock.Anything).Run( func(args mock.Arguments) { numActivityCalls++ }, ).Return(nil) startTime := env.Now() // Time-skip 90 days. env.RegisterDelayedCallback(func() { // Check that the activity has been called 3 times. require.Equal(t, 3, numActivityCalls) // Send the signal to complete the workflow. env.SignalWorkflow("complete", nil) // Expect no more activity calls to have been made - workflow is complete. require.Equal(t, 3, numActivityCalls) // Expect more than 90 days to have passed. require.Equal(t, env.Now().Sub(startTime), time.Hour*24*90) }, time.Hour*24*90) // Execute workflow. env.ExecuteWorkflow(SleepForDaysWorkflow) } ``` ## How to Replay a Workflow Execution Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: Use the [worker.WorkflowReplayer](https://pkg.go.dev/go.temporal.io/sdk/worker#WorkflowReplayer) to replay an existing Workflow Execution from its Event History to replicate errors. For example, the following code retrieves the Event History of a Workflow: ```go import ( "context" "go.temporal.io/api/enums/v1" "go.temporal.io/api/history/v1" "go.temporal.io/sdk/client" ) func GetWorkflowHistory(ctx context.Context, client client.Client, id, runID string) (*history.History, error) { var hist history.History iter := client.GetWorkflowHistory(ctx, id, runID, false, enums.HISTORY_EVENT_FILTER_TYPE_ALL_EVENT) for iter.HasNext() { event, err := iter.Next() if err != nil { return nil, err } hist.Events = append(hist.Events, event) } return &hist, nil } ``` This history can then be used to _replay_. For example, the following code creates a `WorkflowReplayer` and register the `YourWorkflow` Workflow function. Then it calls the `ReplayWorkflowHistory` to _replay_ the Event History and return an error code. ```go import ( "context" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" ) func ReplayWorkflow(ctx context.Context, client client.Client, id, runID string) error { hist, err := GetWorkflowHistory(ctx, client, id, runID) if err != nil { return err } replayer := worker.NewWorkflowReplayer() replayer.RegisterWorkflow(YourWorkflow) return replayer.ReplayWorkflowHistory(nil, hist) } ``` The code above will cause the Worker to re-execute the Workflow's Workflow Function using the original Event History. If a noticeably different code path was followed or some code caused a deadlock, it will be returned in the error code. Replaying a Workflow Execution locally is a good way to see exactly what code path was taken for given input and events. You can replay many Event Histories by registering all the needed Workflow implementation and then calling `ReplayWorkflowHistory` repeatedly. --- # Client - Go SDK Source: https://docs.temporal.io/develop/go/client > This section explains how to implement the Temporal Client with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Temporal Client - [Temporal Client](/develop/go/client/temporal-client) - [Namespaces](/develop/go/client/namespaces) --- # Namespaces - Go SDK Source: https://docs.temporal.io/develop/go/client/namespaces > Register and manage Namespaces in Temporal using CLI or SDK APIs. Isolate Workflow Executions, match development lifecycles, and secure Namespace workflows. This page shows how to do the following: - [Register Namespaces](#register-namespace) - [Manage Namespaces](#manage-namespaces) You can create, update, deprecate, or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs. Use Namespaces to isolate your Workflow Executions according to your needs. For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces. You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line. On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. ### How to register Namespaces Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces. On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. Use [`Register` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) with the `NamespaceClient` interface to register a [Namespace](/namespaces) and set the [Retention Period](/temporal-service/temporal-server#retention-period) for the Workflow Execution Event History for the Namespace. You can also [register Namespaces using the Temporal CLI command-line tool](/cli/command-reference/operator#create). ```go client, err := client.NewNamespaceClient(client.Options{HostPort: ts.config.ServiceAddr}) //... err = client.Register(ctx, &workflowservice.RegisterNamespaceRequest{ Namespace: your-namespace-name, WorkflowExecutionRetentionPeriod: &retention, }) ``` The Retention Period setting using `WorkflowExecutionRetentionPeriod` is mandatory. The minimum value you can set for this period is 1 day. Once registered, set Namespace using `Dial` in a Workflow Client to run your Workflow Executions within that Namespace. See [how to set Namespace in a Client in Go](/develop/go/client/temporal-client#connect-to-temporal-cloud) for details. Note that Namespace registration using this API takes up to 10 seconds to complete. Ensure that you wait for this registration to complete before starting the Workflow Execution against the Namespace. To update your Namespace, use the [`Update` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) with the `NamespaceClient`. To update your Namespace using the Temporal CLI, use the [temporal operator namespace update](/cli/command-reference/operator#update) command. ### How to manage Namespaces You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). - Update information and configuration for a registered Namespace on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace update`](/cli/command-reference/operator#update) Example - Use the [`UpdateNamespace` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) to update configuration on a Namespace. Example ```go //... err = client.Update(context.Background(), &workflowservice.UpdateNamespaceRequest{ Namespace: "your-namespace-name", UpdateInfo: &namespace.UpdateNamespaceInfo{ //updates info for the namespace "your-namespace-name" Description: "updated namespace description", OwnerEmail: "newowner@mail.com", //Data: nil, //State: 0, }, /*other details that you can update: Config: &namespace.NamespaceConfig{ //updates the configuration of the namespace with the following options //WorkflowExecutionRetentionTtl: nil, //BadBinaries: nil, //HistoryArchivalState: 0, //HistoryArchivalUri: "", //VisibilityArchivalState: 0, //VisibilityArchivalUri: "", }, ReplicationConfig: &replication.NamespaceReplicationConfig{ //updates the replication configuration for the namespace //ActiveClusterName: "", //Clusters: nil, //State: 0, }, SecurityToken: "", DeleteBadBinary: "", PromoteNamespace: false, })*/ //... ``` - Get details for a registered Namespace on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace describe`](/cli/command-reference/operator#describe) - Use the [`DescribeNamespace` API](https://pkg.go.dev/go.temporal.io/sdk/client#NamespaceClient) to return information and configuration details for a registered Namespace. Example ```go //... client, err := client.NewNamespaceClient(client.Options{}) //... client.Describe(context.Background(), "default") //... ``` - Get details for all registered Namespaces on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace list`](/cli/command-reference/operator#list) - Use the [`ListNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/operatorservice/v1/service.proto) to return information and configuration details for all registered Namespaces on your Temporal Service. Example ```go //... namespace.Handler.ListNamespaces(context.Context(), &workflowservice.ListNamespacesRequest{ //lists 1 page (1-100) of namespaces on the active Temporal Service. You can set a large PageSize or loop until NextPageToken is nil //PageSize: 0, //NextPageToken: nil, //NamespaceFilter: nil, }) //... ``` - Delete a Namespace: The [`DeleteNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/operatorservice/v1/service.proto) deletes a Namespace. Deleting a Namespace deletes all running and completed Workflow Executions on the Namespace, and removes them from the persistence store and the visibility store. Example: ```go //... client.OperatorService().DeleteNamespace(ctx, &operatorservice.DeleteNamespaceRequest{... //... ``` --- # Temporal Client - Go SDK Source: https://docs.temporal.io/develop/go/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the [Temporal Service](/temporal-service). Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals to Workflow Executions, sending Queries to Workflow Executions, getting the results of a Workflow Execution, and providing Activity Task Tokens. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the Go SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow-execution) - [Get Workflow results](#get-workflow-results) > **⚠️ Caution:** > > A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a > Temporal Client inside an Activity to communicate with a Temporal Service. > ## Connect to development Temporal Service Use the [`Dial()`](https://pkg.go.dev/go.temporal.io/sdk/client#Dial) API available in the [`go.temporal.io/sdk/client`](https://pkg.go.dev/go.temporal.io/sdk/client) package to create a [`Client`](https://pkg.go.dev/go.temporal.io/sdk/client#Client). The `Dial()` API expects connection options such as the Temporal Server address, the Namespace to connect to, and Transport Layer Security (TLS) configuration. You can specify these options in the function call, or specify them using environment variables or a configuration file. We recommend you use environment variables or a configuration file to manage these connection options securely. > **ℹ️ Info:** > Versioning Requirements > > Environment variable and configuration file support were added in Go SDK v1.28.0. > When you are running a Temporal Service locally, such as the [Temporal CLI](/cli/command-reference/server#start-dev), the connection options you must provide are minimal. If you don't provide [`HostPort`](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions), the Client defaults the address and port number to `127.0.0.1:7233`, which is the port of the development Temporal Service. If you don't set a custom Namespace name in the Namespace field, the client connects to the default Namespace. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml`. For a list of all available configuration options, refer to [Environment Configuration](/references/client-environment-configuration) > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly # disabled = false # Use certificate files for mTLS client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a specific profile from the configuration file as follows: ```go package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Load a specific profile from the TOML config file. // This requires a [profile.prod] section in your config. opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{ ConfigFileProfile: "prod", }) if err != nil { log.Fatalf("Failed to load 'prod' profile: %v", err) } c, err := client.Dial(opts) if err != nil { log.Fatalf("Failed to connect using 'prod' profile: %v", err) } defer c.Close() fmt.Printf("✅ Connected to Temporal namespace %q on %s using 'prod' profile\n", c.Options().Namespace, c.Options().HostPort) } ``` **Environment Variables** Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. ```go package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Loads the "default" profile from the standard location and environment variables. c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalf("Failed to create client: %v", err) } defer c.Close() fmt.Printf("✅ Connected to Temporal namespace %q on %s\n", c.Options().Namespace, c.Options().HostPort) } ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. ```go package main import ( "context" "encoding/json" "log" "net/http" "documentation-samples-go/yourapp" "go.temporal.io/sdk/client" ) func main() { // Create a Temporal Client to communicate with the Temporal Cluster. // A Temporal Client should be created once per process. temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() // Start an HTTP server and listen on /start http.HandleFunc("/start", func(w http.ResponseWriter, r *http.Request) { startWorkflowHandler(w, r, temporalClient) }) err = http.ListenAndServe(":8091", nil) if err != nil { log.Fatalln("Unable to run http server", err) } } ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace and Account ID_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the `LoadClientOptions` function in the `envconfig` package to create a Temporal Client using the `cloud` profile as follows: ```go {13-17} package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Replace with the actual path to your TOML file. configFilePath := "/Users/yourname/.config/my-app/temporal.toml" opts, err := envconfig.LoadClientOptions(envconfig.LoadClientOptionsRequest{ ConfigFilePath: configFilePath, ConfigFileProfile: "cloud", }) if err != nil { log.Fatalf("Failed to load client config from custom file: %v", err) } c, err := client.Dial(opts) if err != nil { log.Fatalf("Failed to connect using custom config file: %v", err) } defer c.Close() fmt.Printf("✅ Connected using custom config at: %s\n", configFilePath) } ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your Go application. Import the `envconfig` package to set connection options for the Temporal Client using environment variables. The `MustLoadDefaultClientOptions` function will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. ```go {13} package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Loads the "default" profile from the standard location and environment variables. c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalf("Failed to create client: %v", err) } defer c.Close() fmt.Printf("✅ Connected to Temporal Service") } ``` **Code** You can also provide connections options in your Go code directly. When instantiating a Temporal `client` in your Temporal Go SDK code, provide the following `clientOptions`: ```go clientOptions := client.Options{ HostPort: , Namespace: ., ConnectionOptions: client.ConnectionOptions{TLS: &tls.Config{}}, Credentials: client.NewAPIKeyStaticCredentials(apiKey), } c, err := client.Dial(clientOptions) ``` To update an API key, use the Go `context` object: ```go // Assuming client Credentials created with var myKey string creds := client.NewAPIKeyDynamicCredentials( func(context.Context) (string, error) { return myKey, nil }) // Update by replacing myKey = myKeyUpdated ``` To rotate an mTLS client certificate without restarting your Worker, set `GetClientCertificate` on the `tls.Config` instead of setting `Certificates` directly. Go's standard library `crypto/tls` package calls this function on every new connection, so it always picks up the current certificate: ```go clientCertPath := "/path/to/client.crt" clientKeyPath := "/path/to/client.key" clientOptions := client.Options{ HostPort: , Namespace: ., ConnectionOptions: client.ConnectionOptions{ TLS: &tls.Config{ GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) { cert, err := tls.LoadX509KeyPair(clientCertPath, clientKeyPath) if err != nil { return nil, err } return &cert, nil }, }, }, } c, err := client.Dial(clientOptions) ``` Rotate the certificate by overwriting the files at `clientCertPath` and `clientKeyPath`; the next time the connection re-establishes (for example, after Temporal Cloud's periodic connection recycling), `GetClientCertificate` reads the new files. This works because `ConnectionOptions.TLS` is a real `*tls.Config`, so any option `crypto/tls` supports is available. See this [reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) for a full walkthrough, including automating certificate issuance with cert-manager on Kubernetes. You can use a combination of environment variables, configuration files, and code to set connection options. For example, you can load a base configuration from environment variables or a configuration file, and then override specific options in code. For example, the following code snippet loads the base configuration from environment variables and the default profile with `envconfig.MustLoadDefaultClientOptions()`. It then overrides the `HostPort` and `Namespace` options programmatically. Refer to [Client Options type in the Go SDK](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions) for a list of all available connection options you can override. ```go package main import ( "fmt" "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" ) func main() { // Load the base configuration (e.g., from the default profile). opts := envconfig.MustLoadDefaultClientOptions() // Apply overrides programmatically. opts.HostPort = "localhost:7233" opts.Namespace = "test-namespace" c, err := client.Dial(opts) if err != nil { log.Fatalf("Failed to connect with overridden options: %v", err) } defer c.Close() fmt.Printf("✅ Connected with overridden config to: %s in namespace: %s\n", opts.HostPort, opts.Namespace) } ``` #### v1.26.0 to v1.32.x Create an initial connection: ```go clientOptions := client.Options{ HostPort: , Namespace: ., ConnectionOptions: client.ConnectionOptions{ TLS: &tls.Config{}, DialOptions: []grpc.DialOption{ grpc.WithUnaryInterceptor( func(ctx context.Context, method string, req any, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error { return invoker( metadata.AppendToOutgoingContext(ctx, "temporal-namespace", .), method, req, reply, cc, opts..., ) }, ), }, }, Credentials: client.NewAPIKeyStaticCredentials(apiKey), } c, err := client.Dial(clientOptions) if err != nil { log.Fatalf("error creating temporal client: %v", err) } ``` Update an API key: ```go // Assuming client Credentials created with var myKey string creds := client.NewAPIKeyDynamicCredentials( func(context.Context) (string, error) { return myKey, nil }) // Just update by replacing myKey = myKeyUpdated ``` #### pre v1.26.0 Create an initial connection: ```go // Create headers provider type APIKeyProvider struct { APIKey string Namespace string } func (a *APIKeyProvider) GetHeaders(context.Context) (map[string]string, error) { return map[string]string{"Authorization": "Bearer " + a.APIKey, "temporal-namespace": a.Namespace}, nil } // Use headers provider apiKeyProvider := &APIKeyProvider{APIKey: , Namespace: .} c, err := client.Dial(client.Options{ HostPort: , Namespace: ., HeadersProvider: apiKeyProvider, ConnectionOptions: client.ConnectionOptions{TLS: &tls.Config{ }}, }) ``` Update an API key: ```go apiKeyProvider.APIKey = myKeyUpdated ``` ## Start Workflow Execution [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from within another Workflow Execution, use either the [Child Workflow](/develop/go/workflows/child-workflows) or External Workflow APIs. See the [Customize Workflow Type](/develop/go/workflows/basics#customize-workflow-type) section to see how to customize the name of the Workflow Type. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. To spawn a [Workflow Execution](/workflow-execution), use the `ExecuteWorkflow()` method on the Go SDK [`Client`](https://pkg.go.dev/go.temporal.io/sdk/client#Client). The `ExecuteWorkflow()` API call requires an instance of [`context.Context`](https://pkg.go.dev/context#Context), an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions), a Workflow Type name, and all variables to be passed to the Workflow Execution. The `ExecuteWorkflow()` call returns a Future, which can be used to get the result of the Workflow Execution. ```go package main import ( // ... "go.temporal.io/sdk/client" ) func main() { temporalClient, err := client.Dial(client.Options{}) if err != nil { // ... } defer temporalClient.Close() // ... workflowOptions := client.StartWorkflowOptions{ // ... } workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param) if err != nil { // ... } // ... } func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (YourWorkflowResponse, error) { // ... } ``` If the invocation process has access to the function directly, then the Workflow Type name parameter can be passed as if the function name were a variable, without quotations. ```go workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param) ``` If the invocation process does not have direct access to the statically defined Workflow Definition, for example, if the Workflow Definition is in an un-importable package, or it is written in a completely different language, then the Workflow Type can be provided as a `string`. ```go workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, "YourWorkflowDefinition", param) ``` ### Set Workflow Task Queue In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue). For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task Queue name. Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `TaskQueue` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `string` - Default: None, this is a required field to be set by the developer ```go workflowOptions := client.StartWorkflowOptions{ // ... TaskQueue: "your-task-queue", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` You can configure Task Queues that are host-specific, Worker-specific or Workflow-specific to distribute your application load. For more information, refer to [Task Queues Processing Tuning](/develop/worker-performance/task-queues#task-queues-processing-tuning) and [Worker Versioning](/worker-versioning). ### Set custom Workflow Id Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity identifier, such as an order identifier or customer identifier. Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `ID` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `string` - Default: System generated UUID ```go workflowOptions := client.StartWorkflowOptions{ // ... ID: "Your-Custom-Workflow-Id", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` ### Go StartWorkflowOptions reference Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, and pass the instance to the `ExecuteWorkflow` call. The following fields are available: | Field | Required | Type | | --------------------------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------- | | [`ID`](#id) | No | `string` | | [`TaskQueue`](#taskqueue) | **Yes** | `string` | | [`WorkflowExecutionTimeout`](#workflowexecutiontimeout) | No | `time.Duration` | | [`WorkflowRunTimeout`](#workflowruntimeout) | No | `time.Duration` | | [`WorkflowTaskTimeout`](#workflowtasktimeout) | No | `time.Duration` | | [`WorkflowIDReusePolicy`](#workflowidreusepolicy) | No | [`WorkflowIdReusePolicy`](https://pkg.go.dev/go.temporal.io/api/enums/v1#WorkflowIdReusePolicy) | | [`WorkflowExecutionErrorWhenAlreadyStarted`](#workflowexecutionerrorwhenalreadystarted) | No | `bool` | | [`RetryPolicy`](#retrypolicy) | No | [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) | | [`CronSchedule`](#cronschedule) | No | `string` | | [`Memo`](#memo) | No | `map[string]interface{}` | | [`SearchAttributes`](#searchattributes) | No | `map[string]interface{}` | #### ID Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity identifier, such as an order identifier or customer identifier. Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `ID` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `string` - Default: System generated UUID ```go workflowOptions := client.StartWorkflowOptions{ // ... ID: "Your-Custom-Workflow-Id", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### TaskQueue Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.10.0/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `TaskQueue` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `string` - Default: None; this is a required field to be set by the developer ```go workflowOptions := client.StartWorkflowOptions{ // ... TaskQueue: "your-task-queue", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### WorkflowExecutionTimeout Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `WorkflowExecutionTimeout` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `time.Duration` - Default: Unlimited ```go workflowOptions := client.StartWorkflowOptions{ // ... WorkflowExecutionTimeout: time.Hours * 24 * 365 * 10, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### WorkflowRunTimeout Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `WorkflowRunTimeout` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `time.Duration` - Default: Same as [`WorkflowExecutionTimeout`](#workflowexecutiontimeout) ```go workflowOptions := client.StartWorkflowOptions{ WorkflowRunTimeout: time.Hours * 24 * 365 * 10, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### WorkflowTaskTimeout Create an instance of [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `WorkflowTaskTimeout` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `time.Duration` - Default: `time.Seconds * 10` ```go workflowOptions := client.StartWorkflowOptions{ WorkflowTaskTimeout: time.Second * 10, //... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### WorkflowIDReusePolicy - Type: [WorkflowIdReusePolicy](https://pkg.go.dev/go.temporal.io/api/enums/v1#WorkflowIdReusePolicy) - Default: `enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE` Set a value from the `go.temporal.io/api/enums/v1` package. ```go workflowOptions := client.StartWorkflowOptions{ WorkflowIdReusePolicy: enums.WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### WorkflowExecutionErrorWhenAlreadyStarted - Type: `bool` - Default: `false` ```go workflowOptions := client.StartWorkflowOptions{ WorkflowExecutionErrorWhenAlreadyStarted: false, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### RetryPolicy Create an instance of a [RetryPolicy](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) from the `go.temporal.io/sdk/temporal` package and provide it as the value to the `RetryPolicy` field of the instance of `StartWorkflowOptions`. - Type: [RetryPolicy](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) - Default: None ```go retrypolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, } workflowOptions := client.StartWorkflowOptions{ RetryPolicy: retrypolicy, // ... } workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### CronSchedule - Type: `string` - Default: None ```go workflowOptions := client.StartWorkflowOptions{ CronSchedule: "15 8 * * *", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` [Sample](https://github.com/temporalio/samples-go/tree/main/cron) #### Memo - Type: `map[string]interface{}` - Default: Empty ```go workflowOptions := client.StartWorkflowOptions{ Memo: map[string]interface{}{ "description": "Test search attributes workflow", }, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` #### SearchAttributes - Type: `map[string]interface{}` - Default: Empty. These are the corresponding [Search Attribute value types](/search-attribute#supported-types) in Go: - Keyword = string - Int = int64 - Double = float64 - Bool = bool - Datetime = time.Time - Text = string ```go searchAttributes := map[string]interface{}{ "CustomIntField": 1, "MiscData": "yellow", } workflowOptions := client.StartWorkflowOptions{ SearchAttributes: searchAttributes, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` ### Get Workflow results If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. The `ExecuteWorkflow` call returns an instance of [`WorkflowRun`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowRun), which is the `workflowRun` variable in the following line. ```go workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, app.YourWorkflowDefinition, param) if err != nil { // ... } // ... } ``` The instance of `WorkflowRun` has the following three methods: - `GetWorkflowID()`: Returns the Workflow Id of the invoked Workflow Execution. - `GetRunID()`: Always returns the Run Id of the initial Run (See [Continue As New](#)) in the series of Runs that make up the full Workflow Execution. - `Get`: Takes a pointer as a parameter and populates the associated variable with the Workflow Execution result. To wait on the result of Workflow Execution in the same process that invoked it, call `Get()` on the instance of `WorkflowRun` that is returned by the `ExecuteWorkflow()` call. ```go workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition, param) if err != nil { // ... } var result YourWorkflowResponse err = workflowRun.Get(context.Background(), &result) if err != nil { // ... } // ... } ``` However, the result of a Workflow Execution can be obtained from a completely different process. All that is needed is the [Workflow Id](#). (A [Run Id](#) is optional if more than one closed Workflow Execution has the same Workflow Id.) The result of the Workflow Execution is available for as long as the Workflow Execution Event History remains in the system. Call the `GetWorkflow()` method on an instance of the Go SDK Client and pass it the Workflow Id used to spawn the Workflow Execution. Then call the `Get()` method on the instance of `WorkflowRun` that is returned, passing it a pointer to populate the result. ```go // ... workflowID := "Your-Custom-Workflow-Id" workflowRun := c.GetWorkflow(context.Background, workflowID) var result YourWorkflowResponse err = workflowRun.Get(context.Background(), &result) if err != nil { // ... } // ... ``` **Get last completion result** In the case of a [Temporal Cron Job](/cron-job), you might need to get the result of the previous Workflow Run and use it in the current Workflow Run. To do this, use the [`HasLastCompletionResult`](https://pkg.go.dev/go.temporal.io/sdk/workflow#HasLastCompletionResult) and [`GetLastCompletionResult`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLastCompletionResult) APIs, available from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package, directly in your Workflow code. ```go type CronResult struct { Count int } func YourCronWorkflowDefinition(ctx workflow.Context) (CronResult, error) { count := 1 if workflow.HasLastCompletionResult(ctx) { var lastResult CronResult if err := workflow.GetLastCompletionResult(ctx, &lastResult); err == nil { count = count + lastResult.Count } } newResult := CronResult { Count: count, } return newResult, nil } ``` This will work even if one of the cron Workflow Runs fails. The next Workflow Run gets the result of the last successfully Completed Workflow Run. --- # Data handling - Go SDK Source: https://docs.temporal.io/develop/go/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/go/data-handling/data-conversion) | [PayloadCodec](/develop/go/data-handling/data-encryption) | [ExternalStorage](/develop/go/data-handling/external-storage) | | ------------------------- | ------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------ | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | Offload large payloads to external store | | **Default** | JSON serialization | None (passthrough) | None (all payloads are stored in Event History) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - Go SDK Source: https://docs.temporal.io/develop/go/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the Go SDK, including composite data converters and custom type examples. Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this: ![Temporal converter architecture](/img/info/converter-architecture.png) ## Use a custom Payload Converter Use a [Composite Data Converter](https://pkg.go.dev/go.temporal.io/sdk/converter#CompositeDataConverter) to apply custom, type-specific Payload Converters in a specified order. Defining a new Composite Data Converter is not always necessary to implement custom data handling. You can override the default Converter with a custom Codec, but a Composite Data Converter may be necessary for complex Workflow logic. `NewCompositeDataConverter` creates a new instance of `CompositeDataConverter` from an ordered list of type-specific Payload Converters. The following type-specific Payload Converters are available in the Go SDK, listed in the order that they are applied by the default Data Converter: - [NewNilPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#NilPayloadConverter.ToString) - [NewByteSlicePayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ByteSlicePayloadConverter) - [NewProtoJSONPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ProtoJSONPayloadConverter) - [NewProtoPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#ProtoPayloadConverter) - [NewJSONPayloadConverter()](https://pkg.go.dev/go.temporal.io/sdk/converter#JSONPayloadConverter) The order in which the Payload Converters are applied is important because during serialization the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-nil Payload. To set your custom Payload Converter, use [`NewCompositeDataConverter`](https://pkg.go.dev/go.temporal.io/sdk/converter#NewCompositeDataConverter) and set it as the Data Converter in the Client options. - To replace the default Data Converter with a custom `NewCompositeDataConverter`, use the following. ```go dataConverter := converter.NewCompositeDataConverter(YourCustomPayloadConverter()) ``` - To add your custom type conversion to the default Data Converter, use the following to keep the defaults but set yours just before the default JSON fall through. ```go dataConverter := converter.NewCompositeDataConverter( converter.NewNilPayloadConverter(), converter.NewByteSlicePayloadConverter(), converter.NewProtoJSONPayloadConverter(), converter.NewProtoPayloadConverter(), YourCustomPayloadConverter(), converter.NewJSONPayloadConverter(), ) ``` --- # Payload encryption - Go SDK Source: https://docs.temporal.io/develop/go/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Go SDK. Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data. Implementers can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. ## Use a custom Payload Codec **Step 1: Create a custom Payload Codec** Create a custom [PayloadCodec](https://pkg.go.dev/go.temporal.io/sdk/converter#PayloadCodec) implementation and define your encryption/compression and decryption/decompression logic in the `Encode` and `Decode` functions. The Payload Codec converts bytes to bytes. It must be used in an instance of [CodecDataConverter](https://pkg.go.dev/go.temporal.io/sdk/converter#CodecDataConverter) that wraps a Data Converter to do the [Payload](/dataconversion#payload) conversions, and applies the custom encoding and decoding in `PayloadCodec` to the converted Payloads. The following example from the [Data Converter sample](https://github.com/temporalio/samples-go/blob/main/codec-server/data_converter.go) shows how to create a custom `NewCodecDataConverter` that wraps an instance of a Data Converter with a custom `PayloadCodec`. ```go // Create an instance of Data Converter with your codec. var DataConverter = converter.NewCodecDataConverter( converter.GetDefaultDataConverter(), NewPayloadCodec(), ) //... // Create an instance of PayloadCodec. func NewPayloadCodec() converter.PayloadCodec { return &Codec{} } ``` Implement your encryption/compression logic in the `Encode` function and the decryption/decompression logic in the `Decode` function in your custom `PayloadCodec`, as shown in the following example. ```go // Codec implements converter.PayloadEncoder for snappy compression. type Codec struct{} // Encode implements converter.PayloadCodec.Encode. func (Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) { result := make([]*commonpb.Payload, len(payloads)) for i, p := range payloads { // Marshal proto origBytes, err := p.Marshal() if err != nil { return payloads, err } // Compress b := snappy.Encode(nil, origBytes) result[i] = &commonpb.Payload{ Metadata: map[string][]byte{converter.MetadataEncoding: []byte("binary/snappy")}, Data: b, } } return result, nil } // Decode implements converter.PayloadCodec.Decode. func (Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) { result := make([]*commonpb.Payload, len(payloads)) for i, p := range payloads { // Decode only if it's our encoding if string(p.Metadata[converter.MetadataEncoding]) != "binary/snappy" { result[i] = p continue } // Uncompress b, err := snappy.Decode(nil, p.Data) if err != nil { return payloads, err } // Unmarshal proto result[i] = &commonpb.Payload{} err = result[i].Unmarshal(b) if err != nil { return payloads, err } } return result, nil } ``` ### Handle Payload Codec failures Payload Codec methods can run during Workflow Task processing or in a Client or Activity context. Handle transient failures within the Payload Codec when possible, such as with bounded retries. During Workflow Task processing, the SDK handles an error returned by `Encode` or `Decode` according to the operation. Some operations convert the error to a panic and apply the Worker's configured [`WorkflowPanicPolicy`](https://pkg.go.dev/go.temporal.io/sdk/worker#WorkflowPanicPolicy); other returned errors can fail the Workflow Execution. To apply the policy consistently, panic from the Payload Codec instead of returning an error. With the default [`BlockWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/worker#BlockWorkflow) policy, the panic fails the current Workflow Task so that the Temporal Service can retry it. With the [`FailWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/worker#FailWorkflow) policy, the panic instead fails the Workflow Execution. Panicking outside Workflow Task processing does not request a Workflow Task retry. **Step 2: Set Data Converter to use custom Payload Codec.** Set your custom `PayloadCodec` with an instance of `DataConverter` in your `Dial` client options that you use to create the client. The following example shows how to set your custom Data Converter from a package called `mycodecpackage`. ```go //... c, err := client.Dial(client.Options{ // Set DataConverter here to ensure that Workflow inputs and results are // encoded as required. DataConverter: mycodecpackage.DataConverter, }) //... ``` - Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. - Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. For reference, see the [Encryption](https://github.com/temporalio/samples-go/tree/main/encryption) sample. ### Using a Codec Server A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. For reference, see the [Codec server](https://github.com/temporalio/samples-go/tree/main/codec-server) sample. --- # External Storage - Go SDK Source: https://docs.temporal.io/develop/go/data-handling/external-storage > Offload large payloads to external storage using the claim check pattern in the Go SDK. > **Public Preview** > APIs and configuration may change before General Availability. Join the [#large-payloads Slack > channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for help. The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how to set up External Storage with Amazon S3 or Google Cloud Storage, and how to implement a custom storage driver. For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage). ## Store and retrieve large payloads with Amazon S3 or Google Cloud Storage The Go SDK includes storage drivers for Amazon S3 and Google Cloud Storage. Select your storage backend in the tabs that follow. Only the driver setup differs between the two. Everything after that is the same. ### Prerequisites - A bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see [Durable External Storage](/external-storage#durable-external-storage). - Bucket credentials on both your Temporal Client and your Workers, since each reaches the bucket directly. Refer to [Amazon S3 access control](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html) or [Cloud Storage IAM permissions](https://cloud.google.com/storage/docs/access-control/iam-permissions) for how to grant access. - Permission to write objects on processes that store payloads, and to read objects on processes that retrieve them. Storing also requires read permission, because the drivers check whether an object already exists before uploading. - Install the driver module, the client adapter for your cloud provider's SDK, and that SDK: **Amazon S3** ```sh go get go.temporal.io/sdk/contrib/aws/s3driver \ go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 \ github.com/aws/aws-sdk-go-v2/config \ github.com/aws/aws-sdk-go-v2/service/s3 ``` **Google Cloud Storage** ```sh go get go.temporal.io/sdk/contrib/gcp/gcsdriver \ go.temporal.io/sdk/contrib/gcp/gcsdriver/gcssdk \ cloud.google.com/go/storage ``` ### Procedure 1. Create a storage client, wrap it in the matching driver client, and pass the result to the driver. Each cloud SDK picks up your standard credentials from the environment: **Amazon S3** The AWS SDK reads [environment variables, an IAM role, or your AWS config file](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html). [features/snippets/external_storage/s3_setup/s3_driver_create.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.go) ```go cfg, err := config.LoadDefaultConfig(context.Background(), config.WithRegion("us-east-2"), ) if err != nil { log.Fatalf("load AWS config: %v", err) } driver, err := s3driver.NewDriver(s3driver.Options{ Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)), Bucket: s3driver.StaticBucket("my-temporal-payloads"), }) if err != nil { log.Fatalf("create S3 driver: %v", err) } ``` **Google Cloud Storage** The Google Cloud SDK reads [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). [features/snippets/external_storage/gcs_setup/gcs_driver_create.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/gcs_setup/gcs_driver_create.go) ```go gcsClient, err := storage.NewClient(context.Background()) if err != nil { log.Fatalf("create GCS client: %v", err) } driver, err := gcsdriver.NewDriver(gcsdriver.Options{ Client: gcssdk.NewClient(gcsClient), Bucket: gcsdriver.StaticBucket("my-temporal-payloads"), }) if err != nil { log.Fatalf("create GCS driver: %v", err) } ``` To route payloads to different buckets at runtime, pass a `BucketFunc` as `Bucket` instead of using `StaticBucket`. The function receives the store context and the payload, and returns a bucket name. 2. Configure the driver on `ExternalStorage` and pass it in your Client options. This step is the same for every driver: [features/snippets/external_storage/s3_setup/s3_external_storage_setup.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_external_storage_setup.go) ```go c, err := client.Dial(client.Options{ HostPort: "localhost:7233", ExternalStorage: converter.ExternalStorage{ Drivers: []converter.StorageDriver{driver}, }, }) if err != nil { log.Fatalf("connect to Temporal: %v", err) } defer c.Close() w := worker.New(c, "my-task-queue", worker.Options{}) ``` A Worker inherits this configuration from the Client it is created with. When your Workers run in their own process, repeat this setup there. By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `PayloadSizeThreshold` option, even setting it to 1 to externalize all payloads regardless of size. Refer to [Configure payload size threshold](#configure-payload-size-threshold) for more information. All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business logic. Both drivers upload and download payloads concurrently, and verify a SHA-256 hash of the contents on retrieve. Each driver rejects any single payload larger than `MaxPayloadSize`, which defaults to 50 MiB. The hash forms part of the object key, along with the Namespace, Workflow Id, and Run Id. When one Workflow Run passes the same payload to several Activities, the key is identical each time, so the driver uploads the payload once and the later writes reuse that object. A different Run, Workflow, or Namespace produces a different key, so it stores its own copy even when the bytes are identical. Storage therefore scales with the number of Runs, not with how many times a Run passes a payload around. The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage failures. ## Implement a custom storage driver If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver. Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and [Lifecycle management](/external-storage#lifecycle) for retention requirements. The following example shows a custom driver that uses local disk as the backing store. This example is for local development and testing only. In production, use a durable storage system that is accessible to all Workers: [features/snippets/external_storage/custom_driver/custom_storage_driver.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/custom_driver/custom_storage_driver.go) ```go type LocalDiskStorageDriver struct { storeDir string } func NewLocalDiskStorageDriver(storeDir string) converter.StorageDriver { return &LocalDiskStorageDriver{storeDir: storeDir} } func (d *LocalDiskStorageDriver) Name() string { return "my-local-disk" } func (d *LocalDiskStorageDriver) Type() string { return "local-disk" } func (d *LocalDiskStorageDriver) Store( ctx converter.StorageDriverStoreContext, payloads []*commonpb.Payload, ) ([]converter.StorageDriverClaim, error) { dir := d.storeDir switch info := ctx.Target.(type) { case converter.StorageDriverWorkflowInfo: if info.WorkflowID != "" { dir = filepath.Join(d.storeDir, info.Namespace, info.WorkflowID) } case converter.StorageDriverActivityInfo: // StorageDriverActivityInfo is only used for standalone (non-workflow-bound) // activities. Activities started by a workflow use StorageDriverWorkflowInfo. if info.ActivityID != "" { dir = filepath.Join(d.storeDir, info.Namespace, info.ActivityID) } } if err := os.MkdirAll(dir, 0o755); err != nil { return nil, fmt.Errorf("create store directory: %w", err) } claims := make([]converter.StorageDriverClaim, len(payloads)) for i, payload := range payloads { key := uuid.NewString() + ".bin" filePath := filepath.Join(dir, key) data, err := proto.Marshal(payload) if err != nil { return nil, fmt.Errorf("marshal payload: %w", err) } if err := os.WriteFile(filePath, data, 0o644); err != nil { return nil, fmt.Errorf("write payload: %w", err) } claims[i] = converter.StorageDriverClaim{ ClaimData: map[string]string{"path": filePath}, } } return claims, nil } func (d *LocalDiskStorageDriver) Retrieve( ctx converter.StorageDriverRetrieveContext, claims []converter.StorageDriverClaim, ) ([]*commonpb.Payload, error) { payloads := make([]*commonpb.Payload, len(claims)) for i, claim := range claims { filePath := claim.ClaimData["path"] data, err := os.ReadFile(filePath) if err != nil { return nil, fmt.Errorf("read payload: %w", err) } payload := &commonpb.Payload{} if err := proto.Unmarshal(data, payload); err != nil { return nil, fmt.Errorf("unmarshal payload: %w", err) } payloads[i] = payload } return payloads, nil } ``` The following sections walk through the key parts of the driver implementation. ### 1. Implement the StorageDriver interface A custom driver implements the `converter.StorageDriver` interface with four methods: - `Name()` returns a unique string that identifies the driver instance. The SDK stores this name in the claim check reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`. - `Type()` returns a string that identifies the driver implementation. Unlike `Name()`, this must be the same across all instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` would both return `"aws.s3driver"` as their type, while the local disk driver in the custom driver code sample returns `"local-disk"`. - `Store()` receives a slice of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string key-value pairs that the driver uses to locate the payload later. - `Retrieve()` receives the claims that `Store()` produced and returns the original payloads. ### 2. Store payloads In `Store()`, marshal each Payload protobuf message to bytes with `proto.Marshal(payload)` and write the bytes to your storage system. The application data has already been serialized by the [Payload Converter](/develop/go/data-handling/data-conversion) and [Payload Codec](/develop/go/data-handling/data-encryption) before it reaches the driver. See the [data conversion pipeline](/external-storage#data-pipeline) for more details. Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `ctx.Target` provides identity information (namespace, Workflow ID) depending on the operation. Use a type switch on `StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the concrete values. Consider structuring your storage keys to include this information so that you can identify which Workflow owns each payload. ### 3. Retrieve payloads In `Retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message with `proto.Unmarshal(data, payload)`. The Payload Converter handles deserializing the application data after the driver returns the payload. ### 4. Configure the Client Pass an `ExternalStorage` struct with your driver in the Client options: ```go c, err := client.Dial(client.Options{ ExternalStorage: converter.ExternalStorage{ Drivers: []converter.StorageDriver{NewLocalDiskStorageDriver("/tmp/temporal-payload-store")}, }, }) ``` You can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services. ## Configure payload size threshold You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `PayloadSizeThreshold` option, or set it to 1 to externalize all payloads regardless of size. A value of 0 is interpreted as the default (256 KiB). Payloads smaller than the threshold stay inline in Event History. The size compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data. [features/snippets/external_storage/threshold/threshold_config.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/threshold/threshold_config.go) ```go c, err := client.Dial(client.Options{ ExternalStorage: converter.ExternalStorage{ Drivers: []converter.StorageDriver{driver}, PayloadSizeThreshold: 1, }, }) ``` ## Use multiple storage drivers When you register multiple drivers, you must provide a `DriverSelector` that implements the `StorageDriverSelector` interface. The selector chooses which driver stores each payload. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when migrating between storage backends. Return `nil` from the selector to keep a specific payload inline in Event History. Multiple drivers are useful in scenarios such as: - Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old driver remains available for retrieving existing claims. - Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver based on the runtime environment. Every registered driver needs a distinct name. The S3 and GCS drivers default to `"aws.s3driver"` and `"gcp.gcsdriver"`, so you can register one of each without extra configuration. Registering two drivers of the same type requires setting the `DriverName` option on at least one of them. The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` is only registered so the Worker can retrieve payloads that were previously stored with it: [features/snippets/external_storage/multiple_drivers/multiple_drivers.go](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/multiple_drivers/multiple_drivers.go) ```go type PreferredSelector struct { preferred converter.StorageDriver } func (s *PreferredSelector) SelectDriver( ctx converter.StorageDriverStoreContext, payload *commonpb.Payload, ) (converter.StorageDriver, error) { return s.preferred, nil } func MultipleDriversSetup(preferredDriver, legacyDriver converter.StorageDriver) converter.ExternalStorage { return converter.ExternalStorage{ Drivers: []converter.StorageDriver{preferredDriver, legacyDriver}, DriverSelector: &PreferredSelector{preferred: preferredDriver}, } } ``` ## Multi-region durability with Amazon S3 To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an [S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the driver at the MRAP ARN instead of a bucket name. See [Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs. In code, the only change is the value you pass to `s3driver.StaticBucket`: ```go driver, err := s3driver.NewDriver(s3driver.Options{ Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)), Bucket: s3driver.StaticBucket("arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap"), }) ``` The AWS SDK for Go v2 automatically uses [SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html) signing when the bucket value is an MRAP ARN, so no additional client configuration is required. --- # Integrations - Go SDK Source: https://docs.temporal.io/develop/go/integrations > This section covers integrations with the Go SDK The following integrations are available for the Temporal Go SDK. You can also use the Temporal Go SDK's [Plugin system](/develop/plugins-guide) to build your own integrations. - [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#go) — Monitor and evaluate AI application performance with Braintrust observability. _(Go · Agent observability)_ - [Google ADK](/develop/go/integrations/google-adk) — Run Google ADK agents with durable execution using the Temporal Go SDK. _(Go · Agent framework)_ - [OpenTelemetry v2](/develop/go/integrations/opentelemetry-v2) — Export tracing and metrics from Temporal Go SDK applications with OpenTelemetry. _(Go · Observability)_ --- # Google ADK integration Source: https://docs.temporal.io/develop/go/integrations/google-adk > Run Google ADK agents with durable execution using the Temporal Go SDK and the googleadk contrib integration. Temporal's integration with [Google ADK](https://google.github.io/adk-docs/) (`adk-go`) gives your agents [Durable Execution](/temporal#durable-execution): the agent's orchestration loop runs inside a Temporal Workflow, each LLM call becomes a durable Temporal Activity, and any tool that does I/O runs as an Activity too — so every step is retried, timed out, recorded in Workflow history, and replayable after crashes or restarts. You keep building agents the native ADK way — `llmagent.New(...)` with a `model.LLM`, `tool.Tool`s / `tool.Toolset`s and `SubAgents`, wrapped in `runner.New(...)` and driven by `r.Run(...)`. You change two things: 1. Use `googleadk.NewModel("")` as your agent's `Model`. It is a `model.LLM` whose calls dispatch to the `InvokeModel` Activity; the real model is reconstructed **worker-side**, never in the Workflow. 2. Pass `googleadk.NewContext(workflowCtx)` to `r.Run`, which installs Temporal-deterministic time, UUID, and task-fan-out providers so the agent loop replays deterministically. Tools run **in-workflow by default** (the idiomatic Temporal model: the Workflow is deterministic, and anything touching the network, clock, or disk goes through an Activity). Opt a tool into an Activity with `googleadk.ActivityAsTool`, or use `googleadk.NewMCPToolset` for MCP. > **ℹ️ Info:** > > The `googleadk` contrib module is new. It depends on determinism seams (`platform.WithTimeProvider`, > `WithUUIDProvider`, `WithTaskRunner`) that merged into `google.golang.org/adk/v2` after its latest tagged release, so > `go.mod` pins `adk/v2` to a `main`-branch pseudo-version until a release ships that includes them. > Code snippets in this guide are taken from the [Google ADK plugin samples](https://github.com/temporalio/samples-go/tree/main/googleadk). Refer to the samples for the complete, runnable code. ## Prerequisites - This guide assumes you are already familiar with Google ADK. If you aren't, refer to the [Google ADK documentation](https://google.github.io/adk-docs/) for more details. - If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Set up your local development environment by following the [Set up your local development environment](/develop/go/set-up-your-local-go) guide. Leave the Temporal development server running if you want to test your code locally. - Provide model credentials **worker-side** — for Gemini, set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) in the worker's environment. Credentials are captured in the worker's `ModelFactory` and never cross the Activity boundary into the Workflow. ## Install the plugin Install the `googleadk` contrib module: ```bash go get go.temporal.io/sdk/contrib/googleadk@latest ``` ```go import "go.temporal.io/sdk/contrib/googleadk" ``` ## Run an agent with Durable Execution An agent has two halves: the **worker** registers the real model behind the `InvokeModel` Activity, and the **workflow** builds a vanilla ADK agent and drives it. ### Configure the Worker Configure the Worker with the integration's plugin, which registers the model and MCP Activities at Worker start (and closes any cached MCP toolsets at Worker stop). The real Gemini model lives in the plugin's config, behind the Activity boundary; the API key is read worker-side. Disable the model SDK's own retries so Temporal's `RetryPolicy` is the single source of truth. [googleadk/worker/main.go](https://github.com/temporalio/samples-go/blob/main/googleadk/worker/main.go) ```go // The plugin registers the integration's Activities on the worker (and closes // any cached MCP toolsets at worker stop). The real Gemini model lives in the // factory, behind the Activity boundary; the API key is read worker-side from // the env and never crosses into the workflow. Disable the model SDK's own // retries so Temporal's RetryPolicy is the single source of truth. adkPlugin, err := googleadk.NewPlugin(googleadk.Config{ Models: map[string]googleadk.ModelFactory{ adk.ModelName: func(ctx context.Context, name string) (model.LLM, error) { // nil config reads GEMINI_API_KEY / GOOGLE_API_KEY from the env. return gemini.NewModel(ctx, name, nil) }, }, }) if err != nil { log.Fatalln("Unable to build googleadk plugin", err) } w := worker.New(c, adk.TaskQueue, worker.Options{ Plugins: []worker.Plugin{adkPlugin}, }) w.RegisterWorkflow(adk.AgentWorkflow) // Register GetWeather under the tool name the ActivityAsTool dispatches, so the // agent's get_weather call resolves to this activity. w.RegisterActivityWithOptions(adk.GetWeather, activity.RegisterOptions{Name: adk.WeatherToolName}) if err := w.Run(worker.InterruptCh()); err != nil { log.Fatalln("Unable to start worker", err) } ``` `Config.Models` is optional for providers ADK's registry already knows (for example, `gemini-*`): when a model name is absent, `InvokeModel` falls back to `model.NewLLM`. Supply a factory to inject credentials, disable the model SDK's own retries, or override the default. ### Define the Workflow Build the agent the ordinary ADK way, using `googleadk.NewModel` for the model and passing `googleadk.NewContext(ctx)` to `r.Run`. The `get_weather` tool is an ordinary Temporal Activity exposed to the agent with `googleadk.ActivityAsTool`. [googleadk/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/workflow.go) ```go func AgentWorkflow(ctx workflow.Context, question string) (string, error) { weatherTool, err := googleadk.ActivityAsTool(GetWeather, googleadk.ActivityToolOptions{ Name: WeatherToolName, Description: "Get the current weather for a city.", }) if err != nil { return "", err } // Build the agent the ordinary ADK way. NewModel is a model.LLM that carries // only the model name in-workflow; the real Gemini client lives worker-side. root, err := llmagent.New(llmagent.Config{ Name: "assistant", Description: "a helpful weather assistant", Model: googleadk.NewModel(ModelName), Instruction: "Answer the user's question. Use the get_weather tool when asked about the weather.", Tools: []tool.Tool{weatherTool}, }) if err != nil { return "", err } r, err := runner.New(runner.Config{ AppName: "weather", Agent: root, SessionService: session.InMemoryService(), AutoCreateSession: true, }) if err != nil { return "", err } // NewContext bridges the workflow.Context into the context ADK reads its // determinism/executor seams from. Pass it straight to Run. adkCtx := googleadk.NewContext(ctx) msg := genai.NewContentFromText(question, genai.RoleUser) var answer string for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { return "", err } if ev != nil && ev.Content != nil { for _, p := range ev.Content.Parts { if p != nil && p.Text != "" { answer = p.Text } } } } return answer, nil } ``` The tool itself is an ordinary Temporal Activity — register it on the worker as usual, and expose it to the agent with `ActivityAsTool` (its parameter schema is inferred from the argument type): [googleadk/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/workflow.go) ```go func GetWeather(ctx context.Context, in GetWeatherInput) (GetWeatherOutput, error) { return GetWeatherOutput{City: in.City, Conditions: "sunny, 72°F"}, nil } ``` ### Start the Workflow Start the Workflow like any other and read its result: [googleadk/starter/main.go](https://github.com/temporalio/samples-go/blob/main/googleadk/starter/main.go) ```go question := "What's the weather in San Francisco?" we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, adk.AgentWorkflow, question) if err != nil { log.Fatalln("Unable to execute workflow", err) } log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID()) // Synchronously wait for the workflow completion. var answer string if err := we.Get(context.Background(), &answer); err != nil { log.Fatalln("Unable to get workflow result", err) } log.Println("Agent answer:", answer) ``` ## Tools - **Function tools run in-workflow by default.** Ordinary `functiontool.New(...)` tools run on Temporal's deterministic dispatcher inside the Workflow — no Activity overhead — and their session-state mutations propagate normally. Their code must be deterministic and replay-safe: no direct network, clock, randomness, or goroutines. - **Opt a tool into an Activity when it does I/O.** `googleadk.ActivityAsTool(myActivity, ...)` exposes an existing `func(context.Context, TArgs) (TResults, error)` Temporal Activity to the agent as a tool (shown in the Hello World Workflow above); its call dispatches the Activity, so it is retried, timed out, and visible in the UI. - **MCP, statelessly.** `googleadk.NewMCPToolset(...)` is a workflow-side proxy that lists remote tools via the `ListMcpTools` Activity and executes calls via `CallMcpTool`. The live, stateful `mcptoolset.New(...)` runs worker-side (registered in `Config.MCPToolsets`), never in the Workflow. ## Multi-agent systems Build a coordinator agent with specialist `SubAgents`; ADK wires the parent/child relationship and exposes the built-in `transfer_to_agent` tool automatically. The entire tree — including the transfer hop — runs in the Workflow; only the model calls and any Activity-backed tools leave it. [googleadk/multiagent/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/multiagent/workflow.go) ```go func MultiAgentWorkflow(ctx workflow.Context, question string) (string, error) { weatherTool, err := googleadk.ActivityAsTool(GetWeather, googleadk.ActivityToolOptions{ Name: WeatherToolName, Description: "Get the current weather for a city.", }) if err != nil { return "", err } // The weather specialist owns the get_weather tool. weather, err := llmagent.New(llmagent.Config{ Name: "weather", Description: "answers questions about the current weather in a city", Model: googleadk.NewModel(WeatherModelName), Instruction: "You are a weather specialist. Use the get_weather tool to answer weather questions.", Tools: []tool.Tool{weatherTool}, }) if err != nil { return "", err } // The jokes specialist just tells jokes. jokes, err := llmagent.New(llmagent.Config{ Name: "jokes", Description: "tells a light-hearted joke", Model: googleadk.NewModel(JokesModelName), Instruction: "You are a comedian. Respond with a short, friendly joke.", }) if err != nil { return "", err } // The coordinator delegates to whichever specialist fits the question. ADK // wires the parent/child relationship from SubAgents and exposes the built-in // transfer_to_agent tool automatically. coordinator, err := llmagent.New(llmagent.Config{ Name: "coordinator", Description: "routes the user's request to the right specialist", Model: googleadk.NewModel(CoordinatorModelName), Instruction: "You are a router. Delegate weather questions to the weather agent " + "and requests for a joke to the jokes agent. Do not answer directly.", SubAgents: []agent.Agent{weather, jokes}, }) if err != nil { return "", err } r, err := runner.New(runner.Config{ AppName: "multiagent", Agent: coordinator, SessionService: session.InMemoryService(), AutoCreateSession: true, }) if err != nil { return "", err } adkCtx := googleadk.NewContext(ctx) msg := genai.NewContentFromText(question, genai.RoleUser) var answer string for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { return "", err } if ev == nil || ev.Content == nil { continue } // Keep the last non-empty text produced by any agent in the tree; after a // transfer_to_agent hop this is the specialist's answer. for _, p := range ev.Content.Parts { if p != nil && p.Text != "" { answer = p.Text } } } return answer, nil } ``` ## Human-in-the-loop tool confirmation A sensitive tool calls ADK's `ctx.RequestConfirmation(hint, payload)`, which ends the turn with an `adk_request_confirmation` function call. The Workflow detects pending confirmations with `googleadk.PendingConfirmations`, durably waits for the human's decision (delivered as a Temporal signal), and resumes the agent with `googleadk.ConfirmationResponse`. Because the wait is durable, the Workflow can sit idle for days and survive worker restarts — when the approval signal arrives, the agent resumes exactly where it paused. [googleadk/humanintheloop/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/humanintheloop/workflow.go) ```go func ApprovalWorkflow(ctx workflow.Context, request string) (Result, error) { delTool, err := functiontool.New[DeleteArgs, map[string]any]( functiontool.Config{ Name: DeleteToolName, Description: "Delete a named resource. Requires human confirmation before it runs.", }, deleteResource, ) if err != nil { return Result{}, err } root, err := llmagent.New(llmagent.Config{ Name: "assistant", Description: "an assistant that can delete resources with human approval", Model: googleadk.NewModel(ModelName), Instruction: "Use the delete_resource tool when the user asks to delete something.", Tools: []tool.Tool{delTool}, }) if err != nil { return Result{}, err } r, err := runner.New(runner.Config{ AppName: "hitl", Agent: root, SessionService: session.InMemoryService(), AutoCreateSession: true, }) if err != nil { return Result{}, err } adkCtx := googleadk.NewContext(ctx) msg := genai.NewContentFromText(request, genai.RoleUser) var res Result // Drive the run in passes: each Run call is one pass over the same session. A // pass either completes (no pending confirmation) or pauses awaiting a human. for { var events []*session.Event for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { return Result{}, err } if ev == nil { continue } events = append(events, ev) if ev.Content != nil { for _, p := range ev.Content.Parts { if p != nil && p.Text != "" { res.Answer = p.Text } } } } pending := googleadk.PendingConfirmations(events) if len(pending) == 0 { // The agent finished without (further) confirmations needed. return res, nil } // The agent paused. Durably wait for the human's decision to arrive as a // Temporal signal. This is the whole point: the workflow can sit here for // as long as it takes — across worker restarts — without losing state. // // This handles one pending confirmation per pass — the recommended // pattern (see the googleadk.ConfirmationResponse docs): resuming // several decisions at once can re-dispatch the approved tool calls in // an order that is not replay-stable. Any other pending confirmations // simply surface again on the next pass. var decision googleadk.ConfirmationDecision workflow.GetSignalChannel(ctx, googleadk.ConfirmationSignalName).Receive(ctx, &decision) res.Approved = decision.Confirmed // Match the decision to the pending confirmation and resume the run with // it as the next message. ADK re-dispatches (or blocks) the original tool // call based on Confirmed. if decision.FunctionCallID == "" { decision.FunctionCallID = pending[0].FunctionCallID } msg = googleadk.ConfirmationResponse(decision) } } ``` ## Continue-as-new for long conversations A conversation's history lives in the ADK session. To keep a Workflow's history bounded, snapshot the session with `googleadk.ExportSession` and [continue-as-new](/develop/go/workflows/continue-as-new); rebuild it on the next run with `googleadk.ImportSession`. `SessionSnapshot` is JSON-serializable (session-scoped state plus the full event history), so every value in session state and every tool result must be JSON-encodable. [googleadk/chat/workflow.go](https://github.com/temporalio/samples-go/blob/main/googleadk/chat/workflow.go) ```go func ChatWorkflow(ctx workflow.Context, in ChatInput) error { // A fresh in-memory session service, kept in a local so we can Export it later. svc := session.InMemoryService() adkCtx := googleadk.NewContext(ctx) // Resume a prior conversation if this run was continued-as-new. if in.Snapshot != nil { if _, err := googleadk.ImportSession(adkCtx, svc, in.Snapshot); err != nil { return err } } root, err := llmagent.New(llmagent.Config{ Name: "assistant", Description: "a friendly conversational assistant", Model: googleadk.NewModel(ModelName), Instruction: "You are a helpful assistant. Answer the user, using the conversation history for context.", }) if err != nil { return err } r, err := runner.New(runner.Config{ AppName: AppName, Agent: root, SessionService: svc, AutoCreateSession: true, }) if err != nil { return err } turns := 0 // One agent turn runs at a time: serialize concurrent Updates so they can't // interleave on the shared ADK session. busy := false err = workflow.SetUpdateHandlerWithOptions( ctx, SendMessageUpdateName, func(ctx workflow.Context, text string) (string, error) { if err := workflow.Await(ctx, func() bool { return !busy }); err != nil { return "", err } busy = true defer func() { busy = false }() // Build the ADK context from this Update handler's own workflow.Context so // the model Activity is scheduled on the handler's coroutine. turnCtx := googleadk.NewContext(ctx) var answer string msg := genai.NewContentFromText(text, genai.RoleUser) for ev, err := range r.Run(turnCtx, UserID, SessionID, msg, agent.RunConfig{}) { if err != nil { return "", err } if ev == nil || ev.Content == nil { continue } for _, p := range ev.Content.Parts { if p != nil && p.Text != "" { answer = p.Text } } } turns++ return answer, nil }, workflow.UpdateHandlerOptions{ Validator: func(ctx workflow.Context, text string) error { if text == "" { return fmt.Errorf("message must not be empty") } return nil }, }, ) if err != nil { return err } // Serve messages until Temporal suggests continue-as-new (history getting large) // or the demo turn cap is reached. if err := workflow.Await(ctx, func() bool { return workflow.GetInfo(ctx).GetContinueAsNewSuggested() || (in.MaxTurns > 0 && turns >= in.MaxTurns) }); err != nil { return err } // Let any in-flight Update finish so its turn is captured in the snapshot. if err := workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }); err != nil { return err } snap, err := googleadk.ExportSession(adkCtx, svc, AppName, UserID, SessionID) if err != nil { return err } return workflow.NewContinueAsNewError(ctx, ChatWorkflow, ChatInput{ Snapshot: snap, MaxTurns: in.MaxTurns, }) } ``` ## Streaming `googleadk.NewModel(name, googleadk.WithStreaming(topic, 0))` drives the model in streaming mode: the `InvokeModel` Activity calls the model with `stream=true`, heartbeats, and publishes each chunk to a per-run [`workflowstreams`](https://pkg.go.dev/go.temporal.io/sdk/contrib/workflowstreams) topic for external (UI) consumers, then returns the aggregated final response into the Workflow so replay stays deterministic. Call `googleadk.StreamServer(ctx)` once near the top of the Workflow that drives `r.Run`, and set `agent.RunConfig{StreamingMode: agent.StreamingModeSSE}`: ```go func StreamingAgentWorkflow(ctx workflow.Context, q string) (string, error) { if err := googleadk.StreamServer(ctx); err != nil { // required when streaming return "", err } topic := "run-" + workflow.GetInfo(ctx).WorkflowExecution.ID root, _ := llmagent.New(llmagent.Config{ Model: googleadk.NewModel("gemini-2.0-flash", googleadk.WithStreaming(topic, 0)), // ... }) // ... build the runner, set agent.RunConfig{StreamingMode: agent.StreamingModeSSE}, and drive r.Run } ``` External consumers read chunks with `workflowstreams.NewClient(c, workflowID, ...).Subscribe(...)`. The bidirectional `RunLive` path (hard-coded goroutines/channels) is **not** supported. ## Error handling Model, tool, and MCP failures surface as Temporal `ApplicationError`s tagged `googleadk.ModelError`, `.ToolError`, and `.McpError`. Classify them with `googleadk.IsNonRetryable(err)` rather than string-matching. For model calls, the upstream HTTP status drives retryability (`408`/`409`/`429`/`5xx` are retryable; other `4xx` are not). > **💡 Tip:** > > Disable your model client's own retries in the `ModelFactory`. `InvokeModel` already runs under Temporal's > `RetryPolicy`; leaving the model SDK's retries on retries a transient failure twice over. Let Temporal own retries. > ## Composing with other plugins The integration's plugin only registers its Activities and closes cached MCP toolsets — it uses the default JSON data converter and ships no client/worker interceptor — so it composes with interceptor- or converter-based plugins (for example [`sdk-go/contrib/opentelemetry`](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry)) without conflict. ADK emits its own OpenTelemetry spans; register your tracing interceptor on the worker as usual. ## Testing without a live LLM The plugin ships test helpers so you can unit-test agent Workflows with no network: `FakeModel` (with `TextResponse` / `FunctionCallResponse` builders) and `FakeMCPServer`. Register them through the same `googleadk.Config` your production worker uses — via `googleadk.NewActivities` and `Register` rather than the plugin, since the test environments construct no real Worker and therefore run no plugins. ## Supported and not-yet-supported - **Supported:** single- and multi-agent (`SubAgents`) trees, in-workflow function tools, `ActivityAsTool`, stateless MCP, Gemini built-in tools (executed server-side inside `InvokeModel`), human-in-the-loop tool confirmation, continue-as-new session-state carry, the in-memory session service, and SSE streaming. - **Not yet:** `RunLive` (bidirectional streaming), sub-agent-as-child-workflow, live memory/artifact tools that require in-workflow network I/O, and database/Vertex session services. These raise or are documented rather than silently degrading. ## Samples The [Google ADK plugin samples](https://github.com/temporalio/samples-go/tree/main/googleadk) demonstrate a basic agent with a tool, a [multi-agent](https://github.com/temporalio/samples-go/tree/main/googleadk/multiagent) system, durable [human-in-the-loop](https://github.com/temporalio/samples-go/tree/main/googleadk/humanintheloop) tool approval, and a [continue-as-new chat](https://github.com/temporalio/samples-go/tree/main/googleadk/chat). --- # OpenTelemetry v2 integration Source: https://docs.temporal.io/develop/go/integrations/opentelemetry-v2 > Configure trace propagation, automatic tracing, custom tracing, and metrics with the Go SDK OpenTelemetry v2 plugin. Temporal's OpenTelemetry integration lets you understand the internal state of Temporal applications across Clients, Workflows, Activities, and Nexus Operations by instrumenting them with [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/). OpenTelemetry instruments your applications to give you insight into your deployed environments. Temporal Workflows complicate that picture because a trace can span across different Workers over long stretches of time, which can scatter a trace into disconnected fragments. The OpenTelemetry plugin solves this by propagating OpenTelemetry context across those Temporal boundaries, keeping a trace intact end to end. It can also generate spans and emit metrics for Temporal SDK operations automatically. > **Pre-release** All code snippets in this guide are taken from the [OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2). Refer to the sample for complete code. ## Prerequisites - This guide assumes you are already familiar with OpenTelemetry. If you aren't, refer to the [OpenTelemetry documentation](https://opentelemetry.io/docs/) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/go/set-up-your-local-go) guide. When you're done, leave the Temporal Development Server running if you want to test your code locally. ## Install Add the OpenTelemetry v2 integration to your Go module: ```bash go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest ``` Also add the OpenTelemetry SDK packages and the exporter or metric reader your backend requires. ## Set up the tracer provider A [Tracer Provider](https://opentelemetry.io/docs/concepts/signals/traces/#tracer-provider) is a factory for Tracers, and it configures the Tracers it creates, including how they generate span IDs. A standard Tracer Provider assigns a new random span ID each time a span is created, but Temporal Workflows replay, re-executing the same code and recreating what should be the same span with a different random ID each time. Temporal's replay-safe Tracer Provider avoids this by generating span IDs from a deterministic source tied to the Workflow, so the same span gets the same ID on every replay. Create it and install it as the OpenTelemetry global before you create the plugin or call `Tracer`. [opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go) ```go // ... provider := temporalotel.NewReplaySafeTracerProvider( // WithBatcher performs exporter I/O outside the Workflow goroutine. sdktrace.WithBatcher(exporter), sdktrace.WithResource(resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceName(serviceName), )), ) otel.SetTracerProvider(provider) ``` Your application owns the Tracer Provider for the life of the process. Shut it down before exit so remaining spans can flush through the [trace exporter](https://opentelemetry.io/docs/concepts/signals/traces/#trace-exporters). ## Set up the meter provider A [Meter Provider](https://opentelemetry.io/docs/concepts/signals/metrics/#meter-provider) is a factory for Meters. OpenTelemetry's default global Meter Provider is a no-op, so if you enable `MetricsHandlerOptions`, you need to supply a configured one yourself, either by installing it with `otel.SetMeterProvider` before you create the plugin, or by passing a Meter directly through `MetricsHandlerOptions.Meter`. ## Add the plugin Pass the plugin to your Temporal Client when you create it. Workers made from that Client get the plugin automatically. [opentelemetry-v2/workflow-activity-propagation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/worker/main.go) ```go plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{}) if err != nil { return fmt.Errorf("unable to create plugin: %w", err) } c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}}) if err != nil { return fmt.Errorf("unable to create client: %w", err) } defer c.Close() ``` By default the plugin only performs [context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) so [Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context) can cross Temporal boundaries. ## Add custom spans ### In Workflows A [Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer) creates spans that capture information about a given operation. A standard Tracer stamps a span with the current time and emits it as soon as it completes, but Temporal Workflows replay, re-executing the same code and stamping what should be the same span with a new time and emitting a duplicate span. Temporal's replay-safe `Tracer` avoids this by stamping a span with `workflow.Now`, Temporal's replay-safe clock, and skipping a span that already completed on a previous successful execution. Use it instead of `otel.Tracer` in Workflows. [opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go) ```go // ... func Workflow(ctx workflow.Context, name string) (string, error) { tracer := temporalotel.Tracer(instrumentationName) ctx, span := tracer.Start(ctx, "workflow-operation") defer span.End() ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, }) var result string if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil { return "", err } return result, nil } ``` As in [OpenTelemetry Go](https://opentelemetry.io/docs/languages/go/instrumentation/), `Start` returns a context that contains the active span. Pass that `workflow.Context` to downstream Temporal calls so later spans nest under it as children. ### Outside Workflows In Clients, Activities, and other non-Workflow code, use an ordinary OpenTelemetry [Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer): [opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go) ```go // ... func Activity(ctx context.Context, name string) (string, error) { _, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation") defer span.End() return fmt.Sprintf("Hello, %s!", name), nil } ``` ## Enable automatic instrumentation [opentelemetry-v2/automatic-instrumentation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/automatic-instrumentation/worker/main.go) ```go plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{ TracerOptions: tracing.TracerOptions{ AddTemporalSpans: true, }, MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{ UseMonotonicCounters: true, }, }) if err != nil { return fmt.Errorf("unable to create plugin: %w", err) } ``` ### `AddTemporalSpans` Set `AddTemporalSpans` to `true` to create spans for Temporal SDK operations across Clients, Workflows, Activities, and Nexus Operations. ### `MetricsHandlerOptions` Set `MetricsHandlerOptions` to a non-`nil` value to emit [Temporal SDK metrics](/references/sdk-metrics) through OpenTelemetry. ## Configure context propagation [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) is how OpenTelemetry moves context across process boundaries, injecting it on the way out and extracting it on the way in. The plugin performs this propagation for you across Temporal boundaries, carrying [Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context), which keeps spans linked into one trace, and [baggage](https://opentelemetry.io/docs/concepts/signals/baggage/), optional key-value data that travels with the context. Do not put credentials, tokens, or personal data in baggage since the plugin serializes it into Temporal headers that can be persisted in Workflow Event History. ### `TextMapPropagator` The plugin injects and extracts both with a [TextMapPropagator](https://opentelemetry.io/docs/specs/otel/context/api-propagators/#textmap-propagator). By default that propagator supports [W3C Trace Context](https://www.w3.org/TR/trace-context/) and [W3C Baggage](https://www.w3.org/TR/baggage/). Set `PluginOptions.TextMapPropagator` to override it. ### `HeaderKey` Propagated values are stored in the Temporal header under `_tracer-data`. Set `TracerOptions.HeaderKey` to use a different key. ### `DisableBaggage` Set `DisableBaggage` to `true` to stop propagating baggage. ### `AllowInvalidParentSpans` Set `AllowInvalidParentSpans` to `true` to ignore errors when extracting [Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context) from Temporal headers. Use this when migrating between tracing libraries while Workflows or Activities are still in progress. ## Resources - [OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2) - [OpenTelemetry v2 Go package](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry-v2) - [Traces](https://opentelemetry.io/docs/concepts/signals/traces/) - [Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/) - [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) - [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) - [Go SDK observability guide](/develop/go/platform/observability) --- # Nexus - Go SDK Source: https://docs.temporal.io/develop/go/nexus > This section explains how to use Temporal Nexus with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Temporal Nexus - [Quickstart](/develop/go/nexus/quickstart) - [Feature guide](/develop/go/nexus/feature-guide) - [Standalone Operations](/develop/go/nexus/standalone-operations) --- # Temporal Nexus - Go SDK feature guide Source: https://docs.temporal.io/develop/go/nexus/feature-guide > Use Temporal Nexus within the Go SDK to connect durable executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Use [Temporal Nexus](/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. > **💡 Tip:** > > New to Nexus? Start with the [Nexus Go Quickstart](/develop/go/nexus/quickstart). > This page shows how to do the following: - [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server) - [Create caller and handler Namespaces](#create-caller-handler-namespaces) - [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint) - [Define the Nexus Service contract](#define-nexus-service-contract) - [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers) - [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service) - [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server) - [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud) > **📝 Note:** > > This documentation uses source code derived from the [Go Nexus sample](https://github.com/temporalio/samples-go/tree/main/nexus). > ## Run the Temporal Development Server with Nexus enabled Prerequisites: - [Install the latest Temporal CLI](/develop/run-a-development-server) (v1.3.0 or higher recommended) - [Install the latest Temporal Go SDK](/develop/go/set-up-your-local-go) (v1.33.0 or higher recommended) The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled. ``` temporal server start-dev ``` This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`. ## Create caller and handler Namespaces Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. ``` temporal operator namespace create --namespace my-target-namespace temporal operator namespace create --namespace my-caller-namespace ``` `my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. ## Create a Nexus Endpoint to route requests from caller to handler After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. ``` temporal operator nexus endpoint create \ --name my-nexus-endpoint-name \ --target-namespace my-target-namespace \ --target-task-queue my-handler-task-queue ``` You can also use the Web UI to create the Namespaces and Nexus endpoint. ## Define the Nexus Service contract Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](/dataconversion). The default data converter encodes payloads in the following order: Null, Byte array, Protobuf JSON, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, Protobuf and JSON are common choices. This example uses native Go types. [nexus/service/api.go](https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go) ```go // ... const HelloServiceName = "my-hello-service" // Echo operation const EchoOperationName = "echo" type EchoInput struct { Message string } type EchoOutput EchoInput ``` ## Develop a Nexus Service and Operation handlers Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/cloud/limits#nexus-operation-request-timeout). Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database. Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint. The `temporalnexus` package has builders to create Nexus Operations and other helpers for authoring Operation handlers: - `NewWorkflowRunOperation` \- Run a Workflow as an asynchronous Nexus Operation - `GetClient` \- Get the Temporal Client that the Worker was initialized with for synchronous handlers backed by Temporal primitives such as Signals and Queries This tutorial starts with a sync Operation handler example using the `nexus.NewSyncOperation` method, and then shows how to create an async Operation handler that uses `NewWorkflowRunOperation` to start a handler Workflow from a Nexus Operation. ### Develop a Synchronous Nexus Operation handler The `nexus.NewSyncOperation` builder function is for exposing simple RPC handlers. Use `temporalnexus.GetClient(ctx)` to get the Temporal Client for signaling, querying, and listing Workflows. Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking). [nexus/handler/app.go](https://github.com/temporalio/samples-go/blob/main/nexus/handler/app.go) ```go // ... import ( "context" "fmt" "github.com/nexus-rpc/sdk-go/nexus" "go.temporal.io/sdk/client" "go.temporal.io/sdk/temporalnexus" "go.temporal.io/sdk/workflow" "github.com/temporalio/samples-go/nexus/service" ) // NewSyncOperation is a meant for exposing simple RPC handlers. var EchoOperation = nexus.NewSyncOperation(service.EchoOperationName, func(ctx context.Context, input service.EchoInput, options nexus.StartOperationOptions) (service.EchoOutput, error) { // Use temporalnexus.GetClient to get the client that the worker was initialized with to perform client calls // such as signaling, querying, and listing workflows. Implementations are free to make arbitrary calls to other // services or databases, or perform simple computations such as this one. return service.EchoOutput(input), nil }) ``` ### Use the Temporal Client for Signals, Queries, and Updates A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](/cloud/limits#nexus-operation-request-timeout). The ctx provided to the handler is automatically set with this deadline, so passing it directly to Temporal Client calls will correctly propagate the timeout. Updates should be short-lived to stay within this deadline. The [nexus_messaging](https://github.com/temporalio/samples-go/tree/main/nexus-messaging) sample shows how to create a Nexus Service that uses synchronous operations to send Updates and Queries. Use the Nexus library, as shown below, to get the Client that the Worker was initialized with. In this example, the Workflow Id is derived from the client Id, with the `GetWorkflowID` method. This converts a given client Id (in this case, the client is passing in a user Id) to generate a Workflow Id from it. This way the client only needs the identifier it cares about. [nexus-messaging/callerpattern/handler/app.go](https://github.com/temporalio/samples-go/blob/main/nexus-messaging/callerpattern/handler/app.go) ```go func GetWorkflowID(userID string) string { return WorkflowIDPrefix + userID } var GetLanguagesOperation = nexus.NewSyncOperation(service.GetLanguagesOperationName, func(ctx context.Context, input service.GetLanguagesInput, options nexus.StartOperationOptions) (service.GetLanguagesOutput, error) { c := temporalnexus.GetClient(ctx) workflowID := GetWorkflowID(input.UserID) ... ``` There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-go/tree/main/nexus-messaging/callerpattern/) and [on-demand pattern](https://github.com/temporalio/samples-go/tree/main/nexus-messaging/ondemandpattern/). The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it. ### Develop an Asynchronous Nexus Operation handler to start a Workflow Use the `NewWorkflowRunOperation` constructor, which is the easiest way to expose a Workflow as an operation. See alternatives [here](https://pkg.go.dev/go.temporal.io/sdk/temporalnexus). [nexus/handler/app.go](https://github.com/temporalio/samples-go/blob/main/nexus/handler/app.go) ```go // ... var HelloOperation = temporalnexus.NewWorkflowRunOperation(service.HelloOperationName, HelloHandlerWorkflow, func(ctx context.Context, input service.HelloInput, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) { return client.StartWorkflowOptions{ // Workflow IDs should typically be business meaningful IDs and are used to dedupe workflow starts. // For this example, use a business ID derived from the greeting input so repeated operations // for the same name and language resolve to the same workflow. ID: service.HelloWorkflowID(input), // Task queue defaults to the task queue this operation is handled on. }, nil }) ``` Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. For the `HelloOperation`, `input.ID` is passed as part of the Nexus Service contract. > **💡 Tip:** > RESOURCES > > [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing. > #### Map a Nexus Operation input to multiple Workflow arguments A Nexus Operation can only take one input parameter. If you want a Nexus Operation to start a Workflow that takes multiple arguments use `NewWorkflowRunOperationWithOptions` or `MustNewWorkflowRunOperationWithOptions`. [nexus-multiple-arguments/handler/app.go](https://github.com/temporalio/samples-go/blob/main/nexus-multiple-arguments/handler/app.go) ```go var HelloOperation = temporalnexus.MustNewWorkflowRunOperationWithOptions(temporalnexus.WorkflowRunOperationOptions[service.HelloInput, service.HelloOutput]{ Name: service.HelloOperationName, Handler: func(ctx context.Context, input service.HelloInput, options nexus.StartOperationOptions) (temporalnexus.WorkflowHandle[service.HelloOutput], error) { return temporalnexus.ExecuteUntypedWorkflow[service.HelloOutput]( ctx, options, client.StartWorkflowOptions{ // Workflow IDs should typically be business meaningful IDs and are used to dedupe workflow starts. // For this example, use a business ID derived from the greeting input so repeated operations // for the same name and language resolve to the same workflow. ID: service.HelloWorkflowID(input), }, HelloHandlerWorkflow, input.Name, input.Language, ) }, }) ``` ### Register a Nexus Service in a Worker After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus Service in a Worker. [nexus/handler/worker/main.go](https://github.com/temporalio/samples-go/blob/main/nexus/handler/worker/main.go) ```go package main import ( "log" "os" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" "github.com/nexus-rpc/sdk-go/nexus" "github.com/temporalio/samples-go/nexus/handler" "github.com/temporalio/samples-go/nexus/options" "github.com/temporalio/samples-go/nexus/service" ) const ( taskQueue = "my-handler-task-queue" ) func main() { // The client and worker are heavyweight objects that should be created once per process. clientOptions, err := options.ParseClientOptionFlags(os.Args[1:]) if err != nil { log.Fatalf("Invalid arguments: %v", err) } c, err := client.Dial(clientOptions) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, taskQueue, worker.Options{}) service := nexus.NewService(service.HelloServiceName) err = service.Register(handler.EchoOperation, handler.HelloOperation) if err != nil { log.Fatalln("Unable to register operations", err) } w.RegisterNexusService(service) w.RegisterWorkflow(handler.HelloHandlerWorkflow) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` ## Develop a caller Workflow that uses the Nexus Service Import the Service API package that has the necessary service and operation names and input/output types to execute a Nexus Operation from the caller Workflow: [nexus/caller/workflows.go](https://github.com/temporalio/samples-go/blob/main/nexus/caller/workflows.go) ```go package caller import ( "github.com/temporalio/samples-go/nexus/service" "go.temporal.io/sdk/workflow" ) const ( TaskQueue = "my-caller-workflow-task-queue" endpointName = "my-nexus-endpoint-name" ) func EchoCallerWorkflow(ctx workflow.Context, message string) (string, error) { c := workflow.NewNexusClient(endpointName, service.HelloServiceName) fut := c.ExecuteOperation(ctx, service.EchoOperationName, service.EchoInput{Message: message}, workflow.NexusOperationOptions{}) var res service.EchoOutput if err := fut.Get(ctx, &res); err != nil { return "", err } return res.Message, nil } func HelloCallerWorkflow(ctx workflow.Context, name string, language service.Language) (string, error) { c := workflow.NewNexusClient(endpointName, service.HelloServiceName) fut := c.ExecuteOperation(ctx, service.HelloOperationName, service.HelloInput{Name: name, Language: language}, workflow.NexusOperationOptions{}) var res service.HelloOutput // Optionally wait for the operation to be started. NexusOperationExecution will contain the operation token in // case this operation is asynchronous, which is a handle that can be used to perform additional actions like // cancelling an operation. var exec workflow.NexusOperationExecution if err := fut.GetNexusOperationExecution().Get(ctx, &exec); err != nil { return "", err } if err := fut.Get(ctx, &res); err != nil { return "", err } return res.Message, nil } ``` ### Set Nexus Operation timeouts Nexus Operations support [three types of timeouts](/nexus/operations#timeouts) that control how long the caller is willing to wait at different stages of the Operation lifecycle. Set these timeouts in `NexusOperationOptions` when calling `ExecuteOperation`. #### Schedule-to-Close timeout The [Schedule-to-Close timeout](/nexus/operations#schedule-to-close-timeout) limits the total duration of the Operation from when it is scheduled to when it completes. The Nexus Machinery automatically retries failed requests until this timeout is exceeded. ```go fut := c.ExecuteOperation(ctx, service.HelloOperationName, service.HelloInput{Name: name, Language: language}, workflow.NexusOperationOptions{ ScheduleToCloseTimeout: 10 * time.Minute, }) ``` #### Schedule-to-Start timeout The [Schedule-to-Start timeout](/nexus/operations#schedule-to-start-timeout) limits how long the caller will wait for the Operation to be started by the handler. If not set, no Schedule-to-Start timeout is enforced. ```go fut := c.ExecuteOperation(ctx, service.HelloOperationName, service.HelloInput{Name: name, Language: language}, workflow.NexusOperationOptions{ ScheduleToStartTimeout: 2 * time.Minute, }) ``` #### Start-to-Close timeout The [Start-to-Close timeout](/nexus/operations#start-to-close-timeout) limits how long the caller will wait for an asynchronous Operation to complete after it has been started. This timeout only applies to asynchronous Operations. If not set, no Start-to-Close timeout is enforced. ```go fut := c.ExecuteOperation(ctx, service.HelloOperationName, service.HelloInput{Name: name, Language: language}, workflow.NexusOperationOptions{ StartToCloseTimeout: 5 * time.Minute, }) ``` ### Register the caller Workflow in a Worker After developing the caller Workflow, the next step is to register it with a Worker. [nexus/caller/worker/main.go](https://github.com/temporalio/samples-go/blob/main/nexus/caller/worker/main.go) ```go package main import ( "log" "os" "github.com/temporalio/samples-go/nexus/caller" "github.com/temporalio/samples-go/nexus/options" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" ) func main() { // The client and worker are heavyweight objects that should be created once per process. clientOptions, err := options.ParseClientOptionFlags(os.Args[1:]) if err != nil { log.Fatalf("Invalid arguments: %v", err) } c, err := client.Dial(clientOptions) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, caller.TaskQueue, worker.Options{}) w.RegisterWorkflow(caller.EchoCallerWorkflow) w.RegisterWorkflow(caller.HelloCallerWorkflow) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` ### Develop a starter to start the caller Workflow To initiate the caller Workflow, a starter program is required. [nexus/caller/starter/main.go](https://github.com/temporalio/samples-go/blob/main/nexus/caller/starter/main.go) ```go package main import ( "context" "log" "os" "time" "go.temporal.io/sdk/client" "github.com/temporalio/samples-go/nexus/caller" "github.com/temporalio/samples-go/nexus/options" "github.com/temporalio/samples-go/nexus/service" ) func main() { clientOptions, err := options.ParseClientOptionFlags(os.Args[1:]) if err != nil { log.Fatalf("Invalid arguments: %v", err) } c, err := client.Dial(clientOptions) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() runWorkflow(c, caller.EchoCallerWorkflow, "Nexus Echo 👋") runWorkflow(c, caller.HelloCallerWorkflow, "Nexus", service.ES) } func runWorkflow(c client.Client, workflow interface{}, args ...interface{}) { ctx := context.Background() workflowOptions := client.StartWorkflowOptions{ ID: "nexus_hello_caller_workflow_" + time.Now().Format("20060102150405"), TaskQueue: caller.TaskQueue, } wr, err := c.ExecuteWorkflow(ctx, workflowOptions, workflow, args...) if err != nil { log.Fatalln("Unable to execute workflow", err) } log.Println("Started workflow", "WorkflowID", wr.GetID(), "RunID", wr.GetRunID()) // Synchronously wait for the workflow completion. var result string err = wr.Get(context.Background(), &result) if err != nil { log.Fatalln("Unable get workflow result", err) } log.Println("Workflow result:", result) } ``` ## Make Nexus calls across Namespaces with a development Server Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter. ### Run Workers connected to a local development server Run the Nexus handler Worker: ``` cd handler go run ./worker \ -target-host localhost:7233 \ -namespace my-target-namespace ``` In another terminal window, run the Nexus caller Worker: ``` cd caller go run ./worker \ -target-host localhost:7233 \ -namespace my-caller-namespace ``` ### Start a caller Workflow With the Workers running, the final step in the local development process is to start a caller Workflow. Run the starter: ``` cd caller go run ./starter \ -target-host localhost:7233 \ -namespace my-caller-namespace ``` This will result in: ``` 2024/10/04 19:57:40 Workflow result: Nexus Echo 👋 2024/10/04 19:57:40 Started workflow WorkflowID nexus_hello_caller_workflow_20240723195740 RunID c9789128-2fcd-4083-829d-95e43279f6d7 2024/10/04 19:57:40 Workflow result: ¡Hola! Nexus 👋 ``` ### Canceling a Nexus Operation To cancel a Nexus Operation from within a Workflow, create a Go context using the `workflow.WithCancel` API. This returns a new context and a function that, when called, cancels the context and any SDK method that was passed this context. The future returned by `NexusClient.ExecuteOperation` is resolved when the operation finishes, whether it succeeds, fails, times out, or is canceled. Only asynchronous operations can be canceled in Nexus, as cancelation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancelation request. If ignored, the operation may enter a terminal state. Once the caller Workflow completes, the caller's Nexus Machinery will not make any further attempts to cancel operations that are still running. It's okay to leave operations running in some use cases. To ensure cancelations are delivered, wait for all pending operations to finish before exiting the Workflow. See the [Nexus cancelation sample](https://github.com/temporalio/samples-go/tree/main/nexus-cancelation) for reference. ## Make Nexus calls across Namespaces in Temporal Cloud This section assumes you are already familiar with [how to connect a Worker to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud). The same [source code](https://github.com/temporalio/samples-go/tree/main/nexus) is used in this section, but the Temporal Cloud CLI will be used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install `tcld` and generate certificates Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command (on macOS): ``` brew install temporalio/brew/tcld ``` If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: ``` tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key ``` These certificates will be valid for one year. ### Create caller and handler Namespaces Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. **Temporal CLI** ``` temporal cloud login temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` **tcld** ``` tcld login tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/Namespaces](https://cloud.temporal.io/Namespaces). ### Create a Nexus Endpoint to route requests from caller to handler To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. **Temporal CLI** ``` temporal cloud nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file description.md ``` **tcld** ``` tcld nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file description.md ``` The `--allow-namespace` flag adds caller Namespaces that can use the Nexus Endpoint to its allowlist. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus). ### Run Workers Connected to Temporal Cloud with TLS certificates Run the handler Worker: ``` cd handler go run ./worker \ -target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key' ``` Run the caller Worker: ``` cd caller go run ./worker \ -target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key' ``` ### Start a caller Workflow ``` cd caller go run ./starter \ -target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key' ``` This will result in: ``` 2024/10/04 19:57:40 Workflow result: Nexus Echo 👋 2024/10/04 19:57:40 Workflow result: ¡Hola! Nexus 👋 ``` ### Run Workers Connected to Temporal Cloud with API keys [View the source code](https://github.com/temporalio/samples-go/tree/main/nexus) in the context of the rest of the application code. Run the handler Worker: ``` cd handler go run ./worker \ -target-host .tmprl.cloud:7233 \ -namespace \ -api-key ``` Run the caller Worker: ``` cd caller go run ./worker \ -target-host .tmprl.cloud:7233 \ -namespace \ -api-key ``` ### Start a caller Workflow ``` cd caller go run ./starter \ -target-host .tmprl.cloud:7233 \ -namespace \ -api-key ``` This will result in: ``` 2024/10/04 19:57:40 Workflow result: Nexus Echo 👋 2024/10/04 19:57:40 Workflow result: ¡Hola! Nexus 👋 ``` ## Observability ### Web UI A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history: ![Observability Sync](/img/cloud/nexus/go-sdk-observability-sync.png) An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history: ![Observability Async](/img/cloud/nexus/go-sdk-observability-async.png) ### Temporal CLI Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: ``` temporal workflow describe -w ``` Nexus events are included in the caller's Event history: ``` temporal workflow show -w ``` For **asynchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationStarted` - `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationCompleted` > **📝 Note:** > > `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. > ## Learn more - Read the high-level description of the [Temporal Nexus feature](/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082). - Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus). - Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus). --- # Nexus Go Quickstart Source: https://docs.temporal.io/develop/go/nexus/quickstart > Build a Nexus Service that wraps an existing Temporal Workflow using the Go SDK [Temporal Nexus](/evaluate/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow. > **ℹ️ Info:** > To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus). **Prerequisites:** Complete the [Go SDK Quickstart](/develop/go/set-up-your-local-go) first. You should have `activity.go`, `workflow.go`, `worker/main.go`, and `start/main.go` from that guide. ## What you'll build You have `SayHelloWorkflow` running in the `default` Namespace. By the end of this guide: 1. A Nexus Service will expose `SayHelloWorkflow` as an Operation. 2. A second Namespace will contain a Workflow that calls that Operation. 3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces. ## 1. Define the Nexus Service Create a file called `service.go` that defines the Nexus Service contract. Creating a Nexus Service establishes the contract between your implementation and any callers. It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract. `HelloServiceName` and `HelloOperationName` are string constants that uniquely identify the Service and Operation. `HelloInput` defines the input type for the Operation. `SayHelloWorkflow` returns `string`, so the Operation output type is also `string`. ```go package greeting const ( HelloServiceName = "my-hello-service" HelloOperationName = "say-hello" ) type HelloInput struct { Name string } ``` ## 2. Define the Nexus Operation handlers Create a file called `nexus_handler.go` that implements the Nexus Operation handler. Operation handlers contain the logic that runs when a caller invokes a Nexus Operation. `HelloNexusWorkflow` acts as the handler Workflow. It bridges the Nexus `HelloInput` to `SayHelloWorkflow`'s `string` parameter by extracting `input.Name`. `temporalnexus.NewWorkflowRunOperation` creates an asynchronous Nexus Operation that starts `HelloNexusWorkflow` when invoked. The Options function returns `client.StartWorkflowOptions`, including a stable Workflow ID derived from `options.RequestID`. ```go package greeting import ( "context" "github.com/nexus-rpc/sdk-go/nexus" "go.temporal.io/sdk/client" "go.temporal.io/sdk/temporalnexus" "go.temporal.io/sdk/workflow" ) // HelloNexusWorkflow is the handler Workflow for the Nexus SayHello Operation. // It bridges the Nexus HelloInput to SayHelloWorkflow's string parameter. func HelloNexusWorkflow(ctx workflow.Context, input HelloInput) (string, error) { return SayHelloWorkflow(ctx, input.Name) } var HelloOperation = temporalnexus.NewWorkflowRunOperation( HelloOperationName, HelloNexusWorkflow, func(ctx context.Context, input HelloInput, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) { return client.StartWorkflowOptions{ // RequestID is stable across retries, making it safe to use as a Workflow ID. ID: options.RequestID, }, nil }, ) ``` ## 3. Register the Nexus Service handler in a Worker Update your existing `worker/main.go` to register the Nexus Service Handler. This Worker runs in the `default` Namespace — the same Namespace where `SayHelloWorkflow` is already registered. A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered. This is the same Worker concept used for Workflows and Activities. `nexus.NewService` creates a named Nexus Service. `service.Register` adds the `HelloOperation` to the Service. `w.RegisterNexusService` registers the Service with the Worker so it can receive Nexus Operation requests. `HelloNexusWorkflow` must also be registered so the Worker can execute it as the handler Workflow. ```go package main import ( "log" "my-org/greeting" "github.com/nexus-rpc/sdk-go/nexus" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" ) func main() { // Empty Options defaults to the "default" Namespace. c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "my-task-queue", worker.Options{}) w.RegisterWorkflow(greeting.SayHelloWorkflow) w.RegisterWorkflow(greeting.HelloNexusWorkflow) w.RegisterActivity(greeting.Greet) service := nexus.NewService(greeting.HelloServiceName) err = service.Register(greeting.HelloOperation) if err != nil { log.Fatalln("Unable to register operations", err) } w.RegisterNexusService(service) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` ## 4. Develop the caller Workflow Create a file called `caller_workflow.go` that defines a Workflow which invokes the Nexus Operation. The caller Workflow demonstrates the consumer side of Nexus. Instead of importing handler code directly, the caller only depends on the Service contract. This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams. `workflow.NewNexusClient` creates a client bound to your Nexus Service and Endpoint. `ExecuteOperation` starts the Operation and returns a future. `fut.Get` blocks until the Operation completes and writes the result into `result`. ```go package greeting import ( "time" "go.temporal.io/sdk/workflow" ) const ( CallerTaskQueue = "my-caller-task-queue" NexusEndpoint = "my-nexus-endpoint-name" ) func CallerWorkflow(ctx workflow.Context, name string) (string, error) { c := workflow.NewNexusClient(NexusEndpoint, HelloServiceName) fut := c.ExecuteOperation(ctx, HelloOperationName, HelloInput{Name: name}, workflow.NexusOperationOptions{ ScheduleToCloseTimeout: 10 * time.Second, }) var result string if err := fut.Get(ctx, &result); err != nil { return "", err } return result, nil } ``` ## 5. Create the caller Namespace and Nexus Endpoint Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server. Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. The endpoint name must match the `NexusEndpoint` constant defined in `caller_workflow.go` from step 4. Make sure your local Temporal dev server is running (`temporal server start-dev`). ```bash temporal operator namespace create --namespace my-caller-namespace ``` ```bash temporal operator nexus endpoint create \\ --name my-nexus-endpoint-name \\ --target-namespace default \\ --target-task-queue my-task-queue ``` ## 6. Run and Verify Create `caller/main.go` to start the caller Worker and execute the Workflow. This brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `say-hello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `HelloNexusWorkflow` (which calls `SayHelloWorkflow`), and back to the caller. **Ensure the `nexus-rpc` dependency is synchronized:** In a terminal, from within your project: ```bash go mod tidy ``` **Run the application:** 1. Start the handler Worker in one terminal: ```bash go run worker/main.go ``` 2. Run the caller in another terminal: ```bash go run caller/main.go ``` You should see: ``` Workflow result: Hello Temporal ``` Open the [Temporal Web UI](http://localhost:8233) and switch between Namespaces to see both Workflow Executions. In `my-caller-namespace`, find the `CallerWorkflow` execution — you should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in its history. In `default`, find the `HelloNexusWorkflow` execution that was started by the Nexus Operation. ```go package main import ( "context" "log" "my-org/greeting" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" ) func main() { c, err := client.Dial(client.Options{ Namespace: "my-caller-namespace", }) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, greeting.CallerTaskQueue, worker.Options{}) w.RegisterWorkflow(greeting.CallerWorkflow) if err := w.Start(); err != nil { log.Fatalln("Unable to start worker", err) } defer w.Stop() wr, err := c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{ ID: "nexus-caller-workflow", TaskQueue: greeting.CallerTaskQueue, }, greeting.CallerWorkflow, "Temporal") if err != nil { log.Fatalln("Unable to execute workflow", err) } var result string if err := wr.Get(context.Background(), &result); err != nil { log.Fatalln("Unable to get workflow result", err) } log.Println("Workflow result:", result) } ``` ## Next Steps Now that you have a working Nexus Service, here are some resources to deepen your understanding: - **[Go Nexus Feature Guide](/develop/go/nexus/feature-guide)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls. - **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics. - **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker. - **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns. - **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries. - **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls. - **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests. - **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity. --- # Standalone Nexus Operations - Go SDK Source: https://docs.temporal.io/develop/go/nexus/standalone-operations > Execute Nexus Operations independently without a Workflow using the Temporal Go SDK. > **Pre-release** > Requires Go SDK `v1.46.0` or above. All APIs are experimental and may be subject to backwards-incompatible changes. [Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using `workflow.NewNexusClient()`, you execute a Standalone Nexus Operation directly from a Nexus Client created using `client.NewNexusClient()`. Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/go/nexus/feature-guide) for details on [defining a Service contract](/develop/go/nexus/feature-guide#define-nexus-service-contract), [developing Operation handlers](/develop/go/nexus/feature-guide#develop-nexus-service-operation-handlers), and [registering a Service in a Worker](/develop/go/nexus/feature-guide#register-a-nexus-service-in-a-worker). This page focuses on the client-side APIs that are unique to Standalone Nexus Operations: - [Execute a Standalone Nexus Operation](#execute-operation) - [Get the result of a Standalone Nexus Operation](#get-operation-result) - [List Standalone Nexus Operations](#list-operations) - [Count Standalone Nexus Operations](#count-operations) - [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud) > **📝 Note:** > This documentation uses source code from the > [Go Nexus Standalone sample](https://github.com/temporalio/samples-go/tree/main/nexus-standalone-operations). > ## Prerequisites Standalone Nexus Operations are at Pre-release and require a special Temporal CLI build. ### 1. Install and verify the Pre-release Temporal CLI The `temporal nexus operation` commands require a Pre-release build of the Temporal CLI. See [Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for the platform downloads, then verify: ```bash ./temporal --version # temporal version 1.7.4-standalone-nexus-operations ``` Run it as `./temporal` from the directory where you extracted it. The standard `brew install temporal` build does not include Standalone Nexus Operation support during Pre-release. ### 2. Start a local dev server The Pre-release dev server enables Standalone Nexus Operations by default — no dynamic config is required. Start it with the caller and handler Namespaces pre-created: ```bash ./temporal server start-dev \ --namespace my-caller-namespace \ --namespace my-handler-namespace ``` The starter and Worker connect to two different Namespaces (a caller Namespace and a handler Namespace), mirroring how Nexus crosses Namespace boundaries. To run the examples on this page against the [Go sample](https://github.com/temporalio/samples-go/tree/main/nexus-standalone-operations), create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue: ```bash ./temporal operator nexus endpoint create \ --name my-nexus-endpoint \ --target-namespace my-handler-namespace \ --target-task-queue nexus-handler-queue ``` Then start the sample Worker in the handler Namespace: ```bash TEMPORAL_NAMESPACE=my-handler-namespace go run nexus-standalone-operations/worker/main.go ``` ## Execute a Standalone Nexus Operation To execute a Standalone Nexus Operation, first create a [`NexusClient`](https://pkg.go.dev/go.temporal.io/sdk/client#NexusClient) using `client.NewNexusClient()`, bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `ExecuteOperation()` from application code (for example, a starter program), not from inside a Workflow Definition. `ExecuteOperation` returns a [`NexusOperationHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#NexusOperationHandle) that you can use to get the result of the Operation. [`StartNexusOperationOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartNexusOperationOptions) requires `ID`. `ScheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server. ```go nexusClient, err := c.NewNexusClient(client.NexusClientOptions{ Endpoint: "my-nexus-endpoint", Service: "my-hello-service", }) handle, err := nexusClient.ExecuteOperation(ctx, operationName, input, client.StartNexusOperationOptions{ ID: "unique-operation-id", ScheduleToCloseTimeout: 10 * time.Second, }) ``` See the full [starter sample](https://github.com/temporalio/samples-go/blob/main/nexus-standalone-operations/starter/main.go) for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and counts Operations. To run the starter in the caller Namespace (in a separate terminal from the Worker): ``` TEMPORAL_NAMESPACE=my-caller-namespace go run nexus-standalone-operations/starter/main.go ``` Or use the Temporal CLI to execute a Standalone Nexus Operation: ```bash ./temporal nexus operation execute \ --namespace my-caller-namespace \ --endpoint my-nexus-endpoint \ --service my-hello-service \ --operation echo \ --operation-id my-echo-op \ --input '{"Message":"hello"}' ``` ## Get the result of a Standalone Nexus Operation Use `NexusOperationHandle.Get()` to block until the Operation completes and retrieve its result. This works for both synchronous and asynchronous (Workflow-backed) Operations. ```go var result service.EchoOutput err = handle.Get(context.Background(), &result) if err != nil { log.Fatalln("Operation failed", err) } log.Println("Operation result:", result.Message) ``` If the Operation completed successfully, the result is deserialized into the provided pointer. If the Operation failed, the failure is returned as an error. Or use the Temporal CLI to wait for a result by Operation ID: ```bash ./temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op ``` ## List Standalone Nexus Operations Use [`client.ListNexusOperations()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields operation metadata entries. Note that `ListNexusOperations` is called on the base `client.Client`, not on the `NexusClient`. ```go resp, err := c.ListNexusOperations(context.Background(), client.ListNexusOperationsOptions{ Query: "Endpoint = 'my-nexus-endpoint'", }) if err != nil { log.Fatalln("Unable to list Nexus operations", err) } for metadata, err := range resp.Results { if err != nil { log.Fatalln("Error iterating operations", err) } log.Printf("OperationID: %s, Operation: %s, Status: %v\n", metadata.OperationID, metadata.Operation, metadata.Status) } ``` The `Query` field accepts [List Filter](/list-filter) syntax. For example, `"Endpoint = 'my-endpoint' AND Status = 'Running'"`. Or use the Temporal CLI: ```bash ./temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Count Standalone Nexus Operations Use [`client.CountNexusOperations()`](https://pkg.go.dev/go.temporal.io/sdk/client#Client) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. Note that `CountNexusOperations` is called on the base `client.Client`, not on the `NexusClient`. ```go resp, err := c.CountNexusOperations(context.Background(), client.CountNexusOperationsOptions{ Query: "Endpoint = 'my-nexus-endpoint'", }) if err != nil { log.Fatalln("Unable to count Nexus operations", err) } log.Println("Total Nexus operations:", resp.Count) ``` Or use the Temporal CLI: ```bash ./temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Run Standalone Nexus Operations with Temporal Cloud The code samples on this page use `envconfig.MustLoadDefaultClientOptions()`, so the same code works against Temporal Cloud — just configure the connection via environment variables or a TOML profile. No code changes are needed. For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate generation, and authentication options, see [Make Nexus calls across Namespaces in Temporal Cloud](/develop/go/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) and [Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud). --- # Platform - Go SDK Source: https://docs.temporal.io/develop/go/platform > This section explains how to implement platform with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Platform - [Observability](/develop/go/platform/observability) - [Enriching the UI](/develop/go/platform/enriching-ui) --- # Enriching the user interface - Go SDK Source: https://docs.temporal.io/develop/go/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the Go SDK. Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```go import ( "context" "go.temporal.io/sdk/client" ) func main() { // Create the client c, err := client.Dial(client.Options{}) if err != nil { // Handle error } defer c.Close() // Start workflow options with static summary and details workflowOptions := client.StartWorkflowOptions{ ID: "your-workflow-id", TaskQueue: "your-task-queue", StaticSummary: "Order processing for customer #12345", StaticDetails: "Processing premium order with expedited shipping", } // Start the workflow we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflow, "workflow input") if err != nil { // Handle error } } ``` `StaticSummary` is a single-line description that appears in the Workflow list view, limited to 200 bytes. `StaticDetails` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. ### Inside the Workflow Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```go import ( "go.temporal.io/sdk/workflow" ) func YourWorkflow(ctx workflow.Context, input string) (string, error) { // Get the current details currentDetails := workflow.GetCurrentDetails(ctx) workflow.GetLogger(ctx).Info("Current details", "details", currentDetails) // Set/update the current details workflow.SetCurrentDetails(ctx, "Updated workflow details with new status") return "Workflow completed", nil } ``` ### Adding Summary to Activities and Timers You can attach a metadata parameter `Summary` to Activities when starting them from within a Workflow: ```go import ( "time" "go.temporal.io/sdk/workflow" ) func YourWorkflow(ctx workflow.Context, input string) (string, error) { // Activity options with summary ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, Summary: "Processing user data", } ctx = workflow.WithActivityOptions(ctx, ao) // Execute the activity var result string err := workflow.ExecuteActivity(ctx, YourActivity, input).Get(ctx, &result) if err != nil { return "", err } return result, nil } ``` Similarly, you can attach a `Summary` to timers within a Workflow: ```go import ( "time" "go.temporal.io/sdk/workflow" ) func YourWorkflow(ctx workflow.Context, input string) (string, error) { // Create a timer with options including summary timerFuture := workflow.NewTimerWithOptions(ctx, 5*time.Minute, workflow.TimerOptions{ Summary: "Waiting for payment confirmation", }) // Wait for the timer err := timerFuture.Get(ctx, nil) if err != nil { return "", err } return "Timer completed", nil } ``` The input format for `Summary` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your workflows, activities, and timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the workflow details page, you'll find the workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the workflow - **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available, both in the timeline graph at the top of the tab and in the events table below it. Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the event details. When you do expand an event, the summary is also prominently displayed in the detailed view. --- # Observability - Go SDK Source: https://docs.temporal.io/develop/go/platform/observability > Monitor your Temporal Application state using Metrics, Tracing, Logging, and Visibility features. Emit metrics, configure tracing, customize logging, and use Search Attributes with the Temporal Go SDK for enhanced Workflow Execution insights. This page covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. This section covers features related to viewing the state of the application, including: - [Metrics](#metrics) - [Tracing](#tracing) - [Logging](#logging) - [Visibility](#visibility) ## How to emit metrics Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process. For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics). > **💡 Use the OpenTelemetry v2 integration:** > > To instrument Temporal applications with OpenTelemetry, use the > [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2) > ([Pre-release](/evaluate/development-production-features/release-stages#pre-release)). > It propagates OpenTelemetry context across Temporal boundaries and > is replay-safe when instrumenting Workflows. > - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). - For an end-to-end example that exposes metrics with the Go SDK, refer to the [samples-go](https://github.com/temporalio/samples-go/tree/main/metrics) repo. To emit metrics from the Temporal Client in Go, create a [metrics handler](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Handler) from the [Client Options](https://pkg.go.dev/go.temporal.io/sdk@v1.15.0/internal#ClientOptions) and specify a listener address to be used by Prometheus. ```go client.Options{ MetricsHandler: sdktally.NewMetricsHandler(newPrometheusScope(prometheus.Configuration{ ListenAddress: "0.0.0.0:9090", TimerType: "histogram", } } ``` The Go SDK provides metrics handlers for [Tally](https://pkg.go.dev/go.temporal.io/sdk/contrib/tally) and [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry). Tally offers [extensible custom metrics reporting](https://github.com/uber-go/tally#report-your-metrics), which is exposed through the [`WithCustomMetricsHandler`](/references/server-options#withcustommetricshandler) API. For more information, see the [Go sample for metrics](https://github.com/temporalio/samples-go/tree/main/metrics). ### Configure OpenTelemetry counters as monotonic > **📝 Note:** > > `UseMonotonicCounters` is available in `go.temporal.io/sdk/contrib/opentelemetry` version 0.8.0 and later. > By default, the OpenTelemetry metrics handler represents counters as `Int64UpDownCounter` instruments to preserve compatibility with earlier releases. To represent Temporal SDK counters as monotonic `Int64Counter` instruments, set `UseMonotonicCounters` to `true` when you create the handler: ```go metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{ Meter: otel.GetMeterProvider().Meter("temporal-sdk-go"), UseMonotonicCounters: true, }) temporalClient, err := client.Dial(client.Options{ MetricsHandler: metricsHandler, }) ``` Monotonic counters let exporters and metrics backends classify Temporal SDK counters correctly. The [`MetricsCounter`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsCounter) contract defines counters as ever-increasing. If you create custom counters through the same metrics handler, pass only non-negative values to [`client.MetricsCounter.Inc`](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Counter). Negative values can produce invalid or backend-dependent metric data when `UseMonotonicCounters` is enabled. ## Tracing Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and Child Workflows. > **💡 Use the OpenTelemetry v2 integration:** > > To instrument Temporal applications with OpenTelemetry, use the > [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2) > ([Pre-release](/evaluate/development-production-features/release-stages#pre-release)). > It propagates OpenTelemetry context across Temporal boundaries and > is replay-safe when instrumenting Workflows. > The Go SDK provides tracing interceptors for [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry), [OpenTracing](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentracing), and [Datadog](https://pkg.go.dev/go.temporal.io/sdk/contrib/datadog/tracing). First, create a tracing interceptor for Client instantiation. ```go // OpenTelemetry tracingInterceptor, err := opentelemetry.NewTracingInterceptor(opentelemetry.TracerOptions{}) // OpenTracing tracingInterceptor, err := opentracing.NewInterceptor(opentracing.TracerOptions{}) // Datadog tracingInterceptor, err := tracing.NewTracingInterceptor(tracing.TracerOptions{}) ``` and register it by passing it to [ClientOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions): ```go c, err := client.Dial(client.Options{ Interceptors: []interceptor.ClientInterceptor{tracingInterceptor}, }) ``` You can also register interceptors through a [Plugin](/develop/plugins-guide#interceptors) if you’re building a reusable library. Each tracing interceptor uses its library's native propagation mechanism to serialize trace spans into Temporal headers. For example, OpenTelemetry uses its `TextMapPropagator` with the W3C TraceContext format. The SDK carries these headers across Workflow, Activity, and Child Workflow boundaries, so the tracing library can reconstruct the call graph. For more information, see the documentation for [OpenTelemetry](https://opentelemetry.io/), [OpenTracing](https://opentracing.io), and [Datadog](https://docs.datadoghq.com/tracing/). To build custom context propagation (for example, tenant IDs, auth tokens), see [Context Propagation](/develop/go/best-practices/context-propagation). ## Log from a Workflow Send logs and errors to a logging service, so that when things go wrong, you can see what happened. Loggers create an audit trail and capture information about your Workflow's operation. An appropriate logging level depends on your specific needs. During development or troubleshooting, you might use debug or even trace. In production, you might use info or warn to avoid excessive log volume. You can find the log levels supported by `slog` in [their official documentation](https://pkg.go.dev/log/slog#Level). The Temporal SDK core normally uses `WARN` as its default logging level. In Workflow Definitions you can use [`workflow.GetLogger(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLogger) to write logs. ```go import ( "context" "time" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/workflow" ) // Workflow is a standard workflow definition. // Note that the Workflow and Activity don't need to care that // their inputs/results are being compressed. func Workflow(ctx workflow.Context, name string) (string, error) { // ... workflow.WithActivityOptions(ctx, ao) // Getting the logger from the context. logger := workflow.GetLogger(ctx) // Logging a message with the key value pair `name` and `name` logger.Info("Compressed Payloads workflow started", "name", name) info := map[string]string{ "name": name, } logger.Info("Compressed Payloads workflow completed.", "result", result) return result, nil } ``` ### Provide a custom logger This field sets a custom Logger that is used for all logging actions of the instance of the Temporal Client. The Go SDK supports custom loggers via `log.NewStructuredLogger()`, which wraps Go's standard [`slog.Logger`](https://pkg.go.dev/log/slog) (Go 1.21+). Because most modern logging libraries (zap, zerolog, logrus, etc.) can back a `slog.Handler`, `slog` serves as the universal bridge to third-party loggers. **Using slog directly:** ```go import ( "log/slog" "os" "go.temporal.io/sdk/client" "go.temporal.io/sdk/log" ) func main() { // ... slogHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}) logger := log.NewStructuredLogger(slog.New(slogHandler)) clientOptions := client.Options{ Logger: logger, } temporalClient, err := client.Dial(clientOptions) // ... } ``` **Bridging a third-party logger through slog (example with zap):** ```go import ( "log/slog" "go.uber.org/zap" "go.uber.org/zap/exp/zapslog" "go.temporal.io/sdk/client" "go.temporal.io/sdk/log" ) func main() { // ... zapLogger, _ := zap.NewProduction() handler := zapslog.NewHandler(zapLogger.Core()) logger := log.NewStructuredLogger(slog.New(handler)) clientOptions := client.Options{ Logger: logger, } temporalClient, err := client.Dial(clientOptions) // ... } ``` As an alternative, you can implement the `log.Logger` interface directly. The Temporal samples repo has a [zap adapter](https://github.com/temporalio/samples-go/blob/main/zapadapter/zap_adapter.go) that can be used as a reference. ## Visibility APIs The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service. ### Search Attributes The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI. - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an option when starting the Execution. - In the Workflow by calling `UpsertSearchAttributes`. - Read the value of the Search Attribute: - On the Client by calling `DescribeWorkflow`. - In the Workflow by looking at `WorkflowInfo`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/workflow#list). - In code by calling `ListWorkflowExecutions`. Here is how to query Workflow Executions: The [ListWorkflow()](https://pkg.go.dev/go.temporal.io/sdk/client#Client.ListWorkflow) function retrieves a list of [Workflow Executions](/workflow-execution) that match the [Search Attributes](/search-attribute) of a given [List Filter](/list-filter). The metadata returned from the [Visibility](/temporal-service/visibility) store can be used to get a Workflow Execution's history and details from the [Persistence](/temporal-service/persistence) store. Use a List Filter to define a `request` to pass into `ListWorkflow()`. ```go request := &workflowservice.ListWorkflowExecutionsRequest{ Query: "CloseTime = missing" } ``` This `request` value returns only open Workflows. For more List Filter examples, see the [examples provided for List Filters in the Temporal Visibility guide.](/list-filter#list-filter-examples) ```go resp, err := temporalClient.ListWorkflow(ctx.Background(), request) if err != nil { return err } fmt.Println("First page of results:") for _, exec := range resp.Executions { fmt.Printf("Workflow ID %v\n", exec.Execution.WorkflowId) } ``` ### Set custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using the `temporal operator search-attribute create` command or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. Provide key-value pairs in [`StartWorkflowOptions.SearchAttributes`](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions). Search Attributes are represented as `map[string]interface{}`. The values in the map must correspond to the [Search Attribute's value type](/search-attribute#supported-types): - Bool = `bool` - Datetime = `time.Time` - Double = `float64` - Int = `int64` - Keyword = `string` - Text = `string` If you had custom Search Attributes `CustomerId` of type Keyword and `MiscData` of type Text, you would provide `string` values: ```go func (c *Client) CallYourWorkflow(ctx context.Context, workflowID string, payload map[string]interface{}) error { // ... searchAttributes := map[string]interface{}{ "CustomerId": payload["customer"], "MiscData": payload["miscData"] } options := client.StartWorkflowOptions{ SearchAttributes: searchAttributes // ... } we, err := c.Client.ExecuteWorkflow(ctx, options, app.YourWorkflow, payload) // ... } ``` ### Upsert Search Attributes You can upsert Search Attributes to add or update Search Attributes from within Workflow code. In advanced cases, you may want to dynamically update these attributes as the Workflow progresses. [UpsertSearchAttributes](https://pkg.go.dev/go.temporal.io/sdk/workflow#UpsertSearchAttributes) is used to add or update Search Attributes from within Workflow code. `UpsertSearchAttributes` will merge attributes to the existing map in the Workflow. Consider this example Workflow code: ```go func YourWorkflow(ctx workflow.Context, input string) error { attr1 := map[string]interface{}{ "CustomIntField": 1, "CustomBoolField": true, } workflow.UpsertSearchAttributes(ctx, attr1) attr2 := map[string]interface{}{ "CustomIntField": 2, "CustomKeywordField": "seattle", } workflow.UpsertSearchAttributes(ctx, attr2) } ``` After the second call to `UpsertSearchAttributes`, the map will contain: ```go map[string]interface{}{ "CustomIntField": 2, // last update wins "CustomBoolField": true, "CustomKeywordField": "seattle", } ``` ### Remove a Search Attribute from a Workflow To remove a Search Attribute that was previously set, set it to an empty array: `[]`. **There is no support for removing a field.** However, to achieve a similar effect, set the field to some placeholder value. For example, you could set `CustomKeywordField` to `impossibleVal`. Then searching `CustomKeywordField != 'impossibleVal'` will match Workflows with `CustomKeywordField` not equal to `impossibleVal`, which includes Workflows without the `CustomKeywordField` set. --- # Set up your local with the Go SDK Source: https://docs.temporal.io/develop/go/set-up-your-local-go > Configure your local development environment to get started developing with Temporal # Quickstart Configure your local development environment to get started developing with Temporal. ## Install Go Make sure you have Go installed. These tutorials were produced using Go 1.18. Check your version of Go with the following command: This will return your installed Go version. ```bash go version ``` ```bash go version go1.18.1 darwin/amd64 ``` ## Install the Temporal Go SDK If you are creating a new project using the Temporal Go SDK, you can start by creating a new directory. Next, switch to the new directory. Then, initialize a Go project in that directory. Finally, install the Temporal SDK with `go get`. ```bash mkdir goproject ``` ```bash cd goproject ``` ```bash go mod init my-org/greeting ``` ```bash go get go.temporal.io/sdk ``` ```bash go get go.temporal.io/sdk/client ``` ```bash go mod tidy ``` ## Install Temporal CLI and start the development server The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI: **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal Go SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Create the Activity An Activity is a normal function or method that executes a single, well-defined action (either short- or long-running) that is typically prone to failure. Examples include any action that interacts with the outside world, such as sending emails, making network requests, writing to a database, or calling an API. If an Activity fails, Temporal automatically retries it based on your configuration. Create an Activity file (activity.go): ```go package greeting import ( "context" "fmt" ) func Greet(ctx context.Context, name string) (string, error) { return fmt.Sprintf("Hello %s", name), nil } ``` ### 2. Create the Workflow Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run—and keep running—for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. Create a Workflow file (workflow.go): ```go package greeting import ( "time" "go.temporal.io/sdk/workflow" ) func SayHelloWorkflow(ctx workflow.Context, name string) (string, error) { ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Second * 10, } ctx = workflow.WithActivityOptions(ctx, ao) var result string err := workflow.ExecuteActivity(ctx, Greet, name).Get(ctx, &result) if err != nil { return "", err } return result, nil } ``` ### 3. Create and Run the Worker With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue, it then executes that task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). Create a Worker file (worker/main.go): ```go package main import ( "log" "my-org/greeting" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" ) func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "my-task-queue", worker.Options{}) w.RegisterWorkflow(greeting.SayHelloWorkflow) w.RegisterActivity(greeting.Greet) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` Run the Worker: ```bash go run worker/main.go ``` ### 4. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. Create a separate file called start/main.go: ```go package main import ( "context" "log" "os" greeting "my-org/greeting" "go.temporal.io/sdk/client" ) func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() options := client.StartWorkflowOptions{ ID: "greeting-workflow", TaskQueue: "my-task-queue", } we, err := c.ExecuteWorkflow(context.Background(), options, greeting.SayHelloWorkflow, os.Args[1]) if err != nil { log.Fatalln("Unable to execute workflow", err) } log.Println("Started workflow", "WorkflowID", we.GetID(), "RunID", we.GetRunID()) var result string err = we.Get(context.Background(), &result) if err != nil { log.Fatalln("Unable get workflow result", err) } log.Println("Workflow result:", result) } ``` Then run: ```bash go run start/main.go Temporal ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the workflow and activity - Output: `Workflow result: Hello Temporal` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/go/first_program_in_go/): Create a basic Workflow and run it with the Temporal Go SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Worker Versioning (Legacy) - Go SDK Source: https://docs.temporal.io/develop/go/worker-versioning-legacy > Learn the Go SDK's outdated Worker Versioning APIs. ## How to use Worker Versioning in Go (Deprecated) > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > > See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. > A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Go Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on. ```go // ... workerOptions := worker.Options{ BuildID: buildID, UseBuildIDForVersioning: true, // ... } w := worker.New(c, "your_task_queue_name", workerOptions) // ... ``` > **⚠️ Warning:** > > Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. > ### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the Workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `VersioningIntent` field on the appropriate options struct. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: ```go // ... ao := workflow.ActivityOptions{ VersioningIntent: VersioningIntentUseAssignmentRules, // ...other options } activityCtx := workflow.WithActivityOptions(ctx, ao) var yourActivityResult YourActivityResultType err := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult) // ... ``` #### Specifying versions for Continue-As-New When using the Continue-As-New feature, use the `WithWorkflowVersioningIntent` context modifier: ```go ctx = workflow.WithWorkflowVersioningIntent(ctx, temporal.VersioningIntentUseAssignmentRules) err := workflow.NewContinueAsNewError(ctx, "WorkflowName") ``` ### Tell the Task Queue about your Worker's Build ID (Deprecated) > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. ```go // ... err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDOpAddNewIDInNewDefaultSet{ BuildID: "deadbeef", }, }) ``` This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: ```go // ... err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDOpAddNewCompatibleVersion{ BuildID: "deadbeef", ExistingCompatibleBuildId: "some-existing-build-id", }, }) ``` This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: ```go // ... err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDPromoteIDWithinSet{ BuildID: "some-existing-build-id", }, }) ``` --- # Workers - Go SDK Source: https://docs.temporal.io/develop/go/workers > This section explains how to implement Workers with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Workers - [Run a Worker](/develop/go/workers/run-worker-process) - [Sessions](/develop/go/workers/sessions) - [Serverless Workers](/develop/go/workers/serverless-workers) --- # Run a Worker - Go SDK Source: https://docs.temporal.io/develop/go/workers/run-worker-process > Create and run a Temporal Worker using the Go SDK. This page covers long-lived Workers that you host and run as persistent processes. For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/go/workers/serverless-workers). ## Create and run a Worker Create a [`Worker`](https://pkg.go.dev/go.temporal.io/sdk/worker#Worker) by calling [`worker.New()`](https://pkg.go.dev/go.temporal.io/sdk/worker#New) and passing: 1. A [Temporal Client](/develop/go/client/temporal-client). 2. The name of the Task Queue to poll. 3. A [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) struct (can be empty for defaults). Register your Workflow and Activity types, then call `Run()` to start polling. The Worker blocks while it polls, so run it in a separate terminal from your starter code. [helloworld/worker/main.go](https://github.com/temporalio/samples-go/blob/main/helloworld/worker/main.go) ```go package main import ( "log" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" "go.temporal.io/sdk/worker" "github.com/temporalio/samples-go/helloworld" ) func main() { // The client and worker are heavyweight objects that should be created once per process. c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, "hello-world", worker.Options{}) w.RegisterWorkflow(helloworld.Workflow) w.RegisterActivity(helloworld.Activity) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` `Run()` accepts an interrupt channel so the Worker shuts down on `SIGINT` or `SIGTERM`. You can also call `Start()` and `Stop()` separately for more control over the lifecycle. > **💡 Tip:** > > If you have [`gow`](https://github.com/mitranim/gow) installed, the Worker automatically reloads when you update the file: > > ```bash > go install github.com/mitranim/gow@latest > gow run worker/main.go > ``` > ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Use `RegisterWorkflow()` and `RegisterActivity()` to register types. To register an Activity struct with multiple methods, pass the struct. The Worker gets access to all exported methods. ```go w.RegisterWorkflow(WorkflowA) w.RegisterWorkflow(WorkflowB) w.RegisterActivity(&MyActivities{}) ``` To customize the registered name or other options, use `RegisterWorkflowWithOptions()` or `RegisterActivityWithOptions()`. See [`workflow.RegisterOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#RegisterOptions) and [`activity.RegisterOptions`](https://pkg.go.dev/go.temporal.io/sdk/activity#RegisterOptions). ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/go/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options Pass a [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) struct to `worker.New()` to configure concurrency limits, pollers, timeouts, and other Worker behavior. An empty struct uses defaults that work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `worker.Options`, then set a versioning behavior on each Workflow. [features/snippets/worker/worker.go](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.go) ```go w := worker.New(c, "my-task-queue", worker.Options{ DeploymentOptions: worker.DeploymentOptions{ UseVersioning: true, Version: worker.WorkerDeploymentVersion{ DeploymentName: "my-app", BuildID: "1.0", }, }, }) w.RegisterWorkflowWithOptions(HelloWorkflow, workflow.RegisterOptions{ VersioningBehavior: workflow.VersioningBehaviorPinned, }) ``` Set the behavior per Workflow with `RegisterWorkflowWithOptions()`, or set a default for every Workflow on the Worker with `DefaultVersioningBehavior` in `worker.DeploymentOptions`. A versioning behavior applies only to a Worker that has versioning enabled, and setting `DefaultVersioningBehavior` without `UseVersioning` is an error. With versioning enabled and no default set, a Workflow that does not set its own behavior fails at registration time. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker A Worker started with `Run(worker.InterruptCh())` shuts down when the process receives `SIGINT` or `SIGTERM`. The Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to the `WorkerStopTimeout` set in `worker.Options`. See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Serverless Workers - Go SDK Source: https://docs.temporal.io/develop/go/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the Go SDK. > **Public Preview** > AWS Lambda support is in Public Preview. GCP Cloud Run support is in Pre-release, and its APIs may change in > backwards-incompatible ways. To request Cloud Run access, create a [support ticket](/cloud/support#support-ticket) or > contact your account team, and [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear > when Cloud Run reaches Public Preview. Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes. Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**AWS Lambda**](/develop/go/workers/serverless-workers/aws-lambda) - Use the `lambdaworker` package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle. - [**GCP Cloud Run**](/develop/go/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in. --- # Serverless Workers on AWS Lambda - Go SDK Source: https://docs.temporal.io/develop/go/workers/serverless-workers/aws-lambda > Write a Temporal Worker that runs on AWS Lambda using the Go SDK lambdaworker package. > **Public Preview** The `lambdaworker` package lets you run a Temporal Serverless Worker on AWS Lambda. Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline. You register Workflows and Activities the same way you would with a standard Worker. For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda). ## Create and run a Worker in Lambda Use the `RunWorker` function to start a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers your Workflows and Activities. [lambda-worker/worker/main.go](https://github.com/temporalio/samples-go/blob/main/lambda-worker/worker/main.go) ```go {13-17} package main import ( greeting "github.com/temporalio/samples-go/lambda-worker/greeting" lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" // ... "go.temporal.io/sdk/worker" "go.temporal.io/sdk/workflow" ) func main() { lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ DeploymentName: "my-app", BuildID: "build-1", }, func(opts *lambdaworker.Options) error { opts.TaskQueue = "serverless-task-queue-1" // ... opts.RegisterWorkflowWithOptions(greeting.SampleWorkflow, workflow.RegisterOptions{ VersioningBehavior: workflow.VersioningBehaviorPinned, }) opts.RegisterActivity(greeting.HelloActivity) return nil }) } ``` The `WorkerDeploymentVersion` is required. Worker Deployment Versioning is always enabled for Serverless Workers. Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`. Set it per-Workflow at registration time, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. The `Options` callback gives you access to the same registration methods you use with a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. ## Configure the Temporal connection The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details. Compared with long-lived Workers, the location of the config file is resolved differently, in the following order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. The file is optional. If absent, only environment variables are used. Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options. ## Adjust Worker defaults for Lambda The `lambdaworker` package applies conservative defaults suited to short-lived Lambda invocations. These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment. Except for `ShutdownDeadlineBuffer`, these are the same [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk@v1.42.0/internal#WorkerOptions) available to any Temporal Worker, just with lower values for Lambda's constrained environment. | Setting | Lambda default | |---|---| | `MaxConcurrentActivityExecutionSize` | 2 | | `MaxConcurrentWorkflowTaskExecutionSize` | 10 | | `MaxConcurrentLocalActivityExecutionSize` | 2 | | `MaxConcurrentNexusTaskExecutionSize` | 5 | | `MaxConcurrentActivityTaskPollers` | 1 | | `MaxConcurrentWorkflowTaskPollers` | 2 | | `MaxConcurrentNexusTaskPollers` | 1 | | `WorkerStopTimeout` | 5 seconds | | `DisableEagerActivities` | Always true | | Sticky cache size | 100 | | `ShutdownDeadlineBuffer` | 7 seconds | `DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. `ShutdownDeadlineBuffer` is specific to the `lambdaworker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities). ## Add observability with OpenTelemetry The `lambdaworker/otel` sub-package provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. The underlying metrics and traces are the same ones the Go SDK emits in any environment. For general observability concepts and the full list of available metrics, see [Observability - Go SDK](/develop/go/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). [lambda-worker/worker/main.go](https://github.com/temporalio/samples-go/blob/main/lambda-worker/worker/main.go) ```go {7, 19-21} package main import ( greeting "github.com/temporalio/samples-go/lambda-worker/greeting" lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel" "go.temporal.io/sdk/worker" "go.temporal.io/sdk/workflow" ) func main() { lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ DeploymentName: "my-app", BuildID: "build-1", }, func(opts *lambdaworker.Options) error { opts.TaskQueue = "serverless-task-queue-1" if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { return err } opts.RegisterWorkflowWithOptions(greeting.SampleWorkflow, workflow.RegisterOptions{ VersioningBehavior: workflow.VersioningBehaviorPinned, }) opts.RegisterActivity(greeting.HelloActivity) return nil }) } ``` `ApplyDefaults` configures both metrics and tracing. By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function. The layer runs a collector sidecar that receives telemetry on `localhost:4317` and forwards traces to X-Ray and metrics to CloudWatch. Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle the following `otel-collector-config.yaml` in your Lambda deployment package: [lambda-worker/otel-collector-config.yaml](https://github.com/temporalio/samples-go/blob/main/lambda-worker/otel-collector-config.yaml) ```yaml receivers: otlp: protocols: grpc: endpoint: "localhost:4317" http: endpoint: "localhost:4318" exporters: debug: awsxray: region: us-west-2 awsemf: # AWS EMF exporter for metrics # These are example configurations namespace: TemporalWorkerMetrics log_group_name: /aws/lambda/ region: us-west-2 dimension_rollup_option: NoDimensionRollup resource_to_telemetry_conversion: enabled: true service: pipelines: traces: receivers: [otlp] exporters: [awsxray, debug] metrics: receivers: [otlp] exporters: [awsemf] telemetry: logs: level: debug metrics: address: localhost:8888 ``` Set the following environment variable on the Lambda function to point the Collector at the bundled config: - `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` Enable X-Ray active tracing on the Lambda function: ```bash aws lambda update-function-configuration \ --function-name \ --tracing-config Mode=Active ``` The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role. Without these permissions, the Collector fails silently and no telemetry appears. If you only need metrics or tracing, use `otel.ApplyMetrics` or `otel.ApplyTracing` individually. --- # Serverless Workers on GCP Cloud Run - Go SDK Source: https://docs.temporal.io/develop/go/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the Go SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Go Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific package. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running Go Worker, then set `DeploymentOptions` in [`worker.Options`](https://pkg.go.dev/go.temporal.io/sdk/internal#WorkerOptions) to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```go package main import ( "log" "os" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/envconfig" "go.temporal.io/sdk/worker" "go.temporal.io/sdk/workflow" "example.com/myapp" ) func main() { c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() w := worker.New(c, os.Getenv("TEMPORAL_TASK_QUEUE"), worker.Options{ DeploymentOptions: worker.DeploymentOptions{ UseVersioning: true, Version: worker.WorkerDeploymentVersion{ DeploymentName: "my-app", BuildID: "build-1", }, }, }) w.RegisterWorkflowWithOptions(myapp.MyWorkflow, workflow.RegisterOptions{ VersioningBehavior: workflow.VersioningBehaviorPinned, }) w.RegisterActivity(myapp.MyActivity) if err := w.Run(worker.InterruptCh()); err != nil { log.Fatalln("Unable to start worker", err) } } ``` `DeploymentName` and `BuildID` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `VersioningBehaviorPinned` or `VersioningBehaviorAutoUpgrade`. Set it per Workflow at registration as shown above, or set `DefaultVersioningBehavior` in `DeploymentOptions` to cover every Workflow on the Worker. If a Version is set and neither is specified, registration panics with `workflow type does not have a versioning behavior`. For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/go/workers/run-worker-process). ## Configure the Temporal connection The `envconfig` package loads [Temporal Client](/develop/go/client/temporal-client) configuration from environment variables and an optional TOML config file, so the Worker code carries no Namespace or credentials. Set the non-secret values as environment variables on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager. For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration). `MustLoadDefaultClientOptions` panics if the configuration is invalid. To handle a bad configuration yourself, use `envconfig.LoadDefaultClientOptions` and check the returned error. ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/go/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over: ```go func MyActivity(ctx context.Context, input MyInput) (string, error) { for i := range input.Items { activity.RecordHeartbeat(ctx, i) // ... process input.Items[i] } return "done", nil } ``` For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Go SDK](/develop/go/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). --- # Worker Sessions - Go SDK Source: https://docs.temporal.io/develop/go/workers/sessions > Enable Worker Sessions, change concurrent Sessions, and create Session Workers using the Go SDK for precise Task Routing, ensuring efficient Activity Tasks execution. This page shows how to do the following: - [Enable Worker Sessions](#enable-sessions) - [Change the maximum concurrent Sessions of a Worker](#max-concurrent-sessions) - [Create a Worker Session](#create-a-session) > **💡 Tip:** > > This feature is currently available only in the Go SDK. > A Worker Session is a feature that provides a straightforward API for [Task Routing](/task-routing) to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. ## Enable Worker Sessions Set `EnableSessionWorker` to `true` in the Worker options. ```go {12,15-19} func main() { // The client and worker are heavyweight objects that should be created once per process. temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create client", err) } defer temporalClient.Close() workerOptions := worker.Options{ EnableSessionWorker: true, MaxConcurrentSessionExecutionSize: 1000, } w := worker.New(temporalClient, "fileprocessing", workerOptions) w.RegisterWorkflow(sessions.SomeFileProcessingWorkflow) w.RegisterActivity(&sessions.FileActivities{}) err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start worker", err) } } ``` ### Change the maximum concurrent Sessions of a Worker You can adjust the maximum concurrent Sessions of a Worker. To limit the number of concurrent Sessions running on a Worker, set the `MaxConcurrentSessionExecutionSize` field of `worker.Options` to the desired value. By default, this field is set to a very large value, so there's no need to manually set it if no limitation is needed. If a Worker hits this limitation, it won't accept any new `CreateSession()` requests until one of the existing sessions is completed. If the session can't be created within `CreationTimeout`, `CreateSession()` returns an error . ```go {5} func main() { // ... workerOptions := worker.Options{ EnableSessionWorker: true, MaxConcurrentSessionExecutionSize: 1000, } } ``` ## Create a Worker Session Within the Workflow code use the Workflow APIs to create a Session with whichever Worker picks up the first Activity Task. Use the [`CreateSession`](https://pkg.go.dev/go.temporal.io/sdk/workflow#CreateSession) API to create a Context object that can be passed to calls to spawn Activity Executions. Pass an instance of `workflow.Context` and [`SessionOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SessionOptions) to the `CreateSession` API call and get a Session Context that contains metadata information of the Session. Use the Session Context to spawn all Activity Executions that should belong to the Session. All associated Activity Tasks are then processed by the same Worker Entity. When the `CreateSession` API is called, the Task Queue name that is specified in `ActivityOptions` (or in `StartWorkflowOptions` if the Task Queue name is not specified in `ActivityOptions`) is used, and a Session is created with one of the Workers polling that Task Queue. The Session Context is cancelled if the Worker executing this Session dies or `CompleteSession()` is called. When using the returned Session Context to spawn Activity Executions, a `workflow.ErrSessionFailed` error is returned if the Session framework detects that the Worker executing this Session has died. The failure of Activity Executions won't affect the state of the Session, so you still need to handle the errors returned from your Activities and call `CompleteSession()` if necessary. If the context passed in already contains an open Session, `CreateSession()` returns an error. If all the Workers are currently busy and unable to handle a new Session, the framework keeps retrying until the `CreationTimeout` period you specified in `SessionOptions` has passed before returning an error. (For more details, check the "Concurrent Session Limitation" section.) `CompleteSession()` releases the resources reserved on the Worker, so it's important to call it as soon as you no longer need the Session. It cancels the session context and therefore all the Activity Executions using that Session Context. It is safe to call `CompleteSession()` on a failed Session, meaning that you can call it from a `defer` function after the Session is successfully created. If the Worker goes down between Activities, any scheduled Activities meant for the Session Worker are canceled. If not, you get a `workflow.ErrSessionFailed` error when the next call of `workflow.ExecuteActivity()` is made from that Workflow. ```go {13-17,23-28,37,44,50} package sessions import ( "time" "go.temporal.io/sdk/workflow" ) type FileProcessingWFParam struct { CloudFileLocation string } func SomeFileProcessingWorkflow(ctx workflow.Context, param FileProcessingWFParam) error { activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, } ctx = workflow.WithActivityOptions(ctx, activityOptions) err := processFile(ctx, param) return err } func processFile(ctx workflow.Context, param FileProcessingWFParam) error { sessionOptions := &workflow.SessionOptions{ CreationTimeout: time.Minute, ExecutionTimeout: time.Minute, } sessionCtx, err := workflow.CreateSession(ctx, sessionOptions) if err != nil { return err } defer workflow.CompleteSession(sessionCtx) var a *FileActivities var downloadResult FileActivityResult err = workflow.ExecuteActivity(sessionCtx, a.DownloadFile, param).Get(sessionCtx, &downloadResult) if err != nil { return err } processParam := FileActivityParam(downloadResult) var processResult FileActivityResult err = workflow.ExecuteActivity(sessionCtx, a.ProcessFile, processParam).Get(sessionCtx, &processResult) if err != nil { return err } uploadParam := FileActivityParam(processResult) err = workflow.ExecuteActivity(sessionCtx, a.UploadFile, uploadParam).Get(sessionCtx, nil) return err } ``` ## Additional Session usage information ```go type SessionInfo struct { // A unique Id for the session SessionID string // The hostname of the worker that is executing the session HostName string // ... other unexported fields } func GetSessionInfo(ctx Context) *SessionInfo ``` The Session Context also stores some Session metadata, which can be retrieved by the `GetSessionInfo()` API. If the Context passed in doesn't contain any Session metadata, this API will return a `nil` pointer. ### Recreate Session For long-running Sessions, you may want to use the `ContinueAsNew` feature to split the Workflow into multiple runs when all Activities need to be executed by the same Worker. The `RecreateSession()` API is designed for such a use case. ```go func RecreateSession(ctx Context, recreateToken []byte, sessionOptions *SessionOptions) (Context, error) ``` Its usage is the same as `CreateSession()` except that it also takes in a `recreateToken`, which is needed to create a new Session on the same Worker as the previous one. You can get the token by calling the `GetRecreateToken()` method of the `SessionInfo` object. ```go token := workflow.GetSessionInfo(sessionCtx).GetRecreateToken() ``` **Is there a complete example?** Yes, the [file processing example](https://github.com/temporalio/samples-go/tree/main/fileprocessing) in the [temporalio/samples-go](https://github.com/temporalio/samples-go) repo has been updated to use the session framework. **What happens to my Activity if the Worker dies?** If your Activity has already been scheduled, it will be canceled. If not, you will get a `workflow.ErrSessionFailed` error when you call `workflow.ExecuteActivity()`. **Is the concurrent session limitation per process or per host?** It's per Worker Process, so make sure there's only one Worker Process running on the host if you plan to use this feature. **Future Work** - Right now, a Session is considered failed if the Worker Process dies. However, for some use cases, you may only care whether the Worker host is alive or not. For these use cases, the Session should be automatically re-established if the Worker Process is restarted. - The current implementation assumes that all Sessions are consuming the same type of resource and there's only one global limitation. Our plan is to allow you to specify what type of resource your Session will consume and enforce different limitations on different types of resources. --- # Workflows - Go SDK Source: https://docs.temporal.io/develop/go/workflows > This section explains how to implement Workflows with the Go SDK ![Go SDK Banner](/img/assets/banner-go-temporal.png) ## Workflows - [Workflow basics](/develop/go/workflows/basics) - [Child Workflows](/develop/go/workflows/child-workflows) - [Continue-As-New](/develop/go/workflows/continue-as-new) - [Cancellation](/develop/go/workflows/cancellation) - [Timeouts](/develop/go/workflows/timeouts) - [Message passing](/develop/go/workflows/message-passing) - [Selectors](/develop/go/workflows/selectors) - [Side effects](/develop/go/workflows/side-effects) - [Schedules](/develop/go/workflows/schedules) - [Timers](/develop/go/workflows/timers) - [Dynamic Workflow](/develop/go/workflows/dynamic-workflow) - [Versioning](/develop/go/workflows/versioning) - [Workflow Streams](/develop/go/workflows/workflow-streams) --- # Workflow basics - Go SDK Source: https://docs.temporal.io/develop/go/workflows/basics > This section explains Workflow basics with the Go SDK ## How to develop a basic Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal Go SDK programming model, a [Workflow Definition](/workflow-definition) is an exportable function. Below is an example of a basic Workflow Definition. ```go package yourapp import ( "time" "go.temporal.io/sdk/workflow" ) func YourSimpleWorkflowDefinition(ctx workflow.Context) error { // ... return nil } ``` ### How to define Workflow parameters Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. The first parameter of a Go-based Workflow Definition must be of the [`workflow.Context`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Context) type. It is used by the Temporal Go SDK to pass around Workflow Execution context, and virtually all the Go SDK APIs that are callable from the Workflow require it. It is acquired from the [`go.temporal.io/sdk/workflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow) package. The `workflow.Context` entity operates similarly to the standard `context.Context` entity provided by Go. The only difference between `workflow.Context` and `context.Context` is that the `Done()` function, provided by `workflow.Context`, returns `workflow.Channel` instead of the standard Go `chan`. Additional parameters can be passed to the Workflow when it is invoked. A Workflow Definition may support multiple custom parameters, or none. These parameters can be regular type variables or safe pointers. However, the best practice is to pass a single parameter that is of a `struct` type, so there can be some backward compatibility if new parameters are added. All Workflow Definition parameters must be serializable and can't be channels, functions, variadic, or unsafe pointers. ```go {9-12,14} package yourapp import ( "time" "go.temporal.io/sdk/workflow" ) type YourWorkflowParam struct { WorkflowParamX string WorkflowParamY int } func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) { activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) activityParam := YourActivityParam{ ActivityParamX: param.WorkflowParamX, ActivityParamY: param.WorkflowParamY, } var a *YourActivityObject var activityResult YourActivityResultObject err := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam).Get(ctx, &activityResult) if err != nil { return nil, err } return nil } ``` ### How to define Workflow return parameters Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error. A Go-based Workflow Definition can return either just an `error` or a `customValue, error` combination. Again, the best practice here is to use a `struct` type to hold all custom values. A Workflow Definition written in Go can return both a custom value and an error. However, it's not possible to receive both a custom value and an error in the calling process, as is normal in Go. The caller will receive either one or the other. Returning a non-nil `error` from a Workflow indicates that an error was encountered during its execution and the Workflow Execution should be terminated, and any custom return values will be ignored by the system. ```go {11-14,35-38} package yourapp import ( "time" "go.temporal.io/sdk/workflow" ) // other structs and code type YourWorkflowResultObject struct { WFResultFieldX string WFResultFieldY int } func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) (*YourWorkflowResultObject, error) { activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, activityOptions) activityParam := YourActivityParam{ ActivityParamX: param.WorkflowParamX, ActivityParamY: param.WorkflowParamY, } var a *YourActivityObject var activityResult YourActivityResultObject err := workflow.ExecuteActivity(ctx, a.YourActivityDefinition, activityParam).Get(ctx, &activityResult) if err != nil { return nil, err } workflowResult := &YourWorkflowResultObject{ WFResultFieldX: activityResult.ResultFieldX, WFResultFieldY: activityResult.ResultFieldY, } return workflowResult, nil } ``` ### How to customize Workflow Type in Go In Go, by default, the Workflow Type name is the same as the function name. To customize the Workflow Type, set the `Name` parameter with `RegisterOptions` when registering your Workflow with a Worker. ```go {20,24-27} package main import ( "log" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/client" "go.temporal.io/sdk/worker" "go.temporal.io/sdk/workflow" "documentation-samples-go/yourapp" ) func main() { temporalClient, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer temporalClient.Close() yourWorker := worker.New(temporalClient, "your-custom-task-queue-name", worker.Options{}) yourWorker.RegisterWorkflow(yourapp.YourWorkflowDefinition) registerWFOptions := workflow.RegisterOptions{ Name: "JustAnotherWorkflow", } yourWorker.RegisterWorkflowWithOptions(yourapp.YourSimpleWorkflowDefinition, registerWFOptions) message := "This could be a connection string or endpoint details" number := 100 activities := &yourapp.YourActivityObject{ Message: &message, Number: &number, } registerAOptions := activity.RegisterOptions{ Name: "JustAnotherActivity", } yourWorker.RegisterActivityWithOptions(yourapp.YourSimpleActivityDefinition, registerAOptions) err = yourWorker.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start Worker", err) } } ``` ### How to develop Workflow logic Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow. In Go, Workflow Definition code cannot directly do the following: - Iterate over maps using `range`, because with `range` the order of the map's iteration is randomized. Instead you can collect the keys of the map, sort them, and then iterate over the sorted keys to access the map. This technique provides deterministic results. You can also use a Side Effect or an Activity to process the map instead. - Call an external API, conduct a file I/O operation, talk to another service, etc. (Use an Activity for these.) The Temporal Go SDK has APIs to handle equivalent Go constructs: - `workflow.Now()` This is a replacement for `time.Now()`. - `workflow.Sleep()` This is a replacement for `time.Sleep()`. - `workflow.GetLogger()` This ensures that the provided logger does not duplicate logs during a replay. - `workflow.Go()` This is a replacement for the `go` statement. - `workflow.Channel` This is a replacement for the native `chan` type. Temporal provides support for both buffered and unbuffered channels. - `workflow.Selector` This is a replacement for the `select` statement. Learn more on the [Go SDK Selectors](https://legacy-documentation-sdks.temporal.io/go/selectors) page. - `workflow.Context` This is a replacement for `context.Context`. See [Tracing](/develop/go/platform/observability#tracing) for more information about context propagation. #### Logging Use [`workflow.GetLogger(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLogger) instead of the standard `log` package or `fmt.Println`. The SDK logger skips log messages during replay to avoid duplicates: ```go func MyWorkflow(ctx workflow.Context, name string) (string, error) { logger := workflow.GetLogger(ctx) logger.Info("Starting workflow", "name", name) // ... } ``` For logger configuration, see [Observability: Log from a Workflow](/develop/go/platform/observability#logging). #### Random numbers and UUIDs The Go SDK does not provide a seeded random source or a UUID helper. Generate these inside a [Side Effect](/develop/go/workflows/side-effects), which records the result in the Event History and returns the recorded value on replay: ```go var id string encodedID := workflow.SideEffect(ctx, func(ctx workflow.Context) interface{} { return uuid.New().String() }) encodedID.Get(&id) ``` An Activity works for this too, and is the better choice when the value comes from an external system. A Side Effect is cheaper for purely local generation. #### Current time Use [`workflow.Now(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Now) instead of `time.Now()`. It returns the time of the last Workflow Task, which is consistent across replays: ```go currentTime := workflow.Now(ctx) ``` To wait, use [`workflow.Sleep(ctx, d)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) instead of `time.Sleep`. #### Detecting replay (advanced) Use [`workflow.IsReplaying(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#IsReplaying) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an Interceptor. > **⚠️ Caution:** > > Never use this to affect Workflow business logic. Branching on replay status breaks determinism. > ```go if !workflow.IsReplaying(ctx) { emitMetric("workflow_started", 1) } ``` --- # Cancel a Workflow - Go SDK Source: https://docs.temporal.io/develop/go/workflows/cancellation This page shows the following: - How to handle a Cancellation request within a Workflow. - How to set an Activity Heartbeat Timeout. - How to listen for and handle a Cancellation request within an Activity. - How to send a Cancellation request from a Temporal Client. - Heartbeating after a Cancellation. ## Handle Cancellation in Workflow Workflow Definitions can be written to handle execution cancellation requests with Go's `defer` and the `workflow.NewDisconnectedContext` API. In the Workflow Definition, there is a special Activity that handles clean up should the execution be cancelled. If the Workflow receives a Cancellation Request, but all Activities gracefully handle the Cancellation, and/or no Activities are skipped then the Workflow status will be Complete. It is completely up to the needs of the business process and your use case which determines whether you want to return the Cancellation error to show a Canceled status or Complete status regardless of whether a Cancellation has propagated to and/or skipped Activities. ```go {7-23,28,31,34} const WorkflowId = "example-cancellation-workflow" const TaskQueueName = "cancellation" func YourWorkflow(ctx workflow.Context) error { logger := workflow.GetLogger(ctx) var a *Activities activityOptions := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Minute, HeartbeatTimeout: 5 * time.Second, WaitForCancellation: true, } defer func() { if !errors.Is(ctx.Err(), workflow.ErrCanceled) { return } newCtx, _ := workflow.NewDisconnectedContext(ctx) err := workflow.ExecuteActivity(newCtx, a.CleanupActivity).Get(ctx, nil) if err != nil { logger.Error("CleanupActivity failed", "Error", err) } }() ctx = workflow.WithActivityOptions(ctx, activityOptions) var result string err := workflow.ExecuteActivity(ctx, a.ActivityToBeCanceled).Get(ctx, &result) logger.Info(fmt.Sprintf("ActivityToBeCanceled returns %v, %v", result, err)) err = workflow.ExecuteActivity(ctx, a.ActivityToBeSkipped).Get(ctx, nil) logger.Error("Error from ActivityToBeSkipped", "Error", err) return err } ``` ## Handle Cancellation in an Activity Ensure that the Activity is Heartbeating to receive the Cancellation request and stop execution. ```go func (a *Activities) ActivityToBeCanceled(ctx context.Context) (string, error) { logger := activity.GetLogger(ctx) logger.Info("Activity started, to cancel the Workflow Execution and this Activity, use 'go run cancel/cancel/main.go " + "-w ' or use the CLI: 'temporal workflow cancel --workflow-id '") // A for select statement is a common approach to listening for a Cancellation is an Activity for { select { case <-time.After(1 * time.Second): logger.Info("Heartbeating...") activity.RecordHeartbeat(ctx, "") // Listen for ctx.Done() to know if a Cancellation Request has propagated to the Activity. case <-ctx.Done(): logger.Info("This Activity is canceled!") return "I am canceled by Done", nil } } } ``` ## Request Cancellation Use the `CancelWorkflow` API to cancel a Workflow Execution using its Id. ```go func main() { temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create client", err) } defer temporalClient.Close() // Call the CancelWorkflow API to cancel a Workflow // In this call we are relying on the Workflow Id only. // But a Run Id can also be supplied to ensure the correct Workflow is Canceled. err = temporalClient.CancelWorkflow(context.Background(), cancellation.WorkflowId, "") if err != nil { log.Fatalln("Unable to cancel Workflow Execution", err) } log.Println("Workflow Execution cancelled", "WorkflowID", cancellation.WorkflowId) } ``` ## Heartbeating after a Cancellation Sometimes you may want to continue running your Activity, even after a Cancellation has been issued. You may want to completely ignore the cancellation and continue Activity execution, including Heartbeating, or you may want to send one final Heartbeat after Cancellation. Even though the context is cancelled when the Workflow is Cancelled, you are still able to send Activity Heartbeats. When you call `activity.RecordHeartbeat` after Cancellation has occurred, a `WARN RecordActivityHeartbeat with error Error context canceled` message will be logged, and a `context canceled` error will be returned from the call. However, the Heartbeat **has** still been sent. ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - Go SDK Source: https://docs.temporal.io/develop/go/workflows/child-workflows > Use the Go SDK to start a Child Workflow Execution and set a Parent Close Policy, including details on Workflow Options and future management. This page shows how to do the following: - [Start a Child Workflow Execution](#child-workflows) - [Set a Parent Close Policy](#parent-close-policy) ## Start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In Go, you must explicitly call `GetChildWorkflowExecution()` on the `ChildWorkflowFuture` and then call `Get()` on the returned Future to wait for this Event. See the [Async Child Workflows](#async-child-workflows) section below for a complete example. To spawn a [Child Workflow Execution](/child-workflows) in Go, use the [`ExecuteChildWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ExecuteChildWorkflow) API, which is available from the `go.temporal.io/sdk/workflow` package. The `ExecuteChildWorkflow` call requires an instance of [`workflow.Context`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Context), with an instance of [`workflow.ChildWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowOptions) applied to it, the Workflow Type, and any parameters that should be passed to the Child Workflow Execution. `workflow.ChildWorkflowOptions` contain the same fields as `client.StartWorkflowOptions`. Workflow Option fields automatically inherit their values from the Parent Workflow Options if they are not explicitly set. If a custom `WorkflowID` is not set, one is generated when the Child Workflow Execution is spawned. Use the [`WithChildOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#WithChildOptions) API to apply Child Workflow Options to the instance of `workflow.Context`. The `ExecuteChildWorkflow` call returns an instance of a [`ChildWorkflowFuture`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowFuture). Call the `.Get()` method on the instance of `ChildWorkflowFuture` to wait for the result. ```go func YourWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) { childWorkflowOptions := workflow.ChildWorkflowOptions{} ctx = workflow.WithChildOptions(ctx, childWorkflowOptions) var result ChildResp err := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{}).Get(ctx, &result) if err != nil { // ... } // ... return resp, nil } func YourOtherWorkflowDefinition(ctx workflow.Context, params ChildParams) (ChildResp, error) { // ... return resp, nil } ``` ### Async Child Workflows To asynchronously spawn a Child Workflow Execution, the Child Workflow must have an "Abandon" Parent Close Policy set in the Child Workflow Options. Additionally, the Parent Workflow Execution must wait for the `ChildWorkflowExecutionStarted` Event to appear in its Event History before it completes. If the Parent makes the `ExecuteChildWorkflow` call and then immediately completes, the Child Workflow Execution does not spawn. To be sure that the Child Workflow Execution has started, first call the `GetChildWorkflowExecution` method on the instance of the `ChildWorkflowFuture`, which will return a different Future. Then call the `Get()` method on that Future, which is what will wait until the Child Workflow Execution has spawned. ```go import ( // ... "go.temporal.io/api/enums/v1" ) func YourWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) { childWorkflowOptions := workflow.ChildWorkflowOptions{ ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON, } ctx = workflow.WithChildOptions(ctx, childWorkflowOptions) childWorkflowFuture := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{}) // Wait for the Child Workflow Execution to spawn var childWE workflow.Execution if err := childWorkflowFuture.GetChildWorkflowExecution().Get(ctx, &childWE); err != nil { return err } // ... return resp, nil } func YourOtherWorkflowDefinition(ctx workflow.Context, params ChildParams) (ChildResp, error) { // ... return resp, nil } ``` #### Set a Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. In Go, a Parent Close Policy is set on the `ParentClosePolicy` field of an instance of [`workflow.ChildWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#ChildWorkflowOptions). The possible values can be obtained from the [`go.temporal.io/api/enums/v1`](https://pkg.go.dev/go.temporal.io/api/enums/v1#ParentClosePolicy) package. - `PARENT_CLOSE_POLICY_ABANDON` - `PARENT_CLOSE_POLICY_TERMINATE` - `PARENT_CLOSE_POLICY_REQUEST_CANCEL` The Child Workflow Options are then applied to the instance of `workflow.Context` by using the `WithChildOptions` API, which is then passed to the `ExecuteChildWorkflow()` call. - Type: [`ParentClosePolicy`](https://pkg.go.dev/go.temporal.io/api/enums/v1#ParentClosePolicy) - Default: `PARENT_CLOSE_POLICY_TERMINATE` ```go import ( // ... "go.temporal.io/api/enums/v1" ) func YourWorkflowDefinition(ctx workflow.Context, params ParentParams) (ParentResp, error) { // ... childWorkflowOptions := workflow.ChildWorkflowOptions{ // ... ParentClosePolicy: enums.PARENT_CLOSE_POLICY_ABANDON, } ctx = workflow.WithChildOptions(ctx, childWorkflowOptions) childWorkflowFuture := workflow.ExecuteChildWorkflow(ctx, YourOtherWorkflowDefinition, ChildParams{}) // ... } func YourOtherWorkflowDefinition(ctx workflow.Context, params ChildParams) (ChildResp, error) { // ... return resp, nil } ``` --- # Continue-As-New - Go SDK Source: https://docs.temporal.io/develop/go/workflows/continue-as-new > Use Temporal's Continue-As-New in Go to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for Go developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) - [How to test Continue-as-New?](#how-to-test) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the Go SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically set to `None` for the original caller of the Workflow. [View the source code](https://github.com/temporalio/samples-go/blob/main/safe_message_handler/workflow.go) in the context of the rest of the application code. ```go ClusterManagerInput struct { State *ClusterManagerState TestContinueAsNew bool } func newClusterManager(ctx workflow.Context, wfInput ClusterManagerInput) (*ClusterManager, error) { ```` The test hook in the above snippet is covered [below](#how-to-test). Inside your Workflow, return the [`NewContinueAsNewError`](https://pkg.go.dev/go.temporal.io/sdk/workflow#NewContinueAsNewError) error. This stops the Workflow right away and starts a new one. [View the source code](https://github.com/temporalio/samples-go/blob/main/safe_message_handler/workflow.go) in the context of the rest of the application code. ```go return ClusterManagerResult{}, workflow.NewContinueAsNewError( ctx, ClusterManagerWorkflow, ClusterManagerInput{ State: &cm.state, TestContinueAsNew: cm.testContinueAsNew, }, ) ```` ### Considerations for Workflows with Message Handlers If you use Updates or Signals, don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish in your main Workflow before you return `NewContinueAsNewError`. See the [`AllHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance. ## When is it right to Continue-as-New using the Go SDK? Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `GetInfo(ctx).GetContinueAsNewSuggested()` to check if it's time. ## How to test Continue-as-New using the Go SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, when `TestContinueAsNew == True`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value. A helper method in the Workflow checks it each time it considers using Continue-as-New: [View the source code](https://github.com/temporalio/samples-go/blob/main/safe_message_handler/workflow.go) in the context of the rest of the application code. ```go func (cm *ClusterManager) shouldContinueAsNew(ctx workflow.Context) bool { if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { return true } if cm.maxHistoryLength > 0 && workflow.GetInfo(ctx).GetCurrentHistoryLength() > cm.maxHistoryLength { return true } return false } ``` --- # Dynamic Workflow - Go SDK Source: https://docs.temporal.io/develop/go/workflows/dynamic-workflow > This section explains Dynamic Workflows with the Go SDK ## Set a Dynamic Workflow A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. A Workflow can be registered as dynamic by using `worker.RegisterDynamicWorkflow()`. You must register the Workflow with the Worker before it can be invoked. Only one Dynamic Workflow can be present on a Worker. The Workflow Definition must then accept a single argument of type `converter.EncodedValues`. This code snippet is taken from the [Dynamic Workflow example from samples-go](https://github.com/temporalio/samples-go/tree/main/dynamic-workflows). ```go func DynamicWorkflow(ctx workflow.Context, args converter.EncodedValues) (string, error) { var result string info := workflow.GetInfo(ctx) var arg1, arg2 string err := args.Get(&arg1, &arg2) if err != nil { return "", fmt.Errorf("failed to decode arguments: %w", err) } if info.WorkflowType.Name == "dynamic-activity" { ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{StartToCloseTimeout: 10 * time.Second}) err := workflow.ExecuteActivity(ctx, "random-activity-name", arg1, arg2).Get(ctx, &result) if err != nil { return "", err } } else { result = fmt.Sprintf("%s - %s - %s", info.WorkflowType.Name, arg1, arg2) } return result, nil } ``` --- # Workflow message passing - Go SDK Source: https://docs.temporal.io/develop/go/workflows/message-passing > Develop with Queries, Signals, and Updates with the Temporal Go SDK. A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates. The Workflow implementation defines these endpoints via handler methods that can react to incoming Queries and Updates, and via Signal channels. Temporal Clients use messages to read Workflow state and control its execution. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal Go SDK. ## Handle messages > **ℹ️ Info:** > The code that follows is part of a working message passing [sample](https://github.com/temporalio/samples-go/tree/message-passing/message-passing-intro). Follow these guidelines when writing message handlers: - Values sent in messages, and the return values of message handlers and the main Workflow function, must be [serializable](/dataconversion). - Prefer using a single struct over multiple input parameters. This allows you to add fields without changing the calling signature. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution: ```go type Language string const Chinese Language = "chinese" const English Language = "english" const French Language = "french" const Spanish Language = "spanish" const Portuguese Language = "portuguese" const GetLanguagesQuery = "GetLanguages" type GetLanguagesInput struct { IncludeUnsupported bool } func GreetingWorkflow(ctx workflow.Context) (string, error) { ... greeting := map[Language]string{English: "Hello", Chinese: "你好，世界"} err := workflow.SetQueryHandler(ctx, GetLanguagesQuery, func(input GetLanguagesInput) ([]Language, error) { // 👉 A Query handler returns a value: it can inspect but must not mutate the Workflow state. if input.IncludeUnsupported { return []Language{Chinese, English, French, Spanish, Portuguese}, nil } else { // Range over map is a nondeterministic operation. // It is OK to have a non-deterministic operation in a query function. //workflowcheck:ignore return maps.Keys(greeting), nil } }) ... } ``` - Use [`SetQueryHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetQueryHandler) to set a Query Handler that listens for a Query by name. - The handler must be a function that returns two values, a serializable result and an error. - You can't perform async operations such as executing an Activity in a Query handler. ### Signal Channels A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow. Handle Signal messages by receiving them from their channel: ```go const ApproveSignal = "approve" type ApproveInput struct { Name string } func GreetingWorkflow(ctx workflow.Context) error { logger := workflow.GetLogger(ctx) approverName := "" ... // Block until the language is approved var approveInput ApproveInput workflow.GetSignalChannel(ctx, ApproveSignal).Receive(ctx, &approveInput) approverName = approveInput.Name logger.Info("Received approval", "Approver", approverName) ... } ``` - Pass the Signal's name to [`GetSignalChannel`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetSignalChannel) to get the Signal Channel that listens for Signals of that type. Alternatively, you might want the Workflow to proceed and still be capable of handling external Signals. ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) error { var signal MySignal signalChan := workflow.GetSignalChannel(ctx, "your-signal-name") workflow.Go(ctx, func(ctx workflow.Context) { for { selector := workflow.NewSelector(ctx) selector.AddReceive(signalChan, func(c workflow.ReceiveChannel, more bool) { c.Receive(ctx, &signal) }) selector.Select(ctx) } }) // You could now submit an activity; any signals will still be received while the activity is pending. } ``` In the example above, the Workflow code uses `workflow.GetSignalChannel` to open a `workflow.Channel` for the Signal type (identified by the Signal name). - Before completing the Workflow or using [Continue-As-New](/develop/go/workflows/continue-as-new), make sure to do an asynchronous drain on the Signal channel. Otherwise, the Signals will be lost. The [batch sliding window](https://github.com/temporalio/samples-go/tree/main/batch-sliding-window) sample contains an example: - Delay calling `workflow.GetSignalChannel` until the Workflow initialization needed to process the Signal channel has finished. This is safe because the SDK buffers signals when there are no channels created for them. ```go reportCompletionChannel := workflow.GetSignalChannel(ctx, "ReportCompletion") // Drain signals async for { var recordId int ok := reportCompletionChannel.ReceiveAsync(&recordId) if !ok { break } s.recordCompletion(ctx, recordId) } ``` ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```go type Language string const SetLanguageUpdate = "set-language" func GreetingWorkflow(ctx workflow.Context) error { language := English err = workflow.SetUpdateHandlerWithOptions(ctx, SetLanguageUpdate, func(ctx workflow.Context, newLanguage Language) (Language, error) { // 👉 An Update handler can mutate the Workflow state and return a value. var previousLanguage Language previousLanguage, language = language, newLanguage return previousLanguage, nil }, workflow.UpdateHandlerOptions{ Validator: func(ctx workflow.Context, newLanguage Language) error { if _, ok := greeting[newLanguage]; !ok { // 👉 In an Update validator you return any error to reject the Update. return fmt.Errorf("%s unsupported language", newLanguage) } return nil }, }) ... } ``` - Register an Update handler for a given name using either [workflow.SetUpdateHandler](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) or [workflow.SetUpdateHandlerWithOptions](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandlerWithOptions). - The handler must be a function that accepts a `workflow.Context` as its first parameter. - The function can return either a serializable value with an error or just an error. - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you don't need a validator. - To set a validator, pass the validator function in the [workflow.UpdateHandlerOptions](https://pkg.go.dev/go.temporal.io/sdk@v1.29.1/internal#UpdateHandlerOptions) when calling [workflow.SetUpdateHandlerWithOptions](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandlerWithOptions). The validator must be a function that accepts the same argument types as the handler and returns a single value of type error. - Accepting and rejecting Updates with validators: - To reject an Update you must return an error or panic in the validator. The Workflow's `WorkflowPanicPolicy` determines how panics are handled inside the Handler function. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into History whether the acceptance was automatic or due to a validator function not throwing an error or panicking. - When a validator throws an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History. The caller receives an "Update failed" error. - Use [`workflow.GetCurrentUpdateInfo`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetCurrentUpdateInfo) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - Update handlers can use Activities, Child Workflows, durable [workflow.Sleep](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) Timers, [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) conditions, and more. See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines. - Delay calling [`workflow.SetUpdateHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) until the Workflow initialization needed by Update handlers is finished. This is safe because the SDK buffers messages when there are no registered handlers for them. Note that [`workflow.SetUpdateHandler`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SetUpdateHandler) will immediately invoke the handler of buffered Updates with matching types. This could lead to out-of-order processing of messages with different types. ## Send messages To send Queries, Signals, or Updates, you call methods on a Temporal [Client](https://pkg.go.dev/go.temporal.io/sdk/client#Client). To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition. > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow function, just as you would complete or fail a Workflow Execution. > ### Send a Query Queries are sent from a Temporal Client. Use [`Client.QueryWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.QueryWorkflow) or [`Client.QueryWorkflowWithOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.QueryWorkflowWithOptions). ```go // ... supportedLangResult, err := temporalClient.QueryWorkflow(context.Background(), we.GetID(), we.GetRunID(), message.GetLanguagesQuery, message.GetLanguagesInput{IncludeUnsupported: false}) if err != nil { log.Fatalf("Unable to query workflow: %v", err) } var supportedLang []message.Language err = supportedLangResult.Get(&supportedLang) if err != nil { log.Fatalf("Unable to get query result: %v", err) } log.Println("Supported languages:", supportedLang) // ... ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### Send a Signal from a Client Use [`Client.SignalWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.SignalWorkflow). Pass in both the [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) and [Run Id](/workflow-execution/workflowid-runid#run-id) to uniquely identify the Workflow Execution. If only the Workflow Id is supplied (provide an empty string as the Run Id param), the Workflow Execution that is running receives the Signal. ```go // ... err = temporalClient.SignalWorkflow(context.Background(), we.GetID(), we.GetRunID(), message.ApproveSignal, message.ApproveInput{Name: ""}) if err != nil { log.Fatalf("Unable to signal workflow: %v", err) } // ... ``` - The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution. - The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History. #### Sending a Signal from a Workflow A Workflow can send a Signal to another Workflow, in which case it's called an External Signal. ```go // ... func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) error { ... signal := MySignal { Message: "Some important data", } err := workflow.SignalExternalWorkflow(ctx, "some-workflow-id", "", "your-signal-name", signal).Get(ctx, nil) if err != nil { // ... } // ... } ``` When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. #### Signal-With-Start Signal-With-Start is used from the Client. It takes a Workflow Id, Workflow arguments, a Signal name, and Signal arguments. If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled. Use the [`Client.SignalWithStartWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.SignalWithStartWorkflow) API to start a Workflow Execution (if not already running) and pass it the Signal at the same time. Because the Workflow Execution might not exist, this API does not take a Run Id as a parameter ```go // ... signal := MySignal { Message: "Some important data", } err = temporalClient.SignalWithStartWorkflow(context.Background(), "your-workflow-id", "your-signal-name", signal) if err != nil { log.Fatalln("Error sending the Signal", err) return } ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A Client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. Setting a timeout with a context provides a hard limit on how long a client will wait for a response. If you need a response as soon as the Server receives the request, use a Signal instead. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. Use the [`Client.UpdateWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.UpdateWorkflow) API to send an Update to a Workflow Execution. You must provide the Workflow Id, but specifying a Run Id is optional. If you supply only the Workflow Id (and provide an empty string as the Run Id param), the running Workflow Execution receives the Update. You must provide a `WaitForStage` when calling `UpdateWorkflow()`. This parameter controls the stage the update must reach before returning a handle to the caller: - If `WaitForStage` is set to `WorkflowUpdateStageCompleted`, the handle is returned after the Update completes. - If `WaitForStage` is set to `WorkflowUpdateStageAccepted`, the handle is returned after the Update is accepted (that is, after the validator has run, if there is a validator). You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. ```go ctxWithTimeout, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel() updateHandle, err := temporalClient.UpdateWorkflow(ctxWithTimeout, client.UpdateWorkflowOptions{ WorkflowID: we.GetID(), RunID: we.GetRunID(), UpdateName: message.SetLanguageUpdate, WaitForStage: client.WorkflowUpdateStageAccepted, Args: []interface{}{message.Chinese}, }) if err != nil { log.Fatalf("Unable to update workflow: %v", err) } var previousLang message.Language err = updateHandle.Get(ctxWithTimeout, &previousLang) if err != nil { log.Fatalf("Unable to get update result: %v", err) } ``` #### Update-With-Start > **💡 Tip:** > > For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended. > [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/go/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. Use the [`Client.UpdateWithStartWorkflow`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.UpdateWithStartWorkflow) API call. It returns once the requested Update wait stage has been reached; or when a provided context times out. Use [`WorkflowUpdateHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowUpdateHandle) to retrieve a result from the Update. You will need to provide: - [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions). The [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) is required. Choose "Use Existing" and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure. Not all `StartWorkflowOptions` are allowed. For example, specifying a Cron Schedule will result in an error. Refer to the [API documentation](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions) for further details. - [`UpdateWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/internal#UpdateWorkflowOptions). Same as for [Update Workflow](/develop/go/workflows/message-passing#send-update-from-client), the update name and an update wait stage must be specified. For Update-with-Start, the Workflow Id is optional. When specified, the Id must match the one used in `StartWorkflowOptions`. Since a running Workflow Execution may not already exist, you can't set a Run Id. - [`Client.NewWithStartWorkflowOperation`](https://pkg.go.dev/go.temporal.io/sdk/client#Client.NewWithStartWorkflowOperation). Specify the workflow options, method and arguments. Note that a `WithStartWorkflowOperation` can only be used once. Re-using a previously used operation returns an error from `UpdateWithStartWorkflow`. The following example shows the creation, configuration, and use of UpdateWithStart: ```go ctxWithTimeout, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel() workflowOptions := client.StartWorkflowOptions{ ID: "some-workflow-id", TaskQueue: "some-task-queue", WorkflowIDConflictPolicy: enumspb.WORKFLOW_ID_CONFLICT_POLICY_USE_EXISTING, } updateOptions := client.UpdateWorkflowOptions{ UpdateName: message.SetLanguageUpdate, WaitForStage: client.WorkflowUpdateStageCompleted, } startWorkflowOp := temporalClient.NewWithStartWorkflowOperation(workflowOptions, MyWorkflow) updateHandle, err := temporalClient.UpdateWithStartWorkflow( ctxWithTimeout, client.UpdateWithStartWorkflowOptions{ StartWorkflowOperation: startWorkflowOp, UpdateOptions: updateOptions, }) if err != nil { log.Fatalf("Unable to execute update-with-start: %v", err) } var previousLang message.Language err = updateHandle.Get(ctxWithTimeout, &previousLang) if err != nil { log.Fatalf("Unable to obtain update result: %v", err) } workflowRun, err := startWorkflowOp.Get(ctxWithTimeout) if err != nil { log.Fatalf("Unable to obtain workflow run: %v", err) } ``` For more examples, see the [Go sample for early-return pattern](https://github.com/temporalio/samples-go/tree/main/early-return). ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-go/blob/main/safe_message_handler/README.md) demonstrating safe blocking message handling. > ### Blocking handlers Signal and Update handlers can block. This allows you to use Activities, Child Workflows, durable [workflow.Sleep](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) Timers, [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) conditions, etc. This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls. It's essential to understand the things that could go wrong in order to use blocking handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of blocking Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below. The following code modifies the Update handler from earlier on in this page. The Update handler now makes a blocking call to execute an Activity: ```go func GreetingWorkflow(ctx workflow.Context) error { language := English err = workflow.SetUpdateHandler(ctx, SetLanguageUpdate, func(ctx workflow.Context, newLanguage Language) (Language, error) { if _, ok := greeting[newLanguage]; !ok { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var greeting string err := workflow.ExecuteActivity(ctx, CallGreetingService, newLanguage).Get(ctx, &greeting) if err != nil { return nil, err } greeting[newLanguage] = greeting } var previousLanguage Language previousLanguage, language = language, newLanguage return previousLanguage, nil }) ... } ``` ### Add blocking wait conditions Sometimes, blocking Signal or Update handlers need to meet certain conditions before they should continue. You can use [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) to prevent the code from proceeding until a condition is true. You specify the condition by passing a function that returns `true` or `false`. This is an important feature that helps you control your handler logic. Here are three important use cases for `Workflow.await`: - Waiting until a specific Update has arrived. - Waiting in a handler until it is appropriate to continue. - Waiting in the main Workflow until all active handlers have finished. ```go err = workflow.SetUpdateHandler(ctx, "UpdateHandler", func(ctx workflow.Context, input UpdateInput) error { workflow.Await(ctx, updateUnblockedFunc) ... }) ``` This is necessary if your Update handlers require something in the main Workflow function to be done first, since an Update handler can execute concurrently with the main Workflow function. You can also use `Workflow.await` anywhere else in the handler to wait for a specific condition to become true. This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true. #### Ensure your handlers finish before the Workflow completes `Workflow.await` can ensure your handler completes before a Workflow finishes. When your Workflow uses blocking Update handlers, your main Workflow method can return or Continue-as-New while a handler is still waiting on an async task, such as an Activity. The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying to retrieve Update results. Use [`workflow.Await`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Await) to wait for [`AllHandlersFinished`](https://pkg.go.dev/go.temporal.io/sdk/workflow#AllHandlersFinished) to return `true` to address this problem and allow your Workflow to end smoothly: ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) error { ... err = workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }) return nil } ``` By default, your Worker will log a warning if you allow your Workflow Execution to finish with unfinished Update handler executions. You can silence these warnings on a per-handler basis by setting `UnfinishedPolicy` field on [`workflow.UpdateHandlerOptions`](https://pkg.go.dev/go.temporal.io/sdk/workflow#UpdateHandlerOptions) struct: ```go err = workflow.SetUpdateHandlerWithOptions(ctx, UpdateHandlerName, UpdateFunc, workflow.UpdateHandlerOptions{ UnfinishedPolicy: workflow.HandlerUnfinishedPolicyAbandon, }) ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. #### Use `workflow.Mutex` to prevent concurrent handler execution See [Message handler concurrency](/handling-messages#message-handler-concurrency). Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```go // ... func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) error { ... err := workflow.SetUpdateHandler(ctx, "BadUpdateHandler", func(ctx workflow.Context) error { ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var result Data err := workflow.ExecuteActivity(ctx, FetchData, name).Get(ctx, &result) x = result.x // 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then // there may be times when the Workflow has self.x from one Activity execution and self.y from another. err = workflow.Sleep(ctx, time.Second) if err != nil { return err } y = result.y }) ... } ``` Coordinating access with `workflow.Mutex` corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```go func YourWorkflowDefinition(ctx workflow.Context, param YourWorkflowParam) error { ... err := workflow.SetUpdateHandler(ctx, "SafeUpdateHandler", func(ctx workflow.Context) error { err := mutex.Lock(ctx) if err != nil { return err } defer mutex.Unlock() ao := workflow.ActivityOptions{ StartToCloseTimeout: 10 * time.Second, } ctx = workflow.WithActivityOptions(ctx, ao) var result Data err := workflow.ExecuteActivity(ctx, FetchData, name).Get(ctx, &result) x = data.x // ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow // method, but no other execution of this handler can run until this execution finishes. err = workflow.Sleep(ctx, time.Second) if err != nil { return err } self.y = data.y }) ... } ``` ## Troubleshooting See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Go-specific discussion of this topic. When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The Client can't contact the server** - **The Workflow does not exist** Unlike Signals, for Queries and Updates, the Client waits for a response from the Worker. If an issue occurs during the handler execution by the Worker, the Client may receive an exception. ### Problems when sending an Update - **There is no Workflow Worker polling the Task Queue** Your request will be retried by the SDK Client until the calling context is cancelled. - **Update failed.** Update failures are like [Workflow failures](/references/failures). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity if the activity retries have been set to a finite number - The Workflow author returning an `error` - A panic in the handler, depending on the `WorkflowPanicPolicy` - **The handler caused the Workflow Task to fail** A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a [`FAILED_PRECONDITION`](https://pkg.go.dev/go.temporal.io/api/serviceerror#FailedPrecondition) error. - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use a [`WorkflowUpdateHandle`](https://pkg.go.dev/go.temporal.io/sdk/client#WorkflowUpdateHandle) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`ServiceError`](https://pkg.go.dev/go.temporal.io/api/serviceerror#ServiceError) "workflow execution already completed"`. This will happen if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Problems when sending a Query - **There is no Workflow Worker polling the Task Queue** You'll receive a [`ServiceError`](https://pkg.go.dev/go.temporal.io/api/serviceerror#ServiceError) on which the `status` is `FAILED_PRECONDITION`. - **Query failed.** You'll receive a [`QueryFailed`](https://pkg.go.dev/go.temporal.io/api/serviceerror#QueryFailed) error. Any panic in a Query handler will trigger this error. This differs from Signal and Update, where panics can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. --- # Schedules - Go SDK Source: https://docs.temporal.io/develop/go/workflows/schedules > Schedule Workflows, start them with delays or as Temporal Cron Jobs using the Go SDK. Master scheduling, backfilling, pausing, deleting, and updating Workflows. This page shows how to do the following: - [Scheduled Workflows](#schedule-a-workflow) - [Create a Schedule](#create-schedule) - [Backfill a Schedule](#backfill-schedule) - [Delete a Schedule](#delete-schedule) - [Describe a Schedule](#describe-schedule) - [List Schedules](#list-schedules) - [Pause a Schedule](#pause-schedule) - [Trigger a Schedule](#trigger-schedule) - [Update a Schedule](#update-schedule) - [Start delay](#start-delay) - [Temporal Cron Jobs](#temporal-cron-jobs) ## Scheduled Workflows Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes. Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process. Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy). If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions. ### Create a Schedule Schedules are initiated with the `create` call. The user generates a unique Schedule ID for each new Schedule. To create a Schedule in Go, use `Create()` on the [Client](/encyclopedia/temporal-client). Schedules must be initialized with a Schedule ID, [Spec](/schedule), and [Action](/schedule) in `client.ScheduleOptions{}`. ```go {11-12,14-22} func main() { ctx := context.Background() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() scheduleID := "schedule_id" workflowID := "schedule_workflow_id" scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: scheduleID, Spec: client.ScheduleSpec{}, Action: &client.ScheduleWorkflowAction{ ID: workflowID, Workflow: schedule.ScheduleWorkflow, TaskQueue: "schedule", }, }) if err != nil { log.Fatalln("Unable to create schedule", err) } log.Println("Schedule created", "ScheduleID", scheduleID) _, _ = scheduleHandle.Describe(ctx) } ``` > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > ### Backfill a Schedule Backfilling a Schedule executes [Workflow Tasks](/tasks#workflow-task) ahead of the Schedule's specified time range. This is useful for executing a missed or delayed Action, or for testing the Workflow ahead of time. To backfill a Schedule in Go, use `Backfill()` on `ScheduleHandle`. Specify the start and end times to execute the Workflow, along with the overlap policy. ```go {34-47} func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Create a Workflow to backfill temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() workflowID := "schedule_workflow_" // create paused Workflow now := time.Now() scheduleHandle, _ := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: "backfill-schedule", Spec: client.ScheduleSpec{ Intervals: []client.ScheduleIntervalSpec{ { Every: time.Minute, }, }, }, Action: &client.ScheduleWorkflowAction{ ID: workflowID, Workflow: schedule.ScheduleWorkflow, TaskQueue: "schedule", }, Paused: true, }) err = scheduleHandle.Backfill(ctx, client.ScheduleBackfillOptions{ Backfill: []client.ScheduleBackfill{ { Start: now.Add(-4 * time.Minute), End: now.Add(-2 * time.Minute), Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL, }, { Start: now.Add(-2 * time.Minute), End: now, Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL, }, }, }) if err != nil { log.Fatalln("Unable to Backfill Schedule", err) } } ``` ### Delete a Schedule Deleting a Schedule erases a Schedule. Deletion does not affect any Workflows started by the Schedule. To delete a Schedule, use `Delete()` on the `ScheduleHandle`. ```go {14-20} func main() { ctx := context.Background() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() scheduleID := "schedule_id" // Retrieve the schedule handle by its ID scheduleHandle := temporalClient.ScheduleClient().GetHandle(ctx, scheduleID) defer func() { log.Println("Deleting schedule", "ScheduleID", scheduleHandle.GetID()) err = scheduleHandle.Delete(ctx) if err != nil { log.Fatalln("Unable to delete schedule", err) } }() } ``` ### Describe a Schedule `Describe` retrieves information about the current Schedule configuration. This can include details about the Schedule Spec (such as Intervals), CronExpressions, and Schedule State. To describe a Schedule, use `Describe()` on the ScheduleHandle. ```go {23} func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() // create Schedule scheduleHandle, _ := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: "test-schedule-describe-spec-cron-schedule", Spec: client.ScheduleSpec{ CronExpressions: []string{ "0 12 * * MON", }, }, }) // describe schedule scheduleHandle.Describe(ctx) } ``` ### List Schedules The `List` action returns all available Schedules and their respective Schedule IDs. To return information on all Schedules, use `ScheduleClient.List()`. ```go {29-35} func main() { ctx := context.Background() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() // Create Schedule and Workflow IDs scheduleID := "schedule_" + uuid.New() workflowID := "schedule_workflow_" + uuid.New() // Create the schedule. scheduleHandle, err := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: scheduleID, Spec: client.ScheduleSpec{}, Action: &client.ScheduleWorkflowAction{ ID: workflowID, Workflow: schedule.ScheduleWorkflow, TaskQueue: "schedule", }, }) if err != nil { log.Fatalln("Unable to create schedule", err) } scheduleHandle.GetID() // list schedules listView, _ := temporalClient.ScheduleClient().List(ctx, client.ScheduleListOptions{ PageSize: 1, }) for listView.HasNext() { log.Println(listView.Next()) } } ``` ### Pause a Schedule `Pause` and `Unpause` enable the start or stop of all future Workflow Runs on a given Schedule. Pausing a Schedule halts all future Workflow Runs. Pausing can be enabled by setting `State.Paused` to `true`, or by using `Pause()` on the ScheduleHandle. Unpausing a Schedule allows the Workflow to execute as planned. To unpause a Schedule, use `Unpause()` on `ScheduleHandle`. ```go {18-20,30-32} func main() { ctx := context.Background() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() scheduleID := "schedule_id" scheduleHandle := temporalClient.ScheduleClient().GetHandle(ctx, scheduleID) if scheduleHandle == nil { log.Fatalln("Unable to retrieve schedule") } // Pause the schedule and print the status err = scheduleHandle.Pause(ctx, client.SchedulePauseOptions{ Note: "The Schedule has been paused.", }) if err != nil { log.Fatalln("Unable to pause schedule", err) } fmt.Println("The Schedule has been paused.") // Wait for 5 seconds time.Sleep(5 * time.Second) // Unpause the schedule err = scheduleHandle.Unpause(ctx, client.ScheduleUnpauseOptions{ Note: "The Schedule has been unpaused.", }) if err != nil { log.Fatalln("Unable to unpause schedule", err) } fmt.Println("The Schedule has been unpaused.") } ``` ### Trigger a Schedule Triggering a Schedule immediately executes an Action defined in that Schedule. By default, `trigger` is subject to the Overlap Policy. To trigger a Scheduled Workflow Execution, use `trigger()` on `ScheduleHandle`. ```go {22-27} func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() scheduleHandle, _ := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: "trigger-schedule", Spec: client.ScheduleSpec{}, Action: &client.ScheduleWorkflowAction{}, Paused: true, Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL, }) // Trigger Schedule for i := 0; i < 5; i++ { scheduleHandle.Trigger(ctx, client.ScheduleTriggerOptions{ Overlap: enums.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL, }) time.Sleep(2 * time.Second) } } ``` ### Update a Schedule Updating a Schedule changes the configuration of an existing Schedule. These changes can be made to Workflow Actions, Action parameters, Memos, and the Workflow's Cancellation Policy. Use `Update()` on the ScheduleHandle to modify a Schedule. ```go {20-28} func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() temporalClient, err := client.Dial(client.Options{ HostPort: client.DefaultHostPort, }) if err != nil { log.Fatalln("Unable to create Temporal Client", err) } defer temporalClient.Close() scheduleHandle, _ := temporalClient.ScheduleClient().Create(ctx, client.ScheduleOptions{ ID: "update-schedule", Spec: client.ScheduleSpec{}, Action: &client.ScheduleWorkflowAction{}, Paused: true, }) updateSchedule := func(input client.ScheduleUpdateInput) (*client.ScheduleUpdate, error) { return &client.ScheduleUpdate{ Schedule: &input.Description.Schedule, }, nil } _ = scheduleHandle.Update(ctx, client.ScheduleUpdateOptions{ DoUpdate: updateSchedule, }) } ``` ## Start Delay Use `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule. Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `StartDelay` field, and pass the instance to the `ExecuteWorkflow` call. ```go workflowOptions := client.StartWorkflowOptions{ // ... // Start the workflow in 12 hours StartDelay: time.Hours * 12, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` ## Temporal Cron Jobs > **⚠️ Caution:** > Cron support is not recommended > > We recommend using [Schedules](/schedule) instead of Cron Jobs. > Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules. > A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made. Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set the `CronSchedule` field, and pass the instance to the `ExecuteWorkflow` call. - Type: `string` - Default: None ```go workflowOptions := client.StartWorkflowOptions{ CronSchedule: "15 8 * * *", // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` Temporal Workflow Schedule Cron strings follow this format: ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) │ │ │ │ │ * * * * * ``` --- # Selectors - Go SDK Source: https://docs.temporal.io/develop/go/workflows/selectors > Use Temporal’s Go SDK Selectors with Futures, Timers, and Channels. Ensure deterministic Workflow execution and handle multiple parallel tasks efficiently. This page shows how to do the following: - [Use Selectors with Futures](#use-selectors-with-futures) - [Use Selectors with Timers](#use-selectors-with-timers) - [Use Selectors with Channels](#use-selectors-with-channels) In Go, the `select` statement lets a goroutine wait on multiple communication operations. A `select` **blocks until one of its cases can run**, then it executes that case. It chooses one at random if multiple are ready. However, a normal Go select statement can not be used inside of Workflows directly because of the random nature. Temporal's Go SDK `Selector`s are similar and act as a replacement. They can block on sending and receiving from Channels but as a bonus can listen on Future deferred work. Usage of Selectors to defer and process work (in place of Go's `select`) are necessary in order to ensure deterministic Workflow code execution (though using `select` in Activity code is fine). ## Full API example The API is sufficiently different from `select` that it bears documenting: ```go func SampleWorkflow(ctx workflow.Context) error { // standard Workflow setup code omitted... // API Example: declare a new selector selector := workflow.NewSelector(ctx) // API Example: defer code execution until the Future that represents Activity result is ready work := workflow.ExecuteActivity(ctx, ExampleActivity) selector.AddFuture(work, func(f workflow.Future) { // deferred code omitted... }) // more parallel timers and activities initiated... // API Example: receive information from a Channel var signalVal string channel := workflow.GetSignalChannel(ctx, channelName) selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) { // matching on the channel doesn't consume the message. // So it has to be explicitly consumed here c.Receive(ctx, &signalVal) // do something with received information }) // API Example: block until the next Future is ready to run // important! none of the deferred code runs until you call selector.Select selector.Select(ctx) // Todo: document selector.HasPending } ``` ## Use Selectors with Futures You usually add `Futures` after `Activities`: ```go // API Example: defer code execution until after an activity is done work := workflow.ExecuteActivity(ctx, ExampleActivity) selector.AddFuture(work, func(f workflow.Future) { // deferred code omitted... }) ``` Calling `ExecuteActivity` by itself doesn't result in actual Activity execution. It results in a command that gets batched with others when there's a yield point in the Workflow definition. Your `Futures` won't run until you call `selector.Select(ctx)`. So no calls to `ExecuteActivity` should result in commands being sent to the server until `selector.Select` is called. A call to `selector.Select(ctx)` doesn't actually execute a `Future`, but it _requests_ execution for a `Future`. So this is a point where the SDK will communicate the command to execute the Activity to the server. ```go // API Example: blocking conditionally if somecondition != nil { selector.Select(ctx) } // API Example: popping off all remaining Futures for i := 0; i < len(someArray); i++ { selector.Select(ctx) // this will wait for one branch // you can interrupt execution here } ``` A Future matches only once per Selector instance even if Select is called multiple times. If multiple items are available, the order of matching is not defined. ## Use Selectors with Timers An important use case of futures is setting up a race between a timer and a pending activity, effectively adding a "soft" timeout that doesn't result in any errors or retries of that activity. For example, [the Timer sample](https://github.com/temporalio/samples-go/tree/main/timer) shows how you can write a long running order processing operation where: - if processing takes too long, we send out a notification email to user about the delay, but we won't cancel the operation - if the operation finishes before the timer fires, then we want to cancel the timer. ```go var processingDone bool f := workflow.ExecuteActivity(ctx, OrderProcessingActivity) selector.AddFuture(f, func(f workflow.Future) { processingDone = true // cancel timerFuture cancelHandler() }) // use timer future to send notification email if processing takes too long timerFuture := workflow.NewTimer(childCtx, processingTimeThreshold) selector.AddFuture(timerFuture, func(f workflow.Future) { if !processingDone { // processing is not done yet when timer fires, send notification email _ = workflow.ExecuteActivity(ctx, SendEmailActivity).Get(ctx, nil) } }) // wait the timer or the order processing to finish selector.Select(ctx) ``` We create timers with the `workflow.NewTimer` API. ## Use Selectors with Channels `selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) {})` is the primary mechanism which receives messages from `Channels`. ```go // API Example: receive information from a Channel var signalVal string channel := workflow.GetSignalChannel(ctx, channelName) selector.AddReceive(channel, func(c workflow.ReceiveChannel, more bool) { c.Receive(ctx, &signalVal) // do something with received information }) ``` Merely matching on the channel doesn't consume the message; it has to be explicitly consumed with a `c.Receive(ctx, &signalVal)` call. ## Query Selector state You can use the `selector.HasPending` API to ensure that signals are not lost when a Workflow is closed (for example, by `ContinueAsNew`). ## Learn more Usage of Selectors is best learned by example: - Setting up a race condition between an Activity and a Timer, and conditionally execute ([Timer example](https://github.com/temporalio/samples-go/blob/main/timer/workflow.go)) - Receiving information in a Channel ([Mutex example](https://github.com/temporalio/samples-go/blob/main/mutex/mutex_workflow.go)) - Looping through a list of work and scheduling them all in parallel ([DSL example](https://github.com/temporalio/samples-go/blob/main/dsl/workflow.go)) - Executing activities in parallel, pick the first result, cancel remainder ([Pick First example](https://github.com/temporalio/samples-go/blob/main/pickfirst/pickfirst_workflow.go)) --- # Side Effects - Go SDK Source: https://docs.temporal.io/develop/go/workflows/side-effects > Side Effects in Workflows execute non-deterministic code, storing results in the Workflow Event History to maintain determinism. Use Go's SideEffect function for integration. Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow. This is done by storing the non-deterministic results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history). A Side Effect does not re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History. Side Effects should not fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task. An Activity or a Local Activity may also be used instead of a Side effect, as its result is also persisted in Workflow Execution History. > **📝 Note:** > > You shouldn't modify the Workflow state inside a Side Effect function, because it is not reexecuted during Replay. Side Effect function should be used to return a value. > Use the [`SideEffect`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SideEffect) function from the `go.temporal.io/sdk/workflow` package to execute a [Side Effect](/workflow-execution/event#side-effect) directly in your Workflow. Pass it an instance of `context.Context` and the function to execute. The `SideEffect` API returns a Future, an instance of [`converter.EncodedValue`](https://pkg.go.dev/go.temporal.io/sdk/workflow#SideEffect). Use the `Get` method on the Future to retrieve the result of the Side Effect. **Correct implementation** The following example demonstrates the correct way to use `SideEffect`: ```go encodedRandom := workflow.SideEffect(ctx, func(ctx workflow.Context) interface{} { return rand.Intn(100) }) var random int encodedRandom.Get(&random) // ... } ``` **Incorrect implementation** The following example demonstrates how NOT to use `SideEffect`: ```go // Warning: This is an incorrect example. // This code is non-deterministic. var random int workflow.SideEffect(func(ctx workflow.Context) interface{} { random = rand.Intn(100) return nil }) // random will always be 0 in replay, so this code is non-deterministic. ``` On replay the provided function is not executed, the random number will always be 0, and the Workflow Execution could take a different path, breaking determinism. ## Mutable Side Effects Mutable Side Effects execute the provided function once, and then it looks up the History of the value with the given Workflow ID. - If there is no existing value, then it records the function result as a value with the given Workflow Id on the History. - If there is an existing value, then it compares whether the existing value from the History has changed from the new function results, by calling the equals function. - If the values are equal, then it returns the value without recording a new Marker Event - If the values aren't equal, then it records the new value with the same ID on the History. > **📝 Note:** > > During a Workflow Execution, every new Side Effect call results in a new Marker recorded on the Event History; whereas Mutable Side Effects only records a new Marker on the Event History if the value for the Side Effect ID changes or is set the first time. > > During a Replay, Mutable Side Effects will not execute the function again. Instead, it returns the exact same value that was returned during the Workflow Execution. > To use [`MutableSideEffect()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#MutableSideEffect) in Go, provide a unique name within the scope of the workflow. ```go if err := workflow.MutableSideEffect(ctx, "configureNumber", get, eq).Get(&number); err != nil { panic("can't decode number:" + err.Error()) } ``` --- # Workflow Timeouts - Go SDK Source: https://docs.temporal.io/develop/go/workflows/timeouts > Optimize Workflow Execution with Temporal Go SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts). Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient. Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes. If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer. - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task. Create an instance of [`StartWorkflowOptions`](https://pkg.go.dev/go.temporal.io/sdk/client#StartWorkflowOptions) from the `go.temporal.io/sdk/client` package, set a timeout, and pass the instance to the `ExecuteWorkflow` call. Available timeouts are: - `WorkflowExecutionTimeout` - `WorkflowRunTimeout` - `WorkflowTaskTimeout` ```go workflowOptions := client.StartWorkflowOptions{ // ... // Set Workflow Timeout duration WorkflowExecutionTimeout: 24 * 365 * 10 * time.Hour, // WorkflowRunTimeout: 24 * 365 * 10 * time.Hour, // WorkflowTaskTimeout: 10 * time.Second, // ... } workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` ## Workflow Retry Policy A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. Create an instance of a [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) from the `go.temporal.io/sdk/temporal` package and provide it as the value to the `RetryPolicy` field of the instance of `StartWorkflowOptions`. - Type: [`RetryPolicy`](https://pkg.go.dev/go.temporal.io/sdk/temporal#RetryPolicy) - Default: None ```go retrypolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, } workflowOptions := client.StartWorkflowOptions{ RetryPolicy: retrypolicy, // ... } workflowRun, err := temporalClient.ExecuteWorkflow(context.Background(), workflowOptions, YourWorkflowDefinition) if err != nil { // ... } ``` --- # Timers - Go SDK Source: https://docs.temporal.io/develop/go/workflows/timers > Set Durable Timers in a Workflow using the sleep() or NewTimer() functions in Go with Temporal. Timers persist through Worker and Temporal Service downtime. A Workflow can set a Durable Timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`. A Workflow can sleep for days, months, or even years. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To set a Timer in Go, use the [`NewTimer()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#NewTimer) function and pass the duration you want to wait before continuing. ```go timer := workflow.NewTimer(timerCtx, duration) ``` To set a sleep duration in Go, use the [`sleep()`](https://pkg.go.dev/go.temporal.io/sdk/workflow#Sleep) function and pass the duration you want to wait before continuing. A zero or negative sleep duration causes the function to return immediately. ```go sleep = workflow.Sleep(ctx, 10*time.Second) ``` For more information, see the [Timer](https://github.com/temporalio/samples-go/tree/main/timer) example in the [Go Samples repository](https://github.com/temporalio/samples-go). --- # Versioning - Go SDK Source: https://docs.temporal.io/develop/go/workflows/versioning > Temporal's Go SDK ensures Workflow determinism through Patching APIs and Worker Versioning. Update Workflow code without causing non-deterministic issues, understand versioning best practices, and use dynamic configuration parameters for seamless updating of long-running Workflows. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. > **🚨 Danger:** > Support for the experimental Worker Versioning method before 2025 will be removed from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/go/worker-versioning-legacy) docs if needed. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Patching with GetVersion A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Consider the following Workflow Definition: ```go func YourWorkflow(ctx workflow.Context, data string) (string, error) { ao := workflow.ActivityOptions{ ScheduleToStartTimeout: time.Minute, StartToCloseTimeout: time.Minute, } ctx = workflow.WithActivityOptions(ctx, ao) var result1 string err := workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1) if err != nil { return "", err } var result2 string err = workflow.ExecuteActivity(ctx, ActivityB, result1).Get(ctx, &result2) return result2, err } ``` Suppose you replaced `ActivityA` with `ActivityC` and deployed the updated code. If an existing Workflow Execution was started by the original version of the Workflow code, where `ActivityA` was run, and then resumed running on a new Worker where it was replaced with `ActivityC`, the server side Event History would be out of sync. This would cause the Workflow to fail with a nondeterminism error. To resolve this, you can use `workflow.GetVersion()` to patch to your Workflow: ```go var err error v := workflow.GetVersion(ctx, "Step1", workflow.DefaultVersion, 1) if v == workflow.DefaultVersion { err = workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1) } else { err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1) } if err != nil { return "", err } var result2 string err = workflow.ExecuteActivity(ctx, ActivityB, result1).Get(ctx, &result2) return result2, err ``` When `workflow.GetVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `GetVersion` for this change Id — `Step 1` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example. If you make an additional change, such as replacing ActivityC with ActivityD, you need to add some additional code: ```go v := workflow.GetVersion(ctx, "Step1", workflow.DefaultVersion, 2) if v == workflow.DefaultVersion { err = workflow.ExecuteActivity(ctx, ActivityA, data).Get(ctx, &result1) } else if v == 1 { err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1) } else { err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) } ``` Note that we changed `maxSupported` from 1 to 2. A Workflow that has already passed this `GetVersion()` call before it was introduced returns `DefaultVersion`. A Workflow that was run with `maxSupported` set to 1 returns 1. New Workflows return 2. After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version: ```go v := workflow.GetVersion(ctx, "Step1", 1, 2) if v == 1 { err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1) } else { err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) } ``` You'll note that `minSupported` has changed from `DefaultVersion` to `1`. If an older version of the Workflow Execution history is replayed on this code, it fails because the minimum expected version is 1. After all the Workflow Executions for version 1 have left retention, you can remove version 1 so that your code looks like the following: ```go _ := workflow.GetVersion(ctx, "Step1", 2, 2) err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) ``` Note that we have preserved the call to `GetVersion()`. There are two reasons to preserve this call: 1. This ensures that if there is a Workflow Execution still running for an older version, it will fail here and not proceed. 2. If you need to make additional changes for `Step1`, such as changing ActivityD to ActivityE, you only need to update `maxVersion` from 2 to 3 and branch from there. You need to preserve only the first call to `GetVersion()` for each `changeID`. All subsequent calls to `GetVersion()` with the same change Id are safe to remove. If necessary, you can remove the first `GetVersion()` call, but you need to ensure the following: - All executions with an older version have left retention. - You can no longer use `Step1` for the changeId. If you need to make changes to that same part in the future, such as change from ActivityD to ActivityE, you would need to use a different changeId like `Step1-fix2`, and start minVersion from DefaultVersion again. The code would look like the following: ```go v := workflow.GetVersion(ctx, "Step1-fix2", workflow.DefaultVersion, 1) if v == workflow.DefaultVersion { err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) } else { err = workflow.ExecuteActivity(ctx, ActivityE, data).Get(ctx, &result1) } ``` You can add multiple calls to `GetVersion` in a single Workflow. This can become challenging to manage if you have many long-running Workflows, as you will wind up with many code branches over time. To clean these up, you can gradually deprecate older Workflow versions. ### Deprecating old Workflow versions You can safely remove support for older Workflow versions once you are certain that there are no longer any open Workflow Executions based on that version. You can use the following [List Filter](/list-filter) syntax for this (the 1 near the end of the last line represents the version number): ``` WorkflowType = "PizzaWorkflow" AND ExecutionStatus = "Running" AND TemporalChangeVersion="ChangedNotificationActivityType-1" ``` Since Workflow Executions that were started before `GetVersion` was added to the code won't have the associated Marker in their Event History, you'll need to use a different query to determine if any of those are still running: ``` WorkflowType = "PizzaWorkflow" AND ExecutionStatus = "Running" AND TemporalChangeVersion IS NULL ``` If you have found that there are no longer any open executions for the first two versions of the Workflow, for example, then you could remove support for them by changing the code as shown below: ```go version := GetVersion(ctx, "ChangedNotificationActivityType", 2, 3) if version == 2 { err = workflow.ExecuteActivity(ctx, SendTextMessage).Get(ctx, nil) } else { err = workflow.ExecuteActivity(ctx, SendTweet).Get(ctx, nil) } ``` Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`: ```go func PizzaWorkflow(ctx workflow.Context, order PizzaOrder) (OrderConfirmation, error) { // this function contains the original code } func PizzaWorkflowV2(ctx workflow.Context, order PizzaOrder) (OrderConfirmation, error) { // this function contains the updated code } ``` You can use any name you like for the new function, so long as the first character remains uppercase (this is a requirement for any Workflow Definition, since it must use an exported function). Using some type of version identifier, such as V2 in this example, will make it easier to identify the change. You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types: ```go w.RegisterWorkflow(pizza.PizzaWorkflow) w.RegisterWorkflow(pizza.PizzaWorkflowV2) ``` The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ## Runtime checking The Temporal Go SDK performs a runtime check to help prevent obvious incompatible changes. Adding, removing, or reordering any of these methods without Versioning triggers the runtime check and results in a nondeterminism error: - `workflow.ExecuteActivity()` - `workflow.ExecuteChildWorkflow()` - `workflow.NewTimer()` - `workflow.RequestCancelWorkflow()` - `workflow.SideEffect()` - `workflow.SignalExternalWorkflow()` - `workflow.Sleep()` The runtime check does not perform a thorough check. For example, it does not check on the Activity's input arguments or the Timer duration. Each Temporal SDK implements these sanity checks differently, and they are not a complete check for non-deterministic changes. Instead, you should incorporate [Replay Testing](/develop/go/best-practices/testing-suite#replay) when making revisions. --- # Workflow Streams - Go SDK Source: https://docs.temporal.io/develop/go/workflows/workflow-streams > Stream events from a Workflow to subscribers using the Temporal Go SDK Workflow Streams contrib module. > **Public Preview** [Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream, and keeping a stream running across long-lived Workflows. ## Enable streaming on a Workflow The library ships as `go.temporal.io/sdk/contrib/workflowstreams`. Enable streaming by constructing a `WorkflowStream` once at the top of your Workflow function, before any blocking call. Construction must happen there because the stream's handlers have to be registered before the first publish Signal arrives. Doing it after a blocking call would miss any publishes that arrived before the run body resumed. ```go import ( "go.temporal.io/sdk/contrib/workflowstreams" "go.temporal.io/sdk/workflow" ) type OrderInput struct { OrderID string StreamState *workflowstreams.WorkflowStreamState } func OrderWorkflow(ctx workflow.Context, input OrderInput) error { stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState) if err != nil { return err } // ... rest of the workflow return nil } ``` `NewWorkflowStream` creates the in-memory event log and registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. The `priorState` argument is `nil` on a fresh start and a `*WorkflowStreamState` after a Continue-As-New rollover (see [Stream from long-running Workflows](#continue-as-new)). Construct exactly one `WorkflowStream` per Workflow. The constructor re-registers the handlers unconditionally on every call, so constructing more than one on the same Workflow registers duplicate handlers. The Go SDK doesn't expose an inspection API for existing handlers, so the library can't raise an exception on a duplicate the way the Python SDK does. ## Publish from a Workflow Bind a [topic name](/workflow-streams#topics) with `stream.Topic(name)`, then call `Publish()` on the returned `*WorkflowTopicHandle` to append events. Repeated calls with the same name return the same handle. ```go type StatusEvent struct { State string Progress int Detail string } func OrderWorkflow(ctx workflow.Context, input OrderInput) error { stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState) if err != nil { return err } status := stream.Topic("status") if err := status.Publish(StatusEvent{State: "validating", Detail: "checking inventory"}); err != nil { return err } if err := workflow.ExecuteActivity(ctx, ValidateOrder, input.OrderID).Get(ctx, nil); err != nil { return err } if err := status.Publish(StatusEvent{State: "charging", Progress: 33, Detail: "authorizing payment"}); err != nil { return err } if err := workflow.ExecuteActivity(ctx, ChargePayment, input.OrderID).Get(ctx, nil); err != nil { return err } if err := status.Publish(StatusEvent{State: "shipping", Progress: 66, Detail: "dispatching to warehouse"}); err != nil { return err } if err := workflow.ExecuteActivity(ctx, DispatchOrder, input.OrderID).Get(ctx, nil); err != nil { return err } return status.Publish(StatusEvent{State: "completed", Progress: 100}) } ``` `Publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction. Unlike the Python and TypeScript SDKs, Go topics carry no per-topic type binding. A topic handle is bound only to a name; published values are `any` and subscribers decode each item from its raw payload (see [Subscribe](#subscribe)). To customize per-item serialization, pass `workflowstreams.WithPayloadConverters(...)` to `NewWorkflowStream`, and use the matching converters on the subscriber side. ## Publish from a client Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `Client`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities. Construct one with: ```go workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{}) ``` Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and `defer client.Close(ctx)` to flush on scope exit. When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream; it processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered). ```go import ( "context" "time" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/workflowstreams" ) func PublishStatus(ctx context.Context, temporalClient client.Client, workflowID string) error { streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{ BatchInterval: 200 * time.Millisecond, }) defer streamClient.Close(ctx) status := streamClient.Topic("status") status.Publish(StatusEvent{State: "started"}, false) // ... // Buffer is flushed automatically on Close. return nil } ``` Inside an Activity scheduled by a Workflow, `workflowstreams.NewClientFromActivity()` infers the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input: ```go import ( "context" "time" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/contrib/workflowstreams" ) type Delta struct { Text string } func StreamDeltas(ctx context.Context, orderID string) error { streamClient, err := workflowstreams.NewClientFromActivity(ctx, workflowstreams.Options{}) if err != nil { return err } defer streamClient.Close(ctx) deltas := streamClient.Topic("delta") for delta := range generateDeltas(orderID) { deltas.Publish(delta, false) activity.RecordHeartbeat(ctx) } // Buffer is flushed automatically on Close. return nil } ``` For a [standalone Activity](/develop/go/activities) (one started directly via the Client rather than from a Workflow), there is no parent Workflow context to infer, so `NewClientFromActivity()` returns an error. Fall back to the general pattern with `activity.GetClient(ctx)` and the target Workflow Id threaded through the Activity's input. Two operations give the application explicit control over when batches ship: the `forceFlush` argument on a publish for latency, and `client.Flush(ctx)` for confirmation that prior publications have landed. Pass `true` as the `forceFlush` argument on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the client is open (between construction and `Close`). The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers: ```go deltas.Publish(delta, true) ``` Use it for latency-sensitive events: the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure. Use `client.Flush(ctx)` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. `Close` already flushes on its way out, so the explicit call is only for barriers in the middle: ```go streamClient := workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{}) defer streamClient.Close(ctx) deltas := streamClient.Topic("delta") for _, delta := range firstPhase() { deltas.Publish(delta, false) } if err := streamClient.Flush(ctx); err != nil { return err } checkpointID, err := recordPhaseOneComplete(ctx) // only safe once phase-one events are durable if err != nil { return err } for _, delta := range secondPhase(checkpointID) { deltas.Publish(delta, false) } ``` `Publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up. If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `Publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you. ## Subscribe [Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `workflowstreams.NewClient(temporalClient, workflowID, opts)` from any process that has a Temporal Client, or `NewClientFromActivity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below. Once you have a client, range over `client.Subscribe()`, the counterpart to `Publish()`. It returns an [`iter.Seq2`](https://pkg.go.dev/iter#Seq2) iterator that yields a `WorkflowStreamItem` and an error on each step. Each item's `Data` is the raw payload; decode it at the call site with a payload converter. ```go import ( "context" "fmt" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/workflowstreams" "go.temporal.io/sdk/converter" ) func WatchOrder(ctx context.Context, temporalClient client.Client, orderID string) error { stream := workflowstreams.NewClient(temporalClient, orderID, workflowstreams.Options{}) for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"status"}}) { if err != nil { return err } var evt StatusEvent if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil { return err } fmt.Printf("[%3d%%] %s: %s\n", evt.Progress, evt.State, evt.Detail) if evt.State == "completed" { break } } return nil } ``` `SubscribeOptions` controls the subscription: `Topics` filters by name (empty or nil means all topics), `FromOffset` resumes from a stored global offset (zero means the beginning), and `PollCooldown` sets the minimum interval between polls. The iterator handles re-polling, pagination when a poll response hits the ~1 MB cap, and Workflow-side log truncation transparently. A single-topic convenience method, `streamClient.Topic("status").Subscribe(ctx, fromOffset)`, is equivalent to passing one name in `Topics`. A subscriber doesn't need `Close()` because the background flusher only runs for publishers. ### Heterogeneous topics Every item arrives as a raw `*commonpb.Payload` in `item.Data`, so a single subscription naturally consumes multiple topics whose payload types differ. Pass the topic names in `SubscribeOptions.Topics` (or leave it empty for every topic on the stream), dispatch on `item.Topic`, and decode into the matching type: ```go for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"status", "progress"}}) { if err != nil { return err } switch item.Topic { case "status": var evt StatusEvent if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil { return err } fmt.Printf("[status] %s: %s\n", evt.State, evt.Detail) case "progress": var evt ProgressEvent if err := converter.GetDefaultDataConverter().FromPayload(item.Data, &evt); err != nil { return err } fmt.Printf("[progress] %s\n", evt.Message) } } ``` A single iterator over multiple topics also avoids the cancellation race that two concurrent subscribers would create. Because `item.Data` is the raw payload, it's also the right shape when you want to forward the bytes through to another system without decoding them. ### Closing the stream A subscriber's `for ... range` doesn't know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces: 1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `WatchOrder` example above, `StatusEvent{State: "completed"}` is the minimal form, and the consumer's `if evt.State == "completed" { break }` is the matching half. Each subscription decides what its own end-of-stream marker is. 2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns is surfaced to the iterator and consumed silently, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it. There are two ways to provide that overlap. - [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits: ```go // at the end of the workflow function if err := status.Publish(StatusEvent{State: "completed", Progress: 100}); err != nil { return err } if err := workflow.Sleep(ctx, 30*time.Second); err != nil { return err } return nil ``` - [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives: ```go func ChatWorkflow(ctx workflow.Context, input ChatInput) (string, error) { stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState) if err != nil { return "", err } subscriberDone := false ackCh := workflow.GetSignalChannel(ctx, "subscriber-acknowledged-terminator") workflow.Go(ctx, func(ctx workflow.Context) { ackCh.Receive(ctx, nil) subscriberDone = true }) // ... do work and publish events ... // Returns true if the ack arrived, false on timeout. Either way, fall through. _, _ = workflow.AwaitWithTimeout(ctx, 30*time.Second, func() bool { return subscriberDone }) return result, nil } ``` The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below. You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). `Subscribe()` ends cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `temporalClient.DescribeWorkflowExecution(ctx, workflowID, "")` after the loop returns to inspect the Workflow's status. ## Stream from long-running Workflows Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers. To keep a stream running across them without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add a `*WorkflowStreamState` field to your Workflow input, pass it to `NewWorkflowStream`, and return `stream.NewContinueAsNewError(ctx, wfn, buildArgs)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, then returns a Continue-As-New error built from the args produced by `buildArgs(postDrainState)`: ```go type WorkflowInput struct { ItemsProcessed int StreamState *workflowstreams.WorkflowStreamState } func LongRunningWorkflow(ctx workflow.Context, input WorkflowInput) error { stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState) if err != nil { return err } itemsProcessed := input.ItemsProcessed for { if err := doOneIteration(ctx, stream); err != nil { return err } itemsProcessed++ if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { return stream.NewContinueAsNewError(ctx, LongRunningWorkflow, func(state *workflowstreams.WorkflowStreamState) []any { return []any{WorkflowInput{ ItemsProcessed: itemsProcessed, StreamState: state, }} }) } } } ``` The `*WorkflowStreamState` field is `nil` on a fresh start and a populated snapshot after a rollover. The `buildArgs` callback receives the post-detach `*WorkflowStreamState` as its only argument, so the snapshot is guaranteed to happen *after* pollers detach. To pass other Continue-As-New parameters such as a different task queue, or to use a custom publisher TTL, use the explicit recipe instead. Drain the pollers, wait for handlers to finish, snapshot the state with your chosen TTL, then build the Continue-As-New error yourself (set options such as the task queue on the context first): ```go stream.DetachPollers() _ = workflow.Await(ctx, func() bool { return workflow.AllHandlersFinished(ctx) }) state, err := stream.GetState(30 * time.Minute) // custom publisher TTL if err != nil { return err } ctx = workflow.WithWorkflowTaskQueue(ctx, "other-tq") return workflow.NewContinueAsNewError(ctx, LongRunningWorkflow, WorkflowInput{ ItemsProcessed: itemsProcessed, StreamState: state, }) ``` The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `stream.Truncate(upToOffset)` to keep the carried log itself small. ## Deduplication window See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams. There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting: - **Publisher TTL.** At each Continue-As-New, deduplicate entries whose last-seen time is older than this are dropped. The last-seen time is updated on each *successful* publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. `stream.NewContinueAsNewError(...)` snapshots with a 15-minute default; to tune it, use the explicit recipe above and pass your value to `GetState(publisherTTL)`. - **`MaxRetryDuration`.** A `Client` retries a failed batch for up to this long (default 10 minutes). If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `FlushTimeoutError` is raised. ```go workflowstreams.NewClient(temporalClient, workflowID, workflowstreams.Options{ MaxRetryDuration: 10 * time.Minute, }) ``` On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `FlushTimeoutError` is raised from inside the background flusher and terminates it. Until you call `client.Flush(ctx)` or `client.Close(ctx)`, subsequent publishes accumulate in the buffer with no flusher to ship them. `MaxRetryDuration` must be less than the workflow's publisher TTL to preserve exactly-once delivery. ## Best practices There are a few details to note if you're writing custom message handlers or testing the library's capabilities: - **Construct exactly one `WorkflowStream` per Workflow.** The constructor re-registers the publish Signal, poll Update, and offset Query handlers on every call, so a second construction registers duplicate handlers. Construct it once at the top of the Workflow function. - **`item.Data` is always the raw payload.** Decode it with a converter built from the same `PayloadConverters` used by the publisher. When publishers and subscribers both rely on the defaults, `converter.GetDefaultDataConverter()` matches on both sides. If you pass `WithPayloadConverters` on the Workflow side, build a matching `converter.NewCompositeDataConverter(...)` on the subscriber side. - **The codec chain runs once on the envelope.** Payload codecs (encryption, compression) configured on the Temporal client run on the Signal or Update envelope that carries each batch, never per item, so items are never double-encoded. `PayloadConverters` handle only per-item serialization. - **The client publish path isn't goroutine-safe.** The client buffer is mutated on the publish path and read from the background flusher. Don't call `Publish()` on the same `Client` from multiple goroutines without coordinating; route events to a single owner. ## Example: Stream LLM output The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it. If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees. **activity.go** ```go package main import ( "context" "strings" "time" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/contrib/workflowstreams" ) type TextDelta struct { Text string } type RetryEvent struct { Attempt int32 } func StreamCompletion(ctx context.Context, prompt string) (string, error) { attempt := activity.GetInfo(ctx).Attempt streamClient, err := workflowstreams.NewClientFromActivity(ctx, workflowstreams.Options{ BatchInterval: 200 * time.Millisecond, }) if err != nil { return "", err } defer streamClient.Close(ctx) deltas := streamClient.Topic("delta") retry := streamClient.Topic("retry") closeTopic := streamClient.Topic("close") // Tell consumers an earlier attempt's deltas are stale. if attempt > 1 { retry.Publish(RetryEvent{Attempt: attempt}, true) } var full []string first := true // generateDeltas wraps the model call and yields tokens as they arrive. // Disable provider-side retries; let Temporal own retry policy at the Activity layer. for token := range generateDeltas(prompt) { // forceFlush only on the first delta so the user sees something // immediately; subsequent deltas batch at the 200 ms interval. deltas.Publish(TextDelta{Text: token}, first) first = false full = append(full, token) } closeTopic.Publish(struct{}{}, false) return strings.Join(full, ""), nil } ``` **workflow.go** ```go package main import ( "time" "go.temporal.io/sdk/contrib/workflowstreams" "go.temporal.io/sdk/workflow" ) type ChatInput struct { Prompt string StreamState *workflowstreams.WorkflowStreamState } func ChatWorkflow(ctx workflow.Context, input ChatInput) (string, error) { stream, err := workflowstreams.NewWorkflowStream(ctx, input.StreamState) if err != nil { return "", err } subscriberDone := false ackCh := workflow.GetSignalChannel(ctx, "subscriber-acknowledged-terminator") workflow.Go(ctx, func(ctx workflow.Context) { ackCh.Receive(ctx, nil) subscriberDone = true }) ao := workflow.ActivityOptions{StartToCloseTimeout: 5 * time.Minute} ctx = workflow.WithActivityOptions(ctx, ao) var result string if err := workflow.ExecuteActivity(ctx, StreamCompletion, input.Prompt).Get(ctx, &result); err != nil { return "", err } // Wait for the subscriber to ack the terminal `close` event. The timeout // is a fallback for when no subscriber is attached; with the ack, the // typical case exits as soon as the subscriber confirms. _, _ = workflow.AwaitWithTimeout(ctx, 30*time.Second, func() bool { return subscriberDone }) return result, nil } ``` **consumer.go** ```go package main import ( "context" "strings" "go.temporal.io/sdk/client" "go.temporal.io/sdk/contrib/workflowstreams" "go.temporal.io/sdk/converter" ) func StreamChat(ctx context.Context, temporalClient client.Client, chatID string) (string, error) { // Subscribe-only; no Close needed because the flusher only runs for publishers. stream := workflowstreams.NewClient(temporalClient, chatID, workflowstreams.Options{}) dc := converter.GetDefaultDataConverter() var output []string render := func() { // ... display the accumulated output (terminal redraw, UI update, etc.) } for item, err := range stream.Subscribe(ctx, workflowstreams.SubscribeOptions{Topics: []string{"delta", "retry", "close"}}) { if err != nil { return "", err } switch item.Topic { case "retry": // Earlier attempt's deltas are stale; drop what we've shown. output = output[:0] render() case "delta": var delta TextDelta if err := dc.FromPayload(item.Data, &delta); err != nil { return "", err } output = append(output, delta.Text) render() case "close": // Acknowledge so the Workflow can return without waiting on the fallback timeout. if err := temporalClient.SignalWorkflow(ctx, chatID, "", "subscriber-acknowledged-terminator", nil); err != nil { return "", err } return strings.Join(output, ""), nil } } return strings.Join(output, ""), nil } ``` A few choices in this shape are deliberate: - The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes only the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why. - The Activity publishes a `RETRY` event when `activity.GetInfo(ctx).Attempt > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)). - Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `AwaitWithTimeout` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative). - `forceFlush` is `true` only on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `BatchInterval`; per-delta `forceFlush` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off). ## See also - [Workflow Streams samples (samples-go)](https://github.com/temporalio/samples-go/tree/main/workflowstreams): runnable scenarios covering basic publish/subscribe, reconnecting subscribers, external publishers, bounded logs, and LLM streaming. - [`workflowstreams` API reference](https://pkg.go.dev/go.temporal.io/sdk/contrib/workflowstreams). - [Workflow message passing](/develop/go/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on. --- # Java SDK developer guide Source: https://docs.temporal.io/develop/java > Explore Temporal Java SDK feature guides to master developing Temporal Applications. Build Workflows, Activities, and Workers, connect to Temporal Services, set up a testing suite, handle failure detection, send messages, complete Activities asynchronously, implement Versioning, use Observability, debug applications, schedule Workflows ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Install and get started You can find detailed installation instructions for the Java SDK in the [Quickstart](/develop/java/set-up-your-local-java). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Develop a Workflow](/develop/java/workflows/basics) - [Develop an Activity](/develop/java/activities/basics) - [Start an Activity execution](/develop/java/activities/execution) - [Run Worker processes](/develop/java/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/java/workflows) - [Workflow basics](/develop/java/workflows/basics) - [Child Workflows](/develop/java/workflows/child-workflows) - [Continue-As-New](/develop/java/workflows/continue-as-new) - [Message passing](/develop/java/workflows/message-passing) - [Cancellation](/develop/java/workflows/cancellation) - [Timeouts](/develop/java/workflows/timeouts) - [Schedules](/develop/java/workflows/schedules) - [Timers](/develop/java/workflows/timers) - [Side effects](/develop/java/workflows/side-effects) - [Versioning](/develop/java/workflows/versioning) - [Workflow Streams](/develop/java/workflows/workflow-streams) ## [Activities](/develop/java/activities) - [Activity basics](/develop/java/activities/basics) - [Activity execution](/develop/java/activities/execution) - [Standalone Activities](/develop/java/activities/standalone-activities-quickstart) - [Timeouts](/develop/java/activities/timeouts) - [Asynchronous Activity Completion](/develop/java/activities/asynchronous-activity) - [Benign exceptions](/develop/java/activities/benign-exceptions) ## [Workers](/develop/java/workers) - [Worker processes](/develop/java/workers/run-worker-process) - [Observability](/develop/java/platform/observability) ## [Temporal Client](/develop/java/client) - [Temporal Client](/develop/java/client/temporal-client) - [Namespaces](/develop/java/client/namespaces) ## [Temporal Nexus](/develop/java/nexus) - [Quickstart](/develop/java/nexus/quickstart) - [Feature guide](/develop/java/nexus/feature-guide) - [Standalone Operations](/develop/java/nexus/standalone-operations) ## [Platform](/develop/java/platform) - [Observability](/develop/java/platform/observability) - [Enriching the UI](/develop/java/platform/enriching-ui) ## [Best practices](/develop/java/best-practices) - [Testing](/develop/java/best-practices/testing-suite) - [Debugging](/develop/java/best-practices/debugging) - [Converters and encryption](/develop/java/best-practices/data-handling) ## [Integrations](/develop/java/integrations) - [Parseable integration](https://github.com/parseablehq/temporal-plugin-java/blob/main/INTEGRATION.md) - [Spring AI integration](/develop/java/integrations/spring-ai) - [Spring Boot integration](/develop/java/integrations/spring-boot-integration) ## Temporal Java technical resources - [Java SDK Quickstart - Setup Guide](/develop/java/set-up-your-local-java) - [Java API Documentation](https://javadoc.io/doc/io.temporal/temporal-sdk) - [Java SDK Code Samples](https://github.com/temporalio/samples-java) - [Java SDK GitHub](https://github.com/temporalio/sdk-java) - [Temporal 101 in Java Free Course](https://learn.temporal.io/courses/temporal_101/java/) ## Get connected with the Temporal Java community - [Temporal Java Community Slack](https://temporalio.slack.com/archives/CTT84KXK9) - [Java SDK Forum](https://community.temporal.io/tag/java-sdk) --- # Activities - Java SDK Source: https://docs.temporal.io/develop/java/activities > This section explains how to implement Activities with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Activities - [Activity basics](/develop/java/activities/basics) - [Activity execution](/develop/java/activities/execution) - [Standalone Activities Quickstart](/develop/java/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/java/activities/standalone-activities) - [Timeouts](/develop/java/activities/timeouts) - [Asynchronous Activity Completion](/develop/java/activities/asynchronous-activity) - [Benign exceptions](/develop/java/activities/benign-exceptions) --- # Asynchronous Activity completion - Java SDK Source: https://docs.temporal.io/develop/java/activities/asynchronous-activity > Asynchronously complete an Activity in a Workflow with Temporal. Follow steps to provide identifying information, use Temporal Client, and set the complete() method. This page shows how to asynchronously complete an Activity [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To complete an Activity asynchronously, set the [`ActivityCompletionClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityCompletionClient.html) interface to the `complete()` method. ```java @Override public String composeGreeting(String greeting, String name) { // Get the activity execution context ActivityExecutionContext context = Activity.getExecutionContext(); // Set a correlation token that can be used to complete the activity asynchronously byte[] taskToken = context.getTaskToken(); /** * For the example we will use a {@link java.util.concurrent.ForkJoinPool} to execute our * activity. In real-life applications this could be any service. The composeGreetingAsync * method is the one that will actually complete workflow action execution. */ ForkJoinPool.commonPool().execute(() -> composeGreetingAsync(taskToken, greeting, name)); context.doNotCompleteOnReturn(); // Since we have set doNotCompleteOnReturn(), the workflow action method return value is // ignored. return "ignored"; } // Method that will complete action execution using the defined ActivityCompletionClient private void composeGreetingAsync(byte[] taskToken, String greeting, String name) { String result = greeting + " " + name + "!"; // Complete our workflow activity using ActivityCompletionClient completionClient.complete(taskToken, result); } } ``` Alternatively, set the [`doNotCompleteOnReturn()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityExecutionContext.html#doNotCompleteOnReturn()) method during an Activity Execution. ```java @Override public String composeGreeting(String greeting, String name) { // Get the activity execution context ActivityExecutionContext context = Activity.getExecutionContext(); // Set a correlation token that can be used to complete the activity asynchronously byte[] taskToken = context.getTaskToken(); /** * For the example we will use a {@link java.util.concurrent.ForkJoinPool} to execute our * activity. In real-life applications this could be any service. The composeGreetingAsync * method is the one that will actually complete workflow action execution. */ ForkJoinPool.commonPool().execute(() -> composeGreetingAsync(taskToken, greeting, name)); context.doNotCompleteOnReturn(); // Since we have set doNotCompleteOnReturn(), the workflow action method return value is // ignored. return "ignored"; } ``` When this method is called during an Activity Execution, the Activity Execution does not complete when its method returns. --- # Activity basics - Java SDK Source: https://docs.temporal.io/develop/java/activities/basics > This section explains how to implement Activities with the Java SDK ## Develop an Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). [Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/java/activities/standalone-activities-quickstart). An [Activity Definition](/activities) is a combination of the Temporal Java SDK [Activity](https://www.javadoc.io/static/io.temporal/temporal-sdk/0.19.0/io/temporal/activity/Activity.html) Class implementing a specially annotated interface. An Activity interface is annotated with `@ActivityInterface` and an Activity implementation implements this Activity interface. To handle Activity types that do not have an explicitly registered handler, you can directly implement a dynamic Activity. ```java @ActivityInterface public interface GreetingActivities { String composeGreeting(String greeting, String language); } ``` Each method defined in the Activity interface defines a separate Activity method. You can annotate each method in the Activity interface with the `@ActivityMethod` annotation, but this is completely optional. The following example uses the `@ActivityMethod` annotation for the method defined in the previous example. ```java @ActivityInterface public interface GreetingActivities { @ActivityMethod String composeGreeting(String greeting, String language); } ``` An Activity implementation is a Java class that implements an Activity annotated interface. ```java // Implementation for the GreetingActivities interface example from in the previous section static class GreetingActivitiesImpl implements GreetingActivities { @Override public String composeGreeting(String greeting, String name) { return greeting + " " + name + "!"; } } ``` ## Define Activity parameters There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. An Activity interface can have any number of parameters. All inputs should be serializable by the default Jackson JSON Payload Converter. When implementing Activities, be mindful of the amount of data that you transfer using the Activity invocation parameters or return values as these are recorded in the Workflow Execution Events History. Large Event Histories can adversely impact performance. You can create a custom object, and pass it to the Activity interface, as shown in the following example. ```java @ActivityInterface public interface YourActivities { String getCustomObject(CustomObj customobj); void sendCustomObject(CustomObj customobj, String abc); } ``` The `execute` method in the dynamic Activity interface implementation takes in `EncodedValues` that are inputs to the Activity Execution, as shown in the following example. ```java // Dynamic Activity implementation public static class DynamicActivityImpl implements DynamicActivity { @Override public Object execute(EncodedValues args) { String activityType = Activity.getExecutionContext().getInfo().getActivityType(); return activityType + ": " + args.get(0, String.class) + " " + args.get(1, String.class) + " from: " + args.get(2, String.class); } } ``` For more details, see [Dynamic Activity Reference](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/DynamicActivity.html). ## Define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). Activity return values must be serializable and deserializable by the provided [`DataConverter`](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.17.0/io/temporal/common/converter/DataConverter.html). The `execute` method for `DynamicActivity` can return type Object. Ensure that your Workflow or Client can handle an Object type return or is able to convert the Object type response. - [Data Converter](/dataconversion) - Java DataConverter reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html) ## Customize your Activity Type Each Activity has a Type, which may also be referred to as the Activity 'name'. This name appears in the Workflow Execution Event History in the Summary tab for each Activity Task. The name lets you identify Activity Types called during the Execution. Custom Activity Type names prevent name collisions across interfaces and Workflows. They offer descriptive Activity method names without concerns about re-using those names elsewhere in your project. They also support code management, especially in larger projects with many Activities. For example, you might use a prefix to group related activities together. Custom names also distinguish keys for gathering metrics without name conflicts. The following examples show how to set custom names for your Activity Type. ### Default behavior By default, an Activity Type is the method name with the first letter capitalized: ```java @ActivityInterface public interface GreetingActivities { String sendMessage(String input); @ActivityMethod String composeGreeting(String greeting, String language); } ``` - Method Name: `sendMessage` - Activity Type: `SendMessage` - Method Name: `composeGreeting` - Activity Type: `ComposeGreeting` ### Custom Prefix Using the `namePrefix` parameter in the `@ActivityInterface` annotation adds a prefix to each Activity Type name mentioned in the interface, unless the prefix is specifically overridden: ```java @ActivityInterface(namePrefix = "Messaging_") public interface GreetingActivities { String sendMessage(String input); @ActivityMethod String composeGreeting(String greeting, String language); } ``` - Method Name: `sendMessage` - Activity Type: `Messaging_SendMessage` - Method Name: `composeGreeting` - Activity Type: `Messaging_ComposeGreeting` The Activity Type is capitalized, even when using a prefix. ### Custom Name To override the default name and any inherited prefixes, use the `name` parameter in the `@ActivityMethod` annotation: ```java @ActivityInterface(namePrefix = "Messaging_") public interface GreetingActivities { String sendMessage(String input); @ActivityMethod String composeGreeting(String greeting, String language); @ActivityMethod(name = "farewell") String composeFarewell(String farewell, String language); } ``` Using the `name` parameter won't automatically capitalize the result: - Method Name: `sendMessage` - Activity Type: `Messaging_SendMessage` - Method Name: `composeGreeting` - Activity Type: `Messaging_ComposeGreeting` - Method Name: `composeFarewell` - Activity Type: `farewell` Be cautious with names that contain special characters, as these can be used as metric tags. Systems such as Prometheus may ignore metrics with tags using unsupported characters. --- # Benign exceptions - Java SDK Source: https://docs.temporal.io/develop/java/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the category to `ApplicationErrorCategory.BENIGN` using the [`ApplicationFailure`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ApplicationFailure.html) builder. Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```java import io.temporal.activity.ActivityInterface; import io.temporal.activity.ActivityMethod; import io.temporal.failure.ApplicationErrorCategory; import io.temporal.failure.ApplicationFailure; @ActivityInterface public interface MyActivities { @ActivityMethod String myActivity(); } public class MyActivitiesImpl implements MyActivities { @Override public String myActivity() { try { return callExternalService(); } catch (Exception e) { // Mark this error as benign since it's expected throw ApplicationFailure.newBuilder() .setMessage(e.getMessage()) .setType(e.getClass().getName()) .setCause(e) .setCategory(ApplicationErrorCategory.BENIGN) .build(); } } } ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Activity execution - Java SDK Source: https://docs.temporal.io/develop/java/activities/execution > Shows how to perform Activity execution with the Java SDK ## Start an Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History. A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity implementation code should be _idempotent_. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. Activities are remote procedure calls that must be invoked from within a Workflow using `ActivityStub`. Activities are not executable on their own. You cannot start an Activity Execution by itself. Note that before an Activity Execution is invoked: - Activity options (either [`setStartToCloseTimeout`](/encyclopedia/detecting-activity-failures#start-to-close-timeout) or [`ScheduleToCloseTimeout`](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) are required) must be set for the Activity. For details, see [How to set Activity timeouts](/develop/java/activities/timeouts). - The Activity must be registered with a Worker. See [Worker Program](/develop/java/workers/run-worker-process#run-a-dev-worker) - Activity code must be thread-safe. Activities should only be instantiated using stubs from within a Workflow. An `ActivityStub` returns a client-side stub that implements an Activity interface. You can invoke Activities using `Workflow.newActivityStub`(type-safe) or `Workflow.newUntypedActivityStub` (untyped). Calling a method on the Activity interface schedules the Activity invocation with the Temporal service, and generates an [`ActivityTaskScheduled` Event](/references/events#activitytaskscheduled). Activities can be invoked synchronously or asynchronously. ### Invoking Activities Synchronously In the following example, we use the type-safe `Workflow.newActivityStub` within the "FileProcessingWorkflow" Workflow implementation to create a client-side stub of the `FileProcessingActivities` class. We also define `ActivityOptions` and set `setStartToCloseTimeout` option to one hour. ```java public class FileProcessingWorkflowImpl implements FileProcessingWorkflow { private final FileProcessingActivities activities; public FileProcessingWorkflowImpl() { this.activities = Workflow.newActivityStub( FileProcessingActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofHours(1)) .build()); } @Override public void processFile(Arguments args) { String localName = null; String processedName = null; try { localName = activities.download(args.getSourceBucketName(), args.getSourceFilename()); processedName = activities.processFile(localName); activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName); } finally { if (localName != null) { activities.deleteLocalFile(localName); } if (processedName != null) { activities.deleteLocalFile(processedName); } } } // ... } ``` A Workflow can have multiple Activity stubs. Each Activity stub can have its own `ActivityOptions` defined. The following example shows a Workflow implementation with two typed Activity stubs. ```java public FileProcessingWorkflowImpl() { ActivityOptions options1 = ActivityOptions.newBuilder() .setTaskQueue("taskQueue1") .setStartToCloseTimeout(Duration.ofMinutes(10)) .build(); this.store1 = Workflow.newActivityStub(FileProcessingActivities.class, options1); ActivityOptions options2 = ActivityOptions.newBuilder() .setTaskQueue("taskQueue2") .setStartToCloseTimeout(Duration.ofMinutes(5)) .build(); this.store2 = Workflow.newActivityStub(FileProcessingActivities.class, options2); } ``` To invoke Activities inside Workflows without referencing the interface it implements, use an untyped Activity stub `Workflow.newUntypedActivityStub`. This is useful when the Activity type is not known at compile time, or to invoke Activities implemented in different programming languages. ```java // Workflow code ActivityOptions activityOptions = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(3)) .setTaskQueue("simple-queue-node") .build(); ActivityStub activity = Workflow.newUntypedActivityStub(activityOptions); activity.execute("ComposeGreeting", String.class, "Hello World", "Spanish"); ``` ### Invoking Activities Asynchronously Sometimes Workflows need to perform certain operations in parallel. The Temporal Java SDK provides the `Async` class which includes static methods used to invoke any Activity asynchronously. The calls return a result of type `Promise` which is similar to the Java `Future` and `CompletionStage`. When invoking Activities, use `Async.function` for Activities that return a result, and `Async.procedure` for Activities that return void. In the following asynchronous Activity invocation, the method reference is passed to `Async.function` followed by Activity arguments. ```java Promise localNamePromise = Async.function(activities::download, sourceBucket, sourceFile); ``` The following example shows how to call two Activity methods, "download" and "upload", in parallel on multiple files. ```java public void processFile(Arguments args) { List> localNamePromises = new ArrayList<>(); List processedNames = null; try { // Download all files in parallel. for (String sourceFilename : args.getSourceFilenames()) { Promise localName = Async.function(activities::download, args.getSourceBucketName(), sourceFilename); localNamePromises.add(localName); } List localNames = new ArrayList<>(); for (Promise localName : localNamePromises) { localNames.add(localName.get()); } processedNames = activities.processFiles(localNames); // Upload all results in parallel. List> uploadedList = new ArrayList<>(); for (String processedName : processedNames) { Promise uploaded = Async.procedure( activities::upload, args.getTargetBucketName(), args.getTargetFilename(), processedName); uploadedList.add(uploaded); } // Wait for all uploads to complete. Promise.allOf(uploadedList).get(); } finally { for (Promise localNamePromise : localNamePromises) { // Skip files that haven't completed downloading. if (localNamePromise.isCompleted()) { activities.deleteLocalFile(localNamePromise.get()); } } if (processedNames != null) { for (String processedName : processedNames) { activities.deleteLocalFile(processedName); } } } } ``` ### Activity Execution Context `ActivityExecutionContext` is a context object passed to each Activity implementation by default. You can access it in your Activity implementations via `Activity.getExecutionContext()`. It provides getters to access information about the Workflow that invoked the Activity. Note that the Activity context information is stored in a thread-local variable. Therefore, calls to `getExecutionContext()` succeed only within the thread that invoked the Activity function. Following is an example of using the `ActivityExecutionContext`: ```java public class FileProcessingActivitiesImpl implements FileProcessingActivities { @Override public String download(String bucketName, String remoteName, String localName) { ActivityExecutionContext ctx = Activity.getExecutionContext(); ActivityInfo info = ctx.getInfo(); log.info("namespace=" + info.getActivityNamespace()); log.info("workflowId=" + info.getWorkflowId()); log.info("runId=" + info.getRunId()); log.info("activityId=" + info.getActivityId()); log.info("activityTimeout=" + info.getStartToCloseTimeout(); return downloadFileFromS3(bucketName, remoteName, localDirectory + localName); } ... } ``` For details on getting the results of an Activity Execution, see [Activity Execution Result](#activity-execution-result). ## Set required Activity Timeouts Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. Set your Activity Timeout from the [`ActivityOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) class. Available timeouts are: - ScheduleToCloseTimeout() - ScheduleToStartTimeout() - StartToCloseTimeout() You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. The following uses `ActivityStub`. ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) // .setStartToCloseTimeout(Duration.ofSeconds(2) // .setScheduletoCloseTimeout(Duration.ofSeconds(20)) .build()); ``` The following uses `WorkflowImplementationOptions`. ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "GetCustomerGreeting", // Set Activity Execution timeout ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) // .setStartToCloseTimeout(Duration.ofSeconds(2)) // .setScheduleToStartTimeout(Duration.ofSeconds(5)) .build())) .build(); ``` > **📝 Note:** > > If you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. > ## Java ActivityOptions reference Use [`ActivityOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) to configure how to invoke an Activity Execution. You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. The following table lists all `ActivityOptions` that can be configured for an Activity invocation. | Option | Required | Type | | ------------------------------------------------------ | -------------------------------------------------- | ------------------------ | | [`setScheduleToCloseTimeout`](#scheduletoclosetimeout) | Yes (if `StartToCloseTimeout` is not specified) | Duration | | [`setScheduleToStartTimeout`](#scheduletostarttimeout) | No | Duration | | [`setStartToCloseTimeout`](#starttoclosetimeout) | Yes (if `ScheduleToCloseTimeout` is not specified) | Duration | | [`setHeartbeatTimeout`](#heartbeattimeout) | No | Duration | | [`setTaskQueue`](#taskqueue) | No | String | | [`setRetryOptions`](#retryoptions) | No | RetryOptions | | [`setCancellationType`](#setcancellationtype) | No | ActivityCancellationType | ### ScheduleToCloseTimeout To set a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout), use [`ActivityOptions.newBuilder.setScheduleToCloseTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). This or `StartToCloseTimeout` must be set. - Type: `Duration` - Default: Unlimited. Note that if `WorkflowRunTimeout` and/or `WorkflowExecutionTimeout` are defined in the Workflow, all Activity retries will stop when either or both of these timeouts are reached. You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. - With `ActivityStub` ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "GetCustomerGreeting", ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) .build())) .build(); ``` ### ScheduleToStartTimeout To set a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout), use [`ActivityOptions.newBuilder.setScheduleToStartTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). - Type: `Duration` - Default: Unlimited. This timeout is non-retryable. You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. - With `ActivityStub` ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setScheduleToStartTimeout(Duration.ofSeconds(5)) // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setScheduletoCloseTimeout(Duration.ofSeconds(20)) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "GetCustomerGreeting", ActivityOptions.newBuilder() .setScheduleToStartTimeout(Duration.ofSeconds(5)) .build())) .build(); ``` ### StartToCloseTimeout To set a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout), use [`ActivityOptions.newBuilder.setStartToCloseTimeout​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). This or `ScheduleToClose` must be set. - Type: `Duration` - Default: Defaults to [`ScheduleToCloseTimeout`](#scheduletoclosetimeout) value You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. - With `ActivityStub` ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(2)) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // Set Activity Execution timeout (single run) .setStartToCloseTimeout(Duration.ofSeconds(2)) .build())) .build(); ``` ### HeartbeatTimeout To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), use [`ActivityOptions.newBuilder.setHeartbeatTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). - Type: `Duration` - Default: None You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. - With `ActivityStub` ```java private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build())) .build(); ``` ### TaskQueue - Type: `String` - Default: Defaults to the Task Queue that the Workflow was started with. - With `ActivityStub` ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are required when // setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setTaskQueue("yourTaskQueue") .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setTaskQueue("yourTaskQueue") .build())) .build(); ``` See [Task Queue](/task-queue) ### RetryOptions To set a Retry Policy, known as the [Retry Options](/encyclopedia/retry-policies) in Java, use [`ActivityOptions.newBuilder.setRetryOptions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). - Type: `RetryOptions` - Default: Server-defined Activity Retry policy. - With `ActivityStub` ```java private final ActivityOptions options = ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setMaximumInterval(Duration.ofSeconds(10)) .build()) .build(); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setRetryOptions( RetryOptions.newBuilder() .setDoNotRetry(NullPointerException.class.getName()) .build()) .build())) .build(); ``` ### setCancellationType - Type: `ActivityCancellationType` - Default: `ActivityCancellationType.TRY_CANCEL` - With `ActivityStub` ```java private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder() .setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() .setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED) .build())) .build(); ``` ## Get the result of an Activity Execution The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable. Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available. To get the results of an asynchronously invoked Activity method, use the `Promise` `get` method to block until the Activity method result is available. Sometimes an Activity Execution lifecycle goes beyond a synchronous method invocation. For example, a request can be put in a queue and later a reply comes and is picked up by a different Worker process. The whole request-reply interaction can be modeled as a single Activity. To indicate that an Activity should not be completed upon its method return, call `ActivityExecutionContext.doNotCompleteOnReturn()` from the original Activity thread. Then later, when replies come, complete the Activity using the `ActivityCompletionClient`. To correlate Activity invocation with completion, use either a `TaskToken` or Workflow and Activity Ids. Following is an example of using `ActivityExecutionContext.doNotCompleteOnReturn()`: ```java public class FileProcessingActivitiesImpl implements FileProcessingActivities { public String download(String bucketName, String remoteName, String localName) { ActivityExecutionContext ctx = Activity.getExecutionContext(); // Used to correlate reply byte[] taskToken = ctx.getInfo().getTaskToken(); asyncDownloadFileFromS3(taskToken, bucketName, remoteName, localDirectory + localName); ctx.doNotCompleteOnReturn(); // Return value is ignored when doNotCompleteOnReturn was called. return "ignored"; } ... } ``` When the download is complete, the download service potentially can complete the Activity, or fail it from a different process, for example: ```java public void completeActivity(byte[] taskToken, R result) { completionClient.complete(taskToken, result); } public void failActivity(byte[] taskToken, Exception failure) { completionClient.completeExceptionally(taskToken, failure); } ``` --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/java/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal Java SDK. > **Public Preview** [Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client using `ActivityClient`. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/java/activities/standalone-activities-quickstart). > This page covers the following: - [Start a Standalone Activity without waiting for the result](#start-activity) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [Wait for the result of a Standalone Activity](#get-activity-result) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the > [standaloneactivities](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities) > sample. > ## Start a Standalone Activity without waiting for the result Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue your Activity job, without waiting for it to be executed by your Worker. Use [`ActivityClient.start()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html) to start a Standalone Activity and get a handle without waiting for the result: [StartActivity.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/StartActivity.java) ```java ActivityHandle handle = client.start( GreetingActivities.class, GreetingActivities::composeGreeting, options, "Hello", "World"); System.out.println("Started activity ID: " + ACTIVITY_ID); // Wait for the result later String result = handle.getResult(); System.out.println("Activity result: " + result); ``` With the Temporal Server and Worker running, open a new terminal in the `samples-java` directory and run: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.standaloneactivities.StartActivity ``` Or use the Temporal CLI: ```bash ./temporal activity start \ --type ComposeGreeting \ --activity-id standalone-activity-id \ --task-queue standalone-activity-task-queue \ --start-to-close-timeout 10s \ --input '"Hello"' \ --input '"World"' ``` ## Get a handle to an existing Standalone Activity Use `client.getHandle()` to create a typed handle to a previously started Standalone Activity: ```java ActivityHandle handle = client.getHandle("standalone-activity-id", null, String.class); ``` Pass `null` as the run ID to target the latest run of the given activity ID. You can then use the handle to wait for the result, describe, cancel, or terminate the Activity. ## Wait for the result of a Standalone Activity Under the hood, calling `client.execute()` is the same as calling `client.start()` to durably enqueue the Standalone Activity, and then calling `handle.getResult()` to block until the Activity completes and return the result: ```java String result = handle.getResult(); ``` To wait asynchronously without blocking the calling thread, use `handle.getResultAsync()`, which returns a `CompletableFuture`: ```java CompletableFuture future = handle.getResultAsync(); ``` Or use the Temporal CLI to wait for a result by Activity ID: ```bash ./temporal activity result --activity-id standalone-activity-id ``` ## List Standalone Activities Use [`client.listExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html) to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is a `Stream` that fetches pages from the server on demand as the stream is consumed. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. [ListActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/ListActivities.java) ```java client .listExecutions("TaskQueue = '" + TASK_QUEUE + "'") .forEach( info -> System.out.printf( "ActivityID: %s, Type: %s, Status: %s%n", info.getActivityId(), info.getActivityType(), info.getStatus())); ``` Run it: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.standaloneactivities.ListActivities ``` Or use the Temporal CLI: ```bash ./temporal activity list ``` The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow Visibility](/visibility). For example, `ActivityType = 'composeGreeting' AND Status = 'Running'`. ## Count Standalone Activities Use [`client.countExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html) to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) — not the number of queued tasks. It works the same way as counting Workflow Executions. [CountActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/CountActivities.java) ```java ActivityExecutionCount resp = client.countExecutions("TaskQueue = '" + TASK_QUEUE + "'"); System.out.println("Total activities: " + resp.getCount()); resp.getGroups() .forEach( group -> System.out.println("Group " + group.getGroupValues() + ": " + group.getCount())); ``` Run it: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.standaloneactivities.CountActivities ``` Or use the Temporal CLI: ```bash ./temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The Worker and Client code in the [Standalone Activities Quickstart](/develop/java/activities/standalone-activities-quickstart) use `ClientConfigProfile.load()`, so the same code works against Temporal Cloud — configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/java/activities/standalone-activities-quickstart). --- # Standalone Activities Java Quickstart Source: https://docs.temporal.io/develop/java/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal Java SDK without writing a Workflow. # Quickstart [Standalone Activities](/standalone-activity) are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client using `ActivityClient`. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/java/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **📝 Note:** > > This documentation uses source code from the > [standaloneactivities](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/standaloneactivities) > sample. > ## Get started with Standalone Activities Prerequisites: - **Java** 8+ - **Temporal Java SDK** (v1.35.0 or higher). See the [Java Quickstart](/develop/java/set-up-your-local-java) for install instructions. - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server will now be available for client connections on `localhost:7233`, and the Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-java](https://github.com/temporalio/samples-java) repository to follow along: ```bash git clone https://github.com/temporalio/samples-java.git cd samples-java ``` The sample consists of separate programs in the `standaloneactivities` package: ``` core/src/main/java/io/temporal/samples/standaloneactivities/ ├── GreetingActivities.java # Activity interface ├── GreetingActivitiesImpl.java # Activity implementation ├── StandaloneActivityWorker.java # Worker that processes activity tasks ├── ExecuteActivity.java # Starts an activity and waits for the result ├── StartActivity.java # Starts an activity without blocking ├── ListActivities.java # Lists activity executions └── CountActivities.java # Counts activity executions ``` ## Define your Activity An Activity in the Temporal Java SDK is an interface annotated with `@ActivityInterface`, with methods annotated with `@ActivityMethod`. The way you define a Standalone Activity is identical to how you define an Activity orchestrated by a Workflow. In fact, the same Activity can be executed both as a Standalone Activity and as a Workflow Activity. [GreetingActivities.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/GreetingActivities.java) [GreetingActivitiesImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/GreetingActivitiesImpl.java) ```java @ActivityInterface public interface GreetingActivities { @ActivityMethod String composeGreeting(String greeting, String name); } ``` ```java public class GreetingActivitiesImpl implements GreetingActivities { private static final Logger log = LoggerFactory.getLogger(GreetingActivitiesImpl.class); @Override public String composeGreeting(String greeting, String name) { log.info("Composing greeting..."); return greeting + ", " + name + "!"; } } ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities — you create a `WorkerFactory`, register the Activity implementation, and call `factory.start()`. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to run a Worker](/develop/java/workers/run-worker-process) for more details on Worker setup and configuration options. [StandaloneActivityWorker.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/StandaloneActivityWorker.java) Open a new terminal, navigate to the `samples-java` directory, and run the Worker. Leave this terminal running — the Worker needs to stay up to process activities. ```java ClientConfigProfile profile = ClientConfigProfile.load(); WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(profile.toWorkflowServiceStubsOptions()); WorkflowClient client = WorkflowClient.newInstance(service, profile.toWorkflowClientOptions()); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker(TASK_QUEUE); worker.registerActivitiesImplementations(new GreetingActivitiesImpl()); factory.start(); System.out.println("Worker running on task queue: " + TASK_QUEUE); ``` ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.standaloneactivities.StandaloneActivityWorker ``` ## Execute a Standalone Activity Use [`ActivityClient.execute()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/ActivityClient.html) to execute a Standalone Activity and block until it completes. Call this from your application code, not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to be executed on your Worker, and then returns the typed result. [ExecuteActivity.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/standaloneactivities/ExecuteActivity.java) The typed `execute()` API takes the Activity interface class and an unbound method reference. The SDK uses the method reference to infer the Activity type name and result type at runtime. You can also call Activities by string type name. `StartActivityOptions` requires `id`, `taskQueue`, and at least one of `startToCloseTimeout` or `scheduleToCloseTimeout`. To run it: 1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above). 2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above). 3. Open a new terminal, navigate to the `samples-java` directory, and run the Gradle command. Or use the Temporal CLI. ```java ActivityClient client = ActivityClient.newInstance( service, ActivityClientOptions.newBuilder().setNamespace(profile.getNamespace()).build()); StartActivityOptions options = StartActivityOptions.newBuilder() .setId(ACTIVITY_ID) .setTaskQueue(TASK_QUEUE) .setStartToCloseTimeout(Duration.ofSeconds(10)) .build(); String result = client.execute( GreetingActivities.class, GreetingActivities::composeGreeting, options, "Hello", "World"); System.out.println("Activity result: " + result); ``` ```java // Using a string type name String result = client.execute("ComposeGreeting", String.class, options, "Hello", "World"); ``` ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.standaloneactivities.ExecuteActivity ``` ```bash ./temporal activity execute \\ --type ComposeGreeting \\ --activity-id standalone-activity-id \\ --task-queue standalone-activity-task-queue \\ --start-to-close-timeout 10s \\ --input '"Hello"' \\ --input '"World"' ``` ## Run with Temporal Cloud All code samples on this page use [`ClientConfigProfile.load()`](https://www.javadoc.io/doc/io.temporal/temporal-envconfig/latest/io/temporal/envconfig/ClientConfigProfile.html) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/java/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/java/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/java/activities/basics)**: How to write and register Activities with the Java SDK. --- # Activity Timeouts - Java SDK Source: https://docs.temporal.io/develop/java/activities/timeouts > This section explains how to set Activity Timeouts with the Java SDK ## Activity timeouts Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution. The following timeouts are available in the Activity Options. - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. Set your Activity Timeout from the [`ActivityOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html) class. Available timeouts are: - ScheduleToCloseTimeout - StartToCloseTimeout - ScheduleToStartTimeout You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. The following uses `ActivityStub`. ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) // .setStartToCloseTimeout(Duration.ofSeconds(2) // .setScheduletoCloseTimeout(Duration.ofSeconds(20)) .build()); ``` The following uses `WorkflowImplementationOptions`. ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "GetCustomerGreeting", // Set Activity Execution timeout ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) // .setStartToCloseTimeout(Duration.ofSeconds(2)) // .setScheduleToStartTimeout(Duration.ofSeconds(5)) .build())) .build(); ``` > **📝 Note:** > > If you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. > ### Custom Activity Retry Policy A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To set a Retry Policy, known as the [Retry Options](/encyclopedia/retry-policies) in Java, use [`ActivityOptions.newBuilder.setRetryOptions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). - Type: `RetryOptions` - Default: Server-defined Activity Retry policy. - With `ActivityStub` ```java private final ActivityOptions options = ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setMaximumInterval(Duration.ofSeconds(10)) .build()) .build(); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setRetryOptions( RetryOptions.newBuilder() .setDoNotRetry(NullPointerException.class.getName()) .build()) .build())) .build(); ``` ## Activity next retry delay You may throw an [`ApplicationFailure`](/references/failures#application-failure) with the `NextRetryDelay` field set. This value will replace and override whatever the retry interval would be on the retry policy. For example, if in an activity, you want to base the interval on the number of attempts, you might do: ```java int attempt = Activity.getExecutionContext().getInfo().getAttempt(); throw ApplicationFailure.newFailureWithCauseAndDelay( "Something bad happened on attempt " + attempt, "my_failure_type", null, 3 * Duration.ofSeconds(attempt)); ``` ## Heartbeat an Activity An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. Heartbeats can contain a `details` field describing the Activity's current progress. If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service. To Heartbeat an Activity Execution in Java, use the `Activity.getExecutionContext().heartbeat()` Class method. ```java public class YourActivityDefinitionImpl implements YourActivityDefinition { @Override public String yourActivityMethod(YourActivityMethodParam param) { // ... Activity.getExecutionContext().heartbeat(details); // ... } // ... } ``` The method takes an optional argument, the `details` variable above that represents latest progress of the Activity Execution. This method can take a variety of types such as an exception object, custom object, or string. If the Activity Execution times out, the last Heartbeat `details` are included in the thrown `ActivityTimeoutException`, which can be caught by the calling Workflow. The Workflow can then use the `details` information to pass to the next Activity invocation if needed. In the case of Activity retries, the last Heartbeat's `details` are available and can be extracted from the last failed attempt by using `Activity.getExecutionContext().getHeartbeatDetails(Class detailsClass)` ### Heartbeat Timeout A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). To set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), use [`ActivityOptions.newBuilder.setHeartbeatTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/activity/ActivityOptions.Builder.html). - Type: `Duration` - Default: None You can set Activity Options using an `ActivityStub` within a Workflow implementation, or per-Activity using `WorkflowImplementationOptions` within a Worker. Note that if you define options per-Activity Type options with `WorkflowImplementationOptions.setActivityOptions()`, setting them again specifically with `ActivityStub` in a Workflow will override this setting. - With `ActivityStub` ```java private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build()); ``` - With `WorkflowImplementationOptions` ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "EmailCustomerGreeting", ActivityOptions.newBuilder() // note that either StartToCloseTimeout or ScheduleToCloseTimeout are // required when setting Activity options. .setStartToCloseTimeout(Duration.ofSeconds(5)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build())) .build(); ``` --- # Best practices - Java SDK Source: https://docs.temporal.io/develop/java/best-practices > This section explains how to implement best practices with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Best practices - [Error handling](/develop/java/best-practices/error-handling) - [Testing](/develop/java/best-practices/testing-suite) - [Debugging](/develop/java/best-practices/debugging) - [Converters and encryption](/develop/java/best-practices/data-handling) --- # Data handling - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/java/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/java/best-practices/data-handling/data-encryption) | | ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | | **Default** | JSON serialization | None (passthrough) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the Java SDK. Payload Converters serialize your application objects into a `Payload` and deserialize them back. A `Payload` is a binary form with metadata that Temporal uses to transport data. By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable. You only need a custom Payload Converter when your application uses types that aren't natively supported. ## Default supported types The default Data Converter supports converting multiple types including: - `null` - Byte arrays - Protobuf JSON: if a value is an instance of a Protobuf message, it is encoded with proto3 JSON - [Jackson JSON](https://docs.spring.io/spring-framework/reference/web/webmvc/mvc-controller/ann-methods/jackson.html) - Anything that can be converted to JSON ## Using custom Payload conversion Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. Implementing custom Payload conversion is optional. It is needed only if the [default Data Converter](/default-custom-data-converters#default-data-converter) does not support your custom values. To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options. The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this: ![Temporal converter architecture](/img/info/converter-architecture.png) Create a custom implementation of a [PayloadConverter](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/PayloadConverter.html) interface and use the `withPayloadConverterOverrides` method to implement the custom object conversion with `DefaultDataConverter`. `PayloadConverter` serializes and deserializes method parameters that need to be sent over the wire. You can create a custom implementation of `PayloadConverter` for custom formats, as shown in the following example: ```java /** Payload Converter specific to your custom object */ public class YourCustomPayloadConverter implements PayloadConverter { //... @Override public String getEncodingType() { return "json/plain"; // The encoding type determines which default conversion behavior to override. } @Override public Optional toData(Object value) throws DataConverterException { // Add your convert-to logic here. } @Override public T fromData(Payload content, Class valueClass, Type valueType) throws DataConverterException { // Add your convert-from logic here. } //... } ``` You can also use [specific implementation classes](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.18.1/io/temporal/common/converter/package-summary.html) provided in the Java SDK. For example, to create a custom `JacksonJsonPayloadConverter`, use the following: ```java //... private static JacksonJsonPayloadConverter yourCustomJacksonJsonPayloadConverter() { ObjectMapper objectMapper = new ObjectMapper(); // Add your custom logic here. return new JacksonJsonPayloadConverter(objectMapper); } //... ``` To set your custom Payload Converter, use it with [withPayloadConverterOverrides](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DefaultDataConverter.html#withPayloadConverterOverrides(io.temporal.common.converter.PayloadConverter...)) with a new instance of `DefaultDataConverter` in your `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions. The following example shows how to set a custom `YourCustomPayloadConverter` Payload Converter. ```java //... DefaultDataConverter ddc = DefaultDataConverter.newDefaultInstance() .withPayloadConverterOverrides(new YourCustomPayloadConverter()); WorkflowClientOptions workflowClientOptions = WorkflowClientOptions.newBuilder().setDataConverter(ddc).build(); //... ``` --- # Payload encryption - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Java SDK. Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. ## Custom Payload Codec in Java Create a custom implementation of [`PayloadCodec`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/payload/codec/PayloadCodec.html) and use it in [`CodecDataConverter`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/CodecDataConverter.html) to set a custom Data Converter. The Payload Codec does byte-to-byte conversion and must be set with a Data Converter. Define custom encryption/compression logic in your `encode` method and decryption/decompression logic in your `decode` method. The following example from the [Java encryption sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads/CryptCodec.java) shows how to implement encryption and decryption logic on your payloads in your `encode` and `decode` methods. ```java class YourCustomPayloadCodec implements PayloadCodec { static final ByteString METADATA_ENCODING = ByteString.copyFrom("binary/encrypted", StandardCharsets.UTF_8); private static final String CIPHER = "AES/GCM/NoPadding"; // Define constants that you can add to your encoded Payload to create a new Payload. static final String METADATA_ENCRYPTION_CIPHER_KEY = "encryption-cipher"; static final ByteString METADATA_ENCRYPTION_CIPHER = ByteString.copyFrom(CIPHER, StandardCharsets.UTF_8); static final String METADATA_ENCRYPTION_KEY_ID_KEY = "encryption-key-id"; private static final Charset UTF_8 = StandardCharsets.UTF_8; // See the linked sample for details on the methods called here. @NotNull @Override public List encode(@NotNull List payloads) { return payloads.stream().map(this::encodePayload).collect(Collectors.toList()); } @NotNull @Override public List decode(@NotNull List payloads) { return payloads.stream().map(this::decodePayload).collect(Collectors.toList()); } private Payload encodePayload(Payload payload) { String keyId = getKeyId(); SecretKey key = getKey(keyId); byte[] encryptedData; try { encryptedData = encrypt(payload.toByteArray(), key); // The encrypt method contains your custom encryption logic. } catch (Throwable e) { throw new DataConverterException(e); } // Apply metadata to the encoded Payload that you can verify in your decode method before decoding. // See the sample for details on the metadata values set. return Payload.newBuilder() .putMetadata(EncodingKeys.METADATA_ENCODING_KEY, METADATA_ENCODING) .putMetadata(METADATA_ENCRYPTION_CIPHER_KEY, METADATA_ENCRYPTION_CIPHER) .putMetadata(METADATA_ENCRYPTION_KEY_ID_KEY, ByteString.copyFromUtf8(keyId)) .setData(ByteString.copyFrom(encryptedData)) .build(); } private Payload decodePayload(Payload payload) { // Verify the incoming encoded Payload metadata before applying decryption. if (METADATA_ENCODING.equals( payload.getMetadataOrDefault(EncodingKeys.METADATA_ENCODING_KEY, null))) { String keyId; try { keyId = payload.getMetadataOrThrow(METADATA_ENCRYPTION_KEY_ID_KEY).toString(UTF_8); } catch (Exception e) { throw new PayloadCodecException(e); } SecretKey key = getKey(keyId); byte[] plainData; Payload decryptedPayload; try { plainData = decrypt(payload.getData().toByteArray(), key); // The decrypt method contains your custom decryption logic. decryptedPayload = Payload.parseFrom(plainData); return decryptedPayload; } catch (Throwable e) { throw new PayloadCodecException(e); } } else { return payload; } } private String getKeyId() { // Currently there is no context available to vary which key is used. // Use a fixed key for all payloads. // This still supports key rotation as the key ID is recorded on payloads allowing // decryption to use a previous key. return "test-key-test-key-test-key-test!"; } private SecretKey getKey(String keyId) { // Key must be fetched from KMS or other secure storage. // Hard coded here only for example purposes. return new SecretKeySpec(keyId.getBytes(UTF_8), "AES"); } //... } ``` **Set Data Converter to use custom Payload Codec** Use `CodecDataConverter` with an instance of a Data Converter and the custom `PayloadCodec` in the `WorkflowClient` options that you use in your Worker process and to start your Workflow Executions. For example, to set a custom `PayloadCodec` implementation with `DefaultDataConverter`, use the following code: ```java WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); // Client that can be used to start and signal Workflows WorkflowClient client = WorkflowClient.newInstance( service, WorkflowClientOptions.newBuilder() .setDataConverter( new CodecDataConverter( DefaultDataConverter.newDefaultInstance(), Collections.singletonList(new YourCustomPayloadCodec()))) // Sets the custom Payload Codec created in the previous example with an instance of the default Data Converter. .build()); ``` - Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. - Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. For reference, see the [Encryption](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads) sample. ### Using a Codec Server A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. For reference, see the [Codec server](https://github.com/temporalio/sdk-java/tree/main/temporal-remote-data-encoder) sample. --- # Debugging - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/debugging > Debug your Temporal Java Workflows using your favorite Java IDE's debugger. Set the TEMPORAL_DEBUG environment variable to true during debugging to avoid deadlocks. Use Web UI, Temporal CLI, and logging for development and production. Optimize Worker performance with metrics and the Worker performance guide. In addition to writing unit and integration tests, debugging your Workflows is also a very valuable testing tool. You can debug your Workflow code using a debugger provided by your favorite Java IDE. Note that when debugging your Workflow code, the Temporal Java SDK includes deadlock detection which fails a Workflow Task in case the code blocks over a second without relinquishing execution control. Because of this you can often encounter the `PotentialDeadlockException` Exception while stepping through Workflow code during debugging. To alleviate this issue, you can set the `TEMPORAL_DEBUG` environment variable to true before debugging your Workflow code. Make sure to set `TEMPORAL_DEBUG` to true only during debugging. ## How to debug in a development environment In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). ## How to debug in a production environment You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/java/best-practices/testing-suite#replay) - [Tracing](/develop/java/platform/observability#tracing) - [Logging](/develop/java/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). For more information, see [Observability ▶️ Metrics](/develop/java/platform/observability#metrics) for setting up SDK metrics. Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). --- # Error handling - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/error-handling > Catch the right exception types, wrap checked exceptions, and inspect failures correctly in Temporal Java Workflows and Activities. Temporal represents failures with a small set of typed exceptions that all extend [`TemporalFailure`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TemporalFailure.html). For what each type means and when the Temporal Service raises it, see the [Temporal Failures reference](/references/failures). This page covers the two things that Java Workflow and Activity code get wrong most often: catching a wider exception type than intended, and manually adding checked exceptions to method signatures instead of using the SDK's wrapping helpers. ## Catch `Exception`, never `Throwable` or `Error` Workflow and Activity code should only ever catch `Exception` or a narrower type. Never catch `Throwable` or `Error`. The Java SDK uses subclasses of `Error` as internal control signals that must reach the SDK's own code uncaught: - [`DestroyWorkflowThreadError`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/DestroyWorkflowThreadError.html) interrupts a Workflow thread so the Worker can release it back to the pool, for example when the Workflow Execution is evicted from the [Worker's cache](/develop/java/workers/run-worker-process). If Workflow code catches it, the thread doesn't unwind and eviction can stall. - [`UnsupportedVersion`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/statemachines/UnsupportedVersion.html) is thrown by [`Workflow.getVersion()`](/develop/java/workflows/versioning) when replayed history was produced by code outside the version range the current Workflow code declares. It extends `Error` specifically so that application code won't catch it by mistake. Use this rule of thumb when deciding what to catch in a Workflow, Update, or Signal handler: 1. **`Error`** — never catch it. If you must run cleanup on any exit path, use a [detached Cancellation Scope](/develop/java/workflows/cancellation) rather than a broad catch (see [Cancellation](#cancellation) below). 2. **`CanceledFailure`** — rethrow it, optionally after cleanup in a detached Cancellation Scope. Don't swallow it: cancellation is cooperative, and swallowing it lets the Workflow Execution finish as "Completed" instead of "Canceled." 3. **`ActivityFailure`, `ChildWorkflowFailure`, or `ApplicationFailure`** that you recognize and can recover from — handle it. 4. **Everything else** — rethrow it. A plain `RuntimeException` that isn't recognized above fails the current [Workflow Task](/tasks#workflow-task-execution), which retries indefinitely rather than failing the Workflow Execution. To fail the Workflow Execution deliberately, throw an `ApplicationFailure` (see [Fail a Workflow deliberately](#fail-a-workflow) below). ### Don't swallow failures with a broad catch A `catch (Throwable t)` (or `catch (Exception e)` that only logs and returns) placed around Workflow logic "to be safe" is the most common way this rule gets broken: ```java // Anti-pattern: never do this in Workflow, Update, or Signal code. try { return doWorkflowLogic(); } catch (Throwable t) { logger.error("workflow failed", t); return defaultResult(); } ``` This single catch block causes three separate problems at once: - `DestroyWorkflowThreadError` and `UnsupportedVersion` are swallowed instead of reaching the SDK, which can stall Worker cache eviction and interfere with replay. - `CanceledFailure` is swallowed, so a canceled Workflow Execution reports "Completed" instead of "Canceled." - Every other exception — including real bugs — disappears with only a log line instead of failing the Workflow Task or Workflow Execution, so there's no signal in the Event History that anything went wrong. Catch the narrowest type you can actually recover from, log and rethrow anything you don't recognize, and let `Error` and `CanceledFailure` propagate untouched: ```java try { return doWorkflowLogic(); } catch (ActivityFailure e) { if (e.getCause() instanceof ApplicationFailure appFailure && "ValidationError".equals(appFailure.getType())) { return defaultResult(); } throw e; // don't recognize it — propagate } ``` ## Wrap checked exceptions instead of adding them to method signatures Activity and Workflow method signatures shouldn't declare `throws` for checked exceptions. Instead, wrap a checked exception with [`Activity.wrap()`]() inside an Activity, or [`Workflow.wrap()`]() inside a Workflow, before rethrowing it: ```java static class GreetingActivitiesImpl implements GreetingActivities { @Override public String composeGreeting(String greeting, String name) { try { return callExternalService(greeting, name); // declares `throws IOException` } catch (IOException e) { throw Activity.wrap(e); } } } ``` `wrap()` only does something to a checked exception. Given that, you don't need to reason about what to pass it: - If `e` is a checked exception, `wrap()` returns a `CheckedExceptionWrapper` around it. The SDK unwraps it automatically while propagating the failure and attaches the original exception as the `cause` of the resulting `ApplicationFailure` — so the caller still sees your original exception type and message in the cause chain. - If `e` already extends `RuntimeException`, `wrap()` returns it unchanged. Calling `wrap()` on an unchecked exception is a safe no-op, so you don't need to check the exception type before calling it. - If `e` extends `Error`, `wrap()` rethrows it directly instead of wrapping it, consistent with [never catching `Throwable` or `Error`](#catch-exception-not-throwable). This also answers the two questions that come up most when working with wrapped exceptions: - **Do you need to wrap exceptions you throw yourself?** Only checked exceptions need `wrap()`. Any unhandled exception an Activity or Workflow throws — checked or not — is already converted to an `ApplicationFailure` automatically when it crosses the Activity or Workflow boundary. `wrap()` exists to satisfy the Java compiler when you want to throw a checked exception from a method that doesn't declare it, not to make the exception propagate. - **Do you need to re-wrap an exception after unwrapping it?** No. Once you've unwrapped a cause to inspect it, either rethrow the failure you caught (`throw e;`) or throw a new `ApplicationFailure` with the original exception set as its cause. There's no wrapper left to reapply — `wrap()` only matters at the point where a checked exception would otherwise need a `throws` declaration. ## Read a failure's cause chain An exception thrown from an Activity or Child Workflow arrives at the caller wrapped with context about where it failed. A failure from an Activity called from a Child Workflow called from a parent Workflow looks like this by the time it reaches a synchronous client call: ```text WorkflowFailedException (thrown to the client) └─ ChildWorkflowFailure (the child Workflow Execution failed) └─ ActivityFailure (the Activity Execution failed) └─ ApplicationFailure (what your code actually threw) ``` Each wrapper adds context — `ActivityFailure` carries the Activity Type and Activity Id, `ChildWorkflowFailure` carries the Workflow Type and Workflow Id — while `getCause()` on each layer moves toward what actually failed. See [Temporal Failures reference](/references/failures) for what each of these types means for retry behavior. Two details matter once you reach an `ApplicationFailure`: - **Read `getOriginalMessage()`, not `getMessage()`.** `getMessage()` returns a decorated string such as `message='Invalid credit card number', type='ValidationError', nonRetryable=true` — meant for logs, not parsing. `getOriginalMessage()` returns the exact text you threw. - **Match on `getType()`, a stable `String`, not `instanceof` your original exception class.** `ApplicationFailure` is `final` and the original exception object doesn't survive serialization: when an Activity in another process (or another SDK language) throws, the caller only ever gets an `ApplicationFailure` back, never your custom exception type. `type` defaults to the thrown exception's fully qualified class name unless you set it explicitly with `ApplicationFailure.newFailure(message, type, ...)`. ```java try { return activities.processCreditCard(orderId); } catch (ActivityFailure e) { if (e.getCause() instanceof ApplicationFailure appFailure) { if ("ValidationError".equals(appFailure.getType())) { return Result.rejected(appFailure.getOriginalMessage()); } } throw e; } ``` ## Handle Activity and Child Workflow failures Catch `ActivityFailure` (or `ChildWorkflowFailure`) around a call, not `ApplicationFailure` directly — the Activity or Child Workflow boundary always wraps the underlying failure. Always check for `CanceledFailure` as the cause before handling anything else, and rethrow it unhandled: ```java try { return activities.charge(order); } catch (ActivityFailure e) { if (e.getCause() instanceof CanceledFailure) { throw e; // never swallow cancellation } if (e.getCause() instanceof ApplicationFailure appFailure && "PaymentDeclined".equals(appFailure.getType())) { return Result.declined(appFailure.getOriginalMessage()); } throw e; // don't recognize it — propagate } ``` For deciding which failures should skip retries, see [Non-Retryable Errors](/design-patterns/non-retryable-errors), which has a Java example for both marking a failure non-retryable at the throw site and listing non-retryable types in a `RetryPolicy`. For undoing the effects of Activities that already succeeded before a later step failed, see the [Saga Pattern](/design-patterns/saga-pattern), which has a Java example built on the SDK's `Saga` helper. For the full set of retry strategies, see [Error Handling & Retry Patterns](/design-patterns/error-handling-patterns). ### Cancellation Cancellation is cooperative — the Worker never force-stops running Workflow code. A cancellation request cancels the current [Cancellation Scope](/develop/java/workflows/cancellation), and the next cancelable call inside it (an Activity, Timer, or Child Workflow) throws `CanceledFailure`. If you need cleanup to run after a cancellation — for example, compensating an Activity that already applied its effect — run it in a [detached Cancellation Scope](/develop/java/workflows/cancellation#cancel-activity), since a normal scope is a child of the one that was just canceled and any call inside it would be canceled immediately: ```java try { activities.longRunningWork(); } catch (CanceledFailure e) { Workflow.newDetachedCancellationScope(() -> activities.compensate()).run(); throw e; // rethrow after cleanup so the Workflow Execution ends "Canceled" } ``` ## Centralize failure conversion with a Worker Interceptor Activity code that calls several external services often ends up repeating the same `catch` blocks to convert domain exceptions into `ApplicationFailure` with a consistent `type` and non-retryable classification. A [`WorkerInterceptor`](/develop/plugins-guide#interceptors) that overrides [`ActivityInboundCallsInterceptor.execute()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/interceptors/ActivityInboundCallsInterceptor.html) centralizes that mapping in one place instead of repeating it in every Activity implementation: ```java public final class ErrorNormalizingWorkerInterceptor extends WorkerInterceptorBase { @Override public ActivityInboundCallsInterceptor interceptActivity(ActivityInboundCallsInterceptor next) { return new ActivityInboundCallsInterceptorBase(next) { @Override public ActivityOutput execute(ActivityInput input) { try { return super.execute(input); } catch (ApplicationFailure | TimeoutFailure | CanceledFailure f) { throw f; // already a well-formed Temporal failure — pass through } catch (PaymentDeclinedException e) { throw ApplicationFailure.newNonRetryableFailure(e.getMessage(), "PaymentDeclined", e.toDetail()); } catch (Exception e) { throw Activity.wrap(e); // uniform fallback: type = class name, retryable } } }; } } ``` Register it on the Worker Factory: ```java WorkerFactoryOptions.newBuilder() .setWorkerInterceptors(new ErrorNormalizingWorkerInterceptor()) .build(); ``` Because the interceptor sees the original exception before the SDK's default conversion runs, it's the layer to attach `details` and a non-retryable classification consistently, rather than depending on every Activity implementation to do it the same way. `WorkerInterceptor` and `ActivityInboundCallsInterceptor` are marked `@Experimental`. For the general interceptor model — inbound versus outbound, and the other call categories you can intercept — see [Interceptors](/develop/plugins-guide#interceptors). ## Fail a Workflow deliberately Throwing `ApplicationFailure` from Workflow code is the only way to fail a Workflow Execution deliberately. Any other unhandled exception fails only the current Workflow Task, which the Worker retries indefinitely — this is intentional: a plain bug should be fixable with a new deployment, not permanently fail every Workflow Execution that hit it. ```java if (order.getTotal().compareTo(BigDecimal.ZERO) <= 0) { throw ApplicationFailure.newNonRetryableFailure( "Order total must be positive: " + order.getTotal(), "InvalidOrderTotal"); } ``` If you want specific plain exception types to fail the Workflow Execution instead of retrying the Workflow Task, list them with [`WorkflowImplementationOptions.setFailWorkflowExceptionTypes()`]() when registering the Workflow implementation. Never extend `TemporalFailure` or any of its subclasses in application code — throw `ApplicationFailure` instead. The SDK reserves the other subclasses (`ActivityFailure`, `ChildWorkflowFailure`, `CanceledFailure`, `TimeoutFailure`, `TerminatedFailure`, `ServerFailure`) for its own use. --- # Testing - Java SDK Source: https://docs.temporal.io/develop/java/best-practices/testing-suite > The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, and unit tests. Unit tests can be set up and run using the Temporal Java SDK's TestWorkflowEnvironment and TestWorkflowExtension classes for automated testing, allowing developers to test Workflows The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows, Activities, and Nexus Operations; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities and Nexus Operations, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow, Activity, or Nexus Operation code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks The Temporal Java SDK provides a test framework to facilitate Workflow unit and integration testing. The test framework provides a `TestWorkflowEnvironment` class which includes an in-memory implementation of the Temporal service that supports automatic time skipping. This allows you to easily test long-running Workflows in seconds, without having to change your Workflow code. You can use the provided `TestWorkflowEnvironment` with a Java unit testing framework of your choice, such as JUnit. ### Setup testing dependency To start using the Java SDK test framework, you need to add [`io.temporal:temporal-testing`](https://search.maven.org/artifact/io.temporal/temporal-testing) as a dependency to your project: **[Apache Maven](https://maven.apache.org/):** ```maven io.temporal temporal-testing 1.36.0 test ``` **[Gradle Groovy DSL](https://gradle.org/):** ```groovy testImplementation ("io.temporal:temporal-testing:1.36.0") ``` If you need JUnit4 or JUnit5 extensions: ``` testImplementation("io.temporal:temporal-testing:1.36.0") { capabilities { requireCapability("io.temporal:temporal-testing-junit4") //requireCapability("io.temporal:temporal-testing-junit5") } } ``` Make sure to set the version that matches your dependency version of the [Temporal Java SDK](https://github.com/temporalio/sdk-java). ## Test Activities An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. Temporal provides the `TestActivityEnvironment` and `TestActivityExtension` classes for testing Activities outside the scope of a Workflow. Testing Activities is similar to testing non-Temporal Java code. For example, you can test an Activity for: - Exceptions thrown when invoking the Activity Execution. - Exceptions thrown when checking for the result of the Activity Execution. - Activity's return values. Check that the return value matches the expected value. Here's an example of an Activity that will be referenced in the sections below: ```java package helloworkflow; import java.util.concurrent.TimeUnit; import io.temporal.activity.Activity; import io.temporal.activity.ActivityExecutionContext; import io.temporal.client.ActivityCompletionException; public class GreetActivitiesImpl implements GreetActivities { @Override public String greet(String name) { ActivityExecutionContext context = Activity.getExecutionContext(); for (int i = 0; i < 5; i++) { try { context.heartbeat(null); } catch (ActivityCompletionException e) { throw e; } sleep(1); } return "Hello " + name + "!"; } private void sleep(int seconds) { try { Thread.sleep(TimeUnit.SECONDS.toMillis(seconds)); } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException(e); } } } ``` ### Run an Activity The following code implements unit tests for the `greet` Activity above: ```java package helloworkflow; import static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; import io.temporal.testing.TestActivityEnvironment; public class GreetActivitiesTest { @Test public void testActivityImpl() { TestActivityEnvironment testEnv = TestActivityEnvironment.newInstance(); testEnv.registerActivitiesImplementations(new GreetActivitiesImpl()); GreetActivities activities = testEnv.newActivityStub(GreetActivities.class); String result = activities.greet("Temporal"); assertEquals("Hello Temporal!", result); } } ``` ### Listen to Heartbeats Activities usually issue periodic Heartbeats, a ping that shows that an Activity is making progress and the Worker hasn't crashed. Heartbeats may include details that report task progress in the event an Activity Worker crashes. When testing Activities that support Heartbeats, make sure you can see those Heartbeats in your test code. ```java @Test void testActivityHeartbeat() { TestActivityEnvironment env = TestActivityEnvironment.newInstance(); AtomicInteger heartbeatCount = new AtomicInteger(0); env.setActivityHeartbeatListener( Void.class, heartbeat -> heartbeatCount.incrementAndGet()); env.registerActivitiesImplementations(new GreetActivitiesImpl()); GreetActivities activities = env.newActivityStub(GreetActivities.class); String result = activities.greet("Temporal"); assertEquals("Hello Temporal!", result); assertEquals(5, heartbeatCount.get()); } ``` ### Cancel an Activity Activity cancellation lets Activities know they don't need to continue work and gives time for the Activity to clean up any resources it's created. You can cancel Java-based Activities if they emit Heartbeats. To test an Activity that reacts to Cancellations, make sure that the Activity reacts correctly and cancels. ```java @Test void testCancelActivity() { try (TestWorkflowEnvironment env = TestWorkflowEnvironment.newInstance()) { Worker worker = env.newWorker(TASK_QUEUE); worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetActivitiesImpl()); env.start(); SayHelloWorkflow workflow = env.getWorkflowClient() .newWorkflowStub( SayHelloWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(TASK_QUEUE) .build()); WorkflowClient.start(workflow::sayHello, "Temporal"); env.registerDelayedCallback( Duration.ofSeconds(1), () -> WorkflowStub.fromTyped(workflow).signal("cancelActivity")); try { WorkflowStub.fromTyped(workflow).getResult(String.class); fail("Workflow should have failed because the Activity was canceled"); } catch (WorkflowFailedException e) { assertEquals(ActivityFailure.class, e.getCause().getClass()); ActivityFailure activityFailure = (ActivityFailure) e.getCause(); assertEquals(CanceledFailure.class, activityFailure.getCause().getClass()); } } } ``` ## Testing Workflows ### How to mock Activities When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker. In cases where you do not wish to execute your actual Activity or Nexus Operation implementations during unit testing, you can use a framework such as [Mockito](https://site.mockito.org/) to mock them. The following code implements a unit test that shows how Activities can be mocked: ```java @Test public void testMockedActivity() { GreetActivities activities = mock(GreetActivities.class, withSettings().withoutAnnotations()); when(activities.greet("Temporal")).thenReturn("Hello Temporal!"); assertEquals("Hello Temporal!", activities.greet("Temporal")); } ``` #### Testing with JUnit4 For JUnit4 tests, Temporal provides the `TestWorkflowRule` class which simplifies the Temporal test environment setup, as well as the creation and shutdown of Workflow Workers in your tests. You can now rewrite the above Activity test class as follows: ```java public class GreetActivitiesJUnit4Test { @Rule public TestWorkflowRule testWorkflowRule = TestWorkflowRule.newBuilder() .setWorkflowTypes(SayHelloWorkflowImpl.class) .setActivityImplementations(new GreetActivitiesImpl()) .build(); @Test public void testActivityImpl() { // Get a workflow stub using the same task queue the worker uses. GreetingWorkflow workflow = testWorkflowRule .getWorkflowClient() .newWorkflowStub( GreetingWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build() ); // Execute a workflow waiting for it to complete. String greeting = workflow.sayHello("Temporal"); assertEquals("Hello Temporal!", greeting); testWorkflowRule.getTestEnvironment().shutdown(); } } ``` #### Testing with JUnit5 For JUnit5 tests, Temporal also provides the `TestWorkflowExtension` helper class. This class can be used to simplify the Temporal test environment setup as well as Workflow Worker startup and shutdowns. To start using JUnit5 `TestWorkflowExtension` in your tests with [Gradle](https://gradle.org/), you need to enable capability [`io.temporal:temporal-testing-junit5`]: Now you can use JUnit5 and rewrite the above test class as follows: ```java public class GreetActivitiesJUnit5Test { @RegisterExtension public static final TestWorkflowExtension testWorkflowExtension = TestWorkflowExtension.newBuilder() .setWorkflowTypes(SayHelloWorkflowImpl.class) .setActivityImplementations(new GreetActivitiesImpl()) .build(); @Test public void testActivityImpl( TestWorkflowEnvironment testEnv, Worker worker, SayHelloWorkflow workflow) { // Execute a workflow waiting for it to complete. String greeting = workflow.sayHello("Temporal"); assertEquals("Hello Temporal!", greeting); } } ``` You can find more unit test examples in the [Temporal Java samples](https://github.com/temporalio/samples-java) repository in [its test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples). ### How to mock Nexus Operations When integration testing Workflows with a Worker, you can mock Nexus operations by providing mock Nexus Service handlers to the Worker. Alternatively, you could just mock the Nexus service itself. You can find more example unit tests for Nexus in the [Temporal Java samples](https://github.com/temporalio/samples-java) repository in [this test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples/nexus/caller). These samples show how to call Nexus services in tests using the Temporal testing package and also how to mock them, for both JUnit 4 and 5. Detailed explanatory comments are included in the code in the repository. To mock Nexus handlers, create a `Rule` (for JUnit4) or `Extension` (for JUnit5) from the Temporal testing package and add a call to `setNexusServiceImplementation` to the builder. That sets up the Nexus endpoints needed for testing as well as the Nexus handler workflows defined by the Nexus Service implementation. You will need to create Workers for each handler, using either `setWorkflowTypes` (for JUnit4) or `registerWorkflowImplementationTypes` (for JUnit5). With that in place, you can then mock a Nexus endpoint exactly like any other Workflow like you do with Activity tests. The following are samples derived from [the test package](https://github.com/temporalio/samples-java/tree/main/core/src/test/java/io/temporal/samples/nexus/caller) to demonstrate this. #### Mocking Nexus handlers with JUnit4 [core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowMockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowMockTest.java) ```java public class CallerWorkflowMockTest { @Rule public TestWorkflowRule testWorkflowRule = TestWorkflowRule.newBuilder() .setNexusServiceImplementation(new SampleNexusServiceImpl()) .setWorkflowTypes(HelloCallerWorkflowImpl.class) .build(); @Test public void testHelloWorkflow() { testWorkflowRule .getWorker() // Workflows started by a Nexus service can be mocked just like any other workflow .registerWorkflowImplementationFactory( HelloHandlerWorkflow.class, () -> { HelloHandlerWorkflow wf = mock(HelloHandlerWorkflow.class); when(wf.hello(any())).thenReturn(new SampleNexusService.HelloOutput("Hello Mock World")); return wf; }); // Now create the caller workflow HelloCallerWorkflow workflow = testWorkflowRule .getWorkflowClient() .newWorkflowStub( HelloCallerWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build()); String greeting = workflow.hello("World", SampleNexusService.Language.EN); assertEquals("Hello Mock World", greeting); } } ``` #### Mocking Nexus handlers with JUnit5 [core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowJunit5MockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/CallerWorkflowJunit5MockTest.java) ```java public class CallerWorkflowJunit5MockTest { @RegisterExtension public static final TestWorkflowExtension testWorkflowExtension = TestWorkflowExtension.newBuilder() // Register the Nexus service as usual and mock things in the unit tests as needed .setNexusServiceImplementation(new SampleNexusServiceImpl()) .registerWorkflowImplementationTypes(HelloCallerWorkflowImpl.class) .build(); @Test public void testHelloWorkflow( TestWorkflowEnvironment testEnv, Worker worker, HelloCallerWorkflow workflow) { // Workflows started by a Nexus service can be mocked just like any other workflow worker.registerWorkflowImplementationFactory( HelloHandlerWorkflow.class, () -> { HelloHandlerWorkflow mockHandler = mock(HelloHandlerWorkflow.class); when(mockHandler.hello(any())) .thenReturn(new SampleNexusService.HelloOutput("Hello Mock World")); return mockHandler; }); // Execute a workflow waiting for it to complete. String greeting = workflow.hello("World", SampleNexusService.Language.EN); assertEquals("Hello Mock World", greeting); } } ``` An alternative approach is to mock the Nexus service itself, instead of mocking the handlers. This is useful if you want to test the calling logic, but can't easily mock the Nexus handlers. The code will mock the implementation of the `SampleNexusService` class with the handler methods, but will need those methods stubbed in for the testing framework. Those methods can be directly mocked with static return values or they can return an instance variable which each unit test can modify to return a desired value. #### Mocking the Nexus Service with JUnit4 [core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceMockTest.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceMockTest.java) ```java public class NexusServiceMockTest { private final SampleNexusService mockNexusService = mock(SampleNexusService.class); /** * A test-only Nexus service implementation that delegates to the Mockito mock defined above. The * operation is implemented as a synchronous handler that forward calls to the mock, allowing * full control over return values and verification of inputs. */ @ServiceImpl(service = SampleNexusService.class) public class TestNexusServiceImpl { @OperationImpl @SuppressWarnings("DirectInvocationOnMock") public OperationHandler hello() { return OperationHandler.sync((ctx, details, input) -> mockNexusService.hello(input)); } } // Using OperationHandler.sync for the operation bypasses the need for a backing workflow, // returning results inline just like a synchronous call. @Rule public TestWorkflowRule testWorkflowRule = TestWorkflowRule.newBuilder() .setNexusServiceImplementation(new TestNexusServiceImpl()) .setWorkflowTypes(HelloCallerWorkflowImpl.class) .build(); @Test public void testHelloCallerWithMockedService() { when(mockNexusService.hello(any())) .thenReturn(new SampleNexusService.HelloOutput("Bonjour World")); HelloCallerWorkflow workflow = testWorkflowRule .getWorkflowClient() .newWorkflowStub( HelloCallerWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(testWorkflowRule.getTaskQueue()).build()); String result = workflow.hello("World", SampleNexusService.Language.FR); assertEquals("Bonjour World", result); // Verify the Nexus service was called with the correct name and language verify(mockNexusService) .hello( argThat( input -> "World".equals(input.getName()) && SampleNexusService.Language.FR == input.getLanguage())); // Verify the operation was called exactly once and no other operations were invoked verify(mockNexusService, times(1)).hello(any()); } } ``` #### Mocking the Nexus Service with JUnit5 [core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceJunit5Test.java](https://github.com/temporalio/samples-java/blob/main/core/src/test/java/io/temporal/samples/nexus/caller/NexusServiceJunit5Test.java) ```java public class NexusServiceJunit5Test { private final SampleNexusService mockNexusService = mock(SampleNexusService.class); /** * A test-only Nexus service implementation that delegates to the Mockito mock defined above. The * operation is implemented as a synchronous handler that forward calls to the mock, allowing * full control over return values and verification of inputs. */ @ServiceImpl(service = SampleNexusService.class) public class TestNexusServiceImpl { @OperationImpl @SuppressWarnings("DirectInvocationOnMock") public OperationHandler hello() { return OperationHandler.sync((ctx, details, input) -> mockNexusService.hello(input)); } } // Using OperationHandler.sync for both operations bypasses the need for a backing workflow, // returning results inline just like a synchronous call. @RegisterExtension public final TestWorkflowExtension testWorkflowExtension = TestWorkflowExtension.newBuilder() // If a Nexus service is registered as part of the test as in the following line of code, // the TestWorkflowExtension will, by default, automatically create a Nexus service // endpoint and workflows registered as part of the TestWorkflowExtension will // automatically inherit the endpoint if none is set. .setNexusServiceImplementation(new TestNexusServiceImpl()) // registerWorkflowImplementationTypes will take the classes given and create workers for // them, enabling workflows to run. // Since both operations are mocked with OperationHandler.sync, no backing workflow is // needed for hello — only the caller workflow types need to be registered. .registerWorkflowImplementationTypes(HelloCallerWorkflowImpl.class) // The workflow will start before each test, and will shut down after each test. // See CallerWorkflowTest for an example of how to control this differently if needed. .build(); // The TestWorkflowExtension extension in the Temporal testing library creates the // arguments to the test cases and initializes them from the extension setup call above. @Test public void testHelloWorkflow( TestWorkflowEnvironment testEnv, Worker worker, HelloCallerWorkflow workflow) { // Set the mock value to return when(mockNexusService.hello(any())) .thenReturn(new SampleNexusService.HelloOutput("Hello Mock World")); // Execute a workflow waiting for it to complete. String greeting = workflow.hello("World", SampleNexusService.Language.EN); assertEquals("Hello Mock World", greeting); // Verify the operation was called exactly once and no other operations were invoked verify(mockNexusService, times(1)).hello(any()); // Verify the Nexus service was called with the correct input verify(mockNexusService).hello(argThat(input -> "World".equals(input.getName()))); verifyNoMoreInteractions(mockNexusService); } } ``` ### How to skip time Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Skip time automatically When you execute a Workflow and wait for the result, the test environment automatically skips Workflow timers such as `Workflow.sleep`. This means: - Workflow timers (like `Workflow.sleep`) are fast-forwarded. - Time doesn't skip while Activities and Nexus operations are executing. Nexus operation handlers timeout after 10 seconds and time skipping is allowed while waiting for retries. Here's an example of a Workflow that implements `Workflow.sleep` to wait for a day before calling the Activity: ```java public class SayHelloWorkflowImpl implements SayHelloWorkflow { private final GreetActivities activities = Workflow.newActivityStub( GreetActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .build()); @Override public String sayHello(String name) { // This will complete immediately in TestWorkflowEnvironment because // Workflow time is automatically advanced. Workflow.sleep(Duration.ofDays(1)); return activities.greet(name); } } ``` Here's what the test for that Workflow could look like: ```java @Test void testSleepCompletesWithoutWaitingOneDay() { try (TestWorkflowEnvironment testEnv = TestWorkflowEnvironment.newInstance()) { Worker worker = testEnv.newWorker(TASK_QUEUE); worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetActivitiesImpl()); testEnv.start(); SayHelloWorkflow workflow = testEnv .getWorkflowClient() .newWorkflowStub( SayHelloWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build()); String result = workflow.sayHello("Temporal"); assertEquals("Hello Temporal!", result); } } ``` Use `Workflow.sleep()` in Workflow code, not `Thread.sleep()`. `Workflow.sleep()` creates a Temporal timer that the test environment can skip. `Thread.sleep()` blocks a Java thread in real time and is not controlled by Temporal’s time-skipping test server. #### Skip time manually Use `TestWorkflowEnvironment.sleep(Duration)` when you want to advance virtual time yourself and inspect intermediate Workflow state. Start the Workflow asynchronously with `WorkflowClient.start()`, then call `testEnv.sleep()` from the test. Here's an example of a Workflow test that lets you manually advance time with `testEnv.sleep()`: ```java package helloworkflow; import java.time.Duration; import static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowOptions; import io.temporal.testing.TestWorkflowEnvironment; import io.temporal.worker.Worker; import io.temporal.workflow.QueryMethod; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; public class ManualTimeSkippingTest { private static final String TASK_QUEUE = "manual-time-skipping-test"; @WorkflowInterface public interface ProgressWorkflow { @WorkflowMethod void run(); @QueryMethod int daysElapsed(); } public static class ProgressWorkflowImpl implements ProgressWorkflow { private int daysElapsed = 0; @Override public void run() { for (int i = 0; i < 100; i++) { Workflow.sleep(Duration.ofDays(1)); daysElapsed++; } } @Override public int daysElapsed() { return daysElapsed; } } @Test void manuallyAdvanceWorkflowTime() { try (TestWorkflowEnvironment testEnv = TestWorkflowEnvironment.newInstance()) { Worker worker = testEnv.newWorker(TASK_QUEUE); worker.registerWorkflowImplementationTypes(ProgressWorkflowImpl.class); testEnv.start(); ProgressWorkflow workflow = testEnv .getWorkflowClient() .newWorkflowStub( ProgressWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build()); WorkflowClient.start(workflow::run); assertEquals(0, workflow.daysElapsed()); testEnv.sleep(Duration.ofHours(25)); assertEquals(1, workflow.daysElapsed()); testEnv.sleep(Duration.ofHours(25)); assertEquals(2, workflow.daysElapsed()); } } } ``` ## How to Replay a Workflow Execution Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: To replay Workflow Executions, use the [WorkflowReplayer](https://www.javadoc.io/doc/io.temporal/temporal-testing/latest/io/temporal/testing/WorkflowReplayer.html) class in the `temporal-testing` package. In the following example, Event Histories are downloaded from the server, and then replayed. Note that this requires Advanced Visibility to be enabled. ```java // Note we assume you already have a WorkflowServiceStubs (`service`) and WorkflowClient (`client`) // in scope. ListWorkflowExecutionsRequest listWorkflowExecutionRequest = ListWorkflowExecutionsRequest.newBuilder() .setNamespace(client.getOptions().getNamespace()) .setQuery("TaskQueue = 'mytaskqueue'") .build(); ListWorkflowExecutionsResponse listWorkflowExecutionsResponse = service.blockingStub().listWorkflowExecutions(listWorkflowExecutionRequest); List histories = listWorkflowExecutionsResponse.getExecutionsList().stream() .map( (info) -> { GetWorkflowExecutionHistoryResponse weh = service.blockingStub().getWorkflowExecutionHistory( GetWorkflowExecutionHistoryRequest.newBuilder() .setNamespace(testEnvironment.getNamespace()) .setExecution(info.getExecution()) .build()); return new WorkflowExecutionHistory( weh.getHistory(), info.getExecution().getWorkflowId()); }) .collect(Collectors.toList()); WorkflowReplayer.replayWorkflowExecutions( histories, true, WorkflowA.class, WorkflowB.class, WorkflowC.class); ``` In the next example, a single history is loaded from a JSON file on disk: ```java File file = new File("my_history.json"); WorkflowReplayer.replayWorkflowExecution(file, MyWorkflow.class); ``` In both examples, if Event History is non-deterministic, an error is thrown. You can choose to wait until all histories have been replayed with `replayWorkflowExecutions` by setting the `failFast` argument to `false`. --- # Client - Java SDK Source: https://docs.temporal.io/develop/java/client > This section explains how to implement the Temporal Client with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Temporal Client - [Temporal Client](/develop/java/client/temporal-client) - [Namespaces](/develop/java/client/namespaces) --- # Namespaces - Java SDK Source: https://docs.temporal.io/develop/java/client/namespaces > Register, update, deprecate, and delete Namespaces using Temporal CLI or SDK APIs. Manage Workflow Executions with isolated Namespaces to match your needs. This page shows how to do the following: - [Register a Namespace](#register-namespace) - [Manage Namespaces](#manage-namespaces) You can create, update, deprecate or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs. Use Namespaces to isolate your Workflow Executions according to your needs. For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces. You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line. On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. ## Register a Namespace Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces. On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. Use the [`RegisterNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to register a [Namespace](/namespaces) and set the [Retention Period](/temporal-service/temporal-server#retention-period) for the Workflow Execution Event History for the Namespace. ```java //... import com.google.protobuf.util.Durations; import io.temporal.api.workflowservice.v1.RegisterNamespaceRequest; //... public static void createNamespace(String name) { RegisterNamespaceRequest req = RegisterNamespaceRequest.newBuilder() .setNamespace("your-custom-namespace") .setWorkflowExecutionRetentionPeriod(Durations.fromDays(3)) // keeps the Workflow Execution //Event History for up to 3 days in the Persistence store. Not setting this value will throw an error. .build(); service.blockingStub().registerNamespace(req); } //... ``` The Retention Period setting using `WorkflowExecutionRetentionPeriod` is mandatory. The minimum value you can set for this period is 1 day. Once registered, set Namespace using `WorkflowClientOptions` within a Workflow Client to run your Workflow Executions within that Namespace. See [Connect to a Development Temporal Service](/develop/java/client/temporal-client#connect-to-development-service) for details. Note that Namespace registration using this API takes up to 10 seconds to complete. Ensure that you wait for this registration to complete before starting the Workflow Execution against the Namespace. To update your Namespace use the [UpdateNamespace API](#manage-namespaces) with the NamespaceClient. ## Manage Namespaces You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). - Update information and configuration for a registered Namespace on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace update`](/cli/command-reference/operator#update) Example - Use the [`UpdateNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to update configuration on a Namespace. Example ```java import io.temporal.api.workflowservice.v1.*; //... UpdateNamespaceRequest updateNamespaceRequest = UpdateNamespaceRequest.newBuilder() .setNamespace("your-namespace-name") //the namespace that you want to update .setUpdateInfo(UpdateNamespaceInfo.newBuilder() //has options to update namespace info .setDescription("your updated namespace description") //updates description in the namespace info. .build()) .setConfig(NamespaceConfig.newBuilder() //has options to update namespace configuration .setWorkflowExecutionRetentionTtl(Durations.fromHours(30)) //updates the retention period for the namespace "your-namespace--name" to 30 hrs. .build()) .build(); UpdateNamespaceResponse updateNamespaceResponse = namespaceservice.blockingStub().updateNamespace(updateNamespaceRequest); //... ``` - Get details for a registered Namespace on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace describe`](/cli/command-reference/operator#describe) - Use the [`DescribeNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to return information and configuration details for a registered Namespace. Example ```java import io.temporal.api.workflowservice.v1.*; //... DescribeNamespaceRequest descNamespace = DescribeNamespaceRequest.newBuilder() .setNamespace("your-namespace-name") //specify the namespace you want details for .build(); DescribeNamespaceResponse describeNamespaceResponse = namespaceservice.blockingStub().describeNamespace(descNamespace); System.out.println("Namespace Description: " + describeNamespaceResponse); //... ``` - Get details for all registered Namespaces on your Temporal Service: - With the Temporal CLI: [`temporal operator namespace list`](/cli/command-reference/operator#list) - Use the [`ListNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) to return information and configuration details for all registered Namespaces on your Temporal Service. Example ```java import io.temporal.api.workflowservice.v1.*; //... ListNamespacesRequest listNamespaces = ListNamespacesRequest.newBuilder().build(); ListNamespacesResponse listNamespacesResponse = namespaceservice.blockingStub().listNamespaces(listNamespaces); //lists 1-100 namespaces (1 page) in the active Temporal Service. To list all, set the page size or loop until NextPageToken is nil. //... ``` - Deprecate a Namespace: The [`DeprecateNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) updates the state of a registered Namespace to "DEPRECATED". Once a Namespace is deprecated, you cannot start new Workflow Executions on it. All existing and running Workflow Executions on a deprecated Namespace will continue to run. Example: ```java import io.temporal.api.workflowservice.v1.*; //... DeprecateNamespaceRequest deprecateNamespace = DeprecateNamespaceRequest.newBuilder() .setNamespace("your-namespace-name") //specify the namespace that you want to deprecate .build(); DeprecateNamespaceResponse response = namespaceservice.blockingStub().deprecateNamespace(deprecateNamespace); //... ``` - Delete a Namespace: The [`DeleteNamespace` API](https://github.com/temporalio/api/blob/main/temporal/api/workflowservice/v1/service.proto) deletes a Namespace. Deleting a Namespace deletes all running and completed Workflow Executions on the Namespace, and removes them from the persistence store and the visibility store. Example: ```java //... DeleteNamespaceResponse res = OperatorServiceStubs.newServiceStubs(OperatorServiceStubsOptions.newBuilder() .setChannel(service.getRawChannel()) .validateAndBuildWithDefaults()) .blockingStub() .deleteNamespace(DeleteNamespaceRequest.newBuilder().setNamespace("your-namespace-name").build()); //... ``` --- # Temporal Client - Java SDK Source: https://docs.temporal.io/develop/java/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the [Temporal Service](/temporal-service). Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals to Workflow Executions, sending Queries to Workflow Executions, getting the results of a Workflow Execution, and providing Activity Task Tokens. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the Java SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow-execution) - [Get Workflow results](#get-workflow-results) > **⚠️ Caution:** > > A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a > Temporal Client inside an Activity to communicate with a Temporal Service. > ## Connect to a development Temporal Service Use the `newLocalServiceStubs` method to create a stub that points to the Temporal development service, and then use the [`WorkflowClient.newInstance` method](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newInstance(io.temporal.serviceclient.WorkflowServiceStubs)) to create a Temporal Client. ```java // Create an instance that connects to a Temporal Service running on the local machine, using the default port 7233 WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newLocalServiceStubs(); // Initialize the Client WorkflowClient client = WorkflowClient.newInstance(serviceStub); ``` When you create a new Client with an instance of `newLocalServiceStubs`, the Client connects to the default local port at port 7233. When you don't specify a custom Namespace, the Client connects to the `default` Namespace. To connect to a custom Namespace, use the `WorkflowClientOptions.Builder.setNamespace` method to set the Namespace. Then pass the `clientOptions` to the `WorkflowClient.newInstance` method. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml`. For a list of all available configuration options, refer to [Environment Configuration](/references/client-environment-configuration) > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS is auto-enabled when this TLS config or API key is present, but you can configure it explicitly # disabled = false # Use certificate files for mTLS client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a specific profile from the configuration file as follows. First use `ClientConfigProfile.load` to load the profile from the configuration file. Then use `profile.toWorkflowServiceStubsOptions` and `profile.toWorkflowClientOptions` to convert the profile to `WorkflowServiceStubsOptions` and `WorkflowClientOptions` respectively. Then use `WorkflowClient.newInstance` to create a Temporal Client. ```java {21-25,32-34} import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadFromFile { private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class); public static void main(String[] args) { try { String configFilePath = Paths.get(LoadFromFile.class.getResource("/config.toml").toURI()).toString(); ClientConfigProfile profile = ClientConfigProfile.load( LoadClientConfigProfileOptions.newBuilder() .setConfigFilePath(configFilePath) .build()); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` **Environment Variables** Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. ```java {18-19,26-28} import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadFromFile { private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class); public static void main(String[] args) { try { ClientConfigProfile profile = ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build()); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. ```java // Add the Namespace as a Client Option WorkflowClientOptions clientOptions = WorkflowClientOptions .newBuilder() .setNamespace(namespace) .build(); // Initialize the Client WorkflowClient client = WorkflowClient.newInstance(service, clientOptions); ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through [mTLS](/cloud/certificates). Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace and Account ID_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `cloud` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.cloud] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the `LoadClientOptions` function in the `envconfig` package to create a Temporal Client using the `cloud` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. ```java {25-30,33-35,42-44} import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadProfile { private static final Logger logger = LoggerFactory.getLogger(LoadProfile.class); public static void main(String[] args) { String profileName = "cloud"; try { String configFilePath = Paths.get(LoadProfile.class.getResource("/config.toml").toURI()).toString(); logger.info("--- Loading '{}' profile from {} ---", profileName, configFilePath); // Load specific profile from file with environment variable overrides ClientConfigProfile profile = ClientConfigProfile.load( LoadClientConfigProfileOptions.newBuilder() .setConfigFilePath(configFilePath) .setConfigFileProfile(profileName) .build()); // Demonstrate programmatic override - fix the incorrect address from staging profile ClientConfigProfile.Builder profileBuilder = profile.toBuilder(); profileBuilder.setAddress("localhost:7233"); // Override the incorrect address profile = profileBuilder.build(); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your Java application. Import the `io.temporal.envconfig` package to set connection options for the Temporal Client using environment variables. The `ClientConfigProfile.load` method will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. ```java {17-18,25-27} import io.temporal.client.WorkflowClientOptions; import io.temporal.envconfig.ClientConfigProfile; import io.temporal.envconfig.LoadClientConfigProfileOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import java.nio.file.Paths; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class LoadFromFile { private static final Logger logger = LoggerFactory.getLogger(LoadFromFile.class); public static void main(String[] args) { try { ClientConfigProfile profile = ClientConfigProfile.load(LoadClientConfigProfileOptions.newBuilder().build()); WorkflowServiceStubsOptions serviceStubsOptions = profile.toWorkflowServiceStubsOptions(); WorkflowClientOptions clientOptions = profile.toWorkflowClientOptions(); try { // Create the workflow client using the loaded configuration WorkflowClient client = WorkflowClient.newInstance( WorkflowServiceStubs.newServiceStubs(serviceStubsOptions), clientOptions); // Test the connection by getting system info var systemInfo = client .getWorkflowServiceStubs() .blockingStub() .getSystemInfo( io.temporal.api.workflowservice.v1.GetSystemInfoRequest.getDefaultInstance()); logger.info("✅ Client connected successfully!"); logger.info(" Server version: {}", systemInfo.getServerVersion()); } catch (Exception e) { logger.error("❌ Failed to connect: {}", e.getMessage()); } } catch (Exception e) { logger.error("Failed to load configuration: {}", e.getMessage(), e); System.exit(1); } } } ``` **Code** Now, when instantiating a Temporal `client` in your Temporal Java SDK code, provide the API key with `WorkflowServiceStubsOptions` and the Namespace and Account ID in `WorkflowClient.newInstance`: ```java WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs( WorkflowServiceStubsOptions.newBuilder() .addApiKey( () -> ) .setTarget() .setEnableHttps(true) ... .build()); WorkflowClient client = WorkflowClient.newInstance( service, WorkflowClientOptions.newBuilder().setNamespace(.).build()); ``` To update the API key, update the `stubOptions`: ```java String myKey = ; WorkflowServiceStubsOptions stubOptions = WorkflowServiceStubsOptions.newBuilder() .addApiKey(() -> myKey) .build(); // Update by replacing, this must be done in a thread safe way myKey = "Bearer " + ; ``` To connect to Temporal Cloud using mTLS, you need to provide the mTLS CA certificate and mTLS private key. You can then use the `SimpleSslContextBuilder` to build an SSL context. Then use the `WorkflowServiceStubsOptions.Builder.setSslContext` method to set the SSL context. When you use a remote service, you don't use the `newLocalServiceStubs` convenience method. Instead, set your connection details as stub configuration options: ```java // Set the Service Stub options (SSL context and gRPC endpoint) WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions .newBuilder() .setSslContext(sslContext) .setTarget(gRPCEndpoint) .build(); // Create a stub that accesses a Temporal Service WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions); ``` Each Temporal Cloud service Client has four prerequisites. - The full Namespace Id from the [Cloud Namespace](https://cloud.temporal.io/namespaces) details page - The gRPC endpoint from the [Cloud Namespace](https://cloud.temporal.io/namespaces) details page - Your mTLS private key - Your mTLS x509 Certificate Retrieve these values before building your Client. The following sample generates an SSL context from the mTLS .pem and .key files. Along with the gRPC endpoint, this information configures a service stub for Temporal Cloud. Add the Namespace to your Client build options and initialize the new Client: ```java // Generate an SSL context InputStream clientCertInputStream = new FileInputStream(clientCertPath); InputStream clientKeyInputStream = new FileInputStream(clientKeyPath); SslContext sslContext = SimpleSslContextBuilder.forPKCS8(clientCertInputStream, clientKeyInputStream).build(); // Set the Service Stub options (SSL context and gRPC endpoint) WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions .newBuilder() .setSslContext(sslContext) .setTarget(gRPCEndpoint) .build(); // Create a stub that accesses a Temporal Service WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions); // Set the Client options WorkflowClientOptions clientOptions = WorkflowClientOptions .newBuilder() .setNamespace(namespace) .build(); // Initialize the Client WorkflowClient client = WorkflowClient.newInstance(serviceStub, clientOptions); ``` To rotate an mTLS client certificate without restarting your Worker, build the `SslContext` with gRPC's [`AdvancedTlsX509KeyManager`](https://grpc.github.io/grpc-java/javadoc/io/grpc/util/AdvancedTlsX509KeyManager.html) instead of `SimpleSslContextBuilder`. `updateIdentityCredentialsFromFile` schedules a periodic reread of the certificate and key files, so the same `SslContext` keeps serving fresh credentials for the life of the process: ```java String tlsCertPath = "/path/to/tls.crt"; String tlsKeyPath = "/path/to/tls.key"; // PKCS8 format // Reread the certificate and key files every 5 minutes. AdvancedTlsX509KeyManager keyManager = new AdvancedTlsX509KeyManager(); keyManager.updateIdentityCredentialsFromFile( new File(tlsKeyPath), new File(tlsCertPath), 5, TimeUnit.MINUTES, Executors.newSingleThreadScheduledExecutor()); SslContextBuilder sslContextBuilder = SslContextBuilder.forClient(); GrpcSslContexts.configure(sslContextBuilder); SslContext sslContext = sslContextBuilder.keyManager(keyManager).build(); WorkflowServiceStubsOptions stubsOptions = WorkflowServiceStubsOptions .newBuilder() .setSslContext(sslContext) .setTarget(gRPCEndpoint) .build(); WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions); ``` Rotate the certificate by overwriting the files at `tlsCertPath` and `tlsKeyPath`; `keyManager` picks up the new files on its next scheduled check, and the Worker keeps running against the same `client`/`serviceStub`. See this [reference implementation](https://github.com/temporal-sa/temporal-worker-cert-rotation) (written for the Go SDK, but the "What if I am not using the Go SDK?" section covers this Java approach) for a full walkthrough. ## Start a Workflow Execution [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from within another Workflow Execution, use either the [Child Workflow](/develop/java/workflows/child-workflows) or External Workflow APIs. See the [Customize Workflow Type](/develop/java/workflows/basics#workflow-type) section to see how to customize the name of the Workflow Type. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. Use `WorkflowStub` to start a Workflow Execution from within a Client, and `ExternalWorkflowStub` to start a different Workflow Execution from within a Workflow. See [`SignalwithStart`](/develop/java/workflows/message-passing#signal-with-start) to start a Workflow Execution to receive a Signal from within another Workflow. **Using `WorkflowStub`** `WorkflowStub` is a proxy generated by the `WorkflowClient`. Each time a new Workflow Execution is started, an instance of the Workflow implementation object is created. Then, one of the methods (depending on the Workflow Type of the instance) annotated with `@WorkflowMethod` can be invoked. As soon as this method returns, the Workflow Execution is considered to be complete. You can use a typed or untyped `WorkflowStub` in the client code. - Typed `WorkflowStub` are useful because they are type safe and allow you to invoke your Workflow methods such as `@WorkflowMethod`, `@QueryMethod`, and `@SignalMethod` directly. - An untyped `WorkflowStub` does not use the Workflow interface, and is not type safe. It is more flexible because it has methods from the `WorkflowStub` interface, such as `start`, `signalWithStart`, `getResults` (sync and async), `query`, `signal`, `cancel` and `terminate`. Note that the Temporal Java SDK also provides typed `WorkflowStub` versions for these methods. When using untyped `WorkflowStub`, we rely on the Workflow Type, Activity Type, Child Workflow Type, as well as Query and Signal names. For details, see [Temporal Client](#connect-to-development-service). A Workflow Execution can be started either synchronously or asynchronously. - Synchronous invocation starts a Workflow and then waits for its completion. If the process that started the Workflow crashes or stops waiting, the Workflow continues executing. Because Workflows are potentially long-running, and Client crashes happen, it is not very commonly found in production use. The following example is a type-safe approach for starting a Workflow Execution synchronously. ```java NotifyUserAccounts workflow = client.newWorkflowStub( NotifyUserAccounts.class, WorkflowOptions.newBuilder() .setWorkflowId("notifyAccounts") .setTaskQueue(taskQueue) .build() ); // start the Workflow and wait for a result. workflow.notify(new String[] { "Account1", "Account2", "Account3", "Account4", "Account5", "Account6", "Account7", "Account8", "Account9", "Account10"}); } // notify(String[] accountIds) is a Workflow method defined in the Workflow Definition. ``` - Asynchronous start initiates a Workflow Execution and immediately returns to the caller. This is the most common way to start Workflows in production code. The [`WorkflowClient`](https://github.com/temporalio/sdk-java/blob/main/temporal-sdk/src/main/java/io/temporal/client/WorkflowClient.java) provides some static methods, such as `start`, `execute`, and `signalWithStart`, that help with starting your Workflows asynchronously. The following examples show how to start Workflow Executions asynchronously, with either typed or untyped `WorkflowStub`. - **Typed WorkflowStub Example** ```java // create typed Workflow stub FileProcessingWorkflow workflow = client.newWorkflowStub(FileProcessingWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(taskQueue) .setWorkflowId(workflowId) .build()); // use WorkflowClient.execute to return future that contains Workflow result or failure, or // use WorkflowClient.start to return WorkflowId and RunId of the started Workflow). WorkflowClient.start(workflow::greetCustomer); ``` - **Untyped WorkflowStub Example** ```java WorkflowStub untyped = client.newUntypedWorkflowStub("FileProcessingWorkflow", WorkflowOptions.newBuilder() .setWorkflowId(workflowId) .setTaskQueue(taskQueue) .build()); // blocks until Workflow Execution has been started (not until it completes) untyped.start(argument); ``` You can call a Dynamic Workflow implementation using an untyped `WorkflowStub`. The following example shows how to call the Dynamic Workflow implementation in the Client code. ```java WorkflowClient client = WorkflowClient.newInstance(service); /** * Note that for this part of the client code, the dynamic Workflow implementation must * be known to the Worker at runtime in order to dispatch Workflow tasks, and may be defined * in the Worker definition as:*/ // worker.registerWorkflowImplementationTypes(DynamicGreetingWorkflowImpl.class); /* Create the Workflow stub to call the dynamic Workflow. * Note that the Workflow Type is not explicitly registered with the Worker.*/ WorkflowOptions workflowOptions = WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).setWorkflowId(WORKFLOW_ID).build(); WorkflowStub workflow = client.newUntypedWorkflowStub("DynamicWF", workflowOptions); ``` `DynamicWorkflow` can be used to invoke different Workflow Types. To check what type is running when your Dynamic Workflow `execute` method runs, use `getWorkflowType()` in the implementation code. ```java String type = Workflow.getInfo().getWorkflowType(); ``` See [Workflow Execution Result](#get-workflow-results) for details on how to get the results of the Workflow Execution. **Using `ExternalWorkflowStub`** Use `ExternalWorkflowStub` within a Workflow to invoke, and send Signals to, other Workflows by type. This helps particularly for executing Workflows written in other language SDKs, as shown in the following example. ```java @Override public String yourWFMethod(String name) { ExternalWorkflowStub callOtherWorkflow = Workflow.newUntypedExternalWorkflowStub("OtherWFId"); } ``` See the [Temporal Polyglot](https://github.com/tsurdilo/temporal-polyglot) code for examples of executing Workflows written in other language SDKs. **Recurring start** You can start a Workflow Execution on a regular schedule by using [`setCronSchedule`](/develop/java/workflows/schedules#cron-schedule) Workflow option in the Client code. ### How to set a Workflow's Task Queue In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue). For your code to execute, a Worker Process must be running. This process needs a Worker Entity that is polling the same Task Queue name. Set the Workflow Task Queue with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setTaskQueue`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `String` - Default: none ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") // Set the Task Queue .setTaskQueue(WorkerGreet.TASK_QUEUE) .build()); ``` ### How to set a Workflow Id Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity identifier, such as an order identifier or customer identifier. Set the Workflow Id with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setWorkflowId​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `String` - Default: none ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() // Set the Workflow Id .setWorkflowId("YourWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) .build()); ``` ### Java WorkflowOptions reference Create a [`newWorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in the Temporal Client code, call the instance of the Workflow, and set the Workflow options with the [`WorkflowOptions.Builder`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html) class. The following fields are available: | Option | Required | Type | | ------------------------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------- | | [`WorkflowId`](#workflowid) | No (but recommended) | String | | [`TaskQueue`](#taskqueue) | **Yes** | String | | [`WorkflowExecutionTimeout`](#workflowexecutiontimeout) | No | `Duration` | | [`WorkflowRunTimeout`](#workflowruntimeout) | No | `Duration` | | [`WorkflowTaskTimeout`](#workflowtasktimeout) | No | `Duration` | | [`WorkflowIdReusePolicy`](#workflowidreusepolicy) | No | `WorkflowIdReusePolicy` | | [`RetryOptions`](#retryoptions) | No | [`RetryOptions`](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.17.0/io/temporal/common/RetryOptions.html) | | [`CronSchedule`](#cronschedule) | No | String | | [`Memo`](#memo) | No | string | | [`SearchAttributes`](#searchattributes) | No | `Map` | #### WorkflowId Set the Workflow Id with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setWorkflowId​`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `String` - Default: none ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() // Set the Workflow Id .setWorkflowId("YourWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) .build()); ``` #### TaskQueue Set the Workflow Task Queue with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setTaskQueue`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `String` - Default: none ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") // Set the Task Queue .setTaskQueue(WorkerGreet.TASK_QUEUE) .build()); ``` #### WorkflowExecutionTimeout Set the [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setWorkflowExecutionTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `Duration` - Default: Unlimited ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Execution Timeout duration .setWorkflowExecutionTimeout(Duration.ofSeconds(10)) .build()); ``` #### WorkflowRunTimeout Set the Workflow Run Timeout with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setWorkflowRunTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `Duration` - Default: Same as [WorkflowExecutionTimeout](#workflowexecutiontimeout). ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Run Timeout duration .setWorkflowRunTimeout(Duration.ofSeconds(10)) .build()); ``` #### WorkflowTaskTimeout Set the Workflow Task Timeout with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setWorkflowTaskTimeout`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `Duration` - Default: 10 seconds. - Values: Maximum accepted value is 60 seconds. ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Task Timeout duration .setWorkflowTaskTimeout(Duration.ofSeconds(10)) .build()); ``` #### WorkflowIDReusePolicy - Type: `WorkflowIdReusePolicy` - Default: `AllowDuplicate` - Values: - `enums.AllowDuplicateFailedOnly`: The Workflow can start if the earlier Workflow Execution failed, Canceled, or Terminated. - `AllowDuplicate`: The Workflow can start regardless of the earlier Execution's closure status. - `RejectDuplicate`: The Workflow can not start if there is a earlier Run. ```java //create Workflow stub for GreetWorkflowInterface GreetWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("GreetWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Id Reuse Policy .setWorkflowIdReusePolicy( WorkflowIdReusePolicy.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE) .build()); ``` #### RetryOptions To set a Workflow Retry Options in the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance use [`WorkflowOptions.Builder.setWorkflowRetryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `RetryOptions` - Default: `Null` which means no retries will be attempted. ```java //create Workflow stub for GreetWorkflowInterface GreetWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("GreetWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Retry Options .setRetryOptions(RetryOptions.newBuilder() .build()); ``` #### CronSchedule A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made. Set the Cron Schedule with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setCronSchedule`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). Setting `setCronSchedule` changes the Workflow Execution into a Temporal Cron Job. The default timezone for a Cron is UTC. - Type: `String` - Default: None ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = YourWorker.yourclient.newWorkflowStub( YourWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") .setTaskQueue(YourWorker.TASK_QUEUE) // Set Cron Schedule .setCronSchedule("* * * * *") .build()); ``` For more details, see the [HelloCron Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java). #### Memo - Type: `String` - Default: None ```java //create Workflow stub for GreetWorkflowInterface GreetWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("GreetWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Memo. You can set additional non-indexed info via Memo .setMemo(ImmutableMap.of( "memoKey", "memoValue" )) .build()); ``` #### SearchAttributes Search Attributes are additional indexed information attributed to Workflow and used for search and visibility. These can be used in a query of List/Scan/Count Workflow APIs. The key and its value type must be registered on Temporal server side. - Type: `Map` - Default: None ```java private static void parentWorkflow() { ChildWorkflowOptions childworkflowOptions = ChildWorkflowOptions.newBuilder() // Set Search Attributes .setSearchAttributes(ImmutableMap.of("MySearchAttributeNAme", "value")) .build(); ``` The following Java types are supported: - String - Long, Integer, Short, Byte - Boolean - Double - OffsetDateTime - Collection of the types in this list. ### How to get the result of a Workflow Execution in Java If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. A synchronous Workflow Execution blocks your client thread until the Workflow Execution completes (or fails) and get the results (or error in case of failure). The following example is a type-safe approach for getting the results of a synchronous Workflow Execution. ```java FileProcessingWorkflow workflow = client.newWorkflowStub( FileProcessingWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId(workflowId) .setTaskQueue(taskQueue) .build(); // start sync and wait for results (or failure) String result = workflow.processfile(new Argument()); ``` An asynchronous Workflow Execution immediately returns a value to the caller. The following examples show how to get the results of a Workflow Execution through typed and untyped `WorkflowStub`. - **Typed WorkflowStub Example** ```java // create typed Workflow stub FileProcessingWorkflow workflow = client.newWorkflowStub(FileProcessingWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(taskQueue) .setWorkflowId(workflowId) .build()); // use WorkflowClient.execute (if your Workflow takes in arguments) or WorkflowClient.start (for zero arguments) WorkflowClient.start(workflow::greetCustomer); ``` - **Untyped WorkflowStub Example** ```java WorkflowStub untyped = client.newUntypedWorkflowStub("FileProcessingWorkflow", WorkflowOptions.newBuilder() .setWorkflowId(workflowId) .setTaskQueue(taskQueue) .build()); // blocks until Workflow Execution has been started (not until it completes) untyped.start(argument); ``` If you need to wait for a Workflow Execution to complete after an asynchronous start, the most straightforward way is to call the blocking Workflow instance again. Note that if `WorkflowOptions.WorkflowIdReusePolicy` is not set to `AllowDuplicate`, then instead of throwing `DuplicateWorkflowException`, it reconnects to an existing Workflow and waits for its completion. The following example shows how to do this from a different process than the one that started the Workflow Execution. ```java YourWorkflow workflow = client.newWorkflowStub(YourWorkflow.class, workflowId); // Returns the result after waiting for the Workflow to complete. String result = workflow.yourMethod(); ``` Another way to connect to an existing Workflow and wait for its completion from another process, is to use `UntypedWorkflowStub`. For example: ```java WorkflowStub workflowStub = client.newUntypedWorkflowStub(workflowType, workflowOptions); // Returns the result after waiting for the Workflow to complete. String result = untyped.getResult(String.class); ``` **Get last (successful) completion result** For a Temporal Cron Job, get the result of previous successful runs using `GetLastCompletionResult()`. The method returns `null` if there is no previous completion. The following example shows how to implement this in a Workflow. ```java public String cronWorkflow() { String lastProcessedFileName = Workflow.getLastCompletionResult(String.class); // Process work starting from the lastProcessedFileName. // Business logic implementation goes here. // Updates lastProcessedFileName to the new value. return lastProcessedFileName; } ``` Note that this works even if one of the Cron schedule runs failed. The next schedule will still get the last successful result if it ever successfully completed at least once. For example, for a daily cron Workflow, if the run succeeds on the first day and fails on the second day, then the third day run will get the result from first day's run using these APIs. --- # Integrations - Java SDK Source: https://docs.temporal.io/develop/java/integrations > This section covers integrations with the Java SDK The following integrations are available for the Temporal Java SDK. These integrations are built on the Temporal Java SDK's [Plugin system](/develop/plugins-guide), which you can also use to build your own integrations. - [Parseable](https://github.com/parseablehq/temporal-plugin-java/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Java · Agent observability)_ - [Spring AI](/develop/java/integrations/spring-ai) — Build AI-powered Java applications with durable Spring AI tool calls. _(Java · Agent framework)_ - [Spring Boot](/develop/java/integrations/spring-boot-integration) — Use Temporal natively in Spring Boot with auto-configuration and dependency injection. _(Java · Framework)_ --- # Spring AI integration - Java SDK Source: https://docs.temporal.io/develop/java/integrations/spring-ai > Build durable AI agents in Java with the Temporal Spring AI integration. [Spring AI](https://docs.spring.io/spring-ai/reference/) is an agent framework for Java applications — chat clients, tool calling, vector stores, embeddings, and MCP servers, all wired through Spring Boot. The [Temporal Spring AI integration](https://central.sonatype.com/artifact/io.temporal/temporal-spring-ai) makes Spring AI agents durable: model calls run through Temporal Activities recorded in Event history, and tools are dispatched per their type so each kind lands in the right place in Workflow execution — Activity stubs and Nexus stubs as durable operations, `@SideEffectTool` classes wrapped in `Workflow.sideEffect`, and plain tools running directly in Workflow code. Agents retry on failure and replay deterministically without changing how you write Spring AI code. The integration is built on the Temporal Java SDK's [Plugin system](/develop/plugins-guide) and is distributed as the `io.temporal:temporal-spring-ai` module alongside the existing [Spring Boot integration](/develop/java/integrations/spring-boot-integration). > **Public Preview** ## Prerequisites The integration requires all of the following on your application's classpath. The plugin won't auto-configure if any of these are missing or below the listed minimum: | Dependency | Minimum version | | ----------------- | --------------- | | Java | 17 | | Spring Boot | 3.x | | Spring AI | 1.1.0 | | Temporal Java SDK | 1.35.0 | You also need the [`temporal-spring-boot-starter`](/develop/java/integrations/spring-boot-integration) and a Spring AI model starter (for example, `spring-ai-starter-model-openai`) — `temporal-spring-ai` does not pull in a model provider on its own. ## Add the dependency Add `temporal-spring-ai` alongside `temporal-spring-boot-starter` and a Spring AI model starter (for example, `spring-ai-starter-model-openai`). **[Apache Maven](https://maven.apache.org/):** ```xml io.temporal temporal-spring-ai ${temporal-sdk.version} ``` **[Gradle Groovy DSL](https://gradle.org/):** ```groovy implementation "io.temporal:temporal-spring-ai:${temporalSdkVersion}" ``` When `temporal-spring-ai` is on the classpath, the `SpringAiPlugin` auto-registers `ChatModelActivity` with all Temporal Workers created by the Spring Boot integration. Optional Activities are auto-configured when their dependencies are present: | Feature | Dependency | Registered Activity | | ------------ | --------------- | ------------------------ | | Vector store | `spring-ai-rag` | `VectorStoreActivity` | | Embeddings | `spring-ai-rag` | `EmbeddingModelActivity` | | MCP | `spring-ai-mcp` | `McpClientActivity` | ## Call a chat model from a Workflow Use `ActivityChatModel` as a Spring AI `ChatModel` inside a Workflow. Every call goes through a Temporal Activity, so model responses are durable and retried per your Activity options. Wrap `ActivityChatModel` in a `TemporalChatClient` to build prompts and register tools: [springai/basic/src/main/java/io/temporal/samples/springai/chat/ChatWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/ChatWorkflowImpl.java) ```java @WorkflowInit public ChatWorkflowImpl(String systemPrompt) { // Build an activity-backed chat model. The factory creates the activity stub // internally and registers per-call Summaries on the Temporal UI. ActivityChatModel activityChatModel = ActivityChatModel.forDefault(); // Create an activity stub for weather tools - these execute as durable activities WeatherActivity weatherTool = Workflow.newActivityStub( WeatherActivity.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(3).build()) .build()); // Create deterministic tools - these execute directly in the workflow StringTools stringTools = new StringTools(); // Create side-effect tools - these are wrapped in Workflow.sideEffect() // The result is recorded in history, making replay deterministic TimestampTools timestampTools = new TimestampTools(); // Create chat memory - uses in-memory storage that gets rebuilt on replay ChatMemory chatMemory = MessageWindowChatMemory.builder() .chatMemoryRepository(new InMemoryChatMemoryRepository()) .maxMessages(20) .build(); // Build a TemporalChatClient with tools and memory // - Activity stubs (weatherTool) become durable AI tools // - plain workflow tool classes (stringTools) execute directly in workflow // - @SideEffectTool classes (timestampTools) are wrapped in sideEffect() // - PromptChatMemoryAdvisor maintains conversation history this.chatClient = TemporalChatClient.builder(activityChatModel) .defaultSystem(systemPrompt) .defaultTools(weatherTool, stringTools, timestampTools) .defaultAdvisors(PromptChatMemoryAdvisor.builder(chatMemory).build()) .build(); } ``` `ActivityChatModel.forDefault()` resolves to the default Spring AI `ChatModel` bean. To target a specific model in a multi-model application, pass its bean name to `ActivityChatModel.forModel("openai")`. > **📝 Note:** > > Streaming responses are not currently supported. ## Register tools In Spring AI, [tools](https://docs.spring.io/spring-ai/reference/api/tools.html) are methods the model can choose to call to fetch data or take action — you make them available to a chat client by registering them, typically through `ChatClient.defaultTools(...)` or per-prompt `tools(...)`. The chat client advertises the methods to the model, the model decides which (if any) to call, and the framework runs the chosen method and feeds the result back into the conversation. The Temporal integration extends this by inspecting the type of each tool you register and dispatching it to the appropriate Temporal primitive, so you can mix durable and in-Workflow tools in the same chat client. The integration handles Temporal determinism for you when the tool is durable, and gives you control when it isn't. ### Activity stubs An interface annotated with both `@ActivityInterface` and Spring AI `@Tool` methods is auto-detected and executed as a Temporal Activity. Use this for external calls that need retries and timeouts. [springai/basic/src/main/java/io/temporal/samples/springai/chat/WeatherActivity.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/WeatherActivity.java) ```java @ActivityInterface public interface WeatherActivity { /** * Gets the current weather for a city. * *

The {@code @Tool} annotation makes this method available to the AI model, while the * {@code @ActivityInterface} ensures it executes as a Temporal activity. * * @param city the name of the city * @return a description of the current weather */ @Tool( description = "Get the current weather for a city. Returns temperature, conditions, and humidity.") @ActivityMethod String getWeather( @ToolParam(description = "The name of the city (e.g., 'Seattle', 'New York')") String city); /** * Gets the weather forecast for a city. * * @param city the name of the city * @param days the number of days to forecast (1-7) * @return the weather forecast */ @Tool(description = "Get the weather forecast for a city for the specified number of days.") @ActivityMethod String getForecast( @ToolParam(description = "The name of the city") String city, @ToolParam(description = "Number of days to forecast (1-7)") int days); } ``` ### Nexus service stubs Nexus service stubs with `@Tool` methods are auto-detected and invoked as [Nexus operations](/develop/java/nexus), enabling cross-Namespace tool calls. ### `@SideEffectTool` Classes annotated with `@SideEffectTool` have each `@Tool` method wrapped in `Workflow.sideEffect()`. The result is recorded in history on first execution and replayed from history afterward. Use this for cheap, non-deterministic operations such as timestamps or UUIDs. [springai/basic/src/main/java/io/temporal/samples/springai/chat/TimestampTools.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/TimestampTools.java) ```java @SideEffectTool public class TimestampTools { private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z").withZone(ZoneId.systemDefault()); /** * Gets the current date and time. * *

This is non-deterministic (returns different values each time), but wrapped in sideEffect() * it becomes safe for workflow replay. * * @return the current date and time as a formatted string */ @Tool(description = "Get the current date and time") public String getCurrentDateTime() { return FORMATTER.format(Instant.now()); } /** * Gets the current Unix timestamp in milliseconds. * * @return the current time in milliseconds since epoch */ @Tool(description = "Get the current Unix timestamp in milliseconds") public long getCurrentTimestamp() { return System.currentTimeMillis(); } /** * Generates a random UUID. * * @return a new random UUID string */ @Tool(description = "Generate a random UUID") public String generateUuid() { return UUID.randomUUID().toString(); } /** * Gets the current date and time in a specific timezone. * * @param timezone the timezone ID (e.g., "America/New_York", "UTC", "Europe/London") * @return the current date and time in the specified timezone */ @Tool(description = "Get the current date and time in a specific timezone") public String getDateTimeInTimezone( @ToolParam(description = "Timezone ID (e.g., 'America/New_York', 'UTC', 'Europe/London')") String timezone) { try { ZoneId zoneId = ZoneId.of(timezone); DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z").withZone(zoneId); return formatter.format(Instant.now()); } catch (Exception e) { return "Invalid timezone: " + timezone + ". Use formats like 'America/New_York' or 'UTC'."; } } } ``` ### Plain tools Any class with `@Tool` methods that isn't an Activity stub, Nexus stub, or `@SideEffectTool` runs directly on the Workflow thread. Use this for inherently deterministic tools (such as updating in-memory agent state), or for orchestration of durable primitives as you need, for example calling multiple Activities, child Workflows, wait conditions, or other Temporal durable primitives. [springai/basic/src/main/java/io/temporal/samples/springai/chat/StringTools.java](https://github.com/temporalio/samples-java/blob/main/springai/basic/src/main/java/io/temporal/samples/springai/chat/StringTools.java) ```java public class StringTools { @Tool(description = "Reverse a string, returning the characters in opposite order") public String reverse(@ToolParam(description = "The string to reverse") String input) { if (input == null) { return null; } return new StringBuilder(input).reverse().toString(); } @Tool(description = "Count the number of words in a text") public int countWords(@ToolParam(description = "The text to count words in") String text) { if (text == null || text.isBlank()) { return 0; } return text.trim().split("\\s+").length; } @Tool(description = "Convert text to all uppercase letters") public String toUpperCase(@ToolParam(description = "The text to convert") String text) { if (text == null) { return null; } return text.toUpperCase(java.util.Locale.ROOT); } @Tool(description = "Convert text to all lowercase letters") public String toLowerCase(@ToolParam(description = "The text to convert") String text) { if (text == null) { return null; } return text.toLowerCase(java.util.Locale.ROOT); } @Tool(description = "Check if a string is a palindrome (reads the same forwards and backwards)") public boolean isPalindrome(@ToolParam(description = "The text to check") String text) { if (text == null) { return false; } String normalized = text.toLowerCase(java.util.Locale.ROOT).replaceAll("\\s+", ""); String reversed = new StringBuilder(normalized).reverse().toString(); return normalized.equals(reversed); } } ``` ## Activity options and retry behavior `ActivityChatModel.forDefault()` and `forModel(name)` build the chat Activity stub with sensible defaults: a 2-minute start-to-close timeout, 3 attempts, and `org.springframework.ai.retry.NonTransientAiException` and `java.lang.IllegalArgumentException` classified as non-retryable so a bad API key or invalid prompt fails fast. Pass an `ActivityOptions` directly when you need finer control — a specific Task Queue, [heartbeats](/develop/java/activities/execution#heartbeattimeout), [priority](/develop/task-queue-priority-fairness), or a custom `RetryOptions`: ```java ActivityChatModel chatModel = ActivityChatModel.forDefault( ActivityOptions.newBuilder(ActivityChatModel.defaultActivityOptions()) .setTaskQueue("chat-heavy") .build()); ``` For configuration-driven per-model overrides, declare a `ChatModelActivityOptions` bean. The plugin consults it whenever `forDefault()` or `forModel(name)` runs in a Workflow. Use the special key `ChatModelTypes.DEFAULT_MODEL_NAME` (the literal `"default"`) as a global catch-all that applies to any model not explicitly listed — including models contributed by third-party starters: [springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/ChatModelConfig.java](https://github.com/temporalio/samples-java/blob/main/springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/ChatModelConfig.java) ```java @Bean public ChatModelActivityOptions chatModelActivityOptions() { return new ChatModelActivityOptions( Map.of( "anthropicChatModel", ActivityOptions.newBuilder(ActivityChatModel.defaultActivityOptions()) .setStartToCloseTimeout(Duration.ofMinutes(5)) .setScheduleToCloseTimeout(Duration.ofMinutes(15)) .build())); } ``` Keys that neither match a registered `ChatModel` bean nor equal `"default"` cause plugin construction to fail, so a typo surfaces at startup rather than at first call. `ActivityMcpClient.create()` and `create(ActivityOptions)` work the same way for MCP tool calls, with a 30-second default timeout. ## Provider-specific chat options Provider-specific `ChatOptions` subclasses — for example, `AnthropicChatOptions` to enable extended thinking, or `OpenAiChatOptions` to set `reasoning_effort` — pass through the Activity boundary unchanged. Attach them via `ChatClient.defaultOptions(...)` and the plugin re-applies them on the Activity side before calling the underlying model: [springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/MultiModelWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/springai/multimodel/src/main/java/io/temporal/samples/springai/multimodel/MultiModelWorkflowImpl.java) ```java AnthropicChatOptions thinkingOptions = AnthropicChatOptions.builder() .thinking(AnthropicApi.ThinkingType.ENABLED, 1024) .temperature(1.0) .maxTokens(4096) .build(); chatClients.put( "think", TemporalChatClient.builder(anthropicModel) .defaultSystem( "You are a helpful assistant powered by Anthropic with extended thinking. " + "Use the thinking budget to reason carefully, then give a crisp answer " + "that reflects the reasoning you did.") .defaultOptions(thinkingOptions) .build()); ``` The pass-through relies on the `ChatOptions` subclass overriding `copy()` to return its own type — every provider class shipped with Spring AI does. ## Media in messages Prefer URI-based media when attaching images, audio, or other binary content to chat messages. Raw `byte[]` media gets serialized into every chat Activity's input and result payload, which end up inside Temporal Event history events. Server-side history events have a fixed 2 MiB size limit; to leave headroom for messages, tool definitions, and options, the plugin enforces a **1 MiB default cap** on inline bytes and fails fast with a non-retryable `ApplicationFailure` pointing at the URI alternative. ```java // Preferred — only the URL crosses the Activity boundary. Media image = new Media(MimeTypeUtils.IMAGE_PNG, URI.create("https://cdn.example.com/pic.png")); ``` Override the cap by setting the system property `io.temporal.springai.maxMediaBytes` before your worker starts (positive integer; `0` disables the check). For anything larger than a small thumbnail, route the bytes to a binary store from an Activity and pass only the URL across the conversation. ## Use vector stores, embeddings, and MCP When the corresponding Spring AI modules are on the classpath, the integration registers Activities for vector stores, embeddings, and MCP tool calls. Inject the matching Spring AI types into your Activities or Workflows and use them as you would in any Spring AI application — each operation is executed through a Temporal Activity. You can also register these plugins explicitly, without relying on auto-configuration: ```java new VectorStorePlugin(vectorStore); new EmbeddingModelPlugin(embeddingModel); new McpPlugin(); ``` `ActivityMcpClient` wraps a Spring AI MCP client so that remote MCP tool calls become durable Activity executions. ## Resources - [`temporal-spring-ai` README](https://github.com/temporalio/sdk-java/blob/main/contrib/temporal-spring-ai/README.md) — full reference for the module - [Spring Boot integration](/develop/java/integrations/spring-boot-integration) — required companion module - [Plugin system](/develop/plugins-guide) — how integrations are registered with Workers and Clients --- # Spring Boot integration - Java SDK Source: https://docs.temporal.io/develop/java/integrations/spring-boot-integration > Learn about the Temporal Spring Boot integration This guide introduces the [Temporal Spring Boot](https://central.sonatype.com/artifact/io.temporal/temporal-spring-boot-starter?smo=true) integration. The Temporal Spring Boot integration is the easiest way to get started using the Temporal Java SDK if you are a current [Spring](https://spring.io/) user. This section includes the following topics: - [Setup Dependency](#setup-dependency) - [Connect to your Temporal Service](#connect) - [Configure Workers](#configure-workers) - [Customize Options](#customize-options) - [Interceptors](#interceptors) - [Integrations](#integrations) - [Testing](#testing) ## Setup Dependency To start using the Temporal Spring Boot integration, you need to add [`io.temporal:temporal-spring-boot-starter`](https://search.maven.org/artifact/io.temporal/temporal-spring-boot-starter) as a dependency to your Spring project: > **📝 Note:** > Temporal's Spring Boot integration currently supports Spring Boot 2.x, 3.x, and 4.x **[Apache Maven](https://maven.apache.org/):** ```maven io.temporal temporal-spring-boot-starter 1.31.0 ``` **[Gradle Groovy DSL](https://gradle.org/):** ```groovy implementation ("io.temporal:temporal-spring-boot-starter:1.31.0") ``` ## Connect See the [Temporal Client documentation](/develop/java/client/temporal-client) for more information about connecting to a Temporal Service. To create an autoconfigured `WorkflowClient`, you need to specify some connection details in your `application.yml` file, as described in the next section. ### Connect to your local Temporal Service ```yaml spring.temporal: connection: target: local # you can specify a host:port here for a remote connection ``` This is enough to autowire a `WorkflowClient` in your Spring Boot application: ```java @SpringBootApplication class App { @Autowire private WorkflowClient workflowClient; } ``` ### Connect to a custom Namespace You can also connect to a custom Namespace by specifying the `spring.temporal.namespace` property. ```yaml spring.temporal: connection: target: local # you can specify a host:port here for a remote connection namespace: # you can specify a custom namespace that you are using ``` ## Connect to Temporal Cloud You can also connect to Temporal Cloud, using either an API key or mTLS for authentication. See the [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud) section for more information about connecting to Temporal Cloud. ### Using an API key ```yaml spring.temporal: connection: target: apiKey: namespace: ``` ### Using mTLS ``` spring.temporal: connection: mtls: target: key-file: /path/to/key.key cert-chain-file: /path/to/cert.pem # If you use PKCS12 (.pkcs12, .pfx or .p12), you don't need to set it because the certificates chain is bundled into the key file namespace: ``` ## Configure Workers Temporal's Spring Boot integration supports two configuration methods for Workers: explicit configuration and auto-discovery. ### Explicit configuration ```yaml spring.temporal: workers: - task-queue: your-task-queue-name name: your-worker-name # unique name of the Worker. If not specified, Task Queue is used as the Worker name. workflow-classes: - your.package.YourWorkflowImpl activity-beans: - activity-bean-name1 ``` ### Auto Discovery Auto Discovery allows you to skip specifying Workflow classes, Activity beans, and Nexus Service beans explicitly in the config by referencing Worker Task Queue names or Worker Names on Workflow, Activity implementations, and Nexus Service implementations. Auto-discovery is applied after and on top of an explicit configuration. ``` spring.temporal: workers-auto-discovery: packages: - your.package # enumerate all the packages that contain your workflow implementations. ``` #### What is auto-discovered: - Workflow implementation classes annotated with `io.temporal.spring.boot.WorkflowImpl` - Activity beans present Spring context whose implementations are annotated with `io.temporal.spring.boot.ActivityImpl` - Nexus Service beans present in Spring context whose implementations are annotated with `io.temporal.spring.boot.NexusServiceImpl` - Workers if a Task Queue is referenced by the annotations but not explicitly configured. Default configuration will be used. > **📝 Note:** > `io.temporal.spring.boot.ActivityImpl` and `io.temporal.spring.boot.NexusServiceImpl` should be applied to beans, one way to do this is to annotate your Activity implementation class with `@Component` ``` @Component @ActivityImpl(workers = "myWorker") public class MyActivityImpl implements MyActivity { @Override public String execute(String input) { return input; } } ``` > **📝 Note:** > Auto-discovered Workflow implementation classes, Activity beans, and Nexus Service beans will be registered with the configured Workers if not already registered. ## Interceptors To enable Interceptors, you can create beans by implementing the `io.temporal.common.interceptors.WorkflowClientInterceptor`, `io.temporal.common.interceptors.ScheduleClientInterceptor`, or `io.temporal.common.interceptors.WorkerInterceptor` interface. Interceptors will be registered in the order specified by the `@Order` annotation. ## Integrations The Temporal Spring Boot integration also has built in support for various tools in the Spring ecosystem, such as metrics and tracing. ### Metrics You can set up built-in Spring Boot metrics using [Spring Boot Actuator](https://docs.spring.io/spring-boot/reference/actuator/metrics.html). The Temporal Spring Boot integration will pick up the `MeterRegistry` bean and use it to report Temporal metrics. Alternatively, you can define a custom `io.micrometer.core.instrument.MeterRegistry` bean in the application context. ### Tracing You can set up [Spring Cloud Sleuth](https://spring.io/projects/spring-cloud-sleuth) with an OpenTelemetry export. The Temporal Spring Boot integration will pick up the OpenTelemetry bean configured by `spring-cloud-sleuth-otel-autoconfigure` and use it for Temporal traces. Alternatively, you can define a custom `io.opentelemetry.api.OpenTelemetry` for OpenTelemetry or `io.opentracing.Tracer` for an OpenTracing bean in the application context. ## Customization of Options To programmatically customize the various options that are created by the Spring Boot integration, you can create beans that implement the `io.temporal.spring.boot.TemporalOptionsCustomizer` interface. This will be called after the options in your properties files are applied. Where `OptionsType` may be one of: * `WorkflowServiceStubsOptions.Builder` * `WorkflowClientOptions.Builder` * `WorkerFactoryOptions.Builder` * `WorkerOptions.Builder` * `WorkflowImplementationOptions.Builder` * `TestEnvironmentOptions.Builder` `io.temporal.spring.boot.WorkerOptionsCustomizer` may be used instead of `TemporalOptionsCustomizer` if `WorkerOptions` needs to be customized on the Task Queue or Worker name. `io.temporal.spring.boot.WorkflowImplementationOptionsCustomizer` may be used instead of `TemporalOptionsCustomizer` if `WorkflowImplementationOptions` needs to be customized on Workflow Type. ## Testing The Temporal Spring Boot integration also has easy support for testing your Temporal code. Add the following to your `application.yml` to reconfigure the client to work through `io.temporal.testing.TestWorkflowEnvironment` that uses in-memory Java Test Server: ``` spring.temporal: test-server: enabled: true ``` When `spring.temporal.test-server.enabled:true` is added, the `spring.temporal.connection` section is ignored. This allows wiring the `TestWorkflowEnvironment` bean in your unit tests: ``` @SpringBootTest(classes = Test.Configuration.class) @TestInstance(TestInstance.Lifecycle.PER_CLASS) public class Test { @Autowired ConfigurableApplicationContext applicationContext; @Autowired TestWorkflowEnvironment testWorkflowEnvironment; @Autowired WorkflowClient workflowClient; @BeforeEach void setUp() { applicationContext.start(); } @Test @Timeout(value = 10) public void test() { # ... } @ComponentScan # to discover Activity beans annotated with @Component public static class Configuration {} } ``` See the [Java SDK test frameworks documentation](/develop/java/best-practices/testing-suite#test-frameworks) for more information about testing. --- # Nexus - Java SDK Source: https://docs.temporal.io/develop/java/nexus > This section covers Nexus with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Temporal Nexus - [Quickstart](/develop/java/nexus/quickstart) - [Feature guide](/develop/java/nexus/feature-guide) - [Standalone Operations](/develop/java/nexus/standalone-operations) - [Nexus sync tutorial](https://learn.temporal.io/tutorials/nexus/nexus-sync-tutorial/) --- # Temporal Nexus - Java SDK feature guide Source: https://docs.temporal.io/develop/java/nexus/feature-guide > Use Temporal Nexus within the Java SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Use [Temporal Nexus](/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. > **💡 Tip:** > > New to Nexus? Start with the [Nexus Java Quickstart](/develop/java/nexus/quickstart). > This page shows how to do the following: - [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server) - [Create caller and handler Namespaces](#create-caller-handler-namespaces) - [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint) - [Define the Nexus Service contract](#define-nexus-service-contract) - [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers) - [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service) - [Make Nexus calls across Namespaces with a development Server](#nexus-calls-across-namespaces-dev-server) - [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud) > **📝 Note:** > > This documentation uses source code derived from the > [Java Nexus sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexus). > ## Run the Temporal Development Server with Nexus enabled Prerequisites: - [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/java/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) (v1.3.0 or higher recommended) - [Install the latest Temporal Java SDK](https://learn.temporal.io/getting_started/java/dev_environment/#add-temporal-java-sdk-dependencies) (v1.28.0 or higher recommended) The first step in working with Temporal Nexus involves starting a Temporal server with Nexus enabled. ``` temporal server start-dev ``` This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`. ## Create caller and handler Namespaces Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. ``` temporal operator namespace create --namespace my-target-namespace temporal operator namespace create --namespace my-caller-namespace ``` `my-target-namespace` will contain the Nexus Operation handler, and we will use a Workflow in `my-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. ## Create a Nexus Endpoint to route requests from caller to handler After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. ``` temporal operator nexus endpoint create \ --name my-nexus-endpoint-name \ --target-namespace my-target-namespace \ --target-task-queue my-handler-task-queue ``` You can also use the Web UI to create the Namespaces and Nexus endpoint. ## Define the Nexus Service contract Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](/dataconversion). The default data converter encodes payloads in the following order: Null, Byte array, Protobuf JSON, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, Protobuf and JSON are common choices. This example uses Java classes serialized into JSON. [core/src/main/java/io/temporal/samples/nexus/service/NexusService.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/service/NexusService.java) ```java @Service public interface SampleNexusService { enum Language { EN, FR, DE, ES, TR } class HelloInput { private final String name; private final Language language; @JsonCreator(mode = JsonCreator.Mode.PROPERTIES) public HelloInput( @JsonProperty("name") String name, @JsonProperty("language") Language language) { this.name = name; this.language = language; } @JsonProperty("name") public String getName() { return name; } @JsonProperty("language") public Language getLanguage() { return language; } } class HelloOutput { private final String message; @JsonCreator(mode = JsonCreator.Mode.PROPERTIES) public HelloOutput(@JsonProperty("message") String message) { this.message = message; } @JsonProperty("message") public String getMessage() { return message; } } class EchoInput { private final String message; @JsonCreator(mode = JsonCreator.Mode.PROPERTIES) public EchoInput(@JsonProperty("message") String message) { this.message = message; } @JsonProperty("message") public String getMessage() { return message; } } class EchoOutput { private final String message; @JsonCreator(mode = JsonCreator.Mode.PROPERTIES) public EchoOutput(@JsonProperty("message") String message) { this.message = message; } @JsonProperty("message") public String getMessage() { return message; } } @Operation HelloOutput hello(HelloInput input); @Operation EchoOutput echo(EchoInput input); } ``` ## Develop a Nexus Service and Operation handlers Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/cloud/limits#nexus-operation-request-timeout). Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database. Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint. The `io.temporal.nexus.*` packages have utilities to help create Nexus Operations: - `Nexus.getOperationContext().getWorkflowClient()` \- Get the Temporal Client that the Worker was initialized with for synchronous handlers backed by Temporal primitives such as Signals and Queries - `WorkflowRunOperation.fromWorkflowMethod` \- Run a Workflow as an asynchronous Nexus Operation This example starts with a sync Operation handler example using the `OperationHandler.sync` method, and then shows how to create an async Operation handler that uses `WorkflowRunOperation.fromWorkflowMethod` to start a handler Workflow from a Nexus Operation. ### Develop a Synchronous Nexus Operation handler The `OperationHandler.sync` method is for exposing simple RPC handlers. Use `Nexus.getOperationContext().getWorkflowClient(ctx)` to get the Temporal Client for signaling, querying, and listing Workflows. Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking). [core/src/main/java/io/temporal/samples/nexus/handler/NexusServiceImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/handler/NexusServiceImpl.java) ```java // To create a service implementation, annotate the class with @ServiceImpl and provide the // interface that the service implements. The service implementation class should have methods that // return OperationHandler that correspond to the operations defined in the service interface. @ServiceImpl(service = SampleNexusService.class) public class SampleNexusServiceImpl { @OperationImpl public OperationHandler echo() { // OperationHandler.sync is a meant for exposing simple RPC handlers. return OperationHandler.sync( // The method is for making arbitrary short calls to other services or databases, or // perform simple computations such as this one. Users can also access a workflow client by // calling // Nexus.getOperationContext().getWorkflowClient(ctx) to make arbitrary calls such as // signaling, querying, or listing workflows. (ctx, details, input) -> new SampleNexusService.EchoOutput(input.getMessage())); } // ... } ``` ### Use the Temporal Client for Signals, Queries, and Updates A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](/cloud/limits#nexus-operation-request-timeout). Updates should be short-lived to stay within this deadline. The [nexus_messaging](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging) sample shows how to create a Nexus Service that uses synchronous operations to send Updates and Queries. Use the Nexus library, as shown below, to get the Client that the Worker was initialized with. In this example, the Workflow Id is derived from the client Id, with the "getWorkflowId" method. This converts a given client Id (in this case, the client is passing in a user Id) to generate a Workflow Id from it. This way the client only needs the identifier it cares about. [nexusmessaging/callerpattern/handler/NexusGreetingServiceImpl.java](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexusmessaging/callerpattern/handler/NexusGreetingServiceImpl.java) ```java static final String WORKFLOW_ID_PREFIX = "GreetingWorkflow_for_"; public static String getWorkflowId(String userId) { return WORKFLOW_ID_PREFIX + userId; } private GreetingWorkflow getWorkflowStub(String userId) { return Nexus.getOperationContext() .getWorkflowClient() .newWorkflowStub(GreetingWorkflow.class, getWorkflowId(userId)); } ... ``` There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging/callerpattern/) and [on-demand pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusmessaging/ondemandpattern/). The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it. ### Develop an Asynchronous Nexus Operation handler to start a Workflow Use the `WorkflowRunOperation.fromWorkflowMethod` method, which is the easiest way to expose a Workflow as an operation. [core/src/main/java/io/temporal/samples/nexus/handler/NexusServiceImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/handler/NexusServiceImpl.java) ```java // To create a service implementation, annotate the class with @ServiceImpl and provide the // interface that the service implements. The service implementation class should have methods that // return OperationHandler that correspond to the operations defined in the service interface. @ServiceImpl(service = SampleNexusService.class) public class SampleNexusServiceImpl { // ... @OperationImpl public OperationHandler hello() { // Use the WorkflowRunOperation.fromWorkflowMethod constructor, which is the easiest // way to expose a workflow as an operation. To expose a workflow with a different input // parameters then the operation or from an untyped stub, use the // WorkflowRunOperation.fromWorkflowHandler constructor and the appropriate constructor method // on WorkflowHandle. return WorkflowRunOperation.fromWorkflowMethod( (ctx, details, input) -> Nexus.getOperationContext() .getWorkflowClient() .newWorkflowStub( HelloHandlerWorkflow.class, // Workflow IDs should typically be business meaningful IDs and are used to // dedupe workflow starts. // For this example, we're using the request ID allocated by Temporal when // the // caller workflow schedules // the operation, this ID is guaranteed to be stable across retries of this // operation. // // Task queue defaults to the task queue this operation is handled on. WorkflowOptions.newBuilder().setWorkflowId(details.getRequestId()).build()) ``` Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. In general, the ID should be passed in the Operation input as part of the Nexus Service contract. > **💡 Tip:** > RESOURCES > > [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a > Conflict-Policy of Use-Existing. > #### Map a Nexus Operation input to multiple Workflow arguments A Nexus Operation can only take one input parameter. If you want a Nexus Operation to start a Workflow that takes multiple arguments use the `WorkflowRunOperation.fromWorkflowHandle` method. [core/src/main/java/io/temporal/samples/nexusmultipleargs/handler/NexusServiceImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexusmultipleargs/handler/NexusServiceImpl.java) ```java // To create a service implementation, annotate the class with @ServiceImpl and provide the // interface that the service implements. The service implementation class should have methods that // return OperationHandler that correspond to the operations defined in the service interface. @ServiceImpl(service = SampleNexusService.class) public class SampleNexusServiceImpl { @OperationImpl public OperationHandler echo() { // OperationHandler.sync is a meant for exposing simple RPC handlers. return OperationHandler.sync( // The method is for making arbitrary short calls to other services or databases, or // perform simple computations such as this one. Users can also access a workflow client by // calling // Nexus.getOperationContext().getWorkflowClient(ctx) to make arbitrary calls such as // signaling, querying, or listing workflows. (ctx, details, input) -> new SampleNexusService.EchoOutput(input.getMessage())); } @OperationImpl public OperationHandler hello() { // If the operation input parameters are different from the workflow input parameters, // use the WorkflowRunOperation.fromWorkflowHandler constructor and the appropriate constructor // method on WorkflowHandle to map the Nexus input to the workflow parameters. return WorkflowRunOperation.fromWorkflowHandle( (ctx, details, input) -> WorkflowHandle.fromWorkflowMethod( Nexus.getOperationContext() .getWorkflowClient() .newWorkflowStub( HelloHandlerWorkflow.class, // Workflow IDs should typically be business meaningful IDs and are used // to // dedupe workflow starts. // For this example, we're using the request ID allocated by Temporal // when // the // caller workflow schedules // the operation, this ID is guaranteed to be stable across retries of // this // operation. // // Task queue defaults to the task queue this operation is handled on. WorkflowOptions.newBuilder() .setWorkflowId(details.getRequestId()) .build()) ::hello, input.getName(), input.getLanguage())); } } ``` ### Register a Nexus Service in a Worker After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register a Nexus Service in a Worker. [core/src/main/java/io/temporal/samples/nexus/handler/HandlerWorker.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/handler/HandlerWorker.java) ```java package io.temporal.samples.nexus.handler; import io.temporal.client.WorkflowClient; import io.temporal.samples.nexus.options.ClientOptions; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; public class HandlerWorker { public static final String DEFAULT_TASK_QUEUE_NAME = "my-handler-task-queue"; public static void main(String[] args) { WorkflowClient client = ClientOptions.getWorkflowClient(args); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME); worker.registerWorkflowImplementationTypes(HelloHandlerWorkflowImpl.class); worker.registerNexusServiceImplementation(new SampleNexusServiceImpl()); factory.start(); } } ``` ## Develop a caller Workflow that uses the Nexus Service Import the Service API package that has the necessary service and operation names and input/output types to execute a Nexus Operation from the caller Workflow: [core/src/main/java/io/temporal/samples/nexus/caller/EchoCallerWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/EchoCallerWorkflowImpl.java) ```java package io.temporal.samples.nexus.caller; import io.temporal.samples.nexus.service.SampleNexusService; import io.temporal.workflow.NexusOperationOptions; import io.temporal.workflow.NexusServiceOptions; import io.temporal.workflow.Workflow; import java.time.Duration; public class EchoCallerWorkflowImpl implements EchoCallerWorkflow { SampleNexusService sampleNexusService = Workflow.newNexusServiceStub( SampleNexusService.class, NexusServiceOptions.newBuilder() .setOperationOptions( NexusOperationOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .build()) .build()); @Override public String echo(String message) { return sampleNexusService.echo(new SampleNexusService.EchoInput(message)).getMessage(); } } ``` [core/src/main/java/io/temporal/samples/nexus/caller/HelloCallerWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/HelloCallerWorkflowImpl.java) ```java package io.temporal.samples.nexus.caller; import io.temporal.samples.nexus.service.SampleNexusService; import io.temporal.workflow.NexusOperationHandle; import io.temporal.workflow.NexusOperationOptions; import io.temporal.workflow.NexusServiceOptions; import io.temporal.workflow.Workflow; import java.time.Duration; public class HelloCallerWorkflowImpl implements HelloCallerWorkflow { SampleNexusService sampleNexusService = Workflow.newNexusServiceStub( SampleNexusService.class, NexusServiceOptions.newBuilder() .setOperationOptions( NexusOperationOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .build()) .build()); @Override public String hello(String message, SampleNexusService.Language language) { NexusOperationHandle handle = Workflow.startNexusOperation( sampleNexusService::hello, new SampleNexusService.HelloInput(message, language)); // Optionally wait for the operation to be started. NexusOperationExecution will contain the // operation token in case this operation is asynchronous. handle.getExecution().get(); return handle.getResult().get().getMessage(); } } ``` ### Register the caller Workflow in a Worker After developing the caller Workflow, the next step is to register it with a Worker. [core/src/main/java/io/temporal/samples/nexus/caller/CallerWorker.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/CallerWorker.java) ```java package io.temporal.samples.nexus.caller; import io.temporal.client.WorkflowClient; import io.temporal.samples.nexus.options.ClientOptions; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; import io.temporal.worker.WorkflowImplementationOptions; import io.temporal.workflow.NexusServiceOptions; import java.util.Collections; public class CallerWorker { public static final String DEFAULT_TASK_QUEUE_NAME = "my-caller-workflow-task-queue"; public static void main(String[] args) { WorkflowClient client = ClientOptions.getWorkflowClient(args); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker(DEFAULT_TASK_QUEUE_NAME); worker.registerWorkflowImplementationTypes( WorkflowImplementationOptions.newBuilder() .setNexusServiceOptions( Collections.singletonMap( "SampleNexusService", NexusServiceOptions.newBuilder().setEndpoint("my-nexus-endpoint-name").build())) .build(), EchoCallerWorkflowImpl.class, HelloCallerWorkflowImpl.class); factory.start(); } } ``` ### Develop a starter to start the caller Workflow To initiate the caller Workflow, a starter program is used. [core/src/main/java/io/temporal/samples/nexus/caller/CallerStarter.java](https://github.com/temporalio/samples-java/blob/nexus-snip-sync/core/src/main/java/io/temporal/samples/nexus/caller/CallerStarter.java) ```java package io.temporal.samples.nexus.caller; import io.temporal.api.common.v1.WorkflowExecution; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowOptions; import io.temporal.samples.nexus.options.ClientOptions; import io.temporal.samples.nexus.service.SampleNexusService; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class CallerStarter { private static final Logger logger = LoggerFactory.getLogger(CallerStarter.class); public static void main(String[] args) { WorkflowClient client = ClientOptions.getWorkflowClient(args); WorkflowOptions workflowOptions = WorkflowOptions.newBuilder().setTaskQueue(CallerWorker.DEFAULT_TASK_QUEUE_NAME).build(); EchoCallerWorkflow echoWorkflow = client.newWorkflowStub(EchoCallerWorkflow.class, workflowOptions); WorkflowExecution execution = WorkflowClient.start(echoWorkflow::echo, "Nexus Echo 👋"); logger.info( "Started EchoCallerWorkflow workflowId: {} runId: {}", execution.getWorkflowId(), execution.getRunId()); logger.info("Workflow result: {}", echoWorkflow.echo("Nexus Echo 👋")); HelloCallerWorkflow helloWorkflow = client.newWorkflowStub(HelloCallerWorkflow.class, workflowOptions); execution = WorkflowClient.start(helloWorkflow::hello, "Nexus", SampleNexusService.Language.EN); logger.info( "Started HelloCallerWorkflow workflowId: {} runId: {}", execution.getWorkflowId(), execution.getRunId()); logger.info("Workflow result: {}", helloWorkflow.hello("Nexus", SampleNexusService.Language.ES)); } } ``` ## Make Nexus calls across Namespaces with a development Server Follow the steps below to run the Nexus handler Worker, the Nexus caller Worker, and the starter app. ### Run Workers connected to a local development server Run the Nexus handler Worker: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.handler.HandlerWorker \ --args="-target-host localhost:7233 -namespace my-target-namespace" ``` In another terminal window, run the Nexus caller Worker: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerWorker \ --args="-target-host localhost:7233 -namespace my-caller-namespace" ``` ### Start a caller Workflow With the Workers running, the final step in the local development process is to start a caller Workflow. Run the starter: ```bash ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerStarter \ --args="-target-host localhost:7233 -namespace my-caller-namespace" ``` This will result in: ``` [main] INFO i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9b3de8ba-28ae-42fb-8087-bdedf4cecd39 runId: 404a2529-764d-4d1d-9de5-8a9475e40fba [main] INFO i.t.s.nexus.caller.CallerStarter - Workflow result: Nexus Echo 👋 [main] INFO i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9cb29897-356a-4714-87b7-aa2f00784a46 runId: 7e71e62a-db50-49da-b081-24b61016a0fc [main] INFO i.t.s.nexus.caller.CallerStarter - Workflow result: ¡Hola! Nexus 👋 ``` ### Canceling a Nexus Operation To cancel a Nexus Operation from within a Workflow, create a `CancellationScope` using the `Workflow.newCancellationScope` API. `Workflow.newCancellationScope` takes a `Runnable`. Any SDK methods started in this runnable, such as Nexus operations, will be associated with this scope. `Workflow.newCancellationScope` returns a new scope that, when the `cancel()` method is called, cancels the context and any SDK method that was started in the scope. The promise returned by `Workflow.startNexusOperation` is resolved when the operation finishes, whether it succeeds, fails, times out, or is canceled. Only asynchronous operations can be canceled in Nexus, as cancelation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancelation request. If ignored, the operation may enter a terminal state. When a Nexus operation is started the caller can specify different cancellation types that will control how the caller reacts to cancellation: - `ABANDON` - Do not request cancellation of the operation. - `TRY_CANCEL` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is done. - `WAIT_REQUESTED` - Request cancellation of the operation and wait for confirmation that the request was received. Doesn't wait for actual cancellation. - `WAIT_COMPLETED` - Wait for operation completion. Operation may or may not complete as cancelled. The default is `WAIT_COMPLETED`. Users can set a different option on the `NexusServiceOptions` by calling `.setCancellationType()` on `NexusServiceOptions.Builder`. Once the caller Workflow completes, the caller's Nexus Machinery stops attempting to cancel operations that have not yet been canceled, letting them run to completion. It's okay to leave operations running in some use cases. To ensure cancelations are delivered, wait for all pending operations to deliver their cancellation requests before exiting the Workflow. See the [Nexus cancelation sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuscancellation) for reference. ## Make Nexus calls across Namespaces in Temporal Cloud This section assumes you are already familiar with [how connect a Worker to Temporal Cloud](/develop/java/client/temporal-client#start-workflow-execution). The same [source code](https://github.com/temporalio/samples-go/tree/main/nexus) is used in this section, but the Temporal Cloud CLI will be used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install `tcld` and generate certificates Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command (on macOS): ``` brew install temporalio/brew/tcld ``` If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: ``` tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key ``` These certificates will be valid for one year. ### Create caller and handler Namespaces Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. **Temporal CLI** ``` temporal cloud login temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` **tcld** ``` tcld login tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/Namespaces](https://cloud.temporal.io/Namespaces). ### Create a Nexus Endpoint to route requests from caller to handler To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. **Temporal CLI** ``` temporal cloud nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file ./core/src/main/java/io/temporal/samples/nexus/service/description.md ``` **tcld** ``` tcld nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file ./core/src/main/java/io/temporal/samples/nexus/service/description.md ``` The `--allow-namespace` is used to build an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in Runtime Access Control. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus). ### Run Workers Connected to Temporal Cloud Run the handler Worker: ``` ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.handler.HandlerWorker \ --args="-target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key'" ``` Run the caller Worker: ``` ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerWorker \ --args="-target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key'" ``` ### Start a caller Workflow ``` ./gradlew -q execute -PmainClass=io.temporal.samples.nexus.caller.CallerStarter \ --args="-target-host .tmprl.cloud:7233 \ -namespace \ -client-cert 'path/to/your/ca.pem' \ -client-key 'path/to/your/ca.key'" ``` This will result in: ``` [main] INFO i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9b3de8ba-28ae-42fb-8087-bdedf4cecd39 runId: 404a2529-764d-4d1d-9de5-8a9475e40fba [main] INFO i.t.s.nexus.caller.CallerStarter - Workflow result: Nexus Echo 👋 [main] INFO i.t.s.nexus.caller.CallerStarter - Started workflow workflowId: 9cb29897-356a-4714-87b7-aa2f00784a46 runId: 7e71e62a-db50-49da-b081-24b61016a0fc [main] INFO i.t.s.nexus.caller.CallerStarter - Workflow result: ¡Hola! Nexus 👋 ``` ## Observability ### Web UI A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history: ![Observability Sync](/img/cloud/nexus/go-sdk-observability-sync.png) An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history: ![Observability Async](/img/cloud/nexus/go-sdk-observability-async.png) ### Temporal CLI Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: ``` temporal workflow describe -w ``` Nexus events are included in the caller's Event history: ``` temporal workflow show -w ``` For **asynchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationStarted` - `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationCompleted` > **📝 Note:** > > `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. > ## Learn more - Read the high-level description of the [Temporal Nexus feature](/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082). - Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus). - Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus). --- # Nexus Java Quickstart Source: https://docs.temporal.io/develop/java/nexus/quickstart > Build a Nexus Service that wraps an existing Temporal Workflow using the Java SDK [Temporal Nexus](/evaluate/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow. > **ℹ️ Info:** > To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus). **Prerequisites:** Complete the [Java SDK Quickstart](/develop/java/set-up-your-local-java) first. You should have `SayHelloWorkflow`, `SayHelloWorkflowImpl`, `GreetActivities`, `GreetActivitiesImpl`, and `SayHelloWorker` from that guide. ## What you'll build You have `SayHelloWorkflow` running in the `default` Namespace. By the end of this guide: 1. A Nexus Service will expose `SayHelloWorkflow` as an Operation. 2. A second Namespace will contain a Workflow that calls that Operation. 3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces. ## 1. Define the Nexus Service Create a file called `SayHelloNexusService.java` that defines the Nexus Service contract. Creating a Nexus Service establishes the contract between your implementation and any callers. It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract. The `@Service` annotation declares this as a Nexus Service. The `@Operation` annotation marks `sayHello` as a callable Nexus Operation. `SayHelloWorkflow` returns `String`, so the operation output type is also `String`. ```java package helloworkflow; import io.nexusrpc.Operation; import io.nexusrpc.Service; @Service public interface SayHelloNexusService { @Operation String sayHello(String name); } ``` ## 2. Define the Nexus Operation Handlers Create a file called `SayHelloNexusServiceImpl.java` that implements the Nexus Operation handler. Operation handlers contain the logic that runs when a caller invokes a Nexus Operation. The `@OperationImpl` annotation creates an asynchronous Nexus Operation that starts `SayHelloWorkflow`. The handler bridges the Nexus `String` input to `SayHelloWorkflow`'s `sayHello` method directly. ```java package helloworkflow; import io.nexusrpc.handler.OperationHandler; import io.nexusrpc.handler.OperationImpl; import io.nexusrpc.handler.ServiceImpl; import io.temporal.client.WorkflowOptions; import io.temporal.nexus.Nexus; import io.temporal.nexus.WorkflowRunOperation; @ServiceImpl(service = SayHelloNexusService.class) public class SayHelloNexusServiceImpl { @OperationImpl public OperationHandler sayHello() { return WorkflowRunOperation.fromWorkflowMethod( (ctx, details, name) -> Nexus.getOperationContext() .getWorkflowClient() .newWorkflowStub( SayHelloWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId("say-hello-nexus-" + details.getRequestId()) .build()) ::sayHello ); } } ``` ## 3. Register the Nexus Service Handler in a Worker Update your existing `SayHelloWorker.java` to register the Nexus Service Handler. A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered. This is the same Worker concept used for Workflows and Activities. The registerNexusServiceImplementation parameter registers the handler so it can receive Nexus Operation requests. ```java package helloworkflow; import io.temporal.client.WorkflowClient; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; public class SayHelloWorker { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("my-task-queue"); worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetActivitiesImpl()); worker.registerNexusServiceImplementation(new SayHelloNexusServiceImpl()); System.out.println("Starting SayHelloWorker for task queue 'my-task-queue'..."); factory.start(); } } ``` ## 4. Develop the caller Workflow Create two files — `NexusCallerWorkflow.java` and `NexusCallerWorkflowImpl.java` — that define a Workflow which invokes the Nexus Operation. The caller Workflow demonstrates the consumer side of Nexus. Instead of importing handler code directly, the caller only depends on the Service contract. This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams. The `Workflow.newNexusServiceStub` method creates a client bound to your Nexus Service and Endpoint. ```java package helloworkflow; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; @WorkflowInterface public interface NexusCallerWorkflow { @WorkflowMethod String greetThroughNexus(String name); } ``` ```java package helloworkflow; import io.temporal.workflow.NexusOperationOptions; import io.temporal.workflow.NexusServiceOptions; import io.temporal.workflow.Workflow; import java.time.Duration; public class NexusCallerWorkflowImpl implements NexusCallerWorkflow { private final SayHelloNexusService nexusService = Workflow.newNexusServiceStub( SayHelloNexusService.class, NexusServiceOptions.newBuilder() .setOperationOptions( NexusOperationOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .build()) .build() ); @Override public String greetThroughNexus(String name) { return nexusService.sayHello(name); } } ``` ## 5. Create the caller Namespace and Nexus Endpoint Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server. Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. Make sure your local Temporal dev server is running (`temporal server start-dev`). ```bash temporal operator namespace create --namespace my-caller-namespace ``` ```bash temporal operator nexus endpoint create \\ --name my-nexus-endpoint-name \\ --target-namespace default \\ --target-task-queue my-task-queue ``` ## 6. Run and Verify Create `NexusCallerStarter.java` to start the caller Worker and execute the Workflow. This brings everything together: the caller Worker hosts `NexusCallerWorkflow`, which uses the Nexus stub to invoke `sayHello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller. **Run the application:** 1. Start the handler Worker in one terminal: ```bash mvn compile exec:java \ -Dexec.mainClass="helloworkflow.SayHelloWorker" ``` 2. Run the caller in another terminal: ```bash mvn compile exec:java \ -Dexec.mainClass="helloworkflow.NexusCallerStarter" ``` You should see: ``` Workflow result: Hello Temporal ``` Open the [Temporal Web UI](http://localhost:8233) and switch between Namespaces to see both Workflow Executions. In `my-caller-namespace`, find the `NexusCallerWorkflow` execution — you should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in its history. In `default`, find the `SayHelloWorkflow` execution that was started by the Nexus Operation. ```java package helloworkflow; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.client.WorkflowOptions; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; import io.temporal.worker.WorkflowImplementationOptions; import io.temporal.workflow.NexusServiceOptions; import java.util.Collections; public class NexusCallerStarter { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service, WorkflowClientOptions.newBuilder() .setNamespace("my-caller-namespace") .build() ); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("my-caller-task-queue"); worker.registerWorkflowImplementationTypes( WorkflowImplementationOptions.newBuilder() .setNexusServiceOptions( Collections.singletonMap( "SayHelloNexusService", NexusServiceOptions.newBuilder() .setEndpoint("my-nexus-endpoint-name") .build())) .build(), NexusCallerWorkflowImpl.class ); factory.start(); NexusCallerWorkflow workflow = client.newWorkflowStub( NexusCallerWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue("my-caller-task-queue") .setWorkflowId("nexus-caller-workflow-id") .build() ); String result = workflow.greetThroughNexus("Temporal"); System.out.println("Workflow result: " + result); factory.shutdown(); System.exit(0); } } ``` ## Next Steps Now that you have a working Nexus Service, here are some resources to deepen your understanding: - **[Java Nexus Feature Guide](/develop/java/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls. - **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics. - **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker. - **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns. - **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries. - **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls. - **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests. - **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity. --- # Standalone Nexus Operations - Java SDK Source: https://docs.temporal.io/develop/java/nexus/standalone-operations > Execute Nexus Operations independently without a Workflow using the Temporal Java SDK. > **Pre-release** > Requires Java SDK `v1.36.1` or above. All APIs are experimental and may be subject to backwards-incompatible changes. [Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using `Workflow.newNexusServiceStub()`, you execute a Standalone Nexus Operation directly from a Nexus service client created from a `NexusClient` using `NexusClient.newNexusServiceClient()`. Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/java/nexus/feature-guide) for details on [defining a Service contract](/develop/java/nexus/feature-guide#define-nexus-service-contract), [developing Operation handlers](/develop/java/nexus/feature-guide#develop-nexus-service-operation-handlers), and [registering a Service in a Worker](/develop/java/nexus/feature-guide#register-a-nexus-service-in-a-worker). This page focuses on the client-side APIs that are unique to Standalone Nexus Operations: - [Execute a Standalone Nexus Operation](#execute-operation) - [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result) - [List Standalone Nexus Operations](#list-operations) - [Count Standalone Nexus Operations](#count-operations) - [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud) > **📝 Note:** > This documentation uses source code from the > [Java Nexus Standalone sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusstandalone). > ## Prerequisites Standalone Nexus Operations are at Pre-release and require a special Temporal CLI build. ### 1. Install and verify the Pre-release Temporal CLI The `temporal nexus operation` commands require a Pre-release build of the Temporal CLI. See [Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for the platform downloads, then verify: ```bash ./temporal --version # temporal version 1.7.4-standalone-nexus-operations ``` Run it as `./temporal` from the directory where you extracted it. The standard `brew install temporal` build does not include Standalone Nexus Operation support during Pre-release. ### 2. Start a local dev server The Pre-release dev server enables Standalone Nexus Operations by default — no dynamic config is required. Start it with the caller and handler Namespaces pre-created: ```bash ./temporal server start-dev \ --namespace my-caller-namespace \ --namespace my-handler-namespace ``` The starter and Worker connect to two different Namespaces (a caller Namespace and a handler Namespace), mirroring how Nexus crosses Namespace boundaries. To run the examples on this page against the [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexusstandalone), create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue: ```bash ./temporal operator nexus endpoint create \ --name my-nexus-endpoint \ --target-namespace my-handler-namespace \ --target-task-queue nexus-handler-queue ``` Start the sample handler Worker in the handler Namespace: ```bash TEMPORAL_NAMESPACE=my-handler-namespace \ ./gradlew -q :core:execute -PmainClass=io.temporal.samples.nexusstandalone.handler.HandlerWorker ``` Run the starter in the caller Namespace (from a separate terminal): ```bash TEMPORAL_NAMESPACE=my-caller-namespace \ ./gradlew -q :core:execute -PmainClass=io.temporal.samples.nexusstandalone.StandaloneClientStarter ``` ## Execute a Standalone Nexus Operation To execute a Standalone Nexus Operation, first create a [`NexusClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html), then derive a typed [`NexusServiceClient`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusServiceClient.html) from it with `newNexusServiceClient()`, bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `start()` or `execute()` from application code (for example, a starter program), not from inside a Workflow Definition. `execute()` waits for the Operation to complete and returns the result. Both methods take a [`StartNexusOperationOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/StartNexusOperationOptions.html) whose `id` is required — the SDK never generates one for you. `scheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server. ```java NexusClient nexusClient = NexusClient.newInstance(stubs, options); NexusServiceClient greetingClient = nexusClient.newNexusServiceClient(GreetingNexusService.class, ENDPOINT_NAME); // Block until the operation completes and return its result. GreetingOutput greeting = greetingClient.execute( GreetingNexusService::greet, StartNexusOperationOptions.newBuilder() .setId("greet-" + UUID.randomUUID()) .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .build(), new GreetingInput("World")); ``` `executeAsync()` is the same but returns a `CompletableFuture` instead of blocking. ```java CompletableFuture future = greetingClient.executeAsync( GreetingNexusService::greet, options, new GreetingInput("World")); GreetingOutput greeting = future.get(); ``` See the full [starter sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/nexusstandalone/StandaloneClientStarter.java) for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and counts Operations. Or use the Temporal CLI to execute a Standalone Nexus Operation: ```bash ./temporal nexus operation execute \ --namespace my-caller-namespace \ --endpoint my-nexus-endpoint \ --service GreetingNexusService \ --operation greet \ --operation-id my-greet-op \ --input '{"name":"World"}' ``` ## Start a Standalone Nexus Operation and Wait for the Result `start()` returns a [`NexusOperationHandle`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusOperationHandle.html). Use `NexusOperationHandle.getResult()` to wait until the Operation completes and retrieve its result. This works for both synchronous and asynchronous Operations. ```java // Start an operation and get a NexusOperationHandle. NexusOperationHandle handle = greetingClient.start( GreetingNexusService::startGreeting, options, new GreetingInput("World")); // Block until the operation completes and retrieve its result. GreetingOutput greeting = handle.getResult(); ``` If the Operation completed successfully, the result is returned. If the Operation failed, the failure is thrown as a `NexusOperationException`. Use `getResultAsync()` for a non-blocking `CompletableFuture`, or `getResult(long timeout, TimeUnit unit)` to bound the wait. Or use the Temporal CLI to wait for a result by Operation ID: ```bash ./temporal nexus operation result --namespace my-caller-namespace --operation-id my-greet-op ``` ## List Standalone Nexus Operations Use [`NexusClient.listNexusOperationExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The result is a `Stream` of operation metadata entries. Note that `listNexusOperationExecutions()` is called on a `NexusClient`, not on the typed `NexusServiceClient`. ```java String query = "Endpoint = \"" + ENDPOINT_NAME + "\""; nexusClient .listNexusOperationExecutions(query) .forEach( op -> System.out.printf( "OperationId: %s, Operation: %s, Status: %s%n", op.getOperationId(), op.getOperation(), op.getStatus())); ``` The `query` parameter accepts [List Filter](/list-filter) syntax. For example, `"Endpoint = 'my-endpoint' AND ExecutionStatus = 'Running'"`. Or use the Temporal CLI: ```bash ./temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Count Standalone Nexus Operations Use [`NexusClient.countNexusOperationExecutions()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/NexusClient.html) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. Note that `countNexusOperationExecutions()` is called on a `NexusClient`, not on the typed `NexusServiceClient`. ```java String query = "Endpoint = \"" + ENDPOINT_NAME + "\""; NexusOperationExecutionCount count = nexusClient.countNexusOperationExecutions(query); System.out.println("Total Nexus operations: " + count.getCount()); ``` Passing a `GROUP BY` query (for example, `"GROUP BY ExecutionStatus"`) returns a count per group, available through `NexusOperationExecutionCount.getGroups()`. Or use the Temporal CLI: ```bash ./temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Run Standalone Nexus Operations with Temporal Cloud The code samples referenced on this page build their client from a `ClientConfigProfile` loaded from a TOML profile, so the same code works against Temporal Cloud — just point the profile at your Cloud Namespace (or override the connection via `TEMPORAL_*` environment variables). No code changes are needed. For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate generation, and authentication options, see [Make Nexus calls across Namespaces in Temporal Cloud](/develop/java/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) and [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud). --- # Platform - Java SDK Source: https://docs.temporal.io/develop/java/platform > This section explains how to implement platform with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Platform - [Observability](/develop/java/platform/observability) - [Enriching the UI](/develop/java/platform/enriching-ui) --- # Enriching the user interface - Java SDK Source: https://docs.temporal.io/develop/java/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the Java SDK. Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```java import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowOptions; import io.temporal.serviceclient.WorkflowServiceStubs; public class Main { public static void main(String[] args) { // Create service stubs and workflow client WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient workflowClient = WorkflowClient.newInstance(service); // Create workflow options with static summary and details WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("your-workflow-id") .setTaskQueue("your-task-queue") .setStaticSummary("Order processing for customer #12345") .setStaticDetails("Processing premium order with expedited shipping") .build(); // Create the workflow stub YourWorkflow workflow = workflowClient.newWorkflowStub(YourWorkflow.class, options); // Start the workflow String result = workflow.yourWorkflowMethod("workflow input"); } } ``` `setStaticSummary()` sets a single-line description that appears in the Workflow list view, limited to 200 bytes. `setStaticDetails()` sets multi-line comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use `WorkflowClient.start()` for async execution: ```java // Start workflow asynchronously WorkflowExecution execution = WorkflowClient.start(workflow::yourWorkflowMethod, "workflow input"); ``` ### Inside the Workflow Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```java import io.temporal.workflow.Workflow; public class YourWorkflowImpl implements YourWorkflow { @Override public String yourWorkflowMethod(String input) { // Get the current details String currentDetails = Workflow.getCurrentDetails(); Workflow.getLogger(YourWorkflowImpl.class).info("Current details: " + currentDetails); // Set/update the current details Workflow.setCurrentDetails("Updated workflow details with new status"); return "Workflow completed"; } } ``` ### Adding Summary to Activities and Timers You can attach a `setSummary()` to Activities when starting them from within a Workflow: ```java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.Workflow; import java.time.Duration; public class YourWorkflowImpl implements YourWorkflow { private final YourActivities activities = Workflow.newActivityStub(YourActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setSummary("Processing user data") .build()); @Override public String yourWorkflowMethod(String input) { // Execute the activity with the summary String result = activities.yourActivity(input); return result; } } ``` Similarly, you can attach a `setSummary()` to timers within a Workflow: ```java import io.temporal.workflow.Workflow; import io.temporal.workflow.TimerOptions; import java.time.Duration; public class YourWorkflowImpl implements YourWorkflow { @Override public String yourWorkflowMethod(String input) { // Create a timer with a summary Workflow.newTimer(Duration.ofMinutes(5), TimerOptions.newBuilder() .setSummary("Waiting for payment confirmation") .build()) .get(); // Wait for the timer to fire return "Timer completed"; } } ``` The input format for `setSummary()` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the workflow details page, you'll find the workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the workflow - **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. --- # Observability - Java SDK Source: https://docs.temporal.io/develop/java/platform/observability > Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes. The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. This section covers features related to viewing the state of the application, including: - [Emit metrics](#metrics) - [Set up tracing](#tracing) - [Log from a Workflow](#logging) - [Visibility APIs](#visibility) ## Emit metrics Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process. For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics). - For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide. - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). - For an end-to-end example that exposes metrics with the Java SDK, refer to the [samples-java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) repo. To emit metrics with the Java SDK, use the [`MicrometerClientStatsReporter`](https://github.com/temporalio/sdk-java/blob/69a5d3fa44da4043697b7e57d7ef6691c24287cd/temporal-sdk/src/main/java/io/temporal/common/reporter/MicrometerClientStatsReporter.java#L18) class to integrate with Micrometer MeterRegistry configured for your metrics backend. [Micrometer](https://micrometer.io/docs) is a popular Java framework that provides integration with Prometheus and other backends. The following example shows how to use `MicrometerClientStatsReporter` to define the metrics scope and set it with the `WorkflowServiceStubsOptions`. ```java //... // see the Micrometer documentation for configuration details on other supported monitoring systems. // in this example shows how to set up Prometheus registry and stats reported. PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); StatsReporter reporter = new MicrometerClientStatsReporter(registry); // set up a new scope, report every 10 seconds Scope scope = new RootScopeBuilder() .reporter(reporter) .reportEvery(com.uber.m3.util.Duration.ofSeconds(10)); // for Prometheus collection, expose a scrape endpoint. //... // add metrics scope to WorkflowServiceStub options WorkflowServiceStubsOptions stubOptions = WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build(); //... ``` For more details, see the [Java SDK Samples](https://github.com/temporalio/samples-java/tree/637c2e66fd2dab43d9f3f39e5fd9c55e4f3884f0/core/src/main/java/io/temporal/samples/metrics). For details on configuring a Prometheus scrape endpoint with Micrometer, see the [Micrometer Prometheus Configuring](https://docs.micrometer.io/micrometer/reference/implementations/prometheus.html#_configuring) documentation. ## Set up tracing Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows. Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly. To configure tracing in Java, register the `OpenTracingClientInterceptor()` interceptor. You can register the interceptors on both the Temporal Client side and the Worker side, or through a [Plugin](/develop/plugins-guide#interceptors) if you're building a reusable library. The following code examples demonstrate the `OpenTracingClientInterceptor()` on the Temporal Client. ```java WorkflowClientOptions.newBuilder() //... .setInterceptors(new OpenTracingClientInterceptor()) .build(); ``` ```java WorkflowClientOptions clientOptions = WorkflowClientOptions.newBuilder() .setInterceptors(new OpenTracingClientInterceptor(JaegerUtils.getJaegerOptions(type))) .build(); WorkflowClient client = WorkflowClient.newInstance(service, clientOptions); ``` The following code examples demonstrate the `OpenTracingClientInterceptor()` on the Worker. ```java WorkerFactoryOptions.newBuilder() //... .setWorkerInterceptors(new OpenTracingWorkerInterceptor()) .build(); ``` ```java WorkerFactoryOptions factoryOptions = WorkerFactoryOptions.newBuilder() .setWorkerInterceptors( new OpenTracingWorkerInterceptor(JaegerUtils.getJaegerOptions(type))) .build(); WorkerFactory factory = WorkerFactory.newInstance(client, factoryOptions); ``` For more information, see the Temporal [OpenTracing module](https://github.com/temporalio/sdk-java/blob/main/contrib/temporal-opentracing/README.md). ### Context Propagation Over Nexus Operation Calls Nexus does not use the standard context propagator header structure. Instead, it relies on a Temporal-agnostic protocol designed to connect arbitrary systems. To propagate context over Nexus Operation calls, the context is serialized into a `Map`. This map is special as it will normalize all keys to lowercase. Because Nexus uses this custom format, and because Nexus calls may involve external systems, the `ContextPropagator` interface doesn’t apply to Nexus headers. Context must be explicitly propagated through interceptors, as shown in the [Nexus Context Propagation sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/nexuscontextpropagation). ## Log from a Workflow Logging enables you to record critical information during code execution. Loggers create an audit trail and capture information about your Workflow's operation. An appropriate logging level depends on your specific needs. During development or troubleshooting, you might use debug or even trace. In production, you might use info or warn to avoid excessive log volume. You can find the log levels supported by `slf4j` in [their official documentation](https://www.slf4j.org/apidocs/org/slf4j/ext/XLogger.Level.html). The Temporal SDK core normally uses `WARN` as its default logging level. To get a standard `slf4j` logger in your Workflow code, use the [`Workflow.getLogger`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) method. ```java private static final Logger logger = Workflow.getLogger(DynamicDslWorkflow.class); ``` Logs in replay mode are omitted unless the [`WorkerFactoryOptions.Builder.setEnableLoggingInReplay(boolean)`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactoryOptions.Builder.html#setEnableLoggingInReplay(boolean)) method is set to true. ### How to provide a custom logger Use a custom logger for logging. To set a custom logger, supply your own logging implementation and configuration details the same way you would in any other Java application. ## Visibility APIs The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service. ### How to use Search Attributes The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI. - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an option when starting the Execution. - In the Workflow by calling `upsertTypedSearchAttributes`. - Read the value of the Search Attribute: - On the Client by calling `DescribeWorkflow`. - In the Workflow by looking at `WorkflowInfo`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/workflow#list). - In code by calling `ListWorkflowExecutions`. ### How to set custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. When starting a Workflow Execution with your Client, include the Custom Search Attribute in the options using `WorkflowOptions.newBuilder().setTypedSearchAttributes()`: ```java // In a shared constants file, so all files have access public static final SearchAttributeKey IS_ORDER_FAILED = SearchAttributeKey.forBoolean("isOrderFailed"); ... // In main WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId(workflowID) .setTaskQueue(Constants.TASK_QUEUE_NAME) .setTypedSearchAttributes(generateSearchAttributes()) .build(); PizzaWorkflow workflow = client.newWorkflowStub(PizzaWorkflow.class, options); ... // Further down in the file private static Map generateSearchAttributes(){ return SearchAttributes.newBuilder().set(Constants.IS_ORDER_FAILED, false).build(); } ``` Each `SearchAttribute` object represents a custom attribute name, and the value is a [`SearchAttributeKey`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/SearchAttributeKey.html#forBoolean(java.lang.String)) representing a specific type. Currently the following types are supported: - Boolean - Double - Long - KeyWord - KeyWordList - Text In this example `isOrderFailed` is set as a Search Attribute. This attribute is useful for querying Workflows based on the success/failure of customer orders. ### How to upsert Search Attributes Within the Workflow code, you can dynamically add or update Search Attributes using [`upsertTypedSearchAttributes`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#upsertTypedSearchAttributes(io.temporal.common.SearchAttributeUpdate...)). This method is particularly useful for Workflows whose attributes need to change based on internal logic or external events. ```java import io.temporal.workflow.Workflow; ... // Existing Workflow Logic Map searchAttribute = new HashMap<>(); Distance distance; try { distance = activities.getDistance(address); searchAttribute.put("isOrderFailed", false); Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueSet(false)); } catch (NullPointerException e) { searchAttribute.put("isOrderFailed", true); Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueSet(true)); throw new NullPointerException("Unable to get distance"); } ``` ### How to remove a Search Attribute from a Workflow To remove a Search Attribute that was previously set, set it to an empty Map. ```java // In a shared constants file, so all files have access public static final SearchAttributeKey IS_ORDER_FAILED = SearchAttributeKey.forBoolean("isOrderFailed"); ... Workflow.upsertTypedSearchAttributes(Constants.IS_ORDER_FAILED.valueUnset()); ``` --- # Set up your local with the Java SDK Source: https://docs.temporal.io/develop/java/set-up-your-local-java > Configure your local development environment to get started developing with Temporal # Quickstart Configure your local development environment to get started developing with Temporal. ## Install the Java JDK Make sure you have the Java JDK installed. You can either download a copy directly from Oracle or select an OpenJDK distribution from your preferred vendor. You'll also need either Maven or Gradle installed. **If you don't have Maven:** [Download](https://maven.apache.org/download.cgi) and [install](https://maven.apache.org/install.html) from Apache.org, or use Homebrew: `brew install maven`. **If you don't have Gradle:** [Download](https://gradle.org/install/) from Gradle.org, use [IntelliJ IDEA](https://www.jetbrains.com/idea/) (bundled), or use Homebrew: `brew install gradle`. ```bash java -version ``` ## Create a Project Now that you have your build tool installed, create a project to manage your dependencies and build your Temporal application. Choose your build tool to create the appropriate project structure. For Maven, this creates a standard project with the necessary directories and a basic pom.xml file. For Gradle, this creates a project with build.gradle and the standard Gradle directory structure. **Maven** ```bash mkdir temporal-java-project ``` ```bash cd temporal-java-project ``` ```bash mvn archetype:generate -DgroupId=helloworkflow -DartifactId=temporal-hello-world -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false ``` ```bash cd temporal-hello-world ``` **Gradle** ```bash mkdir temporal-hello-world ``` ```bash cd temporal-hello-world ``` ```bash gradle init --type java-application --project-name temporal-hello-world --package helloworkflow ``` ## Add Temporal Java SDK Dependencies Now add the Temporal SDK dependencies to your project configuration file. For Maven, add the following dependencies to your `pom.xml` file. For Gradle, add the following lines to your `build.gradle` file. Next, you'll configure a local Temporal Service for development. **Maven** ```xml io.temporal temporal-sdk 1.33.0 io.temporal temporal-testing 1.33.0 test ``` **Gradle** ```groovy plugins { id 'application' } repositories { mavenCentral() } dependencies { implementation 'io.temporal:temporal-sdk:1.33.0' testImplementation 'io.temporal:temporal-testing:1.33.0' } // Define the main class for the application application { mainClass = 'helloworkflow.Starter' } // Helper tasks to run the worker and the starter tasks.register('runWorker', JavaExec) { group = 'application' description = 'Run the Temporal worker' classpath = sourceSets.main.runtimeClasspath mainClass = 'helloworkflow.SayHelloWorker' } tasks.register('runStarter', JavaExec) { group = 'application' description = 'Run the workflow starter' classpath = sourceSets.main.runtimeClasspath mainClass = 'helloworkflow.Starter' } ``` ```bash ./gradlew build ``` ## Install Temporal CLI and start the development server The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI: **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal. Keep this running in the background: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal Java SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Create the Activity Interface Create an Activity interface file (GreetActivities.java): _Note that all files for this quickstart will be created under src/main/java/helloworkflow._ ```java package helloworkflow; import io.temporal.activity.ActivityInterface; import io.temporal.activity.ActivityMethod; @ActivityInterface public interface GreetActivities { @ActivityMethod String greet(String name); } ``` An Activity is a method that executes a single, well-defined action (either short or long running), which often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration. You define Activities in Java as an annotated interface, and its implementation. ### 2. Create the Activity Implementation Create an Activity implementation file (GreetActivitiesImpl.java): ```java package helloworkflow; public class GreetActivitiesImpl implements GreetActivities { @Override public String greet(String name) { return "Hello " + name; } } ``` ### 3. Create the Workflow Create a Workflow file (SayHelloWorkflow.java): ```java package helloworkflow; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; @WorkflowInterface public interface SayHelloWorkflow { @WorkflowMethod String sayHello(String name); } ``` Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. You define Workflows in Java as an annotated interface, and its implementation. ### 4. Create the Workflow Implementation Create a Workflow implementation file (SayHelloWorkflowImpl.java): ```java package helloworkflow; import io.temporal.activity.ActivityOptions; import io.temporal.workflow.Workflow; import java.time.Duration; public class SayHelloWorkflowImpl implements SayHelloWorkflow { private final GreetActivities activities = Workflow.newActivityStub( GreetActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(5)) .build() ); @Override public String sayHello(String name) { return activities.greet(name); } } ``` ### 5. Create and Run the Worker Create a Worker file (SayHelloWorker.java): ```java package helloworkflow; import io.temporal.client.WorkflowClient; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.worker.Worker; import io.temporal.worker.WorkerFactory; public class SayHelloWorker { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("my-task-queue"); worker.registerWorkflowImplementationTypes(SayHelloWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetActivitiesImpl()); System.out.println("Starting SayHelloWorker for task queue 'my-task-queue'..."); factory.start(); } } ``` With your Activity and Workflow defined, you need a Worker to execute them. Open a new terminal and run the Worker: **Maven** ```bash cd temporal-hello-world mvn compile exec:java -Dexec.mainClass="helloworkflow.SayHelloWorker" ``` **Gradle** ```bash ./gradlew runWorker ``` A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue, it then executes that task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). ### 6. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. This final step will validate that everything is working correctly with your file labeled `Starter.java`. Create a separate file called `Starter.java`: ```java package helloworkflow; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowOptions; import io.temporal.serviceclient.WorkflowServiceStubs; public class Starter { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); SayHelloWorkflow workflow = client.newWorkflowStub( SayHelloWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue("my-task-queue") .setWorkflowId("say-hello-workflow-id") .build() ); String result = workflow.sayHello("Temporal"); System.out.println("Workflow result: " + result); service.shutdown(); service.awaitTermination(10, java.util.concurrent.TimeUnit.SECONDS); } } ``` While your worker is still running, open a new terminal and run: **Maven** ```bash cd temporal-hello-world mvn compile exec:java -Dexec.mainClass="helloworkflow.Starter" ``` **Gradle** ```bash ./gradlew runStarter ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the workflow and activity - Output: `Workflow result: Hello Temporal` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/java/first_program_in_java/): Create a basic Workflow and run it with the Temporal Java SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Worker Versioning (Legacy) - Java SDK Source: https://docs.temporal.io/develop/java/worker-versioning-legacy > Learn the Java SDK's outdated Worker Versioning APIs. ## How to use Worker Versioning in Java (Deprecated) > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > > See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. > A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Java Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your worker, assign the Build ID--perhaps from an environment variable--and turn it on. ```java // ... WorkerOptions workerOptions = WorkerOptions.newBuilder() .setBuildId(buildId) .setUseBuildIdForVersioning(true) // ... .build(); Worker w = workerFactory.newWorker("your_task_queue_name", workerOptions); // ... ``` > **⚠️ Warning:** > > Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. > ### Specify versions for Activities, Child Workflows, and Continue-as-New > **⚠️ Caution:** > > Java support for this feature is under construction! > By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `setVersioningIntent` method on the `ActivityOptions`, `ChildWorkflowOptions`, or `ContinueAsNewOptions` objects. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: ```java // ... private final MyActivity activity = Workflow.newActivityStub( MyActivity.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .setVersioningIntent(VersioningIntent.VERSIONING_INTENT_USE_ASSIGNMENT_RULES) // ...other options .build() ); // ... ``` ### Tell the Task Queue about your Worker's Build ID (Deprecated) > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. ```java // ... workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.newIdInNewDefaultSet("deadbeef")); ``` This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: ```java // ... workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.newCompatibleVersion("deadbeef", "some-existing-build-id")); ``` This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: ```java // ... workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.promoteBuildIdWithinSet("deadbeef")); ``` You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's default. ```java // ... workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.promoteSetByBuildId("deadbeef")); ``` --- # Workers - Java SDK Source: https://docs.temporal.io/develop/java/workers > This section covers Workers with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Workers - [Run Worker processes](/develop/java/workers/run-worker-process) --- # Run a Worker - Java SDK Source: https://docs.temporal.io/develop/java/workers/run-worker-process > Create and run a Temporal Worker using the Java SDK. This page covers long-lived Workers that you host and run as persistent processes. For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/java/workers/serverless-workers). ## Create and run a Worker Create a [`WorkerFactory`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactory.html) from a [Temporal Client](/develop/java/client/temporal-client), then call `newWorker()` with the Task Queue to poll. Register the Workflow and Activity types the Worker can execute, then call `start()` on the factory. [features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java) ```java WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker("my-task-queue"); worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetingActivitiesImpl()); factory.start(); ``` `start()` starts every Worker the factory created, so one factory can run several Workers in a single process. The call returns immediately and the Workers poll on background threads, so the process has to stay alive. ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Register Workflows by class with `registerWorkflowImplementationTypes()`. The Worker creates a new instance for each Workflow Execution, so a Workflow Type can be registered only once per Worker. Registering two implementations of the same type throws at registration time. Register Activities by instance with `registerActivitiesImplementations()`. One instance serves every Workflow Execution that calls it, so the implementation must be thread-safe. Pass dependencies such as database clients through the constructor. ```java worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class, OrderWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetingActivitiesImpl(databaseClient)); ``` A Worker can register one implementation of [`DynamicWorkflow`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/DynamicWorkflow.html) and one of `DynamicActivity` alongside any number of type-specific implementations. The dynamic implementation handles Workflow and Activity Types that no registered type matches. ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/java/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options Pass [`WorkerOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.html) to `newWorker()` to set concurrency limits, poller counts, and rate limits for a single Worker. Pass [`WorkerFactoryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerFactoryOptions.html) to `newInstance()` for settings shared by every Worker in the process, such as the Workflow cache size. The defaults work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `WorkerOptions`, then set a versioning behavior on each Workflow. [features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java) ```java WorkerOptions options = WorkerOptions.newBuilder() .setDeploymentOptions( WorkerDeploymentOptions.newBuilder() .setVersion(new WorkerDeploymentVersion("my-app", "1.0")) .setUseVersioning(true) .build()) .build(); Worker worker = factory.newWorker("my-task-queue", options); worker.registerWorkflowImplementationTypes(VersionedGreetingWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetingActivitiesImpl()); ``` Annotate the Workflow method with `@WorkflowVersioningBehavior(VersioningBehavior.PINNED)` or `AUTO_UPGRADE`, or set a default for the whole Worker with `setDefaultVersioningBehavior()`. The annotation belongs on the implementation, not the interface. A versioning behavior applies only to a Worker that has versioning enabled. If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker Call `shutdown()` on the factory to stop polling for new Tasks, then `awaitTermination()` to give in-flight Tasks time to finish. [features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java) ```java factory.shutdown(); factory.awaitTermination(30, TimeUnit.SECONDS); ``` Both calls apply to every Worker the factory created. See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Serverless Workers - Java SDK Source: https://docs.temporal.io/develop/java/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the Java SDK. > **Public Preview** > AWS Lambda support is in Public Preview. GCP Cloud Run support is in Pre-release, and its APIs may change in > backwards-incompatible ways. To request Cloud Run access, create a [support ticket](/cloud/support#support-ticket) or > contact your account team, and [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear > when Cloud Run reaches Public Preview. Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes. Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**AWS Lambda**](/develop/java/workers/serverless-workers/aws-lambda) - Use the `temporal-aws-lambda` contrib module to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, observability, and the invocation lifecycle. - [**GCP Cloud Run**](/develop/java/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, container packaging, and handling scale-in. --- # Serverless Workers on AWS Lambda - Java SDK Source: https://docs.temporal.io/develop/java/workers/serverless-workers/aws-lambda > Write a Temporal Worker that runs on AWS Lambda using the Java SDK temporal-aws-lambda contrib module. > **Public Preview** The `temporal-aws-lambda` contrib module lets you run a Temporal Serverless Worker on AWS Lambda. Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline. You register Workflows and Activities the same way you would with a standard Worker. For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda). ## Create and run a Worker in Lambda Create a class that implements AWS Lambda's `RequestHandler` interface and delegates to `LambdaWorker.define` in a static field. Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities. Assign the handler to a static field so it is created once during Lambda cold start and reused across invocations. ```java {12-19} package com.example; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.RequestHandler; import io.temporal.aws.lambda.LambdaWorker; import io.temporal.common.WorkerDeploymentVersion; /** AWS Lambda entry point for the Temporal Worker. */ public class LambdaFunction implements RequestHandler { private static final RequestHandler WORKER = LambdaWorker.define( new WorkerDeploymentVersion("my-app", "build-1"), builder -> { builder.setTaskQueue("my-task-queue"); builder.registerWorkflowImplementationTypes(SampleWorkflowImpl.class); builder.registerActivitiesImplementations(new GreetingActivitiesImpl()); }); @Override public Void handleRequest(Object input, Context context) { return WORKER.handleRequest(input, context); } } ``` For a complete project, see the [Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker). The `WorkerDeploymentVersion` is required. Worker Deployment Versioning is always enabled for Serverless Workers. Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `AutoUpgrade` or `Pinned`. Set it per-Workflow with the `@WorkflowVersioningBehavior` annotation on the Workflow method, or set a worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. [lambda-worker/worker/src/main/java/io/temporal/samples/lambdaworker/SampleWorkflowImpl.java](https://github.com/temporalio/samples-java/blob/main/lambda-worker/worker/src/main/java/io/temporal/samples/lambdaworker/SampleWorkflowImpl.java) ```java {21} package io.temporal.samples.lambdaworker; import io.temporal.activity.ActivityOptions; import io.temporal.common.VersioningBehavior; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowVersioningBehavior; import java.time.Duration; import org.slf4j.Logger; /** Workflow implementation that executes a greeting Activity. */ public class SampleWorkflowImpl implements SampleWorkflow { private static final Logger logger = Workflow.getLogger(SampleWorkflowImpl.class); private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(10)).build()); @Override @WorkflowVersioningBehavior(VersioningBehavior.PINNED) public String getGreeting(String name) { logger.info("SampleWorkflow started for {}", name); String result = activities.createGreeting(name); logger.info("SampleWorkflow completed with {}", result); return result; } } ``` The configure callback receives a `LambdaWorkerOptions.Builder` with the same [registration methods as a standard Worker](/develop/java/workers/run-worker-process#register-types): `registerWorkflowImplementationTypes`, `registerActivitiesImplementations`, and `registerDynamicWorkflowImplementationType`. If you need to assemble options outside the callback, call `LambdaWorkerOptions.newBuilderFromEnvironment()`, configure the builder, and pass the built options to `LambdaWorker.newHandler(...)`. ## Configure the Temporal connection The `temporal-aws-lambda` module automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details. Compared with long-lived Workers, the location of the config file is resolved differently, in the following order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. The file is optional. If absent, only environment variables are used. Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options. ## Adjust Worker defaults for Lambda The `temporal-aws-lambda` module applies conservative defaults suited to short-lived Lambda invocations. These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment. | Setting | Lambda default | |---|---| | `MaxConcurrentActivityExecutionSize` | 2 | | `MaxConcurrentWorkflowTaskExecutionSize` | 10 | | `MaxConcurrentLocalActivityExecutionSize` | 2 | | `MaxConcurrentNexusExecutionSize` | 5 | | `MaxConcurrentWorkflowTaskPollers` | 2 | | `MaxConcurrentActivityTaskPollers` | 1 | | `MaxConcurrentNexusTaskPollers` | 1 | | `WorkflowCacheSize` | 30 | | `MaxWorkflowThreadCount` | 30 | | `GracefulShutdownTimeout` | 5 seconds | | `ShutdownDeadlineBuffer` | 7 seconds | `ShutdownDeadlineBuffer` is specific to the `temporal-aws-lambda` module. It controls the full shutdown window reserved at the end of the Lambda invocation, including graceful shutdown time, shutdown hooks, and service stub cleanup. The default is `GracefulShutdownTimeout` (5s) + 2s. If you change `GracefulShutdownTimeout` without explicitly setting `ShutdownDeadlineBuffer`, the buffer is recomputed as `GracefulShutdownTimeout` + 2s. If you explicitly set `ShutdownDeadlineBuffer`, it must be greater than or equal to `GracefulShutdownTimeout`. If your Worker handles long-running Activities, increase `GracefulShutdownTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities). ## Add observability with OpenTelemetry The `OtelLambdaWorkerConfigurationHelper` class provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. The underlying metrics and traces are the same ones the Java SDK emits in any environment. For general observability concepts and the full list of available metrics, see [Observability - Java SDK](/develop/java/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). > **⚠️ Caution:** > > Traces are currently produced through a compatibility shim, not native OpenTelemetry support. The trace structure will change in a future release. Metrics are not affected. > ```java import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.RequestHandler; import io.temporal.aws.lambda.LambdaWorker; import io.temporal.aws.lambda.OtelLambdaWorkerConfigurationHelper; import io.temporal.common.WorkerDeploymentVersion; public final class Handler implements RequestHandler { private static final RequestHandler WORKER = LambdaWorker.define( new WorkerDeploymentVersion("my-app", "build-1"), builder -> { OtelLambdaWorkerConfigurationHelper.configure(builder); builder.setTaskQueue("serverless-task-queue-1"); builder.registerWorkflowImplementationTypes(SampleWorkflowImpl.class); builder.registerActivitiesImplementations(new SampleActivitiesImpl()); }); @Override public Void handleRequest(Object input, Context context) { return WORKER.handleRequest(input, context); } } ``` `OtelLambdaWorkerConfigurationHelper.configure` configures OpenTelemetry with OTLP trace and metric exporters, uses AWS X-Ray-compatible trace ID generation, installs an OpenTelemetry-backed metrics scope, and registers per-invocation flush hooks. By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. To collect this telemetry, attach the [ADOT Collector layer](https://aws-otel.github.io/docs/getting-started/lambda) to your Lambda function. Java does not need a language-specific ADOT layer because the OTel SDK is included as a dependency of the module. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle the following `otel-collector-config.yaml` in your Lambda deployment package: [lambda-worker/otel-collector-config.template.yaml](https://github.com/temporalio/samples-java/blob/main/lambda-worker/otel-collector-config.template.yaml) ```yaml receivers: otlp: protocols: grpc: endpoint: "localhost:4317" http: endpoint: "localhost:4318" exporters: debug: awsxray: region: ${env:AWS_REGION} awsemf: namespace: TemporalWorkerMetrics log_group_name: /aws/lambda/${env:AWS_LAMBDA_FUNCTION_NAME} region: ${env:AWS_REGION} dimension_rollup_option: NoDimensionRollup resource_to_telemetry_conversion: enabled: true service: pipelines: traces: receivers: [otlp] exporters: [awsxray, debug] metrics: receivers: [otlp] exporters: [awsemf] telemetry: logs: level: info metrics: address: localhost:8888 ``` Set the following environment variable on the Lambda function to point the Collector at the bundled config: - `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` Enable X-Ray active tracing on the Lambda function: ```bash aws lambda update-function-configuration \ --function-name \ --tracing-config Mode=Active ``` The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions to the execution role. Without these permissions, the Collector fails silently and no telemetry appears. If you only need metrics or tracing, use `OtelLambdaWorkerConfigurationHelper.configureMetrics`, `OtelLambdaWorkerConfigurationHelper.configureTracing`, or `OtelLambdaWorkerConfigurationHelper.configureFlushHook` individually. To use an application-owned OpenTelemetry provider, pass a customizer to the two-argument `configure` overload: `OtelLambdaWorkerConfigurationHelper.configure(builder, b -> b.setOpenTelemetry(openTelemetry))`. `setOpenTelemetry` is defined on the helper's own builder, not on `LambdaWorkerOptions.Builder`. In that path, no exporters are created and the helper only installs the metrics scope, interceptors, and per-invocation flush hook. --- # Serverless Workers on GCP Cloud Run - Java SDK Source: https://docs.temporal.io/develop/java/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the Java SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Java Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific package. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running Java Worker, then set `WorkerDeploymentOptions` on [`WorkerOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.html) to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```java package example; import io.temporal.client.WorkflowClient; import io.temporal.client.WorkflowClientOptions; import io.temporal.common.VersioningBehavior; import io.temporal.common.WorkerDeploymentVersion; import io.temporal.serviceclient.WorkflowServiceStubs; import io.temporal.serviceclient.WorkflowServiceStubsOptions; import io.temporal.worker.Worker; import io.temporal.worker.WorkerDeploymentOptions; import io.temporal.worker.WorkerFactory; import io.temporal.worker.WorkerOptions; String apiKey = System.getenv("TEMPORAL_API_KEY"); WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs( WorkflowServiceStubsOptions.newBuilder() .setTarget(System.getenv("TEMPORAL_ADDRESS")) .setEnableHttps(true) .addApiKey(() -> apiKey) .build()); WorkflowClient client = WorkflowClient.newInstance( service, WorkflowClientOptions.newBuilder() .setNamespace(System.getenv("TEMPORAL_NAMESPACE")) .build()); WorkerFactory factory = WorkerFactory.newInstance(client); Worker worker = factory.newWorker( System.getenv("TEMPORAL_TASK_QUEUE"), WorkerOptions.newBuilder() .setDeploymentOptions( WorkerDeploymentOptions.newBuilder() .setUseVersioning(true) .setVersion(new WorkerDeploymentVersion("my-app", "build-1")) .setDefaultVersioningBehavior(VersioningBehavior.PINNED) .build()) .build()); worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class); worker.registerActivitiesImplementations(new GreetingActivitiesImpl()); factory.start(); ``` The two arguments to `WorkerDeploymentVersion` are the deployment name and the build ID, and together they identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`. Setting `setDefaultVersioningBehavior` as shown above covers every Workflow on the Worker. To set the behavior per Workflow instead, annotate the Workflow method with `@WorkflowVersioningBehavior`: ```java import io.temporal.common.VersioningBehavior; import io.temporal.workflow.WorkflowVersioningBehavior; public class GreetingWorkflowImpl implements GreetingWorkflow { @Override @WorkflowVersioningBehavior(VersioningBehavior.PINNED) public String run(String name) { // ... } } ``` For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/java/workers/run-worker-process). ## Configure the Temporal connection Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext. The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace. `addApiKey` takes a supplier, which the SDK calls on each request. Rotate the key by returning a new value from the supplier instead of restarting the Worker. For TLS client certificates instead of an API key, see [Connect to Temporal Cloud](/develop/java/client/temporal-client). ## Package the Worker image Cloud Run runs one JVM per instance, so give the JVM a heap sized to the instance rather than to the host. Java reads the container's memory limit and defaults the maximum heap to a quarter of it, which leaves most of a small instance unused. Set `-XX:MaxRAMPercentage` to raise that share: ```dockerfile CMD ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/worker.jar"] ``` A Cloud Run Worker Pool defaults to 512 MiB per instance. Raise `--memory` when you create the pool if your Worker needs more. ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/java/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over: ```java public class GreetingActivitiesImpl implements GreetingActivities { @Override public String process(List items) { for (int i = 0; i < items.size(); i++) { Activity.getExecutionContext().heartbeat(i); // ... process items.get(i) } return "done"; } } ``` For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Java SDK](/develop/java/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). --- # Workflows - Java SDK Source: https://docs.temporal.io/develop/java/workflows > This section covers Workflows with the Java SDK ![Java SDK Banner](/img/assets/banner-java-temporal.png) ## Workflows - [Workflow basics](/develop/java/workflows/basics) - [Child Workflows](/develop/java/workflows/child-workflows) - [Continue-As-New](/develop/java/workflows/continue-as-new) - [Message passing](/develop/java/workflows/message-passing) - [Cancellation](/develop/java/workflows/cancellation) - [Timeouts](/develop/java/workflows/timeouts) - [Schedules](/develop/java/workflows/schedules) - [Timers](/develop/java/workflows/timers) - [Side effects](/develop/java/workflows/side-effects) - [Versioning](/develop/java/workflows/versioning) - [Workflow Streams](/develop/java/workflows/workflow-streams) --- # Workflow basics - Java SDK Source: https://docs.temporal.io/develop/java/workflows/basics > This section explains how to implement Workflows with the Java SDK ## How to develop a Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal Java SDK programming model, a Workflow Definition comprises a Workflow interface annotated with `@WorkflowInterface` and a Workflow implementation that implements the Workflow interface. The Workflow interface is a Java interface and is annotated with `@WorkflowInterface`. Each Workflow interface must have only one method annotated with `@WorkflowMethod`. ```java // Workflow interface @WorkflowInterface public interface YourWorkflow { @WorkflowMethod String yourWFMethod(Arguments args); } ``` However, when using dynamic Workflows, do not specify a `@WorkflowMethod`, and implement the `DynamicWorkflow` directly in the Workflow implementation code. The `@WorkflowMethod` identifies the method that is the starting point of the Workflow Execution. The Workflow Execution completes when this method completes. You can create [interface inheritance hierarchies](#interface-inheritance) to reuse components across other Workflow interfaces. The interface inheritance approach does not apply to `@WorkflowMethod` annotations. A Workflow implementation implements a Workflow interface. ```java // Define the Workflow implementation which implements our getGreeting Workflow method. public static class GreetingWorkflowImpl implements GreetingWorkflow { ... } ``` To call Activities in your Workflow, call the Activity implementation. Use `ExternalWorkflowStub` to start or send Signals from within a Workflow to other running Workflow Executions. You can also invoke other Workflows as Child Workflows with `Workflow.newChildWorkflowStub()` or `Workflow.newUntypedChildWorkflowStub()` within a Workflow Definition. ## Workflow interface inheritance Workflow interfaces can form inheritance hierarchies. It may be useful for creating reusable components across multiple Workflow interfaces. For example imagine a UI or CLI button that allows a `retryNow` Signal on any Workflow. To implement this feature you can redesign an interface like the following: ```java public interface Retryable { @SignalMethod void retryNow(); } @WorkflowInterface public interface FileProcessingWorkflow extends Retryable { @WorkflowMethod String processFile(Arguments args); @QueryMethod(name="history") List getHistory(); @QueryMethod String getStatus(); @SignalMethod void abandon(); } ``` Then some other Workflow interface can extend just `Retryable`, for example: ```java @WorkflowInterface public interface MediaProcessingWorkflow extends Retryable { @WorkflowMethod String processBlob(Arguments args); } ``` Now if we have two running Workflows, one that implements the `FileProcessingWorkflow` interface and another that implements the `MediaProcessingWorkflow` interface, we can Signal to both using their common interface and knowing their WorkflowIds, for example: ```java Retryable r1 = client.newWorkflowStub(Retryable.class, firstWorkflowId); Retryable r2 = client.newWorkflowStub(Retryable.class, secondWorkflowId); r1.retryNow(); r2.retryNow(); ``` The same technique can be used to query Workflows using a base Workflow interface. Note that this approach does not apply to `@WorkflowMethod` annotations, meaning that when using a base interface, it should not include any `@WorkflowMethod` methods. To illustrate this, let's say that we define the following **invalid** code: ```java // INVALID CODE! public interface BaseWorkflow { @WorkflowMethod void retryNow(); } @WorkflowInterface public interface Workflow1 extends BaseWorkflow {} @WorkflowInterface public interface Workflow2 extends BaseWorkflow {} ``` Any attempt to register both implementations with the Worker will fail. Let's say that we have: ```java worker.registerWorkflowImplementationTypes( Workflow1Impl.class, Workflow2Impl.class); ``` This registration will fail with: ```text java.lang.IllegalStateException: BaseWorkflow workflow type is already registered with the worker ``` ## Define Workflow parameters Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. A method annotated with `@WorkflowMethod` can have any number of parameters. We recommend passing a single parameter that contains all the input fields to allow for adding fields in a backward-compatible manner. Note that all inputs should be serializable by the default Jackson JSON Payload Converter. You can create a custom object and pass it to the Workflow method, as shown in the following example. ```java //... @WorkflowInterface public interface YourWorkflow { @WorkflowMethod String yourWFMethod(CustomObj customobj); // ... } ``` ## Define Workflow return parameters Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error. Workflow method arguments and return values must be serializable and deserializable using the provided [`DataConverter`](https://www.javadoc.io/static/io.temporal/temporal-sdk/1.17.0/io/temporal/common/converter/DataConverter.html). The `execute` method for `DynamicWorkflow` can return type Object. Ensure that your Client can handle an Object type return or is able to convert the Object type response. Related references: - [Data Converter](/dataconversion) - [Java DataConverter reference](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html) ## Customize your Workflow Type Workflows have a Type that is referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. The Workflow Type defaults to the short name of the Workflow interface. In the following example, the Workflow Type defaults to `NotifyUserAccounts`. ```java @WorkflowInterface public interface NotifyUserAccounts { @WorkflowMethod void notify(String[] accountIds); } ``` To overwrite this default naming and assign a custom Workflow Type, use the `@WorkflowMethod` annotation with the `name` parameter. In the following example, the Workflow Type is set to `your-workflow`. ```java @WorkflowInterface public interface NotifyUserAccounts { @WorkflowMethod(name = "your-workflow") void notify(String[] accountIds); } ``` When you set the Workflow Type this way, the value of the `name` parameter does not have to start with an uppercase letter. ## Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `@WorkflowInit` annotation on your constructor, you can give it the same [Workflow parameters](/develop/java/workflows/basics#workflow-parameters) as your `@WorkflowMethod`. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/java/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `@WorkflowMethod` method. That always happens, whether or not you use the `@WorkflowInit` annotation. Here's an example. Notice that the constructor and `getGreeting` must have the same parameters: ```java public class GreetingExample { @WorkflowInterface public interface GreetingWorkflow { @WorkflowMethod String getGreeting(String input); } public static class GreetingWorkflowImpl implements GreetingWorkflow { private final String nameWithTitle; private boolean titleHasBeenChecked; ... // Note the annotation is on a public constructor @WorkflowInit public GreetingWorkflowImpl(String input) { this.nameWithTitle = "Knight " + input; this.titleHasBeenChecked = false; } @Override public String getGreeting(String input) { Workflow.await(() -> titleHasBeenChecked) return "Hello " + nameWithTitle; } } } ``` ## Workflow logic requirements Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code. When defining Workflows using the Temporal Java SDK, the Workflow code must be written to execute effectively once and to completion. The following constraints apply when writing Workflow Definitions: - Do not use mutable global variables in your Workflow implementations. This will ensure that multiple Workflow instances are fully isolated. - Workflow code must be deterministic. If you need to call non-deterministic functions (such as non-seeded random or `UUID.randomUUID()`) directly from the Workflow code, the Temporal SDK provides replay-safe replacements. See [Random numbers and UUIDs](#random-numbers-and-uuids). - For operations like calling external APIs, invoking LLMs, querying databases, or performing I/O, use Activities. Activities run outside Workflow replay and are retried reliably. - Use Temporal-provided functions instead of that rely on system time. For example, use only `Workflow.currentTimeMillis()` to get the current time inside a Workflow. - Use `Async.function` or `Async.procedure`, provided by the Temporal SDK, to execute code asynchronously instead of native Java `Thread` or any other multi-threaded classes like `ThreadPoolExecutor`. - Use only the concurrency features provided by the Workflow class. Multi-threaded code inside a Workflow is executed one thread at a time and under a global lock, so there is no need for explicit synchronization. - Call `Workflow.sleep` instead of `Thread.sleep`. - Use `Promise` and `CompletablePromise` instead of `Future` and `CompletableFuture`. - Use `WorkflowQueue` instead of `BlockingQueue`. - Use `Workflow.getVersion` when making any changes to the Workflow code to avoid deployment of updated Workflow code interfering with already-running Workflows. - Do not access configuration APIs directly from a Workflow because changes in the configuration might affect a Workflow Execution path. Instead, pass it as an argument to a Workflow function or use an Activity to load it. - Use `DynamicWorkflow` when you need a default Workflow that can handle all Workflow Types that are not registered with a Worker. A single implementation can implement a Workflow Type which by definition is dynamically loaded from some external source. All standard `WorkflowOptions` and determinism rules apply to Dynamic Workflow implementations. The SDK provides replay-safe alternatives for common needs. ### Logging Use [`Workflow.getLogger()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) instead of `System.out.println` or a logger you create yourself. The SDK logger skips log messages during replay to avoid duplicates: ```java public class MyWorkflowImpl implements MyWorkflow { private static final Logger logger = Workflow.getLogger(MyWorkflowImpl.class); @Override public String execute(String name) { logger.info("Starting workflow for {}", name); // ... } } ``` For logger configuration, see [Observability: Log from a Workflow](/develop/java/platform/observability#logging). ### Random numbers and UUIDs Use [`Workflow.newRandom()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) to get a `Random` instance seeded per Workflow Execution, and [`Workflow.randomUUID()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) instead of `UUID.randomUUID()`: ```java int value = Workflow.newRandom().nextInt(100); UUID uniqueId = Workflow.randomUUID(); ``` ### Current time Use [`Workflow.currentTimeMillis()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) instead of `System.currentTimeMillis()` or `Instant.now()`. It returns the time of the last Workflow Task, which is consistent across replays: ```java long currentTime = Workflow.currentTimeMillis(); ``` To wait, use [`Workflow.sleep()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html) instead of `Thread.sleep()`. ### Detecting replay (advanced) Use [`WorkflowUnsafe.isReplaying()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/unsafe/WorkflowUnsafe.html) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an Interceptor. `Workflow.isReplaying()` is deprecated in favor of this method. > **⚠️ Caution:** > > Never use this to affect Workflow business logic. Branching on replay status breaks determinism. > ```java import io.temporal.workflow.unsafe.WorkflowUnsafe; if (!WorkflowUnsafe.isReplaying()) { emitMetric("workflow_started", 1); } ``` Java Workflow reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html) --- # Interrupt a Workflow Execution - Java SDK Source: https://docs.temporal.io/develop/java/workflows/cancellation You can interrupt a Workflow Execution in one of the following ways: - [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution. - [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. ## Cancel a Workflow Execution Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a process. - The system records a `WorkflowExecutionCancelRequested` event in the Event History. - A Workflow Task gets scheduled to process the cancelation. - The Workflow code can handle the cancelation and execute any cleanup logic. - The system doesn't forcefully stop the Workflow. To cancel a Workflow Execution in Java, use the [cancel()]() function on the WorkflowStub. ```java WorkflowStub workflowStub = WorkflowStub.fromTyped(workflow); workflowStub.cancel(); ``` ## Cancellation scopes In the Java SDK, Workflows are represented internally by a tree of cancellation scopes, each with cancellation behaviors you can specify. By default, everything runs in the "root" scope. Scopes are created using the [Workflow.newCancellationScope]() constructor Cancellations are applied to cancellation scopes, which can encompass an entire Workflow or just part of one. Scopes can be nested, and cancellation propagates from outer scopes to inner ones. A Workflow's method runs in the outermost scope. Cancellations are handled by catching `CanceledFailure`s thrown by cancelable operations. You can also use the following APIs: - `CancellationScope.current()`: Get the current scope. - `scope.cancel()`: Cancel all operations inside a `scope`. - `scope.getCancellationRequest()`: A promise that resolves when a scope cancellation is requested, such as when Workflow code calls `cancel()` or the entire Workflow is cancelled by an external client. When a `CancellationScope` is cancelled, it propagates cancellation in any child scopes and of any cancelable operations created within it, such as the following: - Activities - Timers (created with the [sleep]() function) - Child Workflows - Nexus Operations ### Cancel with Workflow error When you want to interrupt a Workflow Execution using an error, throw the [`DestroyWorkflowThreadError`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/DestroyWorkflowThreadError.html). Make sure there isn't any code on that thread to catch this error or it could cause issues with the Temporal service. More generally, Workflow code should only ever catch `Exception`, never `Throwable` or `Error` — see [Catch Exception, never Throwable or Error](/develop/java/best-practices/error-handling#catch-exception-not-throwable). ### Cancel an Activity from a Workflow Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate Activity is executed, the Activity Execution should send Heartbeats and set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is still working. When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity to appear canceled the exception needs to be re-raised. > **📝 Note:** > > Unlike regular Activities, [Local Activities](/local-activity) currently do not support cancellation. > To cancel an Activity from a Workflow Execution, call the [cancel()]() method on the CancellationScope that the activity was started in. ```java public class GreetingWorkflowImpl implements GreetingWorkflow { @Override public String getGreeting(String name) { List> results = new ArrayList<>(greetings.length); /* * Create our CancellationScope. Within this scope we call the workflow activity * composeGreeting method asynchronously for each of our defined greetings in different * languages. */ CancellationScope scope = Workflow.newCancellationScope( () -> { for (String greeting : greetings) { results.add(Async.function(activities::composeGreeting, greeting, name)); } }); /* * Execute all activities within the CancellationScope. Note that this execution is * non-blocking as the code inside our cancellation scope is also non-blocking. */ scope.run(); // We use "anyOf" here to wait for one of the activity invocations to return String result = Promise.anyOf(results).get(); // Trigger cancellation of all uncompleted activity invocations within the cancellation scope scope.cancel(); /* * Wait for all activities to perform cleanup if needed. * For the sake of the example we ignore cancellations and * get all the results so that we can print them in the end. * * Note that we cannot use "allOf" here as that fails on any Promise failures */ for (Promise activityResult : results) { try { activityResult.get(); } catch (ActivityFailure e) { if (!(e.getCause() instanceof CanceledFailure)) { throw e; } } } return result; } } ``` ## Terminate a Workflow Execution Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. To terminate a Workflow Execution in Java, use the [terminate()]() function on the WorkflowStub. ```java WorkflowStub untyped = WorkflowStub.fromTyped(myWorkflowStub); untyped.terminate("Sample reason"); ``` ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - Java SDK Source: https://docs.temporal.io/develop/java/workflows/child-workflows > Start a Child Workflow Execution and set a Parent Close Policy using the Java SDK. Manage Child Workflow Events and ensure successful execution. This page shows how to do the following: - [Start a Child Workflow Execution](#start-child-workflow) - [Set a Parent Close Policy](#parent-close-policy) ## Start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In Java, you must explicitly call `Workflow.getWorkflowExecution(child)` to get a `Promise`, then call `.get()` on that Promise to wait for this Event. See the [Parent Close Policy](#parent-close-policy) section below for a complete example. ### Async Child Workflows The first call to the Child Workflow stub must always be its Workflow method (method annotated with `@WorkflowMethod`). Similar to Activities, invoking Child Workflow methods can be made synchronous or asynchronous by using `Async#function` or `Async#procedure`. The synchronous call blocks until a Child Workflow method completes. The asynchronous call returns a `Promise` which can be used to wait for the completion of the Child Workflow method, as in the following example: ```java GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class); Promise greeting = Async.function(child::composeGreeting, "Hello", name); // ... greeting.get() ``` To execute an untyped Child Workflow asynchronously, call `executeAsync` on the `ChildWorkflowStub`, as shown in the following example. ```java //... ChildWorkflowStub childUntyped = Workflow.newUntypedChildWorkflowStub( "GreetingChild", // your workflow type ChildWorkflowOptions.newBuilder().setWorkflowId("childWorkflow").build()); Promise greeting = childUntyped.executeAsync(String.class, String.class, "Hello", name); String result = greeting.get(); //... ``` The following examples show how to spawn a Child Workflow: - Spawn a Child Workflow from a Workflow ```java // Child Workflow interface @WorkflowInterface public interface GreetingChild { @WorkflowMethod String composeGreeting(String greeting, String name); } // Child Workflow implementation not shown // Parent Workflow implementation public class GreetingWorkflowImpl implements GreetingWorkflow { @Override public String getGreeting(String name) { GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class); // This is a blocking call that returns only after child has completed. return child.composeGreeting("Hello", name ); } } ``` - Spawn two Child Workflows (with the same type) in parallel: ```java // Parent Workflow implementation public class GreetingWorkflowImpl implements GreetingWorkflow { @Override public String getGreeting(String name) { // Workflows are stateful, so a new stub must be created for each new child. GreetingChild child1 = Workflow.newChildWorkflowStub(GreetingChild.class); Promise greeting1 = Async.function(child1::composeGreeting, "Hello", name); // Both children will run concurrently. GreetingChild child2 = Workflow.newChildWorkflowStub(GreetingChild.class); Promise greeting2 = Async.function(child2::composeGreeting, "Bye", name); // Do something else here. ... return "First: " + greeting1.get() + ", second: " + greeting2.get(); } } ``` - Send a Signal to a Child Workflow from the parent: ```java // Child Workflow interface @WorkflowInterface public interface GreetingChild { @WorkflowMethod String composeGreeting(String greeting, String name); @SignalMethod void updateName(String name); } // Parent Workflow implementation public class GreetingWorkflowImpl implements GreetingWorkflow { @Override public String getGreeting(String name) { GreetingChild child = Workflow.newChildWorkflowStub(GreetingChild.class); Promise greeting = Async.function(child::composeGreeting, "Hello", name); child.updateName("Temporal"); return greeting.get(); } } ``` - Sending a Query to Child Workflows from within the parent Workflow code is not supported. However, you can send a Query to Child Workflows from Activities using `WorkflowClient`. Related reads: - [How to develop a Workflow Definition](/develop/java/workflows/basics) - Java Workflow reference: [https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/package-summary.html) ## Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. Set [Parent Close Policy](/parent-close-policy) on an instance of `ChildWorkflowOptions` using [`ChildWorkflowOptions.newBuilder().setParentClosePolicy`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/ChildWorkflowOptions.Builder.html). - Type: `ChildWorkflowOptions.Builder` - Default: `PARENT_CLOSE_POLICY_TERMINATE` ```java public void parentWorkflow() { ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder() .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON) .build(); MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, options); Async.procedure(child::, ...); Promise childExecution = Workflow.getWorkflowExecution(child); // Wait for child to start childExecution.get() } ``` In this example, we are: 1. Setting `ChildWorkflowOptions.ParentClosePolicy` to `ABANDON` when creating a Child Workflow stub. 2. Starting Child Workflow Execution asynchronously using `Async.function` or `Async.procedure`. 3. Calling `Workflow.getWorkflowExecution(…)` on the child stub. 4. Waiting for the `Promise` returned by `getWorkflowExecution` to complete. This indicates whether the Child Workflow started successfully (or failed). 5. Completing parent Workflow Execution asynchronously. Steps 3 and 4 are needed to ensure that a Child Workflow Execution starts before the parent closes. If the parent initiates a Child Workflow Execution and then completes immediately after, the Child Workflow will never execute. --- # Continue-As-New - Java SDK Source: https://docs.temporal.io/develop/java/workflows/continue-as-new > Use Temporal's Continue-As-New in Java to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for Java developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) - [How to test Continue-as-New?](#how-to-test) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the Java SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically set to `None` for the original caller of the Workflow. [View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflow.java) in the context of the rest of the application code. ```java class ClusterManagerInput { private final Optional state; private final boolean testContinueAsNew; } @WorkflowMethod ClusterManagerResult run(ClusterManagerInput input); ```` The test hook in the above snippet is covered [below](#how-to-test). Inside your Workflow, call the [`continueAsNew()`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#continueAsNew(io.temporal.workflow.ContinueAsNewOptions,java.lang.Object...)) function with the same type. This stops the Workflow right away and starts a new one. [View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflowImpl.java) in the context of the rest of the application code. ```java Workflow.continueAsNew( new ClusterManagerInput(Optional.of(state), input.isTestContinueAsNew())); ```` ### Considerations for Workflows with Message Handlers If you use Updates or Signals, don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish in your main Workflow before you run `continueAsNew`. ## When is it right to Continue-as-New using the Java SDK? Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `Workflow.getInfo().isContinueAsNewSuggested()` to check if it's time. ## How to test Continue-as-New using the Java SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, when `testContinueAsNew == true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value. A helper method in the Workflow checks it each time it considers using Continue-as-New: [View the source code](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/safemessagepassing/ClusterManagerWorkflowImpl.java) in the context of the rest of the application code. ```java private boolean shouldContinueAsNew() { if (Workflow.getInfo().isContinueAsNewSuggested()) { return true; } // This is just for ease-of-testing. In production, we trust temporal to tell us when to // continue as new. if (maxHistoryLength > 0 && Workflow.getInfo().getHistoryLength() > maxHistoryLength) { return true; } return false; } ``` --- # Workflow message passing - Java SDK Source: https://docs.temporal.io/develop/java/workflows/message-passing > Develop with Queries, Signals, and Updates with the Temporal Java SDK. A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates. The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values. Temporal Clients use messages to read Workflow state and control execution. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal Java SDK. ## Write message handlers Follow these guidelines when writing your message handlers: - Message handlers are defined as methods on the Workflow class, using one of the three annotations: [`@QueryMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/QueryMethod.html), [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html), and [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html). - The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion). - Prefer a single class with multiple fields over using multiple input parameters. A class allows you to add fields without changing the calling signature. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution: ```java public class MessagePassingIntro { public enum Language { CHINESE, ENGLISH, FRENCH, SPANISH, PORTUGUESE, } public static class GetLanguagesInput { public boolean includeUnsupported; public GetLanguagesInput() { this.includeUnsupported = false; } public GetLanguagesInput(boolean includeUnsupported) { this.includeUnsupported = includeUnsupported; } } @WorkflowInterface public interface GreetingWorkflow { ... // 👉 Use the @QueryMethod annotation to define a Query handler in the // Workflow interface. @QueryMethod List getLanguages(GetLanguagesInput input); } public static class GreetingWorkflowImpl implements GreetingWorkflow { ... @Override public List getLanguages(GetLanguagesInput input) { // 👉 The Query handler returns a value: it must not mutate the Workflow state // or perform blocking operations. if (input.includeUnsupported) { return Arrays.asList(Language.values()); } else { return new ArrayList(greetings.keySet()); } } } } ``` - A Query handler must not modify Workflow state. - You can't perform blocking operations such as executing an Activity in a Query handler. - The Query annotation accepts an argument (`name`) as described in the API reference docs for [`@QueryMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/QueryMethod.html). ### Signal handlers A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow: ```java public class MessagePassingIntro { public static class ApproveInput { private String name; public ApproveInput() {} public ApproveInput(String name) { this.name = name; } } @WorkflowInterface public interface GreetingWorkflow { ... // 👉 Use the @SignalMethod annotation to define a Signal handler in the // Workflow interface. @SignalMethod void approve(ApproveInput input); } public static class GreetingWorkflowImpl implements GreetingWorkflow { ... private Boolean approvedForRelease; private String approverName; @Override public void approve(ApproveInput input) { // 👉 The Signal handler mutates the Workflow state but cannot return a value. this.approvedForRelease = true; this.approverName = input.name; } } } ``` - The handler should not return a value. The response is sent immediately from the server, without waiting for the Workflow to process the Signal. - The Signal annotation accepts arguments (`name`, and `unfinished_policy`) as described in the API reference docs for [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html). - Signal (and Update) handlers can be blocking. This allows you to use Activities, Child Workflows, durable [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)), and more. See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using blocking Signal and Update handlers. ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```java public class MessagePassingIntro { @WorkflowInterface public interface GreetingWorkflow { ... // 👉 Use the @UpdateMethod annotation to define an Update handler in the // Workflow interface. @UpdateMethod Language setLanguage(Language language); // 👉 Update validators are optional @UpdateValidatorMethod(updateName = "setLanguage") void setLanguageValidator(Language language); } public static class GreetingWorkflowImpl implements GreetingWorkflow { ... @Override public Language setLanguage(Language language) { // 👉 The Update handler can mutate the Workflow state and return a value. Language previousLanguage = this.language; this.language = language; return previousLanguage; } @Override public void setLanguageValidator(Language language) { // 👉 The Update validator performs validation but cannot mutate the Workflow state. if (!greetings.containsKey(language)) { throw new IllegalArgumentException("Unsupported language: " + language); } } } } ``` - The Update annotation accepts arguments (`name`, and `unfinished_policy`) as described in the API reference docs for [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html). - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them. - Define an Update validator with the `@UpdateValidatorMethod` annotation. Use the `updateName` argument when declaring the validator to connect it to its Update. The validator must return `void` and accept the same argument types as the handler. - Accepting and rejecting Updates with validators: - To reject an Update, throw an exception of any type in the validator. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic. - When a Validator throws an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History. The caller receives an "Update failed" error. - Use [`getCurrentUpdateInfo`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/internal/sync/WorkflowInternal.html#getCurrentUpdateInfo()) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - Signal (and Update) handlers can be blocking, letting them use Activities, Child Workflows, durable [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)) conditions, and more. See [Blocking handlers](#blocking-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines. ## Send messages To send Queries, Signals, or Updates you call methods on a `WorkflowInterface`, often called the "WorkflowStub." Use [newWorkflowStub](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newWorkflowStub(java.lang.Class,io.temporal.client.WorkflowOptions)) to obtain the WorkflowStub. For example: ```java WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkflowOptions workflowOptions = WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).setWorkflowId(WORKFLOW_ID).build(); // Create the workflow client stub. It is used to start the workflow execution. GreetingWorkflow workflow = client.newWorkflowStub(GreetingWorkflow.class, workflowOptions); // Start workflow asynchronously and call its getGreeting workflow method WorkflowClient.start(workflow::getGreetings); ``` To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition. > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution. > ### Send a Query Call a Query method defined within a Workflow from a `WorkflowStub` created in Client code to send a Query to a Workflow Execution: ```java List languages = workflow.getLanguages(new GetLanguagesInput(false)); System.out.println("Supported languages: " + languages); ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### Send a Signal from a Client To send a Signal from Client code, call a Signal method on the WorkflowStub: ```java workflow.approve(new ApproveInput("Me")); ``` - The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution. - The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History. #### Send a Signal from a Workflow A Workflow can send a Signal to another Workflow, known as an _External Signal_. Use [`Workflow.newExternalWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newExternalWorkflowStub(java.lang.Class,io.temporal.api.common.v1.WorkflowExecution)) in your _current_ Workflow to create an `ExternalWorkflowStub` for the other Workflow. Call Signal methods on the external stub to Signal the other Workflow: ```java OtherWorkflow other = Workflow.newExternalWorkflowStub(OtherWorkflow.class, otherWorkflowID); other.mySignalMethod(); ``` When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. #### Signal-With-Start Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running. If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled. To use Signal-With-Start, call `signalWithStart` and pass the name of your Signal with its arguments: ```java public static void signalWithStart() { // WorkflowStub is a client-side stub to a single Workflow instance WorkflowStub untypedWorkflowStub = client.newUntypedWorkflowStub("GreetingWorkflow", WorkflowOptions.newBuilder() .setWorkflowId(workflowId) .setTaskQueue(taskQueue) .build()); untypedWorkflowStub.signalWithStart("setCustomer", new Object[] {customer2}, new Object[] {customer1}); String greeting = untypedWorkflowStub.getResult(String.class); } ``` Here's the `WorkflowInterface` for the previous example. When using Signal-With-Start, the Signal handler (`setCustomer`) will be executed before the Workflow method (`greet`). ```java @WorkflowInterface public interface GreetingWorkflow { @WorkflowMethod String greet(Customer customer); @SignalMethod void setCustomer(Customer customer); @QueryMethod Customer getCustomer(); } ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A Client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. To send an Update to a Workflow Execution, you can: - Call the Update method on a WorkflowStub in Client code and wait for the Update to complete. This code fetches an Update result: ```java Language previousLanguage = workflow.setLanguage(Language.CHINESE); ``` - Send [`startUpdate`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdate(io.temporal.client.UpdateOptions,java.lang.Object...)) to receive an [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) as soon as the Update is accepted or rejected. - Use this `WorkflowUpdateHandle` later to fetch your results. - Blocking Update handlers normally perform long-running asynchronous operations. - `startUpdate` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete. For example: ```java WorkflowUpdateHandle handle = WorkflowStub.fromTyped(workflow) .startUpdate( "setLanguage", WorkflowUpdateStage.ACCEPTED, Language.class, Language.ENGLISH); previousLanguage = handle.getResultAsync().get(); ``` For more details, see the "Blocking handlers" section. To obtain an Update handle, you can: - Use [`startUpdate`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdate(io.temporal.client.UpdateOptions,java.lang.Object...)) to start an Update and return the handle, as shown in the preceding example. - Use [`getUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#getUpdateHandle(java.lang.String,java.lang.Class)) to fetch a handle for an in-progress Update using the Update ID and Workflow ID. You can use the `WorkflowUpdateHandle` to obtain information about the update: - `getExecution()`: Returns the Workflow Execution that this Update was sent to. - `getId()`: Returns the Update's unique ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - `getResultAsync()`: Returns a `CompletableFuture` which can be used to wait for the Update to complete. #### Update-With-Start > **💡 Tip:** > > For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended. > [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/java/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. Use the [`startUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#startUpdateWithStart(io.temporal.workflow.Functions.Func,io.temporal.client.UpdateOptions,io.temporal.client.WithStartWorkflowOperation)) WorkflowClient API. It returns once the requested Update wait stage has been reached; or when the request times out. Use the [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) to retrieve a result from the Update. You will need to provide: - WorkflowStub created from [`WorkflowOptions`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.html). The `WorkflowOptions` require [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) to be specified. Choose "Use Existing" and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure. Not all `WorkflowOptions` are allowed. For example, specifying a Cron Schedule will result in an error. - [`UpdateOptions`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/UpdateOptions.html). Same as for [Update Workflow](/develop/java/workflows/message-passing#send-update-from-client), the update wait stage must be specified. For Update-with-Start, the Workflow Id is optional. When specified, the Id must match the one used in `WorkflowOptions`. Since a running Workflow Execution may not already exist, you can't set a Run Id. - [`WithStartWorkflowOperation`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WithStartWorkflowOperation.html). Specify the workflow method. Note that a `WithStartWorkflowOperation` can only be used once. Re-using a previously used operation returns an error from `startUpdateWithStart`. For example: ```java WorkflowUpdateHandle handle = WorkflowClient.startUpdateWithStart( workflow::setLanguage, Language.ENGLISH, UpdateOptions.newBuilder().setWaitForStage(WorkflowUpdateStage.ACCEPTED).build(), new WithStartWorkflowOperation<>(workflow::getGreetings)); Language previousLanguage = handle.getResultAsync().get(); ``` To obtain the update result directly, use the [`executeUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#executeUpdateWithStart(io.temporal.workflow.Functions.Func,io.temporal.client.UpdateOptions,io.temporal.client.WithStartWorkflowOperation)) WorkflowClient API. It returns once the update result is available; or when the API call times out. The update wait stage on the `UpdateOptions` is optional. When specified, it must be `WorkflowUpdateStage.COMPLETED`. For example: ```java Language previousLanguage = WorkflowClient.executeUpdateWithStart( workflow::setLanguage, Language.ENGLISH, UpdateOptions.newBuilder().build(), new WithStartWorkflowOperation<>(workflow::getGreetings)); ``` For more examples, see the [Java sample for early-return pattern](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/earlyreturn). > **ℹ️ Info:** > NON-TYPE SAFE API CALLS > > In real-world development, sometimes you may be unable to import Workflow Definition method signatures. > When you don't have access to the Workflow Definition or it isn't written in Java, you can use these non-type safe APIs to obtain an untyped WorkflowStub: > > - [`WorkflowClient.newUntypedWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowClient.html#newUntypedWorkflowStub(java.lang.String,io.temporal.client.WorkflowOptions)) > - [`Workflow.newUntypedExternalWorkflowStub`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newUntypedExternalWorkflowStub(java.lang.String)). > > Pass method names instead of method objects to: > > - [`WorkflowStub.query`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#query(java.lang.String,java.lang.Class,java.lang.Object...)) > - [`WorkflowStub.signal`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#signal(java.lang.String,java.lang.Object...)) > - [`WorkflowStub.update`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#update(java.lang.String,java.lang.Class,java.lang.Object...)) > - [`WorkflowStub.startUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#startUpdateWithStart(io.temporal.client.UpdateOptions,java.lang.Object%5B%5D,java.lang.Object%5B%5D)) > - [`WorkflowStub.executeUpdateWithStart`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html#executeUpdateWithStart(io.temporal.client.UpdateOptions,java.lang.Object%5B%5D,java.lang.Object%5B%5D)) > ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). > ### Do blocking operations in handlers Signal and Update handlers can block. This allows you to use [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)), Activities, Child Workflows, [`Workflow.sleep`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep(java.time.Duration)) Timers, etc. This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls. It's essential to understand the things that could go wrong in order to use blocking handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of blocking Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below. The following code modifies the Update handler from earlier on in this page. The Update handler now makes a blocking call to execute an Activity: ```java public static class GreetingWorkflowImpl implements GreetingWorkflow { ... @Override public Language setLanguage(Language language) { if (!greetings.containsKey(language)) { String greeting = activity.greetingService(language); if (greeting == null) { // 👉 An update validator cannot be blocking, so cannot be used to check that the remote // greetingService supports the requested language. Throwing an ApplicationFailure // will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be // added to history. throw ApplicationFailure.newFailure("Greeting service does not support: " + language, "GreetingFailure") } greetings.put(language, greeting); } Language previousLanguage = this.language; this.language = language; return previousLanguage; } } ``` Although a Signal handler can also make blocking calls like this, using an Update handler allows the Client to receive a result or error once the Activity completes. This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, etc. ### Add blocking wait conditions Sometimes, blocking Signal or Update handlers need to meet certain conditions before they should continue. You can use [`Workflow.await`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#await(java.time.Duration,java.util.function.Supplier)) to prevent the code from proceeding until a condition is true. You specify the condition by passing a function that returns `true` or `false`. This is an important feature that helps you control your handler logic. Here are two important use cases for `Workflow.await`: - Waiting in a handler until it is appropriate to continue. - Waiting in the main Workflow until all active handlers have finished. #### Wait for conditions in handlers It's common to use `Workflow.await` in a handler. For example, suppose your Workflow class has a `updateReadyToExecute` method that indicates whether your Update handler should be allowed to start executing. You can use `workflow.wait_condition` in the handler to make the handler pause until the condition is met: ```java @Override public String setLanguage(UpdateInput input) { Workflow.await(() -> this.updateReadyToExecute(input)); ... } ``` Remember: handlers can execute before the main Workflow method starts. You can also use `Workflow.await` anywhere else in the handler to wait for a specific condition to become true. This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true. #### Ensure your handlers finish before the Workflow completes `Workflow.await` can ensure your handler completes before a Workflow finishes. When your Workflow uses blocking Signal or Update handlers, your main Workflow method can return or Continue-as-New while a handler is still waiting on an async task, such as an Activity. The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying to retrieve Update results. Use `Workflow.await` to wait for [`Workflow.isEveryHandlerFinished`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#isEveryHandlerFinished()) to return `true` to address this problem and allow your Workflow to end smoothly: ```java public class MyWorkflowImpl implements MyWorkflow { ... @Override public String run() { ... Workflow.await(() -> Workflow.isEveryHandlerFinished()); return "workflow-result"; } } ``` By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions. You can silence these warnings on a per-handler basis by passing the `unfinishedPolicy` argument to the [`@SignalMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/SignalMethod.html) / [`@UpdateMethod`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/UpdateMethod.html) annotation: ```java @WorkflowInterface public interface MyWorkflow { ... @UpdateMethod(unfinishedPolicy = HandlerUnfinishedPolicy.ABANDON) void myUpdate(); } ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. ### Use `@WorkflowInit` to operate on Workflow input before any handler executes Normally, your Workflows constructor won't have any parameters. However, if you use the `@WorkflowInit` annotation on your constructor, you can give it the same [Workflow parameters](/develop/java/workflows/basics#workflow-parameters) as your `@WorkflowMethod`. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/java/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `@WorkflowMethod` method -- that always happens, whether or not you use the `@WorkflowInit` annotation. This is useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). > **⚠️ Caution:** > > Do not make blocking calls from within your `@WorkflowInit` method. This could result in your Workflow being incompletely initialized at the start, meaning, for example, that Signal, Query, and Update handler registration would be delayed. > Here's an example. Notice that the constructor and `getGreeting` must have the same parameters: ```java public class GreetingExample { @WorkflowInterface public interface GreetingWorkflow { @WorkflowMethod String getGreeting(String input); @UpdateMethod boolean checkTitleValidity(); } public static class GreetingWorkflowImpl implements GreetingWorkflow { private final String nameWithTitle; private boolean titleHasBeenChecked; ... // Note the annotation is on a public constructor @WorkflowInit public GreetingWorkflowImpl(String input) { this.nameWithTitle = "Sir " + input; this.titleHasBeenChecked = false; } @Override public String getGreeting(String input) { Workflow.await(() -> titleHasBeenChecked) return "Hello " + nameWithTitle; } @Override public boolean checkTitleValidity() { // 👉 The handler is now guaranteed to see the workflow input // after it has been processed by the constructor. boolean isValid = activity.checkTitleValidity(nameWithTitle); titleHasBeenChecked = true; return isValid; } } } ``` ### Use locks to prevent concurrent handler execution Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```java public class DataWorkflowImpl implements DataWorkflow { ... @Override public void badSignalHandler() { Data data = activity.fetchData(); this.x = data.x; // 🐛🐛 Bug!! If multiple instances of this method are executing concurrently, then // there may be times when the Workflow has self.x from one Activity execution and self.y from another. Workflow.sleep(Duration.ofSeconds(1)); this.y = data.y; } } ``` Coordinating access with `WorkflowLock` corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```java public class DataWorkflowImpl implements DataWorkflow { WorkflowLock lock = Workflow.newWorkflowLock(); ... @Override public void safeSignalHandler() { try { lock.lock(); Data data = activity.fetchData(); this.x = data.x; // ✅ OK: the scheduler may switch now to a different handler execution, // or to the main workflow method, but no other execution of this handler // can run until this execution finishes. Workflow.sleep(Duration.ofSeconds(1)); this.y = data.y; } finally { lock.unlock() } } } ``` ## Message handler troubleshooting When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The Client can't contact the server**: You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) on which the `cause` is a [`StatusRuntimeException`](https://grpc.github.io/grpc-java/javadoc/io/grpc/StatusRuntimeException.html) and `status` of `UNAVAILABLE` (after some retries). - **The Workflow does not exist**: You'll receive a [`WorkflowNotFoundException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowNotFoundException.html). See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Java-specific discussion of this topic. ### Problems when sending a Signal When using Signal, the above `WorkflowException`s are the only types of exception that will result from the request. In contrast, for Queries and Updates, the client waits for a response from the Worker. If an issue occurs during the handler execution by the Worker, the Client may receive an exception. ### Problems when sending an Update When working with Updates, you may encounter these errors: - **No Workflow Workers are polling the Task Queue**: Your request will be retried by the SDK Client indefinitely. You can impose a timeout with `CompletableFuture.get()` method with a timeout parameter. This throws a `java.util.concurrent.TimeoutException` exception when it expires. - **Update failed**: You'll receive a `WorkflowUpdateException` exception. There are two ways this can happen: - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler. - The Update failed after having been accepted. Update failures are like [Workflow failures](/references/failures). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity (if the Activity retries have been set to a finite number) - The Workflow author throwing `ApplicationFailure` - Any error listed in [getFailWorkflowExceptionTypes](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkflowImplementationOptions.html#getFailWorkflowExceptionTypes()) (empty by default) - **The handler caused the Workflow Task to fail**: A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) exception. - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use a [`WorkflowUpdateHandle`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowUpdateHandle.html) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) "workflow execution already completed"`. This will happen if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Problems when sending a Query When working with Queries, you may encounter these errors: - **There is no Workflow Worker polling the Task Queue**: You'll receive a [`WorkflowServiceException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowServiceException.html) on which the `cause` is a [`StatusRuntimeException`](https://grpc.github.io/grpc-java/javadoc/io/grpc/StatusRuntimeException.html) with a `status` of `FAILED_PRECONDITION`. - **Query failed**: You'll receive a [`WorkflowQueryException`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowQueryException.html) exception if something goes wrong during a Query. Any exception in a Query handler will trigger this error. This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. ## Dynamic components A dynamic Workflow, Activity, Signal, Update, or Query is a kind of unnamed item. Normally, these items are registered by name with the Worker and invoked at runtime. When an unregistered or unrecognized Workflow, Activity, or message request arrives with a recognized method signature, the Worker can use a pre-registered dynamic stand-in. For example, you might send a request to start a Workflow named "MyUnknownWorkflow". After receiving a Workflow Task, the Worker may find that there's no registered Workflow Definitions of that type. It then checks to see if there's a registered dynamic Workflow. If the dynamic Workflow signature matches the incoming Workflow signature, the Worker invokes that just as it would invoke a non-dynamic statically named version. By registering dynamic versions of your Temporal components, the Worker can fall back to these alternate implementations for name mismatches. > **⚠️ Caution:** > > Use dynamic elements judiciously and as a fallback mechanism, not a primary design. > They can introduce long-term maintainability and debugging issues. > Reserve dynamic invocation use for cases where a name is not or can't be known at compile time. > ### Set a Dynamic Workflow Use [`DynamicWorkflow`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/DynamicWorkflow.html) to implement Workflow Types dynamically. Register a Workflow implementation type that extends `DynamicWorkflow` to implement any Workflow Type that is not explicitly registered with the Worker. The dynamic Workflow interface is implemented with the `execute` method. This method takes in `EncodedValues` that are inputs to the Workflow Execution. These inputs can be specified by the Client when invoking the Workflow Execution. ```java public class MyDynamicWorkflow implements DynamicWorkflow { @Override public Object execute(EncodedValues args) { } } ``` ### How to set a Dynamic Activity To handle Activity types that do not have an explicitly registered handler, you can directly implement a dynamic Activity. Use `DynamicActivity` to implement any number of Activity types dynamically. When an Activity implementation that extends `DynamicActivity` is registered, it is called for any Activity type invocation that doesn't have an explicitly registered handler. The dynamic Activity interface is implemented with the `execute` method, as shown in the following example. ```java // Dynamic Activity implementation public static class DynamicGreetingActivityImpl implements DynamicActivity { @Override public Object execute(EncodedValues args) { String activityType = Activity.getExecutionContext().getInfo().getActivityType(); return activityType + ": " + args.get(0, String.class) + " " + args.get(1, String.class) + " from: " + args.get(2, String.class); } } ``` Use `Activity.getExecutionContext()` to get information about the Activity type that should be implemented dynamically. ### How to set a Dynamic Signal You can also implement Signal handlers dynamically. This is useful for library-level code and implementation of DSLs. Use `Workflow.registerListener(Object)` to register an implementation of the `DynamicSignalListener` in the Workflow implementation code. ```java Workflow.registerListener( (DynamicSignalHandler) (signalName, encodedArgs) -> name = encodedArgs.get(0, String.class)); ``` When registered, any Signals sent to the Workflow without a defined handler will be delivered to the `DynamicSignalHandler`. Note that you can only register one `Workflow.registerListener(Object)` per Workflow Execution. `DynamicSignalHandler` can be implemented in both regular and dynamic Workflow implementations. ### How to set a Dynamic Query You can also implement Query handlers dynamically. This is useful for library-level code and implementation of DSLs. Use `Workflow.registerListener(Object)` to register an implementation of the `DynamicQueryListener` in the Workflow implementation code. ```java Workflow.registerListener( (DynamicQueryHandler) (queryName, encodedArgs) -> name = encodedArgs.get(0, String.class)); ``` When registered, any Queries sent to the Workflow without a defined handler will be delivered to the `DynamicQueryHandler`. Note that you can only register one `Workflow.registerListener(Object)` per Workflow Execution. `DynamicQueryHandler` can be implemented in both regular and dynamic Workflow implementations. ### How to set a Dynamic Update You can also implement Update handlers dynamically. This is useful for library-level code and implementation of DSLs. ```java Workflow.registerListener( (DynamicUpdateHandler) (updateName, encodedArgs) -> encodedArgs.get(0, String.class)); ``` When registered, any Updates sent to the Workflow without a defined handler will be delivered to the `DynamicUpdateHandler`. You can only register one `Workflow.registerListener(Object)` per Workflow Execution. `DynamicUpdateHandler` can be implemented in both regular and dynamic Workflow implementations. --- # Schedules - Java SDK Source: https://docs.temporal.io/develop/java/workflows/schedules > Schedule, Backfill, Delete, Describe, List, Pause, Trigger, Update, and set Cron and Start Delays for Workflow Executions in Java using Temporal's ScheduleClient. This page shows how to do the following: - [How to Schedule a Workflow](#schedule-a-workflow) - [How to create a Schedule in Java](#create-schedule) - [How to backfill a Schedule in Java](#backfill-schedule) - [How to delete a Schedule in Java](#delete-schedule) - [How to describe a Schedule in Java](#describe-schedule) - [How to list a Schedule in Java](#list-schedule) - [How to pause a Schedule in Java](#pause-schedule) - [How to trigger a Schedule in Java](#trigger-schedule) - [How to update a Schedule in Java](#update-schedule) - [How to set a Cron Schedule in Java](#cron-schedule) - [Start Delay](#start-delay) ## How to Schedule a Workflow Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes. Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process. Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy). If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions. ### How to create a Schedule in Java The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands. To create a Scheduled Workflow Execution in Java, use the `createSchedule()` method on the `ScheduleClient`. When you create the `ScheduleClient`, you can also use any custom Namespace instead of the default by setting the [`ScheduleClientOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/schedules/ScheduleClientOptions.Builder.html). Schedules must be initialized with a Schedule ID. ```java Schedule schedule = Schedule.newBuilder() .setAction( ScheduleActionStartWorkflow.newBuilder() .setWorkflowType(HelloSchedules.GreetingWorkflow.class) .setArguments("World") .setOptions( WorkflowOptions.newBuilder() .setWorkflowId("WorkflowId") .setTaskQueue("TaskQueue") .build()) .build()) .setSpec(ScheduleSpec.newBuilder() .setIntervals( Collections.singletonList(new ScheduleIntervalSpec(Duration.ofSeconds(5))) ).build() ) .build(); // Create a schedule on the server ScheduleClient scheduleClient = ScheduleClient .newInstance(service, ScheduleClientOptions.newBuilder() .setNamespace("custom_namespace") .build() ); ScheduleHandle handle = scheduleClient.createSchedule("ScheduleId", schedule, ScheduleOptions.newBuilder().build()); ``` > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > ### How to backfill a Schedule in Java The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time. To Backfill a Scheduled Workflow Execution in Java, use the `backfill()` method on the `ScheduleHandle`. ```java ScheduleHandle handle = client.getHandle("schedule-id") Instant now = Instant.now(); handle.backfill( Arrays.asList( new ScheduleBackfill(now.minusMillis(5500), now.minusMillis(2500)), new ScheduleBackfill(now.minusMillis(2500), now))); ``` ### How to delete a Schedule in Java The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule. To delete a Scheduled Workflow Execution in Java, use the `delete()` method on the `Schedule Handle`. ```java ScheduleHandle handle = client.getHandle("schedule-id") handle.delete(); ``` ### How to describe a Schedule in Java The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs. To describe a Scheduled Workflow Execution in Java, use the `describe()` method on the `ScheduleHandle`. ```java ScheduleHandle handle = client.getHandle("schedule-id") ScheduleDescription description = handle.describe(); ``` ### How to list a Schedule in Java The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs. To list all schedules, use the `listSchedules()` asynchronous method on the `ScheduleClient`. If a schedule is added or deleted, it may not be available in the list immediately. ```java Stream scheduleStream = client.listSchedules(); ``` ### How to pause a Schedule in Java The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason. To pause a Scheduled Workflow Execution in Java, use the `pause()` method on the `ScheduleHandle`. You can pass a `note` to the `pause()` method to provide a reason for pausing the schedule. ```java ScheduleHandle handle = client.getHandle("schedule-id") handle.pause("Pausing the schedule for now"); ``` ### How to trigger a Schedule in Java The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time. To trigger a Scheduled Workflow Execution in Java, use the `trigger()` method on the `ScheduleHandle`. ```java ScheduleHandle handle = client.getHandle("schedule-id") handle.trigger(); ``` ### How to update a Schedule in Java The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval. Create a function that takes `ScheduleUpdateInput` and returns `ScheduleUpdate`. To update a Schedule, use a callback to build the update from the description. The following example updates the Schedule to set a limited number of actions. ```java ScheduleHandle handle = client.getHandle("schedule-id") handle.update( (ScheduleUpdateInput input) -> { Schedule.Builder builder = Schedule.newBuilder(input.getDescription().getSchedule()); // Make the schedule paused to demonstrate how to unpause a schedule builder.setState( ScheduleState.newBuilder() .setLimitedAction(true) .setRemainingActions(10) .build()); return new ScheduleUpdate(builder.build()); }); ``` ## How to set a Cron Schedule in Java > **⚠️ Caution:** > Cron support is not recommended > > We recommend using [Schedules](/schedule) instead of Cron Jobs. > Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules. > A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made. Set the Cron Schedule with the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance in the Client code using [`WorkflowOptions.Builder.setCronSchedule`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). Setting `setCronSchedule` changes the Workflow Execution into a Temporal Cron Job. The default timezone for a Cron is UTC. - Type: `String` - Default: None ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = YourWorker.yourclient.newWorkflowStub( YourWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWF") .setTaskQueue(YourWorker.TASK_QUEUE) // Set Cron Schedule .setCronSchedule("* * * * *") .build()); ``` Temporal Workflow Schedule Cron strings follow this format: ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) │ │ │ │ │ * * * * * ``` For more details, see the [Cron Sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloCron.java) ## Start Delay **How to delay the start of a Workflow Execution using Start Delay with the Temporal Java SDK.** Use the `StartDelay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule. Create an instance of [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in the Client code and set `StartDelay` using `setStartDelay`. ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWorkflow") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Start the workflow in 12 hours .setStartDelay(Duration.ofHours(12)) .build()); ``` --- # Side Effects - Java SDK Source: https://docs.temporal.io/develop/java/workflows/side-effects > Side Effects in a Workflow execute non-deterministic code like generating a UUID. The result is saved in Workflow Event History for consistent replays without re-execution. ## Side Effects Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow. This is done by storing the non-deterministic results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history). A Side Effect does not re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History. Side Effects should not fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task. An Activity or a Local Activity may also be used instead of a Side effect, as its result is also persisted in Workflow Execution History. > **📝 Note:** > > You shouldn't modify the Workflow state inside a Side Effect function, because it is not reexecuted during Replay. Side Effect function should be used to return a value. > To use a Side Effect in Java, set the [`sideEffect()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sideEffect(java.lang.Class,io.temporal.workflow.Functions.Func)) function in your Workflow Execution and return the non-deterministic code. ```java int random = Workflow.sideEffect(Integer.class, () -> random.nextInt(100)); if random < 50 { .... } else { .... } ``` Here's another example that uses `sideEffect()`. ```java // implementation of the @WorkflowMethod public void execute() { int randomInt = Workflow.sideEffect( int.class, () -> { Random random = new SecureRandom(); return random.nextInt(); }); String userHome = Workflow.sideEffect(String.class, () -> System.getenv("USER_HOME")); if(randomInt % 2 == 0) { // ... } else { // ... } } ``` Java also provides a deterministic method to generate random numbers or random UUIDs. To generate random numbers in a deterministic method, use [`newRandom()`](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newRandom). ```java // implementation of the @WorkflowMethod public void execute() { int randomInt = Workflow.newRandom().nextInt(); // ... } ``` To generate a random UUID in a deterministic method, use [`randomUUID()`](https://www.javadoc.io/static/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#newRandom()). ```java // implementation of the @WorkflowMethod public void execute() { String randomUUID = Workflow.randomUUID().toString(); // ... } ``` --- # Workflow Timeouts - Java SDK Source: https://docs.temporal.io/develop/java/workflows/timeouts > Optimize Workflow Execution with Temporal Java SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts). Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient. Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes. If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer. - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task. Create an instance of [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) in the Client code and set your timeout. Available timeouts are: - [setWorkflowExecutionTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowExecutionTimeout(java.time.Duration)) - [setWorkflowRunTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowRunTimeout(java.time.Duration)) - [setWorkflowTaskTimeout()](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html#setWorkflowTaskTimeout(java.time.Duration)) ```java //create Workflow stub for YourWorkflowInterface YourWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("YourWorkflow") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Timeout duration .setWorkflowExecutionTimeout(Duration.ofSeconds(10)) // .setWorkflowRunTimeout(Duration.ofSeconds(10)) // .setWorkflowTaskTimeout(Duration.ofSeconds(10)) .build()); ``` ## Workflow Retry Policy **How to set a Workflow Retry Policy in Java.** A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. To set a Workflow Retry Options in the [`WorkflowStub`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowStub.html) instance use [`WorkflowOptions.Builder.setWorkflowRetryOptions`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/client/WorkflowOptions.Builder.html). - Type: `RetryOptions` - Default: `Null` which means no retries will be attempted. ```java //create Workflow stub for GreetWorkflowInterface GreetWorkflowInterface workflow1 = WorkerGreet.greetclient.newWorkflowStub( GreetWorkflowInterface.class, WorkflowOptions.newBuilder() .setWorkflowId("GreetWF") .setTaskQueue(WorkerGreet.TASK_QUEUE) // Set Workflow Retry Options .setRetryOptions(RetryOptions.newBuilder() .build()); ``` --- # Timers - Java SDK Source: https://docs.temporal.io/develop/java/workflows/timers > A Workflow sets a durable Timer for delayed execution. Even if the Worker or Temporal Service is down, the Timer resumes once back up. Efficient and scalable. ## What is a Timer? A Workflow can set a durable Timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`. A Workflow can sleep for months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `Workflow.sleep()` call resolves and your code continues executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To set a Timer in Java, use [`sleep()`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/Workflow.html#sleep) and pass the number of seconds you want to wait before continuing. ```java sleep(5); ``` --- # Versioning - Java SDK Source: https://docs.temporal.io/develop/java/workflows/versioning > The Temporal Platform ensures deterministic Workflow code, offering versioning features in the Java SDK with Workflow Patching APIs and Worker Build Ids for efficient updates. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. > **🚨 Danger:** > Support for the experimental Worker Versioning method before 2025 will be removed from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](/develop/java/worker-versioning-legacy) docs if needed. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Patching with GetVersion A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Consider the following Workflow Definition: ```java public void processFile(Arguments args) { String localName = null; String processedName = null; try { localName = activities.download(args.getSourceBucketName(), args.getSourceFilename()); processedName = activities.processFile(localName); activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName); } finally { if (localName != null) { // File was downloaded. activities.deleteLocalFile(localName); } if (processedName != null) { // File was processed. activities.deleteLocalFile(processedName); } } } ``` Imagine you want to revise this Workflow by adding another Activity to calculate a file checksum. If an existing Workflow Execution was started by the original version of the Workflow code, where there was no `calculateChecksum()` Activity, and then resumed running on a new Worker where this Activity had been added, the server side Event History would be out of sync. This would cause the Workflow to fail with a nondeterminism error. To resolve this, you can use `workflow.GetVersion()` to patch to your Workflow: ```java public void processFile(Arguments args) { String localName = null; String processedName = null; try { localName = activities.download(args.getSourceBucketName(), args.getSourceFilename()); processedName = activities.processFile(localName); int version = Workflow.getVersion("checksumAdded", Workflow.DEFAULT_VERSION, 1); if (version == Workflow.DEFAULT_VERSION) { activities.upload(args.getTargetBucketName(), args.getTargetFilename(), processedName); } else { long checksum = activities.calculateChecksum(processedName); activities.uploadWithChecksum( args.getTargetBucketName(), args.getTargetFilename(), processedName, checksum); } } finally { if (localName != null) { // File was downloaded. activities.deleteLocalFile(localName); } if (processedName != null) { // File was processed. activities.deleteLocalFile(processedName); } } } ``` When `workflow.GetVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `GetVersion` for this change id — `checksumAdded` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example. After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version. ```java public void processFile(Arguments args) { String localName = null; String processedName = null; try { localName = activities.download(args.getSourceBucketName(), args.getSourceFilename()); processedName = activities.processFile(localName); // getVersion call is left here to ensure that any attempt to replay history // for a different version fails. It can be removed later when there is no possibility // of this happening. Workflow.getVersion("checksumAdded", 1, 1); long checksum = activities.calculateChecksum(processedName); activities.uploadWithChecksum( args.getTargetBucketName(), args.getTargetFilename(), processedName, checksum); } finally { if (localName != null) { // File was downloaded. activities.deleteLocalFile(localName); } if (processedName != null) { // File was processed. activities.deleteLocalFile(processedName); } } } ``` ### Adding support for versioned Workflow visibility in the Event History When you invoke `getVersion`, the SDK can record an `UpsertWorkflowSearchAttribute` Event in the history. This adds a [Search Attribute](/search-attribute#default-search-attribute) named `TemporalChangeVersion` that allows you to filter Workflows based on their version. #### Enable automatic upsert (Java SDK v1.30.0+) Starting with Java SDK v1.30.0, you can enable automatic `TemporalChangeVersion` upserts by setting `enableUpsertVersionSearchAttributes` in `WorkflowImplementationOptions`: ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setEnableUpsertVersionSearchAttributes(true) .build(); worker.registerWorkflowImplementationTypes(options, YourWorkflowImpl.class); ``` When enabled, each call to `Workflow.getVersion` automatically upserts the `TemporalChangeVersion` Search Attribute with a keyword list of `"-"` entries. > **📝 Note:** > > This option is backwards compatible and safe to enable on running Workflows. > However, once enabled, you cannot roll back to a version of the SDK that does not support this option. > #### Upsert manually If you are using a version of the Java SDK earlier than v1.30.0, or prefer to control the upsert yourself, you can set the attribute manually. ```java import io.temporal.common.SearchAttributeKey; public static final SearchAttributeKey> TEMPORAL_CHANGE_VERSION = SearchAttributeKey.forKeywordList("TemporalChangeVersion"); ``` Set this attribute when you call `getVersion`: ```java int version = Workflow.getVersion("MovedThankYouAfterLoop", Workflow.DEFAULT_VERSION, 1); if (version != Workflow.DEFAULT_VERSION) { Workflow.upsertTypedSearchAttributes(TEMPORAL_CHANGE_VERSION .valueSet(Arrays.asList("MovedThankYouAfterLoop-" + version))); } ``` For multiple `getVersion` calls, collect all version changes and set the attribute once: ```java List list = new ArrayList(); int versionOne = Workflow.getVersion("versionOne", Workflow.DEFAULT_VERSION, 1); int versionTwo = Workflow.getVersion("versionTwo", Workflow.DEFAULT_VERSION, 1); if (versionOne != Workflow.DEFAULT_VERSION) { list.add("versionOne-" + versionOne); } if (versionTwo != Workflow.DEFAULT_VERSION) { list.add("versionTwo-" + versionTwo); } Workflow.upsertTypedSearchAttributes(TEMPORAL_CHANGE_VERSION.valueSet(list)); ``` Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`: ```java import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; @WorkflowInterface public interface PizzaWorkflow { @WorkflowMethod public OrderConfirmation pizzaWorkflow(PizzaOrder order); } public class PizzaWorkflowImpl{ @Override public OrderConfirmation pizzaWorkflow(PizzaOrder order){ // implementation code omitted for this example } } @WorkflowInterface public interface PizzaWorkflowV2 { @WorkflowMethod public OrderConfirmation pizzaWorkflow(PizzaOrder order); } public class PizzaWorkflowImplV2 implements PizzaWorkflowV2{ @Override public OrderConfirmation pizzaWorkflow(PizzaOrder order){ // implementation code omitted for this example } } ``` It is necessary to create a separate interface because a Workflow Interface can only have one Workflow Method. You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types: ```java worker.registerWorkflowImplementationTypes(PizzaWorkflowImpl.class); worker.registerWorkflowImplementationTypes(PizzaWorkflowImplV2.class); ``` The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ### Testing a Workflow for replay safety To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/java/best-practices/testing-suite#replay). --- # Workflow Streams - Java SDK Source: https://docs.temporal.io/develop/java/workflows/workflow-streams > Stream events from a Workflow to subscribers using the Temporal Java SDK Workflow Streams contrib module. > **Public Preview** [Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream with either the blocking iterator or the non-blocking listener, and keeping a stream running across long-lived Workflows. ## Enable streaming on a Workflow The library ships as the `io.temporal:temporal-workflowstreams` contrib module. Enable streaming by constructing a `WorkflowStream` once via `WorkflowStream.newInstance()`, preferably in a [`@WorkflowInit`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/WorkflowInit.html) constructor. The factory registers the stream's handlers on the current Workflow, and a `@WorkflowInit` constructor runs before any handler dispatch, so polls and offset Queries arriving with the first Workflow Task (for example, from Update-With-Start) are accepted rather than rejected. ```java import io.temporal.workflow.WorkflowInit; import io.temporal.workflowstreams.WorkflowStream; import io.temporal.workflowstreams.WorkflowStreamState; public class OrderInput { public String orderId; public WorkflowStreamState streamState; } public class OrderWorkflowImpl implements OrderWorkflow { private final WorkflowStream stream; @WorkflowInit public OrderWorkflowImpl(OrderInput input) { stream = WorkflowStream.newInstance(input.streamState); } @Override public void execute(OrderInput input) { // ... rest of the workflow } } ``` `WorkflowStream.newInstance` creates the in-memory event log and registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. The `priorState` argument may be `null` and is only needed for Continue-As-New rollovers. Pass `null` or call the no-argument overload on a fresh start, and the carried `WorkflowStreamState` after a rollover. See [Stream from long-running Workflows](#continue-as-new) for more details. Constructing the stream at the top of the Workflow method also works. Signals received earlier are buffered by the SDK, but polls and offset Queries are rejected until the stream exists, so use `@WorkflowInit`. Construct exactly one `WorkflowStream` per Workflow. ## Publish from a Workflow Bind a [topic name](/workflow-streams#topics) with `stream.topic(name)`, then call `publish()` on the returned `WorkflowTopicHandle` to append events. Repeated calls with the same name return the same handle. ```java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowInit; import io.temporal.workflowstreams.WorkflowStream; import io.temporal.workflowstreams.WorkflowTopicHandle; import java.time.Duration; public class StatusEvent { public String state; public int progress; public String detail; public StatusEvent() {} public StatusEvent(String state, int progress, String detail) { this.state = state; this.progress = progress; this.detail = detail; } } public class OrderWorkflowImpl implements OrderWorkflow { private final WorkflowStream stream; private final WorkflowTopicHandle status; private final OrderActivities activities = Workflow.newActivityStub( OrderActivities.class, ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofMinutes(1)).build()); @WorkflowInit public OrderWorkflowImpl(OrderInput input) { stream = WorkflowStream.newInstance(input.streamState); status = stream.topic("status"); } @Override public void execute(OrderInput input) { status.publish(new StatusEvent("validating", 0, "checking inventory")); activities.validateOrder(input.orderId); status.publish(new StatusEvent("charging", 33, "authorizing payment")); activities.chargePayment(input.orderId); status.publish(new StatusEvent("shipping", 66, "dispatching to warehouse")); activities.dispatchOrder(input.orderId); status.publish(new StatusEvent("completed", 100, "")); } } ``` `publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction. Unlike the Python and TypeScript SDKs, Java topics carry no per-topic type binding. A topic handle is bound only to a name; published values are `Object` and subscribers decode each item from its raw payload (see [Subscribe](#subscribe)). To customize per-item serialization, pass `WorkflowStreamOptions.newBuilder().setPayloadConverters(...)` to `WorkflowStream.newInstance`, and use the matching converters on the subscriber side. There is no public accessor for the Worker's configured data converter inside Workflow code, so a custom converter can't be picked up automatically. Pass matching payload converters explicitly to keep Workflow-side publishes consistent with the rest of your Workflow. A pre-built `io.temporal.api.common.v1.Payload` may also be passed to `publish()`, bypassing conversion. ## Publish from a client Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `WorkflowStreamClient`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities. Construct one with: ```java WorkflowStreamClient.newInstance(workflowClient, workflowId) ``` Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and let try-with-resources flush on scope exit (`close()` guarantees a final flush of buffered items). When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream. It processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered). ```java import io.temporal.client.WorkflowClient; import io.temporal.workflowstreams.TopicHandle; import io.temporal.workflowstreams.WorkflowStreamClient; import io.temporal.workflowstreams.WorkflowStreamClientOptions; import java.time.Duration; public void publishStatus(WorkflowClient workflowClient, String workflowId) { WorkflowStreamClientOptions options = WorkflowStreamClientOptions.newBuilder() .setBatchInterval(Duration.ofMillis(200)) .build(); try (WorkflowStreamClient streamClient = WorkflowStreamClient.newInstance(workflowClient, workflowId, options)) { TopicHandle status = streamClient.topic("status"); status.publish(new StatusEvent("started", 0, "")); // ... // Buffer is flushed automatically on close(). } } ``` Inside an Activity scheduled by a Workflow, `WorkflowStreamClient.fromActivity()` infers the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input: ```java import io.temporal.activity.Activity; import io.temporal.workflowstreams.TopicHandle; import io.temporal.workflowstreams.WorkflowStreamClient; public class Delta { public String text; } public void streamDeltas(String orderId) { try (WorkflowStreamClient streamClient = WorkflowStreamClient.fromActivity()) { TopicHandle deltas = streamClient.topic("delta"); for (Delta delta : generateDeltas(orderId)) { deltas.publish(delta); Activity.getExecutionContext().heartbeat(null); } // Buffer is flushed automatically on close(). } } ``` For a [standalone Activity](/develop/java/activities/standalone-activities), there is no parent Workflow context to infer, so `fromActivity()` throws an `IllegalStateException`. Fall back to the general pattern with `Activity.getExecutionContext().getWorkflowClient()` and the target Workflow Id threaded through the Activity's input. Two operations give the application explicit control over when batches ship: the `forceFlush` argument on a publish for latency, and `client.flush()` for confirmation that prior publications have landed. Pass `true` as the `forceFlush` argument on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the client is open (between construction and `close()`). The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers: ```java deltas.publish(delta, /* forceFlush */ true); ``` Use it for latency-sensitive events: the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure. Use `client.flush()` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. `close()` already flushes on its way out, so the explicit call is only for barriers in the middle: ```java try (WorkflowStreamClient streamClient = WorkflowStreamClient.newInstance(workflowClient, workflowId)) { TopicHandle deltas = streamClient.topic("delta"); for (Delta delta : firstPhase()) { deltas.publish(delta); } streamClient.flush(); String checkpointId = recordPhaseOneComplete(); // only safe once phase-one events are durable for (Delta delta : secondPhase(checkpointId)) { deltas.publish(delta); } } ``` Batches also ship early once the buffer reaches `maxBatchSize`, if you set one on `WorkflowStreamClientOptions`; by default only the interval, `forceFlush`, `flush()`, and `close()` trigger a flush. `publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. The value goes through the payload converters immediately, so an unconvertible value fails the `publish()` call rather than a later background flush. From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up. If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you. ## Subscribe [Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `WorkflowStreamClient.newInstance(workflowClient, workflowId)` from any process that has a Temporal Client, or `fromActivity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below. The Java SDK offers two subscriber APIs over one shared poll engine: a blocking iterator for synchronous consumers, and a non-blocking listener that delivers items as callbacks without occupying a caller thread. Neither occupies a thread while a poll is blocked on the server. Polling runs on a small executor shared by all of a client's subscriptions (2 daemon threads by default; see `WorkflowStreamClientOptions.Builder.setPollExecutor`), so many concurrent subscriptions don't mean many threads. Either way, the subscription ends cleanly when the Workflow reaches a terminal state, automatically follows Continue-As-New chains, recovers from Workflow-side log truncation by restarting from the current base offset, handles pagination when a poll response hits the ~1 MB cap, and also ends when the owning `WorkflowStreamClient` is closed. Items carry the raw `io.temporal.api.common.v1.Payload` in `item.getPayload()`; decode at the call site with your data converter. Offsets are global across all topics, not per-topic. ### Blocking iterator `client.subscribe(options)` without a listener returns a `WorkflowStreamSubscription`. A blocking, single-use subscription the consuming thread iterates with a for-each loop. The thread blocks waiting for items while polling still runs on the shared executor. ```java import io.temporal.client.WorkflowClient; import io.temporal.common.converter.DefaultDataConverter; import io.temporal.workflowstreams.SubscribeOptions; import io.temporal.workflowstreams.WorkflowStreamClient; import io.temporal.workflowstreams.WorkflowStreamItem; import io.temporal.workflowstreams.WorkflowStreamSubscription; public void watchOrder(WorkflowClient workflowClient, String orderId) { SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("status").build(); try (WorkflowStreamClient stream = WorkflowStreamClient.newInstance(workflowClient, orderId); WorkflowStreamSubscription subscription = stream.subscribe(options)) { for (WorkflowStreamItem item : subscription) { StatusEvent evt = DefaultDataConverter.STANDARD_INSTANCE.fromPayload( item.getPayload(), StatusEvent.class, StatusEvent.class); System.out.printf("[%3d%%] %s: %s%n", evt.progress, evt.state, evt.detail); if (evt.state.equals("completed")) { break; } } } } ``` `SubscribeOptions` controls the subscription: `setTopics` filters by name (unset means all topics), `setFromOffset` resumes from a stored global offset (zero means the beginning), and `setPollCooldown` sets the minimum interval between polls (default 100 milliseconds). A single-topic convenience method, `streamClient.topic("status").subscribe(fromOffset)`, is equivalent to passing one name in `setTopics`. `subscription.close()` stops the subscription before the next poll; items already fetched still drain. An unrecoverable poll failure is rethrown from `hasNext()`. A subscriber that never publishes doesn't strictly need to close the `WorkflowStreamClient` for flushing (the background flusher only runs for publishers), but closing it releases the poll executor, so keep both in the try-with-resources as above. ### Non-blocking listener Unique to the Java SDK, the second subscriber API inverts control. Instead of parking a thread in an iterator, pass a `WorkflowStreamListener` to `subscribe` and items are delivered as callbacks on the poll executor. `subscribe` returns a `WorkflowStreamSubscriptionHandle` immediately. This is the right shape when one process consumes many streams concurrently — all subscriptions share the client's small executor rather than each pinning a thread. Callbacks are serialized and are never invoked concurrently and must not block. The `CompletionStage` returned by `onNext` is the backpressure boundary. Return `null` or an already-completed stage to receive the next item immediately, or a pending stage to defer both further delivery and the next poll until it completes. A stage that completes exceptionally or an exception thrown directly from `onNext` stops the subscription and is reported to `onError`. ```java import io.temporal.common.converter.DefaultDataConverter; import io.temporal.workflowstreams.SubscribeOptions; import io.temporal.workflowstreams.WorkflowStreamItem; import io.temporal.workflowstreams.WorkflowStreamListener; import io.temporal.workflowstreams.WorkflowStreamSubscriptionHandle; import java.util.concurrent.CompletionStage; SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("status").build(); WorkflowStreamSubscriptionHandle handle = streamClient.subscribe( options, new WorkflowStreamListener() { @Override public CompletionStage onNext(WorkflowStreamItem item) { StatusEvent evt = DefaultDataConverter.STANDARD_INSTANCE.fromPayload( item.getPayload(), StatusEvent.class, StatusEvent.class); System.out.printf( "offset=%d topic=%s state=%s%n", item.getOffset(), item.getTopic(), evt.state); return null; // or a pending stage to apply backpressure } @Override public void onCompleted() { System.out.println("stream ended"); } }); // The calling thread is free; wait on the handle when you need to. handle.getDoneFuture().join(); ``` `onCompleted` fires once when the stream ends cleanly because the Workflow reached a terminal state. `onError` fires once on an unrecoverable failure, including a failure from `onNext`. Its default implementation logs at warn level. `handle.close()` stops the subscription before the next poll without calling `onCompleted`. `handle.getDoneFuture()` completes normally when the stream ends cleanly or the subscription is closed, and exceptionally with the failure passed to `onError`. A single-topic convenience, `streamClient.topic("status").subscribe(fromOffset, listener)`, is also available. To hand work off a callback without blocking the executor, return a stage that completes when the work is done. For example, `CompletableFuture.runAsync(() -> render(item), renderExecutor)` and the next item is delivered only after it completes. ### Heterogeneous topics Every item arrives as a raw `Payload` in `item.getPayload()`, so a single subscription naturally consumes multiple topics whose payload types differ. Pass the topic names to `SubscribeOptions.Builder.setTopics` (or leave it unset for every topic on the stream), dispatch on `item.getTopic()`, and decode into the matching type: ```java SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("status", "progress").build(); try (WorkflowStreamSubscription subscription = stream.subscribe(options)) { for (WorkflowStreamItem item : subscription) { switch (item.getTopic()) { case "status": StatusEvent status = DefaultDataConverter.STANDARD_INSTANCE.fromPayload( item.getPayload(), StatusEvent.class, StatusEvent.class); System.out.printf("[status] %s: %s%n", status.state, status.detail); break; case "progress": ProgressEvent progress = DefaultDataConverter.STANDARD_INSTANCE.fromPayload( item.getPayload(), ProgressEvent.class, ProgressEvent.class); System.out.printf("[progress] %s%n", progress.message); break; } } } ``` A single subscription over multiple topics also avoids the cancellation race that two concurrent subscribers would create. Because `item.getPayload()` is the raw payload, it's also the right shape when you want to forward the bytes through to another system without decoding them. ### Closing the stream A subscriber's loop or listener doesn't know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces: 1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `watchOrder` example above, `new StatusEvent("completed", 100, "")` is the minimal form, and the consumer's `if (evt.state.equals("completed")) break` is the matching half. Each subscription decides what its own end-of-stream marker is. 2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns is consumed by the subscription transparently, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it. There are two ways to provide that overlap. - [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits: ```java // at the end of the workflow method status.publish(new StatusEvent("completed", 100, "")); Workflow.sleep(Duration.ofSeconds(30)); return result; ``` - [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives: ```java public class ChatWorkflowImpl implements ChatWorkflow { private boolean subscriberDone = false; @Override // annotated with @SignalMethod on the workflow interface public void subscriberAcknowledgedTerminator() { subscriberDone = true; } @Override public String complete(ChatInput input) { // ... do work and publish events ... // Returns true if the ack arrived, false on timeout. Either way, fall through. Workflow.await(Duration.ofSeconds(30), () -> subscriberDone); return result; } } ``` The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below. You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). A subscription ends cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but doesn't distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `workflowClient.newUntypedWorkflowStub(workflowId).describe()` after the subscription ends to inspect the Workflow's status. ## Stream from long-running Workflows Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers. To keep a stream running across them without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add a `WorkflowStreamState` field to your Workflow input, pass it to `WorkflowStream.newInstance`, and call `stream.continueAsNew(buildArgs)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, snapshots the stream state, then Continues-As-New with the arguments built by `buildArgs(postDrainState)`. It never returns: ```java public class WorkflowInput { public int itemsProcessed; public WorkflowStreamState streamState; } public class LongRunningWorkflowImpl implements LongRunningWorkflow { private final WorkflowStream stream; private int itemsProcessed; @WorkflowInit public LongRunningWorkflowImpl(WorkflowInput input) { stream = WorkflowStream.newInstance(input.streamState); itemsProcessed = input.itemsProcessed; } @Override public void execute(WorkflowInput input) { while (true) { doOneIteration(); itemsProcessed++; if (Workflow.getInfo().isContinueAsNewSuggested()) { stream.continueAsNew( state -> { WorkflowInput next = new WorkflowInput(); next.itemsProcessed = itemsProcessed; // your own state, carried across next.streamState = state; // the captured stream state return new Object[] {next}; }); } } } } ``` The `streamState` field is `null` on a fresh start and a populated snapshot after a rollover. The `buildArgs` callback receives the post-detach `WorkflowStreamState` as its only argument, so the snapshot is guaranteed to happen *after* pollers detach. To pass other Continue-As-New parameters such as a different Task Queue, or to use a custom publisher TTL, use the explicit recipe instead. Drain the pollers, wait for handlers to finish, snapshot the state with your chosen TTL, then call `Workflow.continueAsNew` yourself: ```java import io.temporal.workflow.ContinueAsNewOptions; stream.detachPollers(); Workflow.await(() -> Workflow.isEveryHandlerFinished()); WorkflowStreamState state = stream.getState(Duration.ofMinutes(30)); // custom publisher TTL WorkflowInput next = new WorkflowInput(); next.itemsProcessed = itemsProcessed; next.streamState = state; Workflow.continueAsNew( ContinueAsNewOptions.newBuilder().setTaskQueue("other-tq").build(), next); ``` The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `stream.truncate(upToOffset)` to keep the carried log itself small. ## Deduplication window See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams. There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting: - **Publisher TTL.** At each Continue-As-New, deduplication entries whose last-seen time is older than this are dropped. The last-seen time is updated on each *successful* publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. `stream.continueAsNew(...)` snapshots with a 15-minute default; to tune it, use the explicit recipe above and pass your value to `getState(publisherTtl)`. - **`maxRetryDuration`.** A `WorkflowStreamClient` retries a failed batch for up to this long (default 10 minutes). If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `FlushTimeoutException` is raised. ```java WorkflowStreamClientOptions.newBuilder() .setMaxRetryDuration(Duration.ofMinutes(10)) .build(); ``` On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `FlushTimeoutException` is raised from inside the background flusher and terminates it. Until you call `client.flush()` or `client.close()` — which surface the deferred exception — subsequent publishes accumulate in the buffer with no flusher to ship them. `maxRetryDuration` must be less than the Workflow's publisher TTL to preserve exactly-once delivery. ## Best practices There are a few details to note if you're writing custom message handlers or testing the library's capabilities: - **Construct exactly one `WorkflowStream` per Workflow, preferably in `@WorkflowInit`.** The factory registers the publish Signal, poll Update, and offset Query handlers on the current Workflow. A `@WorkflowInit` constructor runs before any handler dispatch, so polls and offset Queries arriving with the first Workflow Task are accepted; construction at the top of the Workflow method leaves them rejected until the stream exists. - **`item.getPayload()` is always the raw payload.** Decode it with a converter built from the same payload converters used by the publisher. When publishers and subscribers both rely on the defaults, `DefaultDataConverter.STANDARD_INSTANCE` matches on both sides. If you pass `setPayloadConverters` on the Workflow side or the client side, build a matching converter on the subscriber side. - **The codec chain runs once on the envelope.** Payload codecs (encryption, compression) configured on the Temporal client run on the Signal or Update envelope that carries each batch, never per item, so items are never double-encoded. `setPayloadConverters` handles only per-item serialization; its `PayloadConverter[]` type makes it impossible to slot a codec in per item. - **Listener callbacks must not block.** They run serialized on the client's poll executor, which drives every subscription on the client. Blocking in `onNext` stalls the client's other subscriptions. Hand slow work to your own executor and return the resulting `CompletionStage` for backpressure. - **Size the poll executor for slow Workflows.** The default executor has 2 daemon threads, is created lazily, and is owned (and shut down) by the client; a user-supplied executor is never shut down by the client. The executor runs the short update-admission and delivery steps and poll cooldowns, never the long poll itself, so a small pool serves many subscriptions. The known worst case for pool pressure is a backlogged Workflow pinning a thread in the update-admission call; supply a bigger pool via `setPollExecutor` when running many subscriptions against slow Workflows. - **Cross-language interop depends on the configured data converter.** The handler names, JSON envelope field names, and per-item payload encoding match the other SDKs' packages exactly, so a Java publisher or subscriber interoperates with a Workflow written in any of them and vice versa. One Java-specific caveat: the protocol envelope types are serialized by the Workflow's and client's *configured* data converter. The default Jackson JSON converter produces the wire-compatible snake_case field names; if you configure a non-Jackson JSON converter, it must produce the same field names for cross-language interop. ## Example: Stream LLM output The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it. If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees. **LlmActivitiesImpl.java** ```java import com.openai.client.OpenAIClient; import com.openai.client.okhttp.OpenAIOkHttpClient; import com.openai.core.http.StreamResponse; import com.openai.models.chat.completions.ChatCompletionChunk; import com.openai.models.chat.completions.ChatCompletionCreateParams; import io.temporal.activity.Activity; import io.temporal.workflowstreams.TopicHandle; import io.temporal.workflowstreams.WorkflowStreamClient; import io.temporal.workflowstreams.WorkflowStreamClientOptions; import java.time.Duration; import java.util.concurrent.atomic.AtomicBoolean; public class TextDelta { public String text; public TextDelta() {} public TextDelta(String text) { this.text = text; } } public class RetryEvent { public int attempt; public RetryEvent() {} public RetryEvent(int attempt) { this.attempt = attempt; } } public class CloseEvent {} public class LlmActivitiesImpl implements LlmActivities { @Override public String streamCompletion(String prompt) { WorkflowStreamClientOptions options = WorkflowStreamClientOptions.newBuilder() .setBatchInterval(Duration.ofMillis(200)) .build(); try (WorkflowStreamClient streamClient = WorkflowStreamClient.fromActivity(options)) { TopicHandle deltas = streamClient.topic("delta"); TopicHandle retry = streamClient.topic("retry"); TopicHandle close = streamClient.topic("close"); // Tell consumers an earlier attempt's deltas are stale. int attempt = Activity.getExecutionContext().getInfo().getAttempt(); if (attempt > 1) { retry.publish(new RetryEvent(attempt), /* forceFlush */ true); } // Disable provider-side retries; let Temporal own retry policy at the Activity layer. OpenAIClient openai = OpenAIOkHttpClient.builder().fromEnv().maxRetries(0).build(); ChatCompletionCreateParams params = ChatCompletionCreateParams.builder().model("gpt-4o-mini").addUserMessage(prompt).build(); StringBuilder full = new StringBuilder(); AtomicBoolean first = new AtomicBoolean(true); try (StreamResponse stream = openai.chat().completions().createStreaming(params)) { stream.stream() .forEach( chunk -> chunk.choices().stream() .findFirst() .flatMap(choice -> choice.delta().content()) .filter(text -> !text.isEmpty()) .ifPresent( text -> { // forceFlush only on the first delta so the user sees something // immediately; subsequent deltas batch at the 200 ms interval. deltas.publish(new TextDelta(text), first.getAndSet(false)); full.append(text); })); } close.publish(new CloseEvent()); return full.toString(); } } } ``` **ChatWorkflowImpl.java** ```java import io.temporal.activity.ActivityOptions; import io.temporal.workflow.SignalMethod; import io.temporal.workflow.Workflow; import io.temporal.workflow.WorkflowInit; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; import io.temporal.workflowstreams.WorkflowStream; import java.time.Duration; @WorkflowInterface public interface ChatWorkflow { @WorkflowMethod String complete(ChatInput input); @SignalMethod void subscriberAcknowledgedTerminator(); } public class ChatWorkflowImpl implements ChatWorkflow { private final LlmActivities activities = Workflow.newActivityStub( LlmActivities.class, ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofMinutes(5)).build()); private boolean subscriberDone = false; @WorkflowInit public ChatWorkflowImpl(ChatInput input) { WorkflowStream.newInstance(input.streamState); } @Override public void subscriberAcknowledgedTerminator() { subscriberDone = true; } @Override public String complete(ChatInput input) { String result = activities.streamCompletion(input.prompt); // Wait for the subscriber to ack the terminal `close` event. The timeout // is a fallback for when no subscriber is attached; with the ack, the // typical case exits as soon as the subscriber confirms. Workflow.await(Duration.ofSeconds(30), () -> subscriberDone); return result; } } ``` **Consumer.java** ```java import io.temporal.client.WorkflowClient; import io.temporal.common.converter.DefaultDataConverter; import io.temporal.workflowstreams.SubscribeOptions; import io.temporal.workflowstreams.WorkflowStreamClient; import io.temporal.workflowstreams.WorkflowStreamItem; import io.temporal.workflowstreams.WorkflowStreamSubscription; public String streamChat(WorkflowClient workflowClient, String chatId) { StringBuilder output = new StringBuilder(); Runnable render = () -> { // ... display the accumulated output (terminal redraw, UI update, etc.) }; SubscribeOptions options = SubscribeOptions.newBuilder().setTopics("delta", "retry", "close").build(); try (WorkflowStreamClient stream = WorkflowStreamClient.newInstance(workflowClient, chatId); WorkflowStreamSubscription subscription = stream.subscribe(options)) { for (WorkflowStreamItem item : subscription) { switch (item.getTopic()) { case "retry": // Earlier attempt's deltas are stale; drop what we've shown. output.setLength(0); render.run(); break; case "delta": TextDelta delta = DefaultDataConverter.STANDARD_INSTANCE.fromPayload( item.getPayload(), TextDelta.class, TextDelta.class); output.append(delta.text); render.run(); break; case "close": // Acknowledge so the Workflow can return without waiting on the fallback timeout. workflowClient .newUntypedWorkflowStub(chatId) .signal("subscriberAcknowledgedTerminator"); return output.toString(); } } } return output.toString(); } ``` A few choices in this shape are deliberate: - The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes only the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why. - The Activity publishes a `RETRY` event when `Activity.getExecutionContext().getInfo().getAttempt() > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)). - Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `Workflow.await` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative). - `forceFlush` is `true` only on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `batchInterval`; per-delta `forceFlush` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off). ## See also - [Workflow Streams samples (samples-java)](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/workflowstreams): six runnable scenarios covering basic publish/subscribe, the non-blocking listener, reconnecting subscribers, external publishers, bounded logs, and LLM streaming. - [`temporal-workflowstreams` API reference](https://javadoc.io/doc/io.temporal/temporal-workflowstreams). - [Workflow message passing](/develop/java/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on. - [Converters and encryption](/develop/java/best-practices/data-handling): converters and codecs. --- # PHP SDK developer guide Source: https://docs.temporal.io/develop/php > Explore Temporal PHP SDK guides to master features for Temporal Applications. Learn Workflows, Activities, Testing, Failure Detection, Messages, Observability, and more. ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Install and get started You can find detailed installation instructions for the PHP SDK in the [Quickstart](/develop/php/set-up-your-local-php). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Activity Basics](/develop/php/activities/basics) - [Workflow Basics](/develop/php/workflows/basics) - [Start an Activity Execution](/develop/php/activities/execution) - [Run Worker Processes](/develop/php/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/php/workflows) - [Workflow Basics](/develop/php/workflows/basics) - [Child Workflows](/develop/php/workflows/child-workflows) - [Continue-As-New](/develop/php/workflows/continue-as-new) - [Cancellation](/develop/php/workflows/cancellation) - [Timeouts](/develop/php/workflows/timeouts) - [Message Passing](/develop/php/workflows/message-passing) - [Schedules](/develop/php/workflows/schedules) - [Timers](/develop/php/workflows/timers) - [Side effects](/develop/php/workflows/side-effects) - [Versioning](/develop/php/workflows/versioning) ## [Activities](/develop/php/activities) - [Activity Basics](/develop/php/activities/basics) - [Activity Execution](/develop/php/activities/execution) - [Timeouts](/develop/php/activities/timeouts) - [Asynchronous Activity Completion](/develop/php/activities/asynchronous-activity) ## [Workers](/develop/php/workers) - [Run Worker processes](/develop/php/workers/run-worker-process) ## [Temporal Client](/develop/php/client) - [Temporal Client](/develop/php/client/temporal-client) ## [Platform](/develop/php/platform) - [Observability](/develop/php/platform/observability) - [Enriching the UI](/develop/php/platform/enriching-ui) ## [Best practices](/develop/php/best-practices) - [Testing](/develop/php/best-practices/testing-suite) - [Debugging](/develop/php/best-practices/debugging) ## Temporal PHP technical resources - [PHP SDK Quickstart - Setup Guide](/develop/php/set-up-your-local-php) - [PHP API Documentation](https://php.temporal.io) - [PHP SDK Code Samples](https://github.com/temporalio/samples-php) - [PHP SDK GitHub](https://github.com/temporalio/sdk-php) ## Get connected with the Temporal PHP community - [Temporal PHP Community Slack](https://temporalio.slack.com/archives/C01LK9FAMM0) - [PHP SDK Forum](https://community.temporal.io/tag/php-sdk) --- # Activities - PHP SDK Source: https://docs.temporal.io/develop/php/activities > This section explains how to implement Activities with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Activities - [Activity Basics](/develop/php/activities/basics) - [Activity Execution](/develop/php/activities/execution) - [Timeouts](/develop/php/activities/timeouts) - [Asynchronous Activity Completion](/develop/php/activities/asynchronous-activity) --- # Asynchronous Activity Completion - PHP SDK Source: https://docs.temporal.io/develop/php/activities/asynchronous-activity ## How to asynchronously complete an Activity [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. Sometimes Workflows need to perform certain operations in parallel. Invoking activity stub without the use of `yield` will return the Activity result promise which can be resolved at later moment. Calling `yield` on promise blocks until a result is available. > Activity promise also exposes `then` method to construct promise chains. Read more about Promises > [here](https://github.com/reactphp/promise). Alternatively you can explicitly wrap your code (including `yield` constructs) using `Workflow::async` which will execute nested code in parallel with main Workflow code. Call `yield` on Promise returned by `Workflow::async` to merge execution result back to primary Workflow method. ```php public function greet(string $name): \Generator { // Workflow::async runs it's activities and child workflows in a separate coroutine. Use keyword yield to merge // it back to parent process. $first = Workflow::async( function () use ($name) { $hello = yield $this->greetingActivity->composeGreeting('Hello', $name); $bye = yield $this->greetingActivity->composeGreeting('Bye', $name); return $hello . '; ' . $bye; } ); $second = Workflow::async( function () use ($name) { $hello = yield $this->greetingActivity->composeGreeting('Hola', $name); $bye = yield $this->greetingActivity->composeGreeting('Chao', $name); return $hello . '; ' . $bye; } ); // blocks until $first and $second complete return (yield $first) . "\n" . (yield $second); } ``` **Async completion** There are certain scenarios when moving on from an Activity upon completion of its function is not possible or desirable. For example, you might have an application that requires user input to complete the Activity. You could implement the Activity with a polling mechanism, but a simpler and less resource-intensive implementation is to asynchronously complete a Temporal Activity. There are two parts to implementing an asynchronously completed Activity: 1. The Activity provides the information necessary for completion from an external system and notifies the Temporal service that it is waiting for that outside callback. 2. The external service calls the Temporal service to complete the Activity. The following example demonstrates the first part: [app/src/AsyncActivityCompletion/GreetingActivity.php](https://github.com/temporalio/samples-php/blob/main/app/src/AsyncActivityCompletion/GreetingActivity.php) ```php class GreetingActivity implements GreetingActivityInterface { private LoggerInterface $logger; public function __construct() { $this->logger = new Logger(); } /** * Demonstrates how to implement an Activity asynchronously. * When {@link Activity::doNotCompleteOnReturn()} is called, * the Activity implementation function that returns doesn't complete the Activity. */ public function composeGreeting(string $greeting, string $name): string { // In real life this request can be executed anywhere. By a separate service for example. $this->logger->info(sprintf('GreetingActivity token: %s', base64_encode(Activity::getInfo()->taskToken))); // Send the taskToken to the external service that will complete the Activity. // Return from the Activity a function indicating that Temporal should wait // for an async completion message. Activity::doNotCompleteOnReturn(); // When doNotCompleteOnReturn() is invoked the return value is ignored. return 'ignored'; } } ``` The following code demonstrates how to complete the Activity successfully using `WorkflowClient`: [app/src/AsyncActivityCompletion/CompleteCommand.php](https://github.com/temporalio/samples-php/blob/main/app/src/AsyncActivityCompletion/CompleteCommand.php) ```php $client = $this->workflowClient->newActivityCompletionClient(); // Complete the Activity. $client->completeByToken( base64_decode($input->getArgument('token')), $input->getArgument('message') ); ``` To fail the Activity, you would do the following: ```php // Fail the Activity. $activityClient->completeExceptionallyByToken($taskToken, new \Error("activity failed")); ``` --- # Activity Basics - PHP SDK Source: https://docs.temporal.io/develop/php/activities/basics > This section explains Activity Basics with the PHP SDK ## How to develop a basic Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). Activities are defined as methods of a plain PHP interface annotated with `#[ActivityInterface]`. Following is an example of an interface that defines four Activities: ```php #[ActivityInterface] // Defining an interface for the activities. interface FileProcessingActivities { public function upload(string $bucketName, string $localName, string $targetName): void; #[ActivityMethod("transcode_file")] public function download(string $bucketName, string $remoteName): void; public function processFile(): string; public function deleteLocalFile(string $fileName): void; } ``` ### How to develop Activity Parameters There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. Each method defines a single Activity type. A single Workflow can use more than one Activity interface and call more than one Activity method from the same interface. The only requirement is that Activity method arguments and return values are serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface. The default implementation uses a JSON serializer, but an alternative implementation can be easily configured. ### How to define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). Return values must be serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface. The default implementation uses a JSON serializer, but an alternative implementation can be easily configured. Thus, you can return both primitive types: ```php class GreetingActivity implements GreetingActivityInterface { public function composeGreeting(string $greeting, string $name): string { return $greeting . ' ' . $name; } } ``` And objects: ```php class GreetingActivity implements GreetingActivityInterface { public function composeGreeting(string $greeting, string $name): Greeting { return new Greeting($greeting, $name); } } ``` ### How to customize your Activity Type Activities have a Type that are referred to as the Activity name. The following examples demonstrate how to set a custom name for your Activity Type. An optional `#[ActivityMethod]` attribute can be used to override a default Activity name. You can define your own prefix for all Activity names by adding the `prefix` option to the `ActivityInterface` attribute. (The default prefix is empty.) ```php #[ActivityInterface("file_activities.")] interface FileProcessingActivities { public function upload(string $bucketName, string $localName, string $targetName); #[ActivityMethod("transcode_file")] public function download(string $bucketName, string $remoteName); public function processFile(): string; public function deleteLocalFile(string $fileName); } ``` The `#[ActivityInterface("file_activities.")]` is an attribute that tells the PHP SDK to generate a class to implement the `FileProcessingActivities` interface. The functions define Activities that are used in the Workflow. --- # Activity execution - PHP SDK Source: https://docs.temporal.io/develop/php/activities/execution > Shows how to perform Activity execution with the PHP SDK ## How to start an Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed]) in your Workflow Execution Event History. A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity implementation code should be _idempotent_. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. Activity implementation is an implementation of an Activity interface. The following code example, uses a constructor that takes an Amazon S3 client and a local directory, and uploads a file to the S3 bucket. Then, the code uses a function to download a file from the S3 bucket passing a bucket name, remote name, and local name as arguments. Finally, it uses a function that takes a local file name as an argument and returns a string. ```php // An implementation of an Activity interface. class FileProcessingActivitiesImpl implements FileProcessingActivities { private S3Client $s3Client; private string $localDirectory; public function __construct(S3Client $s3Client, string $localDirectory) { $this->s3Client = $s3Client; $this->localDirectory = $localDirectory; } // Uploading a file to S3. public function upload(string $bucketName, string $localName, string $targetName): void { $this->s3Client->putObject( $bucketName, $targetName, fopen($this->localDirectory . $localName, 'rb+') ); } // Downloading a file from S3. public function download( string $bucketName, string $remoteName, string $localName ): void { $this->s3Client->downloadObject( $bucketName, $remoteName, fopen($this->localDirectory .$localName, 'wb+') ); } // A function that takes a local file name as an argument and returns a string. public function processFile(string $localName): string { // Implementation omitted for brevity. return compressFile($this->localDirectory . $localName); } public function deleteLocalFile(string $fileName): void { unlink($this->localDirectory . $fileName); } } ``` ### How to set the required Activity Timeouts Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. ### How to get the results of an Activity Execution The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable. Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available. `Workflow::newActivityStub`returns a client-side stub an implements an Activity interface. The client-side stub can be used within the Workflow code. It takes the Activity's type and`ActivityOptions` as arguments. Calling (via `yield`) a method on this interface invokes an Activity that implements this method. An Activity invocation synchronously blocks until the Activity completes, fails, or times out. Even if Activity Execution takes a few months, the Workflow code still sees it as a single synchronous invocation. It doesn't matter what happens to the processes that host the Workflow. The business logic code just sees a single method call. ```php class GreetingWorkflow implements GreetingWorkflowInterface { private $greetingActivity; public function __construct() { $this->greetingActivity = Workflow::newActivityStub( GreetingActivityInterface::class, ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 seconds')) ); } public function greet(string $name): \Generator { // This is a blocking call that returns only after the activity has completed. return yield $this->greetingActivity->composeGreeting('Hello', $name); } } ``` If different Activities need different options, like timeouts or a task queue, multiple client-side stubs can be created with different options. ```php $greetingActivity = Workflow::newActivityStub( GreetingActivityInterface::class, ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 seconds')) ); $greetingActivity = Workflow::newActivityStub( GreetingActivityInterface::class, ActivityOptions::new()->withStartToCloseTimeout(\DateInterval::createFromDateString('30 minutes')) ); ``` --- # Activity Timeouts - PHP SDK Source: https://docs.temporal.io/develop/php/activities/timeouts > Optimize Workflow Execution with Temporal PHP SDK - Set Activity Timeouts and Retry Policies efficiently. ## How to set Activity timeouts Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution. The following timeouts are available in the Activity Options. - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. Because Activities are reentrant, only a single stub can be used for multiple Activity invocations. Available timeouts are: - withScheduleToCloseTimeout() - withStartToCloseTimeout() - withScheduleToStartTimeout() ```php $this->greetingActivity = Workflow::newActivityStub( GreetingActivityInterface::class, // Set Activity Timeout duration ActivityOptions::new() ->withScheduleToCloseTimeout(CarbonInterval::seconds(2)) // ->withStartToCloseTimeout(CarbonInterval::seconds(2)) // ->withScheduleToStartTimeout(CarbonInterval::seconds(10)) ); ``` ### How to set an Activity Retry Policy A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To set an Activity Retry, set `{@link RetryOptions}` on `{@link ActivityOptions}`. The follow example creates a new Activity with the given options. ```php $this->greetingActivity = Workflow::newActivityStub( GreetingActivityInterface::class, ActivityOptions::new() ->withScheduleToCloseTimeout(CarbonInterval::seconds(10)) ->withRetryOptions( RetryOptions::new() ->withInitialInterval(CarbonInterval::seconds(1)) ->withMaximumAttempts(5) ->withNonRetryableExceptions([\InvalidArgumentException::class]) ) ); } ``` For an executable code sample, see [ActivityRetry sample](https://github.com/temporalio/samples-php/tree/master/app/src/ActivityRetry) in the PHP samples repository. ### How to set the required Activity Timeouts Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. ## Activity next Retry delay **How to override the next Retry delay following an Activity failure using the Temporal PHP SDK** You may throw an [`ApplicationFailure`](/references/failures#application-failure) with the `nextRetryDelay` field set. This value will replace and override whatever the retry interval would be on the retry policy. For example, if in an Activity, you want to base the interval on the number of attempts, you might do: ```php $attempt = \Temporal\Activity::getInfo()->attempt; throw new \Temporal\Exception\Failure\ApplicationFailure( message: "Something bad happened on attempt $attempt", type: 'my_failure_type', nonRetryable: false, nextRetryDelay: \DateInterval::createFromDateString(\sprintf('%d seconds', $attempt * 3)), ); ``` ## How to Heartbeat an Activity An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. Heartbeats can contain a `details` field describing the Activity's current progress. If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service. Some Activities are long-running. To react to a crash quickly, use the Heartbeat mechanism, `Activity::heartbeat()`, which lets the Temporal Server know that the Activity is still alive. This acts as a periodic checkpoint mechanism for the progress of an Activity. You can piggyback `details` on an Activity Heartbeat. If an Activity times out, the last value of `details` is included in the `TimeoutFailure` delivered to a Workflow. Then the Workflow can pass the details to the next Activity invocation. Additionally, you can access the details from within an Activity via `Activity::getHeartbeatDetails`. When an Activity is retried after a failure `getHeartbeatDetails` enables you to get the value from the last successful Heartbeat. ```php use Temporal\Activity; class FileProcessingActivitiesImpl implements FileProcessingActivities { // ... public function download( string $bucketName, string $remoteName, string $localName ): void { $this->dowloader->downloadWithProgress( $bucketName, $remoteName, $localName, // on progress function ($progress) { Activity::heartbeat($progress); } ); Activity::heartbeat(100); // download complete // ... } // ... } ``` #### How to set a Heartbeat Timeout A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). Some Activities are long-running. To react to a crash quickly, use the Heartbeat mechanism, `Activity::heartbeat()`, which lets the Temporal Server know that the Activity is still alive. This acts as a periodic checkpoint mechanism for the progress of an Activity. You can piggyback `details` on an Activity Heartbeat. If an Activity times out, the last value of `details` is included in the `TimeoutFailure` delivered to a Workflow. Then the Workflow can pass the details to the next Activity invocation. Additionally, you can access the details from within an Activity via `Activity::getHeartbeatDetails`. When an Activity is retried after a failure `getHeartbeatDetails` enables you to get the value from the last successful Heartbeat. ```php use Temporal\Activity; class FileProcessingActivitiesImpl implements FileProcessingActivities { // ... public function download( string $bucketName, string $remoteName, string $localName ): void { $this->dowloader->downloadWithProgress( $bucketName, $remoteName, $localName, // on progress function ($progress) { Activity::heartbeat($progress); } ); Activity::heartbeat(100); // download complete // ... } // ... } ``` --- # Best practices - PHP SDK Source: https://docs.temporal.io/develop/php/best-practices > This section explains how to implement best practices with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Best practices - [Testing](/develop/php/best-practices/testing-suite) - [Debugging](/develop/php/best-practices/debugging) --- # Debugging - PHP SDK Source: https://docs.temporal.io/develop/php/best-practices/debugging > Effectively debug your Workflow in both development and production environments using Web UI, Temporal CLI, and performance metrics for optimal Worker and Server performance. ## Debugging ### How to debug in a development environment In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). ### How to debug in a production environment You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). --- # Testing - PHP SDK Source: https://docs.temporal.io/develop/php/best-practices/testing-suite > The Temporal Application Testing section explains frameworks for Workflow and integration testing, including end-to-end, integration, unit tests, and how to mock Activities in a PHP environment. The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Testing Activities An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. ## Testing Workflows ### How to mock Activities Mock the Activity invocation when unit testing your Workflows. When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker. **RoadRunner config** To mock an Activity in PHP, use [RoadRunner Key-Value storage](https://github.com/spiral/roadrunner-kv) and add the following lines to your `tests/.rr.test.yaml` file. ```yaml # tests/.rr.test.yaml kv: test: driver: memory config: interval: 10 ``` If you want to be able to mock Activities, use `WorkerFactory` from the `Temporal\Testing` Namespace in your PHP Worker: ```php // worker.test.php use Temporal\Testing\WorkerFactory; $factory = WorkerFactory::create(); $worker = $factory->newWorker(); $worker->registerWorkflowTypes(MyWorkflow::class); $worker->registerActivity(MyActivity::class); $factory->run(); ``` Then, in your tests to mock an Activity, use the`ActivityMocker` class. Assume we have the following Activity: ```php #[ActivityInterface(prefix: "SimpleActivity.")] interface SimpleActivityInterface { #[ActivityMethod('doSomething')] public function doSomething(string $input): string; ``` To mock it in the test, you can do this: ```php final class SimpleWorkflowTestCase extends TestCase { private WorkflowClient $workflowClient; private ActivityMocker $activityMocks; protected function setUp(): void { $this->workflowClient = new WorkflowClient(ServiceClient::create('localhost:7233')); $this->activityMocks = new ActivityMocker(); parent::setUp(); } protected function tearDown(): void { $this->activityMocks->clear(); parent::tearDown(); } public function testWorkflowReturnsUpperCasedInput(): void { $this->activityMocks->expectCompletion('SimpleActivity.doSomething', 'world'); $workflow = $this->workflowClient->newWorkflowStub(SimpleWorkflow::class); $run = $this->workflowClient->start($workflow, 'hello'); $this->assertSame('world', $run->getResult('string')); } } ``` In the preceding test case, we do the following: 1. Instantiate `ActivityMocker` in the `setUp()` method of the test. 2. Clear the cache after each test in `tearDown()`. 3. Mock an Activity call to return a string `world`. To mock a failure, use the `expectFailure()` method: ```php $this->activityMocks->expectFailure('SimpleActivity.echo', new \LogicException('something went wrong')); ``` ### How to skip time Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Set up time skipping 1. In the `tests` folder, create `bootstrap.php` with the following contents: ```php declare(strict_types=1); require __DIR__ . '/../vendor/autoload.php'; use Temporal\Testing\Environment; $environment = Environment::create(); $environment->start(); register_shutdown_function(fn () => $environment->stop()); ``` If you don't want to run the test server with all of your tests, you can add a condition to start a test only if the `RUN_TEMPORAL_TEST_SERVER` environment variable is present: ```php if (getenv('RUN_TEMPORAL_TEST_SERVER') !== false) { $environment = Environment::create(); $environment->start('./rr serve -c .rr.silent.yaml --workflow-id tests'); register_shutdown_function(fn() => $environment->stop()); } ``` 2. Add `bootstrap.php` and the `TEMPORAL_ADDRESS` environment variable to `phpunit.xml`: ```xml ``` 3. Add the test server executable to `.gitignore`: ```gitignore temporal-test-server ``` ## How to Replay a Workflow Execution Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: To replay Workflow Executions, use the `\Temporal\Testing\Replay\WorkflowReplayer` class. In the following example, Event Histories are fetching from the Temporal, and then replayed. If the Workflow is non-deterministic, a `NonDeterministicWorkflowException` will be thrown. Note that this requires [Advanced Visibility](/visibility#advanced-visibility) to be enabled. ```php /** * We assume you already have a WorkflowClient and WorkflowReplayer in scope. * @var \Temporal\Client\WorkflowClientInterface $workflowClient * @var \Temporal\Testing\Replay\WorkflowReplayer $replayer */ // Find all workflow executions of type "MyWorkflow" and task queue "MyTaskQueue". $executions = $workflowClient->listWorkflowExecutions( "WorkflowType='MyWorkflow' AND TaskQueue='MyTaskQueue'" ); // Replay each workflow execution. foreach ($executions as $executionInfo) { try { $replayer->replayFromServer( workflowType: $executionInfo->type->name, execution: $executionInfo->execution, ); } catch (\Temporal\Testing\Replay\Exception\ReplayerException $e) { // Handle a replay error. } } ``` In the next example, an Event History is loaded from a JSON file, and the maximum number of replayed Events is limited to 42. ```php $replayer->replayFromJSON( workflowType: 'MyWorkflow', path: 'history.json', lastEventId: 42, // optional ); ``` You can download a Event History using PHP, and then replay it from a memorized History object: ```php $history = $this->workflowClient->getWorkflowHistory( execution: $run->getExecution(), )->getHistory(); (new WorkflowReplayer())->replayHistory($history); ``` --- # Client - PHP SDK Source: https://docs.temporal.io/develop/php/client > This section explains how to implement the Temporal Client with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Temporal Client - [Temporal Client](/develop/php/client/temporal-client) --- # Temporal Client - PHP SDK Source: https://docs.temporal.io/develop/php/client/temporal-client > Connect a Temporal Client to a Temporal Service and start Workflow Executions. This guide covers communication, including sending signals and queries. This guide introduces Temporal Clients. It explains the role and use of Clients and shows you how to configure your PHP Client code to connect to the Temporal Service. This page shows how to do the following: - [Connect to a local development Temporal Service](#connect-to-a-dev-cluster) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow-execution) - [Advanced connection options](#advanced-connection-options) ## How to connect a Temporal Client to a Temporal Service A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the [Temporal Service](/temporal-service). Communication with a Temporal Service includes, but isn't limited to, the following: - Scheduling Workflow Executions. - Starting Workflow Executions. - Sending Signals to Workflow Executions. - Sending Queries to Workflow Executions. - Sending Updates to Workflow Executions. - Getting the results of a Workflow Execution. - Providing an Activity Task Token. > **⚠️ Caution:** > > A Temporal Client cannot be initialized and used inside a Workflow. > However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service. > When you are running a Temporal Service locally (such as the [Temporal CLI](/cli/command-reference/server#start-dev)), the number of connection options you must provide is minimal. Many SDKs default to `127.0.0.1:7233`. In the PHP SDK, different client classes are responsible for different functional areas. The [`ServiceClient`](https://php.temporal.io/classes/Temporal-Client-GRPC-ServiceClient.html) is responsible for the low-level API and connection to the Temporal Service. It is also used in higher-level clients: [`WorkflowClient`](https://php.temporal.io/classes/Temporal-Client-WorkflowClient.html) and [`ScheduleClient`](https://php.temporal.io/classes/Temporal-Client-ScheduleClient.html). > **📝 Note:** > > RoadRunner is not required to work only with the client API; however, the [gRPC extension](https://pecl.php.net/package/grpc) is necessary. > Use `create()` factory methods to create clients. ```php use Temporal\Client\GRPC\ServiceClient; use Temporal\Client\WorkflowClient; $serviceClient = ServiceClient::create('localhost:7233'); $workflowClient = WorkflowClient::create($serviceClient); // Use $workflowClient to work with Workflows ... ``` See the [Advanced connection options](#advanced-connection-options) section for more information on configuring the connection. ## How to connect a Temporal Client to a Temporal Cloud When you connect to [Temporal Cloud](/cloud), you need to provide additional connection and client options that include the following: - The [Temporal Cloud Namespace Id](/cloud/namespaces#temporal-cloud-namespace-id). - The [Namespace's gRPC endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint). An endpoint listing is available at the [Temporal Cloud Website](https://cloud.temporal.io/namespaces) on each Namespace detail page. The endpoint contains the Namespace Id and port. - mTLS CA certificate. - mTLS private key. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server). Use the [`ServiceClient::createSSL()`](https://php.temporal.io/classes/Temporal-Client-GRPC-BaseClient.html#method_createSSL) method to configure a client connection to the Temporal Service. The `$clientKey` argument must be combined with the `$clientPem` to authenticate the Client. ```php use Temporal\Client\ClientOptions; use Temporal\Client\GRPC\ServiceClient; use Temporal\Client\WorkflowClient; $serviceClient = \Temporal\Client\GRPC\ServiceClient::createSSL( address: '.tmprl.cloud:7233', // crt: 'certs/server-root-ca-cert.pem', # ROOT CA to validate the server cert clientKey: 'certs/client-private-key.pem', clientPem: 'certs/client-cert.pem', // overrideServerName: 'tls-sample', ); $workflowClient = WorkflowClient::create( serviceClient: $serviceClient, options: (new ClientOptions()) ->withNamespace('.'), ); ``` To [run Worker processes](/develop/php/workers/run-worker-process#run-a-dev-worker) managed by Temporal Cloud, configure RoadRunner in the same way. ```yml temporal: # ... tls: # root_ca: 'certs/server-root-ca-cert.pem' key: 'certs/client-private-key.pem' cert: 'certs/client-cert.pem' client_auth_type: require_and_verify_client_cert # server_name: 'tls-sample' ``` To set up the [API key](/cloud/api-keys) in the Client, use the [`ServiceClient::withAuthKey()`](https://php.temporal.io/classes/Temporal-Client-GRPC-BaseClient.html#method_withAuthKey) method: ```php $serviceClient = \Temporal\Client\GRPC\ServiceClient::createSSL(/*...*/) ->withAuthKey('your-api-key'); ``` Rotating an mTLS client certificate without restarting the Worker isn't currently supported by the PHP SDK or RoadRunner — the certificate configured on `ServiceClient::createSSL()` or in RoadRunner's `tls:` block is fixed for the life of the process. To rotate a certificate, stage the new certificate alongside the old one on your Temporal Cloud Namespace, then restart your Worker (RoadRunner process) with the new certificate before removing the old one. See [Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime staging sequence. ## How to start a Workflow Execution [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. In the examples following all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from within another Workflow Execution, use either the [Child Workflow](/develop/php/workflows/child-workflows) or External Workflow APIs. See the [Customize Workflow Type](/develop/php/workflows/basics#workflow-type) section to see how to customize the name of the Workflow Type. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. Use Workflow stub to start a Workflow Execution from within a Client. Workflow stub is a proxy generated by the [`WorkflowClient`](https://php.temporal.io/classes/Temporal-Client-WorkflowClient.html). You can use a typed or untyped Workflow stub in the client code. - Typed Workflow stubs are useful because they are type safe and allow you to invoke your Workflow methods such as `#[WorkflowMethod]`, `#[QueryMethod]`, `#[SignalMethod]`, and `#[UpdateMethod]` directly. - An untyped Workflow stub does not use a Workflow interface. It is more flexible because it has methods from the [`WorkflowStubInterface`](https://php.temporal.io/classes/Temporal-Client-WorkflowStubInterface.html), such as `start`, `signal`, `getResults`, `query`, `update`, `cancel`, and `terminate` When using untyped Workflow stub, we rely on the Workflow Type, Activity Type, Child Workflow Type, as well as Query and Signal names. For example, there is a Workflow defined as follows: ```php #[WorkflowInterface] interface AccountTransferWorkflowInterface { #[WorkflowMethod(name: "account.transfer")] public function begin(UuidInterface $transactionId); #[UpdateMethod(name: "pay")] public function move(UuidInterface $from, UuidInterface $to, int $amount); #[UpdateMethod(name: "finish")] public function commit(); #[UpdateMethod(name: "cancel")] public function rollback(string $reason); } ``` In case of a **typed** Workflow stub, you can use the `AccountTransferWorkflowInterface` to call the Workflow methods directly: ```php $stub = $workflowClient->newWorkflowStub(AccountTransferWorkflowInterface::class); $workflowClient->start($stub, $transactionId); $stub->move($from1, $to1, $amount1); $stub->move($from2, $to2, $amount2); $stub->commit(); ``` In case of an **untyped** Workflow stub, you need to specify Workflow Type and method names explicitly: ```php $stub = $workflowClient->newUntypedWorkflowStub('account.transfer'); $workflowClient->start($stub, $transactionId); $stub->update('pay', $from1, $to1, $amount1); $stub->update('pay', $from2, $to2, $amount2); $stub->update('finish'); ``` A Workflow Execution can be started either synchronously or asynchronously. **Synchronous start** A synchronous start initiates a Workflow and then waits for its completion. The started Workflow will not rely on the invocation process and will continue executing even if the waiting process crashes or stops. Be sure to acquire the Workflow interface or class name you want to start. For example: ```php #[WorkflowInterface] interface AccountTransferWorkflowInterface { #[WorkflowMethod(name: "MoneyTransfer")] #[ReturnType(UuidInterface::class)] public function transfer( string $fromAccountId, string $toAccountId, string $referenceId, int $amountCents); } ``` To start the Workflow in sync mode: ```php $accountTransfer = $workflowClient->newWorkflowStub( AccountTransferWorkflowInterface::class, ); $result = $accountTransfer->transfer('fromID', 'toID', 'refID', 1000); ``` **Asynchronous start** An asynchronous start initiates a Workflow Execution and immediately returns to the caller without waiting for a result. This is the most common way to start Workflows in a live environment. To start a Workflow asynchronously, pass the Workflow stub instance and start parameters into the [`WorkflowClient::start()`](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_start) method. ```php $accountTransfer = $workflowClient->newWorkflowStub( AccountTransferWorkflowInterface::class, ); $run = $this->workflowClient->start($accountTransfer, 'fromID', 'toID', 'refID', 1000); ``` After the Workflow is started, you can receive details about the Workflow Execution or result via the [`WorkflowRun`](https://php.temporal.io/classes/Temporal-Workflow-WorkflowRunInterface.html) object methods: ```php $run = $workflowClient->start($accountTransfer, 'fromID', 'toID', 'refID', 1000); // Get the Workflow ID var_dump($run->getExecution()->getID()); // Describe the Workflow Execution var_dump($run->describe()); // Wait for the Workflow to complete and get the result with 10-second timeout var_dump($run->getResult(timeout: 10)); ``` **Recurring start** You can start a Workflow Execution on a regular schedule with [the CronSchedule option](/develop/php/workflows/schedules#temporal-cron-jobs). ### How to set a Workflow's Task Queue In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue). When developing in PHP, the Task Queue name defaults to `"default"`. While setting a meaningful Task Queue name is recommended for better observability, Workflows can be run without setting this option. > **📝 Note:** > > PHP's default is different from most SDKs, which do require an explicit Task Queue name. > For your code to execute, a Worker Process must be running ([how to run Worker Processes](/develop/php/workers/run-worker-process#run-a-dev-worker)). This process needs a Worker Entity that is polling the same Task Queue name. Set the Workflow Task Queue with the Workflow stub in the Client code using [`WorkflowOptions::withTaskQueue()`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html#method_withTaskQueue). ```php $stub = $workflowClient->newWorkflowStub( YourWorkflowInterface::class, WorkflowOptions::new() ->withTaskQueue("Workflow-Task-Queue-1"), ); ``` ### How to set a Workflow Id Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id)that maps to a business process or business entity identifier, such as an order identifier or customer identifier. Set the Workflow Id with the Workflow stub in the Client code using [`WorkflowOptions::withTaskQueue()`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html#method_withWorkflowId). ```php $stub = $workflowClient->newWorkflowStub( YourWorkflowInterface::class, WorkflowOptions::new() ->withWorkflowId("Workflow-Id"), ); ``` ### How to get the results of a Workflow Execution If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. If you need to wait for the completion of a Workflow after an asynchronous start, make a blocking call to the `WorkflowRun::getResult()` method. ```php $stub = $workflowClient->newWorkflowStub(YourWorkflowInterface::class); $run = $workflowClient->start($stub, 'fromID', 'toID', 'refID', 1000); var_dump($run->getResult()); ``` In case of untyped Workflow stub, you can use the [`WorkflowStub::getResult()`](https://php.temporal.io/classes/Temporal-Workflow-WorkflowRunInterface.html#method_getResult) method: ```php $stub = $workflowClient->newUntypedWorkflowStub('account.transfer'); $workflowClient->start($stub, 'fromID', 'toID', 'refID', 1000); var_dump($stub->getResult(timeout: 5.5)); ``` Note that you can specify a timeout for the `getResult()` method in seconds. If the Workflow does not complete within the specified time, a `TimeoutException` will be thrown. See how to limit all RPC calls in the [RPC timeout](#configure-rpc-timeout) section. ## Advanced connection options In PHP, it is common practice to work with resources in blocking mode. Long blocks can quickly exhaust the pool of available workers and lead to application failure. This section introduces features and configuration examples of the PHP SDK when working with the Temporal Client API. ### Connection gRPC connections in the PHP SDK are lazy by default, meaning they are not established until the first call. To force establishing the connection to the Temporal Service, you can call the [`ConnectionInterface::connect()`](https://php.temporal.io/classes/Temporal-Client-GRPC-Connection-ConnectionInterface.html#method_connect) or [`ServiceClient::getServerCapabilities()`](https://php.temporal.io/classes/Temporal-Client-GRPC-ServiceClientInterface.html#method_getServerCapabilities) method. ```php // ... $serviceClient->getConnection()->connect(timeout: 10); // or $serviceClient->getServerCapabilities(); ``` If, for some reason, the established connection is broken, the SDK will automatically attempt to restore it, taking into account the configured retry policy. ### Retry policy Whenever the client fails to connect to the server, an error with a status code is generated. If the status code is `UNKNOWN`, `UNAVAILABLE`, or `RESOURCE_EXHAUSTED`, the client will make another connection attempt. By default, the number of attempts is unlimited, and the interval between them will range from 0.5 to 100 seconds with a backoff coefficient of 2. This means that the client will likely be blocked until it establishes a connection to the server through infinite attempts. If you want to change the default behavior, use the `withRetryPolicy()` method when creating a client service: ```php use Temporal\Client\Common\RpcRetryOptions; use Temporal\Client\GRPC\ServiceClient; use Temporal\Client\WorkflowClient; $serviceClient = ServiceClient::create('localhost:7233'); $workflowClient = WorkflowClient::create($serviceClient) ->withRetryOptions( RpcRetryOptions::new() ->withMaximumAttempts(10) ->withInitialInterval('1 second') // The first retry will be in 1 second ->withBackoffCoefficient(2.5) // Each next retry time will be multiplied by 2.5 ->withMaximumInterval('20 seconds') // The maximum interval between attempts ->withMaximumJitterCoefficient(0.2) // Actual retry time can be +/- 20% of the calculated time ); ``` ### RPC timeout When the client calls the service's RPC, there is no default time limit for waiting for a response. This can result in the code call `$result = $workflowHandle->getResult();` blocking the PHP worker until the Workflow completes. In some cases, this is not the desired behavior, and there may be a need to set a reasonable timeout for waiting for the RPC to complete. Use the `withTimeout()` method to build a client with a timeout for all RPC calls. ```php use Temporal\Client\GRPC\ServiceClient; use Temporal\Client\WorkflowClient; $serviceClient = ServiceClient::create('localhost:7233'); $workflowClient = WorkflowClient::create($serviceClient) ->withTimeout(5.75); // Create a Workflow stub $stub = $workflowClient->newWorkflowStub(AccountTransferWorkflowInterface::class); // If the Workflow does not complete within 5.75 seconds, a TimeoutException will be thrown $result = $stub->transfer('fromID', 'toID', 'refID', 1000); ``` > **📝 Note:** > > The `withTimeout()` method is immutable. > If you need to change the timeout for individual operations, create a new client from the existing one with a specific timeout: `$newClient = $workflowClient->withTimeout(0);` (`0` means no timeout). > --- # Platform - PHP SDK Source: https://docs.temporal.io/develop/php/platform > This section explains how to implement platform with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Platform - [Observability](/develop/php/platform/observability) - [Enriching the UI](/develop/php/platform/enriching-ui) --- # Enriching the user interface - PHP SDK Source: https://docs.temporal.io/develop/php/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the PHP SDK. Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```php use Temporal\Client\WorkflowClient; use Temporal\Client\WorkflowOptions; // Create workflow client $workflowClient = WorkflowClient::create($serviceClient); // Start a workflow with static summary and details $workflow = $workflowClient->newWorkflowStub( YourWorkflow::class, WorkflowOptions::new() ->withWorkflowId('your-workflow-id') ->withTaskQueue('your-task-queue') ->withStaticSummary('Order processing for customer #12345') ->withStaticDetails('Processing premium order with expedited shipping') ); $result = $workflow->yourWorkflowMethod('workflow input'); ``` `withStaticSummary()` sets a single-line description that appears in the Workflow list view, limited to 200 bytes. `withStaticDetails()` sets multi-line comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also start a Workflow asynchronously: ```php // Start workflow asynchronously $workflowClient->start($workflow, 'workflow input'); ``` ### Adding Summary to Activities and Timers You can attach a `summary` to timers within a workflow: ```php use Temporal\Workflow; use Temporal\Workflow\TimerOptions; #[WorkflowInterface] interface YourWorkflow { #[WorkflowMethod] public function yourWorkflowMethod(string $input): string; } class YourWorkflowImpl implements YourWorkflow { public function yourWorkflowMethod(string $input): \Generator { // Create a timer with a summary yield Workflow::timer( 300, // 5 minutes in seconds TimerOptions::new()->withSummary('Waiting for payment confirmation') ); return 'Timer completed'; } } ``` For Activities, you can set a summary using Activity options: ```php use Temporal\Activity\ActivityOptions; use Temporal\Workflow; class YourWorkflowImpl implements YourWorkflow { private YourActivitiesInterface $activities; public function __construct() { $this->activities = Workflow::newActivityStub( YourActivitiesInterface::class, ActivityOptions::new() ->withStartToCloseTimeout('10 seconds') ->withSummary('Processing user data') ); } public function yourWorkflowMethod(string $input): \Generator { // Execute the activity with the summary $result = yield $this->activities->yourActivity($input); return $result; } } ``` The input format for `summary` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the Workflow details page, you'll find the Workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the Workflow - **Current Details** - Displays the dynamic details that can be updated during Workflow Execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. --- # Observability - PHP SDK Source: https://docs.temporal.io/develop/php/platform/observability > Explore the Temporal Developer’s guide on observability to learn about Visibility APIs and Search Attributes, helping you manage Workflow Executions efficiently. The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. This section covers features related to viewing the state of the application, including: - [Log from a Workflow](#logging) - [Visibility](#visibility) ## Log from a Workflow Logging enables you to record critical information during code execution. Loggers create an audit trail and capture information about your Workflow's operation. An appropriate logging level depends on your specific needs. During development or troubleshooting, you might use debug or even trace. In production, you might use info or warn to avoid excessive log volume. You can find the log levels supported by `PSR-3` in [their official documentation](https://www.php-fig.org/psr/psr-3/#5-psrlogloglevel). The Temporal SDK core normally uses `WARN` as its default logging level. To get a PSR-3 compatible logger in your Workflow code, use the [`Workflow::getLogger()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_getLogger) method. ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class MyWorkflow { #[Workflow\WorkflowMethod] public function execute(string $param): \Generator { Workflow::getLogger()->info('Workflow started', ['parameter' => $param]); // Your workflow implementation Workflow::getLogger()->info('Workflow completed'); return 'Done'; } } ``` The Workflow logger automatically enriches log context with the current Task Queue name. Logs in replay mode are omitted unless the [`enableLoggingInReplay`](https://php.temporal.io/classes/Temporal-Worker-WorkerOptions.html#method_withEnableLoggingInReplay) Worker option is set to true. ```php $factory = WorkerFactory::create(); $worker = $factory->newWorker('your-task-queue', WorkerOptions::new() ->withEnableLoggingInReplay(true) ); ``` ### Default Logger By default, PHP SDK uses a [`StderrLogger`](https://php.temporal.io/classes/Temporal-Worker-Logger-StderrLogger.html) that outputs log messages to the standard error stream. These messages are automatically captured by RoadRunner and incorporated into its logging system with the INFO level, ensuring proper log collection in both development and production environments. For more details on RoadRunner's logging capabilities, see the [RoadRunner Logger documentation](https://docs.roadrunner.dev/docs/logging-and-observability/logger). ### How to provide a custom logger You can set a custom PSR-3 compatible logger when creating a Worker: ```php $myLogger = new MyLogger(); $workerFactory = WorkerFactory::create(converter: $converter); $worker = $workerFactory->newWorker( taskQueue: 'my-task-queue', logger: $myLogger, ); ``` ## Visibility APIs The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service. ### How to use Search Attributes The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI. - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an option when starting the Execution. - In the Workflow by calling `UpsertSearchAttributes`. - Read the value of the Search Attribute: - On the Client by calling `DescribeWorkflow`. - In the Workflow by looking at `WorkflowInfo`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/workflow#list). - In code by calling `ListWorkflowExecutions`. Here is how to query Workflow Executions: Use the [listWorkflowExecutions()](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_listWorkflowExecutions) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows. The result is an iterable paginator, so you can use the `foreach` loop to iterate over the results. ```php $paginator = $workflowClient->listWorkflowExecutions('WorkflowType="GreetingWorkflow"'); foreach ($paginator as $info) { echo "Workflow ID: {$info->execution->getID()}\n"; } ``` ### How to set custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. To set custom Search Attributes, use the `withTypedSearchAttributes` method on `WorkflowOptions` for a Workflow stub. Typed search attributes are a `TypedSearchAttributes` collection. ```php $keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime'); $keyOrderId = SearchAttributeKey::forKeyword('OrderId'); $workflow = $workflowClient->newWorkflowStub( OrderWorkflowInterface::class, WorkflowOptions::new() ->withWorkflowExecutionTimeout('10 minutes') ->withTypedSearchAttributes( TypedSearchAttributes::empty() ->withValue($keyOrderId, $orderid) ->withValue($keyDestinationTime, new \DateTimeImmutable('2028-11-05T00:10:07Z')) ), ); ``` ### How to upsert Search Attributes Within the Workflow code, you can dynamically add or update Search Attributes using [`upsertTypedSearchAttributes`](https://php.temporal.io/classes/Temporal-Workflow.html#method_upsertTypedSearchAttributes). This method is particularly useful for Workflows whose attributes need to change based on internal logic or external events. ```php #[Workflow\UpdateMethod] public function postponeDestinationTime(\DateInterval $interval) { // Get the key for the DestinationTime attribute $keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime'); /** @var DateTimeImmutable $destinationTime */ $destinationTime = Workflow::getInfo()->typedSearchAttributes->get($keyDestinationTime); Workflow::upsertTypedSearchAttributes( $keyDestinationTime->valueSet($destinationTime->add($interval)), ); } ``` ### How to remove a Search Attribute from a Workflow To remove a Search Attribute that was previously set, set it to an empty Map. ```php #[Workflow\UpdateMethod] public function unsetDestinationTime() { // Get the key for the DestinationTime attribute $keyDestinationTime = SearchAttributeKey::forDatetime('DestinationTime'); Workflow::upsertTypedSearchAttributes( $keyDestinationTime->valueUnset(), ); } ``` --- # Set up your local development with the PHP SDK Source: https://docs.temporal.io/develop/php/set-up-your-local-php > Configure your local development environment to get started developing with Temporal # Quickstart Configure your local development environment to get started developing with Temporal. ## Install PHP Make sure you have PHP installed. **If you don't have PHP:** Visit the official website to [download and install](https://www.php.net/downloads.php) it. ### GRPC extension GRPC extension is required to work with RoadRunner application server. **If you don't have `ext-grpc` installed:** Visit the official website to [download and install](https://docs.cloud.google.com/php/docs/reference/help/grpc) it. > **💡 Tip:** > GRPC Installation Tip > > On macOS with Apple Silicon (M1/M2/M3/M4) and PHP 8.3, `pecl install grpc` may appear to hang or install indefinitely. > If this happens, try installing a specific version: > > ```bash > pecl install channel://pecl.php.net/grpc-1.78.0RC2 > ``` > > Note: You can find the latest versions at [pecl.php.net/package/grpc](https://pecl.php.net/package/grpc). > ```bash php -v ``` ## Create a Project Now that you have PHP installed, create a project to manage your dependencies and build your Temporal application. ```bash mkdir temporal-hello-world ``` ```bash cd temporal-hello-world ``` ```bash composer init --name="myproject/quickstart" -n ``` ## Add Temporal PHP SDK and Configure Autoloading Install the Temporal SDK, then add PSR-4 autoloading to your `composer.json` so PHP can find your Workflow and Activity classes. Your final `composer.json` should look like this. After updating, run `composer dump-autoload` to regenerate the autoloader. ```bash composer require temporal/sdk ``` ```json { "name": "myproject/quickstart", "require": { "temporal/sdk": "^2.16" }, "autoload": { "psr-4": { "App\\\\": "src/" } } } ``` ```bash composer dump-autoload ``` ## Install RoadRunner application server Install [RoadRunner application server](https://github.com/roadrunner-server/roadrunner). It starts and manages your PHP processes that run Temporal Workers, and connects them to the Temporal Service over gRPC. See [RoadRunner installation instructions](https://docs.roadrunner.dev/docs/general/install) to learn about other installation methods. **CLI** Download RoadRunner with the following command: ```bash ./vendor/bin/rr get ``` When prompted "Do you want create default '.rr.yaml' configuration file?", answer **yes**. You'll replace it with the proper config in the next step. **DLoad** Install DLoad package manager using Composer ```bash composer require --dev internal/dload ``` Create a configuration file named `dload.xml` with the following content: ```xml ``` Finally, download the RoadRunner binary: ```bash ./vendor/bin/dload ``` Create a simple configuration file named `.rr.yaml` with the following content: ```yml version: "3" rpc: listen: tcp://127.0.0.1:6001 server: command: "php worker.php" temporal: address: "127.0.0.1:7233" logs: level: info ``` ## Install Temporal CLI and start the development server The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI: **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ### DLoad package manager Consider using DLoad to delegate all installation and updating processes to the package manager. Add one more download action to the configuration file ```xml ``` The final configuration file should look like this: ```xml ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal. Keep this running in the background: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal PHP SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Create the Activity Create an Activity file (`src/GreetingActivity.php`): ```php withStartToCloseTimeout(5), ); return yield $activity->greet($name); } } ``` ### 3. Create a Worker file Create a Worker file (`worker.php`, under project root directory): ```php newWorker(); // Register Workflows $worker->registerWorkflowTypes(\App\SayHelloWorkflow::class); // Register Activities $worker->registerActivity(\App\GreetingActivity::class); $factory->run(); ``` #### Run the Worker Previously, we created a Worker that executes Workflow and Activity tasks. Now, start the RoadRunner application server to run the Worker by opening up a new terminal window and running this command: ```bash ./rr serve ``` A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue, it then executes the task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). ### 5. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. This final step will validate that everything is working correctly with your file labeled `client.php`. Create a separate file called `client.php`: ```php newWorkflowStub(\App\SayHelloWorkflow::class); $result = $workflowStub->sayHello('Temporal'); echo "Result: {$result}\n"; ``` While your Worker is still running, open a new terminal and run: ```bash php client.php ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the workflow and activity - Output: `Result: Hello, Temporal!` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/php/hello_world_in_php/): Create a basic Workflow and run it with the Temporal PHP SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Workers - PHP SDK Source: https://docs.temporal.io/develop/php/workers > This section explains how to implement Workers with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Workers - [Run Worker processes](/develop/php/workers/run-worker-process) --- # Run Worker processes - PHP SDK Source: https://docs.temporal.io/develop/php/workers/run-worker-process > Shows how to run Worker processes with the PHP SDK ## How to run Worker Processes The [Worker Process](/workers#worker-process) is where Workflow Functions and Activity Functions are executed. - Each [Worker Entity](/workers#worker-entity) in the Worker Process must register the exact Workflow Types and Activity Types it may execute. - Each Worker Entity must also associate itself with exactly one [Task Queue](/task-queue). - Each Worker Entity polling the same Task Queue must be registered with the same Workflow Types and Activity Types. A [Worker Entity](/workers#worker-entity) is the component within a Worker Process that listens to a specific Task Queue. Although multiple Worker Entities can be in a single Worker Process, a single Worker Entity Worker Process may be perfectly sufficient. For more information, see the [Worker tuning guide](/develop/worker-performance). A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively. The [RoadRunner application server](https://roadrunner.dev/) will launch multiple Temporal PHP Worker processes based on provided `.rr.yaml` configuration. Each Worker might connect to one or multiple Task Queues. Workers poll the _Temporal Service_ for tasks, perform those tasks, and communicate task execution results back to the _Temporal Service_. Worker code is developed, deployed, and operated by Temporal customers. To create a worker use `Temporal\WorkerFactory`: ```php newWorker(); // Workflows are stateful. So you need a type to create instances. $worker->registerWorkflowTypes(App\DemoWorkflow::class); // Activities are stateless and thread safe. So a shared instance is used. $worker->registerActivity(App\DemoActivity::class); // In case an activity class requires some external dependencies provide a callback - factory // that creates or builds a new activity instance. The factory should be a callable which accepts // an instance of ReflectionClass with an activity class which should be created. $worker->registerActivity(App\DemoActivity::class, fn(ReflectionClass $class) => $container->create($class->getName())); // start primary loop $factory->run(); ``` You can configure task queue name using first argument of `WorkerFactory`->`newWorker`: ```php $worker = $factory->newWorker('your-task-queue'); ``` As mentioned preceding, you can create as many Task Queue connections inside a single Worker Process as you need. To configure additional WorkerOptions use `Temporal\Worker\WorkerOptions`: ```php use Temporal\Worker\WorkerOptions; $worker = $factory->newWorker( 'your-task-queue', WorkerOptions::new() ->withMaxConcurrentWorkflowTaskPollers(10) ); ``` Make sure to point the Worker file in application server configuration: ```yaml rpc: listen: tcp://127.0.0.1:6001 server: command: 'php worker.php' temporal: address: 'temporal:7233' activities: num_workers: 10 ``` > You can serve HTTP endpoints using the same server setup. To provide the [API key](/cloud/api-keys) to RoadRunner use a `ServiceCredentials` DTO when creating the WorkerFactory: ```php use Temporal\Worker\ServiceCredentials; $workerFactory = \Temporal\WorkerFactory::create( credentials: ServiceCredentials::create()->withApiKey('your-api-key'), ); ``` [How to configure connection to a Temporal Cloud](/develop/php/client/temporal-client#connect-to-temporal-cloud) ### How to register types All Workers listening to the same Task Queue name must be registered to handle the exact same Workflows Types and Activity Types. If a Worker polls a Task for a Workflow Type or Activity Type it does not know about, it fails that Task. However, the failure of the Task does not cause the associated Workflow Execution to fail. Worker listens on a Task Queue and hosts both Workflow and Activity implementations: ```php // Workflows are stateful. So you need a type to create instances: $worker->registerWorkflowTypes(App\DemoWorkflow::class); // Activities are stateless and thread safe: $worker->registerActivity(App\DemoActivity::class); ``` In case an activity class requires some external dependencies provide a callback - factory that creates or builds a new activity instance. The factory should be a callable which accepts an instance of ReflectionClass with an activity class which should be created. ```php $worker->registerActivity( App\DemoActivity::class, fn(ReflectionClass $class) => $container->create($class->getName()) ); ``` If you want to clean up some resources after activity is done, you may register a finalizer. This callback is called after each activity invocation: ```php $worker->registerActivityFinalizer(fn() => $kernel->shutdown()); ``` --- # Workflows - PHP SDK Source: https://docs.temporal.io/develop/php/workflows > This section explains how to implement Workflows with the PHP SDK ![PHP SDK Banner](/img/assets/banner-php-temporal.png) ## Workflows - [Workflow Basics](/develop/php/workflows/basics) - [Child Workflows](/develop/php/workflows/child-workflows) - [Continue-As-New](/develop/php/workflows/continue-as-new) - [Cancellation](/develop/php/workflows/cancellation) - [Timeouts](/develop/php/workflows/timeouts) - [Message Passing](/develop/php/workflows/message-passing) - [Schedules](/develop/php/workflows/schedules) - [Timers](/develop/php/workflows/timers) - [Side effects](/develop/php/workflows/side-effects) - [Versioning](/develop/php/workflows/versioning) --- # Workflow Basics - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/basics > This section explains Workflow Basics with the PHP SDK ## How to develop a basic Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal PHP SDK programming model, Workflows are a class method. Classes must implement interfaces that are annotated with `#[WorkflowInterface]`. The method that is the Workflow must be annotated with `#[WorkflowMethod]`. ```php use Temporal\Workflow\YourWorkflowInterface; use Temporal\Workflow\WorkflowMethod; #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] public function processFile(Argument $args); } ``` ### How to define Workflow parameters Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. A method annotated with `#[WorkflowMethod]` can have any number of parameters. We recommend passing a single parameter that contains all the input fields to allow for adding fields in a backward-compatible manner. Note that all inputs should be serializable to a byte array using the provided [DataConverter](https://github.com/temporalio/sdk-php/blob/master/src/DataConverter/DataConverterInterface.php) interface. The default implementation uses a JSON serializer, but an alternative implementation can be easily configured. You can create a custom object and pass it to the Workflow method, as shown in the following example: ```php #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] public function processFile(Argument $args); } ``` ### How to define Workflow return parameters Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error. A Workflow method returns a Generator. To properly typecast the Workflow's return value in the client code, use the `#[ReturnType()]` attribute. ```php #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] #[ReturnType("string")] public function processFile(Argument $args); } ``` ### How to customize your Workflow Type Workflows have a Type that are referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. To customize a Workflow Type, use the `WorkflowMethod` attribute to specify the name of Workflow. ```php #[WorkflowMethod(name)] ``` If a Workflow Type is not specified, then Workflow Type defaults to the interface name, which is `YourWorkflowDefinitionInterface` in this case. ```php #[WorkflowInterface] interface YourWorkflowDefinitionInterface { #[WorkflowMethod] public function processFile(Argument $args); } ``` ### Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `#[WorkflowInit]` attribute on your constructor, you can give it the same [Workflow parameters](/develop/php/workflows/basics#workflow-parameters) as your `#[WorkflowMethod]`. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/php/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `#[WorkflowMethod]` method. That always happens, whether or not you use the `#[WorkflowInit]` attribute. Here's an example. Notice that the constructor and `getGreeting` must have the same parameters: ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class GreetingExample { private readonly string $nameWithTitle; private bool $titleHasBeenChecked; // Note the attribute is on a public constructor #[Workflow\WorkflowInit] public function __construct(string $input) { $this->nameWithTitle = 'Sir ' . $input; $this->titleHasBeenChecked = false; } #[Workflow\WorkflowMethod] public function getGreeting(string $input) { yield Workflow::await(fn() => $this->titleHasBeenChecked); return "Hello " . $this->nameWithTitle; } } ``` ### How to develop Workflow logic Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with application code outside the Workflow. used inside your Workflow to interact with external (to the Workflow) application code. \*\*Temporal uses the [Microsoft Azure Event Sourcing pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing) to recover the state of a Workflow object including its local variable values. In essence, every time a Workflow state has to be restored, its code is re-executed from the beginning. When replaying, side effects (such as Activity invocations) are ignored because they are already recorded in the Workflow event history. When writing Workflow logic, the replay is not visible, so the code should be written since it executes only once. This design puts the following constraints on the Workflow implementation: - Do not use any mutable global variables because multiple instances of Workflows are executed in parallel. - Do not call any non-deterministic functions like non seeded random or `UUID` directly from the Workflow code. Always do the following in the Workflow implementation code: - Don't perform any IO or service calls as they are not usually deterministic. Use Activities for this. - Only use `Workflow::now()` to get the current time inside a Workflow. - Call `yield Workflow::timer()` instead of `sleep()`. - Do not use any blocking SPL provided by PHP (that is, `fopen`, `PDO`, etc) in **Workflow code**. - Use `yield Workflow::getVersion()` when making any changes to the Workflow code. Without this, any deployment of updated Workflow code might break already open Workflows. - Don't access configuration APIs directly from a Workflow because changes in the configuration might affect a Workflow Execution path. Pass it as an argument to a Workflow function or use an Activity to load it. Workflow method arguments and return values are serializable to a byte array using the provided [DataConverter](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/common/converter/DataConverter.html) interface. The default implementation uses JSON serializer, but you can use any alternative serialization mechanism. Make sure to annotate your `WorkflowMethod` using `ReturnType` to specify concrete return type. > You can not use the default return type declaration as Workflow methods are generators. The values passed to Workflows through invocation parameters or returned through a result value are recorded in the execution history. The entire execution history is transferred from the Temporal service to Workflow workers with every event that the Workflow logic needs to process. A large execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data that you transfer via Activity invocation parameters or return values. Otherwise, no additional limitations exist on Activity implementations.\*\* --- # Cancel a Workflow - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/cancellation ## Cancel an Activity from a Workflow Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate Activity is executed, the Activity Execution should send Heartbeats and set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is still working. When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity to appear canceled the exception needs to be re-raised. > **📝 Note:** > > Unlike regular Activities, [Local Activities](/local-activity) can be canceled if they don't send Heartbeats. Local > Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same > Worker process. > ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/child-workflows > Start a Child Workflow Execution within a parent Workflow using Temporal in PHP. Configure ChildWorkflowOptions, handle Parent Close Policy, and implement asynchronous calls with promises. ## How to start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In PHP, yielding `$child->start()` or `Workflow::executeChildWorkflow()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the yield resolves. See the [Parent Close Policy](#parent-close-policy) section below for an example. Besides Activities, a Workflow can also start other Workflows. `Workflow::executeChildWorkflow` and `Workflow::newChildWorkflowStub` enables the scheduling of other Workflows from within a Workflow's implementation. The parent Workflow has the ability to monitor and impact the lifecycle of the Child Workflow, similar to the way it does for an Activity that it invoked. ```php // Use one stub per child workflow run $child = Workflow::newChildWorkflowStub( ChildWorkflowInterface::class, ChildWorkflowOptions::new() // Do not specify WorkflowId if you want Temporal to generate a unique Id // for the child execution. ->withWorkflowId('BID-SIMPLE-CHILD-WORKFLOW') ->withExecutionStartToCloseTimeout(DateInterval::createFromDateString('30 minutes')) ); // This is a non blocking call that returns immediately. // Use yield $child->workflowMethod(name) to call synchronously. $promise = $child->workflowMethod('value'); // Do something else here. try{ $value = yield $promise; } catch(TemporalException $e) { $logger->error('child workflow failed'); throw $e; } ``` Let's take a look at each component of this call. Before calling `$child->workflowMethod()`, you must configure `ChildWorkflowOptions` for the invocation. These options customize various execution timeouts, and are passed into the Workflow stub defined by the `Workflow::newChildWorkflowStub`. Once stub created you can invoke its Workflow method based on attribute `WorkflowMethod`. The method call returns immediately and returns a `Promise`. This allows you to execute more code without having to wait for the scheduled Workflow to complete. When you are ready to process the results of the Workflow, call the `yield $promise` method on the returned promise object. When a parent Workflow is cancelled by the user, the Child Workflow can be cancelled or abandoned based on a configurable child policy. You can also skip the stub part of Child Workflow initiation and use `Workflow::executeChildWorkflow` directly: ```php // Use one stub per child workflow run $childResult = yield Workflow::executeChildWorkflow( 'ChildWorkflowName', ['args'], ChildWorkflowOptions::new()->withWorkflowId('BID-SIMPLE-CHILD-WORKFLOW'), Type::TYPE_STRING // optional: defines the return type ); ``` #### How to set a Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. In PHP, a [Parent Close Policy](/parent-close-policy) is set via the `ChildWorkflowOptions` object and `withParentClosePolicy()` method. The possible values can be obtained from the [`ParentClosePolicy`](https://github.com/temporalio/sdk-php/blob/master/src/Workflow/ParentClosePolicy.php) class. - `POLICY_TERMINATE` - `POLICY_ABANDON` - `POLICY_REQUEST_CANCEL` Then `ChildWorkflowOptions` object is used to create a new Child Workflow object: ```php $child = Workflow::newUntypedChildWorkflowStub( 'child-workflow', ChildWorkflowOptions::new() ->withParentClosePolicy(ParentClosePolicy::POLICY_ABANDON) ); yield $child->start(); ``` In the snippet above we: 1. Create a new untyped Child Workflow stub with `Workflow::newUntypedChildWorkflowStub`. 2. Provide `ChildWorkflowOptions` object with Parent Close Policy set to `ParentClosePolicy::POLICY_ABANDON`. 3. Start Child Workflow Execution asynchronously using `yield` and method `start()`. We need `yield` here to ensure that a Child Workflow Execution starts before the parent closes. --- # Continue-As-New - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/continue-as-new > Use Temporal's Continue-As-New in PHP to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for PHP developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) - [How to test Continue-as-New?](#how-to-test) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the PHP SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically set to `None` for the original caller of the Workflow. [View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflowInterface.php) in the context of the rest of the application code. ```php final class ClusterManagerInput { public function __construct( public ?ClusterManagerState $state = null, public bool $testContinueAsNew = false, ) {} } #[Workflow\WorkflowInterface] interface MessageHandlerWorkflowInterface { #[Workflow\WorkflowMethod] public function run(ClusterManagerInput $input); } ```` The test hook in the above snippet is covered [below](#how-to-test). Inside your Workflow, call the [`continueAsNew()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_continueAsNew) function with the same type. This stops the Workflow right away and starts a new one. [View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflow.php) in the context of the rest of the application code. ```php Workflow::continueAsNew( Workflow::getInfo()->type->name, [new ClusterManagerInput($this->state, $input->testContinueAsNew)], ); ```` ### Considerations for Workflows with Message Handlers If you use Updates or Signals, don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish in your main Workflow before you run `continueAsNew`. See the [`allHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance. ## When is it right to Continue-as-New using the PHP SDK? Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `Workflow::getInfo()->shouldContinueAsNew` to check if it's time. ## How to test Continue-as-New using the PHP SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, when `testContinueAsNew == true`, this sample creates a test-only variable called `$this->maxHistoryLength` and sets it to a small value. A helper method in the Workflow checks it each time it considers using Continue-as-New: [View the source code](https://github.com/temporalio/samples-php/blob/master/app/src/SafeMessageHandlers/MessageHandlerWorkflow.php) in the context of the rest of the application code. ```php private function shouldContinueAsNew(): bool { if (Workflow::getInfo()->shouldContinueAsNew) { return true; } // This is just for ease-of-testing. In production, we trust temporal to tell us when to continue as new. if ($this->maxHistoryLength !== null && Workflow::getInfo()->historyLength > $this->maxHistoryLength) { return true; } return false; } ``` --- # Workflow message passing - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/message-passing > Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client. ## How to develop with Signals A [Signal](/sending-messages#sending-signals) is a message sent to a running Workflow Execution. Signals are defined in your code and handled in your Workflow Definition. Signals can be sent to Workflow Executions from a Temporal Client or from another Workflow Execution. ### How to define a Signal A Signal has a name and can have arguments. - The name, also called a Signal type, is a string. - The arguments must be [serializable](/dataconversion). Workflows can answer synchronous [Queries](/sending-messages#sending-queries) and receive [Signals](/sending-messages#sending-signals). All interface methods must have one of the following attributes: - **#[WorkflowMethod]** indicates an entry point to a Workflow. It contains parameters that specify timeouts and a Task Queue name. Required parameters (such as `executionStartToCloseTimeoutSeconds`) that are not specified through the attribute must be provided at runtime. - **#[SignalMethod]** indicates a method that reacts to external signals. It must have a `void` return type. - **#[QueryMethod]** indicates a method that reacts to synchronous query requests. It must have a non `void` return type. > It is possible (though not recommended for usability reasons) to annotate concrete class implementation. You can have more than one method with the same attribute (except #[WorkflowMethod]). For example: ```php use Temporal\Workflow\WorkflowInterface; use Temporal\Workflow\WorkflowMethod; use Temporal\Workflow\SignalMethod; use Temporal\Workflow\QueryMethod; #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] public function processFile(Argument $args); #[QueryMethod("history")] public function getHistory(): array; #[QueryMethod("status")] public function getStatus(): string; #[SignalMethod] public function retryNow(): void; #[SignalMethod] public function abandon(): void; } ``` Note that name parameter of Workflow method attributes can be used to specify name of Workflow, Signal and Query types. If name is not specified the short name of the Workflow interface is used. In the preceding code the `#[WorkflowMethod(name)]` is not specified, thus the Workflow Type defaults to `"FileProcessingWorkflow"`. ### How to handle a Signal Workflows listen for Signals by the Signal's name. Use the `#[SignalMethod]` attribute to handle Signals in the Workflow interface: ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class YourWorkflow { private bool $value; #[Workflow\WorkflowMethod] public function run() { yield Workflow::await(fn()=> $this->value); return 'OK'; } #[Workflow\SignalMethod] public function setValue(bool $value) { $this->value = $value; } } ``` In the preceding example, the Workflow updates the protected value. The main Workflow coroutine waits for the value to change by using the `Workflow::await()` function. ### How to send a Signal from a Temporal Client When a Signal is sent successfully from the Temporal Client, the [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Event History of the Workflow that receives the Signal. To send a Signal to a Workflow Execution from a Client, call the Signal method, annotated with `#[SignalMethod]` in the Workflow interface, from the Client code. To send a Signal to a Workflow, use `WorkflowClient->newWorkflowStub` or `WorkflowClient->newUntypedWorkflowStub`: ```php $workflow = $workflowClient->newWorkflowStub(YourWorkflow::class); $run = $workflowClient->start($workflow); // do something $workflow->setValue(true); assert($run->getValue() === true); ``` Use `WorkflowClient->newRunningWorkflowStub` or `WorkflowClient->newUntypedRunningWorkflowStub` with Workflow Id to send Signals to already running Workflows. ```php $workflow = $workflowClient->newRunningWorkflowStub(YourWorkflow::class, 'workflowID'); $workflow->setValue(true); ``` See [Handle Signal](#handle-signal) for details on how to handle Signals in a Workflow. ### How to send a Signal from a Workflow A Workflow can send a Signal to another Workflow, in which case it's called an _External Signal_. When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. To send Signal to a Workflow use `WorkflowClient`->`newWorkflowStub` or `WorkflowClient`->`newUntypedWorkflowStub`: ```php $workflow = $workflowClient->newWorkflowStub(YourWorkflow::class); $run = $workflowClient->start($workflow); // do something $workflow->setValue(true); assert($run->getValue() === true); ``` Use `WorkflowClient`->`newRunningWorkflowStub` or `WorkflowClient->newUntypedRunningWorkflowStub` with Workflow Id to send Signals to a running Workflow. ```php $workflow = $workflowClient->newRunningWorkflowStub(YourWorkflow::class, 'workflowID'); $workflow->setValue(true); ``` ### How to Signal-With-Start Signal-With-Start is used from the Client. It takes a Workflow Id, Workflow arguments, a Signal name, and Signal arguments. If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled. In cases where you may not know if a Workflow is running, and want to send a Signal to it, use `startwithSignal`. If a running Workflow exists, the `startwithSignal` API sends the Signal. If there is no running Workflow, the API starts a new Workflow Run and delivers the Signal to it. ```php $workflow = $workflowClient->newWorkflowStub(YourWorkflow::class); $run = $workflowClient->startWithSignal( $workflow, 'setValue', [true], // signal arguments [] // start arguments ); ``` ## How to develop with Queries A [Query](/sending-messages#sending-queries) is a synchronous operation that is used to get the state of a Workflow Execution. ### How to define a Query A Query has a name and can have arguments. - The name, also called a Query type, is a string. - The arguments must be [serializable](/dataconversion). Workflows can answer synchronous [Queries](/sending-messages#sending-queries) and receive [Signals](/sending-messages#sending-signals). All interface methods must have one of the following attributes: - **#[WorkflowMethod]** indicates an entry point to a Workflow. It contains parameters that specify timeouts and a Task Queue name. Required parameters (such as `executionStartToCloseTimeoutSeconds`) that are not specified through the attribute must be provided at runtime. - **#[SignalMethod]** indicates a method that reacts to external signals. It must have a `void` return type. - **#[QueryMethod]** indicates a method that reacts to synchronous query requests. It must have a non `void` return type. > It is possible (though not recommended for usability reasons) to annotate concrete class implementation. You can have more than one method with the same attribute (except #[WorkflowMethod]). For example: ```php use Temporal\Workflow\WorkflowInterface; use Temporal\Workflow\WorkflowMethod; use Temporal\Workflow\SignalMethod; use Temporal\Workflow\QueryMethod; #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] public function processFile(Argument $args); #[QueryMethod("history")] public function getHistory(): array; #[QueryMethod("status")] public function getStatus(): string; #[SignalMethod] public function retryNow(): void; #[SignalMethod] public function abandon(): void; } ``` Note that name parameter of Workflow method attributes can be used to specify name of Workflow, Signal and Query types. If name is not specified the short name of the Workflow interface is used. In the preceding code the `#[WorkflowMethod(name)]` is not specified, thus the Workflow Type defaults to `"FileProcessingWorkflow"`. ### How to handle a Query Queries are handled by your Workflow. Don't include any logic that causes [Command](/workflow-execution#command) generation within a Query handler (such as executing Activities). Including such logic causes unexpected behavior. You can add custom Query types to handle Queries such as Querying the current state of a Workflow, or Querying how many Activities the Workflow has completed. To do this, you need to set up a Query handler using method attribute `QueryMethod` or `Workflow::registerQuery`. ```php #[Workflow\WorkflowInterface] class YourWorkflow { #[Workflow\QueryMethod] public function getValue() { return 42; } #[Workflow\WorkflowMethod] public function run() { // workflow code } } ``` The handler function can receive any number of input parameters, but all input parameters must be serializable. The following sample code sets up a Query handler that handles the Query type of `currentState`: ```php #[Workflow\WorkflowInterface] class YourWorkflow { private string $currentState; #[Workflow\QueryMethod('current_state')] public function getCurrentState(): string { return $this->currentState; } #[Workflow\WorkflowMethod] public function run() { // Your normal Workflow code begins here, and you update the currentState // as the code makes progress. $this->currentState = 'waiting timer'; try{ yield Workflow::timer(DateInterval::createFromDateString('1 hour')); } catch (\Throwable $e) { $this->currentState = 'timer failed'; throw $e; } $yourActivity = Workflow::newActivityStub( YourActivityInterface::class, ActivityOptions::new()->withScheduleToStartTimeout(60) ); $this->currentState = 'waiting activity'; try{ yield $yourActivity->doSomething('some input'); } catch (\Throwable $e) { $this->currentState = 'activity failed'; throw $e; } $this->currentState = 'done'; return null; } } ``` You can also issue a Query from code using the `QueryWorkflow()` API on a Temporal Client object. Use `WorkflowStub` to Query Workflow instances from your Client code (can be applied to both running and closed Workflows): ```php $workflow = $workflowClient->newWorkflowStub( YourWorkflow::class, WorkflowOptions::new() ); $workflowClient->start($workflow); var_dump($workflow->getCurrentState()); sleep(60); var_dump($workflow->getCurrentState()); ``` ### How to send a Query Queries are sent from a Temporal Client. ## How to develop with Updates An [Update](/sending-messages#sending-updates) is an operation that can mutate the state of a Workflow Execution and return a response. ### Define Update **How to define an Update using the PHP SDK.** Workflow Updates handlers are methods in your Workflow Definition designed to handle updates. These updates can be triggered during the lifecycle of a Workflow Execution. An Update handler has a name, arguments, response, and an optional validator. - The name, also called an Update type, is a string. - The arguments and response must be [serializable](/dataconversion). The [`#[UpdateMethod]`](https://php.temporal.io/classes/Temporal-Workflow-UpdateMethod.html) attribute indicates that the method is used to handle and respond to update requests. ```php #[UpdateMethod] public function myUpdate(string $value); ``` ### Handle Update **How to handle Updates in a Workflow using the PHP SDK.** Workflows listen for Update by the Update's name. Use the `#[UpdateMethod]` attribute to handle Updates in the Workflow interface. The handler method can accept multiple serializable input parameters, but it's recommended using only a single parameter. The function can return a [serializable](/dataconversion) value or `void`. ```php #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] #[ReturnType(ProcessResult::class)] public function processFile(File $file); #[UpdateMethod] public function pauseProcessing(): void; } ``` Update handlers, unlike Query handlers, can change Workflow state. The Updates type defaults to the name of the method. To overwrite this default naming and assign a custom Update type, use the `#[UpdateMethod]` attribute with the `name` parameter. ```php #[WorkflowInterface] interface FileProcessingWorkflow { #[WorkflowMethod] public function processFiles(FileList $files); #[UpdateMethod(name: 'pause')] public function pauseProcessing(): void; } ``` **Register Update Handler dynamically** You can register Update handlers dynamically using the `Workflow::registerUpdate()` method. The third argument is an optional Update validator. The validator function must have the same parameters as the handler and throw an exception if the validation fails. ```php Workflow::registerUpdate( name: 'pause', handler: fn() => $this->paused = true, validator: fn() => $this->paused === false or throw new \Exception('Workflow is already paused'), ); ``` ### Validate Update **How to validate Updates in a Workflow using the PHP SDK.** Validate certain aspects of the data sent to the Workflow using an Update Validator method. For instance, a counter Workflow might never want to accept a non-positive number. Use the [`#[UpdateValidatorMethod]`](https://php.temporal.io/classes/Temporal-Workflow-UpdateValidatorMethod.html) attribute and set the `forUpdate` argument to the name of your Update handler. Your Update Validator should accept the same input parameters as your Update Handler and return `void`. ```php #[WorkflowInterface] interface GreetingWorkflow { #[WorkflowMethod] public function getGreetings(): array; #[UpdateMethod] public function addGreeting(string $name): int; #[UpdateValidatorMethod(forUpdate: 'addGreeting')] public function addGreetingValidator(string $name): void; } ``` ### Send Update from a Client **How to send an Update to a Workflow Execution from a Temporal Client using the PHP SDK.** To send an Update to a Workflow Execution from a Client, call the Update method, annotated with `#[UpdateMethod]` in the Workflow interface, from the Client code. In the following Client code example, start the Workflow `getGreetings` and call the Update method `addGreeting` that is handled in the Workflow. ```php // Create a typed Workflow stub for GreetingsWorkflow $workflow = $workflowClient->newWorkflowStub(GreetingWorkflow::class, $workflowOptions); // Start the Workflow $run = $workflowClient->start($workflow); // Send an update to the Workflow. addGreeting returns // the number of greetings our workflow has received. $count = $workflow->addGreeting("World"); ``` **Async accept** In Workflow Update methods, all Workflow features are available, such as executing Activities and child Workflows, and waiting on timers/conditions. In cases where it's known that the update will take a long time to execute, or you are not interested in the outcome of its execution, you can use the stub method [`startUpdate`](https://php.temporal.io/classes/Temporal-Client-WorkflowStubInterface.html#method_startUpdate) and move on immediately after receiving the validation result. Note that the processing Workflow Worker must be available. Otherwise, the request may block indefinitely or fail due to a timeout. ```php use Ramsey\Uuid\UuidInterface; use Temporal\Client\Update\UpdateOptions; use Temporal\Client\Update\WaitPolicy; use Temporal\Client\Update\LifecycleStage; // Create an untyped Workflow stub for GreetingsWorkflow $stub = $client->newUntypedWorkflowStub('GreetingWorkflow', $workflowOptions); // Start the Workflow $run = $client->start($stub); // Send an Update to the Workflow. UpdateHandle returns $handle = $stub->startUpdate('addGreeting', 'World'); // Use the UpdateHandle to get the Update result with timeout 2.5 seconds $result = $handle->getResult(timeout: 2.5); // You can get more control using UpdateOptions $resultUuid = $stub->startUpdate( UpdateOptions::new('storeGreetings', LifecycleStage::StageCompleted) ->withResultType(UuidInterface::class) )->getResult(); ``` #### Update-With-Start [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. You can: - Use the [`updateWithStart`](https://php.temporal.io/classes/Temporal-Client-WorkflowClientInterface.html#method_updateWithStart) WorkflowClient API. It returns once the requested Update wait stage has been reached; or when the request times out. - Use the [`UpdateHandle`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateHandle.html) to retrieve a result from the Update. You provide: - A WorkflowStub created from [`WorkflowOptions`](https://php.temporal.io/classes/Temporal-Client-WorkflowOptions.html). - The `WorkflowOptions` require a [Workflow Id Conflict Policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) to be specified. - Choose ["Use Existing"](https://php.temporal.io/classes/Temporal-Common-WorkflowIdConflictPolicy.html#enumcase_UseExisting) and use an idempotent Update handler to ensure your code can be executed again in case of a Client failure. Not all `WorkflowOptions` are allowed. For example, specifying a Cron Schedule will result in an error. - Update name or [`UpdateOptions`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateOptions.html). This mirrors the approach used for [Update Workflow](#send-update-from-client). - For Update-with-Start, the Workflow Id is optional. - When specified, the Id must match the one used in `WorkflowOptions`. - Since a running Workflow Execution may not already exist, you can't set a Run Id. For example: ```php $stub = $workflowClient->newUntypedWorkflowStub( ShoppingCartWorkflow::class, WorkflowOptions::new() ->withTaskQueue('service-queue') ->withWorkflowId($cartId) ->withWorkflowIdConflictPolicy(WorkflowIdConflictPolicy::UseExisting), ); $handle = $workflowClient->updateWithStart( workflow: $stub, update: 'addItem', updateArgs: [$itemId, $quantity], ); $price = $handle->getResult(); ``` To wait on the Update result, run the Update with the wait stage set to [`LifecycleStage::StageCompleted`](https://php.temporal.io/classes/Temporal-Client-Update-LifecycleStage.html#enumcase_StageCompleted). This returns once the update result is available; or when the API call times out. For example: ```php $handle = $workflowClient->updateWithStart( workflow: $stub, update: UpdateOptions::new('addItem', LifecycleStage::StageCompleted), updateArgs: [$itemId, $quantity], ); assert($handle->hasResult() === true); $price = $handle->getResult(); ``` ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-php/tree/master/app/src/SafeMessageHandlers) demonstrating safe `async` message handling. ### Add wait conditions to block Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. You can use a wait condition ([`Workflow::await()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_await)) to set a function that prevents the code from proceeding until the condition returns `true`. This is an important feature that helps you control your handler logic. Here are two important use cases for `Workflow::await()`: - Waiting in a handler until it is appropriate to continue. - Waiting in the main Workflow until all active handlers have finished. The condition state you're waiting for can be updated by and reflect any part of the Workflow code. This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method (see [`Workflow::async()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_async). ### Use wait conditions in handlers It's common to use a Workflow wait condition to wait until a handler should start. You can also use wait conditions anywhere else in the handler to wait for a specific condition to become `true`. This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become `true`. Consider a `readyForUpdateToExecute` method that runs before your Update handler executes. The `Workflow::await` method waits until your condition is met: ```php #[UpdateMethod] public function myUpdate(UpdateInput $input) { yield Workflow::await( fn() => $this->readyForUpdateToExecute($input), ); // ... } ``` Remember: Handlers can execute before the main Workflow method starts. ### Ensure your handlers finish before the Workflow completes Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result. The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results. Use [`Workflow::await()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_await) and [`Workflow::allHandlersFinished()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_allHandlersFinished) to address this problem and allow your Workflow to end smoothly: ```php #[WorkflowInterface] class MyWorkflow { #[WorkflowMethod] public function run() { // ... yield Workflow::await(fn() => Workflow::allHandlersFinished()); return "workflow-result"; } } ``` By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions. You can silence these warnings on a per-handler basis by passing the `unfinishedPolicy` argument to the [`UpdateMethod`](https://php.temporal.io/classes/Temporal-Workflow-UpdateMethod.html) / [`SignalMethod`](https://php.temporal.io/classes/Temporal-Workflow-SignalMethod.html) attribute: ```php #[UpdateMethod(unfinishedPolicy: HandlerUnfinishedPolicy::Abandon)] public function myUpdate() { // ... } ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. ### Use `#[WorkflowInit]` to operate on Workflow input before any handler executes Normally, your Workflows constructor won't have any parameters. However, if you use the `#[WorkflowInit]` attribute on your constructor, you can give it the same [Workflow parameters](/develop/php/workflows/basics#workflow-parameters) as your `#[WorkflowMethod]`. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/php/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `#[WorkflowMethod]` method -- that always happens, whether or not you use the `#[WorkflowInit]` attribute. This is useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/sending-messages). Here's an example. Notice that the constructor and `getGreeting` must have the same parameters: ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class GreetingExample { private readonly string $nameWithTitle; private bool $titleHasBeenChecked; // Note the attribute is on a public constructor #[Workflow\WorkflowInit] public function __construct(string $input) { $this->nameWithTitle = 'Sir ' . $input; $this->titleHasBeenChecked = false; } #[Workflow\WorkflowMethod] public function getGreeting(string $input) { yield Workflow::await(fn() => $this->titleHasBeenChecked); return "Hello " . $this->nameWithTitle; } #[Workflow\UpdateMethod] public function checkTitleValidity() { // 👉 The handler is now guaranteed to see the workflow input // after it has been processed by the constructor. $isValid = yield Workflow::executeActivity('activity.checkTitleValidity', [$this->nameWithTitle]); $this->titleHasBeenChecked = true; return $isValid; } } ``` > **📝 Note:** > > By default, the Workflow Handler runs before Signals and Updates in PHP SDK v2. This behavior is incorrect. > To avoid breaking already written Workflows, since PHP SDK v2.11.0, a [feature flag](https://php.temporal.io/classes/Temporal-Worker-FeatureFlags.html#property_workflowDeferredHandlerStart) was added to enhance the behavior of the Workflow Handler. > Make sure to set this flag to `true` to enable the correct behavior. > ### Use `Mutex` to prevent concurrent handler execution Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class MyWorkflow { // ... #[Workflow\SignalMethod] public function badAsyncHandler() { $data = yield Workflow::executeActivity( type: 'fetch_data', args: ['url' => 'http://example.com'], options: ActivityOptions::new()->withStartToCloseTimeout('10 seconds'), ); $this->x = $data->x; # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then # there may be times when the Workflow has $this->x from one Activity execution and $this->y from another. yield Workflow::timer(1); # or await anything else $this->y = $data->y; } } ``` Coordinating access using `Mutex` corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```php use Temporal\Workflow; #[Workflow\WorkflowInterface] class MyWorkflow { // ... private Workflow\Mutex $mutex; public function __construct() { $this->mutex = new Workflow\Mutex(); } #[Workflow\SignalMethod] public function safeAsyncHandler() { $data = yield Workflow::executeActivity( type: 'fetch_data', args: ['url' => 'http://example.com'], options: ActivityOptions::new()->withStartToCloseTimeout('10 seconds'), ); yield Workflow::runLocked($this->mutex, function () use ($data) { $this->x = $data->x; # ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow # method, but no other execution of this handler can run until this execution finishes. yield Workflow::timer(1); # or await anything else $this->y = $data->y; }); } ``` ## Message handler troubleshooting When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The client can't contact the server**: You'll receive a [`ServiceClientException`](https://php.temporal.io/classes/Temporal-Exception-Client-ServiceClientException.html) in case of a server connection error. [How to configure RPC Retry Policy](/develop/php/client/temporal-client#configure-rpc-retry-policy) - **RPC timeout**: You'll receive a [`TimeoutException`](https://php.temporal.io/classes/Temporal-Exception-Client-TimeoutException.html) in case of an RPC timeout. [How to configure RPC timeout](/develop/php/client/temporal-client#configure-rpc-timeout) - **The workflow does not exist**: You'll receive a [`WorkflowNotFoundException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowNotFoundException.html) exception. See [Exceptions in message handlers](/handling-messages#exceptions) for a non–PHP-specific discussion of this topic. ### Problems when sending a Signal When using Signal, the only exception that will result from your requests during its execution is `ServiceClientException`. All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle. For Queries and Updates, the client waits for a response from the Worker. If an issue occurs during the handler Execution by the Worker, the client may receive an exception. ### Problems when sending an Update When working with Updates, you may encounter these errors: - **No Workflow Workers are polling the Task Queue**: Your request will be retried by the SDK Client indefinitely. You can [configure RPC timeout](/develop/php/client/temporal-client#configure-rpc-timeout) to impose a timeout. This raises a [`WorkflowUpdateRPCTimeoutOrCanceledException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateRPCTimeoutOrCanceledException.html). - **Update failed**: You'll receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html) exception. There are two ways this can happen: - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler. - The Update failed after having been accepted. Update failures are like [Workflow failures](/references/failures#errors-in-workflows). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity (if the Activity retries have been set to a finite number) - The Workflow author raising `ApplicationFailure` - **The handler caused the Workflow Task to fail**: A [Workflow Task Failure](/references/failures#errors-in-workflows) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html). - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://php.temporal.io/classes/Temporal-Client-Update-UpdateHandle.html) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`WorkflowUpdateException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowUpdateException.html). This happens if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Problems when sending a Query When working with Queries, you may encounter these errors: - **There is no Workflow Worker polling the Task Queue**: You'll receive a [`WorkflowNotFoundException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowNotFoundException.html). - **Query failed**: You'll receive a [`WorkflowQueryException`](https://php.temporal.io/classes/Temporal-Exception-Client-WorkflowQueryException.html) if something goes wrong during a Query. Any exception in a Query handler will trigger this error. This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. ## Dynamic components Temporal supports Dynamic Queries, Signals, and Updates. These are unnamed handlers that are invoked if no other statically defined handler with the given name exists. Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, or Updates aren't known at run time. > **⚠️ Caution:** > > Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach. > Overusing them can lead to maintainability and debugging issues down the line. > > Instead, Signals, or Queries should be defined statically whenever possible, with clear names that indicate their purpose. > Use static definitions as the primary way of structuring your Workflows. > > Reserve Dynamic Handlers for cases where the handler names are not known at development time and need to be looked up dynamically at runtime. > They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic. > ### How to set a Dynamic Query A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered. Use [`Workflow::registerDynamicQuery()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicQuery) to set a dynamic Query handler. The Query Handler parameters must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments. ```php Workflow::registerDynamicQuery(function (string $name, ValuesInterface $arguments): string { return \sprintf( 'Got query `%s` with %d arguments', $name, $arguments->count(), ); }); ``` ### How to set a Dynamic Signal A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered. Use [`Workflow::registerDynamicSignal()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicSignal) to set a dynamic Signal handler. The Signal Handler parameters must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments. ```php Workflow::registerDynamicSignal(function (string $name, ValuesInterface $arguments): void { Workflow::getLogger()->info(\sprintf( 'Executed signal `%s` with %d arguments', $name, $arguments->count(), )); }); ``` ### How to set a Dynamic Update A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered. Use [`Workflow::registerDynamicUpdate()`](https://php.temporal.io/classes/Temporal-Workflow.html#method_registerDynamicUpdate) to set a dynamic Update handler. The method accepts two arguments: - Update Handler - Update Validator (optional) that should throw an exception if the validation fails Both the Handler and the Validator must accept a `string` name and [`ValuesInterface`](https://php.temporal.io/classes/Temporal-DataConverter-ValuesInterface.html) for the arguments. ```php Workflow::registerDynamicUpdate( static fn(string $name, ValuesInterface $arguments): string => \sprintf( 'Got update `%s` with %d arguments', $name, $arguments->count(), ), static fn(string $name, ValuesInterface $arguments) => \str_starts_with( $name, 'update_', ) or throw new \InvalidArgumentException('Invalid update name'), ); ``` --- # Schedules - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/schedules > Use Workflow Start Delay and Temporal Cron Jobs in PHP. Delay Workflow execution or set up recurring tasks with a Cron Schedule using Temporal Client. This page shows how to do the following: - [How to use Start Delay](#start-delay) - [How to use Temporal Cron Jobs](#temporal-cron-jobs) For recurring automation, Temporal recommends [Schedules](/schedule) instead of Cron Jobs. If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and can affect future scheduled starts through the Schedule's [Overlap Policy](/schedule#overlap-policy). ## How to use Start Delay Use the Workflow [Start Delay](/workflow-execution/timers-delays) functionality if you need to delay the execution of the Workflow without the need for regular launches. Here you simply specify the time to wait before dispatching the first Workflow task. ```php $workflow = $workflowClient->newWorkflowStub( GreeterWorkflowInterface::class, WorkflowOptions::new() ->withWorkflowStartDelay(CarbonInterval::minutes(10)), ); $workflowClient->start($workflow, 'Hello world!'); ``` ## How to use Temporal Cron Jobs > **⚠️ Caution:** > Cron support is not recommended > > We recommend using [Schedules](/schedule) instead of Cron Jobs. > Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules. > A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made. Set your Cron Schedule with `CronSchedule('* * * * *')`. Temporal Workflow Schedule Cron strings follow this format: ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) │ │ │ │ │ * * * * * ``` The following example sets a Cron Schedule in PHP: ```php $workflow = $this->workflowClient->newWorkflowStub( CronWorkflowInterface::class, WorkflowOptions::new() ->withWorkflowId(CronWorkflowInterface::WORKFLOW_ID) ->withCronSchedule('* * * * *') // Execution timeout limits total time. Cron will stop executing after this timeout. ->withWorkflowExecutionTimeout(CarbonInterval::minutes(10)) // Run timeout limits duration of a single workflow invocation. ->withWorkflowRunTimeout(CarbonInterval::minute(1)) ); $output->writeln("Starting CronWorkflow... "); try { $run = $this->workflowClient->start($workflow, 'Antony'); // ... } ``` Setting `withCronSchedule` turns the Workflow Execution into a Temporal Cron Job. For more information, see the [PHP samples](https://github.com/temporalio/samples-php/tree/master/app/src/Cron) for example code or the PHP SDK `WorkflowOptions` [source code](https://github.com/temporalio/sdk-php/blob/master/src/Client/WorkflowOptions.php). > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > --- # Side Effects - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/side-effects > Use Side Effects in PHP to execute non-deterministic code like generating UUIDs or random numbers in a Workflow without compromising its determinism. ## How to use Side Effects in PHP Side Effects are used to execute non-deterministic code, such as generating a UUID or a random number, without compromising determinism in the Workflow. This is done by storing the results of the Side Effect into the Workflow [Event History](/workflow-execution/event#event-history). A Side Effect doesn't re-execute during a Replay. Instead, it returns the recorded result from the Workflow Execution Event History. Side Effects shouldn't fail. An exception that is thrown from the Side Effect causes failure and retry of the current Workflow Task. An Activity or a Local Activity can also be used instead of a Side Effect, as its results are also persisted in Workflow Execution History. > **📝 Note:** > > You shouldn't modify the Workflow state inside a Side Effect, because they're not re-executed during Replay. Side Effect functions should only return a value, and that value can be used in Workflow code to alter state. > To use a Side Effect in PHP, use the `Workflow::sideEffect()` function in your Workflow Definition to run non-deterministic code and return a value. ```php #[Workflow\WorkflowMethod] public function run() { $random = yield Workflow::sideEffect(fn() => random_int(0, 100)); if ($random < 50) { // ... } else { // ... } } ``` --- # Workflow Timeouts - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/timeouts > Optimize Workflow Execution with Temporal PHP SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient. Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes. If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer. Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts). - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task. Create an instance of `WorkflowOptions` in the Client code and set your timeout. Available timeouts are: - `withWorkflowExecutionTimeout()` - `withWorkflowRunTimeout()` - `withWorkflowTaskTimeout()` ```php $workflow = $this->workflowClient->newWorkflowStub( DynamicSleepWorkflowInterface::class, WorkflowOptions::new() ->withWorkflowId(DynamicSleepWorkflow::WORKFLOW_ID) ->withWorkflowIdReusePolicy(WorkflowIdReusePolicy::WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE) // Set Workflow Timeout duration ->withWorkflowExecutionTimeout(CarbonInterval::minutes(2)) // ->withWorkflowRunTimeout(CarbonInterval::minute(2)) // ->withWorkflowTaskTimeout(CarbonInterval::minute(2)) ); ``` ### Workflow retries A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. A Retry Policy can be configured with an instance of the `RetryOptions` object. To enable retries for a Workflow, you need to provide a Retry Policy object via `ChildWorkflowOptions` for Child Workflows or via `WorkflowOptions` for top-level Workflows. ```php $workflow = $this->workflowClient->newWorkflowStub( CronWorkflowInterface::class, WorkflowOptions::new()->withRetryOptions( RetryOptions::new()->withInitialInterval(120) ) ); ``` --- # Timers - PHP SDK Source: https://docs.temporal.io/develop/php/workflows/timers > A Timer in a Workflow sets a durable pause for a fixed time. Even after downtimes, your Workflow resumes execution. Lightweight and scalable, millions of Timers can run on a single Worker. ## What is a Timer? A Workflow can set a durable timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`. A Workflow can sleep for months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To set a Timer in PHP, use `Workflow::timer()` and pass the number of seconds you want to wait before continuing. The following example yields a sleep method for 5 minutes. ```php yield Workflow::timer(300); // sleep for 5 minutes ``` You cannot set a Timer invocation inside the `await` or `awaitWithTimeout` methods. --- # Versioning - PHP SDK feature guide Source: https://docs.temporal.io/develop/php/workflows/versioning > Ensure deterministic Temporal Workflow execution and deploy updates with the PHP SDK's patching and Worker Versioning APIs. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#php-sdk-patching-api). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Patching with GetVersion A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Suppose you have an initial Workflow that runs `prePatchActivity`: ```php #[WorkflowInterface] class MyWorkflow { private $activity; public function __construct() { $this->activity = Workflow::newActivityStub( YourActivityInterface::class, ActivityOptions::new()->withScheduleToStartTimeout(60) ); } #[WorkflowMethod] public function runAsync() { $result = yield $this->activity->prePatchActivity(); } } ``` Suppose you replaced `prePatchActivity` with `postPatchActivity` and deployed the updated code. If an existing Workflow Execution was started by the original version of the Workflow code, where `prePatchActivity` was run, and then resumed running on a new Worker where it was replaced with `postPatchActivity`, the server side Event History would be out of sync. This would cause the Workflow to fail with a nondeterminism error. To resolve this, you can use [Workflow::getVersion](https://php.temporal.io/classes/Temporal-Workflow.html#method_getVersion) to patch to your Workflow: ```php #[WorkflowInterface] class MyWorkflow { // ... #[WorkflowMethod] public function runAsync() { $version = yield Workflow::getVersion('Step 1', Workflow::DEFAULT_VERSION, 1); $result = $version === Workflow::DEFAULT_VERSION ? yield $this->activity->prePatchActivity() : yield $this->activity->postPatchActivity(); } } ``` When `getVersion()` is run for the new Workflow Execution, it records a marker in the Event History so that all future calls to `getVersion()` for this change Id — `Step 1` in the example — on this Workflow Execution will always return the given version number, which is `1` in the example. If you make an additional change, such as adding `anotherPatchActivity()`, you need to add some additional code: ```php #[WorkflowInterface] class MyWorkflow { // ... #[WorkflowMethod] public function runAsync() { $version = yield Workflow::getVersion('Step 1', Workflow::DEFAULT_VERSION, maxSupported: 2); $result = match($version) { Workflow::DEFAULT_VERSION => yield $this->activity->prePatchActivity() 1 => yield $this->activity->postPatchActivity(); 2 => yield $this->activity->anotherPatchActivity(); }; } } ``` Note that we changed `maxSupported` from 1 to 2. A Workflow that has already passed this `getVersion()` call before it was introduced returns `DEFAULT_VERSION`. A Workflow that was run with `maxSupported` set to 1 returns 1. New Workflows return 2. After all the Workflow Executions prior to version 1 have left retention, you can remove the code for that version: ```php #[WorkflowMethod] public function runAsync() { $version = yield Workflow::getVersion('Step 1', minSupported: 1, maxSupported: 2); $result = match($version) { 1 => yield $this->activity->postPatchActivity(); 2 => yield $this->activity->anotherPatchActivity(); }; } ``` You'll note that `minSupported` has changed from `DEFAULT_VERSION` to `1`. If an older version of the Workflow Execution history is replayed on this code, it fails because the minimum expected version is 1. After all the Workflow Executions for version 1 have left retention, you can remove version 1 so that your code looks like the following: ```php #[WorkflowMethod] public function runAsync() { $version = yield Workflow::getVersion('Step 1', minSupported: 2, maxSupported: 2); $result = yield $this->activity->anotherPatchActivity(); } ``` Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `MyWorkflow` as `MyWorkflowV2`: ```php #[WorkflowInterface] class MyWorkflow {} #[WorkflowInterface] class MyWorkflowV2 {} ``` You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types. The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ## Runtime checking The Temporal PHP SDK performs a runtime check to help prevent obvious incompatible changes. Adding, removing, or reordering any of these methods without Versioning triggers the runtime check and results in a nondeterminism error: - `workflow.ExecuteActivity()` - `workflow.ExecuteChildWorkflow()` - `workflow.NewTimer()` - `workflow.RequestCancelWorkflow()` - `workflow.SideEffect()` - `workflow.SignalExternalWorkflow()` - `workflow.Sleep()` The runtime check does not perform a thorough check. For example, it does not check on the Activity's input arguments or the Timer duration. Each Temporal SDK implements these sanity checks differently, and they are not a complete check for non-deterministic changes. Instead, you should incorporate [Replay Testing](/develop/php/best-practices/testing-suite#replay) when making revisions. --- # Plugins guide Source: https://docs.temporal.io/develop/plugins-guide > Best practices for creating plugins for AI Agents # Plugins A **Plugin** is an abstraction that allows you to customize any aspect of your Temporal Worker setup, including registering Workflow and Activity definitions, modifying worker and client options, and more. Using plugins, you can build reusable open-source libraries or build add-ons for engineers at your company. This guide will teach you how to create plugins and give platform engineers general guidance on using and managing Temporal's primitives. Here are some common use cases for plugins: - AI Agent SDKs - Observability, tracing, or logging middleware - Adding reliable built-in functionality such as LLM calls, messaging systems, and payments infrastructure - Encryption or compliance middleware ## How to build a Plugin The recommended way to start building plugins is with a `SimplePlugin`. This abstraction will tackle the vast majority of plugins people want to write. ### Example Plugins If you prefer to learn by getting hands-on with code, check out some existing plugins. - Temporal's Python SDK ships with an [OpenAI Agents SDK](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents) plugin - Temporal's Python SDK ships with a [LangGraph](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/langgraph) plugin - [Temporal client and Worker plugin for Pydantic AI](https://github.com/pydantic/pydantic-ai/blob/d9b4b2540183a4426669b2824c87cdfc36144780/pydantic_ai_slim/pydantic_ai/durable_exec/temporal/__init__.py#L142) - Temporal's TypeScript SDK ships with an [OpenTelemetry Plugin](https://github.com/temporalio/sdk-typescript/blob/main/contrib/interceptors-opentelemetry/src/plugin.ts) ## What you can provide to users in a plugin There are a number of features you can give your users with a plugin. Here's a short list of some of the things you can do. - [Built-in Activities](#built-in-activity) - [Workflow-friendly libraries](#workflow-friendly-libraries) - [Built-in Workflows](#built-in-workflows) - [Built-in Nexus Operations](#built-in-nexus-operations) - [Custom Data Converters](#custom-data-converters) - [Interceptors](#interceptors) - [Context Propagators](#context-propagators) ### Built-in Activity You can provide built-in Activities in a Plugin for users to call from their Workflows. Check out the [Activities](/activities) page for more details on how they work. Refer to the [best practices for creating Activities](/activity-definition#best-practices-for-defining-activities) when you are making Activity plugins. #### Timeouts and retry policies Temporal's Activity retry mechanism gives applications the benefits of Durable Execution. See the [Activity retry policy explanation](/activity-definition#activity-retry-policy) for more details. **Python** [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py @activity.defn async def some_activity() -> None: return None plugin = SimplePlugin("organization.PluginName", activities=[some_activity]) ``` **Go** [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go) ```go func SomeActivity(ctx context.Context) error { // Activity implementation return nil } func createActivityPlugin() (*temporal.SimplePlugin, error) { return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error { options.Registry.RegisterActivityWithOptions( SomeActivity, activity.RegisterOptions{Name: "SomeActivity"}, ) return nil }, }) } ``` **Java** [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java) ```java @ActivityInterface public interface SomeActivity { @ActivityMethod void someActivity(); } public class SomeActivityImpl implements SomeActivity { @Override public void someActivity() { // Activity implementation } } SimplePlugin activityPlugin = SimplePlugin.newBuilder("organization.PluginName") .registerActivitiesImplementations(new SomeActivityImpl()) .build(); ``` **TypeScript** [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts) ```ts const activity = async () => 'activity'; const plugin = new SimplePlugin({ name: 'organization.PluginName', activities: { pluginActivity: activity, }, }); ``` **.NET** [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs) ```cs [Activity] static void SomeActivity() => throw new NotImplementedException(); SimplePlugin activityPlugin = new SimplePlugin( "organization.PluginName", new SimplePluginOptions() { }.AddActivity(SomeActivity)); ``` **Ruby** [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb) ```rb def some_activity # Activity implementation end plugin = Temporalio::SimplePlugin.new( name: 'organization.PluginName', activities: [method(:some_activity)] ) ``` ### Workflow-friendly libraries You can provide a library for use within a Workflow if you'd like to abstract away some Temporal-specific details for your users. Your library will call elements you include in your Plugin such as Activities, Child Workflows, Signals, Updates, Queries, Nexus Operations, Interceptors, Data Converters, and any other code as long as it follows these requirements: - Any code that runs in the Workflow context must be [deterministic](/workflow-definition#deterministic-constraints), meaning it produces the same commands and results when replayed. For example, don't call system time APIs, generate random values, or perform direct network and file I/O from Workflow-context code; move that work to Activities or Nexus Operations. - See [observability](/evaluate/development-production-features/observability) to avoid duplicating observation side effects when Workflows replay. - Put other side effects inside of Activities or [Local Activities](/local-activity). This helps your Workflow handle being restarted, resumed, or executed in a different process from where it originally began without losing correctness or state consistency. - See [testing your Plugin](#testing-your-plugin) to write tests that check for issues with side effects. - It should run quickly since it may be replayed many times during a long Workflow execution. More expensive code should go in Activities or Nexus Operations. A Plugin should allow a user to decompose their Workflows into Activities, as well as Child Workflows and Nexus Calls when needed. This gives users granular control through retries and timeouts, debuggability through the Temporal UI, operability with resets, pauses, and cancels, memoization for efficiency and resumability, and scalability using task queues and Workers. Users use Workflows for: - Orchestration and decision-making - Interactivity via [message-passing](/evaluate/development-production-features/workflow-message-passing) - Tracing and observability #### Making changes to your library Your users may want to keep their Workflows running across deployments of their Worker code. If their deployment includes a new version of your Plugin, changes to your Plugin could break Workflow code that started before the new version was deployed. This can be due to [non-deterministic behavior from code changes](/workflow-definition#non-deterministic-change) in your Plugin. See [testing](#testing-your-plugin) to see how to test for this. And, if you make substantive changes, you need to use [patching](/patching). #### Example of a Workflow library that uses a Plugin in Python - [Implementation of the `OpenAIAgentsPlugin`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents) - [Example of replay testing](https://github.com/temporalio/sdk-python/blob/main/tests/contrib/openai_agents/test_openai_replay.py) ### Built-in Workflows You can provide a built-in Workflow in a `SimplePlugin`. It’s callable as a Child Workflow or standalone. When you want to provide a piece of functionality that's more complex than an Activity, you can: - Use a [Workflow Library](#workflow-friendly-libraries) that runs directly in the end user’s Workflow - Add a Child Workflow Consider adding a Child Workflow when one or more of these conditions applies: - That child should outlive the parent. - The Workflow Event History would otherwise [not scale](/workflow-execution/event#event-history-limits) in parent Workflows. - When you want a separate Workflow ID for the child so that it can be operated independently of the parent's state (canceled, terminated, paused). Any Workflow can be run as a standalone Workflow or as a Child Workflow, so registering a Child Workflow in a `SimplePlugin` is the same as registering any Workflow. **Python** [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py @workflow.defn class HelloWorkflow: @workflow.run async def run(self, name: str) -> str: return f"Hello, {name}!" plugin = SimplePlugin("organization.PluginName", workflows=[HelloWorkflow]) ``` **Go** [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go) ```go func HelloWorkflow(ctx workflow.Context, name string) (string, error) { return "Hello, " + name + "!", nil } func createWorkflowPlugin() (*temporal.SimplePlugin, error) { return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error { options.Registry.RegisterWorkflowWithOptions( HelloWorkflow, workflow.RegisterOptions{Name: "HelloWorkflow"}, ) return nil }, }) } ``` **Java** [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java) ```java @WorkflowInterface public interface HelloWorkflow { @WorkflowMethod String run(String name); } public static class HelloWorkflowImpl implements HelloWorkflow { @Override public String run(String name) { return "Hello, " + name + "!"; } } SimplePlugin workflowPlugin = SimplePlugin.newBuilder("organization.PluginName") .registerWorkflowImplementationTypes(HelloWorkflowImpl.class) .build(); ``` **.NET** [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs) ```cs [Workflow] class SimpleWorkflow { [WorkflowRun] public Task RunAsync(string name) => Task.FromResult($"Hello, {name}!"); } SimplePlugin workflowPlugin = new SimplePlugin( "organization.PluginName", new SimplePluginOptions() { }.AddWorkflow()); ``` **Ruby** [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb) ```rb class HelloWorkflow < Temporalio::Workflow::Definition def execute(name) "Hello, #{name}!" end end plugin = Temporalio::SimplePlugin.new( name: 'organization.PluginName', workflows: [HelloWorkflow] ) ``` ### Built-in Nexus Operations Nexus calls are used from Workflows similar to Activities and you can learn more about [Temporal Nexus](/nexus). Like Activities, Nexus operation inputs and return values must be serializable. **Python** [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py @nexusrpc.service class WeatherService: get_weather_nexus_operation: nexusrpc.Operation[WeatherInput, Weather] @nexusrpc.handler.service_handler(service=WeatherService) class WeatherServiceHandler: @nexusrpc.handler.sync_operation async def get_weather_nexus_operation( self, ctx: nexusrpc.handler.StartOperationContext, input: WeatherInput ) -> Weather: return Weather( city=input.city, temperature_range="14-20C", conditions="Sunny with wind.", ) plugin = SimplePlugin( "organization.PluginName", nexus_service_handlers=[WeatherServiceHandler()] ) ``` **Go** [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go) ```go type WeatherInput struct { City string `json:"city"` } type Weather struct { City string `json:"city"` TemperatureRange string `json:"temperatureRange"` Conditions string `json:"conditions"` } var WeatherService = nexus.NewService("weather-service") var GetWeatherOperation = nexus.NewSyncOperation( "get-weather", func(ctx context.Context, input WeatherInput, options nexus.StartOperationOptions) (Weather, error) { return Weather{ City: input.City, TemperatureRange: "14-20C", Conditions: "Sunny with wind.", }, nil }, ) func createNexusPlugin() (*temporal.SimplePlugin, error) { return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", RunContextBefore: func(ctx context.Context, options temporal.SimplePluginRunContextBeforeOptions) error { options.Registry.RegisterNexusService(WeatherService) return nil }, }) } ``` **Java** [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java) ```java // Example Nexus service implementation public class WeatherService { public Weather getWeather(WeatherInput input) { return new Weather(input.getCity(), "14-20C", "Sunny with wind."); } } public static class Weather { private final String city; private final String temperatureRange; private final String conditions; public Weather(String city, String temperatureRange, String conditions) { this.city = city; this.temperatureRange = temperatureRange; this.conditions = conditions; } // Getters... } public static class WeatherInput { private final String city; public WeatherInput(String city) { this.city = city; } public String getCity() { return city; } } SimplePlugin nexusPlugin = SimplePlugin.newBuilder("organization.PluginName") .registerNexusServiceImplementation(new WeatherService()) .build(); ``` **TypeScript** [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts) ```ts const testServiceHandler = nexus.serviceHandler( nexus.service('testService', { testSyncOp: nexus.operation(), }), { async testSyncOp(_, input) { return input; }, }, ); const plugin = new SimplePlugin({ name: 'organization.PluginName', nexusServices: [testServiceHandler], }); ``` **.NET** [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs) ```cs [NexusService] public interface IStringService { [NexusOperation] string DoSomething(string name); } [NexusServiceHandler(typeof(IStringService))] public class HandlerFactoryStringService { private readonly Func> handlerFactory; public HandlerFactoryStringService(Func> handlerFactory) => this.handlerFactory = handlerFactory; [NexusOperationHandler] public IOperationHandler DoSomething() => handlerFactory(); } SimplePlugin nexusPlugin = new SimplePlugin( "organization.PluginName", new SimplePluginOptions() { }.AddNexusService(new HandlerFactoryStringService(() => OperationHandler.Sync((ctx, name) => $"Hello, {name}"))) ); ``` ### Custom Data Converters A [custom Data Converter](/default-custom-data-converters#custom-data-converter) can alter data formats or provide compression or encryption. Note that you can use an existing Data Converter such as, in Python, `PydanticPayloadConverter` in your Plugin. **Python** [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py def set_converter(converter: DataConverter | None) -> DataConverter: if converter is None or converter == DataConverter.default: return pydantic_data_converter # Should consider interactions with other plugins, # as this will override the data converter. # This may mean failing, warning, or something else return converter plugin = SimplePlugin("organization.PluginName", data_converter=set_converter) ``` **Go** [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go) ```go func createConverterPlugin() (*temporal.SimplePlugin, error) { customConverter := converter.GetDefaultDataConverter() // Or your custom converter return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", DataConverter: customConverter, }) } ``` **Java** [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java) ```java SimplePlugin converterPlugin = SimplePlugin.newBuilder("organization.PluginName") .customizeDataConverter( existingConverter -> { // Customize the data converter // This example keeps the existing converter unchanged // In practice, you might wrap it with additional functionality return existingConverter; }) .build(); ``` **TypeScript** [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts) ```ts const codec: PayloadCodec = { encode(_payloads: Payload[]): Promise { throw new Error(); }, decode(_payloads: Payload[]): Promise { throw new Error(); }, }; const plugin = new SimplePlugin({ name: 'organization.PluginName', dataConverter: (converter: DataConverter | undefined) => ({ ...converter, payloadCodecs: [...(converter?.payloadCodecs ?? []), codec], }), }); ``` **.NET** [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs) ```cs private class Codec : IPayloadCodec { public Task> EncodeAsync(IReadOnlyCollection payloads) => throw new NotImplementedException(); public Task> DecodeAsync(IReadOnlyCollection payloads) => throw new NotImplementedException(); } SimplePlugin converterPlugin = new SimplePlugin( "organization.PluginName", new SimplePluginOptions() { DataConverterOption = new SimplePluginOptions.SimplePluginOption( (converter) => converter with { PayloadCodec = new Codec() } ), }); ``` **Ruby** [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb) ```rb custom_converter = Temporalio::Converters::DataConverter.new( payload_converter: Temporalio::Converters::PayloadConverter.default ) plugin = Temporalio::SimplePlugin.new( name: 'organization.PluginName', data_converter: custom_converter ) ``` ### Interceptors Interceptors are middleware that can run before and after various calls such as Activities, Workflows, and Signals. You can [learn more about interceptors](/develop/python/workers/interceptors) for the details of implementing them. They're used to: - Create side effects such as logging and tracing. - Modify arguments, such as adding headers for authorization or tracing propagation. **Python** [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py class SomeWorkerInterceptor(temporalio.worker.Interceptor): pass # Your implementation class SomeClientInterceptor(temporalio.client.Interceptor): pass # Your implementation plugin = SimplePlugin( "organization.PluginName", interceptors=[SomeWorkerInterceptor(), SomeClientInterceptor()], ) ``` **Go** [features/snippets/plugins/plugins.go](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.go) ```go type SomeWorkerInterceptor struct { interceptor.WorkerInterceptorBase } type SomeClientInterceptor struct { interceptor.ClientInterceptorBase } func createInterceptorPlugin() (*temporal.SimplePlugin, error) { return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", WorkerInterceptors: []interceptor.WorkerInterceptor{&SomeWorkerInterceptor{}}, ClientInterceptors: []interceptor.ClientInterceptor{&SomeClientInterceptor{}}, }) } ``` **Java** [features/snippets/plugins/plugins.java](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.java) ```java public class SomeWorkerInterceptor extends WorkerInterceptorBase { // Your worker interceptor implementation } public class SomeClientInterceptor extends WorkflowClientInterceptorBase { // Your client interceptor implementation } SimplePlugin interceptorPlugin = SimplePlugin.newBuilder("organization.PluginName") .addWorkerInterceptors(new SomeWorkerInterceptor()) .addClientInterceptors(new SomeClientInterceptor()) .build(); ``` **TypeScript** [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts) ```ts class MyWorkflowClientInterceptor implements WorkflowClientInterceptor {} class MyActivityInboundInterceptor implements ActivityInboundCallsInterceptor {} class MyActivityOutboundInterceptor implements ActivityOutboundCallsInterceptor {} const workflowInterceptorsPath = ''; const plugin = new SimplePlugin({ name: 'organization.PluginName', clientInterceptors: { workflow: [new MyWorkflowClientInterceptor()], }, workerInterceptors: { client: { workflow: [new MyWorkflowClientInterceptor()], }, workflowModules: [workflowInterceptorsPath], activity: [ (_: Context) => ({ inbound: new MyActivityInboundInterceptor(), outbound: new MyActivityOutboundInterceptor(), }), ], }, }); ``` **.NET** [features/snippets/plugins/plugins.cs](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.cs) ```cs private class SomeClientInterceptor : IClientInterceptor { public ClientOutboundInterceptor InterceptClient( ClientOutboundInterceptor nextInterceptor) => throw new NotImplementedException(); } private class SomeWorkerInterceptor : IWorkerInterceptor { public WorkflowInboundInterceptor InterceptWorkflow( WorkflowInboundInterceptor nextInterceptor) => throw new NotImplementedException(); public ActivityInboundInterceptor InterceptActivity( ActivityInboundInterceptor nextInterceptor) => throw new NotImplementedException(); } SimplePlugin interceptorPlugin = new SimplePlugin( "organization.PluginName", new SimplePluginOptions() { ClientInterceptors = new List() { new SomeClientInterceptor() }, WorkerInterceptors = new List() { new SomeWorkerInterceptor() }, }); ``` **Ruby** [features/snippets/plugins/plugins.rb](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.rb) ```rb class SomeWorkerInterceptor include Temporalio::Worker::Interceptor::Workflow def intercept_workflow(next_interceptor) # Your interceptor implementation next_interceptor end end class SomeClientInterceptor include Temporalio::Client::Interceptor def intercept_client(next_interceptor) # Your interceptor implementation next_interceptor end end plugin = Temporalio::SimplePlugin.new( name: 'organization.PluginName', client_interceptors: [SomeClientInterceptor.new], worker_interceptors: [SomeWorkerInterceptor.new] ) ``` ### Context Propagators Context propagators pass custom key-value data (such as tracing IDs, tenant IDs, or auth tokens) across Workflow, Activity, and Child Workflow boundaries via Temporal headers. See [Context Propagation](/encyclopedia/context-propagation) for details on how they work. Propagators registered via a Plugin are appended to any propagators already set by the user or by previous plugins. **Go** ```go func createContextPropagatorPlugin() (*temporal.SimplePlugin, error) { return temporal.NewSimplePlugin(temporal.SimplePluginOptions{ Name: "organization.PluginName", ContextPropagators: []workflow.ContextPropagator{NewMyPropagator()}, }) } ``` ### Special considerations for different languages Each of the SDKs has nuances you should be aware of so you can account for it in your code. #### Python You can choose to [run your Workflows in a sandbox in Python](/develop/python/best-practices/python-sdk-sandbox). This lets you run Workflow code in a sandbox environment to help prevent non-determinism errors in your application. To work for users who use sandboxing, your Plugin should specify the Workflow runner that it uses. [features/snippets/plugins/plugins.py](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.py) ```py def workflow_runner(runner: WorkflowRunner | None) -> WorkflowRunner: if not runner: raise ValueError("No WorkflowRunner provided to the plugin.") # If in sandbox, add additional passthrough if isinstance(runner, SandboxedWorkflowRunner): return dataclasses.replace( runner, restrictions=runner.restrictions.with_passthrough_modules("module"), ) return runner plugin = SimplePlugin("organization.PluginName", workflow_runner=workflow_runner) ``` #### TypeScript TypeScript bundles cannot provide built-in Workflows because the TypeScript SDK bundles all Workflow code from a single module. Plugin users must import and re-export any Plugin-provided Workflows from their own Workflow module so the bundle includes them. Users of a plugin which provides Workflow interceptors should always provide the plugin to the bundler if bundling. If you aren't aware of the exact function of the plugin, you can always provide it, as it won't have any adverse effects. [features/snippets/plugins/plugins.ts](https://github.com/temporalio/features/blob/main/features/snippets/plugins/plugins.ts) ```ts const bundle = await bundleWorkflowCode({ workflowsPath: require.resolve('./workflows'), plugins: [plugin], }); const worker = await Worker.create({ connection, taskQueue: 'my-task-queue', workflowBundle: bundle, plugins: [plugin], }); ``` ## Testing your Plugin To test your Plugin, you'll write a normal Temporal Workflow tests, having included the plugin in your Client. Two special concerns are versioning tests, for when you're making changes to your plugin, and testing unwanted side effects. ### Versioning tests When you make changes to your plugin after it has already shipped to users, we recommend that you set up [replay testing](/develop/python/best-practices/testing-suite#replay) on each important change to make sure that you’re not causing non-determinism errors for your users. ### Side effects tests Your Plugin should cater to Workflows resuming in different processes than the ones they started on and then replaying from the beginning, which can happen, for example, after an intermittent failure. You can ensure you're not depending on local side effects by turning Workflow caching off, which will mean that the Workflow replays from the top each time it progresses: **Python** [features/snippets/worker/worker.py](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.py) ```py worker = Worker(client, task_queue="task-queue", max_cached_workflows=0) ``` **Go** [features/snippets/worker/worker.go](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.go) ```go worker.SetStickyWorkflowCacheSize(0) w := worker.New(c, "task-queue", worker.Options{}) ``` **Java** [features/snippets/worker/worker.java](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.java) ```java WorkerFactory factory = WorkerFactory.newInstance( client, WorkerFactoryOptions.newBuilder().setWorkflowCacheSize(0).build()); Worker worker = factory.newWorker("task-queue"); ``` **TypeScript** [features/snippets/worker/worker.ts](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.ts) ```ts const worker = await Worker.create({ connection, taskQueue: 'task-queue', maxCachedWorkflows: 0, }); ``` **.NET** [features/snippets/worker/worker.cs](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.cs) ```cs using var worker = new TemporalWorker( client, new TemporalWorkerOptions("task-queue") { MaxCachedWorkflows = 0 }); ``` **Ruby** [features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb) ```rb worker = Temporalio::Worker.new( client: client, task_queue: 'task-queue', max_cached_workflows: 0 ) ``` Inside this replay regime, you should test for duplicate side effects or other types of failures. When testing for duplicate side effects, it may not be sufficient to simply have a counter that increments once per effect, as activities may be retried. Instead, consider these patterns: - count `ActivityTaskScheduled` events of the expected activity type in event history (one per intended call, independent of retries), - or accumulate activity IDs in a concurrency-safe set and assert on its size (different scheduled activities get different IDs; retries of the same scheduled activity share one). It's harder to test against side effects to global variables, so this practice is best avoided entirely. --- # Python SDK developer guide Source: https://docs.temporal.io/develop/python > Explore Temporal Python SDK feature guides to master developing Temporal Applications. Build Workflows, Activities, and Workers, connect to Temporal Services, set up a testing suite, handle failure detection, send messages, complete Activities asynchronously, implement Versioning, use Observability, debug applications, schedule Workflows ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Install and get started You can find detailed installation instructions for the Python SDK in the [Quickstart](/develop/python/set-up-your-local-python). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Activity basics](/develop/python/activities/basics) - [Workflow basics](/develop/python/workflows/basics) - [Start an Activity execution](/develop/python/activities/execution) - [Run Worker processes](/develop/python/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/python/workflows) - [Workflow basics](/develop/python/workflows/basics) - [Child Workflows](/develop/python/workflows/child-workflows) - [Continue-As-New](/develop/python/workflows/continue-as-new) - [Cancellation](/develop/python/workflows/cancellation) - [Timeouts](/develop/python/workflows/timeouts) - [Message passing](/develop/python/workflows/message-passing) - [Schedules](/develop/python/workflows/schedules) - [Timers](/develop/python/workflows/timers) - [Versioning](/develop/python/workflows/versioning) - [Workflow Streams](/develop/python/workflows/workflow-streams) ## [Activities](/develop/python/activities) - [Activity basics](/develop/python/activities/basics) - [Activity execution](/develop/python/activities/execution) - [Standalone Activities](/develop/python/activities/standalone-activities-quickstart) - [Timeouts](/develop/python/activities/timeouts) - [Asynchronous Activity completion](/develop/python/activities/asynchronous-activity) - [Benign exceptions](/develop/python/activities/benign-exceptions) ## [Workers](/develop/python/workers) - [Worker processes](/develop/python/workers/run-worker-process) ## [Temporal Client](/develop/python/client) - [Temporal Client](/develop/python/client/temporal-client) ## [Temporal Nexus](/develop/python/nexus) - [Quickstart](/develop/python/nexus/quickstart) - [Feature guide](/develop/python/nexus/feature-guide) - [Standalone Operations](/develop/python/nexus/standalone-operations) ## [Platform](/develop/python/platform) - [Observability](/develop/python/platform/observability) - [Enriching the UI](/develop/python/platform/enriching-ui) ## [Best practices](/develop/python/best-practices) - [Testing](/develop/python/best-practices/testing-suite) - [Python SDK sandbox](/develop/python/best-practices/python-sdk-sandbox) - [Debugging](/develop/python/best-practices/debugging) - [Data handling](/develop/python/data-handling) - [Sync vs async](/develop/python/best-practices/python-sdk-sync-vs-async) ## [Integrations](/develop/python/integrations) - [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) - [Deep Agents integration](/develop/python/integrations/deepagents) - [Google ADK integration](/develop/python/integrations/google-adk) - [Google GenAI integration](/develop/python/integrations/google-genai) - [Langfuse integration](https://langfuse.com/integrations/frameworks/temporal) - [LangGraph integration](/develop/python/integrations/langgraph) - [LangSmith integration](/develop/python/integrations/langsmith) - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) - [OpenBox integration](https://docs.openbox.ai/getting-started/temporal) - [Parseable integration](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) - [Pydantic AI integration](https://ai.pydantic.dev/durable_execution/temporal/) - [Strands Agents integration](/develop/python/integrations/strands-agents) - [Tenuo integration](https://tenuo.ai/temporal) ## Temporal Python technical resources - [Python SDK Quickstart - Setup Guide](/develop/python/set-up-your-local-python) - [Python API Documentation](https://python.temporal.io) - [Python SDK Code Samples](https://github.com/temporalio/samples-python) - [Python SDK GitHub](https://github.com/temporalio/sdk-python) - [Temporal 101 in Python Free Course](https://learn.temporal.io/courses/temporal_101/python/) ## Get connected with the Temporal Python community - [Temporal Python Community Slack](https://app.slack.com/client/TNWA8QCGZ) - [Python SDK Forum](https://community.temporal.io/tag/python-sdk) --- # Activities - Python SDK Source: https://docs.temporal.io/develop/python/activities > This section explains how to implement Activities with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Activities - [Activity basics](/develop/python/activities/basics) - [Activity execution](/develop/python/activities/execution) - [Standalone Activities Quickstart](/develop/python/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/python/activities/standalone-activities) - [Timeouts](/develop/python/activities/timeouts) - [Asynchronous Activity completion](/develop/python/activities/asynchronous-activity) - [Benign exceptions](/develop/python/activities/benign-exceptions) --- # Asynchronous Activity completion - Python SDK Source: https://docs.temporal.io/develop/python/activities/asynchronous-activity > Asynchronously complete an Activity using the Temporal Python SDK. Follow three steps for Activity completion and use the Temporal Client for Heartbeat and updates. **How to Asynchronously complete an Activity using the Temporal Python SDK.** [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To mark an Activity as completing asynchronously, do the following inside the Activity. ```python # Capture token for later completion captured_token = activity.info().task_token activity.raise_complete_async() ``` To update an Activity outside the Activity, use the [get_async_activity_handle()](https://python.temporal.io/temporalio.client.Client.html#get_async_activity_handle) method to get the handle of the Activity. ```python handle = my_client.get_async_activity_handle(task_token=captured_token) ``` Then, on that handle, you can call the results of the Activity, `heartbeat`, `complete`, `fail`, or `report_cancellation` method to update the Activity. ```python await handle.complete("Completion value.") ``` --- # Activity basics - Python SDK Source: https://docs.temporal.io/develop/python/activities/basics > This section explains Activity Basics with the Python SDK ## Develop a basic Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/python/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/python/activities/standalone-activities-quickstart). You can develop an Activity Definition by using the `@activity.defn` decorator. Register the function as an Activity with a custom name through a decorator argument, for example `@activity.defn(name="your_activity")`. > **📝 Note:** > > The Temporal Python SDK supports multiple ways of implementing an Activity: > > - Asynchronously using [`asyncio`](https://docs.python.org/3/library/asyncio.html) > - Synchronously multithreaded using > [`concurrent.futures.ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) > - Synchronously multiprocess using > [`concurrent.futures.ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) > and > [`multiprocessing.managers.SyncManager`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager) > > Blocking the async event loop in Python would turn your asynchronous program into a synchronous program that executes > serially, defeating the entire purpose of using `asyncio`. This can also lead to potential deadlock, and unpredictable > behavior that causes tasks to be unable to execute. Debugging these issues can be difficult and time consuming, as > locating the source of the blocking call might not always be immediately obvious. > > Due to this, consider not making blocking calls from within an asynchronous Activity, or use an async safe library to > perform these actions. If you must use a blocking library, consider using a synchronous Activity instead. > ```python from temporalio import activity from your_dataobject import YourParams @activity.defn(name="your_activity") async def your_activity(input: YourParams) -> str: return f"{input.greeting}, {input.name}!" ``` ### Develop Activity Parameters There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. Activity parameters are the function parameters of the function decorated with `@activity.defn`. These can be any data type Temporal can convert, including dataclasses when properly type-annotated. Technically this can be multiple parameters, but Temporal strongly encourages a single dataclass parameter containing all input fields. ```python {2,6} from temporalio import activity from your_dataobject import YourParams @activity.defn(name="your_activity") async def your_activity(input: YourParams) -> str: return f"{input.greeting}, {input.name}!" ``` ### Define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). An Activity Execution can return inputs and other Activity values. The following example defines an Activity that takes an object as input and returns a string. ```python {6,7} from temporalio import activity from your_dataobject import YourParams @activity.defn(name="your_activity") async def your_activity(input: YourParams) -> str: return f"{input.greeting}, {input.name}!" ``` ### Customize your Activity Type Activities have a Type that are referred to as the Activity name. The following examples demonstrate how to set a custom name for your Activity Type. You can customize the Activity name with a custom name in the decorator argument. For example, `@activity.defn(name="your-activity")`. If the name parameter is not specified, the Activity name defaults to the function name. ```python {5} from temporalio import activity from your_dataobject import YourParams @activity.defn(name="your_activity") async def your_activity(input: YourParams) -> str: return f"{input.greeting}, {input.name}!" ``` --- # Benign exceptions - Python SDK Source: https://docs.temporal.io/develop/python/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. **How to mark an Activity error as benign using the Temporal Python SDK** When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the `category` parameter to `ApplicationErrorCategory.BENIGN` when raising an [`ApplicationError`](https://python.temporal.io/temporalio.exceptions.ApplicationError.html). Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```python from temporalio import activity from temporalio.exceptions import ApplicationError, ApplicationErrorCategory @activity.defn async def my_activity() -> str: try: return await call_external_service() except Exception as err: raise ApplicationError( message=str(err), # Mark this error as benign since it's expected category=ApplicationErrorCategory.BENIGN, ) ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Activity execution - Python SDK Source: https://docs.temporal.io/develop/python/activities/execution > Shows how to perform Activity execution with the Python SDK ## Start an Activity Execution **How to start an Activity Execution using the Temporal Python SDK.** Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTask[Closed])in your Workflow Execution Event History. A single instance of the Activities implementation is shared across multiple simultaneous Activity invocations. Activity implementation code should be _idempotent_. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. To spawn an Activity Execution, use the [`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) operation from within your Workflow Definition. `execute_activity()` is a shortcut for [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) that waits on its result. To get just the handle to wait and cancel separately, use `start_activity()`. In most cases, use `execute_activity()` unless advanced task capabilities are needed. A single argument to the Activity is positional. Multiple arguments are not supported in the type-safe form of `start_activity()` or `execute_activity()` and must be supplied by the `args` keyword argument. ```python from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from your_activities_dacx import your_activity from your_dataobject_dacx import YourParams @workflow.defn(name="YourWorkflow") class YourWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( your_activity, YourParams("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` ### Set the required Activity Timeouts **How to set the required Activity Timeouts using the Temporal Python SDK.** Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set in the Activity Options. Activity options are set as keyword arguments after the Activity arguments. Available timeouts are: - schedule_to_close_timeout - schedule_to_start_timeout - start_to_close_timeout ```python {8-15} with workflow.unsafe.imports_passed_through(): from activities import your_activity, YourParams @workflow.defn class YourWorkflow: @workflow.run async def run(self, greeting: str) -> list[str]: activity_timeout_result = await workflow.execute_activity( your_activity, YourParams(greeting, "Activity Timeout option"), # Activity Execution Timeout start_to_close_timeout=timedelta(seconds=10), # schedule_to_start_timeout=timedelta(seconds=10), # schedule_to_close_timeout=timedelta(seconds=10), ) # ... ``` ### Get the results of an Activity Execution **How to get the results of an Activity Execution using the Temporal Python SDK.** The call to spawn an [Activity Execution](/activity-execution) generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable. Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available. Use [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) to start an Activity and return its handle, [`ActivityHandle`](https://python.temporal.io/temporalio.workflow.ActivityHandle.html). Use [`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) to return the results. You must provide either `schedule_to_close_timeout` or `start_to_close_timeout`. `execute_activity()` is a shortcut for `await start_activity()`. An asynchronous `execute_activity()` helper is provided which takes the same arguments as `start_activity()` and `await`s on the result. `execute_activity()` should be used in most cases unless advanced task capabilities are needed. ```python from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from your_activities_dacx import your_activity from your_dataobject_dacx import YourParams @workflow.defn(name="YourWorkflow") class YourWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( your_activity, YourParams("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/python/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal Python SDK. > **Public Preview** Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/python/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/python/activities/standalone-activities-quickstart). > This page covers the following: - [Start a Standalone Activity without waiting for the result](#start-activity) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [Wait for the result of a Standalone Activity](#get-activity-result) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the [hello_standalone_activity](https://github.com/temporalio/samples-python/tree/main/hello_standalone_activity) sample. > ## Start a Standalone Activity without waiting for the result Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue your Activity job, without waiting for it to be executed by your Worker. Use [`client.start_activity()`](https://python.temporal.io/temporalio.client.Client.html#start_activity) to start your Standalone Activity and get a handle: ```python activity_handle = await client.start_activity( compose_greeting, args=[ComposeGreetingInput("Hello", "World")], id="my-standalone-activity-id", task_queue="my-standalone-activity-task-queue", start_to_close_timeout=timedelta(seconds=10), ) ``` With the Temporal Server and Worker running, open a new terminal in the `samples-python` directory and run: ```bash uv run hello_standalone_activity/start_activity.py ``` Or use the Temporal CLI: ```bash temporal activity start \ --type compose_greeting \ --activity-id my-standalone-activity-id \ --task-queue my-standalone-activity-task-queue \ --start-to-close-timeout 10s \ --input '{"greeting": "Hello", "name": "World"}' ``` ## Get a handle to an existing Standalone Activity You can also use `client.get_activity_handle()` to create a handle to a previously started Standalone Activity: ```python activity_handle = client.get_activity_handle( activity_id="my-standalone-activity-id", run_id="the-run-id", ) ``` You can now use the handle to wait for the result, describe, cancel, or terminate the Activity. ## Wait for the result of a Standalone Activity Under the hood, calling `client.execute_activity()` is the same as calling [`client.start_activity()`](https://python.temporal.io/temporalio.client.Client.html#start_activity) to durably enqueue the Standalone Activity, and then calling `await activity_handle.result()` to wait for the activity to be executed and fetch the result: ```python activity_result = await activity_handle.result() ``` Or use the Temporal CLI to wait for a result by Activity ID: ```bash temporal activity result --activity-id my-standalone-activity-id ``` ## List Standalone Activities Use [`client.list_activities()`](https://python.temporal.io/temporalio.client.Client.html#list_activities) to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an async iterator that yields ActivityExecution entries. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. [hello_standalone_activity/list_activities.py](https://github.com/temporalio/samples-python/blob/main/hello_standalone_activity/list_activities.py) ```python import asyncio from temporalio.client import Client from temporalio.envconfig import ClientConfig async def my_application(): connect_config = ClientConfig.load_client_connect_config() connect_config.setdefault("target_host", "localhost:7233") client = await Client.connect(**connect_config) activities = client.list_activities( query="TaskQueue = 'my-standalone-activity-task-queue'", ) async for info in activities: print( f"ActivityID: {info.activity_id}, Type: {info.activity_type}, Status: {info.status}" ) if __name__ == "__main__": asyncio.run(my_application()) ``` Run it: ```bash uv run hello_standalone_activity/list_activities.py ``` Or use the Temporal CLI: ```bash temporal activity list ``` The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow Visibility](/visibility). For example, "ActivityType = 'MyActivity' AND Status = 'Running'". ## Count Standalone Activities Use [`client.count_activities()`](https://python.temporal.io/temporalio.client.Client.html#count_activities) to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) - not the number of queued tasks. It works the same way as counting Workflow Executions. [hello_standalone_activity/count_activities.py](https://github.com/temporalio/samples-python/blob/main/hello_standalone_activity/count_activities.py) ```python import asyncio from temporalio.client import Client from temporalio.envconfig import ClientConfig async def my_application(): connect_config = ClientConfig.load_client_connect_config() connect_config.setdefault("target_host", "localhost:7233") client = await Client.connect(**connect_config) resp = await client.count_activities( query="TaskQueue = 'my-standalone-activity-task-queue'", ) print("Total activities:", resp.count) for group in resp.groups: print(f"Group {group.group_values}: {group.count}") if __name__ == "__main__": asyncio.run(my_application()) ``` Run it: ```bash uv run hello_standalone_activity/count_activities.py ``` Or use the Temporal CLI: ```bash temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The code samples on this page use `ClientConfig.load_client_connect_config()`, so the same code works against Temporal Cloud - just configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/python/activities/standalone-activities-quickstart). --- # Standalone Activities Python Quickstart Source: https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal Python SDK without writing a Workflow. # Quickstart Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/python/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **📝 Note:** > > This documentation uses source code from the [hello_standalone_activity](https://github.com/temporalio/samples-python/tree/main/hello_standalone_activity) sample. > ## Get started with Standalone Activities Prerequisites: - **Python 3.9+** - **[uv](https://docs.astral.sh/uv/)** - Python package manager. Install with Homebrew, or see the [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/) for other platforms. - **Temporal Python SDK** (v1.23.0 or higher) - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server will now be available for client connections on `localhost:7233`, and the Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install uv ``` ```bash uv add temporalio ``` ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-python](https://github.com/temporalio/samples-python) repository to follow along. ```bash git clone https://github.com/temporalio/samples-python.git cd samples-python ``` The sample project is structured as follows: ``` hello_standalone_activity/ ├── my_activity.py ├── worker.py ├── execute_activity.py ├── start_activity.py ├── list_activities.py └── count_activities.py ``` ## Write an Activity function An Activity in the Temporal Python SDK is just a normal function with the `@activity.defn` decorator. It can optionally be an `async def`. The way you write a Standalone Activity is identical to how you write an Activity to be orchestrated by a Workflow. In fact, an Activity can be executed both as a Standalone Activity and as a Workflow Activity. Create [hello_standalone_activity/my_activity.py](https://github.com/temporalio/samples-python/blob/main/hello_standalone_activity/my_activity.py). ```python # my_activity.py from dataclasses import dataclass from temporalio import activity @dataclass class ComposeGreetingInput: greeting: str name: str @activity.defn def compose_greeting(input: ComposeGreetingInput) -> str: activity.logger.info("Running activity with parameter %s" % input) return f"{input.greeting}, {input.name}!" ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities — you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to run a Worker](/develop/python/workers/run-worker-process#run-a-dev-worker) for more details on Worker setup and configuration options. Create [hello_standalone_activity/worker.py](https://github.com/temporalio/samples-python/blob/main/hello_standalone_activity/worker.py). Open a new terminal, navigate to the `samples-python` directory, and run the Worker. Leave this terminal running - the Worker needs to stay up to process activities. ```python import asyncio from concurrent.futures import ThreadPoolExecutor from temporalio.client import Client from temporalio.envconfig import ClientConfig from temporalio.worker import Worker from hello_standalone_activity.my_activity import compose_greeting async def main(): connect_config = ClientConfig.load_client_connect_config() connect_config.setdefault("target_host", "localhost:7233") client = await Client.connect(**connect_config) worker = Worker( client, task_queue="my-standalone-activity-task-queue", activities=[compose_greeting], activity_executor=ThreadPoolExecutor(5), ) print("worker running...", end="", flush=True) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ```bash uv run hello_standalone_activity/worker.py ``` ## Execute a Standalone Activity Use [`client.execute_activity()`](https://python.temporal.io/temporalio.client.Client.html#execute_activity) to execute a Standalone Activity. Call this from your application code, not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to be executed on your Worker, and then fetches the result. Create [hello_standalone_activity/execute_activity.py](https://github.com/temporalio/samples-python/blob/main/hello_standalone_activity/execute_activity.py). To run it: 1. Make sure the Temporal Server is running (from the Get Started step above). 2. Make sure the Worker is running (from the Run a Worker step above). 3. Open a new terminal, navigate to the `samples-python` directory, and run `uv run hello_standalone_activity/execute_activity.py`. Or use the Temporal CLI. You should see: ``` Activity result: Hello, World! ``` ```python import asyncio from datetime import timedelta from temporalio.client import Client from temporalio.envconfig import ClientConfig from hello_standalone_activity.my_activity import ComposeGreetingInput, compose_greeting async def my_application(): connect_config = ClientConfig.load_client_connect_config() connect_config.setdefault("target_host", "localhost:7233") client = await Client.connect(**connect_config) activity_result = await client.execute_activity( compose_greeting, args=[ComposeGreetingInput("Hello", "World")], id="my-standalone-activity-id", task_queue="my-standalone-activity-task-queue", start_to_close_timeout=timedelta(seconds=10), ) print(f"Activity result: {activity_result}") if __name__ == "__main__": asyncio.run(my_application()) ``` ```bash uv run hello_standalone_activity/execute_activity.py ``` ```bash temporal activity execute \\ --type compose_greeting \\ --activity-id my-standalone-activity-id \\ --task-queue my-standalone-activity-task-queue \\ --start-to-close-timeout 10s \\ --input '{"greeting": "Hello", "name": "World"}' ``` ## Run with Temporal Cloud All code samples on this page use [`ClientConfig.load_client_connect_config()`](https://python.temporal.io/temporalio.envconfig.ClientConfig.html) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/python/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/python/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/python/activities/basics)**: How to write and register Activities with the Python SDK. --- # Activity Timeouts - Python SDK Source: https://docs.temporal.io/develop/python/activities/timeouts > Optimize Workflow Execution with Temporal Python SDK - Set Activity Timeouts and Retry Policies efficiently. ## Set Activity timeouts Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution. The following timeouts are available in the Activity Options. - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. Activity options are set as keyword arguments after the Activity arguments. Available timeouts are: - `schedule_to_close_timeout` - `schedule_to_start_timeout` - `start_to_close_timeout` ```python {13-20} from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy with workflow.unsafe.imports_passed_through(): from activities import your_activity, YourParams @workflow.defn class YourWorkflow: @workflow.run async def run(self, greeting: str) -> list[str]: activity_timeout_result = await workflow.execute_activity( your_activity, YourParams(greeting, "Activity Timeout option"), # Activity Execution Timeout start_to_close_timeout=timedelta(seconds=10), # schedule_to_start_timeout=timedelta(seconds=10), # schedule_to_close_timeout=timedelta(seconds=10), ) return activity_timeout_result ``` ### Set an Activity Retry Policy A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To create an Activity Retry Policy in Python, set the [RetryPolicy](https://python.temporal.io/temporalio.common.RetryPolicy.html) class within the [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) or [`execute_activity()`](https://python.temporal.io/temporalio.workflow.html#execute_activity) function. ```python {12-24} from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy with workflow.unsafe.imports_passed_through(): from activities import your_activity, YourParams @workflow.defn class YourWorkflow: @workflow.run async def run(self, greeting: str) -> list[str]: activity_result = await workflow.execute_activity( your_activity, YourParams(greeting, "Retry Policy options"), start_to_close_timeout=timedelta(seconds=10), # Retry Policy retry_policy=RetryPolicy( backoff_coefficient=2.0, maximum_attempts=5, initial_interval=timedelta(seconds=1), maximum_interval=timedelta(seconds=2), # non_retryable_error_types=["ValueError"], ), ) return activity_result ``` ### Override the retry interval with `next_retry_delay` To override the next retry interval set by the current policy, pass `next_retry_delay` when raising an [ApplicationError](/references/failures#application-failure) in an Activity. This value replaces and overrides whatever the retry interval would normally be on the retry policy. For example, you can set the delay interval based on an Activity's attempt count. In the following example, the retry delay starts at 3 seconds after the first attempt. It increases to 6 seconds for the second attempt, 9 seconds for the third attempt, and so forth. This creates a steadily increasing backoff, versus the exponential approach used by [backoff coefficients](/encyclopedia/retry-policies#backoff-coefficient): ```python from temporalio.exceptions import ApplicationError from datetime import timedelta @activity.defn async def my_activity(input: MyActivityInput): try: # Your activity logic goes here except Exception as e: attempt = activity.info().attempt raise ApplicationError( f"Error encountered on attempt {attempt}", next_retry_delay=timedelta(seconds=3 * attempt), ) from e ``` ## Heartbeat an Activity An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered failed and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. Heartbeats can contain a `details` field describing the Activity's current progress. If an Activity gets retried, the Activity can access the `details` from the last Heartbeat that was sent to the Temporal Service. To Heartbeat an Activity Execution in Python, use the [`heartbeat()`](https://python.temporal.io/temporalio.activity.html#heartbeat) API. ```python @activity.defn async def your_activity_definition() -> str: activity.heartbeat("heartbeat details!") ``` In addition to obtaining cancellation information, Heartbeats also support detail data that persists on the server for retrieval during Activity retry. If an Activity calls `heartbeat(123, 456)` and then fails and is retried, `heartbeat_details` returns an iterable containing `123` and `456` on the next Run. #### Set a Heartbeat Timeout A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). [`heartbeat_timeout`](https://python.temporal.io/temporalio.worker.StartActivityInput.html#heartbeat_timeout) is a class variable for the [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) function used to set the maximum time between Activity Heartbeats. ```python workflow.start_activity( activity="your-activity", schedule_to_close_timeout=timedelta(seconds=5), heartbeat_timeout=timedelta(seconds=1), ) ``` `execute_activity()` is a shortcut for [`start_activity()`](https://python.temporal.io/temporalio.workflow.html#start_activity) that waits on its result. To get just the handle to wait and cancel separately, use `start_activity()`. `execute_activity()` should be used in most cases unless advanced task capabilities are needed. ```python workflow.execute_activity( activity="your-activity", name, schedule_to_close_timeout=timedelta(seconds=5), heartbeat_timeout=timedelta(seconds=1), ) ``` --- # Best practices - Python SDK Source: https://docs.temporal.io/develop/python/best-practices > This section explains how to implement best practices with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Best practices - [Error handling](/develop/python/best-practices/error-handling) - [Testing](/develop/python/best-practices/testing-suite) - [Python SDK sandbox](/develop/python/best-practices/python-sdk-sandbox) - [Debugging](/develop/python/best-practices/debugging) - [Data handling](/develop/python/data-handling) - [Sync vs async](/develop/python/best-practices/python-sdk-sync-vs-async) --- # Debugging - Python SDK Source: https://docs.temporal.io/develop/python/best-practices/debugging > Debug Workflows in development and production environments using the Temporal Python SDK, Web UI, Temporal CLI, replay, tracing, logging, and performance metrics. This page shows how to do the following: - [Debug in a development environment](#debug-in-a-development-environment) - [Debug in a production environment](#debug-in-a-production-environment) ### Debug in a development environment **How to debug in a development environment using the Temporal Python SDK.** When developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow. In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). ### How to debug in a production environment **How to debug in a production environment using the Temporal Python SDK.** You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/python/best-practices/testing-suite#replay) - [Tracing](/develop/python/platform/observability#tracing) - [Logging](/develop/python/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). For more information, see [Observability ▶️ Metrics](/develop/python/platform/observability#metrics) for setting up SDK metrics. Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). --- # Error handling - Python SDK Source: https://docs.temporal.io/develop/python/best-practices/error-handling > Learn how to handle errors in Temporal Python applications with retry policies, idempotent Activities, and recovery patterns. Temporal automatically handles many types of failures through retries and Durable Execution. This page shows you how to build on these capabilities to create robust error handling for your applications. **Key concepts:** Not all failures should be handled the same way. **Transient failures** (like brief network hiccups) resolve on their own and should be retried immediately. **Intermittent failures** (like rate limiting) need increasing delays between retries. **Permanent failures** (like invalid input) won't resolve through retries and need different data or code changes. Temporal distinguishes between **Workflow Task failures** (bugs that can be fixed with redeployment) and **Workflow Execution failures** (business logic failures that should stop the Workflow). Task failures retry automatically so you can fix and redeploy without losing state. Execution failures require you to explicitly raise an `ApplicationError`. This page shows how to: - [Make Activities idempotent](#make-activities-idempotent) - [Raise exceptions from Activities](#raise-exceptions-from-activities) - [Raise exceptions from Workflows](#raise-exceptions-from-workflows) - [Handle exceptions in Workflows](#handle-exceptions-in-workflows) - [Configure custom Retry Policies](#configure-custom-retry-policies) - [Mark specific errors as non-retryable](#mark-errors-as-non-retryable) - [Specify non-retryable error types in Retry Policies](#specify-non-retryable-error-types) - [Implement rollback logic with the Saga pattern](#implement-saga-pattern) - [Understand Temporal's failure types](#understand-failure-types) ## Make Activities idempotent **How to make Activities idempotent using the Temporal Python SDK** Because Activities may be retried due to failures, it's strongly recommended to make them idempotent. An idempotent operation produces the same result whether executed once or multiple times. Activities follow an at-least-once execution model. If a Worker executes an Activity successfully but crashes before notifying the Temporal Service, the Activity will be retried. Without idempotence, this could cause duplicate charges in payment processing or create duplicate resources in infrastructure provisioning. ### Use idempotency keys Most external services support idempotency keys—unique identifiers that prevent duplicate operations. When the service receives a request with a key it has already processed, it returns the original result instead of performing the operation again. Create an idempotency key by combining the Workflow Run ID and Activity ID: ```python from temporalio import activity @activity.defn async def process_payment(amount: float, account: str): info = activity.info() idempotency_key = f"{info.workflow_run_id}-{info.activity_id}" # Pass idempotency_key to your payment service result = await payment_service.charge( amount=amount, account=account, idempotency_key=idempotency_key ) return result ``` This value remains constant across Activity retries but is unique among all Workflow Executions. ### Design Activities to be atomic Activities are atomic—they either complete successfully or not. If an Activity performs multiple steps and the last step fails, the entire Activity is retried. Consider this Activity: 1. Look up data in database 2. Call microservice with the data 3. Write result to filesystem If step 3 fails, all three steps execute again on retry. You might split this into three separate Activities so only the failed step retries, but balance this against having a larger Event History with more Activity Executions. ## Raise exceptions from Activities **How to raise exceptions from Activities using the Temporal Python SDK** Use `ApplicationError` to communicate application-specific failures from Activities. Raising it explicitly gives you more control. > **📝 Note:** > > *Any* other exceptions that are raised from your Python code in a Temporal Activity will be converted to an `ApplicationError` internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries. > ```python from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def validate_charge(credit_card_number: str, amount: float): if not is_valid_card(credit_card_number): raise ApplicationError( f"Invalid credit card number: {credit_card_number}", type="InvalidCreditCard", ) if amount <= 0: raise ApplicationError( f"Amount must be positive, got {amount}", type="InvalidAmount", ) return True ``` When raising an `ApplicationError`: - Provide a descriptive `message` - Optionally provide a `type` string to categorize the failure - The error appears in the Event History as an `ActivityTaskFailed` event When an Activity fails, Temporal wraps the exception in an `ActivityError` before surfacing it to the Workflow. The `ActivityError` provides context including: - Activity type that failed - Number of retry attempts - Original cause (the [`ApplicationError`](https://python.temporal.io/temporalio.exceptions.ApplicationError.html) you raised, or `TimeoutError`, [`CancelledError`](https://python.temporal.io/temporalio.exceptions.CancelledError.html), etc.) ## Raise exceptions from Workflows **How to raise exceptions from Workflows using the Temporal Python SDK** The behavior depends on what exception you raise: ### Fail a Workflow Execution To deliberately fail a Workflow Execution, raise an `ApplicationError`: ```python from temporalio import workflow from temporalio.exceptions import ApplicationError @workflow.defn class PizzaDeliveryWorkflow: @workflow.run async def run(self, order): distance = await workflow.execute_activity( calculate_distance, order.address, start_to_close_timeout=timedelta(seconds=10) ) if order.is_delivery and distance.kilometers > 25: workflow.logger.error("Customer outside service area") raise ApplicationError( "Customer lives outside the service area", type="CustomerOutsideServiceArea" ) # Continue with order... ``` One reason to use the Temporal `ApplicationError` class is because it allows you to set an additional `non_retryable` parameter. This way, you can decide whether an error should not be retried automatically by Temporal. This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse. You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/python/activities/timeouts#activity-retries). This puts the Workflow Execution in "Failed" state with no automatic retries. Use this for permanent failures where retrying won't help—like the customer being too far away. ### Trigger a Workflow Task retry Raising any other Python exception (like `ValueError` or `TypeError`) causes a Workflow Task failure, which retries automatically: ```python # This causes a Workflow Task failure (retries automatically) raise ValueError("Unexpected condition") ``` This is intentional. Regular Python exceptions are treated as bugs that can be fixed with a code deployment, not business logic failures. The Workflow Task retries indefinitely, letting you fix the bug and redeploy without losing Workflow state. ## Handle exceptions in Workflows **How to handle exceptions in Workflows using the Temporal Python SDK** Use Python's `try/except` blocks to handle Activity failures in your Workflow: ```python from temporalio import workflow from temporalio.exceptions import ActivityError, ApplicationError from datetime import timedelta @workflow.defn class MoneyTransferWorkflow: @workflow.run async def run(self, details): # Withdraw money try: withdraw_result = await workflow.execute_activity( withdraw, details, start_to_close_timeout=timedelta(seconds=10) ) except ActivityError as e: raise ApplicationError( f"Withdrawal failed: {e.cause}", type="WithdrawalError" ) # Deposit money try: deposit_result = await workflow.execute_activity( deposit, details, start_to_close_timeout=timedelta(seconds=10) ) except ActivityError as e: # Deposit failed - attempt refund try: await workflow.execute_activity( refund, withdraw_result, start_to_close_timeout=timedelta(seconds=10) ) raise ApplicationError( f"Deposit failed but money refunded to source account", type="DepositError" ) except ActivityError as refund_err: raise ApplicationError( f"Deposit failed and refund also failed: {refund_err.cause}", type="CriticalTransferError" ) return f"Transfer complete: {withdraw_result}, {deposit_result}" ``` Common Temporal exceptions you can catch in Workflows: - `ActivityError` - Activity failed after exhausting retries - `ChildWorkflowError` - Child Workflow failed - `CancelledError` - Workflow, Activity, or Timer was canceled - `TimeoutError` - Operation exceeded timeout If these exceptions propagate unhandled, the Workflow Execution fails (or enters "Canceled" state for `CancelledError`). ## Configure custom Retry Policies **How to configure custom Retry Policies using the Temporal Python SDK** Activities have a default Retry Policy with unlimited attempts and exponential backoff. Customize this to match your expected failure patterns. ```python from temporalio import workflow from temporalio.common import RetryPolicy from datetime import timedelta @workflow.defn class OrderWorkflow: @workflow.run async def run(self, order): # Custom retry for rate-limited service retry_policy = RetryPolicy( initial_interval=timedelta(seconds=10), backoff_coefficient=3.0, maximum_interval=timedelta(minutes=5), maximum_attempts=20, ) result = await workflow.execute_activity( call_external_service, order, start_to_close_timeout=timedelta(seconds=30), retry_policy=retry_policy, ) return result ``` Retry Policy attributes: - **`initial_interval`**: Delay before first retry (default: 1 second) - **`backoff_coefficient`**: Multiplier for subsequent delays (default: 2.0) - **`maximum_interval`**: Cap on retry delay (default: 100× initial interval) - **`maximum_attempts`**: Maximum retry attempts (default: unlimited) - **`non_retryable_error_types`**: Error types that shouldn't retry (default: empty) ### Match your Retry Policy to failure types **For transient failures** (brief network issues): Use the defaults or a low `initial_interval` and `backoff_coefficient`. **For intermittent failures** (rate limiting): Increase `initial_interval` and `backoff_coefficient` to space out retries and let the condition resolve. **For cost-sensitive APIs**: Set `maximum_attempts` to limit retries (rare—usually prefer timeouts). ### Use different policies for different Activities You can use different Retry Policies for different Activities, or even multiple policies for the same Activity: ```python fast_retry = RetryPolicy( initial_interval=timedelta(seconds=1), backoff_coefficient=1.5, ) slow_retry = RetryPolicy( initial_interval=timedelta(seconds=30), backoff_coefficient=3.0, ) # Same Activity, different policies await workflow.execute_activity( process_order, order, start_to_close_timeout=timedelta(seconds=10), retry_policy=fast_retry, ) # Later, with different circumstances... await workflow.execute_activity( process_order, order, start_to_close_timeout=timedelta(seconds=10), retry_policy=slow_retry, ) ``` ### Don't use Workflow Retry Policies Unlike Activities, Workflows don't retry by default, and you usually shouldn't add a Retry Policy. Workflows are deterministic and not designed for failure-prone operations. A Workflow failure typically indicates a code bug or bad input data—retrying the entire Workflow repeats the same logic without fixing the underlying issue. If you need retry logic for specific Workflow operations, implement it in your Workflow code rather than using a Workflow Retry Policy. ## Mark specific errors as non-retryable **How to mark specific errors as non-retryable using the Temporal Python SDK** Some failures are permanent and won't resolve through retries. Mark these as non-retryable to fail fast instead of waiting for timeouts. Set the `non_retryable` flag when raising an `ApplicationError`: ```python from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def process_payment(card_number: str, amount: float): if not is_valid_card_format(card_number): # Invalid format will never become valid through retries raise ApplicationError( f"Invalid credit card format: {card_number}", type="InvalidCardFormat", non_retryable=True, ) if amount <= 0: # Invalid amount won't be fixed by retrying raise ApplicationError( f"Amount must be positive: {amount}", type="InvalidAmount", non_retryable=True, ) # Process payment... ``` An `ApplicationError` with `non_retryable=True` will never retry, regardless of the Retry Policy. Use non-retryable errors for: - Invalid input data that prevents the Activity from proceeding - Business rule violations - Authorization failures **Use this sparingly.** In most cases, it's better to let the Retry Policy handle when to stop retrying based on time or attempts. ## Specify non-retryable error types **How to specify non-retryable error types in Retry Policies using the Temporal Python SDK** Sometimes you want the Workflow (caller) to decide which error types shouldn't retry, rather than the Activity (implementer). List error types that shouldn't retry in your Retry Policy: ```python from temporalio import workflow from temporalio.common import RetryPolicy from datetime import timedelta @workflow.defn class CheckoutWorkflow: @workflow.run async def run(self, payment_details): retry_policy = RetryPolicy( non_retryable_error_types=[ "InvalidCardFormat", "InsufficientFunds", "AccountClosed", ] ) try: result = await workflow.execute_activity( process_payment, payment_details, start_to_close_timeout=timedelta(seconds=30), retry_policy=retry_policy, ) return result except ActivityError as e: workflow.logger.error(f"Payment failed: {e.cause}") # Handle the non-retryable error... ``` When an Activity raises an `ApplicationError`, Temporal checks if its `type` is in `non_retryable_error_types`. If it matches, the Activity fails immediately without retries. ### When to use each approach **`non_retryable=True` in the Activity**: Use when the Activity implementer knows the error is permanently unrecoverable. This enforces the constraint for all callers. **`non_retryable_error_types` in the Retry Policy**: Use when the caller wants to decide which errors are unrecoverable based on their business logic. This lets different Workflows make different decisions about the same Activity. ## Implement rollback logic with the Saga pattern **How to implement the Saga pattern using the Temporal Python SDK** The [Saga pattern](/design-patterns/saga-pattern) coordinates a sequence of operations where each operation has a compensating action to undo its effects. If any operation fails, execute compensating actions in reverse order to roll back previous operations. Use this for multi-step processes like: - E-commerce checkout (payment, inventory, shipping) - Distributed transactions across services - Multi-stage data updates ```python from temporalio import workflow from temporalio.exceptions import ActivityError from datetime import timedelta @workflow.defn class OrderWorkflow: @workflow.run async def run(self, order): compensations = [] try: # Reserve inventory compensations.append({ "activity": revert_inventory, "input": order }) await workflow.execute_activity( reserve_inventory, order, start_to_close_timeout=timedelta(seconds=10), ) # Charge payment compensations.append({ "activity": refund_payment, "input": order }) payment_id = await workflow.execute_activity( charge_payment, order, start_to_close_timeout=timedelta(seconds=10), ) # Create shipment compensations.append({ "activity": cancel_shipment, "input": payment_id }) shipment_id = await workflow.execute_activity( create_shipment, order, start_to_close_timeout=timedelta(seconds=10), ) return {"payment_id": payment_id, "shipment_id": shipment_id} except ActivityError as e: workflow.logger.error(f"Order failed: {e.cause}, rolling back...") # Execute compensations in reverse order for compensation in reversed(compensations): try: await workflow.execute_activity( compensation["activity"], compensation["input"], start_to_close_timeout=timedelta(seconds=10), ) except ActivityError as comp_err: # Log compensation failure but continue with others workflow.logger.error(f"Compensation failed: {comp_err.cause}") # Re-raise the original error raise ApplicationError( f"Order failed: {e.cause}", type="OrderFailed" ) ``` Key points: - Add compensating actions to a list **before** executing each Activity - Use `reversed(compensations)` to undo operations in the correct order - Handle compensation failures gracefully (they might fail too) - Temporal manages all state and retry logic, making Saga implementation straightforward ## Understand Temporal's failure types Temporal uses specialized exception types to represent different failure scenarios. All exceptions inherit from [`TemporalError`](https://python.temporal.io/temporalio.exceptions.TemporalError.html). **Do not extend `TemporalError` or its children.** Use the provided exception types to ensure: - Consistent behavior across process and language boundaries - Compatibility with the Temporal Service - Proper serialization via Protocol Buffers ### Common failure types **`ApplicationError`**: Raised by your code to indicate application-specific failures. This is the only Temporal exception you should raise manually. When you raise an `ApplicationError`, you can optionally provide a `type` string and mark it as `non_retryable`. **`ActivityError`**: Wraps exceptions raised from Activities. The `cause` field contains the original error (`ApplicationError`, `TimeoutError`, `CancelledError`, etc.). Catch this in Workflows to handle Activity failures. **`TimeoutError`**: Occurs when an Activity or Workflow exceeds its configured timeout. **`CancelledError`**: Results from cancellation of a Workflow, Activity, or Timer. You can catch and ignore this to continue execution despite cancellation. **`TerminatedError`**: Occurs when a Workflow Execution is forcefully terminated. **`ChildWorkflowError`**: Raised when a Child Workflow Execution fails. **`WorkflowAlreadyStartedError`**: Raised when attempting to start a Workflow with an ID that's already running. **`ServerError`**: Used for exceptions from the Temporal Service itself (like database failures). ### Workflow Task vs Workflow Execution failures **Workflow Task failures** occur when Workflow code raises a non-Temporal exception (like `ValueError`, `TypeError`, or non-determinism errors). These retry automatically, letting you fix bugs and redeploy without losing Workflow state. **Workflow Execution failures** occur when Workflow code raises a Temporal exception like `ApplicationError`. These put the Workflow in "Failed" state with no automatic retries. Example of a permanent failure that should fail the Workflow: ```python if distance.kilometers > MAX_DELIVERY_DISTANCE: # Retrying won't change the distance - this is permanent raise ApplicationError( "Customer lives outside service area", type="OutsideServiceArea" ) ``` ### Protecting sensitive information The default Failure Converter copies exception messages and stack traces as plain text visible in the Web UI. If your exceptions might contain sensitive information, configure a custom Failure Converter to encrypt this data. See the [Securing Application Data course](https://learn.temporal.io/courses/appdatasec/) for details. --- # Temporal Python SDK sandbox environment Source: https://docs.temporal.io/develop/python/best-practices/python-sdk-sandbox > The Temporal Python SDK offers a sandbox environment to run Workflow code, aiming to prevent non-determinism errors in applications by isolating global state and applying restrictions. The Temporal Python SDK enables you to run Workflow code in a sandbox environment to help prevent non-determinism errors in your application. The Temporal Workflow Sandbox for Python is not completely isolated, and some libraries can internally mutate state, which can result in breaking determinism. ## Benefits Temporal's Python SDK uses a sandbox environment for Workflow runs to make developing Workflow code safer. If a Workflow Execution performs a non-deterministic event, an exception is thrown, which results in failing the Task Worker. The Workflow will not progress until the code is fixed. The Temporal Python sandbox offers a mechanism to _pass through modules_ from outside the sandbox. By default, this includes all standard library modules and Temporal modules. For performance and behavior reasons, users should pass through all models, Activities, Nexus services, or other modules that are in separate files whose calls will be deterministic. For more information, see [Passthrough modules](#passthrough-modules). ## How it works The Sandbox environment consists of two main components. - [Global state isolation](#global-state-isolation) - [Restrictions](#restrictions) ### Global state isolation The first component of the Sandbox is a global state isolation. Global state isolation uses `exec` to compile and evaluate statements. Upon the start of a Workflow, the file in which the Workflow is defined is imported into a newly created sandbox. If a module is imported by the file, a known set, which includes all of Python's standard library, is _passed through_ from outside the sandbox. These modules are expected to be free of side effects and have their non-deterministic aspects restricted. For a full list of modules imported, see [Customize the Sandbox](#customize-the-sandbox). ### Restrictions Restrictions prevent known non-deterministic library calls. This is achieved by using proxy objects on modules wrapped around the custom importer set in the sandbox. Restrictions apply at both the Workflow import level and the Workflow run time. A default set of restrictions that prevents most dangerous standard library calls. ## Skip Workflow Sandboxing The following techniques aren't recommended, but they allow you to avoid, skip, or break through the sandbox environment. Skipping Workflow Sandboxing results in a lack of determinism checks. Using the Workflow Sandboxing environment helps prevent non-determinism errors but doesn't completely negate the risk. ### Skip Sandboxing for a block of code To skip a sandbox environment for a specific block of code in a Workflow, use [`sandbox_unrestricted()`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_unrestricted). The Workflow will run without sandbox restrictions. ```python with temporalio.workflow.unsafe.sandbox_unrestricted(): # Your code ``` ### Skip Sandboxing for an entire Workflow To skip a sandbox environment for a Workflow, set the `sandboxed` argument in the [`@workflow.defn`](https://python.temporal.io/temporalio.workflow.html#defn) decorator to false. The entire Workflow will run without sandbox restrictions. ```python @workflow.defn(sandboxed=False) ``` ### Skip Sandboxing for a Worker To skip a sandbox environment for a Worker, set the `workflow_runner` keyword argument of the `Worker` init to [`UnsandboxedWorkflowRunner()`](https://python.temporal.io/temporalio.worker.UnsandboxedWorkflowRunner.html). ## Customize the sandbox When creating the Worker, the `workflow_runner` defaults to [`SandboxedWorkflowRunner()`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxedWorkflowRunner.html). The `SandboxedWorkflowRunner` init accepts a `restrictions` keyword argument that defines a set of restrictions to apply to this sandbox. The [`SandboxRestrictions`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html) dataclass is immutable and contains four fields that can be customized, but only three have notable values. - [`passthrough_modules`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#passthrough_modules) - [`invalid_modules_members`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#invalid_module_members) - [`import_notification_policy`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#import_notificaton_policy) ### Passthrough modules By default, the sandbox completely reloads non-standard-library and non-Temporal modules for every Workflow run. Passing through a module means that the module will not be reloaded every time the Workflow runs. Instead, the module will be imported from outside the sandbox and used directly in the Workflow. This can improve performance because importing a module can be a time-consuming process, and passing through a module can avoid this overhead. > **📝 Note:** > It is important to note that you should only import _known-side-effect-free_ third-party modules: meaning they don't have any unintended consequences when imported and used multiple times. This is because passing through a module means that it will be used multiple times in a Workflow without being reloaded, so any side effects it has won't be repeated. For this reason, it's recommended to only pass through modules that are known to be deterministic, meaning they will always produce the same output given the same input. One way to pass through a module is at import time in the Workflow file using the [`imports_passed_through`](https://python.temporal.io/temporalio.workflow.unsafe.html#imports_passed_through) context manager. ```python # my_workflow_file.py from temporalio import workflow with workflow.unsafe.imports_passed_through(): import pydantic @workflow.defn class MyWorkflow: # ... ``` Alternatively, this can be done at Worker creation time by customizing the runner's restrictions. ```python # my_worker_file.py from temporalio.worker import Worker from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions my_worker = Worker( ..., workflow_runner=SandboxedWorkflowRunner( restrictions=SandboxRestrictions.default.with_passthrough_modules("pydantic") ) ) ``` In both of these cases, now the `pydantic` module will be passed through from outside the sandbox instead of being reloaded for every Workflow run. ### Invalid module members `invalid_module_members` includes modules that cannot be accessed. Checks are compared against the fully qualified path to the item. For example, to remove a restriction on `datetime.date.today()`, see the following example. ```python # my_worker_file.py import dataclasses from temporalio.worker import Worker from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions my_restrictions = dataclasses.replace( SandboxRestrictions.default, invalid_module_members=SandboxRestrictions.invalid_module_members_default.with_child_unrestricted( "datetime", "date", "today", ), ) my_worker = Worker(..., workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions)) ``` Restrictions can also be added by piping (`|`) together [`SandboxMatcher`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxMatcher.html) instances. The following example restricts the `datetime.date` class from being used. ```python # my_worker_file.py import dataclasses from temporalio.worker import Worker from temporalio.worker.workflow_sandbox import ( SandboxedWorkflowRunner, SandboxMatcher, SandboxRestrictions, ) my_restrictions = dataclasses.replace( SandboxRestrictions.default, invalid_module_members=SandboxRestrictions.invalid_module_members_default | SandboxMatcher( children={"datetime": SandboxMatcher(use={"date"})}, ), ) my_worker = Worker(..., workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions)) ``` ### Import Notification Policy The sandbox's import notification policy specifies how the sandbox behaves when it imports modules in a way that may be unintentional. It covers two common scenarios: dynamic imports and enforcing module passthrough. Each can be controlled independently. A dynamic import occurs when a module is imported after the Workflow is loaded into the sandbox. These imports are often invisible and, if they don't do anything restricted by the sandbox, cause memory overhead. By default the [`WARN_ON_DYNAMIC_IMPORT`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#WARN_ON_DYNAMIC_IMPORT) policy setting is enabled and a warning will be emitted when a module that is not in the [passthrough modules](#passthrough-modules) list is dynamically imported. The other notable policy settings apply when a module is imported into the sandbox that was not passed through. These settings are disabled by default and must be explicitly turned on. The [`WARN_ON_UNINTENTIONAL_PASSTHROUGH`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#WARN_ON_UNINTENTIONAL_PASSTHROUGH) setting emits a warning when a module not included in the [passthrough modules](#passthrough-modules) list is imported. Similarly, the [`RAISE_ON_UNINTENTIONAL_PASSTHROUGH`](https://python.temporal.io/temporalio.workflow.SandboxImportNotificationPolicy.html#RAISE_ON_UNINTENTIONAL_PASSTHROUGH) setting will raise an error when a non-passed-through module is imported. The import notification policy can be set for specific imports by using [`sandbox_import_notification_policy`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_import_notification_policy) context manager. ```python # my_workflow_file.py from temporalio import workflow with workflow.unsafe.sandbox_import_notification_policy( workflow.SandboxImportNotificationPolicy.SILENT ): import pydantic @workflow.defn class MyWorkflow: # ... ``` This can also be done at worker creation time by customizing the runner's restrictions. ```python # my_worker_file.py from temporalio.worker import Worker from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions my_worker = Worker( ..., workflow_runner=SandboxedWorkflowRunner( restrictions=SandboxRestrictions.default.with_import_notification_policy( workflow.SandboxImportNotificationPolicy.WARN_ON_DYNAMIC_IMPORT | workflow.SandboxImportNotificationPolicy.WARN_ON_UNINTENTIONAL_PASSTHROUGH ) ) ) ``` The [`sandbox_import_notification_policy`](https://python.temporal.io/temporalio.workflow.unsafe.html#sandbox_import_notification_policy) context manager will always be respected if used in combination with the restrictions customization. For more information on the Python sandbox, see the following resources. - [Python SDK README](https://github.com/temporalio/sdk-python) - [Python API docs](https://python.temporal.io/index.html) --- # Temporal Python SDK synchronous vs. asynchronous Activity implementations Source: https://docs.temporal.io/develop/python/best-practices/python-sdk-sync-vs-async > The Temporal Python SDK supports implementing Activities asynchronously with asyncio, synchronously with ThreadPoolExecutor or ProcessPoolExecutor. Choose the correct method to avoid application errors. The Temporal Python SDK supports multiple ways of implementing an Activity: - Asynchronously using [`asyncio`](https://docs.python.org/3/library/asyncio.html) - Synchronously multithreaded using [`concurrent.futures.ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) - Synchronously multiprocess using [`concurrent.futures.ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and [`multiprocessing.managers.SyncManager`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.managers.SyncManager) It is important to implement your Activities using the correct method, otherwise your application may fail in sporadic and unexpected ways. Which one you should use depends on your use case. This section provides guidance to help you choose the best approach. ## The Python Asynchronous Event Loop and Blocking Calls First, let's look at how async event loops work in Python. The Python async event loop runs in a thread and executes all tasks in its thread. When any task is running in the event loop, the loop is blocked and no other tasks can be running at the same time within that event loop. Whenever a task executes an `await` expression, the task is suspended, and the event loop begins or resumes execution of another task. This means that the event loop can only pass the flow of control when the `await` keyword is executed. If a program makes a blocking call, such as one that reads from a file, makes a synchronous request to a network service, waits for user input, or anything else that blocks the execution, the entire event loop must wait until that execution has completed. Blocking the async event loop in Python would turn your asynchronous program into a synchronous program that executes serially, defeating the entire purpose of using `asyncio`. This can also lead to potential deadlock, and unpredictable behavior that causes tasks to be unable to execute. Debugging these issues can be difficult and time consuming, as locating the source of the blocking call might not always be immediately evident. Due to this, Python developers must be extra careful to not make blocking calls from within an asynchronous Activity, or use an async safe library to perform these actions. For example, making an HTTP call with the popular `requests` library within an asynchronous Activity would lead to blocking your event loop. If you want to make an HTTP call from within an asynchronous Activity, you should use an async-safe HTTP library such as `aiohttp` or `httpx`. Otherwise, use a synchronous Activity. ## Python SDK Worker Execution Architecture Python workers have the following components for executing code: - Your event loop, which runs Tasks from async Activities **plus the rest of the Temporal Worker, such as communicating with the server**. - An executor for executing Activity Tasks from synchronous Activities. A thread pool executor is recommended. - A thread pool executor for executing Workflow Tasks. > See Also: [docs for](https://python.temporal.io/temporalio.worker.Worker.html#__init__) `worker.__init__()` ### Activities - Async Activities and the temporal worker SDK code both run the default asyncio event loop or whatever event loop you give the Worker. - Synchronous Activities run in the `activity_executor`. ### Workflows Since Workflow Tasks have the following three properties, they're run in threads. - are CPU bound - need to be timed out for deadlock detection - need to not block other Workflow Tasks The `workflow_task_executor` is the thread pool these Tasks are run on. The fact that Workflow Tasks run in a thread pool can be confusing at first because Workflow Definitions are `async`. The key differentiator is that the `async` in Workflow Definitions isn't referring to the standard event loop -- it's referring to the Workflow's own event loop. Each Workflow gets its own “Workflow event loop,” which is deterministic, and described in [the Python SDK blog](https://temporal.io/blog/durable-distributed-asyncio-event-loop#temporal-workflows-are-asyncio-event-loops). The Workflow event loop doesn't constantly loop -- it just gets cycled through during a Workflow Task to make as much progress as possible on all of its futures. When it can no longer make progress on any of its futures, then the Workflow Task is complete. ### Number of CPU cores The only ways to use more than one core in a python Worker (considering Python's GIL) are: - Run more than one Worker Process. - Run the synchronous Activities in a process pool executor, but a thread pool executor is recommended. ### A Worker infrastructure option: Separate Activity and Workflow Workers To reduce the risk of event loops or executors getting blocked, some users choose to deploy separate Workers for Workflow Tasks and Activity Tasks. ## Activity Definition **By default, Activities should be synchronous rather than asynchronous**. You should only make an Activity asynchronous if you are certain that it doesn't block the event loop. This is because if you have blocking code in an `async def` function, it blocks your event loop and the rest of Temporal, which can cause bugs that are hard to diagnose, including freezing your worker and blocking Workflow progress (because Temporal can't tell the server that Workflow Tasks are completing). The reason synchronous Activities help is because they run in the `activity_executor` ([docs for](https://python.temporal.io/temporalio.worker.Worker.html#__init__) `worker.__init__()`) rather than in the global event loop, which helps because: - There's no risk of accidentally blocking the global event loop. - If you have multiple Activity Tasks running in a thread pool rather than an event loop, one bad Activity Task can't slow down the others; this is because the OS scheduler preemptively switches between threads, which the event loop coordinator doesn't do. > See Also: > ["Types of Activities" section of Python SDK README](https://github.com/temporalio/sdk-python#types-of-activities) ## How to implement Synchronous Activities The following code is a synchronous Activity Definition. It takes a name (`str`) as input and returns a customized greeting (`str`) as output. It makes a call to a microservice, and when making this call, you'll notice that it uses the `requests` library. This is safe to do in synchronous Activities. ```python import urllib.parse import requests from temporalio import activity class TranslateActivities: @activity.defn def greet_in_spanish(self, name: str) -> str: greeting = self.call_service("get-spanish-greeting", name) return greeting # Utility method for making calls to the microservices def call_service(self, stem: str, name: str) -> str: base = f"http://localhost:9999/{stem}" url = f"{base}?name={urllib.parse.quote(name)}" response = requests.get(url) return response.text ``` The preceding example doesn't share a session across the Activity, so `__init__` was removed. While `requests` does have the ability to create sessions, it's currently unknown if they're thread safe. Due to no longer having or needing `__init__`, the case could be made here to not implement the Activities as a class, but just as decorated functions as shown here: ```python @activity.defn def greet_in_spanish(name: str) -> str: greeting = call_service("get-spanish-greeting", name) return greeting # Utility method for making calls to the microservices def call_service(stem: str, name: str) -> str: base = f"http://localhost:9999/{stem}" url = f"{base}?name={urllib.parse.quote(name)}" response = requests.get(url) return response.text ``` Whether to implement Activities as class methods or functions is a design choice left up to the developer when cross-activity state isn't needed. Both are equally valid implementations. ### How to run Synchronous Activities on a Worker When running synchronous Activities, the Worker needs to have an `activity_executor`. Temporal recommends using a `ThreadPoolExecutor` as shown here: ```python with ThreadPoolExecutor(max_workers=42) as executor: worker = Worker( # ... activity_executor=executor, # ... ) ``` ## How to Implement Asynchronous Activities The following code is an implementation of the preceding Activity, but as an asynchronous Activity Definition. It makes a call to a microservice, accessed through HTTP, to request this greeting in Spanish. This Activity uses the `aiohttp` library to make an async safe HTTP request. Using the `requests` library here would have resulted in blocking code within the async event loop, which will block the entire async event loop. For more in-depth information about this issue, refer to the [Python asyncio documentation](https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code). The following code also implements the Activity Definition as a class, rather than a function. The `aiohttp` library requires an established `Session` to perform the HTTP request. It would be inefficient to establish a `Session` every time an Activity is invoked, so instead this code accepts a `Session` object as an instance parameter and makes it available to the methods. This approach will also be beneficial when the execution is over and the `Session` needs to be closed. In this example, the Activity supplies the name in the URL and retrieves the greeting from the body of the response. ```python import aiohttp import urllib.parse from temporalio import activity class TranslateActivities: def __init__(self, session: aiohttp.ClientSession): self.session = session @activity.defn async def greet_in_spanish(self, name: str) -> str: greeting = await self.call_service("get-spanish-greeting", name) return greeting # Utility method for making calls to the microservices async def call_service(self, stem: str, name: str) -> str: base = f"http://localhost:9999/{stem}" url = f"{base}?name={urllib.parse.quote(name)}" async with self.session.get(url) as response: translation = await response.text() if response.status >= 400: raise ApplicationError( f"HTTP Error {response.status}: {translation}", # We want to have Temporal automatically retry 5xx but not 4xx non_retryable=response.status < 500, ) return translation ``` ### How to run synchronous code from an asynchronous activity If your Activity is asynchronous and you don't want to change it to synchronous, but you need to run blocking code inside it, then you can use python utility functions to run synchronous code in an asynchronous function: - [`loop.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor), which is also mentioned in the ["running blocking code" section of the "developing with asyncio" guide](https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code) - [`asyncio.to_thread()`](https://docs.python.org/3/library/asyncio-task.html#running-in-threads) ## When Should You Use Async Activities Asynchronous Activities have many advantages, such as potential speed up of execution. However, as discussed above, making unsafe calls within the async event loop can cause sporadic and difficult to diagnose bugs. For this reason, we recommend using asynchronous Activities _only_ when you are certain that your Activities are async safe and don't make blocking calls. If you experience bugs that you think may be a result of an unsafe call being made in an asynchronous Activity, convert it to a synchronous Activity and see if the issue resolves. --- # Testing - Python SDK Source: https://docs.temporal.io/develop/python/best-practices/testing-suite > The Temporal Application Testing guide covers Frameworks facilitating Workflow and integration testing, including end-to-end, integration, and unit tests. Use mocked Activities, skip time in tests, and replay Workflow Executions. The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks Some SDKs have support or examples for popular test frameworks, runners, or libraries. One recommended framework for testing in Python for the Temporal SDK is [pytest](https://docs.pytest.org/), which can help with fixtures to stand up and tear down test environments, provide useful test discovery, and make it easy to write parameterized tests. If you do use `pytest`, consider using `-s` (`--show-capture=no`) so you can see the logs live. ## Testing Activities An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. ### Run an Activity If an Activity references its context, you need to mock that context when testing in isolation. To run an Activity in a test, use the [`ActivityEnvironment`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html) class. This class allows you to run any callable inside an Activity context. Use it to test the behavior of your code under various conditions. ### Listen to Heartbeats When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them. To test a Heartbeat in an Activity, use the [`on_heartbeat()`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html#on_heartbeat) property of the [`ActivityEnvironment`](https://python.temporal.io/temporalio.testing.ActivityEnvironment.html) class. This property sets a custom function that is called every time the `activity.heartbeat()` function is called within the Activity. ```python @activity.defn async def activity_with_heartbeats(param: str): activity.heartbeat(f"param: {param}") activity.heartbeat("second heartbeat") env = ActivityEnvironment() heartbeats = [] # Set the `on_heartbeat` property to a callback function that will be called for each Heartbeat sent by the Activity. env.on_heartbeat = lambda *args: heartbeats.append(args[0]) # Use the run method to start the Activity, passing in the function that contains the Heartbeats and any necessary parameters. await env.run(activity_with_heartbeats, "test") # Verify that the expected Heartbeats are received by the callback function. assert heartbeats == ["param: test", "second heartbeat"] ``` ## Testing Workflows The simplest test case we can write is to have the test environment execute the Workflow and then evaluate the results. `WorkflowEnvironment.start_local` configures a local environment for running and testing Workflows: ```python import uuid import pytest from temporalio import activity from temporalio.testing import WorkflowEnvironment from temporalio.worker import Worker from activities import greet from workflows import SayHelloWorkflow @pytest.mark.asyncio async def test_say_hello_workflow(): """Execute the workflow end-to-end with its real activity.""" task_queue_name = str(uuid.uuid4()) async with await WorkflowEnvironment.start_local(ui=True, ui_port=8233) as env: async with Worker( env.client, task_queue=task_queue_name, workflows=[SayHelloWorkflow], activities=[greet], ): result = await env.client.execute_workflow( SayHelloWorkflow.run, "Temporal", id=str(uuid.uuid4()), task_queue=task_queue_name, ) assert result == "Hello Temporal" ``` You can also pass `ui=True` to `start_local` to see the UI. ### How to mock Activities When running unit tests on Workflows, many times you want to test the Workflow logic in isolation. When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker. ```python import uuid from temporalio.client import Client from temporalio.worker import Worker # Import your Activity Definition and real implementation from hello.hello_activity import ( ComposeGreetingInput, GreetingWorkflow, compose_greeting, ) # Define your mocked Activity implementation @activity.defn(name="compose_greeting") async def compose_greeting_mocked(input: ComposeGreetingInput) -> str: return f"{input.greeting}, {input.name} from mocked activity!" async def test_mock_activity(client: Client): task_queue_name = str(uuid.uuid4()) # Provide the mocked Activity implementation to the Worker async with Worker( client, task_queue=task_queue_name, workflows=[GreetingWorkflow], activities=[compose_greeting_mocked], ): # Execute your Workflow as usual assert "Hello, World from mocked activity!" == await client.execute_workflow( GreetingWorkflow.run, "World", id=str(uuid.uuid4()), task_queue=task_queue_name, ) ``` The mocked Activity implementation should have the same signature as the real implementation (including the input and output types) and the same name. When the Workflow invokes the Activity, it invokes the mocked implementation instead of the real one, allowing you to test your Workflow isolated. ### How to skip time Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `WorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Skip time automatically Use the [`start_time_skipping()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#start_time_skipping) method to start a test server process and skip time automatically. In time-skipping mode, Timers, which include sleeps and conditional timeouts, are fast-forwarded except when Activities are running. Use the [`start_local()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#start_local) method for testing a full local Temporal Server. There is no time skipping in this environment, so any Timers will wait the actual amount of time. You can find an example of this being used in [the code above](/develop/python/best-practices/testing-suite#test-workflows). Use the [`from_client()`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#from_client) method for an existing Temporal Server. #### Skip time manually To implement time skipping manually, use the [`sleep`](https://python.temporal.io/temporalio.testing.WorkflowEnvironment.html#sleep) method inside the `WorkflowEnvironment`. This will manually advance time by the duration you specify. ```python from temporalio.testing import WorkflowEnvironment async def test_manual_time_skipping(): async with await WorkflowEnvironment.start_time_skipping() as env: # Your code here # You can use the env.sleep(seconds) method to manually advance time await env.sleep(3) # This will advance time by 3 seconds # Your code here ``` ## How to Replay a Workflow Execution Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: To replay Workflow Executions, use the [`replay_workflows`](https://python.temporal.io/temporalio.worker.Replayer.html#replay_workflows) or [`replay_workflow`](https://python.temporal.io/temporalio.worker.Replayer.html#replay_workflow) methods, passing one or more Event Histories as arguments. In the following example (which, as of server v1.18, requires Advanced Visibility to be enabled), Event Histories are downloaded from the server and then replayed. If any replay fails, the code raises an exception. ```python workflows = client.list_workflows(f"TaskQueue=foo and StartTime > '2022-01-01T12:00:00'") histories = workflows.map_histories() replayer = Replayer( workflows=[MyWorkflowA, MyWorkflowB, MyWorkflowC] ) await replayer.replay_workflows(histories) ``` In the next example, a single history is loaded from a JSON string: ```python replayer = Replayer(workflows=[YourWorkflow]) await replayer.replay_workflow(WorkflowHistory.from_json(history_json_str)) ``` In both examples, if Event History is non-deterministic, an error is thrown. You can choose to wait until all histories have been replayed with `replay_workflows` by setting the `fail_fast` option to `false`. > **📝 Note:** > > If the Event History is exported by [Temporal Web UI](/web-ui) or through [Temporal CLI](/cli), you can pass the JSON file history object as a JSON string or as a Python dictionary through the `json.load()` function, which takes a file object and returns the JSON object. > > :::tip > When fetching event histories directly from the server or exporting them, be aware that the data can be protobuf-encoded (`bytes`). The `Replayer`, however, often works with decoded histories (like a `dict`). > > If you encounter `TypeError` exceptions related to `dict` vs. `bytes` mismatches during replay, ensure your event history is properly decoded before passing it to the Replayer. --- # Client - Python SDK Source: https://docs.temporal.io/develop/python/client > This section explains how to implement the Temporal Client with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Temporal Client - [Temporal Client](/develop/python/client/temporal-client) --- # Temporal Client - Python SDK Source: https://docs.temporal.io/develop/python/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service. Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and Queries to Workflow Executions, getting Workflow results, and more. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the Python SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow-execution) - [Get Workflow results](#get-workflow-results) A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service. ## Connect to development Temporal Service Use [`Client.connect`](https://python.temporal.io/temporalio.client.Client.html#connect) to create a client. Connection options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide these options directly in code, load them from **environment variables**, or a **TOML configuration file** using the [`envconfig`](https://python.temporal.io/temporalio.envconfig.html) helpers. We recommend environment variables or a configuration file for secure, repeatable configuration. When you’re running a Temporal Service locally (such as with the [Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to [Environment Configuration](/develop/environment-configuration#configuration-methods) for more details about configuration files and profiles. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml title="config.toml" # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS auto-enables when TLS config or an API key is present # disabled = false client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a profile from the configuration file using the `ClientConfig.load_client_connect_config` function as follows. In this example, you load the `default` profile for local development: ```python {23-25} import asyncio from pathlib import Path from temporalio.client import Client from temporalio.envconfig import ClientConfig async def main(): """ Loads the default profile from the config.toml file in this directory. """ print("--- Loading default profile from config.toml ---") # For this sample to be self-contained, we explicitly provide the path to # the config.toml file included in this directory. # By default though, the config.toml file will be loaded from # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). config_file = Path(__file__).parent / "config.toml" # load_client_connect_config is a helper that loads a profile and prepares # the config dictionary for Client.connect. By default, it loads the # "default" profile. connect_config = ClientConfig.load_client_connect_config( config_file=str(config_file) ) print(f"Loaded 'default' profile from {config_file}.") print(f" Address: {connect_config.get('target_host')}") print(f" Namespace: {connect_config.get('namespace')}") print(f" gRPC Metadata: {connect_config.get('rpc_metadata')}") print("\nAttempting to connect to client...") try: await Client.connect(**connect_config) # type: ignore print("✅ Client connected successfully!") except Exception as e: print(f"❌ Failed to connect: {e}") if __name__ == "__main__": asyncio.run(main()) ``` **Environment Variables** Use the `envconfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. Set the following environment variables before running your Python application. Replace the placeholder values with your actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the `default` Namespace. You may omit these variables entirely since they're the defaults. ```bash export TEMPORAL_NAMESPACE="default" export TEMPORAL_ADDRESS="localhost:7233" ``` After setting the environment variables, you can create a Temporal Client as follows: ```python {11,19} import asyncio from pathlib import Path from temporalio.client import Client from temporalio.envconfig import ClientConfig async def main(): # load_client_connect_config is a helper that loads a profile and prepares # the config dictionary for Client.connect. By default, it loads the # "default" profile. connect_config = ClientConfig.load_client_connect_config() print(f" Address: {connect_config.get('target_host')}") print(f" Namespace: {connect_config.get('namespace')}") print(f" gRPC Metadata: {connect_config.get('rpc_metadata')}") print("\nAttempting to connect to client...") try: await Client.connect(**connect_config) # type: ignore print("✅ Client connected successfully!") except Exception as e: print(f"❌ Failed to connect: {e}") if __name__ == "__main__": asyncio.run(main()) ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. Use the `connect()` method on the `Client` class to create and connect to a Temporal Client to the Temporal Service. ```python import asyncio from temporalio.client import Client from your_workflows import YourWorkflow async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your name", id="your-workflow-id", task_queue="your-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace and Account ID_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the [`connect` method](https://python.temporal.io/temporalio.client.Client.html#connect) on the `Client` class to create a Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. ```python {14,23-25} import asyncio from pathlib import Path from temporalio.client import Client from temporalio.envconfig import ClientConfig async def main(): """ Demonstrates loading a named profile and overriding values programmatically. """ print("--- Loading 'staging' profile with programmatic overrides ---") config_file = Path(__file__).parent / "config.toml" profile_name = "staging" print( "The 'staging' profile in config.toml has an incorrect address (localhost:9999)." ) print("We'll programmatically override it to the correct address.") # Load the 'staging' profile. connect_config = ClientConfig.load_client_connect_config( profile=profile_name, config_file=str(config_file), ) # Override the target host to the correct address. # This is the recommended way to override configuration values. connect_config["target_host"] = "localhost:7233" print(f"\nLoaded '{profile_name}' profile from {config_file} with overrides.") print( f" Address: {connect_config.get('target_host')} (overridden from localhost:9999)" ) print(f" Namespace: {connect_config.get('namespace')}") print("\nAttempting to connect to client...") try: await Client.connect(**connect_config) # type: ignore print("✅ Client connected successfully!") except Exception as e: print(f"❌ Failed to connect: {e}") if __name__ == "__main__": asyncio.run(main()) ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your Python application. Import the `temporalio.envconfig` package to set connection options for the Temporal Client using environment variables. The `ClientConfig.load_client_connect_config` function will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/develop/environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. After setting the environment variables, use the following code to create the Temporal Client: ```python {11, 19} import asyncio from pathlib import Path from temporalio.client import Client from temporalio.envconfig import ClientConfig async def main(): # load_client_connect_config is a helper that loads a profile and prepares # the config dictionary for Client.connect. By default, it loads the # "default" profile. connect_config = ClientConfig.load_client_connect_config() print(f" Address: {connect_config.get('target_host')}") print(f" Namespace: {connect_config.get('namespace')}") print(f" gRPC Metadata: {connect_config.get('rpc_metadata')}") print("\nAttempting to connect to client...") try: await Client.connect(**connect_config) # type: ignore print("✅ Client connected successfully!") except Exception as e: print(f"❌ Failed to connect: {e}") if __name__ == "__main__": asyncio.run(main()) ``` **Code** You can also specify connection options directly in code to connect to Temporal Cloud. To create an initial connection, provide the endpoint, Namespace and Account ID combination, and API key values to the `Client.connect` method. ```python client = await Client.connect( , namespace=., api_key=, tls=True, ) ``` To connect using mTLS instead of an API key, provide the mTLS certificate and private key as follows: ```python from temporalio.client import Client, TLSConfig # ... async def main(): with open("client-cert.pem", "rb") as f: client_cert = f.read() with open("client-private-key.pem", "rb") as f: client_private_key = f.read() client = await Client.connect( "your-custom-namespace.tmprl.cloud:7233", namespace=".", tls=TLSConfig( client_cert=client_cert, client_private_key=client_private_key, # domain=domain, # TLS domain # server_root_ca_cert=server_root_ca_cert, # ROOT CA to validate the server cert ), ) ``` For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server). Unlike an API key, `TLSConfig` is static: the certificate bytes you pass to `Client.connect` are fixed for the lifetime of that `Client`. To rotate an mTLS client certificate without restarting your Worker, connect a new `Client` with the new certificate, then assign it to the running `Worker`'s `client` property: ```python with open("client-cert-new.pem", "rb") as f: client_cert = f.read() with open("client-private-key-new.pem", "rb") as f: client_private_key = f.read() new_client = await Client.connect( "your-custom-namespace.tmprl.cloud:7233", namespace=".", tls=TLSConfig( client_cert=client_cert, client_private_key=client_private_key, ), ) # worker is the Worker instance already running against the old client worker.client = new_client ``` The Worker starts using `new_client` for subsequent calls to the Temporal Service (Workflow Task completion, Activity Heartbeats, and so on); calls already in flight on the old client finish normally. The new client must use the same `Runtime` as the Worker's current client. ## Start a Workflow Execution [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from within another Workflow Execution, use either the [Child Workflow](/develop/python/workflows/child-workflows) or External Workflow APIs. See the [Customize Workflow Type](/develop/python/workflows/basics#workflow-type) section to see how to customize the name of the Workflow Type. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. To start a Workflow Execution in Python, use either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods in the Client. ```python # ... async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your name", id="your-workflow-id", task_queue="your-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ### Set a Workflow's Task Queue In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue). For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task Queue name. To set a Task Queue in Python, specify the `task_queue` argument when executing a Workflow with either [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) methods. ```python # ... async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your name", id="your-workflow-id", task_queue="your-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ### Set a Workflow Id You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id). When setting a Workflow Id, we recommended mapping it to a business process or business entity identifier, such as an order identifier or customer identifier. To set a Workflow Id in Python, specify the `id` argument when executing a Workflow with either [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) methods. The `id` argument should be a unique identifier for the Workflow Execution. ```python # ... async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your name", id="your-workflow-id", task_queue="your-task-queue", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ### Get the results of a Workflow Execution If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. Use [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`get_workflow_handle()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle) to return a Workflow handle. Then use the [`result`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#result) method to await on the result of the Workflow. To get a handle for an existing Workflow by its Id, you can use [`get_workflow_handle()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle), or use [`get_workflow_handle_for()`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle_for) for type safety. Then use [`describe()`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#describe) to get the current status of the Workflow. If the Workflow does not exist, this call fails. ```python {8-11} import asyncio from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") handle = client.get_workflow_handle( workflow_id="your-workflow-id", ) results = await handle.result() print(f"Result: {results}") if __name__ == "__main__": asyncio.run(main()) ``` --- # Data handling - Python SDK Source: https://docs.temporal.io/develop/python/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/python/data-handling/data-conversion) | [PayloadCodec](/develop/python/data-handling/data-encryption) | [ExternalStorage](/develop/python/data-handling/external-storage) | | ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------- | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | Offload large payloads to external store | | **Default** | JSON serialization | None (passthrough) | None (all payloads are stored in Event History) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - Python SDK Source: https://docs.temporal.io/develop/python/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the Python SDK, including Pydantic and custom type examples. Payload Converters serialize your application objects into a `Payload` and deserialize them back. A `Payload` is a binary form with metadata that Temporal uses to transport data. By default, Temporal uses a `DefaultPayloadConverter` that handles `None`, `bytes`, protobuf messages, and anything JSON-serializable. You only need a custom Payload Converter when your application uses types that aren't natively supported. ## Default supported types The default Data Converter supports converting multiple types including: - `None` - `bytes` - `google.protobuf.message.Message` — As JSON when encoding, but can decode binary proto from other languages - Anything that can be converted to JSON including: - Anything that [`json.dump`](https://docs.python.org/3/library/json.html#json.dump) supports natively - [dataclasses](https://docs.python.org/3/library/dataclasses.html) - Iterables including ones JSON dump may not support by default, for example `set` - [IntEnum, StrEnum](https://docs.python.org/3/library/enum.html) based enumerates - [UUID](https://docs.python.org/3/library/uuid.html) Although Workflows, Updates, Signals, and Queries can all be defined with multiple input parameters, users are strongly encouraged to use a single `dataclass` or Pydantic model parameter so that fields with defaults can be easily added without breaking compatibility. Similar advice applies to return values. Classes with generics may not have the generics properly resolved. The current implementation does not have generic type resolution. Users should use concrete types. ## Use Pydantic models To use Pydantic model instances, install Pydantic and set the Pydantic Data Converter when creating Client instances: ```python from temporalio.contrib.pydantic import pydantic_data_converter client = Client(data_converter=pydantic_data_converter, ...) ``` This Data Converter supports conversion of all [types supported by Pydantic](https://docs.pydantic.dev/latest/api/standard_library_types/) to and from JSON. In addition to Pydantic models, supported types include: - Everything that [`json.dumps()`](https://docs.python.org/3/library/json.html#py-to-json-table) supports by default. - Several standard library types that `json.dumps()` does not support, including dataclasses, types from the datetime module, sets, UUID, etc. - Custom types composed of any of these, with any degree of nesting. For example, a list of Pydantic models with `datetime` fields. See the [Pydantic documentation](https://docs.pydantic.dev/latest/api/standard_library_types/) for full details. > **📝 Note:** > > Pydantic v1 isn't supported by this Data Converter. > If you aren't yet able to upgrade from Pydantic v1, see https://github.com/temporalio/samples-python/tree/main/pydantic_converter_v1 for limited v1 support. > `datetime.date`, `datetime.time`, and `datetime.datetime` can only be used with the Pydantic Data Converter. ## How the default converter works The default converter is a `CompositePayloadConverter` that tries each encoding converter in order until one handles the value. Upon serialization, each `EncodingPayloadConverter` is used in order until one succeeds. Payload Converters can be customized independently of a Payload Codec. ## Custom Payload Converters To handle custom data types, create a new `EncodingPayloadConverter`. For example, to support `IPv4Address` types: ```python class IPv4AddressEncodingPayloadConverter(EncodingPayloadConverter): @property def encoding(self) -> str: return "text/ipv4-address" def to_payload(self, value: Any) -> Optional[Payload]: if isinstance(value, ipaddress.IPv4Address): return Payload( metadata={"encoding": self.encoding.encode()}, data=str(value).encode(), ) else: return None def from_payload(self, payload: Payload, type_hint: Optional[Type] = None) -> Any: assert not type_hint or type_hint is ipaddress.IPv4Address return ipaddress.IPv4Address(payload.data.decode()) class IPv4AddressPayloadConverter(CompositePayloadConverter): def __init__(self) -> None: # Just add ours as first before the defaults super().__init__( IPv4AddressEncodingPayloadConverter(), *DefaultPayloadConverter.default_encoding_payload_converters, ) my_data_converter = dataclasses.replace( DataConverter.default, payload_converter_className=IPv4AddressPayloadConverter, ) ``` ### Customize the JSON converter for custom types If you need your custom type to work in lists, unions, and other collections, customize the existing JSON converter instead of adding a new encoding converter. The JSON converter is the last in the list, so it handles any otherwise unknown type. Customize serialization with a custom `json.JSONEncoder` and deserialization with a custom `JSONTypeConverter`: ```python class IPv4AddressJSONEncoder(AdvancedJSONEncoder): def default(self, o: Any) -> Any: if isinstance(o, ipaddress.IPv4Address): return str(o) return super().default(o) class IPv4AddressJSONTypeConverter(JSONTypeConverter): def to_typed_value( self, hint: Type, value: Any ) -> Union[Optional[Any], _JSONTypeConverterUnhandled]: if issubclass(hint, ipaddress.IPv4Address): return ipaddress.IPv4Address(value) return JSONTypeConverter.Unhandled class IPv4AddressPayloadConverter(CompositePayloadConverter): def __init__(self) -> None: # Replace default JSON plain with our own that has our encoder and type # converter json_converter = JSONPlainPayloadConverter( encoder=IPv4AddressJSONEncoder, custom_type_converters=[IPv4AddressJSONTypeConverter()], ) super().__init__( *[ c if not isinstance(c, JSONPlainPayloadConverter) else json_converter for c in DefaultPayloadConverter.default_encoding_payload_converters ] ) my_data_converter = dataclasses.replace( DataConverter.default, payload_converter_className=IPv4AddressPayloadConverter, ) ``` Now `IPv4Address` can be used in type hints including collections, optionals, etc. --- # Payload encryption - Python SDK Source: https://docs.temporal.io/develop/python/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Python SDK. Payload Codecs transform `Payload` bytes after serialization (by the Payload Converter) and before the data is sent to the Temporal Service. Unlike Payload Converters, codecs run outside the Workflow sandbox, so they can use non-deterministic operations and call external services. The most common use case is encryption: encrypting payloads before they reach the Temporal Service so that sensitive data is never stored in plaintext. ## PayloadCodec interface Implement a `PayloadCodec` with `encode()` and `decode()` methods. These should loop through all of a Workflow's payloads, perform your marshaling, compression, or encryption steps in order, and set an `"encoding"` metadata field. In this example, the `encode` method compresses a payload using Python's [cramjam](https://github.com/milesgranger/cramjam) library to provide `snappy` compression. The `decode()` function implements the `encode()` logic in reverse: ```python import cramjam from temporalio.api.common.v1 import Payload from temporalio.converter import PayloadCodec class CompressionCodec(PayloadCodec): async def encode(self, payloads: Iterable[Payload]) -> List[Payload]: return [ Payload( metadata={ "encoding": b"binary/snappy", }, data=(bytes(cramjam.snappy.compress(p.SerializeToString()))), ) for p in payloads ] async def decode(self, payloads: Iterable[Payload]) -> List[Payload]: ret: List[Payload] = [] for p in payloads: if p.metadata.get("encoding", b"").decode() != "binary/snappy": ret.append(p) continue ret.append(Payload.FromString(bytes(cramjam.snappy.decompress(p.data)))) return ret ``` ## Configure the codec on the Data Converter Add a `data_converter` parameter to your `Client.connect()` options that overrides the default converter with your Payload Codec: ```python from codec import CompressionCodec client = await Client.connect( "localhost:7233", data_converter=dataclasses.replace( temporalio.converter.default(), payload_codec=CompressionCodec() ), ) ``` For reference, see the [Encryption](https://github.com/temporalio/samples-python/tree/main/encryption) sample. ## Codec Server A Codec Server is an HTTP server that runs your `PayloadCodec` remotely, so that the Temporal Web UI and CLI can decode encrypted payloads for display. For more information, see [Codec Server](/codec-server). --- # External Storage - Python SDK Source: https://docs.temporal.io/develop/python/data-handling/external-storage > Offload large payloads to external storage using the claim check pattern in the Python SDK. > **Public Preview** > APIs and configuration may change before General Availability. Join the > [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for > help. The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how to set up External Storage with Amazon S3 and how to implement a custom storage driver. For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage). ## Store and retrieve large payloads with Amazon S3 The Python SDK includes an S3 storage driver. Follow these steps to set it up: ### Prerequisites - An Amazon S3 bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see [Durable External Storage](/external-storage#durable-external-storage). - Install the `aioboto3` extra: `python -m pip install "temporalio[aioboto3]"` ### Procedure 1. Create an S3 client using `aioboto3` and pass it to the `S3StorageDriver`. The driver uses your standard AWS credentials from the environment (environment variables, IAM role, or AWS config file): [features/snippets/external_storage/s3_setup/s3_driver_create.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.py) ```py session = aioboto3.Session(profile_name=AWS_PROFILE, region_name=AWS_REGION) async with session.client("s3") as s3_client: driver = S3StorageDriver( client=new_aioboto3_client(s3_client), bucket="my-temporal-payloads", ) ``` 2. Configure the driver on your `DataConverter` and pass the converter to your Client and Worker: [features/snippets/external_storage/s3_setup/s3_external_storage_setup.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_external_storage_setup.py) ```py data_converter = dataclasses.replace( DataConverter.default, external_storage=ExternalStorage(drivers=[driver]), ) client_config = ClientConfig.load_client_connect_config() client = await Client.connect(**client_config, data_converter=data_converter) worker = Worker( client, task_queue="my-task-queue", workflows=[], activities=[], ) ``` By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `payload_size_threshold` parameter, even setting it to 0 to externalize all payloads regardless of size. Refer to [Configure payload size threshold](#configure-payload-size-threshold) for more information. All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business logic. The driver uploads and downloads payloads concurrently and validates payload integrity on retrieve. The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage failures. For a complete working example that includes a Worker, Codec Server, and S3 driver, see the [External Storage sample](https://github.com/temporalio/samples-python/tree/main/external_storage). ## Implement a custom storage driver If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver. Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and [Lifecycle management](/external-storage#lifecycle) for retention requirements. The following example shows a custom driver that uses local disk as the backing store. This example is for local development and testing only. In production, use a durable storage system that is accessible to all Workers. For example, see the [Redis storage driver sample](https://github.com/temporalio/samples-python/tree/main/external_storage_redis). [features/snippets/external_storage/custom_driver/custom_storage_driver.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/custom_driver/custom_storage_driver.py) ```py class LocalDiskStorageDriver(StorageDriver): def __init__(self, store_dir: str = "/tmp/temporal-payload-store") -> None: self._store_dir = store_dir def name(self) -> str: return "local-disk" async def store( self, context: StorageDriverStoreContext, payloads: Sequence[Payload], ) -> list[StorageDriverClaim]: os.makedirs(self._store_dir, exist_ok=True) prefix = self._store_dir target = context.target if isinstance(target, StorageDriverWorkflowInfo) and target.id: prefix = os.path.join(self._store_dir, target.namespace, target.id) os.makedirs(prefix, exist_ok=True) claims = [] for payload in payloads: key = f"{uuid.uuid4()}.bin" file_path = os.path.join(prefix, key) with open(file_path, "wb") as f: f.write(payload.SerializeToString()) claims.append(StorageDriverClaim(claim_data={"path": file_path})) return claims async def retrieve( self, context: StorageDriverRetrieveContext, claims: Sequence[StorageDriverClaim], ) -> list[Payload]: payloads = [] for claim in claims: file_path = claim.claim_data["path"] with open(file_path, "rb") as f: raw = f.read() payload = Payload() payload.ParseFromString(raw) payloads.append(payload) return payloads ``` The following sections walk through the key parts of the driver implementation. ### 1. Extend the StorageDriver class A custom driver extends the `StorageDriver` abstract class and implements three methods: - `name()` returns a unique string that identifies the driver. The SDK stores this name in the claim check reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. - `store()` receives a list of payloads and returns one `StorageDriverClaim` per payload. A claim is a set of string key-value pairs that the driver uses to locate the payload later. - `retrieve()` receives the claims that `store()` produced and returns the original payloads. ### 2. Store payloads In `store()`, convert each Payload protobuf message to bytes with `payload.SerializeToString()` and write the bytes to your storage system. The application data has already been serialized by the [Payload Converter](/develop/python/data-handling/data-conversion) and [Payload Codec](/develop/python/data-handling/data-encryption) before it reaches the driver. See the [data conversion pipeline](/external-storage#data-pipeline) for more details. Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `context.target` provides identity information (namespace, Workflow ID, or Activity ID) depending on the operation. Consider structuring your storage keys to include this information so that you can identify which Workflow owns each payload. Within that scope, content-addressable keys (such as a SHA-256 hash of the payload bytes) can help deduplicate identical payloads. ### 3. Retrieve payloads In `retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message with `payload.ParseFromString(data)`. The Payload Converter handles deserializing the application data after the driver returns the payload. ### 4. Configure the Data Converter Pass an `ExternalStorage` instance to your `DataConverter` and use the converter when creating your Client and Worker. You can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services: [features/snippets/external_storage/custom_driver/custom_driver_data_converter.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/custom_driver/custom_driver_data_converter.py) ```py data_converter = dataclasses.replace( DataConverter.default, external_storage=ExternalStorage( drivers=[LocalDiskStorageDriver()], ), ) ``` ## Configure payload size threshold You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `payload_size_threshold` parameter, or set it to 0 to externalize all payloads regardless of size. Payloads smaller than the threshold stay inline in Event History. The size compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data. [features/snippets/external_storage/threshold/threshold_config.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/threshold/threshold_config.py) ```py data_converter = dataclasses.replace( DataConverter.default, external_storage=ExternalStorage( drivers=[driver], payload_size_threshold=0, ), ) ``` ## Use multiple storage drivers When you register multiple drivers, you must provide a `driver_selector` function that chooses which driver stores each payload. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when migrating between storage backends. Return `None` from the selector to keep a specific payload inline in Event History. Multiple drivers are useful in scenarios such as: - Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old driver remains available for retrieving existing claims. - Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver based on the runtime environment. The following example registers two drivers but always selects `preferred_driver` for new payloads. The `legacy_driver` is only registered so the Worker can retrieve payloads that were previously stored with it: [features/snippets/external_storage/multiple_drivers/multiple_drivers.py](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/multiple_drivers/multiple_drivers.py) ```py preferred_driver = S3StorageDriver(client=s3_client, bucket="my-bucket") legacy_driver = LegacyStorageDriver() ExternalStorage( drivers=[preferred_driver, legacy_driver], driver_selector=lambda context, payload: preferred_driver, ) ``` ## Multi-region durability To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an [S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the driver at the MRAP ARN instead of a bucket name. See [Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs. In code, the only change is the value you pass as `bucket`: ```py session = aioboto3.Session(profile_name=AWS_PROFILE, region_name=AWS_REGION) async with session.client("s3") as s3_client: driver = S3StorageDriver( client=new_aioboto3_client(s3_client), bucket="arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap", ) ``` `aioboto3` (via `botocore`) automatically uses [SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html) signing when the bucket value is an MRAP ARN. Make sure your `botocore` version is recent enough to support SigV4A. --- # Integrations Source: https://docs.temporal.io/develop/python/integrations > Integrations with other tools and services. The following integrations are available for the Temporal Python SDK. These integrations are built on the Temporal Python SDK's [Plugin system](/develop/plugins-guide), which you can also use to build your own integrations. - [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) — Monitor and evaluate AI application performance with Braintrust observability. _(Python · Agent observability)_ - [Deep Agents](/develop/python/integrations/deepagents) — Make LangChain Deep Agents durable with Temporal Workflows and Activities. _(Python · Agent framework)_ - [Google ADK](/develop/python/integrations/google-adk) — Orchestrate Google ADK agents with durable Temporal Workflows. _(Python · Agent framework)_ - [Google GenAI](/develop/python/integrations/google-genai) — Call Google Gemini models durably from Temporal Workflows with the Google Gen AI SDK. _(Python · Agent framework)_ - [Langfuse](https://langfuse.com/integrations/frameworks/temporal) — Trace and debug LLM calls in Temporal Workflows with Langfuse. _(Python · Agent observability)_ - [LangGraph](/develop/python/integrations/langgraph) — Run LangGraph agent graphs as durable, resumable Temporal Workflows. _(Python · Agent framework)_ - [LangSmith](/develop/python/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(Python · Agent observability)_ - [OpenAI Agents SDK](/develop/python/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(Python · Agent framework)_ - [OpenBox](https://docs.openbox.ai/getting-started/temporal) — Add governance and observability guardrails to AI-powered Temporal Workflows with OpenBox. _(Python · Governance · Agent observability)_ - [Parseable](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(Python · Agent observability)_ - [Pydantic AI](https://ai.pydantic.dev/durable_execution/temporal/) — Build type-safe AI agents with Durable Execution through Pydantic AI. _(Python · Agent framework)_ - [Strands Agents](/develop/python/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(Python · Agent framework)_ - [Tenuo](https://tenuo.ai/temporal) — Add governance and compliance guardrails to AI-powered Temporal Workflows. _(Python · Governance)_ --- # Deep Agents integration Source: https://docs.temporal.io/develop/python/integrations/deepagents Temporal's integration with [Deep Agents](https://github.com/langchain-ai/deepagents) makes an existing LangChain Deep Agent durable by adding one plugin. Build your agent with `create_deep_agent(...)` inside a Workflow, add `DeepAgentsPlugin` to your Client or Worker, and each LLM call and each I/O tool call becomes a Temporal Activity — while the agent's control loop runs, and deterministically replays, inside the Workflow. The code you already wrote against `deepagents` doesn't change. Sub-agents, planning and todo state, the filesystem middleware, human-in-the-loop interrupts, and `agent.ainvoke(...)` all keep working. On top of that you get crash durability, automatic retries and timeouts on every model and tool call, resumable human-in-the-loop, and bounded Workflow history for long-running agents. > **Pre-release** Code snippets in this guide are taken from the [Deep Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/deepagents_plugin). Refer to the samples for the complete code. ## Prerequisites - Python 3.11 or newer. - This guide assumes you are already familiar with Deep Agents. If you aren't, refer to the [Deep Agents documentation](https://github.com/langchain-ai/deepagents) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the Temporal development server running if you want to test your code locally. ## Install the plugin The plugin lives on the Temporal Python SDK's `main` branch and ships as the `temporalio[deepagents]` extra in the next SDK release. Until that release is on PyPI, install it from `main`: ```bash uv add "temporalio[deepagents] @ git+https://github.com/temporalio/sdk-python.git" ``` or with pip: ```bash pip install "temporalio[deepagents] @ git+https://github.com/temporalio/sdk-python.git" ``` This builds the SDK from source, including its Rust core, so expect a few minutes on first install. Once the next release is available, a plain `uv add "temporalio[deepagents]"` (or `pip install "temporalio[deepagents]"`) is all you need. ## Hello World A Deep Agent becomes durable without changing the agent code itself. The Workflow below builds a vanilla `create_deep_agent(...)` and drives it with `await agent.ainvoke(...)` — exactly the code you would write outside Temporal. Because `model=` is a bare `"provider:name"` string, the plugin auto-routes the model call through the `deepagents.invoke_model` Activity, so the LLM call gets Temporal-managed retries and timeouts while the agent's control loop replays deterministically in the Workflow. [deepagents_plugin/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/workflow.py) ```py # No `workflow.unsafe.imports_passed_through()` guard is needed: the plugin # configures the workflow sandbox to pass the deepagents / LangChain import # tree through, so workflow files import them like any other module. from deepagents import create_deep_agent from temporalio import workflow @workflow.defn class HelloWorldAgent: @workflow.run async def run(self, question: str) -> str: agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", system_prompt="You are a helpful assistant. Answer concisely.", ) result = await agent.ainvoke( {"messages": [{"role": "user", "content": question}]} ) return result["messages"][-1].content ``` `DeepAgentsPlugin` is a **client-level** plugin: add it to `Client.connect(...)` and the SDK propagates it to any Worker built from that Client. Add it on exactly one side. The plugin registers the `deepagents.*` Activities and installs the LangChain-aware data converter, so the Worker needs no other wiring. [deepagents_plugin/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/run_worker.py) ```py import asyncio import os from temporalio.client import Client from temporalio.contrib.deepagents import DeepAgentsPlugin from temporalio.worker import Worker from deepagents_plugin.hello_world.workflow import HelloWorldAgent async def main() -> None: client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[DeepAgentsPlugin()], ) worker = Worker( client, task_queue="deepagents-hello-world", workflows=[HelloWorldAgent], ) print("Worker started. Ctrl+C to exit.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Use `model_activity_options` on the plugin to control the timeout and retry policy for each model call. API keys live on the Worker via the model provider (LangChain's `init_chat_model` by default), never in Workflow inputs or history — the Workflow ships only the model name, and the Worker builds the real client. LLM-SDK-side retries are disabled so that Temporal owns retries and timeouts. Start the Workflow like any other: [deepagents_plugin/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/hello_world/run_workflow.py) ```py import asyncio import os from temporalio.client import Client from deepagents_plugin.hello_world.workflow import HelloWorldAgent async def main() -> None: client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")) result = await client.execute_workflow( HelloWorldAgent.run, "What is Temporal in one sentence?", id="deepagents-hello-world", task_queue="deepagents-hello-world", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Choose Workflow or Activity execution per tool A Deep Agent holds its tools in-Workflow. A tool that only reads or writes agent state is pure and belongs there; a tool that does real I/O must not run in Workflow code. The plugin gives you two explicit ways to move a tool's work into an Activity: - `activity_as_tool(my_activity, ...)` surfaces an existing `@activity.defn` function as a Deep Agents tool without re-declaring it. - `tool_as_activity(tool, ...)` wraps a LangChain tool whose body does I/O so its execution runs as a `deepagents.invoke_tool` Activity. An unwrapped, non-builtin tool runs in-Workflow, and the plugin emits a best-effort warning at agent construction (the warning fires through the plugin's `create_deep_agent` seam, so code paths that bypass that seam construct without it). Deep Agents' pure built-ins (`write_todos`, the state-backed file tools) stay in-Workflow by design. This sample wraps an existing activity and an I/O tool, and builds the agent with `create_temporal_deep_agent`, which wraps the model name in a durable `TemporalModel` and scopes `activity_options` to this agent's model calls: [deepagents_plugin/react_agent/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/react_agent/workflow.py) ```py @activity.defn async def get_weather(city: str) -> str: """Return the current weather for a city.""" # A real implementation would call a weather API here; this is a stand-in. return f"It is sunny and 22C in {city}." ``` [deepagents_plugin/react_agent/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/react_agent/workflow.py) ```py @tool def web_search(query: str) -> str: """Search the web for a query and return a short result.""" # Real I/O (an HTTP call) would go here; wrapped with tool_as_activity so it # runs in an activity, not in workflow code. return f"Top result for {query!r}: Temporal makes code durable." @workflow.defn class ReactAgent: @workflow.run async def run(self, question: str) -> str: weather_tool = activity_as_tool( get_weather, start_to_close_timeout=timedelta(seconds=30), ) search_tool = tool_as_activity( web_search, start_to_close_timeout=timedelta(seconds=30), ) agent = create_temporal_deep_agent( model="anthropic:claude-sonnet-4-5", tools=[weather_tool, search_tool], system_prompt=( "You are a research assistant. Use the get_weather and " "web_search tools when they help answer the question." ), # Scopes the model-call activity options to this agent — the # recommended way to set model timeouts/retries per agent. activity_options={"start_to_close_timeout": timedelta(minutes=2)}, ) result = await agent.ainvoke( {"messages": [{"role": "user", "content": question}]} ) return result["messages"][-1].content ``` ## Durable backends A Deep Agent's built-in file tools (`write_file`, `read_file`, `ls`, and so on) delegate to a *backend*. The default `StateBackend` keeps files in agent state — pure Workflow state, replay-safe, and needs no wrapping. A `FilesystemBackend`, `LocalShellBackend`, or `StoreBackend` touches real resources, which must not happen from Workflow code. Wrap such a backend with `TemporalBackend(inner, activity_options=...)` so each file or shell operation the agent's tools invoke becomes a `deepagents.backend_op` Activity instead of running in the Workflow. The agent code is unchanged; only the backend is wrapped. [deepagents_plugin/filesystem_backend/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/filesystem_backend/workflow.py) ```py from datetime import timedelta from deepagents import create_deep_agent from deepagents.backends import FilesystemBackend from temporalio import workflow from temporalio.contrib.deepagents import TemporalBackend @workflow.defn class FilesystemAgent: @workflow.run async def run(self, root_dir: str, instruction: str) -> str: # Wrap the real-I/O backend so every file op runs in an activity. backend = TemporalBackend( # virtual_mode roots every path the agent uses under root_dir, so the # agent's file tools stay sandboxed to this working directory. FilesystemBackend(root_dir=root_dir, virtual_mode=True), activity_options={"start_to_close_timeout": timedelta(seconds=30)}, ) agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", # TemporalBackend delegates the backend protocol to the wrapped # backend at runtime, which the static type can't see through. backend=backend, # type: ignore[arg-type] system_prompt=( "You are a file-savvy assistant. Use the write_file and " "read_file tools to complete the task." ), ) result = await agent.ainvoke( {"messages": [{"role": "user", "content": instruction}]} ) return result["messages"][-1].content ``` ## Sub-agents A coordinator built with `create_deep_agent(..., subagents=[...])` delegates to its sub-agents via the built-in `task` tool. Deep Agents builds each sub-agent as a separate graph, but they inherit the parent's `model` object by default. Because the plugin makes that model object durable, every sub-agent's model call is automatically durable too — you wire the plugin once and the whole agent tree is covered, with no per-sub-agent wiring. [deepagents_plugin/subagents/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/subagents/workflow.py) ```py from deepagents import create_deep_agent from temporalio import workflow @workflow.defn class SubagentsWorkflow: @workflow.run async def run(self, question: str) -> str: agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", system_prompt=( "You are a research coordinator. Delegate deep investigation to " "the researcher sub-agent via the task tool, then synthesize a " "final answer." ), subagents=[ { "name": "researcher", "description": "Researches a topic in depth and reports findings.", "system_prompt": "You research topics thoroughly and report back.", } ], ) result = await agent.ainvoke( {"messages": [{"role": "user", "content": question}]} ) return result["messages"][-1].content ``` ## Human-in-the-loop `create_deep_agent(..., interrupt_on=...)` makes the agent pause before a guarded tool runs. With an in-Workflow `InMemorySaver` checkpointer, LangGraph does *not* raise out of `ainvoke` — it returns the current state with an `__interrupt__` entry describing the pending approval. Because the agent loop runs in the Workflow, that pause surfaces directly in Workflow code. The plugin adds no shim here; the native LangGraph resume protocol is used as-is. The recommended Temporal mapping is to expose the pending approval via a [Query](/develop/python/workflows/message-passing#queries) and resume via an [Update](/develop/python/workflows/message-passing#updates) that feeds the human's decision back with `Command(resume={"decisions": [...]})`. The `InMemorySaver` is replay-safe because its state lives in the Workflow's own memory, rehydrated by deterministic replay, and the stable Workflow ID is used as the `thread_id`. [deepagents_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/human_in_the_loop/workflow.py) ```py from datetime import timedelta from deepagents import create_deep_agent from langchain_core.runnables import RunnableConfig from langgraph.checkpoint.memory import InMemorySaver from langgraph.types import Command from temporalio import workflow from temporalio.contrib.deepagents import tool_as_activity @workflow.defn class HumanInTheLoopAgent: def __init__(self) -> None: self._pending: str | None = None self._decision: str | None = None self._resumed = False @workflow.run async def run(self, city: str) -> str: def book_trip(city: str) -> str: """Book a trip to a city (requires human approval).""" return f"Booked a trip to {city}." trip_tool = tool_as_activity( book_trip, start_to_close_timeout=timedelta(seconds=30) ) agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", tools=[trip_tool], interrupt_on={"book_trip": True}, checkpointer=InMemorySaver(), ) config = RunnableConfig(configurable={"thread_id": workflow.info().workflow_id}) result = await agent.ainvoke( {"messages": [{"role": "user", "content": f"Book a trip to {city}."}]}, config=config, ) # LangGraph returns (not raises) the pending approval under __interrupt__. pending = result.get("__interrupt__") if pending: self._pending = str(getattr(pending[0], "value", pending[0])) # Block until a client approves/rejects via the `resume` update. await workflow.wait_condition(lambda: self._resumed) # No longer paused: the query goes back to reporting None. self._pending = None result = await agent.ainvoke( Command(resume={"decisions": [{"type": self._decision}]}), config=config, ) return result["messages"][-1].content @workflow.query def pending_approval(self) -> str | None: """Return the pending approval prompt, or ``None`` if not paused.""" return self._pending @workflow.update async def resume(self, decision: str) -> None: """Resume the paused agent with ``"approve"`` or ``"reject"``.""" self._decision = decision self._resumed = True @resume.validator def validate_resume(self, decision: str) -> None: # Runs before the update is accepted, keeping invalid decisions out of # workflow history entirely. Only the decisions this workflow feeds to # `Command(resume=...)` are allowed. if decision not in ("approve", "reject"): raise ValueError('decision must be "approve" or "reject"') ``` ## Continue-as-new Long conversations bloat Workflow history until it hits Temporal's limits. `run_deep_agent(agent, input, state_snapshot=...)` snapshots state and continues into a fresh run when a turn ends with pending todos and the server recommends continuing (`workflow.info().is_continue_as_new_suggested()`, the default and recommended mode). Pass `continue_as_new_after=N` to trigger on a fixed history-event count instead. - **Carries forward:** the accumulated messages and the model/tool result cache, so an LLM or tool call completed before the continue-as-new is *not* re-run afterward. - **Does not carry forward:** anything else — the snapshot is exactly the messages plus the result cache. Pending todos only gate *whether* to continue; the todo list itself (and any in-memory checkpointer state) starts fresh in the continued run and is re-derived by the agent from the carried conversation. Your `@workflow.run` method must accept the carried state — its signature is `run(self, input, state_snapshot=None)`. On a continue-as-new, `run_deep_agent` re-invokes the method with `args=[input, snapshot]`, so `input` must be passed straight through, not re-wrapped, or the carried conversation is corrupted. [deepagents_plugin/continue_as_new/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/continue_as_new/workflow.py) ```py from typing import Any from deepagents import create_deep_agent from temporalio import workflow from temporalio.contrib.deepagents import run_deep_agent @workflow.defn class LongResearchAgent: @workflow.run async def run( self, input: dict[str, Any], state_snapshot: dict | None = None ) -> str: agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", system_prompt=( "You are a research agent. Break large tasks into todos and work " "through them until the research is complete." ), ) result = await run_deep_agent( agent, # ``input`` is the messages mapping. Pass it through unchanged: on a # continue-as-new, run_deep_agent re-invokes this method with the # carried input as its first arg, so re-wrapping it here would nest a # dict where a message is expected and corrupt the conversation. input, # No threshold: continue-as-new fires when the agent still has # pending todos and the server suggests continuing — the recommended # mode. Pass continue_as_new_after=N to use a fixed event count. state_snapshot=state_snapshot, ) return result["messages"][-1].content ``` ## Streaming Constructing the plugin with `DeepAgentsPlugin(streaming_topic="...")` flips model dispatch from `deepagents.invoke_model` to `deepagents.invoke_model_streaming`: the streaming Activity coalesces chunk batches and publishes them to a [`WorkflowStream`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/workflow_streams) topic for external subscribers, while the aggregated final message is still returned to the Workflow. The durable result is identical to the non-streaming path. Streaming is async-only, so the Workflow drives an explicit `TemporalModel.astream(...)` and hosts a `WorkflowStream` so external subscribers can attach by Workflow ID. [deepagents_plugin/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/streaming/workflow.py) ```py from langchain_core.messages import HumanMessage from temporalio import workflow from temporalio.contrib.deepagents import TemporalModel from temporalio.contrib.workflow_streams import WorkflowStream STREAMING_TOPIC = "model-chunks" @workflow.defn class StreamingWorkflow: def __init__(self) -> None: # Host the stream so the publish-Signal handler is registered before the # streaming activity (the external publisher) starts publishing. self.stream = WorkflowStream() @workflow.run async def run(self, prompt: str) -> str: model = TemporalModel(model="anthropic:claude-sonnet-4-5") parts: list[str] = [] async for chunk in model.astream([HumanMessage(content=prompt)]): parts.append(str(chunk.content)) return "".join(parts) ``` ## Compose with an observability plugin This plugin carries no tracing context of its own. For observability, compose it with an observability plugin such as [`temporalio.contrib.langsmith`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/langsmith) or `temporalio.contrib.opentelemetry`. Registration order of the two plugins does not matter; the observability plugin captures the LLM calls that `DeepAgentsPlugin` runs as Activities. The Workflow itself is an ordinary Deep Agent — the tracing comes entirely from composing the plugins on the Client. [deepagents_plugin/langsmith_tracing/workflow.py](https://github.com/temporalio/samples-python/blob/main/deepagents_plugin/langsmith_tracing/workflow.py) ```py from deepagents import create_deep_agent from temporalio import workflow @workflow.defn class TracedAgent: @workflow.run async def run(self, question: str) -> str: agent = create_deep_agent( model="anthropic:claude-sonnet-4-5", system_prompt="You are a helpful assistant.", ) result = await agent.ainvoke( {"messages": [{"role": "user", "content": question}]} ) return result["messages"][-1].content ``` For agents built directly as LangGraph graphs rather than a compiled Deep Agent, see the [LangGraph integration](/develop/python/integrations/langgraph). ## Runtime behavior and limitations - **Auto-routing is Workflow-scoped.** While a Worker built with this plugin is running, the plugin patches Deep Agents' model-resolution seam so a bare `model="provider:name"` string is auto-routed through an Activity, regardless of how you imported `create_deep_agent`. This seam is shared by the agent and every sub-agent, and it only rewrites the model when resolved *inside a Workflow*, so importing `deepagents` on a plain client or Activity Worker is unaffected, and the patched seam is restored when the Worker stops. Pass `TemporalModel("provider:name")` yourself if you would rather be explicit. - **Built-ins stay in-Workflow.** Deep Agents' pure built-in tools (`write_todos`, state-backed file tools) run in-Workflow by design and do not need wrapping. Only tools and backends that do real I/O should be moved to Activities. - **Durable checkpointers are not replay-safe.** The default in-Workflow `InMemorySaver` is rehydrated for free by deterministic replay. A durable checkpointer that does its own I/O is not replay-safe from inside a Workflow, and the plugin warns if you pass one — prefer the snapshot plus continue-as-new path above. ## Samples The [Deep Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/deepagents_plugin) demonstrate all supported patterns, including tool choice, durable backends, sub-agents, human-in-the-loop, continue-as-new, streaming, and observability composition. --- # Google ADK integration Source: https://docs.temporal.io/develop/python/integrations/google-adk Temporal's Google ADK integration lets you run [Google ADK](https://google.github.io/adk-docs/) agents inside Temporal Workflows, so an agent keeps its place across Worker restarts, deploys, and transient failures. Temporal gives your agent code [Durable Execution](/temporal#durable-execution). The Google ADK gives you the agent itself: model calls, tools, multi-agent handoffs, and MCP. The integration connects the two so that you write an ordinary ADK agent and run it as a Workflow, without managing sessions, a database, or your own retry logic. The `GoogleAdkPlugin` is what ties them together. It runs each model call and tool call as a Temporal Activity, configures the payload converter that serializes ADK objects, and makes ADK's runtime deterministic so the Workflow can replay. You add the same plugin to your Client and your Worker, and the rest of your agent code stays standard ADK. > **Pre-release** ## Prerequisites - This guide assumes you are already familiar with the Google ADK. If you aren't, refer to the [Google ADK documentation](https://google.github.io/adk-docs/) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the Temporal Development Server running if you want to test your code locally. ## Install Make sure you have the Temporal Python SDK (requires version 1.28.0 or later). Then install the `google-adk`: ```bash uv add "temporalio[google-adk]>=1.28.0" ``` If you use pip: ```bash pip install "temporalio[google-adk]>=1.28.0" ``` ## Define an agent in a Workflow Write your agent the way you normally would with the ADK. The one Temporal-specific piece is `TemporalModel`, which you pass in place of a model name. It runs each model call as an `invoke_model` Activity, so every turn is durable and shows up in the Event history. [google_adk_agents/basic/workflows/hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/workflows/hello_world_workflow.py) ```py @workflow.defn class HelloWorldAgentWorkflow: @workflow.run async def run(self, prompt: str) -> str: # TemporalModel runs each model call as an `invoke_model` activity. agent = Agent( name="hello_world_agent", model=TemporalModel("gemini-2.5-flash"), instruction="You only respond in haikus.", ) # The plugin points ADK's session-id generation at workflow.uuid4(), so # creating a session here is replay-safe. runner = InMemoryRunner(agent=agent, app_name="hello_world_app") session = await runner.session_service.create_session( app_name="hello_world_app", user_id="user" ) final_text = "" async with Aclosing( runner.run_async( user_id="user", session_id=session.id, new_message=types.Content(role="user", parts=[types.Part(text=prompt)]), ) ) as agen: async for event in agen: if event.content and event.content.parts: for part in event.content.parts: if part.text: final_text = part.text return final_text ``` Everything other than `TemporalModel` is normal ADK. You build an `Agent`, drive it with a runner, and read the events it produces. ## Add the plugin to your Worker and Client Build one `GoogleAdkPlugin` and pass the same instance to both the Client and the Worker. The Client side links the code that starts a Workflow to the Workflow itself, and the Worker side runs the agent. [google_adk_agents/basic/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/run_worker.py) ```py plugin = GoogleAdkPlugin() client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin] ) worker = Worker( client, task_queue="google-adk-agents-basic", workflows=[HelloWorldAgentWorkflow], plugins=[plugin], ) await worker.run() ``` Model calls run as Activities on the Worker, so the Worker process is the one that needs your model provider credentials. The samples use Gemini, which reads its key from the `GOOGLE_API_KEY` environment variable. ```bash export GOOGLE_API_KEY="your-api-key" python run_worker.py ``` The ADK supports other model providers as well, for example non-Gemini models through LiteLLM. Change the model name you pass to `TemporalModel` to use one. Start the Workflow the way you would start any other Temporal Workflow. Use a Client that has the plugin so the starting code is linked to the run. [google_adk_agents/basic/run_hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/basic/run_hello_world_workflow.py) ```py client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[GoogleAdkPlugin()], ) result = await client.execute_workflow( HelloWorldAgentWorkflow.run, "Tell me about recursion in programming.", id="google-adk-agents-basic-workflow-id", task_queue="google-adk-agents-basic", ) print(f"Result: {result}") ``` ## Run tools as Activities To expose a Temporal Activity as a tool the agent can call, wrap it with `activity_as_tool`. When the model calls the tool, it runs as its own Activity, so it gets its own retries and timeouts and appears in the Event history. [google_adk_agents/tools/workflows/weather_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/tools/workflows/weather_workflow.py) ```py @workflow.defn class WeatherAgentWorkflow: @workflow.run async def run(self, prompt: str) -> str: # activity_as_tool runs the tool call as a real Temporal activity, so it's # retryable and shows up in history. weather_tool = temporalio.contrib.google_adk_agents.workflow.activity_as_tool( get_weather, start_to_close_timeout=timedelta(seconds=60) ) agent = Agent( name="weather_agent", model=TemporalModel("gemini-2.5-flash"), instruction="Use the get_weather tool to answer weather questions.", tools=[weather_tool], ) ``` `get_weather` is a plain Temporal Activity. Register it on the Worker alongside the Workflow. [google_adk_agents/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/tools/run_worker.py) ```py plugin = GoogleAdkPlugin() client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin] ) worker = Worker( client, task_queue="google-adk-agents-tools", workflows=[WeatherAgentWorkflow], activities=[get_weather], plugins=[plugin], ) await worker.run() ``` ## Coordinate multiple agents The ADK's multi-agent patterns work inside a Workflow. Give each agent its own `TemporalModel`, and pass an `ActivityConfig` when you want to name its model turns or set per-agent timeouts. A coordinator delegates to its `sub_agents` using the ADK's built-in handoff. [google_adk_agents/agent_patterns/workflows/multi_agent_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/agent_patterns/workflows/multi_agent_workflow.py) ```py @workflow.defn class MultiAgentWorkflow: @workflow.run async def run(self, topic: str) -> str: session_service = InMemorySessionService() session = await session_service.create_session( app_name="multi_agent_app", user_id="user" ) # The ActivityConfig summary makes each model turn a named activity in # history. researcher = LlmAgent( name="researcher", model=TemporalModel( "gemini-2.5-flash", activity_config=ActivityConfig(summary="Researcher Agent"), ), instruction="You are a researcher. Find information about the topic.", ) writer = LlmAgent( name="writer", model=TemporalModel( "gemini-2.5-flash", activity_config=ActivityConfig(summary="Writer Agent"), ), instruction="You are a poet. Write a haiku based on the research.", ) # ADK's transfer_to_agent handoff runs durably here. coordinator = LlmAgent( name="coordinator", model=TemporalModel( "gemini-2.5-flash", activity_config=ActivityConfig( start_to_close_timeout=timedelta(seconds=30), summary="Coordinator Agent", ), ), instruction="You are a coordinator. Delegate to researcher then writer.", sub_agents=[researcher, writer], ) ``` The `summary` you set on each `ActivityConfig` becomes the Activity name in the Event history, which makes it easy to see which agent ran and when. ## Use MCP tool servers To give an agent tools from an [MCP](https://modelcontextprotocol.io/) server, use `TemporalMcpToolSet`. It runs the server's list-tools and call-tool operations as Activities because connecting to an MCP server is external I/O. You define a factory function for the toolset and register it with the plugin through a `TemporalMcpToolSetProvider`. [google_adk_agents/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/mcp/run_worker.py) ```py plugin = GoogleAdkPlugin( toolset_providers=[TemporalMcpToolSetProvider("echo", echo_toolset)] ) ``` In the Workflow, give the agent a `TemporalMcpToolSet` with the same name. The `not_in_workflow_toolset` factory lets you run the same agent locally, outside Temporal, by connecting to the MCP server directly. [google_adk_agents/mcp/workflows/echo_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/mcp/workflows/echo_workflow.py) ```py @workflow.defn class EchoMcpWorkflow: @workflow.run async def run(self, prompt: str) -> str: # TemporalMcpToolSet runs the MCP server's list-tools and call-tool # calls as activities. agent = Agent( name="echo_agent", model=TemporalModel("gemini-2.5-flash"), instruction="Use the echo tool to echo back the user's message.", tools=[TemporalMcpToolSet("echo", not_in_workflow_toolset=echo_toolset)], ) ``` ## Stream model output `TemporalModel` can stream a model's output as it is generated, using a Workflow stream from `temporalio.contrib.workflow_streams`. Give the model a `streaming_topic`, host a `WorkflowStream` on the Workflow, and the streaming model call publishes each chunk to the topic as it arrives. A Client subscribes to the topic to read chunks while the Workflow is still running. > **⚠️ Caution:** > > Streaming is an early, experimental part of this integration and is more likely to change than the rest of the API. > Refer to the [streaming sample](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/streaming) for a > complete, working example. > [google_adk_agents/streaming/workflows/streaming_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_adk_agents/streaming/workflows/streaming_workflow.py) ```py @workflow.defn class StreamingAgentWorkflow: @workflow.init def __init__(self, prompt: str) -> None: # The streaming activity publishes LlmResponse chunks to this stream as # they come back from the model. self.stream = WorkflowStream() @workflow.run async def run(self, prompt: str) -> str: # streaming_mode=SSE routes the call through the invoke_model_streaming # activity. model = TemporalModel("gemini-2.5-flash", streaming_topic="responses") agent = Agent( name="streaming_agent", model=model, instruction="You are a helpful assistant.", ) runner = InMemoryRunner(agent=agent, app_name="streaming_app") session = await runner.session_service.create_session( app_name="streaming_app", user_id="user" ) final_text = "" async for event in runner.run_async( user_id="user", session_id=session.id, new_message=types.Content(role="user", parts=[types.Part(text=prompt)]), run_config=RunConfig(streaming_mode=StreamingMode.SSE), ): if event.content and event.content.parts: for part in event.content.parts: if part.text: final_text = part.text return final_text ``` ## Samples The [Google ADK samples](https://github.com/temporalio/samples-python/tree/main/google_adk_agents) cover each pattern in a self-contained, runnable scenario: - [`basic`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/basic): a single agent with `TemporalModel` and one model call. - [`tools`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/tools): a Temporal Activity exposed as a tool with `activity_as_tool`. - [`agent_patterns`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/agent_patterns): a coordinator agent that delegates to sub-agents. - [`mcp`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/mcp): MCP tools run as Activities, with a self-contained echo server. - [`streaming`](https://github.com/temporalio/samples-python/tree/main/google_adk_agents/streaming): token streaming with `WorkflowStream`. --- # Google GenAI integration Source: https://docs.temporal.io/develop/python/integrations/google-genai Temporal's Google GenAI integration lets you call [Gemini](https://ai.google.dev/gemini-api/docs) models from inside Temporal Workflows, so a sequence of model calls keeps its place across Worker restarts, deploys, and transient failures. Temporal gives your code [Durable Execution](/temporal#durable-execution). The [Google Gen AI SDK](https://googleapis.github.io/python-genai/) gives you the model API: content generation, automatic function calling, chat sessions, structured output, files, and MCP. The integration connects the two so that you write ordinary Gemini SDK code and run it as a Workflow, without writing your own retry loop or checkpointing. `GoogleGenAIPlugin` is what ties them together. You build a `genai.Client` with your credentials on the Worker and hand it to the plugin. Inside the Workflow you construct a `TemporalAsyncClient`, which has the same shape as the SDK's async client but routes every API call through a Temporal Activity. Each call gets its own timeout, retry policy, and Event History entry, and your credentials stay on the Worker. > **Public Preview** Code snippets in this guide come from the [Google GenAI plugin samples](https://github.com/temporalio/samples-python/tree/main/google_genai). Refer to the samples for the complete code. ## Prerequisites - This guide assumes you are already familiar with the Google Gen AI SDK. If you aren't, refer to the [Gemini API documentation](https://ai.google.dev/gemini-api/docs) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the Temporal development server running if you want to test your code locally. ## Install the plugin Install the Temporal Python SDK with Google GenAI support (requires `temporalio` 1.31.0 or later): ```bash uv add "temporalio[google-genai]>=1.31.0" ``` If you use pip: ```bash pip install "temporalio[google-genai]>=1.31.0" ``` The [MCP](#use-mcp-tool-servers) path also needs the `mcp` package, which the extra does not install. ## Call a model from a Workflow Construct a `TemporalAsyncClient` in your Workflow and call it the way you would call `genai.Client.aio`. The client takes no credentials — it resolves each call to an Activity that runs on the Worker. [google_genai/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/workflow.py) ```py @workflow.defn class HelloWorldWorkflow: @workflow.run async def run(self, prompt: str) -> str: client = TemporalAsyncClient() response = await client.models.generate_content( model="gemini-2.5-flash", contents=prompt, ) return response.text or "" ``` On the Worker, build a real `genai.Client` with your credentials, wrap it in a `GoogleGenAIPlugin`, and pass the plugin to `Client.connect`. A Worker created from that Temporal Client picks up the plugin, which registers the Activity that makes the API calls and swaps in the Pydantic payload converter that serializes Gemini types. [google_genai/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/run_worker.py) ```py import asyncio import os from google import genai from temporalio.client import Client from temporalio.contrib.google_genai import GoogleGenAIPlugin from temporalio.worker import Worker from google_genai.hello_world.workflow import HelloWorldWorkflow async def main() -> None: # The real genai.Client (with credentials) lives only on the worker. genai_client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"]) plugin = GoogleGenAIPlugin(genai_client) client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin], ) worker = Worker( client, task_queue="google-genai-hello-world", workflows=[HelloWorldWorkflow], ) print("Worker started. Ctrl+C to exit.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Because API calls run as Activities, the Worker process is the one that needs credentials. The samples use the Gemini Developer API, which reads its key from the `GOOGLE_API_KEY` environment variable. ```bash export GOOGLE_API_KEY="your-api-key" uv run google_genai/hello_world/run_worker.py ``` Start the Workflow the way you start any other Temporal Workflow. The starting Client does not need the plugin. [google_genai/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/hello_world/run_workflow.py) ```py import asyncio import os from temporalio.client import Client from google_genai.hello_world.workflow import HelloWorldWorkflow async def main() -> None: client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")) result = await client.execute_workflow( HelloWorldWorkflow.run, "Write a haiku about durable execution.", id="google-genai-hello-world", task_queue="google-genai-hello-world", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Run tools as Activities The Gemini SDK's automatic function calling loop runs inside the Workflow, so you don't write the tool loop yourself. A tool can be either of the following: - A Temporal Activity wrapped with `activity_as_tool`. The model's call to it runs as its own Activity, with its own timeout and retries, and appears in the Event History. Use this for anything that does I/O or is otherwise non-deterministic. - A plain Workflow method passed directly. It runs in the Workflow with no Activity dispatch, so it must be [deterministic](/develop/python/workflows/basics#workflow-logic-requirements). This Workflow passes one of each on a single call. [google_genai/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/tools/workflow.py) ```py @workflow.defn class ToolsWorkflow: @workflow.run async def run(self, prompt: str) -> str: client = TemporalAsyncClient() response = await client.models.generate_content( model="gemini-2.5-flash", contents=prompt, config=types.GenerateContentConfig( tools=[ activity_as_tool( get_weather, activity_config=ActivityConfig( start_to_close_timeout=timedelta(seconds=30), ), ), self.recommend_thing_to_do, ], ), ) return response.text or "" async def recommend_thing_to_do(self, weather: str) -> str: """Recommend something to do given a weather description.""" if "sunny" in weather.lower(): return "Go for a hike." return "Visit a museum." ``` `activity_as_tool` keeps the wrapped function's name, docstring, and type signature, which is what the model uses to decide when to call it. Its `activity_config` must set `start_to_close_timeout` or `schedule_to_close_timeout`; there is no default, and the tool call fails without one. Register the Activity on the Worker alongside the Workflow. [google_genai/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/tools/run_worker.py) ```py worker = Worker( client, task_queue="google-genai-tools", workflows=[ToolsWorkflow], activities=[get_weather], ) ``` ## Hold a multi-turn conversation `client.chats` works inside a Workflow. The chat session keeps its history in Workflow state, and each `send_message` call runs as its own Activity, so a conversation that spans hours or days survives a Worker restart. [google_genai/chat/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/chat/workflow.py) ```py @workflow.defn class ChatWorkflow: @workflow.run async def run(self, prompts: list[str]) -> list[str]: client = TemporalAsyncClient() chat = client.chats.create(model="gemini-2.5-flash") replies: list[str] = [] for prompt in prompts: response = await chat.send_message(prompt) replies.append(response.text or "") return replies ``` To drive the turns from the outside instead of from a list, take each prompt as a [Signal](/develop/python/workflows/message-passing#signals) and wait on it with `workflow.wait_condition`. ## Return structured output The plugin installs Temporal's Pydantic payload converter, so a Pydantic model passes through Temporal payloads unchanged. Pass the model as `response_schema` and read the parsed result from `response.parsed`. [google_genai/structured_output/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/structured_output/workflow.py) ```py class Recipe(BaseModel): name: str ingredients: list[str] steps: list[str] @workflow.defn class StructuredOutputWorkflow: @workflow.run async def run(self, prompt: str) -> Recipe: client = TemporalAsyncClient() response = await client.models.generate_content( model="gemini-2.5-flash", contents=prompt, config=types.GenerateContentConfig( response_mime_type="application/json", response_schema=Recipe, ), ) recipe = response.parsed if not isinstance(recipe, Recipe): # ``parsed`` is None when the model returns malformed JSON. Fail the # workflow with an ApplicationError rather than asserting: an # assertion is a workflow task failure, which Temporal retries # forever, so the run would hang instead of failing visibly. raise ApplicationError( f"Gemini did not return a valid Recipe: {response.text!r}", non_retryable=True, ) return recipe ``` ## Use MCP tool servers To give a model tools from an [MCP](https://modelcontextprotocol.io/) server, register the server on the Worker and reference it by name in the Workflow. Connecting to an MCP server is external I/O, so the plugin holds the connection on the Worker and runs `list_tools` and `call_tool` as Activities against it. Register each server with a factory that yields a connected, initialized `mcp.ClientSession`. [google_genai/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/mcp/run_worker.py) ```py @asynccontextmanager async def echo_session() -> AsyncIterator[ClientSession]: """Yield a connected, initialized session to the stdio echo MCP server.""" params = StdioServerParameters(command=sys.executable, args=[ECHO_SERVER]) async with stdio_client(params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() yield session async def main() -> None: genai_client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"]) plugin = GoogleGenAIPlugin(genai_client, mcp_servers={"echo": echo_session}) ``` In the Workflow, pass a `TemporalMcpClientSession` with the same name in the `tools` list. Automatic function calling discovers and calls the server's tools from there. [google_genai/mcp/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/mcp/workflow.py) ```py @workflow.defn class McpWorkflow: @workflow.run async def run(self, prompt: str) -> str: client = TemporalAsyncClient() session = TemporalMcpClientSession( "echo", cache_tools=True, activity_config=ActivityConfig( start_to_close_timeout=timedelta(seconds=30), ), ) response = await client.models.generate_content( model="gemini-2.5-flash", contents=prompt, config=types.GenerateContentConfig(tools=[session]), ) return response.text or "" ``` `cache_tools=True` reuses the first `list_tools` result for the rest of the run instead of listing tools before every call. The Worker keeps each MCP connection open between uses and disconnects it after five minutes idle; change that with `mcp_connection_idle_timeout` on the plugin. Server-side MCP needs no wiring. Vertex AI's `Tool(mcp_servers=[McpServer(...)])` and the Interactions API's MCP steps run on Google's backend, so they pass through as ordinary request and response data. ## Stream model output `generate_content_stream` can forward chunks to an external subscriber while the Workflow is still running. Set `streaming_topic` on the client and host a `WorkflowStream` in the Workflow's `@workflow.init`; each chunk is published to that topic as it arrives. The Workflow's own iteration over the stream is unchanged. [google_genai/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/streaming/workflow.py) ```py @workflow.defn class StreamingWorkflow: @workflow.init def __init__(self, prompt: str) -> None: # Hosting a WorkflowStream is required when streaming_topic is set. self.stream = WorkflowStream() self._done = False @workflow.run async def run(self, prompt: str) -> str: client = TemporalAsyncClient(streaming_topic="gemini") chunks: list[str] = [] async for chunk in await client.models.generate_content_stream( model="gemini-2.5-flash", contents=prompt, ): chunks.append(chunk.text or "") # Bound the wait: if the subscriber dies without signaling, complete # anyway instead of waiting forever. try: await workflow.wait_condition(lambda: self._done, timeout=FINISH_TIMEOUT) except asyncio.TimeoutError: workflow.logger.warning( "No finish signal after %s; completing without a subscriber.", FINISH_TIMEOUT, ) return "".join(chunks) @workflow.signal def finish(self) -> None: self._done = True ``` A consumer subscribes to the topic with `WorkflowStreamClient`. The published chunks are Pydantic `GenerateContentResponse` objects, so the subscribing Client needs `pydantic_data_converter`. [google_genai/streaming/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/streaming/run_workflow.py) ```py async def consume(client: Client, workflow_id: str) -> None: """Subscribe to the "gemini" topic and print chunks as the model produces them.""" stream = WorkflowStreamClient.create(client, workflow_id) async for item in stream.subscribe( ["gemini"], from_offset=0, result_type=types.GenerateContentResponse, poll_cooldown=timedelta(milliseconds=50), ): chunk: types.GenerateContentResponse = item.data if chunk.text: print(chunk.text, end="", flush=True) if chunk.candidates and chunk.candidates[0].finish_reason: print() return async def main() -> None: # The stream publishes Pydantic GenerateContentResponse chunks, so the # consumer needs the Pydantic data converter to decode them. client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), data_converter=pydantic_data_converter, ) ``` The streaming Activity batches published chunks and flushes them every 100 milliseconds by default. Adjust that with `streaming_batch_interval`. Delivery is at-least-once per Activity attempt: if the streaming Activity retries, the model call re-runs and republishes, so subscribers should tolerate duplicates and treat the Workflow result as the source of truth. ## Upload files and reference them in a prompt `client.files` runs as Activities too, so the file is read on the Worker rather than in the Workflow. Upload it, then pass the returned handle in `contents`. [google_genai/files/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/files/workflow.py) ```py @workflow.defn class FilesWorkflow: @workflow.run async def run(self, file_path: str, prompt: str) -> str: client = TemporalAsyncClient() uploaded = await client.files.upload( file=file_path, config=types.UploadFileConfig(mime_type="text/plain"), ) contents = cast(types.ContentListUnion, [prompt, uploaded]) response = await client.models.generate_content( model="gemini-2.5-flash", contents=contents, ) return response.text or "" ``` The file path resolves on the Worker, so the Worker needs access to it. Operations that require separate Google Cloud credentials, such as `files.register_files`, use the `extra_credentials` you pass to the plugin. ## Use the Interactions API and managed agents `client.interactions` and `client.agents` are server-managed: the state lives on Google's backend and each operation runs as its own Activity. [google_genai/interactions/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/interactions/workflow.py) ```py @workflow.defn class InteractionsWorkflow: @workflow.run async def run(self, prompt: str) -> dict[str, Any]: client = TemporalAsyncClient() # create/get return either an Interaction or a streaming response; without # stream=True the result is always an Interaction. interaction: Any = await client.interactions.create( model="gemini-2.5-flash", input=prompt, ) fetched: Any = await client.interactions.get(interaction.id) await client.interactions.delete(interaction.id) return {"id": interaction.id, "status": str(fetched.status)} ``` Two limits apply to the Interactions API: - It has no automatic function calling. Declare tools as `{"type": "function", ...}` dicts and drive the tool loop yourself, running each call with `workflow.execute_activity` or an `activity_as_tool` callable. - Streamed interactions are batched. The Activity drains the server-sent event stream and the Workflow iterates the collected events. `client.webhooks` is not supported in Workflows. ## Run against Vertex AI To use Vertex AI instead of the Gemini Developer API, set `vertexai=True` on both sides. In the Workflow, pass the project and location as Workflow arguments rather than reading environment variables, which keeps the Workflow deterministic. [google_genai/vertex_ai/workflow.py](https://github.com/temporalio/samples-python/blob/main/google_genai/vertex_ai/workflow.py) ```py @workflow.defn class VertexAIWorkflow: @workflow.run async def run(self, prompt: str, project: str, location: str) -> str: client = TemporalAsyncClient( vertexai=True, project=project, location=location, ) response = await client.models.generate_content( model="gemini-2.5-flash", contents=prompt, ) return response.text or "" ``` The Worker's `genai.Client` uses Application Default Credentials instead of an API key. Run `gcloud auth application-default login`, or set `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file. [google_genai/vertex_ai/run_worker.py](https://github.com/temporalio/samples-python/blob/main/google_genai/vertex_ai/run_worker.py) ```py genai_client = genai.Client( vertexai=True, project=os.environ["GOOGLE_CLOUD_PROJECT"], location=os.environ.get("GOOGLE_CLOUD_LOCATION", "us-central1"), ) plugin = GoogleGenAIPlugin(genai_client) ``` The `vertexai` setting must match on both sides. A Workflow that sets `vertexai=True` against a Worker configured for the Gemini Developer API sends requests the backend can't serve. ## Set timeouts and retries Every API call defaults to a 60-second `start_to_close_timeout` and Temporal's default retry policy. Override that for all of a client's calls with `activity_config`: ```python from datetime import timedelta from temporalio.common import RetryPolicy from temporalio.contrib.google_genai import TemporalAsyncClient from temporalio.workflow import ActivityConfig client = TemporalAsyncClient( activity_config=ActivityConfig( start_to_close_timeout=timedelta(minutes=5), retry_policy=RetryPolicy(maximum_attempts=3), ), ) ``` `activity_as_tool` and `TemporalMcpClientSession` take their own `activity_config`, so tool calls and MCP calls can use different limits than model calls. Let Temporal own retries. The plugin rejects a `genai.Client` configured with `http_options.retry_options`, because an SDK-internal retry loop hides its attempts inside a single Activity and compounds with the Temporal retry policy. ## Samples The [Google GenAI plugin samples](https://github.com/temporalio/samples-python/tree/main/google_genai) cover each pattern as a self-contained, runnable scenario: - [`hello_world`](https://github.com/temporalio/samples-python/tree/main/google_genai/hello_world): one `generate_content` call. - [`tools`](https://github.com/temporalio/samples-python/tree/main/google_genai/tools): an Activity tool and a Workflow-method tool on the same call. - [`streaming`](https://github.com/temporalio/samples-python/tree/main/google_genai/streaming): chunks forwarded to an external subscriber with `WorkflowStream`. - [`chat`](https://github.com/temporalio/samples-python/tree/main/google_genai/chat): a multi-turn conversation with `client.chats`. - [`structured_output`](https://github.com/temporalio/samples-python/tree/main/google_genai/structured_output): typed JSON output through a Pydantic model. - [`mcp`](https://github.com/temporalio/samples-python/tree/main/google_genai/mcp): MCP tools run as Activities, with a self-contained echo server. - [`files`](https://github.com/temporalio/samples-python/tree/main/google_genai/files): a file uploaded with `client.files` and referenced in a prompt. - [`interactions`](https://github.com/temporalio/samples-python/tree/main/google_genai/interactions): server-managed conversations with `client.interactions`. - [`agents`](https://github.com/temporalio/samples-python/tree/main/google_genai/agents): managed agent create, get, list, and delete with `client.agents`. - [`vertex_ai`](https://github.com/temporalio/samples-python/tree/main/google_genai/vertex_ai): the same hello-world flow against Vertex AI. --- # LangGraph integration Source: https://docs.temporal.io/develop/python/integrations/langgraph Temporal's integration with [LangGraph](https://www.langchain.com/langgraph) gives your LangGraph AI agent workflows durable execution, automatic retries, and timeouts via the Temporal platform. The plugin supports both the LangGraph **Graph API** (`StateGraph` with nodes and edges) and the **Functional API** (`@entrypoint` / `@task` decorators). Each graph node and task must specify whether it runs as a Temporal Activity or directly inside the Workflow — Activity nodes get configurable timeouts and retry policies, while Workflow nodes run inline and must be deterministic. > **Public Preview** Code snippets in this guide are taken from the [LangGraph plugin samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin). Refer to the samples for the complete code. ## Prerequisites - This guide assumes you are already familiar with LangGraph. If you aren't, refer to the [LangGraph documentation](https://langchain-ai.github.io/langgraph/) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the Temporal development server running if you want to test your code locally. ## Install the plugin Install the Temporal Python SDK with LangGraph support (requires `temporalio` 1.27.0 or later): ```bash uv add "temporalio[langgraph]" ``` or with pip: ```bash pip install "temporalio[langgraph]" ``` > **📝 Note:** > > Python 3.11 or newer is required for the Functional API (`@entrypoint` / `@task`), for `interrupt()`, and for streaming > from a node running in the Workflow. On older Python versions the plugin loads but emits a warning, and those features > will not work because LangGraph relies on `contextvars` propagation through `asyncio.create_task()`, which is only > available from Python 3.11 onward. > ## Graph API The Graph API uses `StateGraph` to define nodes and edges declaratively. ### Define a graph and Workflow Build a `StateGraph`, then retrieve it inside your Workflow with the `graph()` helper: ```python from datetime import timedelta from langgraph.graph import START, StateGraph from temporalio import workflow from temporalio.contrib.langgraph import graph async def process_query(query: str) -> str: """Process a query and return a response.""" return f"Processed: {query}" def build_graph() -> StateGraph: """Construct a single-node graph.""" g = StateGraph(str) g.add_node( "process_query", process_query, metadata={ "execute_in": "activity", "start_to_close_timeout": timedelta(seconds=10), }, ) g.add_edge(START, "process_query") return g @workflow.defn class HelloWorldWorkflow: @workflow.run async def run(self, query: str) -> str: return await graph("hello-world").compile().ainvoke(query) ``` ### Configure the Worker Create a `LangGraphPlugin` with your graphs and pass it to the Worker: ```python import asyncio from temporalio.client import Client from temporalio.contrib.langgraph import LangGraphPlugin from temporalio.worker import Worker async def main() -> None: client = await Client.connect("localhost:7233") plugin = LangGraphPlugin(graphs={"hello-world": build_graph()}) worker = Worker( client, task_queue="langgraph-hello-world", workflows=[HelloWorldWorkflow], plugins=[plugin], ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ### Set Activity options Pass Activity options as node `metadata` when calling `add_node`. Every node must include `"execute_in"` set to either `"activity"` or `"workflow"`; the plugin raises an error if it's missing. ```python from datetime import timedelta from temporalio.common import RetryPolicy g = StateGraph(str) g.add_node( "my_node", my_node, metadata={ "execute_in": "activity", "start_to_close_timeout": timedelta(seconds=30), "retry_policy": RetryPolicy(maximum_attempts=3), }, ) ``` > **⚠️ Warning:** > > Don't pass a LangGraph `retry_policy=` to `add_node` (or `@task(retry_policy=...)`) — the plugin raises an error if you > do. Use Temporal's `RetryPolicy` via the node's `metadata` (Graph API) or `activity_options` (Functional API) instead. > ### Shared defaults To apply the same Activity options across every node and task, pass `default_activity_options` to `LangGraphPlugin`. Per-node `metadata` (Graph API) and per-task `activity_options` (Functional API) override these defaults key by key: ```python plugin = LangGraphPlugin( graphs={"my-graph": g}, default_activity_options={ "start_to_close_timeout": timedelta(seconds=30), "retry_policy": RetryPolicy(maximum_attempts=3), }, ) ``` To mitigate potential determinism bugs, `execute_in` cannot be set in `default_activity_options` — you must set it on each node or task individually. See [Activity vs. Workflow execution](#activity-vs-workflow-execution). ## Functional API The Functional API uses `@entrypoint` and `@task` decorators, letting you orchestrate tasks with native Python control flow (`while`, `if/else`, `for`) rather than declaring nodes and edges. ### Define tasks and a Workflow ```python from datetime import timedelta from langgraph.func import entrypoint as lg_entrypoint from langgraph.func import task from temporalio import workflow from temporalio.contrib.langgraph import entrypoint @task def agent_think(query: str, history: list[str]) -> dict: """Decide the next action based on query and tool history.""" tool_results = [h for h in history if h.startswith("[Tool]")] if len(tool_results) < 2: return {"action": "tool", "tool_name": "search", "tool_input": query} return {"action": "final", "answer": f"Found: {'; '.join(tool_results)}"} @task def execute_tool(tool_name: str, tool_input: str) -> str: """Execute a tool by name.""" return f"[Tool] Result for {tool_name}({tool_input})" @lg_entrypoint() async def react_agent(query: str) -> dict: """ReAct agent loop: think -> act -> observe -> repeat.""" history: list[str] = [] while True: decision = await agent_think(query, history) if decision["action"] == "final": return {"answer": decision["answer"], "steps": len(history)} result = await execute_tool(decision["tool_name"], decision["tool_input"]) history.append(result) all_tasks = [agent_think, execute_tool] activity_options = { t.func.__name__: { "execute_in": "activity", "start_to_close_timeout": timedelta(seconds=30), } for t in all_tasks } @workflow.defn class ReactAgentWorkflow: @workflow.run async def run(self, query: str) -> dict: return await entrypoint("react-agent").ainvoke(query) ``` ### Configure the Worker with the Functional API ```python from temporalio.contrib.langgraph import LangGraphPlugin plugin = LangGraphPlugin( entrypoints={"react-agent": react_agent}, tasks=all_tasks, activity_options=activity_options, ) worker = Worker( client, task_queue="langgraph-react-agent", workflows=[ReactAgentWorkflow], plugins=[plugin], ) ``` ## Checkpointer If your LangGraph code requires a checkpointer (for example, if you're using interrupts), use `InMemorySaver`. Temporal handles durability, so third-party checkpointers (like PostgreSQL or Redis) are not needed. ```python import langgraph.checkpoint.memory g = graph("my-graph").compile( checkpointer=langgraph.checkpoint.memory.InMemorySaver(), ) ``` ## Runtime context LangGraph's run-scoped context (`context_schema`) is reconstructed on the Activity side, so nodes and tasks can read from `runtime.context`: ```python from langgraph.runtime import Runtime from typing_extensions import TypedDict from temporalio.contrib.langgraph import graph class Context(TypedDict): user_id: str async def my_node(state: State, runtime: Runtime[Context]) -> dict: return {"user": runtime.context["user_id"]} # In the Workflow: g = graph("my-graph").compile() await g.ainvoke({...}, context=Context(user_id="alice")) ``` Your `context` object must be serializable by the configured Temporal payload converter, since it crosses the Activity boundary. ## Continue-as-new Long-running graphs can hit Temporal's per-Event history size limit. Use Temporal's [continue-as-new](/develop/python/workflows/continue-as-new) to start a fresh execution while preserving the results of nodes and tasks that have already completed. The `cache()` helper returns the current task-result cache as a serializable dict. Pass it to `graph(name, cache=...)` or `entrypoint(name, cache=...)` in the new run to skip re-executing nodes that already produced a result. ```python from temporalio import workflow from temporalio.contrib.langgraph import cache, graph @workflow.defn class LongRunningWorkflow: @workflow.run async def run(self, state: State, prior_cache: dict | None = None) -> State: g = graph("my-graph", cache=prior_cache).compile() # ... run some steps, then continue-as-new before history grows too large ... workflow.continue_as_new(args=[state, cache()]) ``` ## Streaming To stream intermediate values (such as LLM tokens or progress updates) out of a running graph, set `streaming_topic` on `LangGraphPlugin`. Calls to LangGraph's `get_stream_writer()` inside a node then publish to the named topic on the Workflow's [`WorkflowStream`](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/workflow_streams), and external subscribers consume the stream with `WorkflowStreamClient`. Activity nodes publish via a batched Temporal signal, controlled by `streaming_batch_interval` (default 100ms). Workflow nodes publish synchronously to the in-Workflow stream, with no signal. When `streaming_topic` is set, your Workflow **must** construct a `WorkflowStream()` in its `@workflow.init` (its `__init__`); otherwise the plugin raises an error. ```python from datetime import timedelta from langgraph.config import get_stream_writer from langgraph.graph import START, StateGraph from typing_extensions import TypedDict from temporalio import workflow from temporalio.contrib.langgraph import LangGraphPlugin, graph from temporalio.contrib.workflow_streams import WorkflowStream, WorkflowStreamClient class State(TypedDict): value: str async def token_node(state: State) -> dict[str, str]: writer = get_stream_writer() for token in ["hello", " ", "world"]: writer({"token": token}) writer({"done": True}) return {"value": "hello world"} @workflow.defn class StreamingWorkflow: def __init__(self) -> None: # Required when streaming_topic is set on the plugin. _ = WorkflowStream() self.app = graph("streaming").compile() @workflow.run async def run(self) -> str: result = await self.app.ainvoke({"value": ""}) return result["value"] ``` Configure the plugin with `streaming_topic`: ```python g = StateGraph(State) g.add_node("token_node", token_node, metadata={"execute_in": "activity"}) g.add_edge(START, "token_node") plugin = LangGraphPlugin( graphs={"streaming": g}, default_activity_options={"start_to_close_timeout": timedelta(seconds=10)}, streaming_topic="tokens", ) ``` An external client subscribes to the topic to consume items as they're published: ```python handle = await client.start_workflow( StreamingWorkflow.run, id="streaming-wf", task_queue="streaming-tq" ) ws_client = WorkflowStreamClient.create(client, handle.id) async for item in ws_client.topic("tokens", type=dict).subscribe(from_offset=0): print(item.data) if item.data.get("done"): break print(await handle.result()) ``` > **📝 Note:** > > Streaming from a node running in the Workflow (`execute_in: "workflow"`) requires Python > 3.11 or newer, because LangGraph relies on `contextvars` propagation through > `asyncio.create_task()`. See the [installation note](#install-the-plugin) above. > ### What's covered, and what isn't `streaming_topic` wires up exactly one LangGraph stream mode: `stream_mode="custom"` — the values written through `get_stream_writer()`. The other modes (`"messages"`, `"values"`, `"updates"`, and `"debug"`) are **not** captured by `streaming_topic`, because they aren't produced by node-side writers; LangGraph's orchestrator emits them as it walks the graph. To stream one of those modes, bridge `astream()` in the Workflow and republish each yielded chunk to a `WorkflowStream` topic yourself: ```python @workflow.defn class AstreamBridge: def __init__(self) -> None: self.stream = WorkflowStream() self.app = graph("g").compile() @workflow.run async def run(self) -> None: topic = self.stream.topic("astream") async for chunk in self.app.astream({...}, stream_mode="messages"): topic.publish(chunk) topic.publish({"done": True}) ``` ### Retry semantics Streaming has **at-least-once** delivery per Activity attempt. When an Activity-wrapped node retries (transient failure, worker crash, and so on), the node function re-runs from scratch and re-publishes its writes — earlier publishes from the failed attempt are not rolled back. Subscribers should be ready to see duplicates and recover idempotently: dedupe on a sequence ID you include in each chunk, or treat the stream as advisory and rely on the Workflow's final result for state. For a complete working example, see the [streaming sample](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin/graph_api/streaming). ## Tracing For tracing your LangGraph Workflows and Activities, we recommend the [Temporal LangSmith plugin](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/langsmith). It composes with `LangGraphPlugin` — pass both plugins to your Worker. ## Stores are not supported LangGraph's `Store` (for example, `InMemoryStore` passed via `graph.compile(store=...)` or `@entrypoint(store=...)`) isn't accessible inside Activity-wrapped nodes: the Store holds live state that can't cross the Activity boundary, and Activities may run on a different worker than the Workflow. If you pass a store, the plugin logs a warning on first use and `runtime.store` is `None` inside nodes. Use Workflow state for per-run memory, or an external database (Postgres, Redis, etc.) configured on each worker if you need shared memory across runs. ## Activity vs. Workflow execution Every graph node and `@task` must specify `execute_in` — set it to `"activity"` to run as a Temporal Activity, or `"workflow"` to run directly inside the Workflow. The plugin raises an error if you forget to set it. `execute_in` must be set per node or task; it cannot be set in `default_activity_options`. Understanding when to use each mode is important for correctness and durability. ### When to use an Activity Use `execute_in: "activity"` when a node does any of the following: - **Makes network calls** — LLM calls, HTTP requests, database queries, or any I/O. Activities can do I/O; Workflows cannot. - **Has non-deterministic behavior** — anything that can return different results on re-execution (random numbers, current time, external data). Workflows must be deterministic. - **Is long-running or may fail** — Activities get configurable timeouts, automatic retries, and heartbeating. If an LLM call times out or a service is unavailable, Temporal retries the Activity without re-running the entire Workflow. - **Calls `interrupt()`** — LangGraph's `interrupt()` is supported in Activity nodes. The plugin serializes the interrupt and propagates it back to the Workflow for human-in-the-loop patterns. ### When to run in the Workflow Use `execute_in: "workflow"` when a node: - **Orchestrates other graphs** — a node that calls `graph("child").compile().ainvoke(state)` to dispatch to a subgraph. The subgraph's own nodes still run as Activities, but the orchestration logic runs in the Workflow. - **Performs pure state transformations** — deterministic data reshaping, merging, or filtering with no I/O. - **Is a lightweight routing step** — when a node's only job is to decide what happens next and you want to avoid the overhead of an Activity round-trip. > **⚠️ Warning:** > > Workflow code must be [deterministic](/develop/python/workflows/basics#workflow-logic-requirements). A node running in > the Workflow **must not** make network calls, use `random`, read the system clock, or do file I/O. Violating this causes > non-determinism errors on replay. > ### Where LangGraph primitives run Not all LangGraph primitives are node functions. Some run in the Workflow context regardless of the `execute_in` setting: | Primitive | Runs in | Notes | | --- | --- | --- | | Node functions | Activity or Workflow | Controlled by `execute_in` in node `metadata` (required) | | `@task` functions | Activity or Workflow | Controlled by `execute_in` in `activity_options` (required) | | Conditional edge functions (`add_conditional_edges`) | Workflow | Always runs in the Workflow. Must be **deterministic** and **async** (sync functions trigger `run_in_executor`, which is not allowed in the Temporal sandbox). | | `interrupt()` | Activity | Call `interrupt()` inside Activity nodes. The plugin serializes the interrupt and propagates it to the Workflow. | | `Command(resume=...)` | Workflow | Used from Workflow code to resume after an interrupt. | | `InMemorySaver` checkpointer | Workflow | Runs in-process. Temporal handles durability — external checkpointers are not needed. | > **💡 Tip:** > > Conditional edge functions like `should_continue` must be `async def`, not plain `def`. Synchronous functions cause > LangGraph to use `run_in_executor`, which is not supported inside Temporal's Workflow sandbox. > > ```python > # ✅ Correct: async conditional edge function > async def should_continue(state: AgentState) -> str: > if state["messages"][-1].startswith("[Agent]") and "Calling" in state["messages"][-1]: > return "tools" > return END > > g.add_conditional_edges("agent", should_continue) > ``` > ### Syntax ```python # Graph API g.add_node("my_node", my_node, metadata={"execute_in": "workflow"}) # Functional API plugin = LangGraphPlugin( tasks=[my_task], activity_options={"my_task": {"execute_in": "workflow"}}, ) ``` ### Example: subgraph orchestration A common pattern is a parent node that runs in the Workflow and dispatches to a child graph whose nodes run as Activities: ```python async def parent_node(state: State) -> dict[str, str]: return await graph("child").compile().ainvoke(state) parent = StateGraph(State) parent.add_node("parent_node", parent_node, metadata={"execute_in": "workflow"}) parent.add_edge(START, "parent_node") plugin = LangGraphPlugin(graphs={"parent": parent, "child": child}) ``` ## Human-in-the-loop LangGraph's `interrupt()` works with Temporal signals and queries to support human-in-the-loop patterns: 1. A graph node calls `interrupt(draft)`, pausing execution. 2. The Workflow exposes the pending draft via a Temporal query. 3. An external process (UI, CLI) queries the draft and sends approval via a Temporal signal. 4. The graph resumes — `interrupt()` returns the signal value and the node completes. See the [human-in-the-loop samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin/graph_api/human_in_the_loop) for complete working examples using both Graph and Functional APIs. ## Samples The [LangGraph plugin samples](https://github.com/temporalio/samples-python/tree/main/langgraph_plugin) demonstrate all supported patterns across both APIs. --- # LangSmith integration Source: https://docs.temporal.io/develop/python/integrations/langsmith Temporal's LangSmith integration lets you trace AI agent Workflows in [LangSmith](https://smith.langchain.com/) alongside every LLM call, tool execution, and Temporal operation. Temporal gives your agent code [durable execution](/temporal#durable-execution). LangSmith adds the observability side, so you can inspect LLM inputs and outputs, follow a request from the Client through to the model, and compare runs over time. The `LangSmithPlugin` is what connects the two. It propagates trace context across Temporal boundaries so that runs started on the Client nest correctly under Workflow and Activity runs on the Worker. It can also create LangSmith runs for Temporal operations themselves: Workflow executions, Activity executions, Signals, Updates, and Queries. > **Public Preview** All code snippets in this guide are taken from the [LangSmith tracing sample](https://github.com/temporalio/samples-python/tree/main/langsmith_tracing). Refer to the sample for complete code. ## Prerequisites - This guide assumes you are already familiar with LangSmith. If you aren't, refer to the [LangSmith documentation](https://docs.smith.langchain.com/) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. When you're done, leave the Temporal Development Server running if you want to test your code locally. ## Configure Workers to use LangSmith Workers execute the code that defines your Workflows and Activities. To trace Workflow and Activity execution in LangSmith, add the `LangSmithPlugin` to your Worker. Follow the steps below to configure your Worker. 1. Install the Temporal Python SDK with the LangSmith extra. ```bash uv add "temporalio[langsmith]>=1.26.0" ``` 2. Add the `LangSmithPlugin` to your Worker. Set `project_name` to the LangSmith project where you want traces to appear. ```python from temporalio.contrib.langsmith import LangSmithPlugin from temporalio.worker import Worker worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity], plugins=[LangSmithPlugin(project_name="my-project")], ) ``` 3. Run the Worker. Ensure the Worker process has access to your LangSmith API key via the `LANGSMITH_API_KEY` environment variable, and enable tracing with `LANGCHAIN_TRACING_V2`. ```bash export LANGSMITH_API_KEY="your-api-key" export LANGCHAIN_TRACING_V2=true python worker.py ``` ## Configure Clients to use LangSmith Add the plugin to any Temporal Client you use on the Client side (typically a starter or API that calls into your Workflows) so that client-side operations like starting a Workflow or sending an Update get linked to the Workflows they trigger. ```python from temporalio.client import Client from temporalio.contrib.langsmith import LangSmithPlugin client = await Client.connect( "localhost:7233", plugins=[LangSmithPlugin(project_name="my-project")], ) ``` > **💡 Tip:** > > Use the same `project_name` on both the Worker and the Client so their traces land in the same LangSmith project. > > **📝 Note:** > > `@traceable` functions on the Client side run outside the plugin's interceptor scope, so they don't pick up > `project_name` from the plugin. If you have a client-side `@traceable` that wraps a call into your Workflow, pass > `project_name` to it explicitly so it lands in the same LangSmith project as the rest of the trace. > ## Trace Activities Any non-deterministic work in a Temporal Workflow (LLM calls, tool executions, database queries, external API calls, and so on) must run inside an Activity. That makes Activities an important place to add LangSmith runs. When you decorate an Activity function with `@traceable`, the run shows up in LangSmith nested under the Workflow that scheduled it. ```python from dataclasses import dataclass from langsmith import traceable from temporalio import activity @traceable(name="Fetch Weather", run_type="tool") @activity.defn async def fetch_weather(city: str) -> str: # Call an external weather API here. ... ``` You can combine `@traceable` with provider-specific LangSmith wrappers to capture more detail. For OpenAI, for example, `wrap_openai` patches the client so that every API call creates its own child run with the model name, prompt, completion, token counts, and latency. You can access this by wrapping the client: ```python from langsmith import traceable from langsmith.wrappers import wrap_openai from openai import AsyncOpenAI from temporalio import activity @dataclass class OpenAIRequest: model: str input: str # wrap_openai patches the client so that every API call adds a ChatOpenAI run under the @traceable. # Set max_retries=0 and use Temporal's Activity retry policy instead. @traceable(name="Call OpenAI", run_type="llm") @activity.defn async def call_openai(request: OpenAIRequest) -> str: client = wrap_openai(AsyncOpenAI(max_retries=0)) response = await client.responses.create( model=request.model, input=request.input, ) return response.output_text ``` LangSmith ships similar wrappers for [Anthropic](https://docs.smith.langchain.com/observability/how-to/integrations#anthropic) and other providers; refer to the LangSmith documentation for the full list. ## Add custom runs with @traceable Decorate functions with `@traceable` to create named runs for your business logic. You control the run name, tags, metadata, and `run_type` (`chain`, `llm`, `tool`, `retriever`). Put `@traceable` on Activities and on private helper methods within your Workflow class that get called from Workflow code. For example: ```python from langsmith import traceable from temporalio import workflow @workflow.defn class ChatbotWorkflow: # Private helper methods can be decorated directly. @traceable(name="Save Note", run_type="tool") def _save_note(self, name: str, content: str) -> str: ... ``` > **⚠️ Warning:** > > Do not put `@traceable` directly on any `@workflow` method (for example, `@workflow.run`, `@workflow.signal`, > `@workflow.update`, `@workflow.query`). Doing so can produce duplicate or orphaned (unknown parent) runs in LangSmith. > If you want to trace the body of one of these methods, move the logic into an inner function and decorate that: > > ```python > @workflow.defn > class MyWorkflow: > @workflow.run > async def run(self, prompt: str) -> str: > # Option 1: Use the @traceable decorator > @traceable(name=f"Ask: {prompt[:60]}", run_type="chain") > async def _run() -> str: > ... > return await _run() > > @workflow.update > async def message_from_user(self, message: str) -> str: > async def _handle_message(self, message: str) -> str: > ... > # Option 2: Use the traceable() function > return await traceable( > name=f"Update: {message[:60]}", > run_type="chain", > )(self._handle_message)(message) > ``` > ## Include Temporal operations as runs By default, `LangSmithPlugin(add_temporal_runs=False)` only propagates LangSmith context so that `@traceable` and `wrap_openai` calls nest correctly. The plugin does not create its own runs. Set `add_temporal_runs=True` if you want runs for the Temporal operations themselves: Workflow executions, Activity executions, Signals, Updates, Queries, and Child Workflows. ```python plugin = LangSmithPlugin( project_name="my-project", add_temporal_runs=True, ) ``` With this on, your LangSmith traces include runs like `StartWorkflow:MyWorkflow`, `RunWorkflow:MyWorkflow`, `StartActivity:call_openai`, and `RunActivity:call_openai`. `Start*` and `Run*` pairs appear as siblings: the `Start*` run is emitted by the side scheduling the operation (for example, the Client), and the `Run*` run is emitted by the side executing it (for example, the Worker). ## Trace hierarchy example With the plugin configured on both Client and Worker, and `add_temporal_runs=True`, a trace for a simple LLM call looks like this: ``` Run Agent (@traceable, client-side) ├── StartWorkflow:MyWorkflow (automatic, LangSmithPlugin) └── RunWorkflow:MyWorkflow (automatic, LangSmithPlugin) └── Ask: What is Temporal? (@traceable, Workflow) ├── StartActivity:call_openai (automatic, LangSmithPlugin) └── RunActivity:call_openai (automatic, LangSmithPlugin) └── Call OpenAI (@traceable, Activity) └── ChatOpenAI (automatic via wrap_openai) ``` Without `add_temporal_runs` (the default), only the `@traceable` and `wrap_openai` runs appear. Context still propagates, so they nest correctly under the client-side run: ``` Run Agent (@traceable, client-side) └── Ask: What is Temporal? (@traceable, Workflow-side) └── Call OpenAI (@traceable, Activity-side) └── ChatOpenAI (automatic via wrap_openai) ``` ## Example sample The [LangSmith tracing sample](https://github.com/temporalio/samples-python/tree/main/langsmith_tracing) puts these patterns together in two working examples: - **`basic/`**: a one-shot Workflow that sends a prompt to OpenAI and returns the response. - **`chatbot/`**: a long-running conversational Workflow with tool calls (save and read notes), Update handlers, and dynamic trace names per message. Each example shows the `LangSmithPlugin` configuration, `@traceable` runs on the Client, Workflow, and Activity, and expected trace output for both `add_temporal_runs=False` and `add_temporal_runs=True`. --- # OpenAI Agents SDK integration Source: https://docs.temporal.io/develop/python/integrations/openai-agents > Run OpenAI Agents SDK agents as durable Temporal Workflows in Python, with model calls executed as Activities. Temporal's integration with the [OpenAI Agents SDK for Python](https://openai.github.io/openai-agents-python/) lets you run agents as Temporal Workflows. Agent orchestration—the agent loop, tool selection, and handoffs—runs inside the Workflow, while model calls run as [Activities](/glossary#activity). Like with other types of API calls, in a [Temporal Application](/glossary#temporal-application), you make LLM calls in your Activities. This integration handles that for you: model calls are executed as Activities, so they retry durably and are not repeated during Workflow replay. Your agents survive Worker restarts and can run for extended periods without losing state. ## Prerequisites - This guide assumes you are already familiar with the OpenAI Agents SDK. If you aren't, refer to the [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-python/) for more details. - If you are new to Temporal, we recommend you read the [Understanding Temporal](/evaluate/understanding-temporal) document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics of Temporal. - Ensure you have set up your local development environment by following the [Set up your local with the Python SDK](/develop/python/set-up-your-local-python) guide. When you are done, leave the Temporal Development Server running if you want to test your code locally. ## Install ```bash uv add "temporalio[openai-agents]" ``` The extra pulls in `openai-agents` and `mcp` alongside the Temporal SDK. Two import paths cover most applications. `temporalio.contrib.openai_agents` holds what you configure on the Worker and Client—`OpenAIAgentsPlugin`, `ModelActivityParameters`, and the MCP and sandbox providers. `temporalio.contrib.openai_agents.workflow` holds what you call from inside a Workflow, such as `activity_as_tool` and the MCP server handles. ## Run your first durable agent A Temporal-backed agent needs three pieces: a Workflow that runs the agent, a Worker configured with the integration plugin, and a Client configured with the same plugin. ### Write the Workflow Inside the Workflow, write ordinary OpenAI Agents SDK code. The plugin redirects `Runner.run` so that each model call becomes an Activity—there is no Temporal-specific runner to learn. [openai_agents/basic/workflows/hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/workflows/hello_world_workflow.py) ```py from agents import Agent, Runner from temporalio import workflow @workflow.defn class HelloWorldAgent: @workflow.run async def run(self, prompt: str) -> str: agent = Agent( name="Assistant", instructions="You only respond in haikus.", ) result = await Runner.run(agent, input=prompt) return result.final_output ``` ### Configure the Worker Register `OpenAIAgentsPlugin` on the Client. The plugin registers the model Activity, configures the Pydantic data converter, propagates tracing context, and registers any configured MCP server or sandbox providers. [openai_agents/basic/run_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/run_worker.py) ```py client = await Client.connect( "localhost:7233", plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=30) ) ), ], ) ``` Workers built from that Client pick up the plugin automatically, so all a Worker has to do is register the Workflow. `ModelActivityParameters` controls how the model Activity is scheduled. Alongside `start_to_close_timeout`, which defaults to 60 seconds, it accepts `schedule_to_close_timeout`, `schedule_to_start_timeout`, `heartbeat_timeout`, `retry_policy`, `cancellation_type`, `versioning_intent`, `task_queue`, `priority`, `summary_override`, `use_local_activity`, `streaming_topic`, and `streaming_batch_interval`. You must ensure the Worker process has access to your model-provider credentials. Most provider SDKs read credentials from environment variables. ### Start the Workflow Attach the same plugin to the Client that starts the Workflow, so payloads are converted the same way on both sides. [openai_agents/basic/run_hello_world_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/run_hello_world_workflow.py) ```py client = await Client.connect( "localhost:7233", plugins=[ OpenAIAgentsPlugin(), ], ) # Execute a workflow result = await client.execute_workflow( HelloWorldAgent.run, "Tell me about recursion in programming.", id="my-workflow-id", task_queue="openai-agents-basic-task-queue", ) print(f"Result: {result}") ``` ## Tools Where a tool runs depends on how you define it. | Tool | Runs in | Use for | | :---------------------------------- | :--------------- | :--------------------------------------------------------- | | `activity_as_tool()` | Temporal Activity | External I/O and other non-deterministic work | | `FunctionTool` / `@function_tool` | Workflow | Deterministic, Workflow-safe computation | | OpenAI-hosted tool | Model provider | Provider-hosted features run as part of the model call | Model calls are always routed through Activities. Tools are not: a `@function_tool` runs in the Workflow unless you back it with an Activity, so any tool that performs I/O needs `activity_as_tool()` or a Nexus Operation. ### Activity-backed tools Use `activity_as_tool` for HTTP calls, database access, file system work, or other I/O. Write an ordinary Temporal Activity: [openai_agents/basic/activities/get_weather_activity.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/activities/get_weather_activity.py) ```py from dataclasses import dataclass from temporalio import activity @dataclass class Weather: city: str temperature_range: str conditions: str @activity.defn async def get_weather(city: str) -> Weather: """ Get the weather for a given city. """ return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.") ``` Then pass it through `activity_as_tool` when you build the agent: [openai_agents/basic/workflows/tools_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/basic/workflows/tools_workflow.py) ```py @workflow.defn class ToolsWorkflow: @workflow.run async def run(self, question: str) -> str: agent = Agent( name="Hello world", instructions="You are a helpful agent.", tools=[ temporal_agents.workflow.activity_as_tool( get_weather, start_to_close_timeout=timedelta(seconds=10) ) ], ) result = await Runner.run(agent, input=question) return result.final_output ``` `activity_as_tool` controls how the agent invokes the Activity; it does not register the Activity with the Worker. Pass the Activity function to the Worker's `activities` argument as well. Because the Activity may not run in the same process as the Workflow, an Activity-backed tool receives a copy of the agent context and cannot mutate it. A tool that runs in the Workflow can. ### Inline and hosted tools For deterministic computation, use the standard `@function_tool` decorator and call it directly from the Workflow. Do not perform network, database, or file system I/O from these tools—use `activity_as_tool` instead. Hosted tools such as `WebSearchTool`, `FileSearchTool`, `CodeInterpreterTool`, and `ImageGenerationTool` are executed by the model provider during the model Activity, so they need no extra wiring: [openai_agents/tools/workflows/web_search_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/tools/workflows/web_search_workflow.py) ```py @workflow.defn class WebSearchWorkflow: @workflow.run async def run(self, question: str, user_city: str = "New York") -> str: agent = Agent( name="Web searcher", instructions="You are a helpful agent.", tools=[ WebSearchTool(user_location={"type": "approximate", "city": user_city}) ], ) result = await Runner.run(agent, question) return result.final_output ``` `LocalShellTool` and `ComputerTool` are not supported, because they assume a single long-lived local process. ### Nexus operation tools Use `nexus_operation_as_tool` to expose a [Nexus](/nexus) Operation as an agent tool. The Workflow starts the Operation through a Nexus client and feeds the result back to the agent, which lets an agent call across a Namespace boundary: ```python from temporalio.contrib.openai_agents.workflow import nexus_operation_as_tool weather_tool = nexus_operation_as_tool( WeatherService.get_weather, service=WeatherService, endpoint="weather-endpoint", ) ``` ### Nested agent tools `Agent.as_tool()` from the OpenAI Agents SDK works unchanged. The nested agent's model calls become Activities like any others, so a multi-agent run stays durable throughout: [openai_agents/agent_patterns/workflows/agents_as_tools_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/agent_patterns/workflows/agents_as_tools_workflow.py) ```py def orchestrator_agent() -> Agent: spanish_agent = Agent( name="spanish_agent", instructions="You translate the user's message to Spanish", handoff_description="An english to spanish translator", ) french_agent = Agent( name="french_agent", instructions="You translate the user's message to French", handoff_description="An english to french translator", ) italian_agent = Agent( name="italian_agent", instructions="You translate the user's message to Italian", handoff_description="An english to italian translator", ) orchestrator_agent = Agent( name="orchestrator_agent", instructions=( "You are a translation agent. You use the tools given to you to translate." "If asked for multiple translations, you call the relevant tools in order." "You never translate on your own, you always use the provided tools." ), tools=[ spanish_agent.as_tool( tool_name="translate_to_spanish", tool_description="Translate the user's message to Spanish", ), french_agent.as_tool( tool_name="translate_to_french", tool_description="Translate the user's message to French", ), italian_agent.as_tool( tool_name="translate_to_italian", tool_description="Translate the user's message to Italian", ), ], ) return orchestrator_agent ``` ## MCP servers Temporal's durability does not extend to [MCP](https://modelcontextprotocol.io/) servers, which run independently of the Workflow. The integration offers two wrappers so you can pick the one that matches how your server behaves. A **stateless** server treats each operation as independent—`get_weather(location)` carries everything it needs—so it can be reconnected to without changing behavior. A **stateful** server keeps session state between calls, as a server where `set_location(location)` precedes `get_weather()` does, and loses that state if the session drops. Prefer stateless when you have the choice: its durability guarantees are stronger. > **⚠️ Warning:** > > Both `stateless_mcp_server()` and `stateful_mcp_server()` accept a `factory_argument` that is passed to the registered > factory. It is an Activity argument, so it is recorded in Workflow history and, without a payload codec, visible in the > Web UI. Do not pass secrets, credentials, or API keys through it—resolve those Worker-side inside the server factory. > ### Stateless MCP servers Register a `StatelessMCPServerProvider` with a factory that creates the server, and give it a name: [openai_agents/mcp/run_file_system_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp/run_file_system_worker.py) ```py file_system_server = StatelessMCPServerProvider( "FileSystemServer", lambda: MCPServerStdio( name="FileSystemServer", params={ "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", samples_dir], }, ), ) # Create client connected to server at the given address config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") client = await Client.connect( **config, plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=60) ), mcp_server_providers=[file_system_server], ), ], ) ``` Reference the same name from Workflow code with `stateless_mcp_server`: [openai_agents/mcp/workflows/file_system_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp/workflows/file_system_workflow.py) ```py server: MCPServer = openai_agents.workflow.stateless_mcp_server( "FileSystemServer" ) agent = Agent( name="Assistant", instructions="Use the tools to read the filesystem and answer questions based on those files.", mcp_servers=[server], ) ``` ### Stateful MCP servers Register a `StatefulMCPServerProvider` instead. The plugin runs a dedicated Worker that holds the connection open for the life of the Workflow run. [openai_agents/mcp/run_memory_research_scratchpad_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp/run_memory_research_scratchpad_worker.py) ```py memory_server_provider = StatefulMCPServerProvider( "MemoryServer", lambda _: MCPServerStdio( name="MemoryServer", params={ "command": "npx", "args": ["-y", "@modelcontextprotocol/server-memory"], }, ), ) # Create client connected to server at the given address config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") client = await Client.connect( **config, plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=60) ), mcp_server_providers=[memory_server_provider], ), ], ) ``` In the Workflow, `stateful_mcp_server` is an async context manager, which ties the session's lifetime to the block: [openai_agents/mcp/workflows/memory_research_scratchpad_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/mcp/workflows/memory_research_scratchpad_workflow.py) ```py async with temporal_openai_agents.workflow.stateful_mcp_server( "MemoryServer", ) as server: with trace(workflow_name="MCP Memory Scratchpad Example"): agent = Agent( name="Research Scratchpad Agent", instructions=( "Use the Memory MCP tools to persist, query, update, and delete notes." " Keep IDs short and consistent. Synthesis must rely only on recalled notes and include simple" " citations of the form '(Note: id)'. Keep the brief to 5 bullets." ), mcp_servers=[server], model_settings=ModelSettings(tool_choice="required"), ) ``` If the dedicated Worker fails—a network problem, or the server itself going away—the session state is gone and Temporal cannot recreate it. The integration raises an `ApplicationError` so your Workflow can decide what to do; recovering means retrying at the application level, not relying on Activity retries. ### Hosted MCP tool For a network-accessible server, `HostedMCPTool` uses an MCP client hosted by OpenAI, so there is nothing to register on the Worker: [openai_agents/hosted_mcp/workflows/simple_mcp_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/hosted_mcp/workflows/simple_mcp_workflow.py) ```py @workflow.defn class SimpleMCPWorkflow: @workflow.run async def run( self, question: str, server_url: str = "https://gitmcp.io/openai/codex" ) -> str: agent = Agent( name="Assistant", tools=[ HostedMCPTool( tool_config={ "type": "mcp", "server_label": "gitmcp", "server_url": server_url, "require_approval": "never", } ) ], ) result = await Runner.run(agent, question) return result.final_output ``` ## Conversation history and human-in-the-loop Because the agent loop runs inside a Workflow, conversation history and pending approvals have to be replay safe. ### Carry conversation history across turns Hold the history in Workflow state and rebuild each turn's input from the previous result. `result.to_input_list()` returns the conversation as input items, and `result.last_agent` records which agent a handoff left you on: ```python self.input_items.append({"content": user_input, "role": "user"}) result = await Runner.run(self.current_agent, self.input_items) self.input_items = result.to_input_list() self.current_agent = result.last_agent ``` Workflow state is replay safe, so history survives Worker restarts within a run. `SQLiteSession` is not supported: it keeps history in a local file, which no longer identifies one conversation once Workers are distributed. ### Handle long-running conversations A chat-style Workflow accumulates history with every turn, and over a long session the event history can grow large enough to hit Temporal's per-Workflow limit. Use [Continue-as-New](/develop/python/workflows/continue-as-new) to start a fresh Execution carrying the history forward. In this example each user turn arrives as a Workflow [Update](/develop/python/workflows/message-passing#updates), so the caller gets the agent's reply back from the same call. The `run` method waits until Temporal suggests continuing, drains in-flight handlers, then continues as new with the accumulated state: [openai_agents/customer_service/workflows/customer_service_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/customer_service/workflows/customer_service_workflow.py) ```py @workflow.run async def run( self, customer_service_state: CustomerServiceWorkflowState | None = None ): await workflow.wait_condition( lambda: workflow.info().is_continue_as_new_suggested() and workflow.all_handlers_finished() ) workflow.continue_as_new( CustomerServiceWorkflowState( printed_history=self.printed_history, current_agent_name=self.current_agent.name, context=self.context, input_items=self.input_items, ) ) ``` ### Add human approval An agent action that should not proceed unattended can pause for a person. A `HostedMCPTool` configured with `require_approval` calls your `on_approval_request` callback, which runs in Workflow context—so it must be deterministic, and it can wait on a Signal or Update to get the answer from outside: [openai_agents/hosted_mcp/workflows/approval_mcp_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/hosted_mcp/workflows/approval_mcp_workflow.py) ```py def approval_callback(request: MCPToolApprovalRequest) -> MCPToolApprovalFunctionResult: """Simple approval callback that logs the request and approves by default. In a real application, user input would be provided through a UI or API. The approval callback executes within the Temporal workflow, so the application can use signals or updates to receive user input. """ workflow.logger.info(f"MCP tool approval requested for: {request.data.name}") result: MCPToolApprovalFunctionResult = {"approve": True} return result ``` ## Sandbox > **⚠️ Caution:** > > Sandbox support is pre-release and may change before general availability. > `SandboxAgent` gives an agent a machine to work on: a shell to run commands in and a filesystem to read and write. The plugin dispatches every sandbox operation—creating the session, each command, each read and write, PTY interaction, and teardown—as its own Temporal Activity. Each one is individually retryable and visible in Workflow history, and the session state is serialized with the Workflow, so a Worker restart part-way through a run resumes against the same session. Register a `SandboxClientProvider` for each backend you want to reach, under a unique name: [openai_agents/sandbox/run_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/sandbox/run_worker.py) ```py client = await Client.connect( "localhost:7233", plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=60) ), # The plugin registers one set of sandbox activities per # provider, prefixed with the provider name. Register several # providers to let one worker serve several backends. sandbox_clients=[ SandboxClientProvider(SANDBOX_PROVIDER, UnixLocalSandboxClient()), ], ), ], ) ``` In the Workflow, `temporal_sandbox_client()` resolves a name to that backend and goes in the `RunConfig`: [openai_agents/sandbox/workflows/local_sandbox_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/sandbox/workflows/local_sandbox_workflow.py) ```py @workflow.defn class LocalSandboxWorkflow: @workflow.run async def run(self, prompt: str) -> str: # A default SandboxAgent already carries the Filesystem, Shell, and # Compaction capabilities, so there are no tools to declare here. agent = SandboxAgent[None]( name="Sandbox Assistant", instructions=( "You have a sandbox with a shell and a filesystem. Use it to do " "the work rather than answering from memory, then report what " "the commands returned." ), ) result = await Runner.run( starting_agent=agent, input=prompt, run_config=RunConfig( sandbox=SandboxRunConfig( # Must match the name registered on the worker. client=temporal_sandbox_client(SANDBOX_PROVIDER), options=UnixLocalSandboxClientOptions(), ), ), ) return result.final_output_as(str, raise_if_incorrect_type=True) ``` The name becomes the prefix of that backend's Activity names, which is what lets several backends share one Worker. A single Workflow can target more than one by calling `temporal_sandbox_client()` once per name. Names must match the Worker's registration exactly. The sample above uses `UnixLocalSandboxClient`, which runs commands on the Worker host and copies the Worker's entire environment into each agent-run process. An agent can read values such as `OPENAI_API_KEY`. Command output returns as an Activity result and is stored in Workflow history unless you protect payloads with a codec. Use `UnixLocalSandboxClient` only for local development with trusted prompts. In production, register a remote client such as `DaytonaSandboxClient` or `E2BSandboxClient` from `agents.extensions.sandbox` instead. Only the Worker changes; the Workflow still names a provider. ## Streaming > **⚠️ Caution:** > > Streaming is experimental and may change before general availability. > `Runner.run_streamed` works inside a Workflow. The model call runs as a streaming Activity that consumes `Model.stream_response` and publishes each event to a [Workflow Stream](/workflow-streams) topic as the model produces it, so an external client can watch a run live while it stays durable. Set the topic on `ModelActivityParameters.streaming_topic` and host a `WorkflowStream` in the Workflow. The topic is required: without it, `run_streamed` raises before scheduling any Activity. [openai_agents/streaming/workflows/stream_text_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/streaming/workflows/stream_text_workflow.py) ```py @workflow.defn class StreamTextWorkflow: @workflow.init def __init__(self, input: StreamTextInput) -> None: # WorkflowStream requires construction from a method named __init__ # (it checks its caller's frame and raises otherwise), and # @workflow.init is what makes the run argument — and the # stream_state it carries across continue-as-new — available here. self.stream = WorkflowStream(prior_state=input.stream_state) self.done = self.stream.topic(TOPIC_DONE, type=bool) @workflow.run async def run(self, input: StreamTextInput) -> str: agent = Agent( name="Joker", instructions="You are a helpful assistant.", ) result = Runner.run_streamed(agent, input=input.prompt) # The workflow only sees these events once the activity returns, so # the loop just counts them. External subscribers receive them as the # activity publishes them. deltas = 0 async for event in result.stream_events(): if event.type == "raw_response_event" and isinstance( event.data, ResponseTextDeltaEvent ): deltas += 1 workflow.logger.info("collected %d delta events", deltas) # In-band terminator so the subscriber can stop without racing the # workflow's completion, then a brief pause to let its next poll # deliver the tail of the stream — the log lives in workflow memory # and is gone once this run completes. self.done.publish(True) await workflow.sleep(DRAIN_INTERVAL) # final_output is typed Any and is None when a run ends without # message output, so assert the str this signature promises rather # than letting a None through. return result.final_output_as(str, raise_if_incorrect_type=True) ``` Subscribe from outside with `WorkflowStreamClient`: [openai_agents/streaming/run_stream_text_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/streaming/run_stream_text_workflow.py) ```py stream = WorkflowStreamClient.create(client, workflow_id) converter = client.data_converter.payload_converter # A single iterator over both topics — one subscriber, no cancellation race # between concurrent ones. result_type=RawValue delivers the underlying # Payload so heterogeneous topics can be decoded per item.topic. The loop # ends on the in-band terminator, or by the iterator exhausting if the # workflow reaches a terminal state without publishing one (e.g. on # failure); either way handle.result() below surfaces the outcome. last_sequence = -1 response_in_flight = False async for item in stream.subscribe( [TOPIC_EVENTS, TOPIC_DONE], result_type=RawValue ): if item.topic == TOPIC_DONE: break # Subscribers receive native OpenAI events, not the agents-SDK # StreamEvent wrappers that stream_events() yields in the workflow. event: Any = converter.from_payload(item.data.payload, EVENT_TYPE) # Every event carries a sequence_number that starts at 0 per response, # so a number that does not advance means a new response is streaming. # That is a retry only if the previous one never completed: each turn # of a multi-turn run is its own response and restarts the count too. # The retry is an independently sampled answer rather than a # continuation, so mark the seam instead of letting the failed # attempt's partial text run into the new one. The workflow's return # value is unaffected — stream_events() there sees only the attempt # that succeeded. sequence = event.sequence_number if sequence <= last_sequence and response_in_flight: print("\n\n[model activity retried — output restarts here]\n") last_sequence = sequence response_in_flight = not isinstance(event, ResponseCompletedEvent) if isinstance(event, ResponseTextDeltaEvent): print(event.delta, end="", flush=True) ``` Two things to know: - Streaming is incompatible with `use_local_activity`, because Local Activities support neither heartbeats nor the Workflow stream signal channel. - Activity retries are visible to stream subscribers but not to `stream_events()`. An attempt that fails mid-response leaves its events on the stream and the retry publishes a second sequence, while `stream_events()` sees only the successful attempt's collected events. ## Tracing OpenAI Agents SDK tracing works across Client, Workflow, and Activity boundaries, with the plugin propagating trace context for you. ### OpenAI hosted traces Hosted tracing needs no setup beyond the plugin. To start a trace on the Client—so the whole Workflow Execution is part of a larger trace—open `plugin.tracing_context()` first: ```python plugin = OpenAIAgentsPlugin() client = await Client.connect("localhost:7233", plugins=[plugin]) with plugin.tracing_context(): with trace("Customer support workflow"): result = await client.execute_workflow( CustomerSupportAgent.run, "Help me with my order", id="customer-support-123", task_queue="my-task-queue", ) ``` `plugin.tracing_context()` is required when starting traces outside a Worker; without it the trace does not propagate into the Workflow. ### OpenTelemetry > **⚠️ Caution:** > > The OpenTelemetry integration is Public Preview and may change before general availability. > If you already collect traces with OpenTelemetry, the integration can emit the agent's spans through your pipeline, so model calls, tools, and orchestration land in the same backend as the rest of your traces. Install the additional dependencies: ```bash uv add openinference-instrumentation-openai-agents opentelemetry-sdk opentelemetry-exporter-otlp ``` Then set a global replay-safe tracer provider before connecting the Client, and turn on instrumentation in the plugin: ```python from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.export import SimpleSpanProcessor from temporalio.contrib.opentelemetry import create_tracer_provider tracer_provider = create_tracer_provider() tracer_provider.add_span_processor( SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317")) ) trace.set_tracer_provider(tracer_provider) client = await Client.connect( "localhost:7233", plugins=[OpenAIAgentsPlugin(use_otel_instrumentation=True)], ) ``` The provider must come from `create_tracer_provider()`. It replays safely, exporting spans only when a Workflow actually completes rather than on every replay, and generates deterministic span identifiers so they correlate across replays. Passing any other provider raises a `ValueError`. To call the OpenTelemetry API directly from Workflow code, allow the module through the Workflow sandbox and open an agents-SDK span first, so your spans are parented rather than becoming roots: ```python import opentelemetry.trace from agents import custom_span from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner, SandboxRestrictions worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], workflow_runner=SandboxedWorkflowRunner( SandboxRestrictions.default.with_passthrough_modules("opentelemetry") ), ) # Inside the Workflow: with custom_span("Workflow coordination"): tracer = opentelemetry.trace.get_tracer(__name__) with tracer.start_as_current_span("Custom workflow span"): ... ``` ## Resources - [OpenAI Agents SDK samples](https://github.com/temporalio/samples-python/tree/main/openai_agents) — runnable examples for the patterns in this guide. - [`temporalio.contrib.openai_agents` README](https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/openai_agents/README.md) — the full plugin reference, including the complete feature-support matrix. - [OpenAI Agents SDK for Python](https://openai.github.io/openai-agents-python/) - [Temporal Plugins guide](/develop/plugins-guide) — the Plugin system this integration is built on, which you can also use to build your own integrations. --- # Strands Agents integration Source: https://docs.temporal.io/develop/python/integrations/strands-agents > Run Strands Agents AI workflows with durable execution using the Temporal Python SDK and Strands plugin. Temporal's integration with [Strands Agents](https://strandsagents.com/) is an [SDK Plugin](/develop/plugins-guide) that gives your Strands agents [Durable Execution](/temporal#durable-execution) via the Temporal platform. The plugin routes model invocations, tool calls, MCP tool calls, and hooks through Temporal Activities, so every step your agent takes is recorded in Workflow history and can survive crashes, restarts, and infrastructure failures. > **Public Preview** Code snippets in this guide are taken from the [Strands Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/strands_plugin). Refer to the samples for the complete code. ## Get started Install the plugin, then run a minimal Strands agent inside a Temporal Workflow. ### Prerequisites - This guide assumes you are already familiar with Strands Agents. If you are not, refer to the [Strands Agents documentation](https://strandsagents.com/) for more details. - If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Set up your local development environment by following the [Set up your local development environment](/develop/python/set-up-your-local-python) guide. Leave the Temporal development server running if you want to test your code locally. ### Install the plugin Install the Temporal Python SDK with Strands Agents support (requires `temporalio` 1.28.0 or later): ```bash uv add "temporalio[strands-agents]" ``` or with pip: ```bash pip install "temporalio[strands-agents]" ``` ### Run a Strands agent with Durable Execution The following example runs a Strands agent inside a Temporal Workflow. Model calls execute as Temporal Activities, which means they get automatic retries, timeouts, and durable execution. If the Worker process crashes mid-conversation, Temporal replays the Workflow and resumes from the last completed Activity. **1. Define the Workflow** Create a Workflow that holds a `TemporalAgent` and invokes it with a prompt. The `start_to_close_timeout` sets the maximum time each model call Activity can run: [strands_plugin/hello_world/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/workflow.py) ```py from datetime import timedelta from temporalio import workflow from temporalio.contrib.strands import TemporalAgent @workflow.defn class HelloWorldWorkflow: def __init__(self) -> None: self.agent = TemporalAgent(start_to_close_timeout=timedelta(seconds=60)) @workflow.run async def run(self, prompt: str) -> str: result = await self.agent.invoke_async(prompt) return str(result) ``` > **⚠️ Caution:** > > Inside a Workflow, always call `agent.invoke_async(message)`, not `agent(message)`. The synchronous form spawns a worker > thread, which the Workflow sandbox blocks. > **2. Start a Worker** Create a Worker that registers the Workflow and the `StrandsPlugin`. The plugin automatically registers the Activities that handle model calls: [strands_plugin/hello_world/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/run_worker.py) ```py import asyncio import os from temporalio.client import Client from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker from strands_plugin.hello_world.workflow import HelloWorldWorkflow async def main() -> None: plugin = StrandsPlugin() client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin], ) worker = Worker( client, task_queue="strands-hello-world", workflows=[HelloWorldWorkflow], ) print("Worker started. Ctrl+C to exit.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` **3. Run the Workflow** Start the Workflow from a separate client script. This example sends the prompt "Write a haiku about durable execution" and prints the agent's response: [strands_plugin/hello_world/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hello_world/run_workflow.py) ```py import asyncio import os from temporalio.client import Client from strands_plugin.hello_world.workflow import HelloWorldWorkflow async def main() -> None: client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")) result = await client.execute_workflow( HelloWorldWorkflow.run, "Write a haiku about durable execution.", id="strands-hello-world", task_queue="strands-hello-world", ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Build the agent Customize which model provider your agent uses, add tools that run as Activities, subscribe to lifecycle events with hooks, and connect to MCP servers. ### Choose and configure models By default, `StrandsPlugin` uses Strands' own default model (`BedrockModel`). To use a different model, pass a `models` mapping to `StrandsPlugin` on the Worker. When you provide a custom `models` mapping, each `TemporalAgent` must specify which model to use by name. Each entry in the mapping pairs a name with a factory function that creates a model provider (such as `AnthropicModel` or `BedrockModel`). The provider is created on first use and reused for the Worker's lifetime: ```python from strands.models.anthropic import AnthropicModel from strands.models.bedrock import BedrockModel # Workflow @workflow.defn class MultiModelWorkflow: def __init__(self) -> None: self.agent_a = TemporalAgent( model="claude", start_to_close_timeout=timedelta(seconds=60), ) self.agent_b = TemporalAgent( model="bedrock", start_to_close_timeout=timedelta(seconds=60), ) # Worker Worker(..., plugins=[StrandsPlugin(models={ "claude": lambda: AnthropicModel(client_args={"api_key": "..."}), "bedrock": lambda: BedrockModel(), })]) ``` Each `TemporalAgent` carries its own Activity options (timeouts, retry policy, task queue, streaming topic) and dispatches to a shared model Activity, which resolves the model name against the registered factories at runtime. A model name not present in the `models` mapping raises `ValueError` inside the Activity. ### Run non-deterministic tools as Activities Strands tools that perform I/O, access external services, or produce non-deterministic results need to run as Temporal Activities rather than inline in the Workflow. Wrap each tool in an `@activity.defn` function, register the Activities on the Worker, and pass them to the agent using `activity_as_tool`. Define an Activity for the tool: [strands_plugin/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/workflow.py) ```py @activity.defn async def fetch_weather(city: str) -> dict: """Stub weather lookup — replace with a real HTTP call in production.""" return { "city": city, "temperature_f": 72, "conditions": "sunny", } ``` Pass the Activity to the agent in the Workflow using `activity_as_tool`: [strands_plugin/tools/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/workflow.py) ```py @workflow.defn class ToolsWorkflow: def __init__(self) -> None: self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), tools=[ letter_counter, activity_as_tool( fetch_weather, start_to_close_timeout=timedelta(seconds=30), ), activity_as_tool( environment_activity, start_to_close_timeout=timedelta(seconds=30), ), ], ) @workflow.run async def run(self, prompt: str) -> str: result = await self.agent.invoke_async(prompt) return str(result) ``` Register the Activity functions on the Worker: [strands_plugin/tools/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/tools/run_worker.py) ```py import asyncio import os from temporalio.client import Client from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker from strands_plugin.tools.workflow import ( ToolsWorkflow, environment_activity, fetch_weather, ) async def main() -> None: plugin = StrandsPlugin() client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin], ) worker = Worker( client, task_queue="strands-tools", workflows=[ToolsWorkflow], activities=[fetch_weather, environment_activity], ) print("Worker started. Ctrl+C to exit.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` If you are using built-in `strands_tools`, wrap them in a thin async function decorated with `@activity.defn` so they run as Temporal Activities. ### React to agent lifecycle events Strands' [hook system](https://strandsagents.com/docs/user-guide/concepts/agents/hooks/) lets you subscribe callbacks to events in the agent lifecycle, such as invocation start/end, model call before/after, tool call before/after, and message added. Use hooks to add logging, metrics, or custom logic at each stage. Pass `hooks=[MyHookProvider()]` to `TemporalAgent`. Hook callbacks fire in Workflow context, so deterministic callbacks work without any extra setup. For callbacks that need I/O (audit logging, metrics, alerting), use `activity_as_hook` to dispatch the work as a Temporal Activity. The following example shows both patterns in one `HookProvider`. The `_record` callback runs in Workflow context (deterministic), while `persist_tool_call` runs as an Activity (I/O-safe): [strands_plugin/hooks/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hooks/workflow.py) ```py @activity.defn async def persist_tool_call(tool_name: str) -> None: # In production, write to a database / S3 / your audit pipeline. activity.logger.info(f"audit: tool {tool_name} completed") ``` [strands_plugin/hooks/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/hooks/workflow.py) ```py class AuditHook(HookProvider): def __init__(self) -> None: self.fired: list[str] = [] def register_hooks(self, registry: HookRegistry, **kwargs: object) -> None: registry.add_callback(AfterToolCallEvent, self._record) registry.add_callback( AfterToolCallEvent, activity_as_hook( persist_tool_call, activity_input=lambda event: event.tool_use["name"], start_to_close_timeout=timedelta(seconds=15), ), ) def _record(self, event: AfterToolCallEvent) -> None: self.fired.append(event.tool_use["name"]) ``` > **⚠️ Caution:** > > Hook callbacks run in Workflow context, so they must be > [deterministic](/develop/python/workflows/basics#workflow-logic-requirements). Do not use `time.time()`, `uuid.uuid4()`, > or I/O inside hook callbacks. Use `activity_as_hook` for anything that requires I/O. > The `activity_input` parameter extracts serializable values from the event to pass as the Activity's input. Use a dataclass or Pydantic model for multiple values. This is needed because hook events hold references to `Agent`, `AgentTool` instances, and other objects that cannot cross the Activity boundary. ### Connect to MCP servers If your agent needs access to tools provided by an [MCP](https://modelcontextprotocol.io/) server, configure the MCP clients on the Worker and reference them by name in the Workflow. `StrandsPlugin(mcp_clients=...)` takes a mapping of `name` to `MCPClient` factory, mirroring the `models` pattern. The plugin registers a per-server Activity and connects at Worker startup to enumerate available tools. In the Workflow, `TemporalMCPClient(server="name")` is a handle that references the server by name and carries per-call Activity options. Define the Workflow with a `TemporalMCPClient`: [strands_plugin/mcp/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/mcp/workflow.py) ```py from datetime import timedelta from temporalio import workflow from temporalio.contrib.strands import TemporalAgent, TemporalMCPClient @workflow.defn class MCPWorkflow: def __init__(self) -> None: echo = TemporalMCPClient( server="echo", start_to_close_timeout=timedelta(seconds=30), ) self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), tools=[echo], ) @workflow.run async def run(self, prompt: str) -> str: result = await self.agent.invoke_async(prompt) return str(result) ``` Register the MCP client factory on the Worker: [strands_plugin/mcp/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/mcp/run_worker.py) ```py # ... from mcp import StdioServerParameters, stdio_client from strands.tools.mcp.mcp_client import MCPClient from temporalio.client import Client from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker # ... def _make_echo_client() -> MCPClient: return MCPClient( lambda: stdio_client( StdioServerParameters( command=sys.executable, args=[str(ECHO_SERVER)], ) ) ) # ... async def main() -> None: plugin = StrandsPlugin(mcp_clients={"echo": _make_echo_client}) client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin], ) worker = Worker( client, task_queue="strands-mcp", workflows=[MCPWorkflow], ) print("Worker started. Ctrl+C to exit.") await worker.run() ``` Each factory returns a fully configured `MCPClient`, so you can pass options like `tool_filters`, `prefix`, `elicitation_callback`, or `tasks_config` to it. > **ℹ️ Info:** > > The plugin connects to each MCP server once at Worker startup to enumerate tools. The schema is frozen for the Worker's > lifetime. Restart Workers to pick up MCP server changes. If a server is unavailable at startup, the Worker fails to > start. > ## Interact with the agent Control the shape of agent responses, stream output in real time, and pause the agent for human approval. ### Add human approval gates Some agent actions, such as deleting resources or sending messages, may require human approval before proceeding. Strands offers two ways to interrupt an agent and wait for a response. Both work with the plugin. In each case, `agent.invoke_async()` returns `AgentResult(stop_reason="interrupt", interrupts=[...])` instead of raising. Pair this with a Signal handler that supplies responses, then resume by calling `agent.invoke_async(responses)`. #### Interrupt from a hook A hook on an interruptible event such as `BeforeToolCallEvent` can pause the agent by calling `event.interrupt(name, reason=...)`. The hook runs in Workflow context, so it must be deterministic. Define the approval hook: [strands_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/human_in_the_loop/workflow.py) ```py class ApprovalHook(HookProvider): def register_hooks(self, registry: HookRegistry, **kwargs: object) -> None: registry.add_callback(BeforeToolCallEvent, self._gate) def _gate(self, event: BeforeToolCallEvent) -> None: if event.tool_use["name"] != "delete_file": return approval = event.interrupt( "approval", reason=f"approve delete of {event.tool_use['input']['path']}?", ) if approval != "approve": event.cancel_tool = "denied" ``` The Workflow waits for a Signal carrying the approval response, then resumes the agent: [strands_plugin/human_in_the_loop/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/human_in_the_loop/workflow.py) ```py @workflow.defn class HumanInTheLoopWorkflow: def __init__(self) -> None: self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), tools=[delete_file], hooks=[ApprovalHook()], ) self._approval: str | None = None self._pending_reason: str | None = None @workflow.signal def approve(self, response: str) -> None: self._approval = response @workflow.query def pending_approval(self) -> str | None: return self._pending_reason @workflow.run async def run(self, prompt: str) -> str: result = await self.agent.invoke_async(prompt) while result.stop_reason == "interrupt": interrupts = list(result.interrupts or []) self._pending_reason = interrupts[0].reason if interrupts else None await workflow.wait_condition(lambda: self._approval is not None) response = self._approval self._approval = None self._pending_reason = None responses: list[InterruptResponseContent] = [ {"interruptResponse": {"interruptId": i.id, "response": response}} for i in interrupts ] result = await self.agent.invoke_async(responses) return str(result) ``` #### Interrupt from a tool A `@strands.tool` function can raise `InterruptException(Interrupt(...))` directly. The agent stops with the interrupt, and the Workflow handles the resume the same way as for hooks: ```python from strands import tool from strands.interrupt import Interrupt, InterruptException @tool def delete_thing(name: str) -> str: raise InterruptException( Interrupt(id=f"delete:{name}", name="approval", reason=f"delete {name}?") ) ``` The same approach works from an `activity_as_tool`-wrapped Activity. The plugin's failure converter preserves the `Interrupt` payload across the Activity boundary, so `AgentResult.interrupts` is populated the same way. Define the Activity that raises the interrupt: [strands_plugin/activity_interrupt/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/activity_interrupt/workflow.py) ```py @activity.defn async def delete_thing(name: str) -> str: if name not in _APPROVED: _APPROVED.add(name) raise InterruptException( Interrupt( id=f"delete:{name}", name="approval", reason=f"approve delete of protected resource '{name}'?", ) ) return f"deleted {name}" ``` > **⚠️ Caution:** > > Activity-tool interrupts rely on the plugin's failure converter, which is installed via the client's data converter. > Attach `StrandsPlugin` to the **client** (not just the Worker) for Activity-tool interrupts to work. > Workers built from that client pick up the plugin automatically: [strands_plugin/activity_interrupt/run_worker.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/activity_interrupt/run_worker.py) ```py import asyncio import os from temporalio.client import Client from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker from strands_plugin.activity_interrupt.workflow import ( ActivityInterruptWorkflow, delete_thing, ) async def main() -> None: plugin = StrandsPlugin() # The plugin MUST be on the client so its failure converter is installed. client = await Client.connect( os.environ.get("TEMPORAL_ADDRESS", "localhost:7233"), plugins=[plugin], ) worker = Worker( client, task_queue="strands-activity-interrupt", workflows=[ActivityInterruptWorkflow], activities=[delete_thing], ) print("Worker started. Ctrl+C to exit.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ### Return structured data from an agent To have the agent return a typed object instead of free-form text, pass a `structured_output_model` to `TemporalAgent`. The plugin defaults to the [`pydantic_data_converter`](/develop/python/data-handling/data-conversion), so Pydantic types serialize cleanly across the Activity and Workflow boundary: [strands_plugin/structured_output/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/structured_output/workflow.py) ```py from datetime import timedelta from pydantic import BaseModel, Field from temporalio import workflow from temporalio.contrib.strands import TemporalAgent class PersonInfo(BaseModel): name: str = Field(description="Name of the person") age: int = Field(description="Age of the person") occupation: str = Field(description="Occupation of the person") @workflow.defn class StructuredOutputWorkflow: def __init__(self) -> None: self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), structured_output_model=PersonInfo, ) @workflow.run async def run(self, prompt: str) -> PersonInfo: result = await self.agent.invoke_async(prompt) assert isinstance(result.structured_output, PersonInfo) return result.structured_output ``` ### Stream agent output to clients For long-running agent calls, you may want to forward model output chunks to an external consumer as they arrive rather than waiting for the full response. Pass `streaming_topic="..."` to `TemporalAgent` and host a `WorkflowStream` on the Workflow. Each `StreamEvent` is published from inside the model Activity. Subscribers read events through `WorkflowStreamClient`. Chunks are batched on `streaming_batch_interval` (default 100 ms). Define the Workflow with a `WorkflowStream` and a streaming topic: [strands_plugin/streaming/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/streaming/workflow.py) ```py from datetime import timedelta from temporalio import workflow from temporalio.contrib.strands import TemporalAgent from temporalio.contrib.workflow_streams import WorkflowStream @workflow.defn class StreamingWorkflow: def __init__(self) -> None: self.stream = WorkflowStream() self.agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), streaming_topic="events", ) @workflow.run async def run(self, prompt: str) -> str: result = await self.agent.invoke_async(prompt) return str(result) ``` Subscribe to the stream from a client: [strands_plugin/streaming/run_workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/streaming/run_workflow.py) ```py import asyncio import os from datetime import timedelta from strands.types.streaming import StreamEvent from temporalio.client import Client from temporalio.contrib.workflow_streams import WorkflowStreamClient from strands_plugin.streaming.workflow import StreamingWorkflow async def main() -> None: client = await Client.connect(os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")) workflow_id = "strands-streaming" handle = await client.start_workflow( StreamingWorkflow.run, "Count from 1 to 5, one number per sentence.", id=workflow_id, task_queue="strands-streaming", ) async def consume() -> None: stream = WorkflowStreamClient.create(client, workflow_id) async for item in stream.subscribe( ["events"], from_offset=0, result_type=StreamEvent, poll_cooldown=timedelta(milliseconds=50), ): event: StreamEvent = item.data if "contentBlockDelta" in event: delta = event["contentBlockDelta"].get("delta", {}) if "text" in delta: print(delta["text"], end="", flush=True) elif "messageStop" in event: print() return consume_task = asyncio.create_task(consume()) result = await handle.result() await asyncio.wait_for(consume_task, timeout=10.0) print(f"Final result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` ## Run in production Configure retry policies, handle long-running chat sessions, and add distributed tracing. ### Configure retries `TemporalAgent` disables Strands' built-in `ModelRetryStrategy` so that retries are handled exclusively by Temporal. Configure retries with `retry_policy` on `TemporalAgent` for model calls, and on the Activity options accepted by `activity_as_tool`, `activity_as_hook`, and `TemporalMCPClient` for their respective calls: ```python from temporalio.common import RetryPolicy TemporalAgent( start_to_close_timeout=timedelta(seconds=60), retry_policy=RetryPolicy(maximum_attempts=3), ) ``` Passing `retry_strategy=...` to `TemporalAgent(...)` raises `ValueError`. Remove the argument (or pass `retry_strategy=None`) and use `retry_policy` instead. ### Handle long-running chat sessions A chat-style Workflow accumulates message history with every turn. Over a long session, the Workflow's event history can grow large enough to hit Temporal's per-Workflow history limit. To avoid this, use [Continue-as-New](/develop/python/workflows/continue-as-new) to start a fresh Workflow execution while carrying the agent's message history forward as input. In this example, each user turn arrives as a Workflow [Update](/develop/python/workflows/message-passing#updates), so the caller gets the agent's reply back from the same call. The `run` method creates the agent, then waits until either the chat ends or Temporal suggests continue-as-new. When it does, the Workflow drains any in-flight updates and starts a fresh execution with the agent's accumulated messages: [strands_plugin/continue_as_new/workflow.py](https://github.com/temporalio/samples-python/blob/main/strands_plugin/continue_as_new/workflow.py) ```py import asyncio from dataclasses import dataclass, field from datetime import timedelta from strands.types.content import Messages from temporalio import workflow from temporalio.contrib.strands import TemporalAgent @dataclass class ChatInput: messages: Messages = field(default_factory=list) @workflow.defn class ChatWorkflow: def __init__(self) -> None: self._done = False self._lock = asyncio.Lock() self._agent: TemporalAgent | None = None @workflow.update async def turn(self, prompt: str) -> str: await workflow.wait_condition(lambda: self._agent is not None) async with self._lock: assert self._agent is not None result = await self._agent.invoke_async(prompt) return str(result).strip() @workflow.signal def end_chat(self) -> None: self._done = True @workflow.query def messages(self) -> Messages: return list(self._agent.messages) if self._agent else [] @workflow.run async def run(self, input: ChatInput) -> None: self._agent = TemporalAgent( start_to_close_timeout=timedelta(seconds=60), messages=list(input.messages), ) await workflow.wait_condition( lambda: self._done or workflow.info().is_continue_as_new_suggested() ) await workflow.wait_condition(workflow.all_handlers_finished) if not self._done: workflow.continue_as_new(ChatInput(messages=self._agent.messages)) ``` ### Add tracing with OpenTelemetry To get distributed traces across model, tool, and MCP Activities, combine `StrandsPlugin` with the [OpenTelemetry plugin](/develop/python/platform/observability#tracing). Register `OpenTelemetryPlugin` on the client and `StrandsPlugin` on the Worker. Workers built from that client pick up the OpenTelemetry plugin automatically: ```python import opentelemetry.trace from temporalio.client import Client from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider from temporalio.contrib.strands import StrandsPlugin from temporalio.worker import Worker opentelemetry.trace.set_tracer_provider(create_tracer_provider()) client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) Worker( client, task_queue="strands", workflows=[MyWorkflow], plugins=[StrandsPlugin()], ) ``` Set the tracer provider before connecting the client. ### Snapshots are not supported `TemporalAgent.take_snapshot()` and `TemporalAgent.load_snapshot()` raise `NotImplementedError`. Temporal's event history already persists Workflow state durably at a finer granularity than Strands snapshots, so snapshots are redundant inside a Workflow. ### Samples The [Strands Agents plugin samples](https://github.com/temporalio/samples-python/tree/main/strands_plugin) demonstrate all supported patterns end-to-end. --- # Nexus - Python SDK Source: https://docs.temporal.io/develop/python/nexus > This section explains how to use Temporal Nexus with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Temporal Nexus - [Quickstart](/develop/python/nexus/quickstart) - [Feature guide](/develop/python/nexus/feature-guide) - [Standalone Operations](/develop/python/nexus/standalone-operations) --- # Temporal Nexus - Python SDK feature guide Source: https://docs.temporal.io/develop/python/nexus/feature-guide > Use Temporal Nexus within the Python SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Use [Temporal Nexus](/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. > **💡 Tip:** > > New to Nexus? Start with the [Nexus Python Quickstart](/develop/python/nexus/quickstart). > This page shows how to do the following: - [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server) - [Create caller and handler Namespaces](#create-caller-handler-namespaces) - [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint) - [Define the Nexus Service contract](#define-nexus-service-contract) - [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers) - [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service) - [Understand exceptions in Nexus Operations](#exceptions-in-nexus-operations) - [Cancel a Nexus Operation](#canceling-a-nexus-operation) - [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)
> **📝 Note:** > > This documentation uses source code derived from the [Python Nexus sample](https://github.com/temporalio/samples-python/tree/main/hello_nexus). > ## Run the Temporal Development Server with Nexus enabled Prerequisites: - [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/python/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) (`v1.3.0` or higher recommended) - [Install the latest Temporal Python SDK](https://learn.temporal.io/getting_started/python/dev_environment/#add-temporal-python-sdk-dependencies) (`v1.14.1` or higher) The first step in working with Temporal Nexus involves starting a Temporal Server with Nexus enabled. ``` temporal server start-dev ``` This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`. ## Create caller and handler Namespaces Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. ``` temporal operator namespace create --namespace my-target-namespace temporal operator namespace create --namespace my-caller-namespace ``` For this example, `my-target-namespace` will contain the Nexus Operation handler, and you will use a Workflow in `my-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. ## Create a Nexus Endpoint to route requests from caller to handler After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. ``` temporal operator nexus endpoint create \ --name my-nexus-endpoint-name \ --target-namespace my-target-namespace \ --target-task-queue my-handler-task-queue ``` You can also use the Web UI to create the Namespaces and Nexus endpoint. ## Define the Nexus Service contract Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](/dataconversion). The default data converter encodes payloads in the following order: Null, Byte array, Protobuf JSON, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, Protobuf and JSON are common choices. This example uses Python dataclasses serialized into JSON. [hello_nexus/service.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/service.py) ```python from dataclasses import dataclass import nexusrpc @dataclass class MyInput: name: str @dataclass class MyOutput: message: str @nexusrpc.service class MyNexusService: my_sync_operation: nexusrpc.Operation[MyInput, MyOutput] my_workflow_run_operation: nexusrpc.Operation[MyInput, MyOutput] ``` ## Develop a Nexus Service handler and Operation handlers Nexus Operation handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/cloud/limits#nexus-operation-request-timeout). Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database. Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors (for example: worker timeouts), blocking all Operations from the caller to that Endpoint. The `nexusrpc.handler` and `temporalio.nexus` modules have utilities to help create Nexus Operations: - `nexusrpc.handler.sync_operation` - Create a synchronous operation handler - `nexus.workflow_run_operation` - Create an asynchronous operation handler that starts a Workflow ### Develop a Synchronous Nexus Operation handler The `@nexusrpc.handler.sync_operation` decorator is for exposing simple RPC handlers. [hello_nexus/handler/service_handler.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/handler/service_handler.py) ```python import nexusrpc @nexusrpc.handler.service_handler(service=MyNexusService) class MyNexusServiceHandler: @nexusrpc.handler.sync_operation async def my_sync_operation( self, ctx: nexusrpc.handler.StartOperationContext, input: MyInput ) -> MyOutput: return MyOutput(message=f"Hello {input.name} from sync operation!") ``` A synchronous operation handler must return quickly (less than `10s`). Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking). ### Use the Temporal Client for Signals, Queries, and Updates A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](/cloud/limits#nexus-operation-request-timeout). Updates should be short-lived to stay within this deadline. The [nexus_messaging](https://github.com/temporalio/samples-python/tree/main/nexus_messaging) sample shows how to create a Nexus Service that uses synchronous operations to send Updates and Queries. Use `nexus.client()` to get the Client that the Worker was initialized with. In this example, the Workflow Id is derived from the client Id, with the "get_workflow_id" method. This takes a given client Id (in this case, the client is passing in a user ID) to generate a Workflow Id from it. This way the client only needs the identifier it cares about. [nexus_messaging/callerpattern/handler/service_handler.py](https://github.com/temporalio/samples-python/blob/main/nexus_messaging/callerpattern/handler/service_handler.py) ```python from temporalio import nexus def get_workflow_id(user_id: str) -> str: return f"{WORKFLOW_ID_PREFIX}{user_id}" @nexusrpc.handler.service_handler(service=NexusGreetingService) class NexusGreetingServiceHandler: def _get_workflow_handle( self, user_id: str ) -> WorkflowHandle[GreetingWorkflow, str]: return nexus.client().get_workflow_handle_for( GreetingWorkflow.run, get_workflow_id(user_id) ) ... ``` There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-python/blob/main/nexus_messaging/callerpattern/) and [on demand pattern](https://github.com/temporalio/samples-python/blob/main/nexus_messaging/ondemandpattern/). The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it. In addition to `nexus.client()`, you can use `nexus.info()` to access information about the currently-executing Nexus Operation including its Task Queue. ### Develop an Asynchronous Nexus Operation handler to start a Workflow Use the `@nexus.workflow_run_operation` decorator, which is the easiest way to expose a Workflow as an operation. [hello_nexus/handler/service_handler.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/handler/service_handler.py) ```python import nexusrpc from temporalio import nexus @nexusrpc.handler.service_handler(service=MyNexusService) class MyNexusServiceHandler: @nexus.workflow_run_operation async def my_workflow_run_operation( self, ctx: nexus.WorkflowRunOperationContext, input: MyInput ) -> nexus.WorkflowHandle[MyOutput]: return await ctx.start_workflow( WorkflowStartedByNexusOperation.run, input, id=str(uuid.uuid4()), ) ``` Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. In general, the ID should be passed in the Operation input as part of the Nexus Service contract. > **💡 Tip:** > RESOURCES > > [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing. > #### Map a Nexus Operation input to multiple Workflow arguments A Nexus Operation can only take one input parameter. If you want a Nexus Operation to start a Workflow that takes multiple arguments use the `ctx.start_workflow` method. [nexus_multiple_args/handler/service_handler.py](https://github.com/temporalio/samples-python/blob/main/nexus_multiple_args/handler/service_handler.py) ```py @nexusrpc.handler.service_handler(service=MyNexusService) class MyNexusServiceHandler: """ Service handler that demonstrates multiple argument handling in Nexus operations. """ # This is a nexus operation that is backed by a Temporal workflow. # The key feature here is that it demonstrates how to map a single input object # (HelloInput) to a workflow that takes multiple individual arguments. @nexus.workflow_run_operation async def hello( self, ctx: nexus.WorkflowRunOperationContext, input: HelloInput ) -> nexus.WorkflowHandle[HelloOutput]: """ Start a workflow with multiple arguments unpacked from the input object. """ return await ctx.start_workflow( HelloHandlerWorkflow.run, args=[ input.name, # First argument: name input.language, # Second argument: language ], id=f"hello-multi-args-{input.name}-{input.language}", ) ``` ### Register your Nexus Service handler in a Worker After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register your Nexus Service handler in a Worker. At this stage you can pass any arguments you need to your service handler's `__init__` method. [hello_nexus/handler/worker.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/handler/worker.py) ```python async def main(): client = await Client.connect("localhost:7233", namespace=NAMESPACE) worker = Worker( client, task_queue=TASK_QUEUE, workflows=[WorkflowStartedByNexusOperation], nexus_service_handlers=[MyNexusServiceHandler()], ) await worker.run() ``` ## Develop a caller Workflow that uses the Nexus Service To execute a Nexus Operation from the caller Workflow, import the necessary service definition and operation input/output types: [hello_nexus/caller/workflows.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/caller/workflows.py) ```python from temporalio import workflow with workflow.unsafe.imports_passed_through(): from hello_nexus.service import MyInput, MyNexusService, MyOutput @workflow.defn class CallerWorkflow: @workflow.run async def run(self, name: str) -> tuple[MyOutput, MyOutput]: nexus_client = workflow.create_nexus_client( service=MyNexusService, endpoint=NEXUS_ENDPOINT, ) # Start the nexus operation and wait for the result in one go, using execute_operation. wf_result = await nexus_client.execute_operation( MyNexusService.my_workflow_run_operation, MyInput(name), ) # Alternatively, you can use start_operation to obtain the operation handle and # then `await` the handle to obtain the result. sync_operation_handle = await nexus_client.start_operation( MyNexusService.my_sync_operation, MyInput(name), ) sync_result = await sync_operation_handle return sync_result, wf_result ``` ### Register the caller Workflow in a Worker and start the caller Workflow After developing the caller Workflow, the next step is to register it with a Worker. Finally, the caller Workflow must be started using `client.start_workflow()` or `client.execute_workflow()` These steps are the same as for any normal Workflow. The Python sample combines them in a single application. See [hello_nexus/caller/app.py](https://github.com/temporalio/samples-python/blob/main/hello_nexus/caller/app.py) for reference. ## Exceptions in Nexus operations Temporal provides general guidance on [Errors in Nexus operations](/references/failures#errors-in-nexus-operations). In Python, there are three Nexus-specific exception classes: - [`nexusrpc.OperationError`](https://nexus-rpc.github.io/sdk-python/nexusrpc.OperationError.html): this is the exception type you should raise in a Nexus operation to indicate that it has failed according to its own application logic and should not be retried. - [`nexusrpc.HandlerError`](https://nexus-rpc.github.io/sdk-python/nexusrpc.HandlerError.html): you can raise this exception type in a Nexus operation with a specific [HandlerErrorType](https://nexus-rpc.github.io/sdk-python/nexusrpc.HandlerErrorType.html). The error will be marked retryable or non-retryable according to the type, following the [Nexus spec](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors). The non-retryable handler error types are `BAD_REQUEST`, `UNAUTHENTICATED`, `UNAUTHORIZED`, `NOT_FOUND`, `NOT_IMPLEMENTED`; the retryable types are `RESOURCE_EXHAUSTED`, `INTERNAL`, `UNAVAILABLE`, `UPSTREAM_TIMEOUT`. - [`temporalio.exceptions.NexusOperationError`](https://python.temporal.io/temporalio.exceptions.NexusOperationError.html): this is the error raised inside a Workflow when a Nexus operation fails for any reason. Use the `__cause__` attribute on the exception to access the cause chain. ## Canceling a Nexus Operation To cancel a Nexus Operation from within a Workflow, call `handle.cancel()` on the operation handle. Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancellation request. If ignored, the operation may enter a terminal state. When a Nexus operation is started, the caller can specify different cancellation types that control how the caller reacts to cancellation: - `ABANDON` - Do not request cancellation of the operation. - `TRY_CANCEL` - Initiate a cancellation request and immediately report cancellation to the caller. Note that this type doesn't guarantee that cancellation is delivered to the operation handler if the caller exits before the delivery is done. - `WAIT_REQUESTED` Request cancellation of the operation and wait for confirmation that the request was received. Doesn't wait for actual cancellation. - `WAIT_COMPLETED` - Wait for operation completion. Operation may or may not complete as cancelled. The default is `WAIT_COMPLETED`. Users can set a different option for `cancellation_type` when starting or executing an operation. Once the caller Workflow completes, the caller's Nexus Machinery will not make any further attempts to cancel operations that are still running. It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending operations to finish before exiting the Workflow. See the [Nexus cancellation sample](https://github.com/temporalio/samples-python/tree/main/nexus_cancel) for reference. ## Make Nexus calls across Namespaces in Temporal Cloud This section assumes you are already familiar with how to connect a Worker to Temporal Cloud. The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install `tcld` and generate certificates Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command (on macOS): ``` brew install temporalio/brew/tcld ``` If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: ``` tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key ``` These certificates will be valid for one year. ### Create caller and handler Namespaces Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. **Temporal CLI** ``` temporal cloud login temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` **tcld** ``` tcld login tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/Namespaces](https://cloud.temporal.io/Namespaces). ### Create a Nexus Endpoint to route requests from caller to handler To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. **Temporal CLI** ``` temporal cloud nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file hello_nexus/endpoint_description.md ``` **tcld** ``` tcld nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file hello_nexus/endpoint_description.md ``` The `--allow-namespace` is used to build an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in Runtime Access Control. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus). ## Observability ### Web UI A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history: ![Observability Sync](/img/cloud/nexus/go-sdk-observability-sync.png) An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history: ![Observability Async](/img/cloud/nexus/go-sdk-observability-async.png) ### Temporal CLI Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: ``` temporal workflow describe -w ``` Nexus events are included in the caller's Event history: ``` temporal workflow show -w ``` For **asynchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationStarted` - `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationCompleted` > **📝 Note:** > > `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. > ## Learn more - Read the high-level description of the [Temporal Nexus feature](/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082). - Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus). - Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus). --- # Nexus Python Quickstart Source: https://docs.temporal.io/develop/python/nexus/quickstart > Build a Nexus Service that wraps an existing Temporal Workflow using the Python SDK [Temporal Nexus](/evaluate/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow. > **ℹ️ Info:** > To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus). **Prerequisites:** Complete the [Python SDK Quickstart](/develop/python/set-up-your-local-python) first. You should have `activities.py`, `workflows.py`, `worker.py`, and `starter.py` from that guide. ## What you'll build You have `SayHelloWorkflow` running in the `default` Namespace. By the end of this guide: 1. A Nexus Service will expose `SayHelloWorkflow` as an Operation. 2. A second Namespace will contain a Workflow that calls that Operation. 3. The caller Workflow will get back `"Hello Temporal"` — the same result, but across Namespaces. ## 1. Define the Nexus Service Create a file called `service.py` that defines the Nexus Service contract. Creating a Nexus Service establishes the contract between your implementation and any callers. It provides type safety when invoking Nexus Operations and ensures that operation handlers fulfill the contract. The `@nexusrpc.service` decorator declares a service with typed operations. `SayHelloWorkflow` returns `str`, so the operation output type is `str`. ```python from dataclasses import dataclass import nexusrpc @dataclass class MyInput: name: str @nexusrpc.service class SayHelloNexusService: say_hello: nexusrpc.Operation[MyInput, str] ``` ## 2. Define the Nexus Operation handlers Create a file called `handler.py` that implements the Nexus Operation handler. Operation handlers contain the logic that runs when a caller invokes a Nexus Operation. The `@nexus.workflow_run_operation` decorator creates an asynchronous Nexus Operation that starts a Workflow. The handler bridges the Nexus `MyInput` dataclass to `SayHelloWorkflow`'s `str` parameter by extracting `input.name`. ```python import uuid import nexusrpc.handler from temporalio import nexus from service import MyInput, SayHelloNexusService from workflows import SayHelloWorkflow @nexusrpc.handler.service_handler(service=SayHelloNexusService) class SayHelloNexusServiceHandler: @nexus.workflow_run_operation async def say_hello( self, ctx: nexus.WorkflowRunOperationContext, input: MyInput ) -> nexus.WorkflowHandle[str]: return await ctx.start_workflow( SayHelloWorkflow.run, input.name, id=f"say-hello-nexus-{uuid.uuid4()}", # Task queue defaults to the task queue this Operation is handled on. ) ``` ## 3. Register the Nexus Service handler in a Worker Update your existing `worker.py` to register the Nexus Service handler. A Worker will only poll for and process incoming Nexus requests if the Nexus Service handlers are registered. This is the same Worker concept used for Workflows and Activities. The `nexus_service_handlers` parameter registers the handler so it can receive Nexus Operation requests. ```python import asyncio from temporalio.client import Client from temporalio.worker import Worker from workflows import SayHelloWorkflow from activities import greet from handler import SayHelloNexusServiceHandler async def main(): client = await Client.connect("localhost:7233") worker = Worker( client, task_queue="my-task-queue", workflows=[SayHelloWorkflow], activities=[greet], nexus_service_handlers=[SayHelloNexusServiceHandler()], ) print("Worker started.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ## 4. Develop the caller Workflow Create a file called `caller.py` that defines a Workflow which invokes the Nexus Operation. The caller Workflow demonstrates the consumer side of Nexus. Instead of importing handler code directly, the caller only depends on the Service contract. This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams. The `workflow.create_nexus_client()` method creates a client bound to your Nexus Service and Endpoint. `execute_operation` starts the operation and waits for the result. ```python from datetime import timedelta from temporalio import workflow from service import MyInput, SayHelloNexusService NEXUS_ENDPOINT = "my-nexus-endpoint-name" @workflow.defn class CallerWorkflow: @workflow.run async def run(self, name: str) -> str: nexus_client = workflow.create_nexus_client( service=SayHelloNexusService, endpoint=NEXUS_ENDPOINT, ) return await nexus_client.execute_operation( SayHelloNexusService.say_hello, MyInput(name=name), schedule_to_close_timeout=timedelta(seconds=10), ) ``` ## 5. Create the caller Namespace and Nexus Endpoint Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server. Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. The endpoint name must match the variable defined in `caller.py` from step 4. Make sure your local Temporal dev server is running (`temporal server start-dev`). ```bash temporal operator namespace create --namespace my-caller-namespace ``` ```bash temporal operator nexus endpoint create \\ --name my-nexus-endpoint-name \\ --target-namespace default \\ --target-task-queue my-task-queue ``` ## 6. Run and Verify Create a file called `caller_starter.py` to start the caller Worker and execute the Workflow. This step brings everything together: the caller Worker hosts `CallerWorkflow`, which uses the Nexus client to invoke `say_hello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running `SayHelloWorkflow`, and back to the caller. **Run the application:** 1. Start the handler Worker in one terminal: ```bash source env/bin/activate python3 worker.py ``` 2. Run the caller in another terminal: ```bash source env/bin/activate python3 caller_starter.py ``` You should see: ``` Workflow result: Hello Temporal ``` Open the [Temporal Web UI](http://localhost:8233) and find the `CallerWorkflow` execution. You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history. ```python import asyncio import uuid from temporalio.client import Client from temporalio.worker import Worker from caller import CallerWorkflow CALLER_TASK_QUEUE = "my-caller-task-queue" NAMESPACE = "my-caller-namespace" async def main(): client = await Client.connect("localhost:7233", namespace=NAMESPACE) async with Worker( client, task_queue=CALLER_TASK_QUEUE, workflows=[CallerWorkflow], ): result = await client.execute_workflow( CallerWorkflow.run, "Temporal", id=f"caller-workflow-{uuid.uuid4()}", task_queue=CALLER_TASK_QUEUE, ) print("Workflow result:", result) if __name__ == "__main__": asyncio.run(main()) ``` ## Next Steps Now that you have a working Nexus Service, here are some resources to deepen your understanding: - **[Python Nexus Feature Guide](/develop/python/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls. - **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics. - **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker. - **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns. - **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries. - **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls. - **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests. - **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity. --- # Standalone Nexus Operations - Python SDK Source: https://docs.temporal.io/develop/python/nexus/standalone-operations > Execute Nexus Operations independently without a Workflow using the Temporal Python SDK. > **Pre-release** > Requires Python SDK `1.30.0` or above. All APIs are experimental and may be subject to backwards-incompatible changes. [Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using `workflow.create_nexus_client()`, you execute a Standalone Nexus Operation directly from a Nexus Client created using `client.create_nexus_client()`. Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/python/nexus/feature-guide) for details on [defining a Service contract](/develop/python/nexus/feature-guide#define-nexus-service-contract), [developing Operation handlers](/develop/python/nexus/feature-guide#develop-nexus-service-operation-handlers), and [registering a Service in a Worker](/develop/python/nexus/feature-guide#register-a-nexus-service-in-a-worker). This page focuses on the client-side APIs that are unique to Standalone Nexus Operations: - [Execute a Standalone Nexus Operation](#execute-operation) - [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result) - [List Standalone Nexus Operations](#list-operations) - [Count Standalone Nexus Operations](#count-operations) - [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud) > **📝 Note:** > This documentation uses source code from the > [Python Nexus Standalone sample](https://github.com/temporalio/samples-python/tree/main/nexus_standalone_operations). > ## Prerequisites Standalone Nexus Operations are at Pre-release and require a special Temporal CLI build. ### 1. Install and verify the Pre-release Temporal CLI The `temporal nexus operation` commands require a Pre-release build of the Temporal CLI. See [Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for the platform downloads, then verify: ```bash ./temporal --version # temporal version 1.7.4-standalone-nexus-operations ``` Run it as `./temporal` from the directory where you extracted it. The standard `brew install temporal` build does not include Standalone Nexus Operation support during Pre-release. ### 2. Start a local dev server The Pre-release dev server enables Standalone Nexus Operations by default — no dynamic config is required. Start it with the caller and handler Namespaces pre-created: ```bash ./temporal server start-dev \ --namespace my-caller-namespace \ --namespace my-handler-namespace ``` The starter and Worker connect to two different Namespaces (a caller Namespace and a handler Namespace), mirroring how Nexus crosses Namespace boundaries. To run the examples on this page against the [Python sample](https://github.com/temporalio/samples-python/tree/main/nexus_standalone_operations), create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue: ```bash ./temporal operator nexus endpoint create \ --name my-nexus-endpoint \ --target-namespace my-handler-namespace \ --target-task-queue nexus-handler-queue ``` Start the sample Worker in the handler Namespace: ```bash TEMPORAL_NAMESPACE=my-handler-namespace uv run nexus_standalone_operations/worker.py ``` Run the starter in the caller Namespace (from a separate terminal): ```bash TEMPORAL_NAMESPACE=my-caller-namespace uv run nexus_standalone_operations/starter.py ``` ## Execute a Standalone Nexus Operation To execute a Standalone Nexus Operation, first create a [`NexusClient`](https://python.temporal.io/temporalio.client.NexusClient.html) using `client.create_nexus_client()`, bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `start_operation()` or `execute_operation()` from application code (for example, a starter program), not from inside a Workflow Definition. `execute_operation` waits for the Operation to complete and returns the result. Both methods require `id`. `schedule_to_close_timeout` is optional and defaults to the maximum allowed by the Temporal server. ```python nexus_client = client.create_nexus_client( service=MyNexusService, endpoint=ENDPOINT_NAME ) # Await the result of the operation immediately. echo_result = await nexus_client.execute_operation( MyNexusService.echo, EchoInput(message="hello"), id=f"echo-{uuid.uuid4()}", schedule_to_close_timeout=timedelta(seconds=10), ) ``` See the full [starter sample](https://github.com/temporalio/samples-python/blob/main/nexus_standalone_operations/starter.py) for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and counts Operations. Or use the Temporal CLI to execute a Standalone Nexus Operation: ```bash ./temporal nexus operation execute \ --namespace my-caller-namespace \ --endpoint my-nexus-endpoint \ --service MyNexusService \ --operation echo \ --operation-id my-echo-op \ --input '{"message":"hello"}' ``` ## Start a Standalone Nexus Operation and Wait for the Result `start_operation` returns a [`NexusOperationHandle`](https://python.temporal.io/temporalio.client.NexusOperationHandle.html). Use `NexusOperationHandle.result()` to wait until the Operation completes and retrieve its result. This works for both synchronous and asynchronous Operations. ```python # Start an operation and get a NexusOperationHandle handle = await nexus_client.start_operation( MyNexusService.hello, HelloInput(name="World"), id=f"hello-{uuid.uuid4()}", schedule_to_close_timeout=timedelta(seconds=10), ) # Await the result try: hello_result = await handle.result() print(hello_result) except err: print(err) raise ``` If the Operation completed successfully, the result is returned. If the Operation failed, the failure is raised as an error. Or use the Temporal CLI to wait for a result by Operation ID: ```bash ./temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op ``` ## List Standalone Nexus Operations Use [`client.list_nexus_operations()`](https://python.temporal.io/temporalio.client.Client.html#list_nexus_operations) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The result contains an iterator that yields operation metadata entries. Note that `list_nexus_operations` is called on the base `client.Client`, not on the `NexusClient`. ```python query = f'Endpoint = "{ENDPOINT_NAME}"' async for op in client.list_nexus_operations(query): print( f" OperationId: {op.operation_id},", f" Operation: {op.operation},", f" Status: {op.status.name}", ) ``` The `query` parameter accepts [List Filter](/list-filter) syntax. For example, `"Endpoint = 'my-endpoint' AND Status = 'Running'"`. Or use the Temporal CLI: ```bash ./temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Count Standalone Nexus Operations Use [`client.count_nexus_operations()`](https://python.temporal.io/temporalio.client.Client.html#count_nexus_operations) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. Note that `count_nexus_operations` is called on the base `client.Client`, not on the `NexusClient`. ```python query = f'Endpoint = "{ENDPOINT_NAME}"' count = await client.count_nexus_operations(query) print(f"Total Nexus operations: {count.count}") ``` Or use the Temporal CLI: ```bash ./temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Run Standalone Nexus Operations with Temporal Cloud The code samples referenced on this page use [`ClientConfig.load_client_connect_config()`](https://python.temporal.io/temporalio.envconfig.ClientConfig.html#load_client_connect_config), so the same code works against Temporal Cloud — just configure the connection via environment variables or a TOML profile. No code changes are needed. For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate generation, and authentication options, see [Make Nexus calls across Namespaces in Temporal Cloud](/develop/python/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) and [Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud). --- # Platform - Python SDK Source: https://docs.temporal.io/develop/python/platform > This section explains how to implement platform with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Platform - [Observability](/develop/python/platform/observability) - [Enriching the UI](/develop/python/platform/enriching-ui) --- # Enriching the user interface - Python SDK Source: https://docs.temporal.io/develop/python/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the Python SDK. Temporal supports adding context to Workflows and events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```python # Start a Workflow with static summary and details handle = await client.start_workflow( YourWorkflow.run, "workflow input", id="your-workflow-id", task_queue="your-task-queue", static_summary="Order processing for customer #12345", static_details="Processing premium order with expedited shipping" ) ``` `static_summary` is a single-line description that appears in the Workflow list view, limited to 200 bytes. `static_details` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use the `execute_workflow` method with the same parameters: ```python result = await client.execute_workflow( YourWorkflow.run, "workflow input", id="your-workflow-id", task_queue="your-task-queue", static_summary="Order processing for customer #12345", static_details="Processing premium order with expedited shipping" ) ``` ### Inside the Workflow Within a Workflow, you can get and set the _current Workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```python @workflow.defn class YourWorkflow: @workflow.run async def run(self, input: str) -> str: # Get the current details current_details = workflow.get_current_details() print(f"Current details: {current_details}") # Set/update the current details workflow.set_current_details("Updated workflow details with new status") return "Workflow completed" ``` ### Adding Summary to Activities and Timers You can attach a metadata parameter `summary` to Activities when starting them from within a Workflow: ```python @workflow.defn class YourWorkflow: @workflow.run async def run(self, input: str) -> str: # Start an activity with a summary result = await workflow.execute_activity( your_activity, input, start_to_close_timeout=timedelta(seconds=10), summary="Processing user data" ) return result ``` Similarly, you can attach a `summary` to Timers within a Workflow: ```python @workflow.defn class YourWorkflow: @workflow.run async def run(self, input: str) -> str: # Create a timer with a summary await workflow.sleep(timedelta(minutes=5), summary="Waiting for payment confirmation") return "Timer completed" ``` The input format for `summary` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the Workflow details page, you'll find the Workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the Workflow - **Current Details** - Displays the dynamic details that can be updated during Workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding events, providing immediate context without requiring you to expand the Event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. --- # Observability - Python SDK Source: https://docs.temporal.io/develop/python/platform/observability > Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes. The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. This section covers features related to viewing the state of the application, including: - [Emit metrics](#metrics) - [Set up tracing](#tracing) - [Log from a Workflow](#logging) - [Visibility APIs](#visibility) ## Emit metrics Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process. For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics). - For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide. - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). - For an end-to-end example that exposes metrics with the Python SDK, refer to the [samples-python](https://github.com/temporalio/samples-python/tree/main/prometheus) repo. Metrics in Python are configured globally; therefore, you should set a Prometheus endpoint before any other Temporal code. The following example exposes a Prometheus endpoint on port `9000`. ```python from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig # Create a new runtime that has telemetry enabled. Create this first to avoid # the default Runtime from being lazily created. new_runtime = Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_address="0.0.0.0:9000"))) my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime) ``` ## Set up tracing Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows. Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly. To configure tracing in Python, install the `opentelemetry` dependencies. ```bash # This command installs the `opentelemetry` dependencies. pip install temporalio[opentelemetry] ``` Then the [`temporalio.contrib.opentelemetry.TracingInterceptor`](https://python.temporal.io/temporalio.contrib.opentelemetry.TracingInterceptor.html) class can be set as an interceptor as an argument of [`Client.connect()`](https://python.temporal.io/temporalio.client.Client.html#connect). When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution. ## Log from a Workflow Logging enables you to record critical information during code execution. Loggers create an audit trail and capture information about your Workflow's operation. An appropriate logging level depends on your specific needs. During development or troubleshooting, you might use debug or even trace. In production, you might use info or warn to avoid excessive log volume. You can log from a Workflow using Python's standard library, by importing the logging module `logging`. You can find the log levels supported by the `logging` module in [their official documentation](https://docs.python.org/3/library/logging.html#logging-levels). The Temporal SDK core normally uses `WARN` as its default logging level. Set your logging configuration to a level you want to expose logs to. The following example sets the logging information level to `INFO`. ```python logging.basicConfig(level=logging.INFO) ``` Then in your Workflow, set your [`logger`](https://python.temporal.io/temporalio.workflow.html#logger) and level on the Workflow. The following example logs the Workflow. ```python {11} from temporalio import workflow @workflow.defn class GreetingWorkflow: def __init__(self) -> None: self._greeting = "" @workflow.run async def run(self, name: str) -> None: workflow.logger.info("Workflow input parameter: %s" % name) self._greeting = f"Hello, {name}!" @workflow.query def greeting(self) -> str: return self._greeting ``` ### Custom logger Use a custom logger for logging. Use the built-in [Logging facility for Python](https://docs.python.org/3/library/logging.html) to set a custom logger. ## Visibility APIs The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service. ### Use Search Attributes The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service in the Temporal CLI or Web UI. - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text` - Replace `CustomKeywordField` with the name of your Search Attribute. - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList` - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an option when starting the Execution. - In the Workflow by calling `upsert_search_attributes`. - Read the value of the Search Attribute: - On the Client by calling `DescribeWorkflow`. - In the Workflow by looking at `WorkflowInfo`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/operator#list-2) - In code by calling `ListWorkflowExecutions`. Here is how to query Workflow Executions: Use the [list_workflows()](https://python.temporal.io/temporalio.client.Client.html#list_workflows) method on the Client handle and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows. ```python {30-31} import asyncio from temporalio.client import Client from greeting_workflow import GreetingWorkflow async def main(): client = await Client.connect("localhost:7233") handle = await client.start_workflow( GreetingWorkflow.run, id="search-attributes-workflow-id", task_queue="search-attributes-task-queue", search_attributes={"CustomKeywordField": ["old-value"]}, ) print( "First search attribute values: ", (await handle.describe()).search_attributes.get("CustomKeywordField"), ) await asyncio.sleep(3) print( "Second search attribute values: ", (await handle.describe()).search_attributes.get("CustomKeywordField"), ) await asyncio.sleep(3) print( "Empty search attribute values: ", (await handle.describe()).search_attributes.get("CustomKeywordField"), ) async for workflow in client.list_workflows('WorkflowType="GreetingWorkflow"'): print(f"Workflow: {workflow.id}") if __name__ == "__main__": asyncio.run(main()) ``` ### How to set custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create` or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. Use `SearchAttributeKey` to create your Search Attributes. Then, when starting a Workflow execution using `client.start_workflow()`, include the Custom Search Attributes by passing instances of `SearchAttributePair()` containing each of your keys and starting values to a parameter called `search_attributes`. If you had Custom Search Attributes `CustomerId` of type `Keyword` and `MiscData` of type `Text`, you could provide these starting values: ```python customer_id_key = SearchAttributeKey.for_keyword("CustomerId") misc_data_key = SearchAttributeKey.for_text("MiscData") handle = await client.start_workflow( GreetingWorkflow.run, id="search-attributes-workflow-id", task_queue="search-attributes-task-queue", search_attributes=TypedSearchAttributes([ SearchAttributePair(customer_id_key, "customer_1"), SearchAttributePair(misc_data_key, "customer_1_data") ]), ) ``` In this example, `CustomerId` and `MiscData` are set as Search Attributes. These attributes are useful for querying Workflows based on the customer ID or the date the order was placed. ### Upsert Search Attributes You can upsert Search Attributes to add or update Search Attributes from within Workflow code. To upsert custom Search Attributes, use the [`upsert_search_attributes()`](https://python.temporal.io/temporalio.workflow.html#upsert_search_attributes) method to pass a list of `SearchAttributeUpdate()`. These can be created via value_set calls on Search Attribute keys: ```python workflow.upsert_search_attributes([ customer_id_key.value_set("customer_2") ]) ``` ### Remove a Search Attribute from a Workflow To remove a Search Attribute that was previously set, use `value_unset call` on the Search Attribute key. ```python workflow.upsert_search_attributes([ customer_id_key.value_unset() ]) ``` --- # Set up your local with the Python SDK Source: https://docs.temporal.io/develop/python/set-up-your-local-python > Configure your local development environment to get started developing with Temporal # Quickstart Configure your local development environment to get started developing with Temporal. ## Install Python Make sure you have Python installed. Check your version of Python with the following command. ```bash python3 -V ``` ```bash python 3.13.3 ``` ## Install the Temporal Python SDK You should install the Temporal Python SDK in your project using a virtual environment. Create a directory for your Temporal project, switch to the new directory, create a Python virtual environment, activate it, and then install the Temporal SDK. Next, you'll configure a local Temporal Service for development. ```bash mkdir temporal-project ``` ```bash cd temporal-project ``` ```bash python3 -m venv env ``` ```bash source env/bin/activate ``` ```bash pip install temporalio ``` ## Install Temporal CLI The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI. **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal Python SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Create the Activity Create an Activity file (activities.py): ```python from temporalio import activity @activity.defn async def greet(name: str) -> str: return f"Hello {name}" ``` An Activity is a normal function or method that executes a single, well-defined action (either short or long running), which often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration. ### 2. Create the Workflow Create a Workflow file (workflows.py): ```python from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import greet @workflow.defn class SayHelloWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( greet, name, schedule_to_close_timeout=timedelta(seconds=10), ) ``` Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. ### 3. Create the Worker Create a Worker file (worker.py): ```python import asyncio from temporalio.client import Client from temporalio.worker import Worker from temporalio import workflow with workflow.unsafe.imports_passed_through(): from workflows import SayHelloWorkflow from activities import greet async def main(): client = await Client.connect("localhost:7233") worker = Worker( client, task_queue="my-task-queue", workflows=[SayHelloWorkflow], activities=[greet], ) print("Worker started.") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Run the Worker by opening up a new terminal: ```bash source env/bin/activate python3 worker.py ``` Keep this terminal running - you should see "Worker started" displayed. With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues the Workflow or Activity task from the Task Queue, it then executes that task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). ### 4. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. This final step will validate that everything is working correctly with your file labeled `starter.py`. Create a separate file called `starter.py`: ```python import asyncio import uuid from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( "SayHelloWorkflow", "Temporal", id=f"say-hello-workflow-{uuid.uuid4()}", task_queue="my-task-queue", ) print("Workflow result:", result) if __name__ == "__main__": asyncio.run(main()) ``` While the Worker is still running, run the following command in a new terminal: ```bash source env/bin/activate python3 starter.py ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the Workflow and Activity - Output: `Workflow result: Hello Temporal` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/python/first_program_in_python/): Create a basic Workflow and run it with the Temporal Python SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Workers - Python SDK Source: https://docs.temporal.io/develop/python/workers > This section explains how to implement Workers with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Workers - [Worker processes](/develop/python/workers/run-worker-process) - [Interceptors](/develop/python/workers/interceptors) --- # Interceptors - Python SDK Source: https://docs.temporal.io/develop/python/workers/interceptors Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to add common behavior across many calls, such as tracing and context propagation. This is similar to using middleware in web frameworks such as [Django](https://docs.djangoproject.com/en/5.2/topics/http/middleware/), [Starlette](https://www.starlette.io/middleware/), and [Flask](https://flask.palletsprojects.com/en/stable/lifecycle/#middleware). There are two types of interceptors--inbound and outbound. * Outbound interceptors wrap network calls, running before they reach the network and after they return. * Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns. Concretely, there are five categories of inbound and outbound calls that you can modify in this way: | | [Outbound Client](https://python.temporal.io/temporalio.client.OutboundInterceptor.html) | [Inbound Workflow](https://python.temporal.io/temporalio.worker.WorkflowInboundInterceptor.html) | [Outbound Workflow](https://python.temporal.io/temporalio.worker.WorkflowOutboundInterceptor.html) | [Inbound Activity](https://python.temporal.io/temporalio.worker.ActivityInboundInterceptor.html) | [Outbound Activity](https://python.temporal.io/temporalio.worker.ActivityOutboundInterceptor.html) | | --- | --- | --- | --- | --- | --- | | **Description** | Wraps calls from your application to the Temporal Client to start a Workflow or send [Messages](/encyclopedia/workflow-message-passing/) to it | Wraps calls arriving into a [Workflow Execution](/workflow-execution), such as executing the Workflow, handling [Messages](/encyclopedia/workflow-message-passing/) | Wraps calls a [Workflow](/workflow-definition) makes to the SDK, such as scheduling [Activities](/activities), starting [Child Workflows](/child-workflows), and invoking [Nexus Operations](/nexus) | Wraps calls arriving into an [Activity Execution](/activity-execution) | Wraps calls an [Activity](/activities) makes to the SDK, such as sending [Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) and reading Activity info | | **Runs on** | Client | Worker (Workflow sandbox) | Worker (Workflow sandbox) | Worker (Activity context) | Worker (Activity context) | | **Example methods** | `start_workflow()`, `signal_workflow()`, `list_workflows()` | `execute_workflow()`, `handle_query()`, `handle_signal()`, `handle_update_handler()` | `start_activity()`, `start_child_workflow()`, `signal_child_workflow()`, `start_nexus_operation()` | `execute_activity()` | `info()`, `heartbeat()` | These are not exhaustive lists; refer to the linked API docs for each category. > **⚠️ Warning:** > Workflow interceptors and replay > > Workflow inbound and outbound interceptor methods also execute during [replay](/develop/python/best-practices/testing-suite#replay). Use replay-safe APIs for logging, randomness, and time in these interceptors. > See [Develop Workflow logic](/develop/python/workflows/basics#workflow-logic-requirements) for details. > > If you want to write generic code shared by all inbound Workflow call handlers but want to skip read-only operations, check `workflow.unsafe.is_read_only()`. > > Activity and Client interceptors are not affected by replay. > ## Register an Interceptor Registering an interceptor means supplying an interceptor instance to the SDK so Temporal can invoke it when matching Client or Worker calls occur. Once registered, the interceptor runs as part of the call path and can observe or modify request and response data. ### Register on the Client Pass interceptors in the `interceptors` argument of `Client.connect()`. Client interceptors modify outbound calls such as starting and signaling Workflows. ```python client = await Client.connect( "localhost:7233", interceptors=[TracingInterceptor()], ) ``` The `interceptors` list can contain multiple interceptors. In this case they form a chain: a method implemented on an interceptor instance in the list can perform side effects, and modify the data, before passing it on to the corresponding method on the next interceptor in the list. ### Register via a Plugin If you're building a reusable library or want to bundle interceptors with other primitives, you can register them through a [Plugin](/develop/plugins-guide#interceptors). ### Register on the Worker only If your interceptor doesn't affect the Client, you can pass interceptors in the `interceptors` argument of `Worker()`. Worker interceptors modify inbound and outbound Workflow and Activity calls. ```python worker = Worker( client, task_queue="my-task-queue", interceptors=[SomeWorkerInterceptor()], # ... ) ``` > **📝 Note:** > > If your interceptor class inherits from both `client.Interceptor` and `worker.Interceptor`, pass it to > `Client.connect()` rather than the `Worker()` constructor. The Worker will use interceptors from its underlying Client > automatically. > ## How to implement Interceptors in Python Interceptors run as a chain. Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage. ### Implementing Client call Interceptors To modify outbound Client calls, define a class inheriting from [`client.Interceptor`](https://python.temporal.io/temporalio.client.Interceptor.html), and implement the method `intercept_client()` to return an instance of [`OutboundInterceptor`](https://python.temporal.io/temporalio.client.OutboundInterceptor.html) that implements the subset of outbound Client calls that you wish to modify. This example implements an Interceptor on outbound Client calls that sets a certain key in the outbound `headers` field. A User ID is context-propagated by being sent in a header field with outbound requests: ```python class ContextPropagationInterceptor( temporalio.client.Interceptor, temporalio.worker.Interceptor ): def __init__( self, payload_converter: temporalio.converter.PayloadConverter = temporalio.converter.default().payload_converter, ) -> None: self._payload_converter = payload_converter def intercept_client( self, next: temporalio.client.OutboundInterceptor ) -> temporalio.client.OutboundInterceptor: return _ContextPropagationClientOutboundInterceptor( next, self._payload_converter ) def set_header_from_context( input: _InputWithHeaders, payload_converter: temporalio.converter.PayloadConverter ) -> None: user_id_val = user_id.get() if user_id_val: input.headers = { **input.headers, HEADER_KEY: payload_converter.to_payload(user_id_val), } class _ContextPropagationClientOutboundInterceptor( temporalio.client.OutboundInterceptor ): def __init__( self, next: temporalio.client.OutboundInterceptor, payload_converter: temporalio.converter.PayloadConverter, ) -> None: super().__init__(next) self._payload_converter = payload_converter async def start_workflow( self, input: temporalio.client.StartWorkflowInput ) -> temporalio.client.WorkflowHandle[Any, Any]: set_header_from_context(input, self._payload_converter) return await super().start_workflow(input) ``` It often happens that your Worker and Client interceptors will share code because they implement closely related logic. In the Python SDK, you will typically want to create an interceptor class that inherits from _both_ `client.Interceptor` and `worker.Interceptor` as above, since their method sets do not overlap. You can then [register](#register) this interceptor in your client/starter code. Your interceptor classes need not implement every method; the default implementation is always to pass the data on to the next method in the interceptor chain. During execution, when the SDK encounters an Inbound Activity call, it will look to the first Interceptor instance, get hold of the appropriate intercepted method, and call it. The intercepted method will perform its function then call the same method on the next Interceptor in the chain. At the end of the chain the SDK will call the "real" SDK method. ### Implementing Worker call Interceptors To modify inbound and outbound Workflow and Activity calls, define a class inheriting from `worker.Interceptor`. This is an interface with two methods named `intercept_activity` and `workflow_interceptor_class`, which you can use to configure interceptions of Activity and Workflow calls, respectively. `intercept_activity` returns an `ActivityInboundInterceptor`. This example demonstrates using an interceptor to measure [Schedule-To-Start](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) and Schedule-To-Close latency. Notice how the interceptor wraps the call: it records Schedule-To-Start before `execute_activity`, then records Schedule-To-Close after it completes: ```python from datetime import datetime, timezone from temporalio import activity from temporalio.worker import ( ActivityInboundInterceptor, ExecuteActivityInput, Interceptor, Worker, ) class SimpleWorkerInterceptor(Interceptor): def intercept_activity( self, next: ActivityInboundInterceptor ) -> ActivityInboundInterceptor: return ActivityMetricsInterceptor(next) class ActivityMetricsInterceptor(ActivityInboundInterceptor): async def execute_activity(self, input: ExecuteActivityInput): info = activity.info() meter = activity.metric_meter() attrs = {"workflow_type": info.workflow_type} # Before the activity executes: record Schedule-To-Start schedule_to_start = info.started_time - info.current_attempt_scheduled_time meter.create_histogram_timedelta( "custom_activity_schedule_to_start_latency", description="Time between activity scheduling and start", unit="duration", ).record(schedule_to_start, attrs) # Execute the activity result = await self.next.execute_activity(input) # After the activity completes: record Schedule-To-Close elapsed = datetime.now(timezone.utc) - info.current_attempt_scheduled_time meter.create_histogram_timedelta( "custom_activity_schedule_to_close_latency", description="Time between activity scheduling and completion", unit="duration", ).record(elapsed, attrs) return result client = await Client.connect( "localhost:7233", ) worker = Worker( client, interceptors=[SimpleWorkerInterceptor()], # ... ) ``` The `workflow_interceptor_class` returns a `WorkflowInboundInterceptor` that works similarly to `ActivityInboundInterceptor`. --- # Run a Worker - Python SDK Source: https://docs.temporal.io/develop/python/workers/run-worker-process > Create and run a Temporal Worker using the Python SDK. This page covers long-lived Workers that you host and run as persistent processes. For Workers that run on serverless compute like AWS Lambda, see [Serverless Workers](/develop/python/workers/serverless-workers). ## Create and run a Worker Create a `Worker` with a [Temporal Client](/develop/python/client/temporal-client), the Task Queue to poll, and the Workflows and Activities it can execute. Call `run()` to start polling. [features/snippets/worker/worker.py](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.py) ```py client = await Client.connect("localhost:7233") worker = Worker( client, task_queue="my-task-queue", workflows=[HelloWorkflow], activities=[some_activity], ) await worker.run() ``` `run()` does not return on its own. It polls until you call `shutdown()`, then returns once shutdown is complete. A Worker is also an async context manager: `async with worker:` starts it on entry and shuts it down on exit. See [Shut down a Worker](#shut-down-a-worker). ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Pass a list of Workflows in `workflows`, a list of Activities in `activities`, or both. Activities defined with `async def` run on the Worker's event loop. Activities defined with a plain `def` are synchronous and require an executor, so pass one in `activity_executor`: ```python worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_sync_activity], activity_executor=ThreadPoolExecutor(5), ) ``` The same executor can be shared across multiple Workers. ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/python/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options The `Worker` constructor takes keyword arguments that control concurrency limits, pollers, timeouts, and caching, including `max_concurrent_activities`, `max_concurrent_workflow_tasks`, and `max_cached_workflows`. The defaults work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `deployment_config`, then set a default versioning behavior for the Workflows on the Worker. [features/snippets/worker/worker.py](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.py) ```py worker = Worker( client, task_queue="my-task-queue", workflows=[HelloWorkflow], activities=[some_activity], deployment_config=WorkerDeploymentConfig( version=WorkerDeploymentVersion( deployment_name="my-app", build_id="1.0", ), use_worker_versioning=True, default_versioning_behavior=VersioningBehavior.PINNED, ), ) ``` To set the behavior per Workflow instead of on the Worker, pass `versioning_behavior` to `@workflow.defn`. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker Shut a Worker down by leaving the `async with` block, which calls `shutdown()` for you. To keep the Worker running until the process is interrupted, create an `asyncio.Event` and wait on it inside the block. Set that event from the entry point that catches `KeyboardInterrupt`: ```python interrupt_event = asyncio.Event() if __name__ == "__main__": loop = asyncio.new_event_loop() try: loop.run_until_complete(main()) except KeyboardInterrupt: interrupt_event.set() loop.run_until_complete(loop.shutdown_asyncgens()) ``` Waiting on `interrupt_event` inside the block holds the Worker open. Once the event is set, the block exits, and the Worker stops polling for new Tasks and waits for in-flight Tasks to finish, up to `graceful_shutdown_timeout`. [features/snippets/worker/worker.py](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.py) ```py worker = Worker( client, task_queue="my-task-queue", workflows=[HelloWorkflow], activities=[some_activity], graceful_shutdown_timeout=timedelta(seconds=30), ) async with worker: await interrupt_event.wait() ``` See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Serverless Workers - Python SDK Source: https://docs.temporal.io/develop/python/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the Python SDK. > **Public Preview** > AWS Lambda support is in Public Preview. GCP Cloud Run support is in Pre-release, and its APIs may change in > backwards-incompatible ways. To request Cloud Run access, create a [support ticket](/cloud/support#support-ticket) or > contact your account team, and [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear > when Cloud Run reaches Public Preview. Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes. Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**AWS Lambda**](/develop/python/workers/serverless-workers/aws-lambda) - Use the `lambda_worker` contrib package to run a Worker as a Lambda function. Covers setup, configuration, Lambda-tuned defaults, and observability. - [**GCP Cloud Run**](/develop/python/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in. --- # Serverless Workers on AWS Lambda - Python SDK Source: https://docs.temporal.io/develop/python/workers/serverless-workers/aws-lambda > Write a Temporal Worker that runs on AWS Lambda using the Python SDK lambda_worker package. > **Public Preview** The `lambda_worker` contrib package lets you run a Temporal Serverless Worker on AWS Lambda. Deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline. You register Workflows and Activities the same way you would with a standard Worker. For a full end-to-end deployment guide covering AWS IAM setup, compute configuration, and verification, see [Deploy a Serverless Worker on AWS Lambda](/production-deployment/worker-deployments/serverless-workers/aws-lambda). ## Create and run a Worker in Lambda Use the `run_worker` function to create a Lambda handler that runs a Temporal Worker. Pass a `WorkerDeploymentVersion` and a configure callback that registers your Workflows and Activities. [lambda_worker/lambda_function.py](https://github.com/temporalio/samples-python/blob/lambda-worker/lambda_worker/lambda_function.py) ```py {15-18} from activities import hello_activity from temporalio.common import WorkerDeploymentVersion from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker # ... from workflows import TASK_QUEUE, SampleWorkflow def configure(config: LambdaWorkerConfig) -> None: config.worker_config["task_queue"] = TASK_QUEUE config.worker_config["workflows"] = [SampleWorkflow] config.worker_config["activities"] = [hello_activity] # ... lambda_handler = run_worker( WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"), configure, ) ``` `run_worker` takes a `WorkerDeploymentVersion` and a configure callback, and returns a Lambda handler. The `WorkerDeploymentVersion` identifies the [Worker Deployment](/worker-versioning#deployments) and [Build ID](/worker-versioning#deployment-versions) for this Worker. The deployment name groups related Workers across versions, and the Build Id identifies a specific release of your Worker code. Worker Versioning is required for Serverless Workers. The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. Each Workflow must have a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`. Set it per-Workflow in the `@workflow.defn` decorator, or set a worker-level default with `default_versioning_behavior` in the worker config. ```python {5} from temporalio import workflow from temporalio.common import VersioningBehavior @workflow.defn(versioning_behavior=VersioningBehavior.PINNED) class MyWorkflow: @workflow.run async def run(self, input: str) -> str: ... ``` ## Configure the Temporal connection The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables. Refer to [Environment Configuration](/develop/environment-configuration) for more details. Compared with long-lived Workers, the location of the config file is resolved differently, in the following order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. The file is optional. If absent, only environment variables are used. Encrypt sensitive values like TLS keys or API keys. Refer to [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars-encryption.html) for options. ## Adjust Worker defaults for Lambda The `lambda_worker` package applies conservative defaults suited to short-lived Lambda invocations. These differ from standard Worker defaults to avoid overcommitting resources in a constrained environment. | Setting | Lambda default | |---|---| | `max_concurrent_activities` | 2 | | `max_concurrent_workflow_tasks` | 10 | | `max_concurrent_local_activities` | 2 | | `max_concurrent_nexus_tasks` | 5 | | `workflow_task_poller_behavior` | `SimpleMaximum(2)` | | `activity_task_poller_behavior` | `SimpleMaximum(1)` | | `nexus_task_poller_behavior` | `SimpleMaximum(1)` | | `graceful_shutdown_timeout` | 5 seconds | | `max_cached_workflows` | 30 | | `disable_eager_activity_execution` | Always `True` | | `shutdown_deadline_buffer` | 7 seconds | `disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. `shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. For guidance on how these values relate, see [Tuning for long-running Activities](/serverless-workers/aws-lambda#tuning-for-long-running-activities). ## Add observability with OpenTelemetry The `lambda_worker.otel` module provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. The underlying metrics and traces are the same ones the Python SDK emits in any environment. For general observability concepts and the full list of available metrics, see [Observability - Python SDK](/develop/python/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). [lambda_worker/lambda_function.py](https://github.com/temporalio/samples-python/blob/lambda-worker/lambda_worker/lambda_function.py) ```py from activities import hello_activity from temporalio.common import WorkerDeploymentVersion from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker from temporalio.contrib.aws.lambda_worker.otel import apply_defaults from workflows import TASK_QUEUE, SampleWorkflow def configure(config: LambdaWorkerConfig) -> None: config.worker_config["task_queue"] = TASK_QUEUE config.worker_config["workflows"] = [SampleWorkflow] config.worker_config["activities"] = [hello_activity] apply_defaults(config) lambda_handler = run_worker( WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"), configure, ) ``` `apply_defaults` configures both metrics and tracing. By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. To collect this telemetry, attach the [ADOT Python Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda/lambda-python) to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle the following `otel-collector-config.yaml` in your Lambda deployment package: [lambda_worker/otel-collector-config.yaml](https://github.com/temporalio/samples-python/blob/lambda-worker/lambda_worker/otel-collector-config.yaml) ```yaml receivers: otlp: protocols: grpc: endpoint: "localhost:4317" http: endpoint: "localhost:4318" exporters: debug: awsxray: region: us-west-2 awsemf: # AWS EMF exporter for metrics # These are example configurations namespace: TemporalWorkerMetrics log_group_name: /aws/lambda/ region: us-west-2 dimension_rollup_option: NoDimensionRollup resource_to_telemetry_conversion: enabled: true service: pipelines: traces: receivers: [otlp] exporters: [awsxray, debug] metrics: receivers: [otlp] exporters: [awsemf] telemetry: logs: level: debug metrics: address: localhost:8888 ``` Set the following environment variable on the Lambda function: - `OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` Enable X-Ray active tracing on the Lambda function: ```bash aws lambda update-function-configuration \ --function-name \ --tracing-config Mode=Active ``` The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Attach the [`AWSXRayDaemonWriteAccess`](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSXRayDaemonWriteAccess.html) managed policy, or add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData` permissions. Without these permissions, the Collector fails silently and no telemetry appears. If you only need metrics or tracing, use `build_metrics_telemetry_config` or `apply_tracing` individually. --- # Serverless Workers on GCP Cloud Run - Python SDK Source: https://docs.temporal.io/develop/python/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the Python SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Python Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific package. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running Python Worker, then pass `deployment_config` to `Worker()` to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```python import asyncio import os from temporalio.client import Client from temporalio.common import VersioningBehavior, WorkerDeploymentVersion from temporalio.envconfig import ClientConfig from temporalio.worker import Worker, WorkerDeploymentConfig from my_activities import my_activity from my_workflows import MyWorkflow async def main() -> None: client = await Client.connect(**ClientConfig.load_client_connect_config()) worker = Worker( client, task_queue=os.environ["TEMPORAL_TASK_QUEUE"], workflows=[MyWorkflow], activities=[my_activity], deployment_config=WorkerDeploymentConfig( version=WorkerDeploymentVersion( deployment_name="my-app", build_id="build-1", ), use_worker_versioning=True, default_versioning_behavior=VersioningBehavior.PINNED, ), ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` `deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`. Setting `default_versioning_behavior` as shown above covers every Workflow on the Worker. To set the behavior per Workflow instead, pass `versioning_behavior` to the `@workflow.defn` decorator: ```python from temporalio import workflow from temporalio.common import VersioningBehavior @workflow.defn(versioning_behavior=VersioningBehavior.PINNED) class MyWorkflow: @workflow.run async def run(self, name: str) -> str: ... ``` For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/python/workers/run-worker-process). ## Configure the Temporal connection The `temporalio.envconfig` package loads [Temporal Client](/develop/python/client/temporal-client) configuration from environment variables and an optional TOML config file, so the Worker code carries no Namespace or credentials. Set the non-secret values as environment variables on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager. For the full list of supported variables, the config file format, and profiles, see [Environment configuration](/develop/environment-configuration). `ClientConfig.load_client_connect_config()` returns the keyword arguments for `Client.connect`, which is why the Worker above unpacks it with `**`. To inspect or change values before connecting, load the profile instead and convert it yourself: ```python from temporalio.envconfig import ClientConfigProfile profile = ClientConfigProfile.load() connect_config = profile.to_client_connect_config() client = await Client.connect(**connect_config) ``` ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/python/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over: ```python from temporalio import activity @activity.defn async def my_activity(items: list[str]) -> str: for i, item in enumerate(items): activity.heartbeat(i) # ... process item return "done" ``` For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Python SDK](/develop/python/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). --- # Workflows - Python SDK Source: https://docs.temporal.io/develop/python/workflows > This section explains how to implement Workflows with the Python SDK ![Python SDK Banner](/img/assets/banner-python-temporal.png) ## Workflows - [Workflow basics](/develop/python/workflows/basics) - [Child Workflows](/develop/python/workflows/child-workflows) - [Continue-As-New](/develop/python/workflows/continue-as-new) - [Cancellation](/develop/python/workflows/cancellation) - [Timeouts](/develop/python/workflows/timeouts) - [Message passing](/develop/python/workflows/message-passing) - [Schedules](/develop/python/workflows/schedules) - [Timers](/develop/python/workflows/timers) - [Versioning](/develop/python/workflows/versioning) - [Workflow Streams](/develop/python/workflows/workflow-streams) --- # Workflow Basics - Python SDK Source: https://docs.temporal.io/develop/python/workflows/basics > This section explains Workflow Basics with the Python SDK ## Develop a basic Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal Python SDK programming model, Workflows are defined as classes. Specify the `@workflow.defn` decorator on the Workflow class to identify a Workflow. Use the `@workflow.run` to mark the entry point method to be invoked. This must be set on one asynchronous method defined on the same class as `@workflow.defn`. Run methods have positional parameters. ```python {9,11} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from your_activities_dacx import your_activity from your_dataobject_dacx import YourParams @workflow.defn(name="YourWorkflow") class YourWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( your_activity, YourParams("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` ### Define Workflow parameters Temporal Workflows may have any number of custom parameters. However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. Workflow parameters are the method parameters of the singular method decorated with `@workflow.run`. These can be any data type Temporal can convert, including [`dataclasses`](https://docs.python.org/3/library/dataclasses.html) when properly type-annotated. Technically this can be multiple parameters, but Temporal strongly encourages a single `dataclass` parameter containing all input fields. ```python {3} from dataclasses import dataclass @dataclass class YourParams: greeting: str name: str ``` ### Define Workflow return parameters Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error. To return a value of the Workflow, use `return` to return an object. To return the results of a Workflow Execution, use either `start_workflow()` or `execute_workflow()` asynchronous methods. For performance and behavior reasons, users should pass through all modules, including Activities, Nexus services, and third-party plugins, whose calls will be deterministic using [`imports_passed_through`](https://python.temporal.io/temporalio.workflow.unsafe.html#imports_passed_through) or at Worker creation time by customizing the runner's restrictions with [`with_passthrough_modules`](https://python.temporal.io/temporalio.worker.workflow_sandbox.SandboxRestrictions.html#with_passthrough_modules). ```python {5-7} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from your_activities_dacx import your_activity from your_dataobject_dacx import YourParams @workflow.defn(name="YourWorkflow") class YourWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( your_activity, YourParams("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` ### Customize your Workflow Type Workflows have a Type that are referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. You can customize the Workflow name with a custom name in the decorator argument. For example, `@workflow.defn(name="your-workflow-name")`. If the name parameter is not specified, the Workflow name defaults to the unqualified class name. ```python from temporalio import workflow with workflow.unsafe.imports_passed_through(): from your_activities_dacx import your_activity from your_dataobject_dacx import YourParams @workflow.defn(name="YourWorkflow") class YourWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( your_activity, YourParams("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` ### Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflow `__init__` method won't have any parameters. However, if you use the `@workflow.init` decorator on your `__init__` method, you can give it the same Workflow parameters as your `@workflow.run` method. The SDK will then ensure that your `__init__` method receives the Workflow input arguments that the Client sent. The Workflow input arguments are also passed to your `@workflow.run` method. That always happens, whether or not you use the `@workflow.init` decorator. Here's an example. Notice that `__init__` and `get_greeting` must have the same parameters, with the same type annotations: ```python @dataclass class MyWorkflowInput: name: str @workflow.defn class WorkflowRunSeesWorkflowInitWorkflow: @workflow.init def __init__(self, workflow_input: MyWorkflowInput) -> None: self.name_with_title = f"Knight {workflow_input.name}" self.title_has_been_checked = False @workflow.run async def get_greeting(self, workflow_input: MyWorkflowInput) -> str: await workflow.wait_condition(lambda: self.title_has_been_checked) return f"Hello, {self.name_with_title}" # other Workflow code ``` ### Develop Workflow logic Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). Each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code. Workflow code must be deterministic because the Temporal Server may [replay](/develop/python/best-practices/testing-suite#replay) your Workflow to reconstruct its state. This means: - no threading - no randomness - no external calls to processes - no network I/O - no global state mutation - no system date or time All API safe for Workflows used in the [`temporalio.workflow`](https://python.temporal.io/temporalio.workflow.html) must run in the implicit [`asyncio` event loop](https://docs.python.org/3/library/asyncio-eventloop.html) and be _deterministic_. The SDK provides replay-safe alternatives for common needs: #### Logging Use [`workflow.logger`](https://python.temporal.io/temporalio.workflow.html#logger) instead of `print()` or the standard `logging` module. The SDK logger automatically suppresses log messages during replay to avoid duplicates: ```python @workflow.defn class MyWorkflow: @workflow.run async def run(self, name: str) -> str: workflow.logger.info("Starting workflow", name) # ... ``` For logger configuration, see [Observability: Log from a Workflow](/develop/python/platform/observability#logging). #### Random numbers and UUIDs Use [`workflow.random()`](https://python.temporal.io/temporalio.workflow.html#random) to get a deterministic `random.Random` instance seeded per Workflow Execution. Never use `random.random()` or other `random` module functions directly. For UUIDs, use [`workflow.uuid4()`](https://python.temporal.io/temporalio.workflow.html#uuid4) instead of `uuid.uuid4()`: ```python value = workflow.random().randint(1, 100) unique_id = workflow.uuid4() ``` #### Current time Use [`workflow.now()`](https://python.temporal.io/temporalio.workflow.html#now) instead of `datetime.now()` or `time.time()`. The SDK returns the time of the last Workflow Task, which is consistent across replays: ```python current_time = workflow.now() ``` #### Detecting replay (advanced) Use [`workflow.unsafe.is_replaying`](https://python.temporal.io/temporalio.workflow.html#is_replaying) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an [Interceptor](/develop/python/workers/interceptors). > **⚠️ Caution:** > > Never use this to affect Workflow business logic — branching on replay status breaks determinism. > ```python if not workflow.unsafe.is_replaying(): emit_metric("workflow_started", 1) ``` If your goal is to always take action when something new is happening, check that `workflow.unsafe.is_replaying_history_events()` is false instead. This will be false during read-only operations like queries and update validators. This is what the SDK's built-in logger and tracing interceptors use internally. --- # Cancellation - Python SDK Source: https://docs.temporal.io/develop/python/workflows/cancellation You can interrupt a Workflow Execution in one of the following ways: - [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution. - [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. ## Cancel a Workflow Execution Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a process. - The system records a `WorkflowExecutionCancelRequested` event in the Event History. - A Workflow Task gets scheduled to process the cancelation. - The Workflow code can handle the cancelation and execute any cleanup logic. - The system doesn't forcefully stop the Workflow. To cancel a Workflow Execution in Python, use the [cancel()](https://python.temporal.io/temporalio.client.WorkflowHandle.html#cancel) function on the Workflow handle. ```python await client.get_workflow_handle("your_workflow_id").cancel() ``` ### Cancel an Activity from a Workflow Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate Activity is executed, the Activity Execution should send Heartbeats and set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is still working. When an Activity is canceled, an error is raised in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done in a `finally` clause or inside a caught cancel error. However, for the Activity to appear canceled the exception needs to be re-raised. > **📝 Note:** > > Unlike regular Activities, [Local Activities](/local-activity) can be canceled if they don't send Heartbeats. Local > Activities are handled locally, and all the information needed to handle the cancellation logic is available in the same > Worker process. > To cancel an Activity from a Workflow Execution, call the [cancel()](https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel) method on the Activity handle that is returned from [start_activity()](https://python.temporal.io/temporalio.workflow.html#start_activity). ```python @activity.defn async def cancellable_activity(input: ComposeArgsInput) -> NoReturn: try: while True: print("Heartbeating cancel activity") await asyncio.sleep(0.5) activity.heartbeat("some details") except asyncio.CancelledError: print("Activity cancelled") raise @activity.defn async def run_activity(input: ComposeArgsInput): print("Executing activity") return input.arg1 + input.arg2 @workflow.defn class GreetingWorkflow: @workflow.run async def run(self, input: ComposeArgsInput) -> None: activity_handle = workflow.start_activity( cancellable_activity, ComposeArgsInput(input.arg1, input.arg2), start_to_close_timeout=timedelta(minutes=5), heartbeat_timeout=timedelta(seconds=30), ) await asyncio.sleep(3) activity_handle.cancel() ``` > **📝 Note:** > > The Activity handle is a Python task. By calling `cancel()`, you're essentially requesting the task to be canceled. > ## Terminate a Workflow Execution Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. To terminate a Workflow Execution in Python, use the [terminate()](https://python.temporal.io/temporalio.client.WorkflowHandle.html#terminate) function on the Workflow handle. ```python await client.get_workflow_handle("your_workflow_id").terminate() ``` ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - Python SDK Source: https://docs.temporal.io/develop/python/workflows/child-workflows > Start a Child Workflow Execution and set a Parent Close Policy using the Temporal Python SDK. Ensure proper progress logging and specify Parent Workflow behavior upon closure. This page shows how to do the following: - [Start a Child Workflow Execution](#child-workflows) - [Set a Parent Close Policy](#parent-close-policy) ## Start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In Python, awaiting `start_child_workflow()` or `execute_child_workflow()` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves. If you start a Child Workflow from a non-main coroutine (for example, a Signal or Update handler), make sure the Parent Workflow doesn't complete before that call resolves. To spawn a Child Workflow Execution in Python, use the [`execute_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#execute_child_workflow) function which starts the Child Workflow and waits for completion or use the [`start_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#start_child_workflow) function to start a Child Workflow and return its handle. This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running. > **📝 Note:** > > `execute_child_workflow()` is a helper function for `start_child_workflow()` plus `await handle`. > ```python from temporalio import workflow from dataobject import ComposeGreetingInput from temporalio.workflow import ParentClosePolicy @workflow.defn class ComposeGreetingWorkflow: @workflow.run async def run(self, input: ComposeGreetingInput) -> str: return f"{input.greeting}, {input.name}!" @workflow.defn class GreetingWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_child_workflow( ComposeGreetingWorkflow.run, ComposeGreetingInput("Hello", name), id="hello-child-workflow-workflow-child-id", parent_close_policy=ParentClosePolicy.ABANDON, ) ``` ### Set a Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. Set the `parent_close_policy` parameter inside the [`start_child_workflow`](https://python.temporal.io/temporalio.workflow.html#start_child_workflow) function or the [`execute_child_workflow()`](https://python.temporal.io/temporalio.workflow.html#execute_child_workflow) function to specify the behavior of the Child Workflow when the Parent Workflow closes. ```python {1,19} from temporalio.workflow import ParentClosePolicy # ... @workflow.defn class ComposeGreetingWorkflow: @workflow.run async def run(self, input: ComposeGreetingInput) -> str: return f"{input.greeting}, {input.name}!" @workflow.defn class GreetingWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_child_workflow( ComposeGreetingWorkflow.run, ComposeGreetingInput("Hello", name), id="hello-child-workflow-workflow-child-id", parent_close_policy=ParentClosePolicy.ABANDON, ) ``` --- # Continue-As-New - Python SDK Source: https://docs.temporal.io/develop/python/workflows/continue-as-new > Use Temporal's Continue-As-New in Python to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for Python developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) - [How to test Continue-as-New?](#how-to-test) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the Python SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically set to `None` for the original caller of the Workflow. [View the source code](https://github.com/temporalio/samples-python/blob/main/message_passing/safe_message_handlers/workflow.py) in the context of the rest of the application code. ```python @dataclass class ClusterManagerInput: state: Optional[ClusterManagerState] = None test_continue_as_new: bool = False @workflow.run async def run(self, input: ClusterManagerInput) -> ClusterManagerResult: ```` The test hook in the above snippet is covered [below](#how-to-test). Inside your Workflow, call the [`continue_as_new()`](https://python.temporal.io/temporalio.workflow.html#continue_as_new) function with the same type. This stops the Workflow right away and starts a new one. [View the source code](https://github.com/temporalio/samples-python/blob/main/message_passing/safe_message_handlers/workflow.py) in the context of the rest of the application code. ```python workflow.continue_as_new( ClusterManagerInput( state=self.state, test_continue_as_new=input.test_continue_as_new, ) ) ```` ### Considerations for Workflows with Message Handlers If you use Updates or Signals, don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish in your main Workflow before you run `continue_as_new`. See the [`all_handlers_finished`](message-passing#wait-for-message-handlers) example for guidance. ## When is it right to Continue-as-New using the Python SDK? Use Continue-as-New when your Workflow might encounter degraded performance or [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `workflow.info().is_continue_as_new_suggested()` to check if it's time. ## How to test Continue-as-New using the Python SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, when `test_continue_as_new == True`, this sample creates a test-only variable called `self.max_history_length` and sets it to a small value. A helper method in the Workflow checks it each time it considers using Continue-as-New: [View the source code](https://github.com/temporalio/samples-python/blob/main/message_passing/safe_message_handlers/workflow.py) in the context of the rest of the application code. ```python def should_continue_as_new(self) -> bool: if workflow.info().is_continue_as_new_suggested(): return True # For testing if ( self.max_history_length and workflow.info().get_current_history_length() > self.max_history_length ): return True return False ``` --- # Workflow message passing - Python SDK Source: https://docs.temporal.io/develop/python/workflows/message-passing > Develop with Queries, Signals, and Updates with the Temporal Python SDK. A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates. The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values. Temporal Clients use messages to read Workflow state and control its execution. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal Python SDK. ## Write message handlers > **ℹ️ Info:** > The code that follows is part of a working message passing [sample](https://github.com/temporalio/samples-python/tree/main/message_passing/introduction). Follow these guidelines when writing your message handlers: - Message handlers are defined as methods on the Workflow class, using one of the three decorators: [`@workflow.query`](https://python.temporal.io/temporalio.workflow.html#query), [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal), and [`@workflow.update`](https://python.temporal.io/temporalio.workflow.html#update). - The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion). - Prefer [data classes](https://docs.python.org/3/library/dataclasses.html) to multiple input parameters. Data class parameters allow you to add fields without changing the calling signature. Keep in mind that serialization and deserialization can fail with the default data converter if the new field does not have a default value. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution: ```python class Language(IntEnum): Chinese = 1 English = 2 French = 3 @dataclass class GetLanguagesInput: include_unsupported: bool @workflow.defn class GreetingWorkflow: def __init__(self) -> None: self.greetings = { Language.CHINESE: "你好，世界", Language.ENGLISH: "Hello, world", } @workflow.query def get_languages(self, input: GetLanguagesInput) -> list[Language]: # 👉 A Query handler returns a value: it can inspect but must not mutate the Workflow state. if input.include_unsupported: return list(Language) else: return list(self.greetings) ``` - The Query decorator can accept arguments. Refer to the API docs: [`@workflow.query`](https://python.temporal.io/temporalio.workflow.html#query). - A Query handler uses `def`, not `async def`. You can't perform async operations like executing an Activity in a Query handler. ### Signal handlers A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow: ```python @dataclass class ApproveInput: name: str @workflow.defn class GreetingWorkflow: ... @workflow.signal def approve(self, input: ApproveInput) -> None: # 👉 A Signal handler mutates the Workflow state but cannot return a value. self.approved_for_release = True self.approver_name = input.name ``` - The Signal decorator can accept arguments. Refer to the API docs: [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal). - The handler should not return a value. The response is sent immediately from the server, without waiting for the Workflow to process the Signal. - Signal (and Update) handlers can be `async def`. This allows you to use Activities, Child Workflows, durable [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers. ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```python class Language(IntEnum): Chinese = 1 English = 2 French = 3 @workflow.defn class GreetingWorkflow: ... @workflow.update def set_language(self, language: Language) -> Language: # 👉 An Update handler can mutate the Workflow state and return a value. previous_language, self.language = self.language, language return previous_language @set_language.validator def validate_language(self, language: Language) -> None: if language not in self.greetings: # 👉 In an Update validator you raise any exception to reject the Update. raise ValueError(f"{language.name} is not supported") ``` - The Update decorator can take arguments (like, `name`, `dynamic` and `unfinished_policy`) as described in the API reference docs for [`workflow.update`](https://python.temporal.io/temporalio.workflow.html#update). - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them. - The SDK automatically provides a validator decorator named `@.validator`. The validator must accept the same argument types as the handler and return `None`. - Accepting and rejecting Updates with validators: - To reject an Update, raise an exception of any type in the validator. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic. - When a Validator raises an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History. The caller receives an "Update failed" error. - Use [`workflow.current_update_info`](https://python.temporal.io/temporalio.workflow.html#current_update_info) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - Update (and Signal) handlers can be `async def`, letting them use Activities, Child Workflows, durable [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for safe usage guidelines. ## Send messages To send Queries, Signals, or Updates, you call methods on a [WorkflowHandle](https://python.temporal.io/temporalio.client.WorkflowHandle.html) object: - Use [start_workflow](https://python.temporal.io/temporalio.client.Client.html#start_workflow) to start a Workflow and return its handle. - Use [get_workflow_handle_for](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle_for) to retrieve a typed Workflow handle by its Workflow Id. For example: ```python client = await Client.connect("localhost:7233") workflow_handle = await client.start_workflow( GreetingWorkflow.run, id="greeting-workflow-1234", task_queue="my-task-queue" ) ``` To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition. > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution. > ### Send a Query Use [`WorkflowHandle.query`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#query) to send a Query to a Workflow Execution: ```python supported_languages = await workflow_handle.query( GreetingWorkflow.get_languages, GetLanguagesInput(supported_only=True) ) ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### Send a Signal from a Client Use [`WorkflowHandle.signal`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#signal) to send a Signal: ```python await workflow_handle.signal(GreetingWorkflow.approve, ApproveInput(name="me")) ``` - The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution. - The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History. #### Send a Signal from a Workflow A Workflow can send a Signal to another Workflow, known as an _External Signal_. You'll need a Workflow handle for the external Workflow. Use [`get_external_workflow_handle_for`](https://python.temporal.io/temporalio.workflow.html#get_external_workflow_handle_for): ```python {20-25} from typing import Optional from temporalio import workflow @workflow.defn class WorkflowA: def __init__(self) -> None: self._signal: Optional[str] = None @workflow.run async def run(self) -> str: await workflow.wait_condition(lambda: self._signal is not None) return self._signal @workflow.signal def your_signal(self, value: str) -> None: self._signal = value @workflow.defn class WorkflowB: @workflow.run async def run(self) -> None: handle = workflow.get_external_workflow_handle_for(WorkflowA.run, "workflow-a") await handle.signal(WorkflowA.your_signal, "signal argument") ``` When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. #### Signal-With-Start Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running. To use Signal-With-Start, call the [`start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) method and pass the `start_signal` argument with the name of your Signal: ```python {6-14} import asyncio from temporalio.client import Client from wf_signal import GreetingWorkflow async def main(): client = await Client.connect("localhost:7233") await client.start_workflow( GreetingWorkflow.run, id="your-signal-with-start-workflow", task_queue="signal-tq", start_signal="submit_greeting", start_signal_args=["User Signal with Start"], ) handle = client.get_workflow_handle_for( GreetingWorkflow, "your-signal-with-start-workflow" ) await handle.signal(GreetingWorkflow.exit) result = await handle.result() print(result) # ... ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. To send an Update to a Workflow Execution, you can: - Call [`execute_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#execute_update) and wait for the Update to complete. This code fetches an Update result: ```python previous_language = await workflow_handle.execute_update( GreetingWorkflow.set_language, Language.Chinese ) ``` - Send [`start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update) to receive an [`UpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html) as soon as the Update is accepted. - Use this `UpdateHandle` later to fetch your results. - `async def` Update handlers normally perform long-running asynchronous operations, such as executing an Activity. - `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete. For example: ```python # Wait until the update is accepted update_handle = await workflow_handle.start_update( HelloWorldWorkflow.set_greeting, HelloWorldInput("World"), wait_for_stage=client.WorkflowUpdateStage.ACCEPTED, ) # Wait until the update is completed update_result = await update_handle.result() ``` For more details, see the "Async handlers" section. To obtain an Update handle, you can: - Use [`start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update) to start an Update and return the handle, as shown in the preceding example. - Use [`get_update_handle_for`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#get_update_handle_for) to fetch a handle for an in-progress Update using the Update ID. #### Update-With-Start > **💡 Tip:** > > For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended. > [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/python/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. Use [`execute_update_with_start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_update_with_start_workflow) to start the Update and wait for the result in one go. Alternatively, use [`start_update_with_start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_update_with_start_workflow) to start the Update and receive a [`WorkflowUpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html), and then use `await update_handle.result()` to retrieve the result from the Update. These calls return once the requested Update wait stage has been reached, or when the request times out. You will need to provide a [`WithStartWorkflowOperation`](https://python.temporal.io/temporalio.client.WithStartWorkflowOperation.html) to define the Workflow that will be started if necessary, and its arguments. You must specify a [WorkflowIdConflictPolicy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`. Note that a `WithStartWorkflowOperation` can only be used once. Here's an example taken from the [lazy_initialization](https://github.com/temporalio/samples-python/blob/main/message_passing/update_with_start/lazy_initialization/starter.py) sample: ```python start_op = WithStartWorkflowOperation( ShoppingCartWorkflow.run, id=cart_id, id_conflict_policy=common.WorkflowIDConflictPolicy.USE_EXISTING, task_queue="my-task-queue", ) try: price = Decimal( await temporal_client.execute_update_with_start_workflow( ShoppingCartWorkflow.add_item, ShoppingCartItem(sku=item_id, quantity=quantity), start_workflow_operation=start_op, ) ) except WorkflowUpdateFailedError: price = None workflow_handle = await start_op.workflow_handle() return price, workflow_handle ``` > **ℹ️ Info:** > SEND MESSAGES WITHOUT TYPE SAFETY > > In real-world development, sometimes you may be unable to import Workflow Definition method signatures. > When you don't have access to the Workflow Definition or it isn't written in Python, you can still use APIs that aren't type-safe, and dynamic method invocation. > Pass method names instead of method objects to: > > - [`Client.start_workflow`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) > - [`WorkflowHandle.query`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#query) > - [`WorkflowHandle.signal`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#signal) > - [`WorkflowHandle.execute_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#execute_update) > - [`WorkflowHandle.start_update`](https://python.temporal.io/temporalio.client.WorkflowHandle.html#start_update) > > Use these non-type safe APIs: > > - [`get_workflow_handle`](https://python.temporal.io/temporalio.client.Client.html#get_workflow_handle) > - [`get_external_workflow_handle`](https://python.temporal.io/temporalio.workflow.html#get_external_workflow_handle). > ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow), [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing), and [this sample](https://github.com/temporalio/samples-python/blob/message-passing/message_passing/safe_message_handlers/README.md) demonstrating safe `async` message handling. > ### Use async handlers Signal and Update handlers can be `async def` as well as `def`. Using `async def` allows you to use `await` with Activities, Child Workflows, [`asyncio.sleep`](https://docs.python.org/3/library/asyncio-task.html#asyncio.sleep) Timers, [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) conditions, etc. This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at `await` calls. It's essential to understand the things that could go wrong in order to use `async def` handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, the [Safe message handlers](https://github.com/temporalio/samples-python/tree/main/message_passing/safe_message_handlers) sample, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below. The following code executes an Activity that makes a network call to a remote service. It modifies the Update handler from earlier on this page, turning it into an `async def`: ```python @activity.defn async def call_greeting_service(to_language: Language) -> Optional[str]: await asyncio.sleep(0.2) # Pretend that we are calling a remote service. greetings = { Language.Arabic: "مرحبا بالعالم", Language.Chinese: "你好，世界", Language.English: "Hello, world", Language.French: "Bonjour, monde", Language.Hindi: "नमस्ते दुनिया", Language.Spanish: "Hola mundo", } return greetings.get(to_language) @workflow.defn class GreetingWorkflow: def __init__(self) -> None: self.lock = asyncio.Lock() ... ... @workflow.update async def set_language(self, language: Language) -> Language: if language not in self.greetings: # 👉 Use a lock here to ensure that multiple calls to set_language are processed in order. async with self.lock: greeting = await workflow.execute_activity( call_greeting_service, language, start_to_close_timeout=timedelta(seconds=10), ) if greeting is None: # 👉 An update validator cannot be async, so cannot be used to check that the remote # call_greeting_service supports the requested language. Raising ApplicationError # will fail the Update, but the WorkflowExecutionUpdateAccepted event will still be # added to history. raise ApplicationError( f"Greeting service does not support {language.name}" ) self.greetings[language] = greeting previous_language, self.language = self.language, language return previous_language ``` After updating the code to use a `async def`, your Update handler can schedule an Activity and await the result. Although an `async def` Signal handler can also execute an Activity, using an Update handler allows the Client to receive a result or error once the Activity completes. This lets your client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, etc. ### Add wait conditions to block Sometimes, `async def` Signal or Update handlers need to meet certain conditions before they should continue. You can use [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) to prevent the code from proceeding until a condition is true. You specify the condition by passing a function that returns `True` or `False`, and you can optionally set a timeout. This is an important feature that helps you control your handler logic. Here are three important use cases for `workflow.wait_condition`: - Wait for a Signal or Update to arrive. - Wait in a handler until it's appropriate to continue. - Wait in the main Workflow until all active handlers have finished. #### Wait for a Signal or Update to arrive It's common to use `workflow.condition` to wait for a particular Signal or Update to be sent by a Client: ```python @workflow.defn class GreetingWorkflow: def __init__(self) -> None: self.approved_for_release = False self.approver_name: Optional[str] = None @workflow.signal def approve(self, input: ApproveInput) -> None: self.approved_for_release = True self.approver_name = input.name @workflow.run async def run(self) -> str: await workflow.wait_condition(lambda: self.approved_for_release) ... return self.greetings[self.language] ``` #### Use wait conditions in handlers It's common to use a Workflow wait condition to wait until a handler should start. You can also use wait conditions anywhere else in the handler to wait for a specific condition to become `True`. This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become `True`. Consider a `ready_for_update_to_execute` method that runs before your Update handler executes. The `workflow.wait_condition` method waits until your condition is met: ```python @workflow.update async def my_update(self, update_input: UpdateInput) -> str: await workflow.wait_condition( lambda: self.ready_for_update_to_execute(update_input) ) ``` You can also use wait conditions anywhere else in the handler to wait for a specific condition to become true. This allows you to write handlers that pause at multiple points, each time waiting for a required condition to become true. #### Ensure your handlers finish before the Workflow completes Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses `async def` Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result. The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results. Use [`workflow.wait_condition`](https://python.temporal.io/temporalio.workflow.html#wait_condition) and [`all_handlers_finished`](https://python.temporal.io/temporalio.workflow.html#all_handlers_finished) to address this problem and allow your Workflow to end smoothly: ```python @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> str: ... await workflow.wait_condition(workflow.all_handlers_finished) return "workflow-result" ``` By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions. You can silence these warnings on a per-handler basis by passing the `unfinished_policy` argument to the [`@workflow.signal`](https://python.temporal.io/temporalio.workflow.html#signal) / [`workflow.update`](https://python.temporal.io/temporalio.workflow.html#update) decorator: ```python @workflow.update(unfinished_policy=workflow.HandlerUnfinishedPolicy.ABANDON) async def my_update(self) -> None: ... ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. ### Use `@workflow.init` to operate on Workflow input before any handler executes Normally, your Workflow `__init__` method won't have any parameters. However, if you use the `@workflow.init` decorator on your `__init__` method, you can give it the same [Workflow parameters](/develop/python/workflows/basics#workflow-parameters) as your `@workflow.run` method. The SDK will then ensure that your `__init__` method receives the Workflow input arguments that the [Client sent](/develop/python/client/temporal-client#start-workflow-execution). (The Workflow input arguments are also passed to your `@workflow.run` method -- that always happens, whether or not you use the `@workflow.init` decorator.) This is useful if you have message handlers that need access to workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Here's an example. Notice that `__init__` and `get_greeting` must have the same parameters, with the same type annotations: ```python @dataclass class MyWorkflowInput: name: str @workflow.defn class WorkflowRunSeesWorkflowInitWorkflow: @workflow.init def __init__(self, workflow_input: MyWorkflowInput) -> None: self.name_with_title = f"Sir {workflow_input.name}" self.title_has_been_checked = False @workflow.run async def get_greeting(self, workflow_input: MyWorkflowInput) -> str: await workflow.wait_condition(lambda: self.title_has_been_checked) return f"Hello, {self.name_with_title}" @workflow.update async def check_title_validity(self) -> bool: # 👉 The handler is now guaranteed to see the workflow input # after it has been processed by __init__. is_valid = await workflow.execute_activity( check_title_validity, self.name_with_title, schedule_to_close_timeout=timedelta(seconds=10), ) self.title_has_been_checked = True return is_valid ``` ### Use `asyncio.Lock` to prevent concurrent handler execution Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```python @workflow.defn class MyWorkflow: @workflow.signal async def bad_async_handler(self): data = await workflow.execute_activity( fetch_data, start_to_close_timeout=timedelta(seconds=10) ) self.x = data.x # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then # there may be times when the Workflow has self.x from one Activity execution and self.y from another. await asyncio.sleep(1) # or await anything else self.y = data.y ``` Coordinating access using `asyncio.Lock` corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```python @workflow.defn class MyWorkflow: def __init__(self) -> None: ... self.lock = asyncio.Lock() ... @workflow.signal async def safe_async_handler(self): async with self.lock: data = await workflow.execute_activity( fetch_data, start_to_close_timeout=timedelta(seconds=10) ) self.x = data.x # ✅ OK: the scheduler may switch now to a different handler execution, or to the main workflow # method, but no other execution of this handler can run until this execution finishes. await asyncio.sleep(1) # or await anything else self.y = data.y ``` ## Message handler troubleshooting When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The client can't contact the server**: You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `UNAVAILABLE` (after some retries; see the `retry_config` argument to [`Client.connect`](https://python.temporal.io/temporalio.client.Client.html#connect)). - **The workflow does not exist**: You'll receive an [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `NOT_FOUND`. See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Python-specific discussion of this topic. ### Problems when sending a Signal When using Signal, the only exceptions that will result from your requests during its execution are the `RPCError`s described above. For Queries and Updates, the Client waits for a response from the Worker, and therefore additional errors may occur during the handler Execution by the Worker. ### Problems when sending an Update When working with Updates, in addition to the `RPCError`s described above, you may encounter these errors: - **No Workflow Workers are polling the Task Queue**: Your request will be retried by the SDK Client indefinitely. You can use [`asyncio.timeout`](https://docs.python.org/3/library/asyncio-task.html#timeouts) to impose a timeout. This raises a [`temporalio.client.WorkflowUpdateRPCTimeoutOrCancelledError`](https://python.temporal.io/temporalio.client.WorkflowUpdateRPCTimeoutOrCancelledError.html) exception. - **Update failed**: You'll receive a [`temporalio.client.WorkflowUpdateFailedError`](https://python.temporal.io/temporalio.client.WorkflowUpdateFailedError.html) exception. There are two ways this can happen: - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler. - The Update failed after having been accepted. Update failures are like [Workflow failures](/references/failures#errors-in-workflows). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity (if the Activity retries have been set to a finite number) - The Workflow author raising [`ApplicationError`](/references/failures#application-failure) - Any error listed in [workflow_failure_exception_types](https://python.temporal.io/temporalio.worker.Worker.html) (empty by default) - **The handler caused the Workflow Task to fail**: A [Workflow Task Failure](/references/failures#errors-in-workflows) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception. - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use an [`UpdateHandle`](https://python.temporal.io/temporalio.client.WorkflowUpdateHandle.html) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception with a `status` attribute of [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `NOT_FOUND`. This happens if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Problems when sending a Query When working with Queries, in addition to the `RPCError`s described above, you may encounter these errors: - **There is no Workflow Worker polling the Task Queue**: You'll receive a [`temporalio.service.RPCError`](https://python.temporal.io/temporalio.service.RPCError.html) exception on which the `status` attribute is [`RPCStatusCode`](https://python.temporal.io/temporalio.service.RPCStatusCode.html) `FAILED_PRECONDITION`. - **Query failed**: You'll receive a [`temporalio.client.WorkflowQueryFailedError`](https://python.temporal.io/temporalio.client.WorkflowQueryFailedError.html) exception if something goes wrong during a Query. Any exception in a Query handler will trigger this error. This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. ## Dynamic components A dynamic Workflow, Activity, Signal, Update, or Query is a kind of unnamed item. Normally, these items are registered by name with the Worker and invoked at runtime. When an unregistered or unrecognized Workflow, Activity, or message request arrives with a recognized method signature, the Worker can use a pre-registered dynamic stand-in. For example, you might send a request to start a Workflow named "MyUnknownWorkflow". After receiving a Workflow Task, the Worker may find that there's no registered Workflow Definitions of that type. It then checks to see if there's a registered dynamic Workflow. If the dynamic Workflow signature matches the incoming Workflow signature, the Worker invokes that just as it would invoke a non-dynamic statically named version. By registering dynamic versions of your Temporal components, the Worker can fall back to these alternate implementations for name mismatches. > **⚠️ Caution:** > > Use dynamic elements judiciously and as a fallback mechanism, not a primary design. > They can introduce long-term maintainability and debugging issues. > Reserve dynamic invocation use for cases where a name is not or can't be known at compile time. > ### Set a dynamic Signal, Query, or Update handler A dynamic Signal, Query, or Update refers to a special stand-in handler. It's used when an unregistered handler request arrives. Consider a Signal, where you might send something like `workflow.signal(MyWorkflow.my_signal_method, my_arg)`. This is a type-safe compiler-checked approach that guarantees a method exists. There's also a non-type-safe string-based form: `workflow.signal('some-name', my_arg)`. When sent to the server, the name is checked only after arriving at the Worker. This is where "dynamic handlers" come in. After failing to find a handler with a matching name and type, the Worker checks for a registered dynamic stand-in handler. If found, the Worker uses that instead. You must opt handlers into dynamic access. Add `dynamic=True` to the handler decorator (for example, `@workflow.signal(dynamic=True)`) to make a handler dynamic. The handler's signature must accept `(self, name: str, args: Sequence[RawValue])`. Use a [payload_converter](https://python.temporal.io/temporalio.workflow.html#payload_converter) function to convert `RawValue` objects to your required type. For example: ```python from typing import Sequence from temporalio.common import RawValue ... @workflow.signal(dynamic=True) async def dynamic_signal(self, name: str, args: Sequence[RawValue]) -> None: ... ``` This sample creates a `dynamic_signal` Signal. When an unregistered or unrecognized Signal arrives with a matching signature, dynamic assignment uses this handler to manage the Signal. It is responsible for transforming the sequence contents into usable data in a form that the method's logic can process and act on. ### Set a dynamic Workflow A dynamic Workflow refers to a special stand-in Workflow Definition. It's used when an unknown Workflow Execution request arrives. Consider the "MyUnknownWorkflow" example described earlier. The Worker may find there's no registered Workflow Definitions of that name or type. After failing to find a Workflow Definition with a matching type, the Worker looks for a dynamic stand-in. If found, it invokes that instead. To participate, your Workflow must opt into dynamic access. Adding `dynamic=True` to the `@workflow.defn` decorator makes the Workflow Definition eligible to participate in dynamic invocation. You must register the Workflow with the [Worker](https://python.temporal.io/temporalio.worker.html) before it can be invoked. The Workflow Definition's primary Workflow method must accept a single argument of type `Sequence[temporalio.common.RawValue]`. Use a [payload_converter](https://python.temporal.io/temporalio.workflow.html#payload_converter) function to convert `RawValue` objects to your required type. For example: ```python {19-30} from dataclasses import dataclass from datetime import timedelta from typing import Sequence from temporalio import activity, workflow from temporalio.common import RawValue @dataclass class YourDataClass: greeting: str name: str @activity.defn() async def default_greeting(input: YourDataClass) -> str: return f"{input.greeting}, {input.name}!\nActivity Type: {activity.info().activity_type}" @workflow.defn(dynamic=True) class DynamicWorkflow: @workflow.run async def run(self, args: Sequence[RawValue]) -> str: name = workflow.payload_converter().from_payload(args[0].payload, str) return await workflow.execute_activity( default_greeting, YourDataClass("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` This Workflow converts the first `Sequence` element to a string, and uses that to execute an Activity. ### Set a dynamic Activity A dynamic Activity is a stand-in implementation. It's used when an Activity Task with an unknown Activity type is received by the Worker. To participate, your Activity must opt into dynamic access. Adding `dynamic=True` to the `@activity.defn` decorator makes the Activity Definition eligible to participate in dynamic invocation. You must register the Activity with the [Worker](https://python.temporal.io/temporalio.worker.html) before it can be invoked. The Activity Definition must then accept a single argument of type `Sequence[temporalio.common.RawValue]`. Use a [payload_converter](https://python.temporal.io/temporalio.activity.html#payload_converter) function to convert `RawValue` objects to your required types. For example: ```python {14-19,27-35} from dataclasses import dataclass from datetime import timedelta from typing import Sequence from temporalio import activity, workflow from temporalio.common import RawValue @dataclass class YourDataClass: greeting: str name: str @activity.defn(dynamic=True) async def dynamic_greeting(args: Sequence[RawValue]) -> str: arg1 = activity.payload_converter().from_payload(args[0].payload, YourDataClass) return ( f"{arg1.greeting}, {arg1.name}!\nActivity Type: {activity.info().activity_type}" ) @activity.defn() async def default_greeting(input: YourDataClass) -> str: return f"{input.greeting}, {input.name}!\nActivity Type: {activity.info().activity_type}" @workflow.defn class GreetingWorkflow: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( "unregistered_activity", YourDataClass("Hello", name), start_to_close_timeout=timedelta(seconds=10), ) ``` This example invokes an unregistered Activity by name. The Worker resolves it using the registered dynamic Activity instead. When possible, prefer to use compiler-checked type-safe arguments rather than Activity name strings. --- # Schedules - Python SDK Source: https://docs.temporal.io/develop/python/workflows/schedules > Schedule, Create, Backfill, Delete, Describe, List, Pause, Trigger, and Update a Scheduled Workflow, along with Temporal Cron Jobs and Start Delay options. This page shows how to do the following: - [Schedule a Workflow](#schedule-a-workflow) - [Create a Scheduled Workflow](#create) - [Backfill a Scheduled Workflow](#backfill) - [Delete a Scheduled Workflow](#delete) - [Describe a Scheduled Workflow](#describe) - [List a Scheduled Workflow](#list) - [Pause a Scheduled Workflow](#pause) - [Trigger a Scheduled Workflow](#trigger) - [Update a Scheduled Workflow](#update) - [Temporal Cron Jobs](#temporal-cron-jobs) - [Start Delay](#start-delay) ## Schedule a Workflow Scheduling Workflows is a crucial aspect of any automation process, especially when dealing with time-sensitive tasks. By scheduling a Workflow, you can automate repetitive tasks, reduce the need for manual intervention, and ensure timely execution of your business processes. Use any of the following actions to help Schedule a Workflow Execution and take control over your automation process. Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy). If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions. ### Create a Scheduled Workflow The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands. To create a Scheduled Workflow Execution in Python, use the [create_schedule()](https://python.temporal.io/temporalio.client.Client.html#create_schedule) asynchronous method on the Client. Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution. Set the `action` parameter to `ScheduleActionStartWorkflow` to start a Workflow Execution. Optionally, you can set the `spec` parameter to `ScheduleSpec` to specify the schedule or set the `intervals` parameter to `ScheduleIntervalSpec` to specify the interval. Other options include: `cron_expressions`, `skip`, `start_at`, and `jitter`. ```python {18-32} import asyncio from datetime import timedelta from temporalio.client import ( Client, Schedule, ScheduleActionStartWorkflow, ScheduleIntervalSpec, ScheduleSpec, ScheduleState, ) from your_workflows import YourSchedulesWorkflow async def main(): client = await Client.connect("localhost:7233") await client.create_schedule( "workflow-schedule-id", Schedule( action=ScheduleActionStartWorkflow( YourSchedulesWorkflow.run, "my schedule arg", id="schedules-workflow-id", task_queue="schedules-task-queue", ), spec=ScheduleSpec( intervals=[ScheduleIntervalSpec(every=timedelta(minutes=2))] ), state=ScheduleState(note="Here's a note on my Schedule."), ), ) if __name__ == "__main__": asyncio.run(main()) ``` > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > ### Backfill a Scheduled Workflow The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time. To Backfill a Scheduled Workflow Execution in Python, use the [backfill()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#backfill) asynchronous method on the Schedule Handle. ```python {13-19} import asyncio from datetime import datetime, timedelta from temporalio.client import Client, ScheduleBackfill, ScheduleOverlapPolicy async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) now = datetime.utcnow() ( await handle.backfill( ScheduleBackfill( start_at=now - timedelta(minutes=10), end_at=now - timedelta(minutes=9), overlap=ScheduleOverlapPolicy.ALLOW_ALL, ), ), ) print(f"Result: {handle}") if __name__ == "__main__": asyncio.run(main()) ``` ### Delete a Scheduled Workflow The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule. To delete a Scheduled Workflow Execution in Python, use the [delete()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#delete) asynchronous method on the Schedule Handle. ```python {12} import asyncio from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) await handle.delete() if __name__ == "__main__": asyncio.run(main()) ``` ### Describe a Scheduled Workflow The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs. To describe a Scheduled Workflow Execution in Python, use the [describe()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#delete) asynchronous method on the Schedule Handle. You can get a complete list of the attributes of the Scheduled Workflow Execution from the [ScheduleDescription](https://python.temporal.io/temporalio.client.ScheduleDescription.html) class. ```python {12-14} import asyncio from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) desc = await handle.describe() print(f"Returns the note: {desc.schedule.state.note}") if __name__ == "__main__": asyncio.run(main()) ``` ### List a Scheduled Workflow The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs. To list all schedules, use the [list_schedules()](https://python.temporal.io/temporalio.client.Client.html#list_schedules) asynchronous method on the Client. If a schedule is added or deleted, it may not be available in the list immediately. ```python {9} import asyncio from temporalio.client import Client async def main() -> None: client = await Client.connect("localhost:7233") async for schedule in await client.list_schedules(): print(f"List Schedule Info: {schedule.info}.") if __name__ == "__main__": asyncio.run(main()) ``` ### Pause a Scheduled Workflow The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason. To pause a Scheduled Workflow Execution in Python, use the [pause()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#pause) asynchronous method on the Schedule Handle. You can pass a `note` to the `pause()` method to provide a reason for pausing the schedule. ```python {12} import asyncio from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) await handle.pause(note="Pausing the schedule for now") if __name__ == "__main__": asyncio.run(main()) ``` ### Trigger a Scheduled Workflow The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time. To trigger a Scheduled Workflow Execution in Python, use the [trigger()](https://python.temporal.io/temporalio.client.ScheduleHandle.html#trigger) asynchronous method on the Schedule Handle. ```python {12} import asyncio from temporalio.client import Client async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) await handle.trigger() if __name__ == "__main__": asyncio.run(main()) ``` ### Update a Scheduled Workflow The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval. Create a function that takes `ScheduleUpdateInput` and returns `ScheduleUpdate`. To update a Schedule, use a callback to build the update from the description. The following example updates the Schedule to use a new argument. ```python {17-24} import asyncio from temporalio.client import ( Client, ScheduleActionStartWorkflow, ScheduleUpdate, ScheduleUpdateInput, ) async def main(): client = await Client.connect("localhost:7233") handle = client.get_schedule_handle( "workflow-schedule-id", ) async def update_schedule_simple(input: ScheduleUpdateInput) -> ScheduleUpdate: schedule_action = input.description.schedule.action if isinstance(schedule_action, ScheduleActionStartWorkflow): schedule_action.args = ["my new schedule arg"] return ScheduleUpdate(schedule=input.description.schedule) await handle.update(update_schedule_simple) if __name__ == "__main__": asyncio.run(main()) ``` ## Temporal Cron Jobs > **⚠️ Caution:** > Cron support is not recommended > > We recommend using [Schedules](/schedule) instead of Cron Jobs. > Schedules were built to provide a better developer experience, including more configuration options and the ability to update or pause running Schedules. > A [Temporal Cron Job](/cron-job) is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution. A Cron Schedule is provided as an option when the call to spawn a Workflow Execution is made. You can set each Workflow to repeat on a schedule with the `cron_schedule` option from either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods. ```python {11-17} import asyncio from temporalio.client import Client from your_workflow import CronWorkflow async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( CronWorkflow.run, id="your-workflow-id", task_queue="your-task-queue", cron_schedule="* * * * *", ) print(f"Results: {result}") if __name__ == "__main__": asyncio.run(main()) ``` Temporal Workflow Schedule Cron strings follow this format: ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday) │ │ │ │ │ * * * * * ``` ## Start Delay Use the `start_delay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule. Use the `start_delay` option in either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods in the Client. ```python async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your name", id="your-workflow-id", task_queue="your-task-queue", start_delay=timedelta(hours=1, minutes=20, seconds=30) ) print(f"Result: {result}") if __name__ == "__main__": asyncio.run(main()) ``` --- # Workflow Timeouts - Python SDK Source: https://docs.temporal.io/develop/python/workflows/timeouts > Optimize Workflow Execution with Temporal Python SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient. Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes. If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer. Workflow timeouts are set when [starting the Workflow Execution](#workflow-timeouts). - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)** - restricts the maximum amount of time that a single Workflow Execution can be executed. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout):** restricts the maximum amount of time that a single Workflow Run can last. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout):** restricts the maximum amount of time that a Worker can execute a Workflow Task. Set the timeout to either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods. Available timeouts are: - `execution_timeout` - `run_timeout` - `task_timeout` ```python {11-20} import asyncio from datetime import timedelta from temporalio.client import Client from your_workflows import YourWorkflow from temporalio.common import RetryPolicy async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your timeout argument", id="your-workflow-id", task_queue="your-task-queue", # Set Workflow Timeout duration execution_timeout=timedelta(seconds=2), # run_timeout=timedelta(seconds=2), # task_timeout=timedelta(seconds=2), ) handle = await client.execute_workflow( YourWorkflow.run, "your retry policy argument", id="your-workflow-id", task_queue="your-task-queue", retry_policy=RetryPolicy(maximum_interval=timedelta(seconds=2)), ) print(f"Result: {result}") print(f"Handle: {handle}") if __name__ == "__main__": asyncio.run(main()) ``` ## Workflow retries A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to retry a Workflow Execution in the event of a failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. Set the Retry Policy to either the [`start_workflow()`](https://python.temporal.io/temporalio.client.Client.html#start_workflow) or [`execute_workflow()`](https://python.temporal.io/temporalio.client.Client.html#execute_workflow) asynchronous methods. ```python {21-27} import asyncio from datetime import timedelta from temporalio.client import Client from your_workflows import YourWorkflow from temporalio.common import RetryPolicy async def main(): client = await Client.connect("localhost:7233") result = await client.execute_workflow( YourWorkflow.run, "your timeout argument", id="your-workflow-id", task_queue="your-task-queue", # Set Workflow Timeout duration execution_timeout=timedelta(seconds=2), # run_timeout=timedelta(seconds=2), # task_timeout=timedelta(seconds=2), ) handle = await client.execute_workflow( YourWorkflow.run, "your retry policy argument", id="your-workflow-id", task_queue="your-task-queue", retry_policy=RetryPolicy(maximum_interval=timedelta(seconds=2)), ) print(f"Result: {result}") print(f"Handle: {handle}") if __name__ == "__main__": asyncio.run(main()) ``` --- # Timers - Python SDK Source: https://docs.temporal.io/develop/python/workflows/timers > Set durable Timers with Temporal Workflows using sleep() or timer(), ensuring code execution resumes after downtime. Sleep for months using resource-light operations in Python. A Workflow can set a durable Timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`. A Workflow can sleep for months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To set a Timer in Python, call the [`asyncio.sleep()`](https://docs.python.org/3/library/asyncio-task.html#sleeping) function and pass the duration in seconds you want to wait before continuing. ```python {11} import asyncio from temporalio import workflow @workflow.defn class LoopingWorkflow: @workflow.run async def run(self, iteration: int) -> None: if iteration == 5: return await asyncio.sleep(10) workflow.continue_as_new(iteration + 1) ``` --- # Versioning - Python SDK Source: https://docs.temporal.io/develop/python/workflows/versioning > Ensure deterministic Temporal Workflow execution and safely deploy updates using the Python SDK's patching and Worker Versioning APIs, for scalable long-running Workflows. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. > **🚨 Danger:** > Support for the experimental Worker Versioning method before 2025 will be removed from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](/worker-versioning) for guidance. You can still refer to the [Worker Versioning Legacy](worker-versioning-legacy) docs if needed. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Adding a patch A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Suppose you have an initial Workflow version called `pre_patch_activity`: ```python {5,11-14} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import pre_patch_activity @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> None: self._result = await workflow.execute_activity( pre_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) @workflow.query def result(self) -> str: return self._result ``` Now, you want to update your code to run `post_patch_activity` instead. This represents your desired end state. ```python {5,11-14} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import post_patch_activity @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> None: self._result = await workflow.execute_activity( post_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) @workflow.query def result(self) -> str: return self._result ``` The problem is that you cannot deploy `post_patch_activity` directly until you're certain there are no more running Workflows created using the `pre_patch_activity` code, otherwise you are likely to cause a nondeterminism error. Instead, you'll need to deploy `post_patched_activity` and use the [patched](https://python.temporal.io/temporalio.workflow.html#patched) function to determine which version of the code to execute. Patching is a three-step process: 1. Patch in any new, updated code using the `patched()` function. Run the new patched code alongside old code. 2. Remove old code and use `deprecate_patch()` to mark a particular patch as deprecated. 3. Once there are no longer any open Workflow Executions of the previous version of the code, remove `deprecate_patch()`. Let's walk through this process in sequence. ### Patching in new code Using `patched` inserts a marker into the Event History. During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case, `my-patch`). This ensures you can safely deploy code from `post_patch_activity` as a "feature flag" alongside the original version (`pre_patch_activity`). ```python {12-21} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import post_patch_activity, pre_patch_activity @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> None: if workflow.patched("my-patch"): self._result = await workflow.execute_activity( post_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) else: self._result = await workflow.execute_activity( pre_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) @workflow.query def result(self) -> str: return self._result ``` ### Deprecating patches After ensuring that all Workflows started with `pre_patch_activity` code have left retention, you can [deprecate the patch](https://python.temporal.io/temporalio.workflow.html#deprecate_patch). Once your Workflows are no longer running the pre-patch code paths, you can deploy your code with `deprecate_patch()`. These Workers will be running the most up-to-date version of the Workflow code, which no longer requires the patch. Deprecated patches serve as a bridge between the final stage of the patching process and the final state that no longer has patches. They function similarly to regular patches by adding a marker to the Event History. However, this marker won't cause a replay failure when the Workflow code doesn't produce it. ```python {12} from datetime import timedelta from temporalio import workflow with workflow.unsafe.imports_passed_through(): from activities import post_patch_activity @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> None: workflow.deprecate_patch("my-patch") self._result = await workflow.execute_activity( post_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) @workflow.query def result(self) -> str: return self._result ``` ### Removing a patch Once your pre-patch Workflows have left retention, you can then safely deploy Workers that no longer use either the `patched()` or `deprecate_patch()` calls: ```python # ... @workflow.defn class MyWorkflow: @workflow.run async def run(self) -> None: self._result = await workflow.execute_activity( post_patch_activity, schedule_to_close_timeout=timedelta(minutes=5), ) ``` Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Detailed Description of the Patched Function This video provides an overview of how the `patched()` function works:
For a more in-depth explanation, refer to the [Patching](/patching) Encyclopedia entry. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `PizzaWorkflow` as `PizzaWorkflowV2`: ```python @workflow.defn(name="PizzaWorkflow") class PizzaWorkflow: @workflow.run async def run(self, name: str) -> str: # this function contains the original code @workflow.defn(name="PizzaWorkflowV2") class PizzaWorkflowV2: @workflow.run async def run(self, name: str) -> str: # this function contains the updated code ``` You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types: ```python worker = Worker( client, task_queue="your-task-queue", workflows=[PizzaWorkflow, PizzaWorkflowV2], ) ``` The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ### Testing a Workflow for replay safety To determine whether your Workflow needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/python/best-practices/testing-suite#replay). --- # Worker Versioning (Legacy) - Python SDK Source: https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy > Learn the Python SDK's outdated Worker Versioning APIs. ## (Deprecated) How to use Worker Versioning in Python > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to > [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > > See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more > information. > A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Python Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on. ```python # ... worker = Worker( task_queue="your_task_queue_name", build_id=build_id, use_worker_versioning=True, # ... register workflows & activities, etc ) # ... ``` > **⚠️ Warning:** > > Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. > ### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to > [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `versioning_intent` argument available on the methods you use to invoke these commands. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: ```python # ... await workflow.execute_activity( say_hello, "hi", versioning_intent=VersioningIntent.USE_ASSIGNMENT_RULES, start_to_close_timeout=timedelta(seconds=5), ) # ... ``` ### Tell the Task Queue about your Worker's Build ID (Deprecated) > **⚠️ Caution:** > > This section is for a deprecated Worker Versioning API. Please redirect your attention to > [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). > Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. ```python # ... await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpAddNewDefault("deadbeef") ) ``` This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: ```python # ... await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpAddNewCompatible("deadbeef", "some-existing-build-id") ) ``` This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: ```python # ... await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpPromoteBuildIdWithinSet("deadbeef") ) ``` You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's default build. ```python # ... await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpPromoteSetByBuildId("deadbeef") ) ``` --- # Workflow Streams - Python SDK Source: https://docs.temporal.io/develop/python/workflows/workflow-streams > Stream events from a Workflow to subscribers using the Temporal Python SDK Workflow Streams contrib module. > **Public Preview** [Workflow Streams](/workflow-streams) adds a durable event channel to a Workflow, letting outside observers follow its progress in real time. This page walks through enabling a stream, publishing events from Workflows and Activities, subscribing to a stream, and keeping a stream running across long-lived Workflows. ## Enable streaming on a Workflow You can import Workflow Streams from `temporalio.contrib.workflow_streams`. Enable streaming by constructing a `WorkflowStream` from your Workflow's `@workflow.init` method because the stream's handlers have to be registered before the first publish Signal arrives. Doing it from `@workflow.run` raises a `RuntimeError` and would miss any publishes that arrived before the run body started executing. ```python from dataclasses import dataclass from temporalio import workflow from temporalio.contrib.workflow_streams import WorkflowStream @dataclass class OrderInput: order_id: str @workflow.defn class OrderWorkflow: @workflow.init def __init__(self, input: OrderInput) -> None: self.stream = WorkflowStream() ``` Constructing `WorkflowStream` creates the in-memory event log and dynamically registers the publish Signal, subscribe Update, and offset Query handlers on the current Workflow. Constructing more than one stream on the same Workflow also raises a `RuntimeError`. If your Workflow is long-running, see [Stream from long-running Workflows](#continue-as-new) for how to carry stream state across rollovers so subscribers don't see gaps. ## Publish from a Workflow You can [publish](/workflow-streams#publishing) when you bind a [topic name](/workflow-streams#topics) to its event type once via `self.stream.topic("name", type=Type)`, then call `publish()` on the returned handle to append events. The handle records the per-stream binding from topic name to value type so call sites don't have to repeat the type on every publish and so subscribers reading the same handle decode to the matching type. ```python from dataclasses import dataclass @dataclass class StatusEvent: state: str progress: int = 0 detail: str = "" @workflow.defn class OrderWorkflow: @workflow.init def __init__(self, input: OrderInput) -> None: self.stream = WorkflowStream() self.status = self.stream.topic("status", type=StatusEvent) @workflow.run async def run(self, input: OrderInput) -> None: self.status.publish(StatusEvent(state="validating", detail="checking inventory")) await validate_order(input.order_id) self.status.publish(StatusEvent(state="charging", progress=33, detail="authorizing payment")) await charge_payment(input.order_id) self.status.publish(StatusEvent(state="shipping", progress=66, detail="dispatching to warehouse")) await dispatch_order(input.order_id) self.status.publish(StatusEvent(state="completed", progress=100)) ``` `publish()` runs the payload converter to encode each value. The codec chain (encryption, compression, etc.) runs once on the Signal or Update envelope that carries the batch, never per item, so encryption and compression are applied exactly once each direction. The `type=` argument is optional and defaults to `Any`. Pass it when you want the binding recorded so re-binding the same name to an unequal type raises and exception or so subscribers can pick up the type from the same handle. ## Publish from a client Any process that has a Temporal Client and the target Workflow Id can publish to that Workflow's stream by constructing a `WorkflowStreamClient`. This is the general pattern and covers HTTP backends, starters, one-off scripts, other Workflows' Activities, and standalone Activities. Construct one with: ```python WorkflowStreamClient.create(client, workflow_id) ``` Then use it the same way you would the Workflow-side handle: bind a topic, publish through it, and let the async context manager flush on exit. When events originate in an Activity, publish from the Activity directly rather than returning them for the Workflow to forward. The Workflow hosts the stream but doesn't read its own stream; it processes the Activity's return value and emits its own lifecycle events. Keeping Workflow state independent of streamed output is what lets retried Activity attempts surface to subscribers without polluting the Workflow's durable state. See [How events are delivered](/workflow-streams#how-events-are-delivered). ```python from datetime import timedelta from temporalio.client import Client from temporalio.contrib.workflow_streams import WorkflowStreamClient async def publish_status(workflow_id: str) -> None: temporal_client = await Client.connect("localhost:7233") stream_client = WorkflowStreamClient.create( temporal_client, workflow_id=workflow_id, batch_interval=timedelta(milliseconds=200), ) async with stream_client: status = stream_client.topic("status", type=StatusEvent) status.publish(StatusEvent(state="started")) ... # Buffer is flushed on context manager exit. ``` Inside an Activity scheduled by a Workflow, `WorkflowStreamClient.from_within_activity()` is used to infer the Temporal Client and the parent Workflow Id from the Activity context, so you don't have to thread them through the Activity's input: ```python from temporalio import activity from temporalio.contrib.workflow_streams import WorkflowStreamClient @activity.defn async def stream_deltas(order_id: str) -> None: client = WorkflowStreamClient.from_within_activity() async with client: deltas = client.topic("delta", type=Delta) for delta in generate_deltas(order_id): deltas.publish(delta) activity.heartbeat() # Buffer is flushed on context manager exit. ``` For a [standalone Activity](/develop/python/activities/standalone-activities) (one started directly via `Client.start_activity` rather than from a Workflow), there is no parent Workflow context to infer, so `from_within_activity()` raises an exception. Fall back to the general pattern with `activity.client()` and the target Workflow Id threaded through the Activity's input. Two operations give the application explicit control over when batches ship: `force_flush=True` on a publish for latency, and `await client.flush()` for confirmation that prior publications have landed. Pass `force_flush=True` on a publish to wake the background flusher so the current buffer ships without waiting for the next interval. The flusher only runs while the Workflow Stream client is entered (`async with client`). Otherwise, `force_flush=True` queues the wake event, but nothing ships until you enter the context or call `await client.flush()`. The call returns immediately after appending to the buffer and signaling the flusher. It doesn't wait for delivery to the Workflow or to subscribers: ```python deltas.publish(delta, force_flush=True) ``` Use it for latency-sensitive events like, the first delta of a response so the user sees something fast, or punctuated events like `RETRY` and `STATUS_CHANGE`. See [Tuning](/workflow-streams#tuning) for the trade-off against history pressure. Use `await client.flush()` when you need a mid-stream barrier. Successful completion of the flush is proof that the Temporal server has received all prior publications, so subsequent work that depends on those events being durable can proceed. The client stays open for further publishing afterward. Exiting `async with client` already flushes on its way out, so the explicit call is only for barriers in the middle: ```python async with client: deltas = client.topic("delta", type=Delta) for delta in first_phase(): deltas.publish(delta) await client.flush() checkpoint_id = await record_phase_one_complete() # only safe once phase-one events are durable in the Workflow log for delta in second_phase(checkpoint_id): deltas.publish(delta) ``` `publish()` is non-blocking and applies no backpressure. From an Activity or other client, it appends to the client's in-memory buffer and returns. From inside a Workflow, it appends synchronously to the in-memory log. [Subscribers](/workflow-streams#subscribing) pull from the Workflow's log on their own schedule, so a slow subscriber doesn't slow down [publishers](/workflow-streams#publishing). If a publisher emits faster than batches can ship to the server, the buffer grows: the process uses more memory, the stream falls further behind real time, and at the limit Signals can't keep up. If your application needs to bound this (to cap memory, to keep the stream close to real time, or to apply a policy when the publisher overruns the network), apply that policy upstream of `publish()`. The choice (block, drop, error, sample) is application-specific, and Workflow Streams doesn't pick one for you. ## Subscribe [Subscribing](/workflow-streams#subscribing) uses the same client construction as publishing: `WorkflowStreamClient.create(client, workflow_id)` from any process that has a Temporal Client or `from_within_activity()` inside an Activity. Subscribing from an Activity is less common in practice, so the general client case is the primary example below. Once you have a client, iterate a topic handle's `subscribe()`, the counterpart to `publish()`. The handle's bound type drives decoding, so each `item.data` arrives as `T` via the client's payload converter. The codec chain is applied once at the Update envelope, not per item. ```python from temporalio.client import Client from temporalio.contrib.workflow_streams import WorkflowStreamClient async def watch_order(order_id: str) -> None: temporal_client = await Client.connect("localhost:7233") stream = WorkflowStreamClient.create(temporal_client, workflow_id=order_id) status = stream.topic("status", type=StatusEvent) async for item in status.subscribe(): evt = item.data print(f"[{evt.progress:3d}%] {evt.state}: {evt.detail}") if evt.state == "completed": break ``` The iterator handles re-polling, pagination when a poll response hits the ~1 MB cap, and Workflow-side log truncation transparently. Callers don't need to wrap the iterator for the common cases. Two edge cases are worth knowing: - An RPC timeout where Continue-As-New cannot be followed ends the iterator silently (no exception raised) - A validator rejection during a Continue-As-New handoff can surface as a `WorkflowUpdateFailedError`. See the [API reference](https://python.temporal.io/temporalio.contrib.workflow_streams.html) for details. ### Heterogeneous topics A topic handle binds one name to one type, so it only fits a single-type subscription. To consume multiple topics whose payload types differ, call `client.subscribe()` directly with a list of names (or `subscribe([])` for every topic) and pass `result_type=temporalio.common.RawValue` so each item arrives as the underlying `Payload` wrapped in a `RawValue`. Dispatch on `item.topic` and decode the wrapped payload with the client's payload converter: ```python from temporalio.common import RawValue converter = temporal_client.data_converter.payload_converter async for item in stream.subscribe(["status", "progress"], result_type=RawValue): if item.topic == "status": evt = converter.from_payload(item.data.payload, StatusEvent) print(f"[status] {evt.state}: {evt.detail}") elif item.topic == "progress": evt = converter.from_payload(item.data.payload, ProgressEvent) print(f"[progress] {evt.message}") ``` A single iterator over multiple topics also avoids the cancellation race that two concurrent subscribers would create. `RawValue` is also the right shape when you want to forward the bytes through to another system without decoding them. Omitting `result_type` entirely or passing `result_type=None` decodes each item with the converter's default rules. For the stock JSON converter, that means a Python primitive, `dict`, or `list`. This works for fully homogeneous streams, but not for the dispatch-by-topic pattern above, where each topic has its own concrete dataclass. ### Closing the stream A subscriber's `async for` does not know when the publisher is done. How you [close a stream](/workflow-streams#closing-the-stream) depends on what the application needs. As one example, a common pattern combines two pieces: 1. **An in-band terminator.** The Workflow or its Activity publishes a sentinel event the subscriber recognizes and breaks on. In the `watch_order` example above, `StatusEvent(state="completed")` is the minimal form, and the consumer's `if evt.state == "completed": break` is the matching half. Each subscription decides what its own end-of-stream marker is. 2. **A brief overlap before the Workflow returns.** A poll Update that is still in flight when the Workflow returns surfaces to the client as `AcceptedUpdateCompletedWorkflow`, and no new polls can complete after that. If the Workflow returns immediately after publishing the terminator, subscribers may miss it. There are two ways to provide that overlap. - [Fixed sleep](/workflow-streams#fixed-sleep). Sleep between the terminator and the return so any in-flight poll has time to fetch the terminator before the Workflow exits: ```python # at the end of @workflow.run self.status.publish(StatusEvent(state="completed", progress=100)) await workflow.sleep(timedelta(seconds=30)) return result ``` - [Acknowledgment handshake](/workflow-streams#acknowledgment-handshake). The subscriber sends a Signal once it has the terminator; the Workflow waits up to a timeout, returning as soon as the ack arrives: ```python @workflow.signal async def subscriber_acknowledged_terminator(self) -> None: self.subscriber_done = True @workflow.run async def run(self, input: ChatInput) -> str: ... try: await workflow.wait_condition( lambda: self.subscriber_done, timeout=timedelta(seconds=30), ) except TimeoutError: pass # No subscriber attached; the run still completes cleanly. return result ``` The full pattern is wired into the [Stream LLM output](#stream-llm-output) example below. You can [inspect the terminal status](/workflow-streams#inspecting-terminal-status). `subscribe()` exits cleanly when the Workflow reaches `COMPLETED`, `FAILED`, `CANCELED`, `TERMINATED`, or `TIMED_OUT`, but does not distinguish among them. If your application needs to know which (to display success or failure to the user, log the outcome, or decide whether to retry), call `await temporal_client.get_workflow_handle(workflow_id).describe()` after the loop returns to inspect the Workflow's status. ## Stream from long-running Workflows Workflows that run for hours or accumulate thousands of events need to periodically roll over via [Continue-As-New](/workflow-streams#stream-from-long-running-workflows) to keep history bounded. Subscribers automatically follow these rollovers, but the client retained from `WorkflowStreamClient.create()` or `from_within_activity()` is required (clients constructed directly with a single handle cannot re-target the new run). To keep a stream running across rollovers without subscribers seeing a gap, carry both your application state and the stream state across the boundary. Add a `WorkflowStreamState | None` field to your Workflow input, pass it to the constructor, and call `WorkflowStream.continue_as_new(build_args)` to invoke the rollover. The helper drains waiting subscribers, waits for in-flight handlers to finish, then calls `workflow.continue_as_new` with the args produced by `build_args(post_drain_state)`: ```python from dataclasses import dataclass, field from temporalio import workflow from temporalio.contrib.workflow_streams import WorkflowStream, WorkflowStreamState @dataclass class AppState: items_processed: int = 0 @dataclass class WorkflowInput: app_state: AppState = field(default_factory=AppState) stream_state: WorkflowStreamState | None = None @workflow.defn class LongRunningWorkflow: @workflow.init def __init__(self, input: WorkflowInput) -> None: self.app_state = input.app_state self.stream = WorkflowStream(prior_state=input.stream_state) @workflow.run async def run(self, input: WorkflowInput) -> None: while True: await do_one_iteration(self) if workflow.info().is_continue_as_new_suggested(): await self.stream.continue_as_new( lambda stream_state: [ WorkflowInput( app_state=self.app_state, stream_state=stream_state, ) ] ) ``` The `| None` on the `stream_state` field type is required: `prior_state` is `None` on a fresh start and a `WorkflowStreamState` instance after a rollover. Always use the concrete type, not `Any`. With `Any`, the data converter rebuilds the field as a plain `dict` and `WorkflowStream(prior_state=...)` raises an `AttributeError` accessing `.log` / `.base_offset` / `.publishers` on the dict. To pass other Continue-As-New parameters such as `task_queue`, `retry_policy`, `run_timeout`, use the explicit recipe instead: ```python self.stream.detach_pollers() await workflow.wait_condition(workflow.all_handlers_finished) workflow.continue_as_new( args=[WorkflowInput(app_state=self.app_state, stream_state=self.stream.get_state())], task_queue="other-tq", ) ``` The carried `WorkflowStreamState` includes the entire in-memory log of the previous run, so streams that carry large items can hit Temporal's per-payload size limit at the rollover. Offload the bytes via [External Storage](/external-storage) so each item is a small reference rather than the full payload, and combine that with `truncate()` to keep the carried log itself small. ## Deduplication window See [How events are delivered](/workflow-streams#how-events-are-delivered) for more details on subscriber and publisher behavior. See [Tuning](/workflow-streams#tuning) for more details on how to change your settings to meet the requirements for your Workflow Streams. There are two limits on the [deduplication window](/workflow-streams#deduplication-window) worth highlighting: - `publisher_ttl`: At each Continue-As-New, deduplicate entries whose `last_seen` is older than this are dropped. `last_seen` is updated on each *successful* publish, so a publisher that retries through a long partition without success can still time out. Tune upward if your publishers can be silent for extended windows: ```python WorkflowStream.continue_as_new(publisher_ttl=...) ``` - `max_retry_duration`: A `WorkflowStreamClient` retries a failed batch for up to this long. If the duration elapses with the batch still pending, the client gives up, the pending batch is dropped, and a `TimeoutError` is raised. On timeout, the dropped batch is at-most-once: it may or may not have reached the Workflow. One operational caveat: the `TimeoutError` raises from inside the background flusher task and terminates it. Until you call `await client.flush()` or exit the `async with` block, subsequent publishes accumulate in the buffer with no flusher to ship them. ## Best practices There are a few details to note if you're writing custom message handlers or testing the library's capabilities: - **`WorkflowStreamClient` is asyncio-only.** The client buffer is mutated on the publish path and read from the flusher inside a single event loop. Don't call `publish()` from a Worker thread. - **Custom handlers read stream state on the first activation.** `WorkflowStream` registers its publish-Signal handler dynamically from `__init__`, so on the first activation a publish Signal can be queued before class-level `@workflow.signal` or `@workflow.update` handlers have run. A handler that observes state set by stream initialization in that same activation can see pre-publish state. The fix is to make the handler `async def` and `await` once before reading state. `asyncio.sleep(0)` is a no-op yield that suffices and adds no history events. Don't substitute `workflow.sleep(0)`, which records a timer event. Once the first activation completes, the handler is permanent and the race doesn't recur. - **Type bindings aren't shared across publishers.** Each `WorkflowStream` and each `WorkflowStreamClient` records topic types only for its own instance. If two publishers bind the same topic name to different types, the mismatch is not caught at publish, and the subscriber gets a decode error when it processes events from the mismatched publisher. ## Example: Stream LLM output The headline use case fits the publish/subscribe shapes documented above. An Activity calls the model and publishes deltas as they arrive. The Workflow starts the Activity and waits for the consumer to acknowledge end-of-stream. The consumer subscribes, accumulates the deltas, and clears its accumulated state on `RETRY` before continuing. The shape works for a terminal client, a desktop UI, or a Server-Sent Events (SSE) endpoint forwarding to a browser. Anything that holds the displayed state calls `render()` to display it. If your Activity can retry, the consumer side has to account for it. A retried attempt is a fresh publisher, so its output appears in the stream alongside the output from the previous attempt. In the LLM streaming pattern below, that means the failed attempt's partial deltas and the retried attempt's full output both reach a subscribed UI unless the UI resets on a `RETRY` event. The example wires up that pattern. See [How events are delivered](/workflow-streams#how-events-are-delivered) for the precise guarantees. **activity.py** ```python from openai import AsyncOpenAI @dataclass class TextDelta: text: str @activity.defn async def stream_completion(prompt: str) -> str: stream_client = WorkflowStreamClient.from_within_activity( batch_interval=timedelta(milliseconds=200), ) # Disable provider-side retries; let Temporal own retry policy at the Activity layer. openai_client = AsyncOpenAI(max_retries=0) async with stream_client: deltas = stream_client.topic("delta", type=TextDelta) retry = stream_client.topic("retry", type=dict) close = stream_client.topic("close") # Tell consumers an earlier attempt's deltas are stale. if activity.info().attempt > 1: retry.publish({"attempt": activity.info().attempt}, force_flush=True) full: list[str] = [] first = True oai_stream = await openai_client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], stream=True, ) async for chunk in oai_stream: if not chunk.choices: continue text = chunk.choices[0].delta.content if not text: continue # force_flush only on the first delta so the user sees something # immediately; subsequent deltas batch at the 200 ms interval. deltas.publish(TextDelta(text=text), force_flush=first) first = False full.append(text) close.publish({}) return "".join(full) ``` **workflow.py** ```python @workflow.defn class ChatWorkflow: @workflow.init def __init__(self, input: ChatInput) -> None: self.stream = WorkflowStream() self.subscriber_done: bool = False @workflow.signal async def subscriber_acknowledged_terminator(self) -> None: self.subscriber_done = True @workflow.run async def run(self, input: ChatInput) -> str: result = await workflow.execute_activity( stream_completion, input.prompt, start_to_close_timeout=timedelta(minutes=5), ) # Wait for the subscriber to ack the terminal `close` event. # The timeout is a fallback for when no subscriber is attached; # with the ack, the typical case exits as soon as the subscriber confirms. try: await workflow.wait_condition( lambda: self.subscriber_done, timeout=timedelta(seconds=30), ) except TimeoutError: pass # No subscriber; the run still completes cleanly. return result ``` **consumer.py** ```python async def stream_chat(chat_id: str) -> str: # Subscribe-only; no `async with` needed because the flusher only runs for publishers. stream = WorkflowStreamClient.create(temporal_client, workflow_id=chat_id) converter = temporal_client.data_converter.payload_converter output: list[str] = [] def render() -> None: ... # display the accumulated output (terminal redraw, UI update, etc.) async for item in stream.subscribe( ["delta", "retry", "close"], result_type=RawValue ): if item.topic == "retry": # Earlier attempt's deltas are stale; drop what we've shown. output.clear() render() elif item.topic == "delta": delta = converter.from_payload(item.data.payload, TextDelta) output.append(delta.text) render() elif item.topic == "close": # Acknowledge so the Workflow can return without a sleep. await temporal_client.get_workflow_handle(chat_id).signal( ChatWorkflow.subscriber_acknowledged_terminator ) break return "".join(output) ``` A few choices in this shape are deliberate: - The Activity is the publisher because it owns the non-deterministic LLM call. The Workflow processes the Activity's return value, never reading its own stream. See [Publish from a client](#publish-from-a-client) for why. - The Activity publishes a `RETRY` event when `activity.info().attempt > 1`. This lets the UI respond appropriately to the failure, typically by clearing accumulated deltas before the next attempt's deltas arrive (see [How events are delivered](/workflow-streams#how-events-are-delivered)). - Termination uses an *ack handshake*: the consumer signals the Workflow once it has received the `close` event, so the Workflow can return as soon as the subscriber confirms. The `wait_condition` timeout is the fallback when no subscriber is attached (see [Closing the stream](#closing-the-stream) for the simpler fixed-sleep alternative). - `force_flush=True` is used on the first delta and on the `RETRY` sentinel, where latency matters. Subsequent deltas batch at the 200 ms `batch_interval`; per-delta `force_flush=True` would generate one Signal per token (see [Tuning](/workflow-streams#tuning) for the trade-off). ## See also - [Workflow Streams samples (samples-python)](https://github.com/temporalio/samples-python/tree/main/workflow_streams): four runnable scenarios covering basic publish/subscribe, reconnecting subscribers, external publishers, and bounded logs. - [`temporalio.contrib.workflow_streams` API reference](https://python.temporal.io/temporalio.contrib.workflow_streams.html). - [Workflow message passing](/develop/python/workflows/message-passing): Signals, Updates, and Queries that Workflow Streams is built on. - [Payload conversion](/develop/python/data-handling/data-conversion): converters and codecs. --- # Ruby SDK developer guide Source: https://docs.temporal.io/develop/ruby ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Install and get started You can find detailed installation instructions for the Ruby SDK in the [Quickstart](/develop/ruby/set-up-local-ruby). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Workflow basics](/develop/ruby/workflows/basics) - [Activity basics](/develop/ruby/activities/basics) - [Start an Activity execution](/develop/ruby/activities/execution) - [Run Worker processes](/develop/ruby/workers/run-worker-process) From there, you can dive deeper into any of the Temporal primitives to start building Workflows that fit your use cases. ## [Workflows](/develop/ruby/workflows) - [Workflow basics](/develop/ruby/workflows/basics) - [Child Workflows](/develop/ruby/workflows/child-workflows) - [Continue-As-New](/develop/ruby/workflows/continue-as-new) - [Cancellation](/develop/ruby/workflows/cancellation) - [Timeouts](/develop/ruby/workflows/timeouts) - [Message Passing](/develop/ruby/workflows/message-passing) - [Schedules](/develop/ruby/workflows/schedules) - [Timers](/develop/ruby/workflows/timers) - [Futures](/develop/ruby/workflows/futures) - [Dynamic Workflow](/develop/ruby/workflows/dynamic-workflow) - [Versioning](/develop/ruby/workflows/versioning) ## [Activities](/develop/ruby/activities) - [Activity basics](/develop/ruby/activities/basics) - [Activity execution](/develop/ruby/activities/execution) - [Standalone Activities](/develop/ruby/activities/standalone-activities-quickstart) - [Timeouts](/develop/ruby/activities/timeouts) - [Asynchronous Activity completion](/develop/ruby/activities/asynchronous-activity) - [Dynamic Activity](/develop/ruby/activities/dynamic-activity) - [Benign exceptions](/develop/ruby/activities/benign-exceptions) ## [Workers](/develop/ruby/workers) - [Worker processes](/develop/ruby/workers/run-worker-process) - [Observability](/develop/ruby/platform/observability) ## [Temporal Client](/develop/ruby/client) - [Temporal Client](/develop/ruby/client/temporal-client) ## [Platform](/develop/ruby/platform) - [Observability](/develop/ruby/platform/observability) - [Enriching the UI](/develop/ruby/platform/enriching-ui) ## [Integrations](/develop/ruby/integrations) - [Rails integration](/develop/ruby/integrations/rails-integration) ## [Best practices](/develop/ruby/best-practices) - [Error handling](/develop/ruby/best-practices/error-handling) - [Testing](/develop/ruby/best-practices/testing-suite) - [Debugging](/develop/ruby/best-practices/debugging) - [Converters and encryption](/develop/ruby/best-practices/data-handling) ## Temporal Ruby technical resources - [Ruby SDK Quickstart - Setup Guide](/develop/ruby/set-up-local-ruby) - [Ruby SDK Code Samples](https://github.com/temporalio/samples-ruby) - [Ruby API Documentation](https://ruby.temporal.io/) - [Ruby SDK GitHub](https://github.com/temporalio/sdk-ruby) - [Temporal 101 in Ruby Free Course](https://learn.temporal.io/courses/temporal_101/ruby/) ## Get connected with the Temporal Ruby community - [Temporal Ruby Community Slack](https://temporalio.slack.com/archives/C052K5QFBNW) - [Ruby SDK Forum](https://community.temporal.io/tag/ruby-sdk) --- # Activities - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities > This section explains how to implement Activities with the Ruby SDK ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Activities - [Activity basics](/develop/ruby/activities/basics) - [Activity execution](/develop/ruby/activities/execution) - [Standalone Activities Quickstart](/develop/ruby/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/ruby/activities/standalone-activities) - [Timeouts](/develop/ruby/activities/timeouts) - [Asynchronous Activity completion](/develop/ruby/activities/asynchronous-activity) - [Dynamic Activity](/develop/ruby/activities/dynamic-activity) - [Benign exceptions](/develop/ruby/activities/benign-exceptions) --- # Asynchronous Activity completion - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/asynchronous-activity > Asynchronously complete an Activity in Temporal using the Ruby SDK. Follow simple steps to allow an Activity Function to return without the Activity Execution completing. ## How to asynchronously complete an Activity This page describes how to asynchronously complete an Activity. [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To mark an Activity as completing asynchronously, do the following inside the Activity. ```ruby # Capture token for later completion captured_token = Temporalio::Activity::Context.current.info.task_token # Raise a special exception that says an activity will be completed somewhere else raise Temporalio::Activity::CompleteAsyncError ``` To update an Activity outside the Activity, use the [async_activity_handle](https://ruby.temporal.io/Temporalio/Client.html#async_activity_handle-instance_method) method on the client to get the handle of the Activity. ```ruby handle = my_client.async_activity_handle(captured_token) ``` Then, on that handle, you can use `heartbeat`, `complete`, `fail`, or `report_cancellation` methods to update the Activity. ```ruby handle.complete('completion value') ``` --- # Activity basics - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/basics > This section explains Activity basics with the Ruby SDK ## Develop an Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/ruby/activities/standalone-activities-quickstart). You can develop an Activity Definition by creating a class that extends `Temporalio::Activity::Definition`. To register a class as an Activity with a custom name, use the `activity_name` class method in the class definition. Otherwise, the activity name is the unqualified class name. ```ruby class MyActivity < Temporalio::Activity::Definition def execute(input) "#{input['greeting']}, #{input['name']}!" end end ``` Activity implementation code should be _idempotent_. Learn more about [idempotency](/activity-definition#idempotency). There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single hash or object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a method signature. The `execute` method in your Activity can technically accept multiple parameters of any data type that Temporal can convert. However, Temporal strongly encourages using a single parameter object to simplify versioning and maintainability. ### Activity Concurrency and Executors > **📝 Note:** > > This section covers advanced concurrency and execution options that most users will not need when getting started. > By default, activities run in the "thread pool executor" (that is, `Temporalio::Worker::ActivityExecutor::ThreadPool`). This default is shared across all workers and is a naive thread pool that continually makes threads as needed when none are idle/available to handle incoming work. If a thread sits idle long enough, it will be killed. The maximum number of concurrent activities a worker will run at a time is configured via its `tuner` option. The default is `Temporalio::Worker::Tuner.create_fixed` which defaults to 100 activities at a time for that worker. When this value is reached, the worker will stop asking for work from the server until there are slots available again. In addition to the thread pool executor, there is also a fiber executor in the default executor set. To use fibers, call `activity_executor :fiber` class method at the top of the activity class (the default of this value is `:default` which is the thread pool executor). Activities can only choose the fiber executor if the worker has been created and run in a fiber, but thread pool executor is always available. Currently due to [an issue](https://github.com/temporalio/sdk-ruby/issues/162), workers can only run in a fiber on Ruby versions 3.3 and newer. --- # Benign exceptions - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. **How to mark an Activity error as benign using the Temporal Ruby SDK** When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the `category` parameter to `Temporalio::Error::ApplicationError::Category::BENIGN` when raising an `ApplicationError`. Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```ruby require 'temporalio/activity' class MyActivity < Temporalio::Activity::Definition def execute begin call_external_service rescue StandardError => e # Mark this error as benign since it's expected raise Temporalio::Error::ApplicationError.new( e.message, category: Temporalio::Error::ApplicationError::Category::BENIGN ) end end end ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Dynamic Activities - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/dynamic-activity > This section explains Dynamic Activities with the Ruby SDK ## Set a Dynamic Activity A Dynamic Activity in Temporal is an Activity that is invoked dynamically at runtime if no other Activity with the same name is registered. An Activity can be made dynamic by invoking `activity_dynamic` class method at the top of the definition. You must register the Activity with the Worker before it can be invoked. Only one Dynamic Activity can be present on a Worker. Often, dynamic is used in conjunction with `activity_raw_args` which does not convert arguments but instead passes them through as a splatted array of `Temporalio::Converters::RawValue` instances. ```ruby class MyDynamicActivity < Temporalio::Activity::Definition # Make this the dynamic activity and accept raw args activity_dynamic activity_raw_args def execute(*raw_args) raise Temporalio::Error::ApplicationError, 'One arg expected' unless raw_args.size == 1 # Use payload converter to convert it input = Temporalio::Activity::Context.current.payload_converter.from_payload(raw_args.first.payload) "#{input['greeting']}, #{input['name']}!" end end ``` --- # Activity execution - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/execution > Shows how to perform Activity execution with the Ruby SDK ## Start Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in the set of three [Activity Task](/tasks#activity-task) related Events ([ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and `ActivityTask[Closed]`)in your Workflow Execution Event History. The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. A large Execution history can thus adversely impact the performance of your Workflow. Therefore, be mindful of the amount of data you transfer through Activity invocation parameters or Return Values. Otherwise, no additional limitations exist on Activity implementations. To spawn an Activity Execution, use the `execute_activity` operation from within your Workflow Definition. ```ruby class MyWorkflow < Temporalio::Workflow::Definition # Customize the name workflow_name :MyDifferentWorkflowName def execute(name) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) end end ``` Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout). These values are set as keyword parameters. The Activity result is the returned from the `execute_activity` call. --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/ruby/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal Ruby SDK. > **Public Preview** Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a [`Temporalio::Client`](https://ruby.temporal.io/Temporalio/Client.html). The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/ruby/activities/standalone-activities-quickstart). > This page covers the following: - [Start a Standalone Activity without waiting for the result](#start-activity) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [Wait for the result of a Standalone Activity](#get-activity-result) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the > [standalone_activity](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity) > sample. > ## Start a Standalone Activity without waiting for the result Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue your Activity job, without waiting for it to be executed by your Worker. Use [`Temporalio::Client#start_activity`](https://ruby.temporal.io/Temporalio/Client.html#start_activity-instance_method) to start a Standalone Activity and get a handle without waiting for the result: [start_activity.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/start_activity.rb) ```ruby handle = client.start_activity( StandaloneActivity::MyActivities::ComposeGreeting, 'Hello', 'World', id: 'standalone-activity-id', task_queue: 'standalone-activity-sample', start_to_close_timeout: 10 ) puts "Started Activity with id=#{handle.id} run_id=#{handle.run_id}" # Wait for the result later puts "Activity result: #{handle.result}" ``` With the Temporal Server and Worker running, open a new terminal in the `samples-ruby` directory and run: ```bash bundle exec ruby standalone_activity/start_activity.rb ``` Or use the Temporal CLI: ```bash temporal activity start \ --type ComposeGreeting \ --activity-id standalone-activity-id \ --task-queue standalone-activity-sample \ --start-to-close-timeout 10s \ --input '"Hello"' \ --input '"World"' ``` ## Get a handle to an existing Standalone Activity Use [`Temporalio::Client#activity_handle`](https://ruby.temporal.io/Temporalio/Client.html#activity_handle-instance_method) to create an [`ActivityHandle`](https://ruby.temporal.io/Temporalio/Client/ActivityHandle.html) for a previously started Standalone Activity: ```ruby handle = client.activity_handle('standalone-activity-id') ``` Pass no run ID (the default) to target the latest run of the given Activity ID, or pass `activity_run_id:` to target a specific run. You can then use the handle to wait for the result, describe, cancel, or terminate the Activity: ```ruby handle.result # block until the activity completes; returns the result handle.describe # fetch metadata (status, timestamps, attempt, last failure, etc.) handle.cancel # request cancellation handle.terminate # force-close the activity ``` ## Wait for the result of a Standalone Activity Under the hood, calling `client.execute_activity` is the same as calling `client.start_activity` to durably enqueue the Standalone Activity, and then calling `handle.result` to block until the Activity completes and return the result: ```ruby result = handle.result ``` Or use the Temporal CLI to wait for a result by Activity ID: ```bash temporal activity result --activity-id standalone-activity-id ``` ## List Standalone Activities Use [`Temporalio::Client#list_activities`](https://ruby.temporal.io/Temporalio/Client.html#list_activities-instance_method) to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an `Enumerator` of [`ActivityExecution`](https://ruby.temporal.io/Temporalio/Client/ActivityExecution.html) values that fetches pages from the server on demand as the enumerator is consumed. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. [list_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/list_activities.rb) ```ruby client.list_activities("TaskQueue = 'standalone-activity-sample'").each do |execution| puts "#{execution.activity_id} #{execution.activity_type} #{execution.status}" end ``` Run it: ```bash bundle exec ruby standalone_activity/list_activities.rb ``` Or use the Temporal CLI: ```bash temporal activity list ``` The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow Visibility](/visibility). For example, `ActivityType = 'ComposeGreeting' AND Status = 'Running'`. ## Count Standalone Activities Use [`Temporalio::Client#count_activities`](https://ruby.temporal.io/Temporalio/Client.html#count_activities-instance_method) to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) — not the number of queued tasks. It works the same way as counting Workflow Executions. [count_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/count_activities.rb) ```ruby result = client.count_activities("TaskQueue = 'standalone-activity-sample'") puts "Total: #{result.count}" result.groups.each do |group| puts " #{group.group_values.join(',')} => #{group.count}" end ``` Run it: ```bash bundle exec ruby standalone_activity/count_activities.rb ``` Or use the Temporal CLI: ```bash temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The Worker and Client code in the [Standalone Activities Quickstart](/develop/ruby/activities/standalone-activities-quickstart) use [`Temporalio::EnvConfig::ClientConfig.load_client_connect_options`](https://ruby.temporal.io/Temporalio/EnvConfig/ClientConfig.html#load_client_connect_options-class_method), so the same code works against Temporal Cloud — configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/ruby/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/ruby/activities/standalone-activities-quickstart). --- # Standalone Activities Ruby Quickstart Source: https://docs.temporal.io/develop/ruby/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal Ruby SDK without writing a Workflow. # Quickstart Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a [`Temporalio::Client`](https://ruby.temporal.io/Temporalio/Client.html). The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/ruby/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **📝 Note:** > > This documentation uses source code from the > [standalone_activity](https://github.com/temporalio/samples-ruby/tree/main/standalone_activity) > sample. > ## Get started with Standalone Activities Prerequisites: - **Ruby** 3.3+ - **Temporal Ruby SDK** (v1.5.0 or higher). See the [Ruby Quickstart](/develop/ruby/set-up-local-ruby) for install instructions. - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server will now be available for client connections on `localhost:7233`, and the Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-ruby](https://github.com/temporalio/samples-ruby) repository to follow along: ```bash git clone https://github.com/temporalio/samples-ruby.git cd samples-ruby bundle install ``` The sample consists of separate programs in the `standalone_activity` directory: ``` standalone_activity/ ├── my_activities.rb # Activity definition ├── worker.rb # Worker that processes activity tasks ├── execute_activity.rb # Starts an activity and waits for the result ├── start_activity.rb # Starts an activity without blocking ├── list_activities.rb # Lists activity executions └── count_activities.rb # Counts activity executions ``` ## Define your Activity An Activity in the Temporal Ruby SDK is a subclass of [`Temporalio::Activity::Definition`](https://ruby.temporal.io/Temporalio/Activity/Definition.html) that implements an `execute` method. The way you define a Standalone Activity is identical to how you define an Activity orchestrated by a Workflow. In fact, the same Activity can be executed both as a Standalone Activity and as a Workflow Activity. [my_activities.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/my_activities.rb) ```ruby require 'temporalio/activity' module StandaloneActivity module MyActivities class ComposeGreeting < Temporalio::Activity::Definition def execute(greeting, name) "#{greeting}, #{name}!" end end end end ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities — you create a [`Temporalio::Worker`](https://ruby.temporal.io/Temporalio/Worker.html), register the Activity class, and call `worker.run`. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to run a Worker](/develop/ruby/workers/run-worker-process) for more details on Worker setup and configuration options. [worker.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/worker.rb) Open a new terminal, navigate to the `samples-ruby` directory, and run the Worker. Leave this terminal running — the Worker needs to stay up to process activities. ```ruby args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options args[0] ||= 'localhost:7233' args[1] ||= 'default' client = Temporalio::Client.connect(*args, **kwargs) worker = Temporalio::Worker.new( client:, task_queue: 'standalone-activity-sample', activities: [StandaloneActivity::MyActivities::ComposeGreeting] ) puts 'Starting worker (ctrl+c to exit)' worker.run(shutdown_signals: ['SIGINT']) ``` ```bash bundle exec ruby standalone_activity/worker.rb ``` ## Execute a Standalone Activity Use [`Temporalio::Client#execute_activity`](https://ruby.temporal.io/Temporalio/Client.html#execute_activity-instance_method) to execute a Standalone Activity and block until it completes. Call this from your application code, not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to be executed on your Worker, and then returns the result. [execute_activity.rb](https://github.com/temporalio/samples-ruby/blob/main/standalone_activity/execute_activity.rb) The first argument is the Activity to run. It can be the [`Activity::Definition`](https://ruby.temporal.io/Temporalio/Activity/Definition.html) subclass, an instance of one, or a string/symbol name. Positional arguments after it are passed to the Activity's `execute` method. The call requires `id`, `task_queue`, and at least one of `start_to_close_timeout` or `schedule_to_close_timeout`. To run it: 1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above). 2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above). 3. Open a new terminal, navigate to the `samples-ruby` directory, and run the execute command. Or use the Temporal CLI. ```ruby result = client.execute_activity( StandaloneActivity::MyActivities::ComposeGreeting, 'Hello', 'World', id: 'standalone-activity-id', task_queue: 'standalone-activity-sample', start_to_close_timeout: 10 ) puts "Activity result: #{result}" ``` ```bash bundle exec ruby standalone_activity/execute_activity.rb ``` ```bash temporal activity execute \\ --type ComposeGreeting \\ --activity-id standalone-activity-id \\ --task-queue standalone-activity-sample \\ --start-to-close-timeout 10s \\ --input '"Hello"' \\ --input '"World"' ``` ## Run with Temporal Cloud All code samples on this page use [`Temporalio::EnvConfig::ClientConfig.load_client_connect_options`](https://ruby.temporal.io/Temporalio/EnvConfig/ClientConfig.html#load_client_connect_options-class_method) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/ruby/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/ruby/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/ruby/activities/basics)**: How to write and register Activities with the Ruby SDK. --- # Activity Timeouts - Ruby SDK Source: https://docs.temporal.io/develop/ruby/activities/timeouts > Optimize Workflow Execution with Temporal Ruby SDK - Set Activity Timeouts and Retry Policies efficiently. ## Activity timeouts Each Activity Timeout controls a different aspect of how long an Activity Execution can take: - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout)** - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout)** - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout)** At least one of `start_to_close_timeout` or `schedule_to_close_timeout` is required. ```ruby Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60 ) ``` ### Activity Retry Policy By default, Activities use a system Retry Policy. You can override it by specifying a custom Retry Policy. To create an Activity Retry Policy in Ruby, set the `retry_policy` parameter when executing an activity. ```ruby Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60, retry_policy: Temporalio::RetryPolicy.new(max_interval: 10) ) ``` ### Override the retry interval with `next_retry_delay` If you raise an application-level error, you can override the Retry Policy's delay by specifying a new delay. ```ruby raise Temporalio::Error::ApplicationError.new( 'Some error', type: 'SomeErrorType', next_retry_delay: 3 * Temporalio::Activity::Context.current.info.attempt ) ``` ## Heartbeat an Activity A Heartbeat is a periodic signal from the Worker to the Temporal Service indicating the Activity is still alive and making progress. - Heartbeats are used to detect Worker failure. - Cancellations are delivered via Heartbeats. - Heartbeats may contain custom progress details. ```ruby class MyActivity < Temporalio::Activity::Definition def execute # This is a naive loop simulating work, but similar heartbeat logic # applies to other scenarios as well loop do # Send heartbeat Temporalio::Activity::Context.current.heartbeat # Sleep before heartbeating again sleep(3) end end end ``` ### Heartbeat Timeout The Heartbeat Timeout sets the maximum duration between Heartbeats before the Temporal Service considers the Activity failed. ```ruby Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60, heartbeat_timeout: 5 ) ``` --- # Best practices - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices > This section explains how to implement best practices with the Ruby SDK ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Best practices - [Error handling](/develop/ruby/best-practices/error-handling) - [Testing](/develop/ruby/best-practices/testing-suite) - [Debugging](/develop/ruby/best-practices/debugging) - [Converters and encryption](/develop/ruby/best-practices/data-handling) --- # Data handling - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/ruby/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/ruby/best-practices/data-handling/data-encryption) | | ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | | **Default** | JSON serialization | None (passthrough) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the Ruby SDK. ## Payload conversion Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back. ### Conversion sequence The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this: ![Temporal converter architecture](/img/info/converter-architecture.png) ## Supported Data Types Data converters are used to convert raw Temporal payloads to/from actual Ruby types. A custom data converter can be set via the `data_converter` keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (for example, for compression or encryption). Failure converters convert exceptions to/from serialized failures. Data converters are in the `Temporalio::Converters` module. The default data converter uses a default payload converter, which supports the following types: - `nil` - "bytes" (that is, `String` with `Encoding::ASCII_8BIT` encoding) - `Google::Protobuf::MessageExts` instances - [JSON module](https://docs.ruby-lang.org/en/master/JSON.html) for everything else This means that normal Ruby objects will use `JSON.generate` when serializing and `JSON.parse` when deserializing (with `create_additions: true` set by default). So a Ruby object will often appear as a hash when deserialized. Also, hashes that are passed in with symbol keys end up with string keys when deserialized. While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct. The default payload converter is a collection of "encoding payload converters". On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one). The encoding converter sets an `encoding` metadata value which is used to know which converter to use on deserialize. Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation. **NOTE:** For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for workflows/activities and translate to/from existing models as needed. See the next section on how to do this with ActiveModel objects. #### ActiveModel By default, ActiveModel objects do not natively support the `JSON` module. A mixin can be created to add this support for ActiveModel, for example: ```ruby module ActiveModelJSONSupport extend ActiveSupport::Concern include ActiveModel::Serializers::JSON included do def as_json(*) super.merge(::JSON.create_id => self.class.name) end def to_json(*args) as_json.to_json(*args) end def self.json_create(object) object = object.dup object.delete(::JSON.create_id) new(**object.symbolize_keys) end end end ``` Now if `include ActiveModelJSONSupport` is present on any ActiveModel class, on serialization `to_json` will be used which will use `as_json` which calls the super `as_json` but also includes the fully qualified class name as the JSON `create_id` key. On deserialization, Ruby JSON then uses this key to know what class to call `json_create` on. --- # Payload encryption - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Ruby SDK. Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. ## Custom Payload Codec Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`. The Payload Codec needs to extend `Temporalio::Converters::PayloadCodec` and implement `encode` and `decode` methods. These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field. Do not mutate the existing payloads. Here is an example of an encryption codec that just uses base64 in each direction: ```ruby class Base64Codec < Temporalio::Converters::PayloadCodec def encode(payloads) payloads.map do |p| Temporalio::Api::Common::V1::Payload.new( # Set our specific encoding. We may also want to add a key ID in here for use by # the decode side metadata: { 'encoding' => 'binary/my-payload-encoding' }, data: Base64.strict_encode64(p.to_proto) ) end end def decode(payloads) payloads.map do |p| # Ignore if it doesn't have our expected encoding next p unless p.metadata['encoding'] == 'binary/my-payload-encoding' Temporalio::Api::Common::V1::Payload.decode( Base64.strict_decode64(p.data) ) end end end ``` ### Set Data Converter to use custom Payload Codec When creating a client, the default `DataConverter` can be updated with the payload codec like so: ```ruby my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new) ) ``` - Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. - Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/command-reference/workflow#show) command or when browsing the Web UI to view output. For reference, see the [Encryption](https://github.com/temporalio/samples-ruby/tree/main/encryption) sample. ## Using a Codec Server A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. --- # Debugging - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/debugging > Debug Workflows in development and production environments using Temporal Ruby SDK. Use logging, debugger, CLI, replay, tracing, and more. ## Debugging This page shows how to do the following: - [Debug in a development environment](#debug-in-a-development-environment) - [Debug in a production environment](#debug-in-a-production-environment) ## Debug in a development environment In developing Workflows, you can use the normal development tools of logging and a debugger to see what’s happening in your Workflow. In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). The Web UI provides insight into your Workflows, making it easier to identify issues and monitor the state of your Workflows in real time. ## Debug in a production environment For production Workflows, debugging options include: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/ruby/best-practices/testing-suite#replay-test) - [Tracing](/develop/ruby/platform/observability#tracing) - [Logging](/develop/ruby/platform/observability#logging) You can analyze Worker performance using: - [Metrics](/develop/ruby/platform/observability#metrics) - [Worker performance guide](/develop/worker-performance) To monitor Server performance: - Use [Cloud metrics](/cloud/metrics/) if you're on Temporal Cloud - Or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics) if running your own deployment --- # Error handling - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/error-handling > Handle errors with Temporal Ruby SDK ## Raise and Handle Exceptions In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language. Temporal uses several different error classes internally — for example, [`CancelledError`](https://ruby.temporal.io/Temporalio/Error/CanceledError.html) in the Ruby SDK, to handle a Workflow cancellation. You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic. The one Temporal error class that you will typically raise deliberately is [`ApplicationError`](https://ruby.temporal.io/Temporalio/Error/ApplicationError.html). In fact, *any* other exceptions that are raised from your Ruby code in a Temporal Activity will be converted to an `ApplicationError` internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries. In other words, these two code samples do the same thing: ```ruby class MyError < StandardError end class SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise MyError.new('Simulated failure') end end ``` ```ruby class SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise Temporalio::Error::ApplicationError.new('Simulated failure', type: 'MyError') end end ``` Depending on your implementation, you may decide to use either method. One reason to use the Temporal `ApplicationError` class is because it allows you to set an additional `non_retryable` parameter. This way, you can decide whether an error should not be retried automatically by Temporal. This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse: ```ruby class SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise Temporalio::Error::ApplicationError.new('Simulated failure', non_retryable: true) end end ``` You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](/develop/ruby/activities/timeouts#activity-retries). ## Failing Workflows One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately. The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold. Activities will not actually *return* a failure to your Workflow until this condition or another non-retryable condition is met. At this point, you can decide how to handle an error returned by your Activity the way you would in any other program. For example, you could implement a [Saga Pattern](/design-patterns/saga-pattern) — see this [Ruby sample](https://github.com/temporalio/samples-ruby/tree/main/saga) — that uses `rescue` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure. **You will only fail a Workflow by manually raising an `ApplicationError` from the Workflow code.** You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue: ```ruby class SagaWorkflow < Temporalio::Workflow::Definition def execute(details) Temporalio::Workflow.execute_activity(Activities::SomethingThatFails, details,start_to_close_timeout: 30) rescue StandardError raise Temporalio::Error::ApplicationError.new('Fail the Workflow') ``` This works differently in a Workflow than raising exceptions from Activities. In an Activity, any Ruby exceptions or custom exceptions are converted to a Temporal `ApplicationError`. In a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationError` will only fail that particular [Workflow Task](/tasks#workflow-task-execution) and be retried. This includes any typical Ruby `RuntimeError`s that are raised automatically. These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly. --- # Testing - Ruby SDK Source: https://docs.temporal.io/develop/ruby/best-practices/testing-suite > The Ruby test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution. This page shows how to do the following: - [Understand types of tests](#types-of-tests) - [Use compatible test frameworks](#test-frameworks) - [Test Workflows](#testing-workflows) - [Test Activities](#test-activities) - [Replay tests](#replay-test) The Ruby test-suite feature guide describes the frameworks that facilitate Workflow and integration testing. ## Types of Tests In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks **Compatible testing frameworks** The Ruby SDK is compatible with any testing framework and does not have a specific recommendation. Most Ruby SDK samples use [minitest](https://github.com/minitest/minitest). ## Testing Workflows Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code. Using the time-skipping Workflow test environment can help there. ### Testing Workflows with standard server A non-time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_local` which supports all standard Temporal features. It is actually the real Temporal dev server packaged in the Temporal CLI, lazily downloaded on first use, and run as a sub-process in the background. Assuming tests properly use separate Task Queues, the same server can and should be reused across tests. Here's a simple example of a Workflow: ```ruby class SimpleWorkflow < Temporalio::Workflow::Definition def execute(name) "Hello, #{name}!" end end ``` Here's how a test of that Workflow may appear in minitest: ```ruby def test_simple_workflow # Start local server that is stopped when block is done Temporalio::Testing::WorkflowEnvironment.start_local do |env| # Start worker that is stopped when block is done worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SimpleWorkflow] ) worker.run do # Execute workflow and check result result = env.client.execute_workflow( SimpleWorkflow, 'some-name', id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) assert_equal 'Hello, some-name!', result end end end ``` While this is just a demonstration, a local server is often used as a fixture across many tests. In minitest for instance, users often start the environment lazily (with no block), and shut it down inside a block passed to `Minitest.after_run`. ### Testing Workflows with time skipping Sometimes there is a need to test Workflows that run a long time or to test that timeouts occur. A time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_time_skipping` which is a reimplementation of the Temporal server with special time skipping capabilities. Like `start_local`, this also lazily downloads the process to run when first called. Note, unlike `start_local`, this class is not thread safe nor safe for use with independent tests. It can be technically be reused, but only for one test at a time because time skipping is locked/unlocked at the environment level. Developers are encouraged to run it per test needed. #### Automatic time skipping Here's a simple example of a Workflow that waits a day: ```ruby class WaitADayWorkflow < Temporalio::Workflow::Definition def execute Temporalio::Workflow.sleep(1 * 24 * 60 * 60) 'all done' end end ``` A regular integration test of this Workflow on a normal server would be way too slow. However, the time-skipping server automatically skips to the next event when we wait on the result. Here's a test for that Workflow in minitest: ```ruby def test_wait_a_day_workflow # Start time-skipping test server that is stopped when block is done Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| # Start worker that is stopped when block is done worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [WaitADayWorkflow] ) worker.run do # Execute workflow and check result result = env.client.execute_workflow( WaitADayWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) assert_equal 'all done', result end end end ``` This test will run almost instantly. This is because by calling `execute_workflow` on our client, we are actually calling `start_workflow` + `result`, and `result` automatically skips time as much as it can (basically until the end of the workflow or until an activity is run). To disable automatic time-skipping while waiting for a workflow result, run code in a block passed to `env.auto_time_skipping_disabled`. #### Manual time skipping Until a Workflow is waited on, all time skipping in the time-skipping environment is done manually via `WorkflowEnvironment#sleep`. Here's a Workflow that waits for a Signal or times out: ```ruby class SignalWorkflow < Temporalio::Workflow::Definition def execute # Wait for signal or timeout in 45 seconds Temporalio::Workflow.timeout(45 * 60) do Temporalio::Workflow.wait_condition { @signal_received } end 'got signal' rescue Timeout::Error 'got timeout' end workflow_signal def some_signal @signal_received = true end end ``` To test a normal Signal in minitest, you might: ```ruby def test_signal_workflow Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SignalWorkflow] ) worker.run do handle = env.client.start_workflow( SignalWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) handle.signal(SignalWorkflow.some_signal) assert_equal 'got signal', handle.result end end end ``` But how would you test the timeout part? Like so: ```ruby def test_signal_workflow_timeout Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SignalWorkflow] ) worker.run do handle = env.client.start_workflow( SignalWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) # Advance 50 seconds env.sleep(50) assert_equal 'got timeout', handle.result end end end ``` ### Mocking Activities When testing Workflows, often you don't want to actually run the Activities. Activities are just classes that extend `Temporalio::Activity::Definition`. Simply write different/empty/fake/asserting ones and pass those to the Worker to have different activities called during the test. ## Testing Activities Unit testing an Activity or any code that could run in an Activity is done via the `Temporalio::Testing::ActivityEnvironment` class. Simply instantiate the class, and any code inside the block to `run` will be invoked inside the activity context. Several things about the activity environment can be customized via parameters when constructing the environment including setting the info, providing a proc to call back on each heartbeat, setting the cancellation to be used, etc. ## Replay test Given a Workflow's history, it can be replayed locally to check for things like non-determinism errors. For example, assuming the `history_json` parameter below is given a JSON string of history exported from the CLI or web UI for workflow `MyWorkflow`, the following method will replay it: ```ruby def replay_from_json(history_json) # Create a replayer replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow]) # Replay the history history = Temporalio::WorkflowHistory.from_history_json(history_json) replayer.replay_workflow(history) end ``` If there is a non-determinism, this will raise an exception. Event history can be loaded from more than just JSON. It can be fetched individually from a Workflow handle, or even in a list. For example, the following code will check that all Workflow histories for a certain Workflow type (that is, workflow class) are safe with the current Workflow code. ```ruby # Create a replayer replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow]) # Replay all workflows from a list replayer.replay_workflows(client.list_workflows("WorkflowType = 'MyWorkflow'")).each do |result| # Raise if any failed (could have just set raise_on_replay_failure: true, but this # demonstrates iterating over the results) raise result.replay_failure if result.replay_failure end ``` --- # Client - Ruby SDK Source: https://docs.temporal.io/develop/ruby/client > This section explains how to implement the Temporal Client with the Ruby SDK ![.NET SDK Banner](/img/assets/banner-ruby-temporal.png) ## Temporal Client - [Temporal Client](/develop/ruby/client/temporal-client) --- # Temporal Client - Ruby SDK Source: https://docs.temporal.io/develop/ruby/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service. Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and Queries to Workflow Executions, getting Workflow results, and more. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the Ruby SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow) - [Get Workflow results](#get-workflow-results) A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service. ## Connect to development Temporal Service Use [`Client.connect`](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a client. Connection options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide these options directly in code, load them from **environment variables**, or a **TOML configuration file** using the [`EnvConfig`](https://ruby.temporal.io/Temporalio/EnvConfig.html) helpers. We recommend environment variables or a configuration file for secure, repeatable configuration. When you’re running a Temporal Service locally (such as with the [Temporal CLI dev server](/cli/command-reference/server#start-dev)), the required options are minimal. If you don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to [Environment Configuration](/references/client-environment-configuration) for more details about configuration files and profiles. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml title="config.toml" # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS auto-enables when TLS config or an API key is present # disabled = false client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a profile from the configuration file using the `ClientConfig.load_client_connect_options` function as follows. In this example, you load the `default` profile for local development: ```ruby require 'temporalio/client' require 'temporalio/env_config' def main puts '--- Loading default profile from config.toml ---' # For this sample to be self-contained, we explicitly provide the path to # the config.toml file included in this directory. # By default though, the config.toml file will be loaded from # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). config_file = File.join(__dir__, 'config.toml') # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( config_source: Pathname.new(config_file) ) puts "Loaded 'default' profile from #{config_file}." puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end main if $PROGRAM_NAME == __FILE__ ``` **Environment Variables** Use the `EnvConfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. Set the following environment variables before running your application. Replace the placeholder values with your actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the `default` Namespace. You may omit these variables entirely since they're the defaults. ```bash export TEMPORAL_NAMESPACE="default" export TEMPORAL_ADDRESS="localhost:7233" ``` After setting the environment variables, you can create a Temporal Client as follows: ```ruby {9} require 'temporalio/client' require 'temporalio/env_config' def main # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile and also reads from environment variables. The environment # variables take precedence over the config file. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options() puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end main if $PROGRAM_NAME == __FILE__ ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. Use the `connect` class method on the `Temporalio::Client` class to create and connect to a Temporal Client to the Temporal Service. ```ruby client = Temporalio::Client.connect('localhost:7233', 'default') ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace and Account ID_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the [`Client.connect` method](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. ```ruby {8,14-16} require 'temporalio/client' require 'temporalio/env_config' def main puts "--- Loading 'staging' profile with programmatic overrides ---" config_file = File.join(__dir__, 'config.toml') profile_name = 'staging' puts "The 'staging' profile in config.toml has an incorrect address (localhost:9999)." puts "We'll programmatically override it to the correct address." # Load the 'staging' profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( profile: profile_name, config_source: Pathname.new(config_file) ) # Override the target host to the correct address. # This is the recommended way to override configuration values. args[0] = 'localhost:7233' puts "\nLoaded '#{profile_name}' profile from #{config_file} with overrides." puts " Address: #{args[0]} (overridden from localhost:9999)" puts " Namespace: #{args[1]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end main if $PROGRAM_NAME == __FILE__ ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your application. Require the `temporalio/env_config` module to set connection options for the Temporal Client using environment variables. The `Temporalio::EnvConfig::ClientConfig.load_client_connect_options` method will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. After setting the environment variables, use the following code to create the Temporal Client: ```ruby {9, 17} require 'temporalio/client' require 'temporalio/env_config' def main # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile. It also reads from environment variables. The environment # variables take precedence over the config file. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options() puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" end end main if $PROGRAM_NAME == __FILE__ ``` **Code** You can also specify connection options directly in code to connect to Temporal Cloud. To create an initial connection, provide the endpoint, Namespace and Account ID combination, and API key values to the `Client.connect` method. ```ruby client = Temporalio::Client.connect( '', # Endpoint '.', # Namespace api_key: '', tls: true ) ``` To connect using mTLS instead of an API key, provide the mTLS certificate and private key as follows: ```ruby client = Temporalio::Client.connect( '', # Endpoint '.', # Namespace tls: Temporalio::Client::Connection::TLSOptions.new( client_cert: File.read('my-client-cert.pem'), client_private_key: File.read('my-client-key.pem') ) ) ``` For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server). `TLSOptions` is static: the certificate you pass to `Client.connect` is fixed for the lifetime of that client. To rotate an mTLS client certificate without restarting your Worker, connect a new client with the new certificate, then assign it to the running `Worker`'s `client`: ```ruby new_client = Temporalio::Client.connect( '', # Endpoint '.', # Namespace tls: Temporalio::Client::Connection::TLSOptions.new( client_cert: File.read('my-client-cert-new.pem'), client_private_key: File.read('my-client-key-new.pem') ) ) # my_worker is the Worker instance already running against the old client my_worker.client = new_client ``` The Worker starts using `new_client` for subsequent calls to the Temporal Service; calls already in flight on the old client finish normally. ## Start a Workflow To start a Workflow Execution, supply: - A Task Queue - A Workflow Type - Input arguments - Workflow options such as Workflow Id To start a Workflow Execution in Ruby, use either the `start_workflow` or `execute_workflow` methods in the Client. You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) and [Task Queue](/task-queue) in the parameters given to the method. ```ruby result = my_client.execute_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue' ) puts "Result: #{result}" ``` ## Get Workflow results Once a Workflow Execution is started, the Workflow Id and Run Id can be used to uniquely identify it. You can block until the result is available, or retrieve it later using the handle. You can also use Queries to access Workflow state and results while the Workflow is running. Use `start_workflow` or `workflow_handle` on the Client to return a Workflow handle. Then use the `result` method to await on the result of the Workflow. ```ruby handle = my_client.workflow_handle('my-workflow-id') result = handle.result puts "Result: #{result}" ``` --- # Integrations - Ruby SDK Source: https://docs.temporal.io/develop/ruby/integrations > This section covers integrations with the Ruby SDK The following integrations are available for the Temporal Ruby SDK. - [Rails](/develop/ruby/integrations/rails-integration) — Integrate Temporal durable workflows into Ruby on Rails applications. _(Ruby · Framework)_ --- # Rails integration - Ruby SDK Source: https://docs.temporal.io/develop/ruby/integrations/rails-integration > Use the Temporal Ruby SDK with Ruby on Rails, including conventions for ActiveRecord, ActiveModel, and handling lazy/eager loading. Temporal Ruby SDK is a generic Ruby library that can work in any Ruby environment. However, there are some common conventions for Rails users to be aware of. See the [rails_app sample](https://github.com/temporalio/samples-ruby/tree/main/rails_app) for an example of using Temporal from Rails. ## ActiveRecord For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for Workflows/Activities and translate to/from existing models as needed. See the [ActiveModel section](/develop/ruby/best-practices/data-handling/data-conversion#active-model) on how to do this with ActiveModel objects. ## Lazy/Eager Loading By default, Rails eagerly loads all application code on application start in production, but lazily loads it in non-production environments. Temporal Workflows by default disallow use of IO during the Workflow run. With lazy loading enabled in dev/test environments, when an Activity class is referenced in a Workflow before it has been explicitly required, it can give an error like: ``` Cannot access File path from inside a workflow. If this is known to be safe, the code can be run in a Temporalio::Workflow::Unsafe.illegal_call_tracing_disabled block. ``` This comes from bootsnap via zeitwerk because it is lazily loading a class/module at Workflow runtime. It is not good to lazily load code during a Workflow run because it can be side effecting. Workflows and the classes they reference should be eagerly loaded. To resolve this, either always eagerly load (for example, `config.eager_load = true`) or explicitly require what is used by a workflow at the top of the file. Note, this only affects non-production environments. --- # Platform - Ruby SDK Source: https://docs.temporal.io/develop/ruby/platform > This section explains how to implement platform with the Ruby SDK ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Platform - [Observability](/develop/ruby/platform/observability) - [Enriching the UI](/develop/ruby/platform/enriching-ui) --- # Enriching the user interface - Ruby SDK Source: https://docs.temporal.io/develop/ruby/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the Ruby SDK. Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: ```ruby require 'temporalio/client' # Create client client = Temporalio::Client.connect('localhost:7233') # Start a workflow with static summary and details handle = client.start_workflow( 'YourWorkflow', 'workflow input', id: 'your-workflow-id', task_queue: 'your-task-queue', static_summary: 'Order processing for customer #12345', static_details: 'Processing premium order with expedited shipping' ) ``` `static_summary:` is a single-line description that appears in the Workflow list view, limited to 200 bytes. `static_details:` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use `execute_workflow` for synchronous execution: ```ruby # Execute workflow synchronously result = client.execute_workflow( 'YourWorkflow', 'workflow input', id: 'your-workflow-id', task_queue: 'your-task-queue', static_summary: 'Order processing for customer #12345', static_details: 'Processing premium order with expedited shipping' ) ``` #### Inside the Workflow Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```ruby require 'temporalio' class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Get the current details current_details = Temporalio::Workflow.current_details Temporalio::Workflow.logger.info("Current details: #{current_details}") # Set/update the current details Temporalio::Workflow.current_details = 'Updated workflow details with new status' 'Workflow completed' end end ``` #### Adding Summary to Activities and Timers You can attach a `summary:` to activities when starting them from within a Workflow: ```ruby require 'temporalio' class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Execute an activity with a summary result = Temporalio::Workflow.execute_activity( 'YourActivity', input, start_to_close_timeout: 10, summary: 'Processing user data' ) result end end ``` Similarly, you can attach a `summary:` to timers within a Workflow: ```ruby require 'temporalio' class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Create a timer with a summary Temporalio::Workflow.sleep(300, summary: 'Waiting for payment confirmation') 'Timer completed' end end ``` The input format for `summary:` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the workflow details page, you'll find the workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the workflow - **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available: Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an event, the summary is also prominently displayed in the detailed view. --- # Observability - Ruby SDK Source: https://docs.temporal.io/develop/ruby/platform/observability > Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility using the Ruby SDK. This page covers capabilities related to viewing the state of the application, including: - [Metrics](#metrics) - [Tracing](#tracing) - [Logging](#logging) - [Visibility](#visibility) The observability guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application). This includes viewing [Workflow Executions](/workflow-execution) tracked by the [Temporal Platform](/temporal#temporal-platform), as well as inspecting state at any point during execution. ## Emit metrics Each Temporal SDK can optionally emit metrics from either the Client or Worker process. Metrics can be scraped by systems like Prometheus, and graphs can be created using tools like Grafana. - For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide. - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). Metrics in Ruby are configured on the `metrics` argument of the `telemetry` argument when creating a global `Temporalio::Runtime`. That object should be created globally and should be used for all clients; therefore, you should configure this before any other Temporal code. ## Set a Prometheus endpoint The following example exposes a Prometheus endpoint on port `9000`. ```ruby Temporalio::Runtime.default = Temporalio::Runtime.new( telemetry: Temporalio::Runtime::TelemetryOptions.new( metrics: Temporalio::Runtime::MetricsOptions.new( prometheus: Temporalio::Runtime::PrometheusMetricsOptions.new( bind_address: '0.0.0.0:9000' ) ) ) ) ``` ### Custom metric handling Instead of Prometheus or OpenTelemetry, an instance of `Temporalio::Runtime::MetricBuffer` can be provided as a `buffer` argument to the `MetricsOptions`. `retrieve_updates` can then be periodically called on the buffer to get metric updates. ## Setup Tracing Tracing enables observability into the sequence of calls across your application, including Workflows and Activities. OpenTelemetry tracing for clients, activities, and workflows can be enabled using the `Temporalio::Contrib::OpenTelemetry::TracingInterceptor`. Specifically, when creating a client, set the interceptor like so: ```ruby require 'opentelemetry/api' require 'opentelemetry/sdk' require 'temporalio/client' require 'temporalio/contrib/open_telemetry' # ... assumes my_otel_tracer_provider is a tracer provider created by the user my_tracer = my_otel_tracer_provider.tracer('my-otel-tracer') my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', interceptors: [Temporalio::Contrib::OpenTelemetry::TracingInterceptor.new(my_tracer)] ) ``` When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution. ## Log from a Workflow Logging enables you to capture and persist important execution details from your Workflow and Activity code. Logging uses the Ruby standard logging APIs. You can find the log levels supported in [their official documentation](https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html). The Temporal SDK core normally uses `WARN` as its default logging level. The `logger` can be set when connecting a client. The following example shows logging on the console and sets the level to `INFO`. ```ruby require 'logger' require 'temporalio/client' my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', logger: Logger.new($stdout, level: Logger::INFO) ) ``` You can log from a Workflow using `Temporalio::Workflow.logger` which is a special instance of Ruby's `Logger` that appends workflow details to every log and does not log during replay. ```ruby Temporalio::Workflow.logger.info("Some log #{some_value}") ``` There's also one for use in activities that appends Activity details to every log: ```ruby Temporalio::Activity::Context.current.logger.info("Some log #{some_value}") ``` ## Use Visibility APIs Visibility refers to Temporal features for listing, filtering, and inspecting Workflow Executions. ### Use Search Attributes - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime`, and `ExecutionStatus` are automatically indexed. - [Custom Search Attributes](/search-attribute#custom-search-attribute) let you store domain-specific metadata for Workflows. The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. - [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: - Create a new Search Attribute in your Temporal Service in the CLI or Web UI. - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text` - Replace `CustomKeywordField` with the name of your Search Attribute. - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList` - Set the value of the Search Attribute for a Workflow Execution: - On the Client by including it as an argument when starting the Execution. - In the Workflow by calling `Temporalio::Workflow.upsert_search_attributes`. - Read the value of the Search Attribute: - On the Client by calling `describe` on a `WorkflowHandle`. - In the Workflow by looking at `Temporalio::Workflow.search_attributes`. - Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter): - [In the Temporal CLI](/cli/command-reference/operator#list-2) - In code by calling `list_workflows`. ### List Workflow Executions Use the [list_workflows](https://ruby.temporal.io/Temporalio/Client.html#list_workflows-instance_method) method on the Client and pass a [List Filter](/list-filter) as an argument to filter the listed Workflows. The result is a lazy enumerator/enumerable. ```ruby my_client.list_workflows("WorkflowType='GreetingWorkflow'").each do |wf| puts "Workflow: #{wf.id}" end ``` ### Set Custom Search Attributes After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create`or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. To set custom Search Attributes, use the `search_attributes` parameter for `start_workflow` or `execute_workflow`. Keys should be predefined for reuse. ```ruby # Predefined search attribute key, usually a global somewhere MY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new( 'my-keyword', Temporalio::SearchAttributes::IndexedValueType::KEYWORD ) # ... # Start workflow with the search attribute set handle = my_client.start_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue', search_attributes: Temporalio::SearchAttributes.new({ MY_KEYWORD_KEY => 'some-value' }) ) ``` ### Upsert Search Attributes You can upsert Search Attributes to add, update, or remove Search Attributes from within Workflow code. To upsert custom Search Attributes, use the [`upsert_search_attributes`](https://ruby.temporal.io/Temporalio/Workflow.html#upsert_search_attributes-class_method) method with a set of updates. Keys should be predefined for reuse. ```ruby # Predefined search attribute key, usually a global somewhere MY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new( 'my-keyword', Temporalio::SearchAttributes::IndexedValueType::KEYWORD ) # ... class MyWorkflow < Temporalio::Workflow::Definition def execute # ... Temporalio::Workflow.upsert_search_attributes(MY_KEYWORD_KEY.value_set('some-new-value')) # ... end end ``` --- # Set up your local with the Ruby SDK Source: https://docs.temporal.io/develop/ruby/set-up-local-ruby > Configure your local development environment to get started developing with Temporal # Quickstart This guide walks you through setting up the Temporal Ruby SDK and running your first Workflow. In just a few steps, you'll install the SDK and start a local development server. To validate that your local environment is correctly installed, we will execute a Workflow that will output "Hello, Temporal". ## Installation This step sets up a new Ruby project using Bundler and installs the Temporal Ruby SDK. We recommend using [Bundler](https://bundler.io/) to manage your Ruby project dependencies, including the Temporal SDK. These tutorials assume Ruby 3.4.3 or higher. Follow the steps to create a directory, initialize the project with a `Gemfile`, and add the Temporal SDK. **Note:** - Only macOS ARM/x64 and Linux ARM/x64 are supported. - Source gem is published but **cannot be built directly**. - Windows (MinGW) is not supported. - `fibers`/`async` are only supported on Ruby **3.3+**. - See [Platform Support](#) for full details. **1. Check your Ruby version:** ```bash ruby -v ``` You should see output like `ruby 3.4.3`. Ruby 3.2+ is required. We recommend Ruby 3.4.3. **2. Create your project folder:** ```bash mkdir temporal-project cd temporal-project ``` **3. Initialize with Bundler:** ```bash bundle init ``` **4. Add the Temporal Ruby SDK:** ```bash bundle add temporalio ``` You should see output like: ```bash Fetching gem metadata from https://rubygems.org/... Resolving dependencies... Installing temporalio 0.4.0 (arm64-darwin) Bundle complete! 1 Gemfile dependency, 6 gems now installed. ``` **5. Install dependencies:** ```bash bundle install ``` ## Install Temporal CLI The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI. **macOS** Install the Temporal CLI using Homebrew: ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal Ruby SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Create the Activity Create an Activity file (say_hello_activity.rb): ```ruby require 'temporalio/activity' # Implementation of a simple activity class SayHelloActivity < Temporalio::Activity::Definition def execute(name) "Hello, #{name}!" end end ``` ### 2. Create the Workflow Create a Workflow file (say_hello_workflow.rb): ```ruby require 'temporalio/workflow' require_relative 'say_hello_activity' class SayHelloWorkflow < Temporalio::Workflow::Definition def execute(name) Temporalio::Workflow.execute_activity( SayHelloActivity, name, schedule_to_close_timeout: 300 ) end end ``` ### 3. Create and Run the Worker With your Activity and Workflow defined, you need a Worker to execute them. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](/evaluate/understanding-temporal#workers) and a [deep dive into Workers](/workers). Create a Worker file (worker.rb): ```ruby require 'temporalio/client' require 'temporalio/worker' require_relative 'say_hello_activity' require_relative 'say_hello_workflow' # Create a client client = Temporalio::Client.connect('localhost:7233', 'default') # Create a worker with the client, activities, and workflows worker = Temporalio::Worker.new( client:, task_queue: 'my-task-queue', workflows: [SayHelloWorkflow], # There are various forms an activity can take, see "Activities" section for details activities: [SayHelloActivity] ) # Run the worker until SIGINT. This can be done in many ways, see "Workers" section for details. worker.run(shutdown_signals: ['SIGINT']) ``` Run the Worker: ```bash ruby worker.rb ``` ### 4. Execute the Workflow Now that your Worker is running, it's time to start a Workflow Execution. Create a separate file called starter.rb: ```ruby require 'temporalio/client' require_relative 'say_hello_workflow' # Create a client client = Temporalio::Client.connect('localhost:7233', 'default') # Run workflow result = client.execute_workflow( SayHelloWorkflow, 'Temporal', # This is the input to the workflow id: 'my-workflow-id', task_queue: 'my-task-queue' ) puts "Result: #{result}" ``` Then run: ```bash ruby starter.rb ``` ### Verify Success If everything is working correctly, you should see: - Worker processing the workflow and activity - Output: `Workflow result: Hello, Temporal!` - Workflow Execution details in the [Temporal Web UI](http://localhost:8233) - [Run your first Temporal Application](https://learn.temporal.io/getting_started/ruby/first_program_in_ruby/): Create a basic Workflow and run it with the Temporal Ruby SDK - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Workers - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workers > This section explains how to implement Workers with the Ruby SDK ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Workers - [Worker processes](/develop/ruby/workers/run-worker-process) --- # Run a Worker - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workers/run-worker-process > Create and run a Temporal Worker using the Ruby SDK. ## Create and run a Worker Create a `Temporalio::Worker` with a [Temporal Client](/develop/ruby/client/temporal-client), the Task Queue to poll, and the Workflows and Activities it can execute. Call `run` to start polling. [features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb) ```rb worker = Temporalio::Worker.new( client: client, task_queue: 'my-task-queue', workflows: [GreetingWorkflow], activities: [SayHello] ) worker.run ``` `run` blocks until the Worker shuts down. To run several Workers in one process, use `Temporalio::Worker.run_all`, which returns once every Worker it was given has stopped. ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Pass Workflow classes in `workflows` and Activities in `activities`. For Activities you can pass either the class or an instance: ```ruby worker = Temporalio::Worker.new( client:, task_queue: 'my-task-queue', workflows: [GreetingWorkflow, OrderWorkflow], activities: [SayHello, ChargeCard.new(payment_client)] ) ``` Passing a class makes the Worker instantiate it for each Activity Execution. Passing an instance reuses that object for every execution, which is how Activities share state such as a database client. A shared instance must be thread-safe. ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/ruby/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options `Temporalio::Worker.new` takes keyword arguments that control concurrency limits, pollers, timeouts, and caching, including `max_concurrent_activities`, `max_concurrent_workflow_tasks`, and `max_cached_workflows`. The defaults work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `deployment_options`, then set a versioning behavior on each Workflow. [features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb) ```rb worker = Temporalio::Worker.new( client: client, task_queue: 'my-task-queue', workflows: [VersionedGreetingWorkflow], activities: [SayHello], deployment_options: Temporalio::Worker::DeploymentOptions.new( version: Temporalio::WorkerDeploymentVersion.new( deployment_name: 'my-app', build_id: '1.0' ), use_worker_versioning: true ) ) ``` Declare the behavior in the Workflow class with `workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED` or `AUTO_UPGRADE`, or set a default for the whole Worker with `default_versioning_behavior` on `DeploymentOptions`. A versioning behavior applies only to a Worker that has versioning enabled. If a Workflow declares one and its Worker does not enable versioning, the server rejects the Workflow Task and the Task retries instead of failing outright. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker Pass the signals that should stop the Worker to `run`. The Worker stops polling for new Tasks and waits for in-flight Tasks to finish. [features/snippets/worker/worker.rb](https://github.com/temporalio/features/blob/main/features/snippets/worker/worker.rb) ```rb worker.run(shutdown_signals: %w[SIGINT SIGTERM]) ``` You can also pass a block to `run`, which shuts the Worker down when the block completes, or stop the Worker with a `Temporalio::Cancellation`. `Temporalio::Worker.run_all` takes the same `shutdown_signals`, `cancellation`, and block, and applies them to every Worker it was given. See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Serverless Workers - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the Ruby SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. Serverless Workers run on compute that Temporal starts and stops for you, rather than on long-lived processes you operate. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**GCP Cloud Run**](/develop/ruby/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in. --- # Serverless Workers on GCP Cloud Run - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the Ruby SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Ruby Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific gem. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running Ruby Worker, then pass `deployment_options` to `Temporalio::Worker.new` to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```ruby require 'temporalio/client' require 'temporalio/worker' client = Temporalio::Client.connect( ENV.fetch('TEMPORAL_ADDRESS'), ENV.fetch('TEMPORAL_NAMESPACE'), api_key: ENV.fetch('TEMPORAL_API_KEY'), tls: true ) worker = Temporalio::Worker.new( client:, task_queue: ENV.fetch('TEMPORAL_TASK_QUEUE'), workflows: [GreetingWorkflow], activities: [SayHello], deployment_options: Temporalio::Worker::DeploymentOptions.new( version: Temporalio::WorkerDeploymentVersion.new( deployment_name: 'my-app', build_id: 'build-1' ), use_worker_versioning: true, default_versioning_behavior: Temporalio::VersioningBehavior::PINNED ) ) worker.run ``` `deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. Every Workflow needs a [versioning behavior](/worker-versioning#versioning-behaviors), either `PINNED` or `AUTO_UPGRADE`. Setting `default_versioning_behavior` as shown above covers every Workflow on the Worker. To set the behavior per Workflow instead, call `workflow_versioning_behavior` in the Workflow class: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED def execute(name) # ... end end ``` If versioning is on and neither is set, the Worker raises an error at startup rather than polling. For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/ruby/workers/run-worker-process). ## Configure the Temporal connection Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext. The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace. For the shared configuration format that other Temporal tools read, see [Environment configuration](/develop/environment-configuration). ## Package the Worker image The Ruby SDK ships precompiled gems per platform, so install it in the image rather than building the native extension from source: ```dockerfile FROM ruby:3.3-slim WORKDIR /app RUN gem install temporalio --no-document COPY worker.rb ./ CMD ["ruby", "worker.rb"] ``` Installing through Bundler in a container can select the source gem instead of the precompiled one, which then fails to build without a Rust toolchain. If you use Bundler, add the target platform to the lockfile with `bundle lock --add-platform x86_64-linux`. ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/ruby/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over: ```ruby class Process < Temporalio::Activity::Definition def execute(items) items.each_with_index do |item, i| Temporalio::Activity::Context.current.heartbeat(i) # ... process item end 'done' end end ``` For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Ruby SDK](/develop/ruby/platform/observability) and the [SDK metrics reference](/references/sdk-metrics). --- # Workflows - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows > This section explains how to implement Workflows with the Ruby SDK ![Ruby SDK Banner](/img/assets/banner-ruby-temporal.png) ## Workflows - [Workflow basics](/develop/ruby/workflows/basics) - [Child Workflows](/develop/ruby/workflows/child-workflows) - [Continue-As-New](/develop/ruby/workflows/continue-as-new) - [Cancellation](/develop/ruby/workflows/cancellation) - [Timeouts](/develop/ruby/workflows/timeouts) - [Message passing](/develop/ruby/workflows/message-passing) - [Schedules](/develop/ruby/workflows/schedules) - [Timers](/develop/ruby/workflows/timers) - [Futures](/develop/ruby/workflows/futures) - [Dynamic Workflow](/develop/ruby/workflows/dynamic-workflow) - [Versioning](/develop/ruby/workflows/versioning) --- # Workflow basics - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/basics > This section explains Workflow basics with the Ruby SDK ## Develop a Workflow Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal Ruby SDK programming model, Workflows are defined as classes. Have the Workflow class extend `Temporalio::Workflow::Definition` to define a Workflow. The entrypoint is the `execute` method. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute(name) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) end end ``` Temporal Workflows may have any number of custom parameters. However, we strongly recommend that hashes or objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. ### Customize Workflow Type Workflows have a Type that are referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. You can customize the Workflow name with a custom name in a `workflow_name` class method call on the class. The Workflow name defaults to the unqualified class name. ```ruby class MyWorkflow < Temporalio::Workflow::Definition # Customize the name workflow_name :MyDifferentWorkflowName def execute(name) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) end end ``` ### Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). The `workflow_init` class method above `initialize` gives it access to [Workflow input](/handling-messages#workflow-initializers). When you use the `workflow_init` on your constructor, you give the constructor the same Workflow parameters as your `execute` method. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/ruby/client/temporal-client#start-workflow). The Workflow input arguments are also passed to your `execute` method. That always happens, whether or not you use the `workflow_init` class method above `initialize`. Here's an example. The constructor and `execute` must have the same parameters with the same types: ```ruby class WorkflowInitWorkflow < Temporalio::Workflow::Definition workflow_init def initialize(input) @name_with_title = "Knight #{input['name']}" end def execute(input) Temporalio::Workflow.wait_condition { @title_has_been_checked } "Hello, #{@name_with_title}" end end ``` ## Workflow logic requirements Temporal Workflows [must be deterministic](/workflow-definition#deterministic-constraints), which includes Ruby Workflows. This means there are several things Workflows cannot do such as: - Perform IO (network, disk, stdio, etc) - Access/alter external mutable state - Do any threading - Do anything using the system clock (for example, `Time.Now`) - Make any random calls - Make any not-guaranteed-deterministic calls To prevent illegal Workflow calls, a call tracer is put on the Workflow thread that raises an exception if any illegal calls are made. Which calls are illegal is configurable in the Worker options. The SDK provides replay-safe alternatives for common needs. ### Logging Use [`Temporalio::Workflow.logger`](https://ruby.temporal.io/Temporalio/Workflow.html#logger-class_method) instead of `puts` or a `Logger` you create yourself. The `Logger` class is on the default illegal call list, and the SDK logger appends Workflow details to every log and skips logging during replay: ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute(name) Temporalio::Workflow.logger.info("Starting workflow for #{name}") # ... end end ``` For logger configuration, see [Observability: Log from a Workflow](/develop/ruby/platform/observability#logging). ### Random numbers and UUIDs Use [`Temporalio::Workflow.random`](https://ruby.temporal.io/Temporalio/Workflow.html#random-class_method) to get a `Random` instance seeded per Workflow Execution. The SDK requires `random/formatter`, so this instance also has the standard library's [`Random::Formatter#uuid`](https://rubyapi.org/4.0/o/random/formatter#method-i-uuid) method. Use it instead of `SecureRandom.uuid`: ```ruby value = Temporalio::Workflow.random.rand(1..100) unique_id = Temporalio::Workflow.random.uuid ``` Don't use `SecureRandom`, `Kernel#rand`, `Kernel#srand`, or `Random.new` in Workflow code. They're on the default illegal call list, and the call tracer raises a `Temporalio::Workflow::NondeterminismError` when it detects them. Access the instance each time you need it rather than storing it in an instance variable. The SDK may recreate it with a different seed, such as after a Workflow reset. ### Current time Use [`Temporalio::Workflow.now`](https://ruby.temporal.io/Temporalio/Workflow.html#now-class_method) instead of `Time.now`. It returns the UTC time of the last Workflow Task, which is consistent across replays: ```ruby current_time = Temporalio::Workflow.now ``` To wait, use [`Temporalio::Workflow.sleep`](https://ruby.temporal.io/Temporalio/Workflow.html#sleep-class_method) instead of `Kernel#sleep`. ### Detecting replay (advanced) Use [`Temporalio::Workflow::Unsafe.replaying?`](https://ruby.temporal.io/Temporalio/Workflow/Unsafe.html#replaying?-class_method) to guard code that should only run on the first execution, such as emitting metrics or sending external notifications from an Interceptor. > **⚠️ Caution:** > > Never use this to affect Workflow business logic. Branching on replay status breaks determinism. > ```ruby unless Temporalio::Workflow::Unsafe.replaying? emit_metric('workflow_started', 1) end ``` If your goal is to always take action when something new is happening, check that [`Temporalio::Workflow::Unsafe.replaying_history_events?`](https://ruby.temporal.io/Temporalio/Workflow/Unsafe.html#replaying_history_events?-class_method) is false instead. That is false during read-only operations like Queries and Update validators. This is what the SDK's built-in logger uses internally. --- # Cancellation - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/cancellation This page shows how to interrupt a Workflow Execution. You can interrupt a Workflow Execution in one of the following ways: - [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution. - [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. ## Cancellation To give a Workflow and its Activities the ability to be cancelled, do the following: - Handle a Cancellation request within a Workflow. - Set Activity Heartbeat Timeouts. - Listen for and handle a Cancellation request within an Activity. - Send a Cancellation request from a Temporal Client. ## Handle Cancellation in Workflow Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on Cancellation to perform cleanup. Cancellation Requests on Workflows cancel the `Temporalio::Workflow.cancellation` which is a `Temporalio::Cancellation` that effectively serves as a cancellation token. This is the cancellation that is implicitly used for all calls within the workflow as well (for example, Timers, Activities, etc) and therefore cancellation is propagated to them to be handled and bubble out. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute # Whether this workflow waits on the activity to handle the cancellation or not is # dependent upon the cancellation_type parameter. We leave the default here which # sends the cancellation but does not wait on it to be handled. Temporalio::Workflow.execute_activity(MyActivity, start_to_close_timeout: 100) rescue Temporalio::Error => e # For this sample, we only want to execute cleanup when it's a cancellation raise unless Temporalio::Error.canceled?(e) # Call a cleanup activity. We have to do this with a new/detached cancellation # because the default workflow-level one is already canceled at this point. Temporalio::Workflow.execute_activity( MyCleanupActivity, start_to_close_timeout: 100, cancellation: Temporalio::Cancellation.new ) # Re-raise the original exception raise end end ``` ## Handle Cancellation in an Activity Ensure that the Activity is [Heartbeating](/develop/ruby/activities/timeouts#activity-heartbeats) to receive the Cancellation request and stop execution. Also make sure that the [Heartbeat Timeout](/develop/ruby/activities/timeouts#heartbeat-timeout) is set on the Activity Options when calling from the Workflow. An Activity Cancellation Request raises a `Temporalio::Error::CanceledError` in the Activity. ```ruby class MyActivity < Temporalio::Activity::Definition def execute # This is a naive loop simulating work, but similar heartbeat/cancellation logic # applies to other scenarios as well loop do # Send heartbeat Temporalio::Activity::Context.current.heartbeat # Sleep before heartbeating again sleep(3) end rescue Temporalio::Error::CanceledError raise 'Canceled!' end end ``` ## Request Cancellation Use `cancel` on the `WorkflowHandle` to cancel a Workflow Execution. ```ruby # Get a workflow handle by its workflow ID. This could be made specific to a run by # passing run ID. This could also just be a handle that is returned from # start_workflow instead. handle = my_client.workflow_handle('my-workflow-id') # Send cancellation. This returns when cancellation is received by the server. Wait on # the handle's result to wait for cancellation to be applied. handle.cancel ``` By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation is used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute # Create a new cancellation linked to the workflow one, so that it inherits # cancellation that comes from the workflow. Users can choose to make it # completely detached by not providing a parent. cancellation, cancel_proc = Temporalio::Cancellation.new( Temporalio::Workflow.cancellation ) # Start the activity in the background. Whether this workflow waits on the activity # to handle the cancellation or not is dependent upon the cancellation_type # parameter. We leave the default here which sends the cancellation but does not wait # on it to be handled. future = Temporalio::Future.new do Temporalio::Workflow.execute_activity( MyActivity, start_to_close_timeout: 100, cancellation: ) end # Wait 5 minutes, then cancel it Temporalio::Workflow.sleep(5 * 60) cancel_proc.call # Wait on the activity which will raise an activity error with a cause of # cancellation which will fail the workflow future.wait end end ``` ## Termination To Terminate a Workflow Execution in Ruby, use the `terminate` method on the Workflow handle. ```ruby # Get a workflow handle by its workflow ID. This could be made specific to a run by # passing run ID. This could also just be a handle that is returned from # start_workflow instead. handle = my_client.workflow_handle('my-workflow-id') # Terminate handle.terminate ``` Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the UI when that happens. ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/child-workflows > Start a Child Workflow Execution and set a Parent Close Policy using Temporal Ruby SDK. This page shows how to do the following: - [Start a Child Workflow Execution](#child-workflows) using the Ruby SDK - [Set a Parent Close Policy](#parent-close-policy) using the Ruby SDK ## Start a Child Workflow Execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In Ruby, calling `start_child_workflow` or `execute_child_workflow` internally waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call returns. To spawn a Child Workflow Execution in Ruby, use the `execute_child_workflow` method which starts the Child Workflow and waits for completion or use the `start_child_workflow` method to start a Child Workflow and return its handle. This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running. > **📝 Note:** > > `execute_child_workflow` is a helper method for `start_child_workflow(...).result`. > ```ruby Temporalio::Workflow.execute_child_workflow(MyChildWorkflow, 'my-workflow-arg') ``` ## Set a Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy option is set to terminate the Child Workflow Execution. Set the `parent_close_policy` parameter for `execute_child_workflow` or `start_child_workflow` to specify the behavior of the Child Workflow when the Parent Workflow closes. ```ruby Temporalio::Workflow.execute_child_workflow( MyChildWorkflow, 'my-workflow-arg', parent_close_policy: Temporalio::Workflow::ParentClosePolicy::ABANDON ) ``` --- # Continue-As-New - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/continue-as-new > Use Continue-As-New with the Temporal Ruby SDK to manage Workflow Event Histories, ensuring optimal performance by starting new Executions seamlessly. This page describes how to Continue-As-New using the Temporal Ruby SDK. [Continue-As-New](/workflow-execution/continue-as-new) enables a Workflow Execution to close successfully and create a new Workflow Execution in a single atomic operation if the number of Events in the Event History is becoming too large. The Workflow Execution spawned from the use of Continue-As-New has the same Workflow Id, a new Run Id, and a fresh Event History and is passed all the appropriate parameters. > **⚠️ Caution:** > > As a precautionary measure, the Workflow Execution's Event History is limited to [51,200 Events](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L441) or [50 MB](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L425) and will warn you after 10,240 Events or 10 MB. > To prevent a Workflow Execution Event History from exceeding this limit and failing, use Continue-As-New to start a new Workflow Execution with a fresh Event History. A very large Event History can adversely affect the performance of a Workflow Execution. For example, in the case of a Workflow Worker failure, the full Event History must be pulled from the Temporal Service and given to another Worker via a Workflow Task. If the Event history is very large, it may take some time to load it. The Continue-As-New feature enables developers to complete the current Workflow Execution and start a new one atomically. The new Workflow Execution has the same Workflow Id, but a different Run Id, and has its own Event History. ## Continue-As-New in Ruby To Continue-As-New in Ruby, raise a `Temporalio::Workflow::ContinueAsNewError` from inside your Workflow, which will stop the Workflow immediately and Continue-As-New. ```ruby raise Temporalio::Workflow::ContinueAsNewError.new('my-new-arg') ``` > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution. > --- # Dynamic Workflow - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/dynamic-workflow > This section explains Dynamic Workflows with the Ruby SDK ## Set a Dynamic Workflow A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. A Workflow can be made dynamic by invoking `workflow_dynamic` class method at the top of the definition. You must register the Workflow with the Worker before it can be invoked. Only one Dynamic Workflow can be present on a Worker. Often, dynamic is used in conjunction with `workflow_raw_args` which does not convert arguments but instead passes them through as a splatted array of `Temporalio::Converters::RawValue` instances. ```ruby class MyDynamicWorkflow < Temporalio::Workflow::Definition # Make this the dynamic workflow and accept raw args workflow_dynamic workflow_raw_args def execute(*raw_args) # Require a single arg for our workflow raise Temporalio::Error::ApplicationError, 'One arg expected' unless raw_args.size == 1 # Use payload converter to convert it name = Temporalio::Workflow.payload_converter.from_payload(raw_args.first.payload) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) end end ``` --- # Workflow futures - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/futures > This section explains Workflow futures with the Ruby SDK ## Workflow Futures `Temporalio::Workflow::Future` can be used for running things in the background or concurrently. Temporal provides Workflow-safe wrappers around some core language features in cases like these. `Temporalio::Workflow::Future` is a safe wrapper around `Fiber.schedule` for running multiple Activities at once. The Ruby SDK also provides `Workflow.wait_condition` for awaiting a result. Futures are never used implicitly, but they work with all Workflow code and constructs. For instance, to run 3 activities and wait for them all to complete, something like this can be written: ```ruby # Start 3 activities in background fut1 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity1, schedule_to_close_timeout: 300) end fut2 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity2, schedule_to_close_timeout: 300) end fut3 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity3, schedule_to_close_timeout: 300) end # Wait for them all to complete Temporalio::Workflow::Future.all_of(fut1, fut2, fut3).wait Temporalio::Workflow.logger.info("Got: #{fut1.result}, #{fut2.result}, #{fut3.result}") ``` Or, say, to wait on the first of 5 activities or a timeout to complete: ```ruby # Start 5 activities act_futs = 5.times.map do |i| Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity, "my-arg-#{i}", schedule_to_close_timeout: 300) end end # Start a timer sleep_fut = Temporalio::Workflow::Future.new { Temporalio::Workflow.sleep(30) } # Wait for first act result or sleep fut act_result = Temporalio::Workflow::Future.any_of(sleep_fut, *act_futs).wait # Fail if timer done first raise Temporalio::Error::ApplicationError, 'Timer expired' if sleep_fut.done? # Print act result otherwise Temporalio::Workflow.logger.info("Act result: #{act_result}") ``` There are several other details not covered here about futures, such as how exceptions are handled, how to use a setter proc instead of a block, etc. See the [API documentation](https://ruby.temporal.io/Temporalio/Workflow/Future.html) for details. --- # Message passing - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/message-passing > Develop with Queries, Signals, and Updates using the Temporal Ruby SDK. A Workflow can act like a stateful service that receives messages: Queries, Signals, and Updates. These messages interact with the Workflow via handler methods defined in the Workflow code. Clients use messages to read Workflow state or change its behavior. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview. ## Write message handlers > **ℹ️ Info:** > The code that follows is part of a [working solution](https://github.com/temporalio/samples-ruby/tree/main/message_passing_simple). Follow these guidelines when writing your message handlers: - Message handlers are defined as methods on the Workflow class, decorated by calling one of three class methods before defining the handler method: `workflow_query`, `workflow_signal`, and `workflow_update`. - These also implicitly create class-methods with the same name as the instance methods for use by callers. - The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion). - Prefer single hash/object input parameter to multiple input parameters. Hash/object parameters allow you to add fields without changing the calling signature. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution. Define as a method: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition # ... workflow_query def languages(input) # A query handler returns a value: it can inspect but must not mutate the Workflow state. if input['include_unsupported'] CallGreetingService.greetings.keys.sort else @greetings.keys.sort end end # ... end ``` Or as an attribute reader: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition # This is the equivalent of: # workflow_query # def language # @language # end workflow_query_attr_reader :language # ... end ``` - The `workflow_query` class method can accept arguments. See the API reference docs: [`workflow_query`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_query-class_method). - A Query handler must not modify Workflow state. - You can't perform async blocking operations such as executing an Activity in a Query handler. ### Signal handlers A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition # ... workflow_signal def approve(input) # A signal handler mutates the workflow state but cannot return a value. @approved_for_release = true @approver_name = input['name'] end # ... end ``` - The `workflow_signal` class method can accept arguments. Refer to the API docs: [`workflow_signal`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_signal-class_method). - The handler should not return a value. The response is sent immediately from the server, without waiting for the Workflow to process the Signal. - Signal (and Update) handlers can be asynchronous and blocking. This allows you to use Activities, Child Workflows, durable Timers, wait conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Signal and Update handlers. ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition # ... workflow_update def set_language(new_language) # rubocop:disable Naming/AccessorMethodName # An update handler can mutate the workflow state and return a value. prev = @language.to_sym @language = new_language.to_sym prev end workflow_update_validator(:set_language) def validate_set_language(new_language) # In an update validator you raise any exception to reject the update. raise "#{new_language} is not supported" unless @greetings.include?(new_language.to_sym) end # ... end ``` - The `workflow_update` class method can take arguments as described in the API reference docs for [`workflow_update`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method). - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them. - Define an Update validator with the [`workflow_update_validator`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method) class method invoked before defining the method. The first parameter when declaring the validator is the name of the Update handler method. The validator must accept the same argument types as the handler and should not return a value. - Accepting and rejecting Updates with validators: - To reject an Update, raise an exception of any type in the validator. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic. - When a Validator raises an error, the Update is rejected, the Update is not run, and `WorkflowExecutionUpdateAccepted` _won't_ be added to the Event History. The caller receives an "Update failed" error. - Use [`current_update_info`](https://ruby.temporal.io/Temporalio/Workflow.html#current_update_info-class_method) to obtain information about the current Update. This includes the Update ID, which can be useful for deduplication when using Continue-As-New: see [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). - Update (and Signal) handlers can be asynchronous and blocking. This allows you to use Activities, Child Workflows, durable Timers, wait conditions, and more. See [Async handlers](#async-handlers) and [Workflow message passing](/encyclopedia/workflow-message-passing) for guidelines on safely using async Update and Signal handlers. ## Send messages To send Queries, Signals, or Updates you call methods on a [`WorkflowHandle`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html) instance. To obtain the Workflow handle, you can: - Use [`Client#start_workflow`](https://ruby.temporal.io/Temporalio/Client.html#start_workflow-instance_method) to start a Workflow and return its handle. - Use the [`Client#workflow_handle`](https://ruby.temporal.io/Temporalio/Client.html#workflow_handle-instance_method) method to retrieve a Workflow handle by its Workflow Id. For example: ```ruby client = Temporalio::Client.connect('localhost:7233', 'default') handle = client.start_workflow( MessagePassingSimple::GreetingWorkflow, id: 'message-passing-simple-sample-workflow-id', task_queue: 'message-passing-simple-sample' ) ``` To check the argument types required when sending messages -- and the return type for Queries and Updates -- refer to the corresponding handler method in the Workflow Definition. > **⚠️ Warning:** > Using Continue-as-New and Updates > > - Temporal _does not_ support Continue-as-New functionality within Update handlers. > - Complete all handlers _before_ using Continue-as-New. > - Use Continue-as-New from your main Workflow Definition method, just as you would complete or fail a Workflow Execution. > ### Send a Query Call a Query method with [`WorkflowHandle#query`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html#query-instance_method): ```ruby supported_languages = handle.query(MessagePassingSimple::GreetingWorkflow.languages, { include_unsupported: false }) ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### From a Client Use [`WorkflowHandle#signal`](https://ruby.temporal.io/Temporalio/Client/WorkflowHandle.html#signal-instance_method) from Client code to send a Signal: ```ruby handle.signal(MessagePassingSimple::GreetingWorkflow.approve, { name: 'John Q. Approver' }) ``` - The call returns when the server accepts the Signal; it does _not_ wait for the Signal to be delivered to the Workflow Execution. - The [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the Workflow's Event History. #### From a Workflow A Workflow can send a Signal to another Workflow, known as an _External Signal_. In this case you need to obtain a Workflow handle for the external Workflow. Use `Temporalio::Workflow.external_workflow_handle`, passing a running Workflow Id, to retrieve a Workflow handle: ```ruby class WorkflowB < Temporalio::Workflow::Definition def execute handle = Temporalio::Workflow.external_workflow_handle('workflow-a-id') handle.signal(WorkflowA.some_signal, 'some signal arg') end end ``` When an External Signal is sent: - A [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated) Event appears in the sender's Event History. - A [WorkflowExecutionSignaled](/references/events#workflowexecutionsignaled) Event appears in the recipient's Event History. #### Signal-With-Start Signal-With-Start allows a Client to send a Signal to a Workflow Execution, starting the Execution if it is not already running. If there's a Workflow running with the given Workflow Id, it will be signaled. If there isn't, a new Workflow will be started and immediately signaled. To use Signal-With-Start, call `signal_with_start_workflow` with a `WithStartWorkflowOperation`: ```ruby client = Temporalio::Client.connect('localhost:7233', 'default') # Create start-workflow operation for use with signal-with-start start_workflow_operation = Temporalio::Client::WithStartWorkflowOperation.new( MyWorkflow, 'my-workflow-input', id: 'my-workflow-id', task_queue: 'my-workflow-task-queue' ) # Perform signal-with-start handle = client.signal_with_start_workflow( MyWorkflow.my_signal, 'signal-input', start_workflow_operation: ) ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A Client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. To send an Update to a Workflow Execution, you can: - Call the Update method with `execute_update` from the Workflow handle and wait for the Update to complete. This code fetches an Update result: ```ruby prev_language = handle.execute_update(MessagePassingSimple::GreetingWorkflow.set_language, :chinese) ``` 2. Use `start_update` to receive a handle as soon as the Update is accepted. It returns a `WorkflowUpdateHandle` - Use this `WorkflowUpdateHandle` later to fetch your results. - Asynchronous Update handlers normally perform long-running async Activities. - `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete. For example: ```ruby # Start an update and then wait for it to complete update_handle = handle.start_update( MessagePassingSimple::GreetingWorkflow.apply_language_with_lookup, :arabic, wait_for_stage: Temporalio::Client::WorkflowUpdateWaitStage::ACCEPTED ) prev_language = update_handle.result ``` For more details, see the "Async handlers" section. #### Update-With-Start > **💡 Tip:** > Stability > > In [Public Preview](/evaluate/development-production-features/release-stages#public-preview) in Temporal Cloud. > > Minimum Temporal Server version [Temporal Server version 1.26](https://github.com/temporalio/temporal/releases/tag/v1.26.2) > [Update-with-Start](/sending-messages#update-with-start) lets you [send an Update](/develop/ruby/workflows/message-passing#send-update-from-client) that checks whether an already-running Workflow with that ID exists: - If the Workflow exists, the Update is processed. - If the Workflow does not exist, a new Workflow Execution is started with the given ID, and the Update is processed before the main Workflow method starts to execute. Use `execute_update_with_start_workflow` to start the Update and wait for the result in one go. Alternatively, use `start_update_with_start_workflow` to start the Update and receive a `WorkflowUpdateHandle`, and then use `update_handle.result` to retrieve the result from the Update. These calls return once the requested Update wait stage has been reached, or when the request times out. - You will need to provide a `WithStartWorkflowOperation` to define the Workflow that will be started if necessary, and its arguments. - You must specify an [id_conflict_policy](/workflow-execution/workflowid-runid#workflow-id-conflict-policy) when creating the `WithStartWorkflowOperation`. Note that a `WithStartWorkflowOperation` can only be used once. Here's an example: ```ruby client = Temporalio::Client.connect('localhost:7233', 'default') # Create start-workflow operation for use with update-with-start start_workflow_operation = Temporalio::Client::WithStartWorkflowOperation.new( MyWorkflow, 'my-workflow-input', id: 'my-workflow-id', task_queue: 'my-workflow-task-queue', id_conflict_policy: Temporalio::WorkflowIDConflictPolicy::USE_EXISTING ) # Perform update-with-start and get update result update_result = client.execute_with_start_workflow( MyWorkflow.my_update, 'update-input', start_workflow_operation: ) # The workflow handle is on the start operation, here's an example of waiting on # workflow result workflow_result = start_workflow_operation.workflow_handle.result ``` ## Message handler patterns This section covers common write operations, such as Signal and Update handlers. It doesn't apply to pure read operations, like Queries or Update Validators. > **💡 Tip:** > > For additional information, see [Inject work into the main Workflow](/handling-messages#injecting-work-into-main-workflow) and [Ensuring your messages are processed exactly once](/handling-messages#exactly-once-message-processing). > ### Add async handlers Signal and Update handlers can be asynchronous as well as blocking. Using asynchronous calls allows you to wait for Activities, Child Workflows, Durable Timers, wait conditions, etc. This expands the possibilities for what can be done by a handler but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls. It's essential to understand the things that could go wrong in order to use asynchronous handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, and the [Controlling handler concurrency](#control-handler-concurrency) and [Waiting for message handlers to finish](#wait-for-message-handlers) sections below. The following code is an Activity that simulates a network call to a remote service: ```ruby class CallGreetingService < Temporalio::Activity::Definition def execute(to_language) # Simulate a network call sleep(0.2) # This intentionally returns nil on not found CallGreetingService.greetings[to_language.to_sym] end def self.greetings @greetings ||= { arabic: 'مرحبا بالعالم', chinese: '你好，世界', english: 'Hello, world', french: 'Bonjour, monde', hindi: 'नमस्ते दुनिया', portuguese: 'Olá mundo', spanish: 'Hola mundo' } end end ``` The following code is a Workflow Update for asynchronous use of the preceding Activity: ```ruby class GreetingWorkflow < Temporalio::Workflow::Definition # ... workflow_update def apply_language_with_lookup(new_language) # Call an activity if it's not there. unless @greetings.include?(new_language.to_sym) # We use a mutex so that, if this handler is executed multiple times, each execution # can schedule the activity only when the previously scheduled activity has # completed. This ensures that multiple calls to apply_language_with_lookup are # processed in order. @apply_language_mutex ||= Mutex.new @apply_language_mutex.synchronize do greeting = Temporalio::Workflow.execute_activity( CallGreetingService, new_language, start_to_close_timeout: 10 ) # The requested language might not be supported by the remote service. If so, we # raise ApplicationError, which will fail the update. The # WorkflowExecutionUpdateAccepted event will still be added to history. (Update # validators can be used to reject updates before any event is written to history, # but they cannot be async, and so we cannot use an update validator for this # purpose.) raise Temporalio::Error::ApplicationError, "Greeting service does not support #{new_language}" unless greeting @greetings[new_language.to_sym] = greeting end end set_language(new_language) end end ``` After updating the code for asynchronous calls, your Update handler can schedule an Activity and await the result. Although an async Signal handler can initiate similar network tasks, using an Update handler allows the Client to receive a result or error once the Activity completes. This lets your Client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, etc. ### Use wait conditions Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. Using a wait condition with [`wait_condition`](https://ruby.temporal.io/Temporalio/Workflow.html#wait_condition-class_method) sets a function that prevents the code from proceeding until the condition is truthy. This is an important feature that helps you control your handler logic. Here are two important use cases for `wait_condition`: - Waiting in a handler until it is appropriate to continue. - Waiting in the main Workflow until all active handlers have finished. The condition state you're waiting for can be updated by and reflect any part of the Workflow code. This includes the main Workflow method, other handlers, or child coroutines spawned by the main Workflow method, and so forth. #### In handlers Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. Using a wait condition with [`wait_condition`](https://ruby.temporal.io/Temporalio/Workflow.html#wait_condition-class_method) sets a function that prevents the code from proceeding until the condition is truthy. This is an important feature that helps you control your handler logic. Consider a `ready_for_update_to_execute` method that runs before your Update handler executes. The `wait_condition` call waits until your condition is met: ```ruby workflow_update def my_update(my_update_input) Temporalio::Workflow.wait_condition { ready_for_update_to_execute(my_update_input) } # ... end ``` Remember: Handlers can execute before the main Workflow method starts. #### Before finishing the Workflow Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result. The Workflow completing may interrupt the handler before it finishes crucial work and cause Client errors when trying retrieve Update results. Use `Temporalio::Workflow.all_handlers_finished?` to address this problem and allow your Workflow to end smoothly: ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute # ... Temporalio::Workflow.wait_condition { Temporalio::Workflow.all_handlers_finished? } 'workflow-result' end end ``` By default, your Worker will log a warning when you allow a Workflow Execution to finish with unfinished handler executions. You can silence these warnings on a per-handler basis by passing the `unfinished_policy` argument to the [`workflow_signal`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_signal-class_method) / [`workflow_update`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_update-class_method) class methods: ```ruby workflow_update unfinished_policy: Temporalio::Workflow::HandlerUnfinishedPolicy::ABANDON def my_update # ... ``` See [Finishing handlers before the Workflow completes](/handling-messages#finishing-message-handlers) for more information. ### Use workflow_init to access input early The `workflow_init` class method above `initialize` gives it access to [Workflow input](/handling-messages#workflow-initializers). When you use the `workflow_init` on your constructor, you give the constructor the same Workflow parameters as your `execute` method. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/ruby/client/temporal-client#start-workflow). The Workflow input arguments are also passed to your `execute` method -- that always happens, whether or not you use the `workflow_init` class method above `initialize`. Here's an example. The constructor and `execute` must have the same parameters with the same types: ```ruby class WorkflowInitWorkflow < Temporalio::Workflow::Definition workflow_init def initialize(input) @name_with_title = "Sir #{input['name']}" end def execute(input) Temporalio::Workflow.wait_condition { @title_has_been_checked } "Hello, #{@name_with_title}" end workflow_update def check_title_validity # The handler is now guaranteed to see some workflow input since it was # processed by the constructor valid = Temporalio::Workflow.execute_activity( CheckTitleValidityActivity, @name_with_title, start_to_close_timeout: 100 ) @title_has_been_checked = true valid end end ``` ### Use locks to prevent concurrent handler execution Concurrent processes can interact in unpredictable ways. Incorrectly written [concurrent message-passing](/handling-messages#message-handler-concurrency) code may not work correctly when multiple handler instances run simultaneously. Here's an example of a pathological case: ```ruby class MyWorkflow < Temporalio::Workflow::Definition # ... workflow_signal def bad_handler data = Temporalio::Workflow.execute_activity( FetchDataActivity, start_to_close_timeout: 100 ) @x = data['x'] # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then # there may be times when the Workflow has @x from one Activity execution and @y # from another. Temporalio::Workflow.sleep(1) @y = data['y'] end end ``` Coordinating access with `Mutex`, a mutual exclusion lock, corrects this code. Locking makes sure that only one handler instance can execute a specific section of code at any given time: ```ruby class MyWorkflow < Temporalio::Workflow::Definition # ... workflow_signal def safe_handler @mutex ||= Mutex.new @mutex.synchronize do data = Temporalio::Workflow.execute_activity( FetchDataActivity, start_to_close_timeout: 100 ) @x = data['x'] # 🐛🐛 Bug!! If multiple instances of this handler are executing concurrently, then # there may be times when the Workflow has @x from one Activity execution and @y # from another. Temporalio::Workflow.sleep(1) @y = data['y'] end end end ``` For additional concurrency options, `wait_condition` can be used to do more advanced things such as using an integer attribute + `wait_condition` as a semaphore. ## Troubleshooting When sending a Signal, Update, or Query to a Workflow, your Client might encounter the following errors: - **The Client can't contact the server**: You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is an `UNAVAILABLE` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html) (after some retries). - **The Workflow does not exist**: You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is a `NOT_FOUND` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html). See [Exceptions in message handlers](/handling-messages#exceptions) for a non–Ruby-specific discussion of this topic. ### Signal issues When using Signal, the only exception that will result from your requests during its execution is `RPCError`. All handlers may experience additional exceptions during the initial (pre-Worker) part of a handler request lifecycle. For Queries and Updates, the Client waits for a response from the Worker. If an issue occurs during the handler Execution by the Worker, the Client may receive an exception. ### Update issues When working with Updates, you may encounter these errors: - **No Workflow Workers are polling the Task Queue**: Your request will be retried by the SDK Client indefinitely. Use a `Cancellation` in your [RPC options](https://ruby.temporal.io/Temporalio/Client/RPCOptions.html) to cancel the Update. This raises a [WorkflowUpdateRPCTimeoutOrCanceledError](https://ruby.temporal.io/Temporalio/Error/WorkflowUpdateRPCTimeoutOrCanceledError.html) exception. - **Update failed**: You'll receive a [`WorkflowUpdateFailedError`](https://ruby.temporal.io/Temporalio/Error/WorkflowUpdateFailedError.html) exception. There are two ways this can happen: - The Update was rejected by an Update validator defined in the Workflow alongside the Update handler. - The Update failed after having been accepted. Update failures are like [Workflow failures](/references/failures). Issues that cause a Workflow failure in the main method also cause Update failures in the Update handler. These might include: - A failed Child Workflow - A failed Activity (if the Activity retries have been set to a finite number) - The Workflow author raising `ApplicationError` - Any error listed in `workflow_failure_exception_types` on the Worker or [`workflow_failure_exception_type`](https://ruby.temporal.io/Temporalio/Workflow/Definition.html#workflow_failure_exception_type-class_method) on the Workflow (empty by default) - **The handler caused the Workflow Task to fail**: A [Workflow Task Failure](/references/failures) causes the server to retry Workflow Tasks indefinitely. What happens to your Update request depends on its stage: - If the request hasn't been accepted by the server, you receive a `FAILED_PRECONDITION` [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception. - If the request has been accepted, it is durable. Once the Workflow is healthy again after a code deploy, use an [`WorkflowUpdateHandle`](https://ruby.temporal.io/Temporalio/Client/WorkflowUpdateHandle.html) to fetch the Update result. - **The Workflow finished while the Update handler execution was in progress**: You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) "workflow execution already completed". This will happen if the Workflow finished while the Update handler execution was in progress, for example because - The Workflow was canceled or failed. - The Workflow completed normally or continued-as-new and the Workflow author did not [wait for handlers to be finished](/handling-messages#finishing-message-handlers). ### Query issues When working with Queries, you may encounter these errors: - **There is no Workflow Worker polling the Task Queue**: You'll receive a [`Temporalio::Error::RPCError`](https://ruby.temporal.io/Temporalio/Error/RPCError.html) exception whose `code` is a `FAILED_PRECONDITION` constant defined in [`Code`](https://ruby.temporal.io/Temporalio/Error/RPCError/Code.html). - **Query failed**: You'll receive a [`WorkflowQueryFailedError`](https://ruby.temporal.io/Temporalio/Error/WorkflowQueryFailedError.html) exception if something goes wrong during a Query. Any exception in a Query handler will trigger this error. This differs from Signal and Update requests, where exceptions can lead to Workflow Task Failure instead. - **The handler caused the Workflow Task to fail.** This would happen, for example, if the Query handler blocks the thread for too long without yielding. ## Dynamic handlers Temporal supports Dynamic Queries, Signals, Updates, Workflows, and Activities. These are unnamed handlers that are invoked if no other statically defined handler with the given name exists. Dynamic Handlers provide flexibility to handle cases where the names of Queries, Signals, Updates, Workflows, or Activities, aren't known at run time. > **⚠️ Caution:** > > Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach. > Overusing them can lead to maintainability and debugging issues down the line. > > Instead, Signals, Queries, Workflows, or Activities should be defined statically whenever possible, with clear names that indicate their purpose. > Use static definitions as the primary way of structuring your Workflows. > > Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime. > They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic. > ### Dynamic Query A Dynamic Query in Temporal is a Query method that is invoked dynamically at runtime if no other Query with the same name is registered. A Query can be made dynamic by setting `dynamic` to `true` on the `workflow_query` class method. Only one Dynamic Query can be present on a Workflow. The Query Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`. The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types. ```ruby workflow_query dynamic: true, raw_args: true def dynamic_query(query_name, *args) first_param = Temporalio::Workflow.payload_converter.from_payload( args.first || raise 'Missing first parameter' ) "Got parameter #{first_param} for query #{query_name}" end ``` ### Dynamic Signal A Dynamic Signal in Temporal is a Signal that is invoked dynamically at runtime if no other Signal with the same input is registered. A Signal can be made dynamic by setting `dynamic` to `true` on the `workflow_signal` class method. Only one Dynamic Signal can be present on a Workflow. The Signal Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`. The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types. ```ruby workflow_signal dynamic: true, raw_args: true def dynamic_signal(signal_name, *args) first_param = Temporalio::Workflow.payload_converter.from_payload( args.first || raise 'Missing first parameter' ) @pending_things << "Got parameter #{first_param} for signal #{signal_name}" end ``` ### Dynamic Update A Dynamic Update in Temporal is an Update that is invoked dynamically at runtime if no other Update with the same input is registered. An Update can be made dynamic by setting `dynamic` to `true` on the `workflow_update` class method. Only one Dynamic Update can be present on a Workflow. The Update Handler parameters must accept a string name as the first parameter. Often users set `raw_args` to `true` and set the second parameter as `*args` which will be an array of `Temporalio::Converters::RawValue`. The [Temporalio::Workflow.payload_converter](https://ruby.temporal.io/Temporalio/Workflow.html#payload_converter-class_method) property is used to convert the raw value instances to proper types. ```ruby workflow_update dynamic: true, raw_args: true def dynamic_update(update_name, *args) first_param = Temporalio::Workflow.payload_converter.from_payload( args.first || raise 'Missing first parameter' ) @pending_things << "Got parameter #{first_param} for update #{update_name}" end ``` --- # Schedules - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/schedules > Manage and optimize Scheduled Workflows using the Temporal Ruby SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and more. This page shows how to do the following: - [Schedule a Workflow](#schedule-a-workflow) - [Create a Scheduled Workflow](#create-a-workflow) - [Backfill a Scheduled Workflow](#backfill-a-scheduled-workflow) - [Delete a Scheduled Workflow](#delete-a-scheduled-workflow) - [Describe a Scheduled Workflow](#describe-a-scheduled-workflow) - [List a Scheduled Workflow](#list-a-scheduled-workflow) - [Pause a Scheduled Workflow](#pause-a-scheduled-workflow) - [Trigger a Scheduled Workflow](#trigger-a-scheduled-workflow) - [Update a Scheduled Workflow](#update-a-scheduled-workflow) - [Use Start Delay](#start-delay) ## Schedule a Workflow Scheduling Workflows is a crucial aspect of automation. By scheduling a Workflow, you can automate repetitive tasks, reduce manual intervention, and ensure timely execution. Use the following actions to manage Scheduled Workflows. Schedule behavior is governed by the Schedule's [Overlap Policy](/schedule#overlap-policy). If a Workflow Execution started by a Schedule is [Paused](/cli/command-reference/workflow#pause), it remains open and counts as the running execution for overlap decisions. ### Create a Scheduled Workflow The create action enables you to create a new Schedule. When you create a new Schedule, a unique Schedule ID is generated, which you can use to reference the Schedule in other Schedule commands. To create a Scheduled Workflow Execution in Ruby, use the [create_schedule](https://ruby.temporal.io/Temporalio/Client.html#create_schedule-instance_method) method on the Client. Then pass the Schedule ID and the Schedule object to the method to create a Scheduled Workflow Execution. Set the Schedule's `action` member to an instance of `Temporalio::Client::Schedule::Action::StartWorkflow` to schedule a Workflow Execution. ```ruby handle = my_client.create_schedule( 'my_schedule_id', Temporalio::Client::Schedule.new( action: Temporalio::Client::Schedule::Action::StartWorkflow.new( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue' ), spec: Temporalio::Client::Schedule::Spec.new( intervals: [ Temporalio::Client::Schedule::Spec::Interval.new( every: 5 * 24 * 60 * 60.0, # 5 days ) ] ) ) ) ``` > **💡 Tip:** > Schedule Auto-Deletion > > Once a Schedule has completed creating all its Workflow Executions, the Temporal Service deletes it since it won’t fire again. > The Temporal Service doesn't guarantee when this removal will happen. > ### Backfill a Scheduled Workflow The backfill action executes Actions ahead of their specified time range. This command is useful when you need to execute a missed or delayed Action, or when you want to test the Workflow before its scheduled time. To backfill a Scheduled Workflow Execution in Ruby, use the [backfill](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#backfill-instance_method) method on the Schedule Handle. ```ruby handle = my_client.schedule_handle('my-schedule-id') now = Time.now(in: 'UTC') handle.backfill( Temporalio::Client::Schedule::Backfill.new( start_at: now - (4 * 60), end_at: now - (2 * 60), overlap: Temporalio::Client::Schedule::OverlapPolicy::ALLOW_ALL ) ) ``` ### Delete a Scheduled Workflow The delete action enables you to delete a Schedule. When you delete a Schedule, it does not affect any Workflows that were started by the Schedule. To delete a Scheduled Workflow Execution in Ruby, use the [delete](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#delete-instance_method) method on the Schedule Handle. ```ruby handle = my_client.schedule_handle('my-schedule-id') handle.delete ``` ### Describe a Scheduled Workflow The describe action shows the current Schedule configuration, including information about past, current, and future Workflow Runs. This command is helpful when you want to get a detailed view of the Schedule and its associated Workflow Runs. To describe a Scheduled Workflow Execution in Ruby, use the [describe](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#describe-instance_method) method on the Schedule Handle. ```ruby handle = my_client.schedule_handle('my-schedule-id') desc = handle.describe puts "Schedule info: #{desc.info}" ``` ### List a Scheduled Workflow The list action lists all the available Schedules. This command is useful when you want to view a list of all the Schedules and their respective Schedule IDs. To list all schedules, use the [list_schedules](https://ruby.temporal.io/Temporalio/Client.html#list_schedules-instance_method) asynchronous method on the Client. This returns an enumerator/enumerable. If a schedule is added or deleted, it may not be available in the list immediately. ```ruby my_client.list_schedules.each do |sched| puts "Schedule info: #{sched}" end ``` ### Pause a Scheduled Workflow The pause action enables you to pause and unpause a Schedule. When you pause a Schedule, all the future Workflow Runs associated with the Schedule are temporarily stopped. This command is useful when you want to temporarily halt a Workflow due to maintenance or any other reason. To pause a Scheduled Workflow Execution in Ruby, use the [pause](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#pause-instance_method) method on the Schedule Handle. You can pass a note to the `pause` method to provide a reason for pausing the schedule. ```ruby handle = my_client.schedule_handle('my-schedule-id') handle.pause(note: 'Pausing the schedule for now') ``` ### Trigger a Scheduled Workflow The trigger action triggers an immediate action with a given Schedule. By default, this action is subject to the Overlap Policy of the Schedule. This command is helpful when you want to execute a Workflow outside of its scheduled time. To trigger a Scheduled Workflow Execution in Ruby, use the [trigger](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#trigger-instance_method) method on the Schedule Handle. ```ruby handle = my_client.schedule_handle('my-schedule-id') handle.trigger ``` ### Update a Scheduled Workflow The update action enables you to update an existing Schedule. This command is useful when you need to modify the Schedule's configuration, such as changing the start time, end time, or interval. To update a Scheduled Workflow Execution in Ruby, use the [update](https://ruby.temporal.io/Temporalio/Client/ScheduleHandle.html#update-instance_method) method on the Schedule Handle. This method accepts a block which itself accepts an update input object and is expected to return an update with a new schedule to update, or `nil` to not update. ```ruby handle = my_client.schedule_handle('my-schedule-id') handle.update do |input| # Return a new schedule with the action updated Temporalio::Client::Schedule::Update.new( schedule: input.description.schedule.with( # Update the action action: Temporalio::Client::Schedule::Action::StartWorkflow.new( MyNewWorkflow, 'some-new-input', id: 'my-workflow-id', task_queue: 'my-task-queue' ) ) ) end ``` ## Use Start Delay Use the `start_delay` to schedule a Workflow Execution at a specific one-time future point rather than on a recurring schedule. Use the `start_delay` parameter on either the `start_workflow` or `execute_workflow` methods in the Client. ```ruby handle = my_client.start_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue', start_delay: 3 * 60 * 60 # 3 hours ) ``` --- # Workflow Timeouts - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/timeouts > Optimize Workflow Execution with Temporal Ruby SDK - Set Workflow Timeouts and Retry Policies efficiently. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. - **[Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)**: Limits how long the full Workflow Execution can run. - **[Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout)**: Limits the duration of an individual run of a Workflow Execution. - **[Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout)**: Limits the time allowed for a Worker to process a Workflow Task. Set these values as keyword parameter options when starting a Workflow. ```ruby result = my_client.execute_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue', execution_timeout: 5 * 60 ) ``` ### Workflow retries A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience. Use a [Retry Policy](/encyclopedia/retry-policies) to automatically retry Workflow Executions on failure. Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations. The `retry_policy` can be set when calling `start_workflow` or `execute_workflow`. ```ruby result = my_client.execute_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue', retry_policy: Temporalio::RetryPolicy.new(max_interval: 10) ) ``` --- # Timers - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/timers This page describes how to set a Durable Timer using the Temporal Ruby SDK. A [Durable Timer](/workflow-execution/timers-delays) is used to pause the execution of a Workflow for a specified duration. A Workflow can sleep for days or even months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the Durable Timer call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To add a Timer in a Workflow, use `Temporalio::Workflow.sleep`. _Technically_ `Kernel#sleep` works, but the workflow form allows one to set a summary to view in the UI. ```ruby # Sleep for 72 hours Temporalio::Workflow.sleep(72 * 60 * 60, summary: 'my timer') ``` There is also a `Temporalio::Workflow.timeout` method that accepts a block and works like standard Ruby `Timeout.timeout` if needing the ability to timeout a set of code. --- # Versioning - Ruby SDK Source: https://docs.temporal.io/develop/ruby/workflows/versioning > Use the Ruby SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. Since Workflow Executions in Temporal can run for long periods — sometimes months or even years — it's common to need to make changes to a Workflow Definition, even while a particular Workflow Execution is in progress. The Temporal Platform requires that Workflow code is [deterministic](/workflow-definition#deterministic-constraints). If you make a change to your Workflow code that would cause non-deterministic behavior on Replay, you'll need to use one of our Versioning methods to gracefully update your running Workflows. This only applies to Workflow orchestration logic. Non-deterministic work such as API calls, and database queries should be placed in Activities, which Temporal retries reliably. With Versioning, you can modify your Workflow Definition so that new executions use the updated code, while existing ones continue running the original version. There are two primary Versioning methods that you can use: - [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). The Worker Versioning feature allows you to tag your Workers and programmatically roll them out in versioned deployments, so that old Workers can run old code paths and new Workers can run new code paths. - [Versioning with Patching](#patching). This method works by adding branches to your code tied to specific revisions. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. ## Worker Versioning Temporal's [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) feature allows you to tag your Workers and programmatically roll them out in Deployment Versions, so that old Workers can run old code paths and new Workers can run new code paths. This way, you can pin your Workflows to specific revisions, avoiding the need for patching. ## Versioning with Patching ### Adding a patch A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow Executions, create a patch. Suppose you have an initial Workflow that runs `PrePatchActivity`: ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute result = Temporalio::Workflow.execute_activity( PrePatchActivity, start_to_close_timeout: 100 ) # ... end end ``` Now, you want to update your code to run `PostPatchActivity` instead. This represents your desired end state. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute result = Temporalio::Workflow.execute_activity( PostPatchActivity, start_to_close_timeout: 100 ) # ... end end ``` The problem is that you cannot deploy this new revision directly until you're certain there are no more running Workflows created using the `PrePatchActivity` code, otherwise you are likely to cause a nondeterminism error. Instead, you'll need to use the [`patched`](https://ruby.temporal.io/Temporalio/Workflow.html#patched-class_method) function to check which version of the code should be executed. Patching is a three-step process: 1. Patch in any new, updated code using the `patched()` function. Run the new patched code alongside old code. 2. Remove old code and use `deprecate_patch()` to mark a particular patch as deprecated. 3. Once there are no longer any open Workflow Executions of the previous version of the code, remove `deprecate_patch()`. Let's walk through this process in sequence. ### Patching in new code Using `patched` inserts a marker into the Event History. During Replay, if a Worker encounters a history with that marker, it will fail the Workflow task when the Workflow code doesn't produce the same patch marker (in this case `my-patch`). This ensures you can safely deploy new code paths alongside the original branch. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute if Temporalio::Workflow.patched('my-patch') result = Temporalio::Workflow.execute_activity( PostPatchActivity, start_to_close_timeout: 100 ) else result = Temporalio::Workflow.execute_activity( PrePatchActivity, start_to_close_timeout: 100 ) end # ... end end ``` ### Deprecating patches After ensuring that all Workflows started with `v1` code have left retention, you can [deprecate the patch](https://ruby.temporal.io/Temporalio/Workflow.html#deprecate_patch-class_method). Once your Workflows are no longer running the pre-patch code paths, you can deploy your code with `deprecate_patch()`. These Workers will be running the most up-to-date version of the Workflow code, which no longer requires the patch. The `deprecate_patch()` function works similarly to the `patched()` function by recording a marker in the Event history. This marker does not fail replay when Workflow code does not emit it. Deprecated patches serve as a bridge between the pre-patch code paths and the post-patch code paths, and are useful for avoiding errors resulting from patched code paths in your Event history. ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute Temporalio::Workflow.deprecate_patch('my-patch') result = Temporalio::Workflow.execute_activity( PostPatchActivity, start_to_close_timeout: 100 ) # ... end end ``` ### Removing a patch Once the pre-patch Workflows have left retention, you can then safely deploy Workers that no longer use either the `patched()` or `deprecate_patch()` calls: Patching allows you to make changes to currently running Workflows. It is a powerful method for introducing compatible changes without introducing non-determinism errors. ### Workflow cutovers To understand why Patching is useful, it's helpful to demonstrate cutting over an entire Workflow. Since incompatible changes only affect open Workflow Executions of the same type, you can avoid determinism errors by creating a whole new Workflow when making changes. To do this, you can copy the Workflow Definition function, giving it a different name, and register both names with your Workers. For example, you would duplicate `MyWorkflow` as `MyWorkflowV2`: ```ruby class MyWorkflow < Temporalio::Workflow::Definition def execute # ... end end class MyWorkflowV2 < Temporalio::Workflow::Definition def execute # ... end end ``` You would then need to update the Worker configuration, and any other identifier strings, to register both Workflow Types: ```ruby client = Temporalio::Client.connect('localhost:7233', 'default') worker = Temporalio::Worker.new( client:, task_queue: 'my-task-queue', workflows: [MyWorkflow, MyWorkflowV2] ) ``` The downside of this method is that it requires you to duplicate code and to update any commands used to start the Workflow. This can become impractical over time. This method also does not provide a way to version any still-running Workflows -- it is essentially just a cutover, unlike Patching. ### Testing a Workflow for replay safety To determine whether your Workflow your needs a patch, or that you've patched it successfully, you should incorporate [Replay Testing](/develop/ruby/best-practices/testing-suite#replay-test). --- # Run a development server Source: https://docs.temporal.io/develop/run-a-development-server > Shows how to run a development Temporal Service ## How to install the Temporal CLI and run a development server This page describes how to install the [Temporal CLI](/cli) and run a development Temporal Service. The local development Temporal Service comes packaged with the [Temporal Web UI](/web-ui). For information on deploying and running a self-hosted production Temporal Service, see the [Self-hosted guide](/self-hosted-guide), or sign up for [Temporal Cloud](/cloud) and let us run your production Temporal Service for you. Temporal CLI is a tool for interacting with a Temporal Service from the command line and it includes a distribution of the Temporal Server and Web UI. This local development Temporal Service runs as a single process with zero runtime dependencies and it supports persistence to disk and in-memory mode through SQLite. **Install the Temporal CLI** The Temporal CLI is available on macOS, Windows, and Linux. ### macOS **How to install the Temporal CLI on macOS** Choose one of the following install methods to install the Temporal CLI on macOS: **Install the Temporal CLI with Homebrew** ```bash brew install temporal ``` **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed. - Download for Darwin amd64: https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64 - Download for Darwin arm64: https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64 2. Extract the downloaded archive. 3. Add the `temporal` binary to your PATH. ### Linux **How to install the Temporal CLI on Linux** Choose one of the following install methods to install the Temporal CLI on Linux: **Install the Temporal CLI with Homebrew** ```bash brew install temporal ``` **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed. - Download for Linux amd64: https://temporal.download/cli/archive/latest?platform=linux&arch=amd64 - Download for Linux arm64: https://temporal.download/cli/archive/latest?platform=linux&arch=arm64 2. Extract the downloaded archive. 3. Add the `temporal` binary to your PATH. ### Windows **How to install the Temporal CLI on Windows** Follow these instructions to install the Temporal CLI on Windows: **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed and download the binary. - Download for Windows amd64: https://temporal.download/cli/archive/latest?platform=windows&arch=amd64 - Download for Windows arm64: https://temporal.download/cli/archive/latest?platform=windows&arch=arm64 2. Extract the downloaded archive. 3. Add the `temporal.exe` binary to your PATH. ### Start the Temporal Development Server Start the Temporal Development Server by using the `server start-dev` command. ```bash temporal server start-dev ``` This command automatically starts the Web UI, creates the default [Namespace](/namespaces), and uses an in-memory database. The Temporal Server should be available on `localhost:7233` and the Temporal Web UI should be accessible at [`http://localhost:8233`](http://localhost:8233/). The server's startup configuration can be customized using command line options. For a full list of options, run: ```bash temporal server start-dev --help ``` --- # Rust SDK developer guide Source: https://docs.temporal.io/develop/rust ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Install and get started You can find detailed installation instructions for the Rust SDK in the [Quickstart](/develop/rust/quickstart). Once your local Temporal Service is set up, continue building with the following resources: - [Develop a Workflow](/develop/rust/workflows/basics) - [Develop an Activity](/develop/rust/activities/basics) - [Start an Activity execution](/develop/rust/activities/execution) - [Run Worker processes](/develop/rust/workers/worker-process) ## [Workflows](/develop/rust/workflows) - [Workflow basics](/develop/rust/workflows/basics) - [Child Workflows](/develop/rust/workflows/child-workflows) - [Continue-As-New](/develop/rust/workflows/continue-as-new) - [Message passing](/develop/rust/workflows/message-passing) - [Cancellation](/develop/rust/workflows/cancellation) - [Timers](/develop/rust/workflows/timers) - [Timeouts](/develop/rust/workflows/timeouts) ## [Activities](/develop/rust/activities) - [Activity basics](/develop/rust/activities/basics) - [Activity execution](/develop/rust/activities/execution) - [Timeouts](/develop/rust/activities/timeouts) ## [Workers](/develop/rust/workers) - [Worker processes](/develop/rust/workers/worker-process) ## [Temporal Client](/develop/rust/client) - [Temporal Client](/develop/rust/client/temporal-client) ## [Temporal Nexus](/develop/rust/nexus) - [Feature guide](/develop/rust/nexus/feature-guide) ## Temporal Rust technical resources - [Rust SDK Quickstart - Setup Guide](/develop/rust/quickstart) - [Rust API Documentation](https://docs.rs/temporalio-sdk/latest/temporalio_sdk/) - [Rust SDK GitHub](https://github.com/temporalio/sdk-rust/tree/main/crates/sdk) ## Get connected with the Temporal Rust community - [Temporal Rust Community Slack](https://temporalio.slack.com/archives/C08G723SFNZ/p1773935454727179) --- # Activities - Rust SDK Source: https://docs.temporal.io/develop/rust/activities ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Activities - [Activity basics](/develop/rust/activities/basics) - [Activity execution](/develop/rust/activities/execution) - [Timeouts](/develop/rust/activities/timeouts) --- # Activity basics - Rust SDK Source: https://docs.temporal.io/develop/rust/activities/basics > This section explains how to implement Activities with the Rust SDK ## Develop a basic Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with the world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, you need to define the [Activity Definition](/activity-definition). The `#[activities]` macro marks an `impl` block as containing Activity definitions. Each method decorated with `#[activity]` becomes an Activity that can be invoked from a Workflow. Here's an example of an Activity: ```rust use temporalio_sdk::activities::{ActivityContext, ActivityError}; use temporalio_macros::activities; pub struct GreetingActivities; #[activities] impl GreetingActivities { #[activity] pub async fn greet(_ctx: ActivityContext, name: String) -> Result { Ok(format!("Hello, {}!", name)) } #[activity] pub async fn send_notification(_ctx: ActivityContext, message: String) -> Result<(), ActivityError> { println!("Sending notification: {}", message); Ok(()) } } ``` ### Define Activity parameters There is a limit of 6 parameters that an [Activity Definition](/activity-definition) may support. There is also a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. We recommend that you use a single struct as an argument that wraps all the application data passed to Activities. This way you can change what data is passed to the Activity without breaking the function signature. Each Activity method must: - Be `async` (return a future) - Take `ActivityContext` as the first parameter - Return `Result` where `T` is the return type - Be `pub` (public) The `ActivityContext` parameter provides access to Activity execution information and capabilities like heartbeating. If you don't need it, you can use `_ctx` as a parameter name. Activities can also take `Arc` and be registered using an instance. Here's an example using `Arc`: ```rust struct SleeperActivities { acts_started: Arc, acts_done: Arc, } #[activities] impl SleeperActivities { #[activity] pub async fn sleeper( self: Arc, ctx: ActivityContext, _: String, ) -> Result<(), ActivityError> { self.acts_started.add_permits(1); // just wait to be cancelled ctx.cancelled().await; self.acts_done.add_permits(1); Err(ActivityError::cancelled()) } } ``` Activity parameters should be serializable and deserializable using serde. Use `#[derive(Serialize, Deserialize)]` on your data types: ```rust use serde::{Serialize, Deserialize}; use temporalio_macros::activities; use temporalio_sdk::activities::{ActivityContext, ActivityError}; #[derive(Serialize, Deserialize)] pub struct GreetingInput { pub greeting: String, pub name: String, } pub struct GreetingActivities; #[activities] impl GreetingActivities { #[activity] pub async fn compose_greeting( _ctx: ActivityContext, input: GreetingInput, ) -> Result { Ok(format!("{} {}!", input.greeting, input.name)) } } ``` ### Define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction. Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). The return type of an Activity is `Result`. The `T` type must implement `Serialize`. Use `ApplicationFailure::new` for errors that should be retried, and `ApplicationFailure::non_retryable` for permanent failures: ```rust #[derive(serde::Serialize, serde::Deserialize, Debug, Clone, PartialEq, Eq, Hash)] pub struct ProcessedData { pub processed: String, } #[activities] impl MyActivities { #[activity] pub async fn process_data( _ctx: ActivityContext, input: String, ) -> Result { // If an error should be retried if !validate_input(&input) { return Err(ApplicationFailure::builder("Invalid input format") .next_retry_delay(Duration::from_secs(5)) .build() .into()); } // If an error should not be retried if input.len() > 1000000 { return Err( ApplicationFailure::non_retryable("Input too large").into(), ); } let result = ProcessedData { processed: input.to_uppercase(), }; Ok(result) } } ``` ### Customize your Activity Type Activities have a Type that refers to the Activity name. The Activity name is used to identify Activity Types in the Workflow Execution Event History, Visibility Queries, and Metrics. By default, the Activity name is the method name. You can customize it by providing a `name` parameter to the `#[activity]` macro: ```rust #[activities] impl GreetingActivities { #[activity(name = "compose_greeting")] pub async fn greet(_ctx: ActivityContext, name: String) -> Result { Ok(format!("Hello, {}!", name)) } #[activity(name = "send_email")] pub async fn send_notification(_ctx: ActivityContext, message: String) -> Result<(), ActivityError> { println!("Sending notification: {}", message); Ok(()) } } ``` --- # Activity execution - Rust SDK Source: https://docs.temporal.io/develop/rust/activities/execution > Shows how to perform Activity execution with the Rust SDK ## Start an Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). The call to spawn an Activity Execution generates the [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command. This results in a set of three [Activity Task](/tasks#activity-task) related Events in your Workflow Execution Event History: [ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and ActivityTaskClosed. A single instance of the Activity implementation may be used across multiple concurrent Activity invocations. Activity implementation code should be *idempotent*. Values passed to Activities as input parameters or returned as results are recorded in the Workflow Execution history. This history is replayed to Workflow Workers during recovery. Large payloads can negatively impact Workflow performance. Be mindful of the size of data passed to and from Activities. Otherwise, there are no strict limitations on Activity implementations. To spawn an Activity Execution, use the Workflow context’s Activity execution APIs within your Workflow code. In Rust, Activities are typically executed using `ctx.execute_activity(...)`, which returns a `Future` that can be awaited. ```rust #[workflow_methods] impl GreetingWorkflow { #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); // Execute an activity let greeting = ctx .execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ) .await?; println!("{}", greeting); Ok(greeting) } } ``` ### Set the required Activity Timeouts Activity Execution semantics rely on several timeout parameters. You need to set at least one of these: * [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) * [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout) These are configured as part of the Activity options when scheduling the Activity. Available timeouts include: - `start_to_close_timeout` - `schedule_to_close_timeout` - `with_start_to_close_timeout` - `with_schedule_to_close_timeout` ```rust #[workflow_methods] impl GreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); // Execute an activity let greeting = ctx .execute_activity( MyActivities::greet, name, ActivityOptions::schedule_to_close_timeout(Duration::from_secs(30)), ) .await?; println!("{}", greeting); Ok(greeting) } } ``` ### Get the results of an Activity Execution Spawning an [Activity Execution](/activity-execution) generates a [ScheduleActivityTask](/references/commands#scheduleactivitytask) Command and returns a `Future` to the Workflow. Workflows can either: * `await` the result immediately (blocking progress), or * store the `Future` and await it later to allow concurrent execution. In Rust, calling `.await` on the Activity invocation returns the result. If you need more control (for example, parallel execution), you can create multiple Activity futures and await them selectively. You must provide either `schedule_to_close_timeout` or `start_to_close_timeout`. ```rust #[workflow_methods] impl GreetingWorkflow { #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); // Execute an activity let greeting = ctx .execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ) .await?; println!("{}", greeting); Ok(greeting) } } ``` For concurrent execution: ```rust use temporalio_sdk::workflows::join; #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); // Execute an activity let greeting = ctx.execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ); let language = ctx.execute_activity( MyActivities::call_greeting_service, ActivityLanguages::English, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ); // Run in parallel let (greeting_res, language_res) = join!(greeting, language); Ok(format!("{} ({})", greeting_res?, language_res?)) } ``` Use direct `.await` in most cases. More advanced patterns, like parallel execution or cancellation, can be built using Rust’s async primitives. --- # Activity Timeouts - Rust SDK Source: https://docs.temporal.io/develop/rust/activities/timeouts > Optimize Workflow Execution with the Temporal Rust SDK by configuring Activity Timeouts, Retry Policies, and Heartbeats. ## Set Activity timeouts Each Activity timeout controls the maximum duration of a different aspect of an Activity Execution. The following timeouts are available in Activity options: - [Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout): the maximum amount of time allowed for the overall [Activity Execution](/activity-execution). - [Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout): the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution). - [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout): the maximum amount of time allowed from when an [Activity Task](/tasks#activity-task) is scheduled to when a [Worker](/workers#worker) starts that Activity Task. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close Timeout or the Schedule-To-Close Timeout set. Temporal strongly recommends setting a Start-To-Close Timeout because the service relies on it to detect lost Activity Tasks and trigger retries when appropriate. In Rust, these values are configured as part of the Activity options when scheduling an Activity from a Workflow. The Rust SDK is currently pre-release and its API is still evolving, so exact method names may change over time. Available timeout fields include: - `schedule_to_close_timeout` - `schedule_to_start_timeout` - `start_to_close_timeout` ```rust let greeting = ctx.execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ).await?; ``` ### Set an Activity Retry Policy A Retry Policy works together with timeouts to provide fine-grained control over Activity failure handling. Activities automatically use a default [Retry Policy](/encyclopedia/retry-policies) unless you provide a custom one. In Rust, configure the Retry Policy as part of the Activity options when scheduling the Activity from Workflow code. Because the Rust SDK API is still evolving, treat the following as representative of the current style rather than a guaranteed stable surface. ```rust let language = ctx.execute_activity( MyActivities::call_greeting_service, ActivityLanguages::English, ActivityOptions::with_start_to_close_timeout(Duration::from_secs(30)) .retry_policy( RetryPolicy::builder() .initial_interval(Duration::from_secs(10)) .backoff_coefficient(2.0) .maximum_interval(Duration::from_secs(100)) .maximum_attempts(5) .non_retryable_error_types(["NonRetryableError"]) .build(), ) .build(), ).await?; ``` ### Override the retry interval with `explicit_delay` To override the next retry interval set by the current policy, return a failure from an Activity with a custom next retry delay. That value replaces the interval the Retry Policy would otherwise use for the next retry attempt. This is useful when retry timing depends on runtime state such as the current attempt number. For example, you can increase the delay linearly with each attempt instead of using the exponential backoff defined by a backoff coefficient: ```rust use temporalio_macros::activities; use temporalio_sdk::{ ApplicationFailure, activities::{ActivityContext, ActivityError}, }; use std::sync::atomic::AtomicUsize; struct TestGreetActivities { counter: AtomicUsize, } #[activities] impl TestGreetActivities { #[activity] pub async fn greet(_ctx: ActivityContext, name: String) -> Result { if name == "ziggy" { return Err(ApplicationFailure::builder("Ziggy is not a valid name") // next retry will be after 5 seconds .next_retry_delay(std::time::Duration::from_secs(5)) .build() .into()); } Ok(format!("Hello, {}!", name)) } } ``` ## Heartbeat an Activity An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a signal from the [Worker Process](/workers#worker-process) executing the Activity to the [Temporal Service](/temporal-service). Each heartbeat tells the service that the [Activity Execution](/activity-execution) is still making progress and that the Worker has not crashed. If the service does not receive a heartbeat within the configured [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout), the Activity can time out and be retried according to its Retry Policy. Heartbeats may be throttled by the Worker, so not every heartbeat call is necessarily sent immediately to the Temporal Service. Activity cancellation is also delivered through heartbeat processing, which means Activities that don't heartbeat cannot receive cancellation promptly. ([Temporal Docs][3]) Heartbeats can include `details` that describe current progress. If the Activity fails and is retried, the retried attempt can retrieve the details from the most recently recorded heartbeat. The Rust SDK exposes activity context support for heartbeat details. To heartbeat an Activity in Rust, call the heartbeat API from inside the Activity with `record_heartbeat`: ```rust pub async fn greet(ctx: ActivityContext, name: String) -> Result { ctx.record_heartbeat("greet activity started".to_string()) .await?; if name == "ziggy" { return Err(ApplicationFailure::new("Ziggy is not a valid name").into()); } Ok(format!("Hello, {}!", name)) } ``` ### Set a Heartbeat Timeout A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works together with Activity heartbeats and sets the maximum time allowed between heartbeats. Configure it as part of the Activity options when scheduling the Activity. ```rust let language = ctx.execute_activity( MyActivities::call_greeting_service, ActivityLanguages::English, ActivityOptions::with_start_to_close_timeout(Duration::from_secs(30)) .heartbeat_timeout(Duration::from_secs(5)) .build(), ); ``` --- # Client - Rust SDK Source: https://docs.temporal.io/develop/rust/client ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Temporal Client - [Temporal Client](/develop/rust/client/temporal-client) --- # Temporal Client - Rust SDK Source: https://docs.temporal.io/develop/rust/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) lets your application communicate with the Temporal Service. Use it to start Workflow Executions, send Signals, run Queries, fetch Workflow results, and more. This page shows how to do the following using the Rust SDK and Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Start a Workflow Execution](#start-workflow-execution) - [Get Workflow results](#get-workflow-results) A Temporal Client can't be created and used inside Workflow code. However, using a Temporal Client inside an Activity is acceptable when you need to communicate with the Temporal Service. ## Connect to development Temporal Service In Rust, create a client by establishing a `Connection` and then constructing a `Client`. You can provide connection options directly in code or load them from environment variables. When you are running Temporal locally, the minimal setup is typically a local server address and the `default` Namespace. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file supports multiple profiles, each with its own connection options. If you don't specify a configuration file path, the SDK looks in the default OS-specific location. Environment variables take precedence over values from the configuration file. For example, the following TOML file defines two profiles: ```toml title="temporal.toml" # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` Load the configuration and connect with the `prod` profile as follows: ```rust use temporalio_client::{ Client, ClientOptions, Connection, envconfig::LoadClientConfigProfileOptions, }; #[tokio::main] async fn main() -> Result<(), Box> { let (conn_opts, client_opts) = ClientOptions::load_from_config( LoadClientConfigProfileOptions::builder() .config_file_profile("prod".to_owned()) .build(), )?; let connection = Connection::connect(conn_opts).await?; let client = Client::new(connection, client_opts)?; ... Ok(()) } ``` **Environment Variables** You can also configure the Temporal Client with environment variables using `envconfig`. This is useful for local development, CI, and production deployments. ```rust use temporalio_client::{ Client, ClientOptions, Connection, }; use temporalio_sdk::{Runtime, Worker, WorkerOptions}; #[tokio::main] async fn main() -> Result<(), Box> { let runtime = Runtime::new_assume_tokio(Default::default())?; let (conn_opts, client_opts) = ClientOptions::load_from_config(Default::default())?; let connection = Connection::connect(conn_opts).await?; let client = Client::new(connection, client_opts)?; let worker_options = WorkerOptions::new("hello-world") .register_workflow::()? .register_activities(GreetingActivities) .build(); let mut worker = Worker::new(&runtime, client, worker_options)?; println!("Worker started on task queue: hello-world"); worker.run().await?; Ok(()) } ``` **Code** You can also specify connection options directly in code. This is convenient for local development and testing. ```rust #[tokio::main] async fn main() -> Result<(), Box> { let (conn_opts, client_opts) = ClientOptions::load_from_config(Default::default())?; let connection = Connection::connect(conn_opts).await?; let client = Client::new(connection, client_opts)?; let wf_handle = client .start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build(), ) .await?; } ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an API key or mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your authentication credentials: - For API key authentication, provide the API key. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your _Namespace_ and _Account ID_ combination in the format `.` - The recommended gRPC endpoint for your Namespace, such as `..tmprl.cloud:7233` For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can define a Temporal Cloud profile in `temporal.toml`: ```toml [profile.api] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS instead of an API key: ```toml [profile.mtls] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" [profile.mtls.tls] client_cert_path = "/path/to/client.pem" client_key_path = "/path/to/client.key" ``` Then load the profile and connect: ```rust use temporalio_client::{ Client, ClientOptions, Connection, envconfig::LoadClientConfigProfileOptions, }; #[tokio::main] async fn main() -> Result<(), Box> { let (conn_opts, client_opts) = ClientOptions::load_from_config( LoadClientConfigProfileOptions::builder() .config_file_profile("api".to_owned()) .build(), )?; // Client setup let connection = Connection::connect(conn_opts).await?; let client = Client::new(connection, client_opts)?; println!("Connected to Temporal Cloud!"); Ok(()) } ``` **Environment Variables** The following environment variables are commonly used to connect to Temporal Cloud: * `TEMPORAL_NAMESPACE` * `TEMPORAL_ADDRESS` * `TEMPORAL_API_KEY` * `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH` * `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH` After setting the environment variables, load the configuration and connect: ```rust use temporalio_client::{ Client, ClientOptions, Connection, envconfig::LoadClientConfigProfileOptions, }; use temporalio_sdk::{Runtime, Worker, WorkerOptions}; #[tokio::main] async fn main() -> Result<(), Box> { let runtime = Runtime::new_assume_tokio(Default::default())?; let (conn_opts, client_opts) = ClientOptions::load_from_config(LoadClientConfigProfileOptions::default())?; let connection = Connection::connect(conn_opts).await?; let client = Client::new(connection, client_opts)?; let worker_options = WorkerOptions::new("hello-world") .register_workflow::()? .register_activities(GreetingActivities) .build(); let mut worker = Worker::new(&runtime, client, worker_options)?; println!("Worker started on task queue: hello-world"); worker.run().await?; Ok(()) } ``` **Code** You can also specify connection options directly in code for Temporal Cloud. ```rust use std::str::FromStr; use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, Url}; #[tokio::main] async fn main() -> Result<(), Box> { let connection_options = ConnectionOptions::new(Url::from_str( "https://your-namespace.a1b2c.tmprl.cloud:7233", )?) .api_key("your-api-key") .build(); let connection = Connection::connect(connection_options).await?; let _client = Client::new( connection, ClientOptions::new("your-namespace").build(), )?; println!("Connected to Temporal Cloud!"); Ok(()) } ``` With the default features, an mTLS client certificate is read when `Connection::connect` creates the connection. The `dynamic-tls` feature provides `TlsOptions::client_cert_resolver` for rotating client certificates without restarting the Worker; see [Update certificates using Temporal Cloud UI/tcld](/cloud/certificates#manage-certificates) for the zero-downtime staging sequence. ## Start a Workflow Execution To start a Workflow Execution, supply: - the Workflow Type - the Workflow input - a [Task Queue](/task-queue) that a Worker is polling - a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) Starting a Workflow Execution creates the first [WorkflowExecutionStarted](/references/events#workflowexecutionstarted) Event in the Event History, followed by the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. In Rust, use `start_workflow()` to start a Workflow and return a handle. ```rust let handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "my-task-queue", "greetings-workflow-10", ).build() ).await?; ``` ### Set a Workflow's Task Queue In most cases, the Task Queue is a required Workflow option. For a Workflow to make progress, at least one Worker must be polling the same Task Queue. In Rust, set the Task Queue in `WorkflowStartOptions`: ```rust let handle = client .start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "your-task-queue", "your-workflow-id" ).build(), ).await?; ``` ### Set a Workflow Id You must set a [Workflow Id](/workflow-execution/workflowid-runid#workflow-id). A Workflow Id should usually map to a business process or business entity identifier, such as an order ID or customer ID. In Rust, set the Workflow Id in `WorkflowStartOptions`: ```rust let handle = client .start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "your-task-queue", "your-workflow-id" ).build(), ).await?; ``` ## Get the results of a Workflow Execution If starting a Workflow succeeds, you get a Workflow handle. You can use that handle to wait for the result, describe the Workflow, or interact with it through Signals, Queries, and Updates. To get the result of a newly started Workflow: ```rust let handle = client .start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "your-task-queue", "your-workflow-id" ).build(), ).await?; let result = handle .get_result(WorkflowGetResultOptions::default()) .await?; println!("Result: {:?}", result); ``` --- # Nexus - Rust SDK Source: https://docs.temporal.io/develop/rust/nexus ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Temporal Nexus - [Feature guide](/develop/rust/nexus/feature-guide) --- # Nexus feature guide - Rust SDK Source: https://docs.temporal.io/develop/rust/nexus/feature-guide > Use Temporal Nexus within the Rust SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. [Nexus](/nexus) is a tool for coordinating asynchronous operations between Temporal and external systems. Service handlers allow Workflows to receive inbound requests through Nexus. ## Call a Nexus Operation from a Workflow You can start a Nexus operation from a Workflow using `ctx.start_nexus_operation()`: ```rust use std::time::Duration; use temporalio_common::protos::coresdk::AsJsonPayloadExt; use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{ ApplicationFailure, NexusOperationOptions, WorkflowContext, WorkflowContextView, WorkflowResult, }; #[workflow] pub struct GreetingWorkflow { pub name: String, } #[workflow_methods] impl GreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); let nexus_started = ctx .start_nexus_operation( NexusOperationOptions::builder() .endpoint("my-endpoint") .service("my-service") .operation("my-operation") .input(name.as_json_payload().map_err(ApplicationFailure::new)?) .start_to_close_timeout(Duration::from_secs(10)) .build(), ) .await; let nexus_started = match nexus_started { Ok(started) => started, Err(failure) => return Ok(format!("Nexus start failed: {failure:?}")), }; let nexus_result = nexus_started.result().await; println!("Nexus result: {:?}", nexus_result); Ok(format!("nexus result: {:?}", nexus_result)) } } ``` ### Nexus Operation arguments - `endpoint` - The Nexus endpoint name - `service` - The service name - `operation` - The operation name - `input` - The input payload (optional) - `start_to_close_timeout` - How long the operation can run - `schedule_to_close_timeout` - How long the caller waits for the operation to complete --- # Quickstart - Rust SDK Source: https://docs.temporal.io/develop/rust/quickstart > Configure your local development environment to get started developing with Temporal and the Rust SDK # Quickstart Configure your local development environment to get started developing with Temporal using the Rust SDK. ## Install Rust Make sure you have Rust installed on your system. You can download and install Rust from [rustup.rs](https://rustup.rs/). After installation, verify it's working by checking the version. You'll also need Cargo, which is Rust's package manager and comes bundled with Rust. ```bash rustc --version ``` ## Create a Project Now that you have Rust and Cargo installed, create a new Rust project to manage your dependencies. ```bash mkdir temporal-rust-project ``` ```bash cd temporal-rust-project ``` ```bash cargo init --name temporal-hello-world ``` ## Add Temporal Rust SDK Dependencies Now update your project's `Cargo.toml` file to match the example and run `cargo build`. The core dependencies you'll need are: - `temporalio-sdk` - The Rust SDK for Temporal - `tokio` - Async runtime required by the SDK - `serde` - For serialization/deserialization Next, you'll configure a local Temporal Service for development. ```toml [package] name = "temporal-hello-world" version = "0.1.0" edition = "2024" [dependencies] futures = "0.3" serde = { version = "1", features = ["derive"] } temporalio-client = "0.7.0" temporalio-common = "0.7.0" temporalio-macros = "0.7.0" temporalio-sdk = "0.7.0" temporalio-workflow = "0.7.0" tokio = { version = "1", features = ["full"] } ``` If you run into issues installing the dependencies, try: ```bash brew install protobuf ``` ## Install Temporal CLI and start the development server The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](/cli). Choose your operating system to install Temporal CLI: **macOS** ```bash brew install temporal ``` **Windows** Download the Temporal CLI archive for your architecture: - [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) - [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. **Linux** Download the Temporal CLI for your architecture: - [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) - [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH: ```bash sudo mv temporal /usr/local/bin ``` ## Start the development server Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at http://localhost:8233. Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. After installing, open a new Terminal. Keep this running in the background: ```bash temporal server start-dev ``` #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: ```bash temporal server start-dev --ui-port 8080 ``` The Temporal Web UI will now be available at http://localhost:8080. ## Run Hello World: Test Your Installation Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: - The Temporal Rust SDK is properly installed - Your local Temporal Service is running - You can successfully create and execute Workflows and Activities - The communication between components is functioning correctly ### 1. Define your Activity Create a `/src/activities.rs` file with the Activity definition: ```rust use temporalio_macros::activities; use temporalio_sdk::activities::{ActivityContext, ActivityError}; pub struct MyActivities; #[activities] impl MyActivities { #[activity] pub async fn greet(_ctx: ActivityContext, name: String) -> Result { Ok(format!("Hello, {}!", name)) } } ``` ### 2. Define your Workflow Create a `/src/workflows.rs` file with the Workflow definition: ```rust use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{ActivityOptions, WorkflowContext, WorkflowContextView, WorkflowResult}; use std::time::Duration; use crate::activities::MyActivities; #[workflow] pub struct GreetingWorkflow { name: String, } #[workflow_methods] impl GreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); // Execute an activity let greeting = ctx .execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ) .await?; println!("{}", greeting); Ok(greeting) } } ``` ### 3. Create and Run a Worker Update your `/src/main.rs` file with the following: ```rust use temporalio_client::{ Client, ClientOptions, Connection, envconfig::LoadClientConfigProfileOptions, }; use temporalio_sdk::{Runtime, Worker, WorkerOptions}; mod workflows; mod activities; use crate::workflows::GreetingWorkflow; use crate::activities::MyActivities; #[tokio::main] async fn main() -> Result<(), Box> { let runtime = Runtime::new_assume_tokio(Default::default())?; // Set up client connection options, loading from config if available let (connection_options, client_options) = ClientOptions::load_from_config( LoadClientConfigProfileOptions::default(), )?; let connection = Connection::connect(connection_options).await?; let client = Client::new(connection, client_options)?; let worker_options = WorkerOptions::new("my-task-queue") .register_activities(MyActivities) .register_workflow::()? .build(); let mut worker = Worker::new(&runtime, client, worker_options)?; worker.run().await?; Ok(()) } ``` Open a new terminal and run the Worker with: ```bash cargo run ``` ### 4. Start a Workflow You can now start a Workflow execution using the client. In a new terminal, run the Workflow with: ```bash temporal workflow start \ --type GreetingWorkflow \ --task-queue my-task-queue \ --input '"Ziggy"' ``` ## Next steps Now that you have the basics working, explore the following resources to build more sophisticated applications: - [Develop a Workflow](/develop/rust/workflows/basics) - Learn how to write complex workflow logic - [Develop an Activity](/develop/rust/activities/basics) - Understand activity patterns and best practices - [Worker Processes](/develop/rust/workers/worker-process) - Configure and scale workers - [Using the Temporal Client](/develop/rust/client/temporal-client) - Start workflows and interact with the Temporal Service - [Take a Temporal 101 course](https://learn.temporal.io/courses/): Learn Temporal concepts and build your first application with a guided course --- # Workers - Rust SDK Source: https://docs.temporal.io/develop/rust/workers ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Workers - [Worker processes](/develop/rust/workers/worker-process) --- # Serverless Workers - Rust SDK Source: https://docs.temporal.io/develop/rust/workers/serverless-workers > Write Temporal Workers that run on serverless compute using the Rust SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. Serverless Workers run on compute that Temporal starts and stops for you, rather than on long-lived processes you operate. For a general overview of how Serverless Workers work, see [Serverless Workers](/serverless-workers). For the end-to-end deployment guide, see [Deploy a Serverless Worker](/production-deployment/worker-deployments/serverless-workers). ## Supported providers - [**GCP Cloud Run**](/develop/rust/workers/serverless-workers/cloud-run) - Run a standard Worker on a Cloud Run worker pool. Covers the versioned Worker setup, connection configuration, and handling scale-in. --- # Serverless Workers on GCP Cloud Run - Rust SDK Source: https://docs.temporal.io/develop/rust/workers/serverless-workers/cloud-run > Run a Temporal Worker on a GCP Cloud Run worker pool using the Rust SDK. > **Pre-release** > Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. > Create a [support ticket](/cloud/support#support-ticket) or contact your account team for access, and > [sign up for updates](https://temporal.io/pages/serverless-workers-updates) to hear when Cloud Run reaches Public Preview. The Rust SDK is in [Public Preview](/evaluate/development-production-features/release-stages#public-preview), and its API can change between releases. The code on this page is written against `temporalio-sdk` 0.7.0. On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Rust Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. A Cloud Run Worker needs no Cloud Run-specific crate. The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers. For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run). ## Create a versioned Worker Build the Worker as you would any long-running Rust Worker, then set `deployment_options` on `WorkerOptions` to declare the Worker Deployment Version and turn versioning on. The following Worker reads its connection settings and Task Queue from the environment, so the same image can run against any Namespace: ```rust use std::str::FromStr; use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, TlsOptions, Url}; use temporalio_common::worker::{ VersioningBehavior, WorkerDeploymentOptions, WorkerDeploymentVersion, }; use temporalio_sdk::{Runtime, Worker, WorkerOptions}; #[tokio::main] async fn main() -> Result<(), Box> { let address = std::env::var("TEMPORAL_ADDRESS")?; let namespace = std::env::var("TEMPORAL_NAMESPACE")?; let api_key = std::env::var("TEMPORAL_API_KEY")?; let runtime = Runtime::new_assume_tokio(Default::default())?; let connection_options = ConnectionOptions::new(Url::from_str(&format!("https://{address}"))?) .api_key(api_key) .tls_options(TlsOptions::default()) .build(); let connection = Connection::connect(connection_options).await?; let client = Client::new(connection, ClientOptions::new(namespace).build())?; let worker_options = WorkerOptions::new(std::env::var("TEMPORAL_TASK_QUEUE")?) .deployment_options( WorkerDeploymentOptions::new(WorkerDeploymentVersion { deployment_name: "my-app".to_owned(), build_id: "build-1".to_owned(), }) .use_worker_versioning(true) .default_versioning_behavior(VersioningBehavior::Pinned) .build(), ) .register_workflow::()? .register_activities(MyActivities) .build(); let mut worker = Worker::new(&runtime, client, worker_options)?; worker.run().await?; Ok(()) } ``` `deployment_name` and `build_id` together identify the Worker Deployment Version. Both values must match the version you create with `temporal worker deployment create-version` in the deployment guide, or the Worker polls under a version the WCI does not manage. The Rust SDK sets the versioning behavior on the Worker rather than per Workflow, so `default_versioning_behavior` covers every Workflow the Worker registers. Setting it to `VersioningBehavior::Unspecified` is an error at startup. See [versioning behaviors](/worker-versioning#versioning-behaviors) for what `Pinned` and `AutoUpgrade` mean. For general Worker setup and options that are not specific to Cloud Run, see [Run a Worker](/develop/rust/workers/worker-process). ## Configure the Temporal connection Read the Namespace, address, and Task Queue from environment variables you set on the Worker Pool, and mount the Temporal Cloud API key or TLS material from Secret Manager rather than passing it in plaintext. The Worker above reads `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE`, so the same image can run against any Namespace. Setting `api_key` alone does not enable TLS. `Connection::connect` applies TLS only when `tls_options` is set, so a Temporal Cloud connection needs `.tls_options(TlsOptions::default())` as well, or it fails with `Connecting to HTTPS without TLS enabled`. To rotate the key without restarting the Worker, call `set_api_key` on the connected Client. For the shared configuration format that other Temporal tools read, see [Environment configuration](/develop/environment-configuration). ## Package the Worker image Compile the Worker in one stage and copy the binary into a runtime image: ```dockerfile FROM rust:1.92-slim AS build RUN apt-get update \ && apt-get install -y --no-install-recommends pkg-config libssl-dev protobuf-compiler libprotobuf-dev \ && rm -rf /var/lib/apt/lists/* WORKDIR /src COPY Cargo.toml ./ COPY src ./src RUN cargo build --release FROM debian:bookworm-slim RUN apt-get update \ && apt-get install -y --no-install-recommends ca-certificates \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY --from=build /src/target/release/my-worker /app/worker CMD ["/app/worker"] ``` The build stage needs `libprotobuf-dev` as well as `protobuf-compiler`. The compiler package alone installs `protoc` without the well-known type definitions, and the build then fails with `google/protobuf/duration.proto: File not found`. The runtime stage needs `ca-certificates`. The Worker reads TLS roots from the operating system's certificate store, and `debian:bookworm-slim` ships without one. ## Keep Activities safe across scale-in The WCI decides when to remove an instance from Task Queue activity, not from what an individual instance is doing. An instance running a long Activity can be stopped mid-execution. Use [Activity Heartbeats](/develop/rust/activities/timeouts#activity-heartbeats) so a retry resumes from the last recorded progress instead of starting over. For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle). ## Add observability A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else. For how to configure metrics and telemetry, see the runtime options you pass to `Runtime::new_assume_tokio` and the [SDK metrics reference](/references/sdk-metrics). --- # Run a Worker - Rust SDK Source: https://docs.temporal.io/develop/rust/workers/worker-process > Create and run a Temporal Worker using the Rust SDK. The Rust SDK is in [Public Preview](/evaluate/development-production-features/release-stages#public-preview), and its API can change between releases. The code on this page is written against `temporalio-sdk` 0.7.0. ## Create and run a Worker Start with these dependencies. The `#[workflow]` and `#[activities]` macros expand to code that refers to the `temporalio-workflow` and `futures` crates, so both must be direct dependencies of your crate even though your own code never names them: ```toml [dependencies] temporalio-sdk = "0.7.0" temporalio-client = "0.7.0" temporalio-common = "0.7.0" temporalio-macros = "0.7.0" temporalio-workflow = "0.7.0" futures = "0.3" tokio = { version = "1", features = ["full"] } ``` A Worker needs a `Runtime` and a connected [Temporal Client](/develop/rust/client/temporal-client). Build `WorkerOptions` with the Task Queue to poll, register the Workflows and Activities the Worker can execute, then call `run()`: ```rust use std::str::FromStr; use temporalio_client::{Client, ClientOptions, Connection, ConnectionOptions, Url}; use temporalio_sdk::{Runtime, Worker, WorkerOptions}; #[tokio::main] async fn main() -> Result<(), Box> { let runtime = Runtime::new_assume_tokio(Default::default())?; let connection_options = ConnectionOptions::new(Url::from_str("http://localhost:7233")?).build(); let connection = Connection::connect(connection_options).await?; let client = Client::new(connection, ClientOptions::new("default").build())?; let worker_options = WorkerOptions::new("my-task-queue") .register_workflow::()? .register_activities(GreetingActivities) .build(); let mut worker = Worker::new(&runtime, client, worker_options)?; worker.run().await?; Ok(()) } ``` `run()` polls until the Worker shuts down. `ClientOptions::new()` takes the Namespace, and `Connection::connect()` takes the server address. ## Register Workflows and Activities All Workers polling the same Task Queue must register the same Workflow Types and Activity Types. A Task Queue does not route by type, so any Worker polling it can receive any Task on that queue. A Worker that receives a Task for a type it did not register fails that Task. Register each Workflow with `register_workflow::()` and each Activity implementer with `register_activities()`. `register_workflow` returns a `Result`, so it needs `?` or other error handling: ```rust let worker_options = WorkerOptions::new("my-task-queue") .register_workflow::()? .register_workflow::()? .register_activities(GreetingActivities) .build(); ``` `register_activities()` takes an instance, so Activities can share state such as a database client through the fields of that value. ## Connect to Temporal Cloud To run a Worker against Temporal Cloud, configure the Client connection with your Namespace address and authentication credentials. See [Connect to Temporal Cloud](/develop/rust/client/temporal-client#connect-to-temporal-cloud) for setup instructions. ## Configure Worker options `WorkerOptions` controls the Task Queue, cache size, poller behavior, and slot allocation, including `max_cached_workflows`, `workflow_task_poller_behavior`, and `tuner`. Set `client_identity_override` to give the Worker an identity that is more useful than the default of `{pid}@{hostname}`. The defaults work for most cases. To tune these values against real load, see [Worker performance](/develop/worker-performance) and the [Worker tuning reference](/develop/worker-tuning-reference). ## Run a versioned Worker Set a Worker Deployment Version and enable versioning in `deployment_options`, then set the default versioning behavior for the Workflows on the Worker. ```rust use temporalio_common::worker::{ VersioningBehavior, WorkerDeploymentOptions, WorkerDeploymentVersion, }; let worker_options = WorkerOptions::new("my-task-queue") .deployment_options( WorkerDeploymentOptions::new(WorkerDeploymentVersion { deployment_name: "my-app".to_owned(), build_id: "1.0".to_owned(), }) .use_worker_versioning(true) .default_versioning_behavior(VersioningBehavior::Pinned) .build(), ) .register_workflow::()? .register_activities(GreetingActivities) .build(); ``` The Rust SDK sets the versioning behavior on the Worker, not per Workflow, so `default_versioning_behavior` covers every Workflow the Worker registers. Setting it to `VersioningBehavior::Unspecified` is an error at startup. See [Worker Versioning](/worker-versioning) for the available versioning behaviors and how new versions roll out. ## Shut down a Worker `run()` borrows the Worker mutably, so call `shutdown_handle()` before starting the Worker. Calling the handle it returns initiates shutdown, which stops polling for new Tasks and lets in-flight Tasks finish. ```rust let mut worker = Worker::new(&runtime, client, worker_options)?; let shutdown = worker.shutdown_handle(); tokio::spawn(async move { tokio::signal::ctrl_c().await.expect("failed to listen for ctrl-c"); shutdown(); }); worker.run().await?; ``` See [Worker shutdown](/encyclopedia/workers/worker-shutdown) for what happens to in-flight Workflow Tasks and Activities. --- # Workflows - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows ![Rust SDK Banner](/img/assets/banner-rust-temporal.png) ## Workflows - [Workflow basics](/develop/rust/workflows/basics) - [Child Workflows](/develop/rust/workflows/child-workflows) - [Continue-As-New](/develop/rust/workflows/continue-as-new) - [Message passing](/develop/rust/workflows/message-passing) - [Cancellation](/develop/rust/workflows/cancellation) - [Timers](/develop/rust/workflows/timers) - [Timeouts](/develop/rust/workflows/timeouts) --- # Workflow basics - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/basics > This section explains how to implement Workflows with the Rust SDK ## How to develop a Workflow Workflows are the fundamental unit of a Temporal Application and it all starts with the development of a [Workflow Definition](/workflow-definition). In the Temporal Rust SDK programming model, a Workflow Definition is made of a Workflow struct and associated methods decorated with macros. A Workflow is defined by: 1. A struct that holds the Workflow state 2. A `#[run]` method that contains the main Workflow logic 3. An optional `#[init]` method that initializes the Workflow 4. Optional `#[signal]`, `#[query]`, and `#[update]` methods for external interaction ```rust use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{WorkflowResult, WorkflowContextView, WorkflowContext}; #[workflow] pub struct GreetingWorkflow { name: String, } #[workflow_methods] impl GreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); Ok(format!("Hello, {}!", name)) } } ``` The `#[workflow]` macro marks the struct as a Workflow. The `#[workflow_methods]` macro is applied to the `impl` block containing the Workflow methods. ### Workflow struct The Workflow struct holds the state of your Workflow Execution. This state is persisted and recovered during replays. All fields in a Workflow struct should be serializable. ### Workflow initialization The `#[init]` method is optional and is called when the Workflow first starts. It receives the initial Workflow input parameters and initializes the Workflow struct: ```rust #[init] fn new(ctx: &WorkflowContextView, name: String, age: u32) -> Self { Self { name, age, started_at: ctx.start_time(), } } ``` The `#[init]` method receives a `WorkflowContextView`, which provides read-only access to Workflow execution information. #### Use Workflow constructors Workflow constructors are useful if you have message handlers that need access to Workflow input: see [Initializing the Workflow first](/handling-messages#workflow-initializers). Normally, your Workflows constructor won't have any parameters. However, if you use the `#[init]` annotation on your constructor, you can give it the same [Workflow parameters](/develop/rust/workflows/basics#workflow-parameters) as your `#[run]`. The SDK will then ensure that your constructor receives the Workflow input arguments that the [Client sent](/develop/rust/client/temporal-client#start-workflow-execution). The Workflow input arguments are also passed to your `#[run]` method. That always happens, whether or not you use the `#[init]` annotation. Here's an example. Notice that the constructor and `get_greeting` must have the same parameters: ```rust #[workflow_methods] impl WorkflowRunSeesWorkflowInitWorkflow { #[init] fn new(_ctx: &WorkflowContextView, workflow_input: MyWorkflowInput) -> Self { Self { name_with_title: format!("Knight {}", workflow_input.name), title_has_been_checked: false, } } #[run] pub async fn get_greeting( ctx: &mut WorkflowContext, _workflow_input: MyWorkflowInput, ) -> WorkflowResult { ctx.wait_condition(|state| state.title_has_been_checked) .await?; let name_with_title = ctx.state(|state| state.name_with_title.clone()); Ok(format!("Hello, {}", name_with_title)) } } ``` ### Run method The `#[run]` method is required and contains the main Workflow logic. It: - Must be `async` - Must be `public` - Receives a mutable `WorkflowContext` - Returns `WorkflowResult` where T is the Workflow return type - Executes exactly once per Workflow execution ```rust #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { // Execute activities, timers, child workflows, etc. let result = ctx .execute_activity( MyActivities::greet, name, ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ) .await?; Ok(result) } ``` ## Define Workflow parameters Temporal Workflows may have any number of custom parameters. However, we strongly recommend that structs are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. All Workflow Definition parameters must be serializable. A method annotated with `#[init]` can have any number of parameters. We recommend passing a single struct that contains all the input fields: ```rust use serde::{Serialize, Deserialize}; #[derive(Serialize, Deserialize)] pub struct ProcessingInput { pub data: Vec, pub timeout_seconds: u32, } #[workflow] pub struct ProcessingWorkflow { data: Vec, timeout_seconds: u32, } #[workflow_methods] impl ProcessingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, input: ProcessingInput) -> Self { Self { data: input.data, timeout_seconds: input.timeout_seconds, } } #[run] pub async fn run(_ctx: &mut WorkflowContext) -> WorkflowResult { // Use the initialized state Ok("Processing complete".to_string()) } } ``` All Workflow input should be serializable by `serde`. ## Define Workflow return parameters Workflow return values must also be serializable. Returning results, returning errors, or throwing exceptions is fairly idiomatic in each language that is supported. However, Temporal APIs that must be used to get the result of a Workflow Execution will only ever receive one of either the result or the error. The return type of a Workflow is `WorkflowResult` where `T` implements `Serialize`. Success is represented by `Ok(value)` and failure by `Err(...)`: ```rust #[run] async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { // Can return a complex result type let result = ProcessingResult { status: "completed".to_string(), records_processed: 100, }; Ok(result) } ``` ## Customize your Workflow Type Workflows have a Type that is referred to as the Workflow name. By default, the Workflow type is the name of the Workflow struct. You can customize it by providing a `name` parameter to the `#[run]` macro: ```rust #[workflow] pub struct GreetingWorkflow { name: String, } #[workflow_methods] impl GreetingWorkflow { #[run(name = "my-custom-workflow")] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { Ok(ctx.state(|s| format!("Hello, {}!", s.name))) } } ``` The Workflow Type defaults to the struct name if not specified. For example, this Workflow would have the type `GreetingWorkflow`: ```rust #[workflow] pub struct GreetingWorkflow { // ... } ``` ## Workflow logic requirements Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints). For non-deterministic operations like API calls, and database queries, use [Activities](/develop/rust/activities/basics). Workflow code must be deterministic because the Temporal Server may replay your Workflow to reconstruct its state. This means: ### Don't use nondeterministic functions - No direct system time access - use `ctx.workflow_time()` instead of `SystemTime::now()` - No nondeterministic random number generation - use `ctx.random()` instead - No external I/O (network, filesystem, etc.) - perform these in Activities instead - No UUID generation via random means - use `ctx.uuid4()` instead - Do not use `tokio` or `futures` concurrency primitives directly in Workflow code. Many of them, like `tokio::select!`, `tokio::spawn`, `futures::select!`, introduce non-deterministic behavior that will break Workflow replay. Instead, use the deterministic wrappers provided in `temporalio_sdk::workflows`: - `select!` — deterministic select (polls in declaration order) - `join!` — deterministic join for a fixed number of futures - `join_all` — deterministic join for a dynamic collection of futures ### Use Workflow-safe primitives The Rust SDK provides: - `ctx.timer()` - Wait for a duration - `ctx.wait_condition(closure)` - Wait until a condition is true - `workflows::select!` - Deterministic select statement - `ctx.execute_activity()` - Execute Activities - `ctx.execute_local_activity()` - Execute local Activities - `ctx.start_child_workflow()` - Start Child Workflows - `ctx.cancelled()` - Wait for Workflow cancellation ```rust use std::time::Duration; #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { // Good - deterministic timer ctx.timer( TimerOptions::builder(Duration::from_secs(60)) .summary("important timer".to_owned()) .build(), ) .await; // Good - deterministic wait for condition ctx.wait_condition(|s| s.data.len() >= 3).await?; // Bad - nondeterministic sleep // tokio::time::sleep(Duration::from_secs(10)).await; // Bad - nondeterministic time // SystemTime::now() Ok("Done".to_string()) } ``` ## Access Workflow State Use `ctx.state()` for read-only access and `ctx.state_mut()` for mutable access to your Workflow state: ```rust #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { // Read-only access let name = ctx.state(|s| s.name.clone()); // Mutable access (for signal handlers or update handlers) // Available in sync methods Ok(name) } ``` In synchronous [`Signal`](/develop/rust/workflows/message-passing#signals) and [`Update`](/develop/rust/workflows/message-passing#updates) handlers, you can mutate state directly via `&mut self`. ## Workflow return types The `#[run]` method must return `WorkflowResult`. This is a type alias for `Result`. For application failures, construct an `ApplicationFailure` and convert it into `WorkflowTermination`: ```rust use temporalio_sdk::ApplicationFailure; #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { if some_validation_fails { return Err(ApplicationFailure::new("validation_failed: Input is invalid").into()); } Ok("Success".to_string()) } ``` Workflow errors will cause the Workflow Execution to fail and the error details will be available to clients. --- # Cancellation - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/cancellation You can interrupt a Workflow Execution in one of the following ways: - [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution. - [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. ## Cancel a Workflow Execution Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a `SIGTERM` to a process. - The system records a `WorkflowExecutionCancelRequested` event in the Event History. - A Workflow Task gets scheduled to process the cancelation. - The Workflow code can handle the cancelation and execute any cleanup logic. - The system doesn't forcefully stop the Workflow. To cancel a Workflow Execution in Rust, use the `cancel` method on the Workflow handler: ```rust let handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build() ).await?; handle.cancel(WorkflowCancelOptions::builder().reason("No longer needed").build()).await?; ``` ### Cancel an Activity from a Workflow Canceling an Activity from within a Workflow requires that the Activity Execution sends Heartbeats and sets a Heartbeat Timeout. If the Heartbeat is not invoked, the Activity cannot receive a cancellation request. When any non-immediate Activity is executed, the Activity Execution should send Heartbeats and set a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) to ensure that the server knows it is still working. When an Activity is canceled, an error is returned in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done when handling the cancellation error. However, for the Activity to appear canceled the error must be propagated. Example of a cancellable Activity in Rust: ```rust #![allow(unreachable_pub)] use temporalio_macros::activities; use temporalio_sdk::activities::{ActivityContext, ActivityError}; pub struct CancellationActivities; #[activities] impl CancellationActivities { #[activity] pub async fn long_running_cancellable_activity( ctx: ActivityContext, _input: (), ) -> Result { loop { if ctx.is_cancelled() { return Err(ActivityError::cancelled()); } ctx.record_heartbeat(()).await?; tokio::time::sleep(std::time::Duration::from_millis(200)).await; } } #[activity] pub async fn cleanup(_ctx: ActivityContext, _input: ()) -> Result { Ok("cleanup done".to_string()) } } ``` Canceling the Activity from a Workflow: ```rust fn activity_opts() -> ActivityOptions { ActivityOptions::with_start_to_close_timeout(Duration::from_secs(300)) .heartbeat_timeout(Duration::from_secs(5)) .build() } #[workflow] #[derive(Default)] pub struct CancellationWorkflow; #[workflow_methods] impl CancellationWorkflow { #[run] pub async fn run(ctx: &mut WorkflowContext, _input: ()) -> WorkflowResult { let result = ctx .execute_activity( CancellationActivities::long_running_cancellable_activity, (), activity_opts(), ) .await; match result { Ok(value) => Ok(value), Err(ActivityExecutionError::Cancelled(_)) => { let reason = ctx.cancellation_token().reason().unwrap_or_default(); let cleanup_result = ctx .execute_activity( CancellationActivities::cleanup, (), ActivityOptions::with_start_to_close_timeout(Duration::from_secs(10)) .cancellation_token(WorkflowCancellationToken::new()) .build(), ) .await?; Ok(format!("Cancelled (reason={reason}), {cleanup_result}")) } Err(err) => Err(err.into()), } } } ``` ## Terminate a Workflow Execution Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. - The system records a `WorkflowExecutionTerminated` event in the Event History. - The termination forcefully and immediately stops the Workflow Execution. - The Workflow code gets no chance to handle termination. - A Workflow Task doesn't get scheduled. To terminate a Workflow Execution in Rust, use the `terminate` method on the client. ```rust let handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build() ).await?; handle.terminate(WorkflowTerminateOptions::builder() .reason("Emergency shutdown") .build() ).await?; ``` ## Reset a Workflow Execution Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. **Web UI** 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. **Temporal CLI** Use the `temporal workflow reset` command to reset a Workflow Execution: ```bash temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" ``` For example: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" ``` By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: ```bash temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem ``` Monitor the new Workflow Execution after resetting to ensure it completes successfully. --- # Child Workflows - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/child-workflows > Start a Child Workflow Execution and set a Parent Close Policy using the Rust SDK. Manage Child Workflow Events. This page shows how to do the following: - [Start a Child Workflow execution](#start-child-workflow) - [Set a Parent Close Policy](#parent-close-policy) ## Start a Child Workflow execution A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API. When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted)) are logged in the Workflow Execution Event History. The [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started. In Rust, awaiting `ctx.start_child_workflow()` waits for this Event before returning, so the Child Workflow is guaranteed to have started once the call resolves. To start a Child Workflow in Rust, use `ctx.start_child_workflow()`: ```rust use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{ChildWorkflowOptions, WorkflowContext, WorkflowContextView, WorkflowResult}; // child workflow #[workflow] pub struct ComposeGreetingWorkflow { pub name: String, } #[workflow_methods] impl ComposeGreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); Ok(format!("Hello from child: {}", name)) } } // parent workflow #[workflow] pub struct GreetingWorkflow { pub name: String, } #[workflow_methods] impl GreetingWorkflow { #[init] fn new(_ctx: &WorkflowContextView, name: String) -> Self { Self { name } } #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.name.clone()); let started = ctx .start_child_workflow( ComposeGreetingWorkflow::run, name, ChildWorkflowOptions::workflow_id("greeting-child-en".to_string()), ) .await?; let result = started.result().await?; Ok(format!("ComposeGreetingWorkflow result: {result}")) } } ``` ### Specify Child Workflow options Use [`ChildWorkflowOptions`](https://docs.rs/temporalio-sdk/0.7.0/temporalio_sdk/struct.ChildWorkflowOptions.html) to customize Child Workflow behavior. ### Execute multiple Child Workflows in parallel You can start multiple Child Workflows and wait for all of them: ```rust pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult>> { let name = ctx.state(|s| s.name.clone()); let en_greeting_child = ctx .start_child_workflow( ComposeEnGreetingWorkflow::run, name.clone(), ChildWorkflowOptions::workflow_id("greeting-child-en".to_string()), ) .await?; let es_greeting_child = ctx .start_child_workflow( ComposeEsGreetingWorkflow::run, name, ChildWorkflowOptions::workflow_id("greeting-child-es".to_string()), ) .await?; let en_result = en_greeting_child.result().await; let es_result = es_greeting_child.result().await; let combined = vec![en_result, es_result]; print!("Combined greetings: {:?}", combined); ... } ``` Both child Workflows run in parallel and the parent waits for both to complete. ## Parent Close Policy A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out). The default Parent Close Policy is set to terminate the Child Workflow Execution. Set Parent Close Policy using the [`parent_close_policy`](https://docs.rs/temporalio-sdk/0.7.0/temporalio_sdk/enum.ParentClosePolicy.html) field in `ChildWorkflowOptions`: ```rust use temporalio_sdk::ParentClosePolicy; let es_greeting_child = ctx .start_child_workflow( ComposeEsGreetingWorkflow::run, name, ChildWorkflowOptions::builder() .workflow_id("greeting-child-es".to_string()) .parent_close_policy(ParentClosePolicy::Abandon) .build(), ) .await?; ``` ### Parent Close Policy options - `Terminate` (default) - The Child Workflow will be terminated immediately when the parent closes - `Abandon` - The Child Workflow will continue running even if the parent closes - `RequestCancel` - The Child Workflow will receive a cancellation request when the parent closes --- # Continue-As-New - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/continue-as-new > Use Temporal's Continue-As-New in Rust to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. This page answers the following questions for Rust developers: - [What is Continue-As-New?](#what) - [How to Continue-As-New?](#how) - [When is it right to Continue-as-New?](#when) ## What is Continue-As-New? [Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow execution close successfully and creates a new Workflow execution. You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits. The new Workflow execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History. It also receives your Workflow's usual parameters. ## How to Continue-As-New using the Rust SDK First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run. This state is typically passed as a parameter or stored in the Workflow struct. Inside your Workflow, call `ctx.continue_as_new()` and propagate its result: ```rust use std::time::Duration; use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{ActivityOptions, ContinueAsNewOptions, WorkflowContext, WorkflowResult}; use crate::activities::MyActivities; #[workflow] #[derive(Default)] pub struct GreetingWorkflow; #[workflow_methods] impl GreetingWorkflow { #[run(name = "greeting-workflow-1")] pub async fn run(ctx: &mut WorkflowContext, name: String) -> WorkflowResult { let greeting = ctx .execute_activity( MyActivities::greet, name.clone(), ActivityOptions::start_to_close_timeout(Duration::from_secs(30)), ) .await?; println!("{}", greeting); if name == "Ziggy" { Ok(greeting) } else { ctx.continue_as_new("New Name".to_string(), ContinueAsNewOptions::default())?; } } } ``` The `ctx.continue_as_new()` method accepts the input to pass to the next Workflow Run. ## When is it right to Continue-as-New using the Rust SDK? Use Continue-as-New when your Workflow might encounter degraded performance or [Event History Limits](/workflow-execution/event#event-history). Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New. Call `ctx.continue_as_new_suggested()` to check if it's time. ## How to test Continue-as-New using the Rust SDK Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests. For example, if you have an internal value like `test_continue_as_new == True`, this sample takes a variable called `max_history_length` and that can be set to a small value. A helper method in the Workflow impl checks it each time it considers using Continue-as-New: ```rust use serde::{Deserialize, Serialize}; use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{WorkflowContext, WorkflowResult}; #[derive(Serialize, Deserialize)] pub struct GreetingInput { pub name: String, pub max_history_length: u32, } ... #[workflow_methods] impl GreetingWorkflow { #[run] pub async fn run( ctx: &mut WorkflowContext, input: GreetingInput, ) -> WorkflowResult { // your Workflow code here if Self::should_continue_as_new(ctx, input.max_history_length) { // Continue as new } } fn should_continue_as_new( ctx: &WorkflowContext, max_history_length: u32, ) -> bool { if ctx.continue_as_new_suggested() { return true; } // For testing if max_history_length > 0 && ctx.history_length() > max_history_length { return true; } false } } ``` ## Best practices 1. Pass all necessary state: When continuing as new, include all state the next run needs. 2. Use meaningful iteration markers: Include iteration numbers or timestamps to track progress. 3. Test your state passing: Ensure parameters serialize and deserialize correctly. 4. Don't continue-as-new too frequently: It's better to have some Event History than to continue-as-new on every execution. 5. Consider batch sizes: Find the right balance between batch size and number of continues as new. --- # Workflow message passing - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/message-passing > Develop with Queries, Signals, and Updates with the Temporal Rust SDK. A Workflow can act like a stateful web service that receives messages: Queries, Signals, and Updates. The Workflow implementation defines these endpoints via handler methods that can react to incoming messages and return values. Temporal Clients use messages to read Workflow state and control its execution. See [Workflow message passing](/encyclopedia/workflow-message-passing) for a general overview of this topic. This page introduces these features for the Temporal Rust SDK. ## Write message handlers Follow these guidelines when writing your message handlers: - Message handlers are defined as methods on your Workflow struct and registered with the Workflow runtime. - The parameters and return values of handlers and the main Workflow function must be [serializable](/dataconversion). - Prefer structs to multiple input parameters to allow for forward-compatible changes. ### Query handlers A [Query](/sending-messages#sending-queries) is a synchronous operation that retrieves state from a Workflow Execution: ```rust // workflows.rs use std::collections::HashMap; use temporalio_macros::{workflow, workflow_methods}; use temporalio_sdk::{WorkflowContext, WorkflowContextView, WorkflowResult}; #[derive(serde::Serialize, serde::Deserialize, Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum Language { Chinese, English, French, } #[derive(serde::Serialize, serde::Deserialize)] pub struct GetLanguagesInput { pub include_unsupported: bool, } #[workflow] pub struct GreetingsWorkflow { pub greetings: HashMap, } #[workflow_methods] impl GreetingsWorkflow { #[init] fn new(_ctx: &WorkflowContextView) -> Self { let mut greetings = HashMap::new(); greetings.insert(Language::Chinese, "你好，世界".to_string()); greetings.insert(Language::English, "Hello, world".to_string()); Self { greetings } } #[run(name = "greetings-workflow-10")] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.greetings.clone()); Ok(format!("Hola: {:?}", name)) } #[query] pub fn get_languages( &self, _ctx: &WorkflowContextView, input: GetLanguagesInput, ) -> Vec { if input.include_unsupported { vec![Language::Chinese, Language::English, Language::French] } else { self.greetings.keys().copied().collect() } } } ``` ```rust // main.rs let wf_handle = client .start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build(), ) .await?; let supported_languages = wf_handle .query( GreetingsWorkflow::get_languages, GetLanguagesInput { include_unsupported: true, }, WorkflowQueryOptions::default(), ) .await?; ``` - Query handlers can't mutate Workflow state. - Query handlers can't perform async operations, like executing Activities. ### Signal handlers A [Signal](/sending-messages#sending-signals) is an asynchronous message sent to a running Workflow Execution to change its state and control its flow: ```rust #[derive(serde::Serialize, serde::Deserialize)] pub struct ApproveInput { pub name: String, } // Other structs ... #[workflow_methods] impl GreetingsWorkflow { // Other Workflow logic ... #[signal] pub fn approve(&mut self, _ctx: &mut SyncWorkflowContext, input: ApproveInput) { self.approved_for_release = true; self.approver_name = Some(input.name); } } ``` ```rust // main.rs wf_handle .signal( GreetingsWorkflow::approve, ApproveInput { name: "Ziggy".to_string(), }, WorkflowSignalOptions::default(), ) .await?; ``` * Signal handlers do not return values. * They can trigger async work (Activities, timers) depending on SDK capabilities. ### Update handlers and validators An [Update](/sending-messages#sending-updates) is a trackable synchronous request sent to a running Workflow Execution. It can change the Workflow state, control its flow, and return a result. The sender must wait until the Worker accepts or rejects the Update. The sender may wait further to receive a returned value or an exception if something goes wrong: ```rust // workflows.rs ... #[derive(serde::Serialize, serde::Deserialize)] pub struct SetLanguageInput { pub language: Language, } ... #[workflow_methods] impl GreetingsWorkflow { // Other Workflow logic ... #[update] pub fn set_language( &mut self, _ctx: &mut SyncWorkflowContext, input: SetLanguageInput, ) -> Language { let previous_language = self.language; self.language = input.language; previous_language } #[update_validator(set_language)] fn validate_set_language( &self, _ctx: &WorkflowContextView, input: &SetLanguageInput, ) -> Result<(), Box> { if !self.greetings.contains_key(&input.language) { Err("Not a valid language".into()) } else { Ok(()) } } } ``` - About validators: - Use validators to reject an Update before it is written to History. Validators are always optional. If you don't need to reject Updates, you can skip them. - If you choose to create validators for your Updates, they will reject Updates before they're applied. - Accepting and rejecting Updates with validators: - To reject an Update, raise an exception of any type in the validator. - Without a validator, Updates are always accepted. - Validators and Event History: - The `WorkflowExecutionUpdateAccepted` event is written into the History whether the acceptance was automatic or programmatic. - When a Validator raises an error, the Update is rejected and `WorkflowExecutionUpdateAccepted` won't be added to the Event History. The caller receives an "Update failed" error. - Update and Signal handlers can be async, letting you use Activities, Child Workflows, and more. See [Async handlers](#async-handlers) for safe usage guidelines. ## Send messages To send Queries, Signals, or Updates, use a Workflow handle from the client: ```rust let client = Client::new(connection, ClientOptions::new("default").build())?; let wf_handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10").build() ).await?; ``` ### Send a Query ```rust let supported_languages = wf_handle.query( GreetingsWorkflow::get_languages, GetLanguagesInput { include_unsupported: true }, WorkflowQueryOptions::default() ).await?; ``` - Sending a Query doesn’t add events to a Workflow's Event History. - You can send Queries to closed Workflow Executions within a Namespace's Workflow retention period. This includes Workflows that have completed, failed, or timed out. Querying terminated Workflows is not safe and, therefore, not supported. - A Worker must be online and polling the Task Queue to process a Query. ### Send a Signal You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed. #### From a Client ```rust wf_handle.signal( GreetingsWorkflow::approve, ApproveInput { name: "Ziggy".to_string() }, WorkflowSignalOptions::default() ).await?; ``` - The call returns when the server accepts the Signal; it does not wait for the Signal to be delivered to the Workflow Execution. #### From a Workflow ```rust ctx.external_workflow("workflow-id-1", Some("run-id-1".into())) .signal( GreetingsWorkflow::approve, ApproveInput { name: "Ziggy".to_string() }, SignalWorkflowOptions::default(), ) .await?; ``` ### Signal-With-Start ```rust let signal_input = vec![ApproveInput { name: "Ziggy".to_string(), } .as_json_payload()?] .into_payloads(); let wf_handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new("my-task-queue", "greetings-workflow-10") .start_signal( WorkflowStartSignal::new("approve") .maybe_input(signal_input) .build(), ) .build(), ).await?; ``` ### Send an Update An Update is a synchronous, blocking call that can change Workflow state, control its flow, and return a result. A client sending an Update must wait until the Server delivers the Update to a Worker. Workers must be available and responsive. If you need a response as soon as the Server receives the request, use a Signal instead. You can't send Updates directly from one Workflow to another. If you need to send Updates across Workflows, like to Child Workflows, use an Activity. - `WorkflowExecutionUpdateAccepted` is added to the Event History when the Worker confirms that the Update passed validation. - `WorkflowExecutionUpdateCompleted` is added to the Event History when the Worker confirms that the Update has finished. To send an Update to a Workflow Execution, you can call `execute_update` and wait for the Update to complete. This code fetches an Update result: ```rust let previous_language = wf_handle.execute_update( GreetingsWorkflow::set_language, SetLanguageInput { language: Language::French }, WorkflowExecuteUpdateOptions::default() ).await?; ``` #### Update-With-Start You can also send `start_update` to receive an `UpdateHandle` as soon as the Update is accepted. - Use this `UpdateHandle` later to fetch your results. - Async Update handlers normally perform long-running asynchronous operations, such as executing an Activity. - `start_update` only waits until the Worker has accepted or rejected the Update, not until all asynchronous operations are complete. For example: ```rust let update_handle = main_wf_handle.start_update( GreetingsWorkflow::set_language, SetLanguageInput { language: Language::French }, WorkflowStartUpdateOptions::default() ).await?; ``` - Updates are synchronous and return results. - Worker must accept the Update before it proceeds. ## Message handler patterns ### Async handlers Signal and Update handlers can be `async fn` as well as `fn`. Using `async fn` allows you to use await with Activities, Child Workflows, Timers, etc. This expands the possibilities for what can be done by a handler, but it also means that handler executions and your main Workflow method are all running concurrently, with switching occurring between them at await calls. It's essential to understand the things that could go wrong in order to use `async fn` handlers safely. See [Workflow message passing](/encyclopedia/workflow-message-passing) for guidance on safe usage of async Signal and Update handlers, the Safe message handlers sample and the sections below. The following code executes an Activity that makes a network call to a remote service: ```rust #[derive(serde::Serialize, serde::Deserialize, Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum Language { Arabic, Chinese, English, French, Hindi, Spanish, } pub struct GreetingActivities; #[activities] impl GreetingActivities { #[activity] pub async fn call_greeting_service(_ctx: ActivityContext, to_language: Language) -> Result, ActivityError> { // Pretend that we are calling a remote service. sleep(Duration::from_millis(200)).await; let greetings = HashMap::from([ (Language::Arabic, "مرحبا بالعالم".to_string()), (Language::Chinese, "你好，世界".to_string()), (Language::English, "Hello, world".to_string()), (Language::French, "Bonjour, monde".to_string()), (Language::Hindi, "नमस्ते दुनिया".to_string()), (Language::Spanish, "Hola mundo".to_string()), ]); Ok(greetings.get(&to_language).cloned()) } } ``` After updating the code to use an `async fn`, your Update handler can schedule an Activity and await the result. Although an `async fn` Signal handler can also execute an Activity, using an Update handler allows the client to receive a result or error once the Activity completes. This lets your client track the progress of asynchronous work performed by the Update's Activities, Child Workflows, etc. Here's how you could start that Activity from within your Workflow: ```rust #[update] async fn set_language_activity( ctx: &mut WorkflowContext, language: Language, ) -> Result> { let needs_greeting = ctx.state(|s| !s.greetings.contains_key(&language)); if needs_greeting { ctx.wait_condition(|s| !s.approved_for_release).await?; ctx.state_mut(|s| { s.approved_for_release = true; }); let greeting = ctx .execute_activity( GreetingActivities::call_greeting_service, Language::French, ActivityOptions::start_to_close_timeout(Duration::from_secs(10)), ) .await; ctx.state_mut(|s| { s.approved_for_release = false; }); let greeting = greeting?; if let Some(greeting) = greeting { ctx.state_mut(|s| { s.greetings.insert(language, greeting); }); } } let previous_language = ctx.state(|s| s.language); ctx.state_mut(|s| { s.language = language; }); Ok(previous_language) } ``` ### Add wait conditions to block Sometimes, async Signal or Update handlers need to meet certain conditions before they should continue. You can use `ctx.wait_condition` to prevent the code from proceeding until a condition is true. You specify the condition by passing a function that returns a boolean and you can optionally set a timeout. This is an important feature that helps you control your handler logic. Here are three important use cases for `ctx.wait_condition`: - Wait for a Signal or Update to arrive. - Wait in a handler until it's appropriate to continue. - Wait in the main Workflow until all active handlers have finished. It's common to use `ctx.wait_condition` to wait for a particular Signal or Update to be sent by a client. In your Workflow, you will have something like: ```rust #[run] pub async fn run(ctx: &mut WorkflowContext) -> WorkflowResult { let name = ctx.state(|s| s.greetings.clone()); ctx.wait_condition(|s| !s.approved_for_release).await?; // Other Workflow logic ... } ``` ### Ensure handlers finish before completion Workflow wait conditions can ensure your handler completes before a Workflow finishes. When your Workflow uses async Signal or Update handlers, your main Workflow method can return or continue-as-new while a handler is still waiting on an async task, such as an Activity result. The Workflow completing may interrupt the handler before it finishes crucial work and cause client errors when trying retrieve Update results. Use `ctx.wait_condition` to address this problem and allow your Workflow to end smoothly. --- # Workflow Timeouts - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/timeouts > Optimize Workflow Execution with the Temporal Rust SDK by configuring Workflow Timeouts and Retry Policies. ## Workflow timeouts Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution. In most cases, **Workflow Timeouts are not recommended**. Temporal Workflows are designed to be long-running and resilient. Setting timeouts can unnecessarily limit their ability to tolerate delays or extended processing. If you need to trigger logic after a specific duration, use a **Timer inside the Workflow** instead of a timeout. You can configure Workflow timeouts when starting a Workflow Execution. Available timeouts include: - [Workflow Execution Timeout](/encyclopedia/detecting-workflow-failures#workflow-execution-timeout): Maximum total time a Workflow Execution can run. - [Workflow Run Timeout](/encyclopedia/detecting-workflow-failures#workflow-run-timeout): Maximum time a single Workflow Run can last. - [Workflow Task Timeout](/encyclopedia/detecting-workflow-failures#workflow-task-timeout): Maximum time a Worker can take to complete a Workflow Task. In Rust, you set these via `WorkflowStartOptions` when starting or executing a Workflow. Available fields: - `execution_timeout` - `run_timeout` - `task_timeout` ```rust let wf_handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "my-task-queue", "greetings-workflow-10", ) // Set timeouts .execution_timeout(Duration::from_secs(3600)) .run_timeout(Duration::from_secs(600)) .task_timeout(Duration::from_secs(10)) .build() ).await?; ``` ## Workflow retries A Retry Policy can be used alongside timeouts to control how Workflow Executions are retried after failure. Workflow Executions do **not retry by default**. Retry Policies should only be applied when restarting the entire Workflow is safe and intentional. To enable retries, configure a [Retry Policy](/encyclopedia/retry-policies) when starting the Workflow. ```rust let wf_handle = client.start_workflow( GreetingsWorkflow::run, (), WorkflowStartOptions::new( "my-task-queue", "greetings-workflow-10", ) .retry_policy( RetryPolicy::builder() .initial_interval(Duration::from_secs(1)) .backoff_coefficient(2.0) .maximum_interval(Duration::from_secs(100)) .maximum_attempts(5) .non_retryable_error_types(["NonRetryableError"]) .build(), ) .build() ).await?; ``` Retry Policies define retry behavior such as backoff intervals, maximum attempts, and retry conditions. --- # Timers - Rust SDK Source: https://docs.temporal.io/develop/rust/workflows/timers > Set durable Timers in a Workflow using the Rust SDK. Timers persist through Worker and Temporal Service downtime. A Workflow can set a Durable Timer for a fixed time period. In some SDKs, the function is called `sleep()`, and in others, it's called `timer()`. A Workflow can sleep for days, months, or even years. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the `sleep()` call will resolve and your code will continue executing. Sleeping is a resource-light operation: it doesn't tie up the process, and you can run millions of Timers off a single Worker. To set a Timer in Rust, use the `timer()` function and pass the duration you want to wait before continuing. ```rust ctx.timer( TimerOptions::builder(Duration::from_secs(60)) .summary("important timer".to_owned()) .build(), ) .await; ``` --- # Safely deploying changes to Workflow code Source: https://docs.temporal.io/develop/safe-deployments > Safely deploy changes to existing Workflow code by validating first for determinism errors before deploying to production. Making changes safely to existing Workflow code requires care. Your Workflow code--as opposed to your Activity code--must be [deterministic](/workflow-definition#deterministic-constraints). This means your changes to that code have to be as well. Changes to your Workflow code that qualify as non-deterministic need to be protected by either using [Worker Versioning](/production-deployment/worker-deployments/worker-versioning) to pin your Workflows to specific code revisions, or by using the [patching APIs](/workflow-definition#workflow-versioning) within your Workflow code. > **📝 Note:** > We strongly recommend using Worker Versioning as users see improved error rates when adopting it. In this article, we’ll provide some advice on how you can safely validate changes to your Workflow code, ensuring that you won’t experience unexpected non-determinism errors in production when rolling them out. > **⚠️ Caution:** > Eager start does not respect Worker versioning. An eagerly started Workflow may run on any available local Worker even if that Worker is not the Current or Ramping version of its Worker deployment. ## Use Replay Testing before and during your deployments The best way to verify that your code won’t cause non-determinism errors once deployed is to make use of [replay testing](/workflow-execution#replay). Replay testing takes one or more existing [Event Histories](/workflow-execution/event#event-history) that ran against a previous version of Workflow code and runs them against your _current_ Workflow code, verifying that it is compatible with the provided history. In the case of Worker Versioning, you may have a [pinned Workflow](/worker-versioning#pinned) that you're switching over to the [current Worker deployment version](/worker-versioning#versioning-definitions) and you want to make sure that the changes don't introduce non-determinism errors. Or you may have an [Auto-Upgrade Workflow](/worker-versioning#auto-upgrade) that you want to run automated tests on to ensure the deployments don't trigger errors. There are multiple points in your development lifecycle where running replay tests can make sense. They exist on a spectrum, with shortest time to feedback on one end, and most representative of a production deployment on the other. - During development, replay testing lets you get feedback as early as possible on whether your changes are compatible. For example, you might include some integration tests that run your Workflows against the Temporal Test Server to produce histories which you then check in. You can use those checked-in histories for replay tests to verify you haven’t made breaking changes. - During pre-deployment validation (such as during some automated deployment validation) you can get feedback in a more representative environment. For example, you might fetch histories from a live Temporal environment (whether production or some kind of pre-production) and use them in replay tests. - At deployment time, your environment _is_ production, but you are using the new code to replay recent real-world Workflow histories. When you're writing changes to Workflow code, you can fetch some representative histories from your pre-production or production Temporal environment and verify they work with your changes. You can do the same with the pre-merge CI pipeline. However, if you are using encrypted Payloads, which is a typical and recommended setup in production, you may not be able to decrypt the fetched histories. Additionally if your Workflows contain any PII (which should be encrypted), make sure this information is scrubbed for the purposes of your tests, or err on the side of caution and don’t use this method. With that constraint in mind, we’ll focus on how you can perform replay tests in a production deployment of a Worker with new Workflow code. The core of how replay testing is done is the same regardless of when you choose to do it, so you can apply some of the lessons here to earlier stages in your development process. ## Implement a deployment-time replay test The key to a successful safe deployment is to break it into two phases: a verification phase, where you’ll run the replay test, followed by the actual deployment of your new Worker code. You can accomplish this by wrapping your Worker application with some code that can choose whether it will run in verification mode, or in production. This is most easily done if you do not deploy your Workers side-by-side with other application code, which is a recommended best practice. If you do deploy your Workers as part of some other application, you will likely need to separate out a different entry point specifically for verification. ### Run a replay and real Worker with the same code The following code demonstrates how the same entry point could be used to either verify the new code using replay testing, or to actually run the Worker. ```python import argparse import asyncio from datetime import datetime, timedelta from temporalio.client import Client from temporalio.worker import Worker, Replayer async def main(): parser = argparse.ArgumentParser(prog='MyTemporalWorker') parser.add_argument('mode', choices=['verify', 'run']) args = parser.parse_args() temporal_url = "localhost:7233" task_queue = "your-task-queue" my_workflows = [YourWorkflow] my_activities = [your_activity] client = await Client.connect(temporal_url) ``` Everything up to this point is standard. You import the Workflow and Activity code, instantiate a parser with two modes, and create your Task Queue, Workflow, and Activity. You can pass in the `args.mode` from any appropriate spot in your code. If the mode is set to `verify`, you conduct the replay testing by specifying the time period to test, and passing in the Workflows corresponding to that time period. Note that the Workflows are consumed as histories, using [the `map_histories()` function](https://python.temporal.io/temporalio.client.WorkflowExecutionAsyncIterator.html#map_histories). ```python if args.mode == 'verify': start_time = (datetime.now() - timedelta(hours=10)).isoformat(timespec='seconds') workflows = client.list_workflows( f"TaskQueue={task_queue} and StartTime > '{start_time}'", limit = 100) histories = workflows.map_histories() replayer = Replayer( workflows=my_workflows, ) await replayer.replay_workflows(histories) return ``` If any of the Workflows fail to replay, an error will be thrown. If no errors occur, you can return successfully to indicate success here, or communicate with an endpoint you've defined to indicate success or failure of the verification. You could switch to the `run` mode, and have this Worker transition to a real Worker that will start pulling from the Task Queue and processing Workflows: ```python else: worker = Worker( client, task_queue=task_queue, workflows=my_workflows, activities=my_activities, ) await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` ### Use the multi-modal Worker The most straightforward way to use this bimodal Worker is to deploy one instance of it at the beginning of your deployment process in verify mode, see that it passes, and then proceed to deploy the rest of your new workers in run mode. --- # Task Queue Priority and Fairness Source: https://docs.temporal.io/develop/task-queue-priority-fairness > Control execution order within a Task Queue using priority levels, and use Fairness to stop tenants from blocking each other. [Task Queue Priority](#task-queue-priority) and [Task Queue Fairness](#task-queue-fairness) are two ways to manage the distribution of work within a Task Queue. Priority allows [Tasks](/tasks) to be executed in Priority order. Fairness prevents one set of Tasks from blocking others within the same priority level. You can use Priority and Fairness individually or combine them to express Fairness within a Priority level. ## Task Queue Priority **Task Queue Priority** lets you control the execution order of Workflows, Activities, and Child Workflows based on assigned priority values within a Task Queue. Each priority level acts as a sub-queue that separates Tasks so that high priority Tasks can cut in front of low priority Tasks. ![Flowchart of how Priority dispatches Tasks from highest to lowest priority queue](/img/develop/task-queue-priority-fairness/priority-details.png) Priority is enforced within a single Task Queue [partition](/task-queue#task-ordering). A Task Queue by default uses multiple partitions and randomly distributes Tasks across them, so partitions are usually balanced and priority ordering closely approximates the Task Queue as a whole. When partitions become imbalanced, lower-priority Tasks on a lighter partition can dispatch ahead of higher-priority Tasks waiting on a heavier one. ### When to use Priority If you need a way to specify the order your Tasks execute in, you can use Priority to manage that. Priority lets you differentiate between your Tasks, like batch and real-time Tasks, so that you can use a single pool of Workers for efficient resource allocation, while ensuring real-time Tasks are processed ahead of batch Tasks. You can also use this as a way to run urgent Tasks immediately and override others. For example, if you are running an e-commerce platform, you may want to process payment related Tasks before less time-sensitive Tasks like internal inventory management. ### How to use Priority Priority is enabled by default in both Temporal Cloud and self-hosted Temporal. To disable Priority in self-hosted Temporal, set the [dynamic config](/temporal-service/configuration#dynamic-configuration) `matching.useNewMatcher` to `false` on a Task Queue, Namespace, or globally. To use Priority, you need to set a _priority key_ at the Workflow, Activity, or Child Workflow level to a value within the integer range `[1,5]`. A lower value implies higher priority, so `1` is the highest priority level. If you don't specify a Priority, a Task defaults to a Priority of `3`. Activities and Child Workflows will inherit their Workflow's priority unless they explicitly specify their own priority. When Priority is enabled, all Tasks within a Task Queue will be processed in Priority order. For example, all priority level `1` Tasks will start executing before the first priority level `2` Task, and so on. Lower priority Tasks will be blocked until all higher priority Tasks have started. Tasks are scheduled by default to run in first-in-first-out (FIFO) order within each priority level. If you need greater control of task ordering within a priority level, such as preventing large tenants from overwhelming small tenants, check out [the Fairness section](#task-queue-fairness). You can set a Workflow's priority key via the CLI like so: ``` temporal workflow start \ --type ChargeCustomer \ --task-queue my-task-queue \ --workflow-id my-workflow-id \ --input '{"customerId":"12345"}' \ --priority-key 1 ``` You can set priority keys for a Workflow within the SDK like so: **Go** ```go workflowOptions := client.StartWorkflowOptions{ ID: "my-workflow-id", TaskQueue: "my-task-queue", Priority: temporal.Priority{PriorityKey: 5}, } we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow) ``` **Java** ```java WorkflowOptions options = WorkflowOptions.newBuilder() .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder().setPriorityKey(5).build()) .build(); WorkflowClient client = WorkflowClient.newInstance(service); MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options); workflow.run(); ```` **Python** ```python await client.start_workflow( MyWorkflow.run, args="hello", id="my-workflow-id", task_queue="my-task-queue", priority=Priority(priority_key=1), ) ```` **.NET** ```csharp var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority(1), } ); ``` You can set priority keys for an Activity within the SDK like so: **Go** ```go ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{PriorityKey: 3}, } ctx := workflow.WithActivityOptions(ctx, ao) err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil) ``` **Java** ```java ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder().setPriorityKey(3).build()) .build(); MyActivity activity = Workflow.newActivityStub(MyActivity.class, options); activity.perform(); ```` **Python** ```python await workflow.execute_activity( say_hello, "hi", priority=Priority(priority_key=3), start_to_close_timeout=timedelta(seconds=5), ) ```` **TypeScript** **.NET** ```csharp await Workflow.ExecuteActivityAsync( () => SayHello("hi"), new() { StartToCloseTimeout = TimeSpan.FromSeconds(5), Priority = new(3), } ); ``` You can set priority keys for a Child Workflow within the SDK like so: **Go** ```go cwo := workflow.ChildWorkflowOptions{ WorkflowID: "child-workflow-id", TaskQueue: "child-task-queue", Priority: temporal.Priority{PriorityKey: 1}, } ctx := workflow.WithChildOptions(ctx, cwo) err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil) ``` **Java** ```java ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder() .setTaskQueue("child-task-queue") .setWorkflowId("child-workflow-id") .setPriority(Priority.newBuilder().setPriorityKey(1).build()) .build(); MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions); child.run(); ```` **Python** ```python await workflow.execute_child_workflow( MyChildWorkflow.run, args="hello child", priority=Priority(priority_key=1), ) ```` **.NET** ```csharp await Workflow.ExecuteChildWorkflowAsync( (MyChildWorkflow wf) => wf.RunAsync("hello child"), new() { Priority = new(1) } ); ``` ## Task Queue Fairness Task Queue Fairness lets you distribute Tasks based on _fairness keys_ and _fairness weights_ within a Task Queue. Each fairness key creates its own "virtual queue", allowing you to organize Tasks into logical groups like tenants, applications, or workload types. These virtual queues operate using a round-robin dispatch mechanism, meaning the system cycles through each fairness key in turn when selecting the next Task to dispatch. This prevents any single fairness key from hogging Worker capacity, even if one key has a much larger backlog than the others. By default, each fairness key is weighted equally in the round-robin, with a _fairness weight_ of 1.0. This behavior can be customized by assigning a different fairness weight to a key. For example, Tasks belonging to a fairness key with a weight of 2.0 will be dispatched twice as often as keys with the default weight. ### When to use Fairness Fairness is intended to address common situations like: - Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones. - Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another without limiting overall capacity when one band is empty. It sequences Tasks in the Task Queue probabilistically using a weighted distribution based on: - Fairness weights you set - The current backlog of Tasks - A [data structure](https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch) that tracks how you've distributed Tasks for different fairness keys As an example, imagine a workload with three tenants, _tenant-big_, _tenant-mid_, _tenant-small_, that have varying numbers of Tasks at all times. Your _tenant-big_ has a large number of Tasks that can overwhelm your Task Queue and prevent _tenant-mid_ and _tenant-small_ from running their Tasks. With Fairness, you can give each tenant a different fairness key to make sure _tenant-big_ doesn't use all of the Task Queue resources and block the others. In this case, _tenant-mid_ and _tenant-small_ will have Tasks run in between _tenant-big_ Tasks so that they are executed "fairly". ### How to use Fairness Fairness is available for both self-hosted Temporal instances and Temporal Cloud. To enable Fairness for a Namespace in Temporal Cloud, navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Note that Fairness is a paid feature in Temporal Cloud. For more information, see [Fairness pricing](/cloud/pricing#fairness-pricing). If you're self-hosting Temporal, set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) on the relevant Task Queues or Namespaces. To use Fairness, you need to set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, if you weight _premium-tier_ at 5.0, _basic-tier_ at 3.0, and _free-tier_ at 2.0, then 50% of dispatched Tasks come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_. If there are Tasks in the Task Queue backlog that have the same fairness key, then they're dispatched in [FIFO order](/task-queue#task-ordering). ![Flowchart of how Fairness selects a virtual queue weighted by fairness weight when the backlog has Tasks](/img/develop/task-queue-priority-fairness/fairness-details.png) You can set a Workflow's fairness key and weight via the CLI like so: ``` temporal workflow start \ --type ChargeCustomer \ --task-queue my-task-queue \ --workflow-id my-workflow-id \ --input '{"customerId":"12345"}' \ --priority-key 1 \ --fairness-key a-key \ --fairness-weight 3.14 ``` You can set fairness keys and weights for a Workflow within the SDK like so. Select a concept to highlight the matching lines: **Go** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [5], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [6], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [7], }, ]} > ```go workflowOptions := client.StartWorkflowOptions{ ID: "my-workflow-id", TaskQueue: "my-task-queue", Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, }, } we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow) ``` **Java** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [4], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [5], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [6], }, ]} > ```java WorkflowOptions options = WorkflowOptions.newBuilder() .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder() .setPriorityKey(5) .setFairnessKey("a-key") .setFairnessWeight(3.14) .build()) .build(); WorkflowClient client = WorkflowClient.newInstance(service); MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options); workflow.run(); ``` **Python** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [7], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [8], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [9], }, ]} > ```python await client.start_workflow( MyWorkflow.run, args="hello", id="my-workflow-id", task_queue="my-task-queue", priority=Priority( priority_key=3, fairness_key="a-key", fairness_weight=3.14, ), ) ``` **Ruby** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [6], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [7], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [8], }, ]} > ```ruby client.start_workflow( MyWorkflow, "input-arg", id: "my-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 ) ) ``` **TypeScript** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [4], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [5], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [6], }, ]} > ```ts const handle = await startWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14, }, }); ``` **.NET** annotations={[ { label: 'Priority key', description: 'Priority key in the range [1, 5]. Lower values run first. If unset, Tasks default to priority 3.', lines: [8], }, { label: 'Fairness key', description: 'Groups Tasks into a virtual queue (for example, by tenant or workload type) so no single group monopolizes the Task Queue.', lines: [9], }, { label: 'Fairness weight', description: 'Relative dispatch weight for this fairness key. Default is 1.0. Higher weights get a larger share of dispatches.', lines: [10], }, ]} > ```csharp var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) } ); ``` You can set fairness keys and weights for an Activity within the SDK like so: **Go** ```go ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, }, } ctx := workflow.WithActivityOptions(ctx, ao) err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil) ```` **Java** ```java ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder().setPriorityKey(3).setFairnessKey("a-key").setFairnessWeight(3.14).build()) .build(); MyActivity activity = Workflow.newActivityStub(MyActivity.class, options); activity.perform(); ```` **Python** ```python await workflow.execute_activity( say_hello, "hi", priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14), start_to_close_timeout=timedelta(seconds=5), ) ``` **Ruby** ```ruby client.start_activity( MyActivity, "input-arg", id: "my-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 ) ) ``` **TypeScript** ```ts const handle = await startWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 }, }); ``` **.NET** ```csharp var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) } ); ``` You can set fairness keys and weights for a Child Workflow within the SDK like so: **Go** ```go cwo := workflow.ChildWorkflowOptions{ WorkflowID: "child-workflow-id", TaskQueue: "child-task-queue", Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, }, } ctx := workflow.WithChildOptions(ctx, cwo) err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil) ``` **Java** ```java ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder() .setTaskQueue("child-task-queue") .setWorkflowId("child-workflow-id") .setPriority(Priority.newBuilder().setPriorityKey(1).setFairnessKey("a-key").setFairnessWeight(3.14).build()) .build(); MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions); child.run(); ``` **Python** ```python await workflow.execute_child_workflow( MyChildWorkflow.run, args="hello child", priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14), ) ``` **Ruby** ```ruby client.start_child_workflow( MyChildWorkflow, "input-arg", id: "my-child-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 ) ) ``` **TypeScript** ```ts const handle = await startChildWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 }, }); ``` **.NET** ```csharp var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) } ); ``` Tasks that do not have a `fairness_key` set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental: you can assign fairness keys to some tenants but not others. Unkeyed Tasks do not bypass Fairness; they compete as one group alongside all explicitly keyed Tasks. > **ℹ️ Info:** > > There should only be one fairness weight assigned to each fairness key within a Task Queue. Having multiple fairness weights on a fairness key will result in unspecific behavior. > ### Choosing between Priority, Fairness, and both - **Priority alone** when you need strict priority ordering - for example, separating real-time Tasks from batch Tasks. - **Fairness alone** when you need tier or tenant isolation so no group is starved, but you don't need to preempt any group ahead of another. - **Both** when you have a tiered SLA hierarchy - Priority for the broad tier (for example, paid vs. free), Fairness for per-tenant equity within a tier. When you use Priority and Fairness together, the next Task to dispatch is chosen by walking three rules in order: 1. **Priority tier (strict).** Tasks at a higher priority always dispatch before tasks at lower priorities, regardless of fairness keys or weights. 2. **Fairness key within a tier (weighted).** Within a priority tier, each fairness key is a virtual queue. Keys are dispatched proportional to their weights - a key with weight 2.0 is dispatched twice as often as one with weight 1.0. 3. **FIFO within a key.** Tasks that share a priority tier _and_ fairness key dispatch in the order they were enqueued. These rules apply within a Task Queue partition. ![Flowchart of how Priority and Fairness combine: poller checks priority levels in order, then within each level Fairness selects a virtual queue weighted by fairness weight](/img/develop/task-queue-priority-fairness/priority-fairness.png) ### Inheritance Each field of Priority (`priority_key`, `fairness_key`, `fairness_weight`) is resolved independently. ![Flowchart of how a Priority field is resolved: check for a Task Queue weight override (fairness_weight only), then an explicit value, then inherit from the calling Workflow, otherwise use the default](/img/develop/task-queue-priority-fairness/inheritance.png) **Activity inheritance order** (highest precedence first): 1. [Fairness weight overrides](#fairness-weight-overrides) on the Task Queue (`fairness_weight` only) 2. Value set explicitly in the Activity options 3. Inherited from the calling Workflow 4. Default value (`priority_key=3`, `fairness_key=""`, `fairness_weight=1.0`) **Workflow inheritance order** (highest precedence first): 1. [Fairness weight overrides](#fairness-weight-overrides) on the Task Queue (`fairness_weight` only) 2. Value set explicitly in the Workflow start options 3. Inherited from the parent Workflow (Child Workflows only) 4. Default value (`priority_key=3`, `fairness_key=""`, `fairness_weight=1.0`) Continue-As-New inherits from the current execution unless explicit values are passed. ### Enabling or disabling Fairness with an active backlog When Fairness is enabled on a Namespace, Task Queues in the Namespace begin honoring fairness keys on Tasks for dispatch ordering. Existing queued Tasks are dispatched first, in their original priority + FIFO order. Fairness keys on Tasks already in the backlog do not retroactively affect their dispatch order. When Fairness is disabled on a Namespace, Task Queues in the Namespace stop honoring fairness keys for dispatch ordering. The existing fairness-ordered backlog is dispatched first, in its original fairness order. After the backlog drains, Task Queues dispatch in priority + FIFO order. In both directions, the existing backlog is dispatched before any new Tasks queued under the new mode. New Tasks dispatch only after the backlog fully drains. Tasks are not lost in either transition. ### Set rate limits at the Task Queue level Within a Task Queue, you can set dispatch rate limits for the whole queue using `queue-rps-limit` and for each fairness key using `fairness-key-rps-limit-default`. ``` temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --queue-rps-limit 500 \ --queue-rps-limit-reason "overall limit" \ --fairness-key-rps-limit-default 33.3 \ --fairness-key-rps-limit-reason "per-key limit" ``` **Whole queue rate limits:** applies to the whole queue regardless of the fairness key. This is the same setting as is exposed through the [Worker Options](/develop/worker-tuning-reference#io-configuration-options) in the SDKs, and when set via the API, takes precedence over the limit set through Worker Options. **Fairness key rate limits:** The per-fairness-key rate limit works in conjunction with Task Queue Fairness. If you think of Fairness as dividing the queue into one virtual queue for each key, then the per-fairness-key rate limit is a limit on each individual virtual queue. Some important notes on the per-fairness-key limit: - The whole queue limit and per-fairness-key limit may be set independently: none, one or the other, or both may be set. If both are set, then the more restrictive one applies. - The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. So if the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a particular key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second. - Since the dispatch rate for each key should be proportional to its weight, if any key is hitting the per-key limit, then nearly all of them are. The way it works is if the next Task to be dispatched hits the per-key limit, then dispatch will wait until it can go. - Usually there isn't actually any blocking, but there can be when the fairness weight for a key is changed between when a Task is scheduled and when it's dispatched. If the fairness weight for a key is lowered, for example, the new lower per-key rate limit will be respected. Since those Tasks were originally scheduled with the higher rate, they will block other Tasks as they're dispatched. This limitation will be improved in the future. ### Fairness weight overrides You can override the weights of up to 1000 keys through the config API. When an override is set for a key, the weight attached to the Task, through Workflow or Activity priority metadata, will be ignored, and the overridden weight will be used instead. Weight overrides are stored per Task Queue, including type, so they must be set for both Workflow and Activity Task Queues to take effect for both. Set overrides with `temporal task-queue config set`: ``` temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --fairness-key-weight premium=5.0 \ --fairness-key-weight basic=1.0 ``` To unset a single key's override, pass `key=default`. To clear all overrides on the Task Queue, use `--fairness-key-weight-clear-all`. ### Limitations of Fairness - There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more. - Fairness is enforced within a single Task Queue [partition](/task-queue#task-ordering). When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition. - The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog of Tasks, you won't be able to. - When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_. - During server restarts, Temporal preserves fairness state for the top 100 keys. Other keys rebuild their fairness state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering reduces this distortion by spreading keys' initial positions according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). - Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers. --- # TypeScript SDK developer guide Source: https://docs.temporal.io/develop/typescript ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Install and get started You can find detailed installation instructions for the TypeScript SDK in the [Quickstart](/develop/typescript/set-up-your-local-typescript). There's also a short walkthrough of how to use the Temporal primitives (Activities, Workflows, and Workers) to build and run a Temporal application to get you up and running. Once your local Temporal Service is set up, continue building with the following resources: - [Develop a Workflow](/develop/typescript/workflows/basics) - [Develop an Activity](/develop/typescript/activities/basics) - [Start an Activity execution](/develop/typescript/activities/execution) - [Run Worker processes](/develop/typescript/workers/run-worker-process) ## [Workflows](/develop/typescript/workflows) - [Workflow basics](/develop/typescript/workflows/basics) - [Child Workflows](/develop/typescript/workflows/child-workflows) - [Continue-As-New](/develop/typescript/workflows/continue-as-new) - [Message passing](/develop/typescript/workflows/message-passing) - [Cancellation](/develop/typescript/workflows/cancellation) - [Cancellation scopes](/develop/typescript/workflows/cancellation-scopes) - [Timeouts](/develop/typescript/workflows/timeouts) - [Schedules](/develop/typescript/workflows/schedules) - [Timers](/develop/typescript/workflows/timers) - [Versioning](/develop/typescript/workflows/versioning) - [Workflow Streams](/develop/typescript/workflows/workflow-streams) ## [Activities](/develop/typescript/activities) - [Activity basics](/develop/typescript/activities/basics) - [Activity execution](/develop/typescript/activities/execution) - [Timeouts](/develop/typescript/activities/timeouts) - [Asynchronous Activity](/develop/typescript/activities/asynchronous-activity) - [Benign exceptions](/develop/typescript/activities/benign-exceptions) ## [Workers](/develop/typescript/workers) - [Worker processes](/develop/typescript/workers/run-worker-process) - [Interceptors](/develop/typescript/workers/interceptors) ## [Temporal Client](/develop/typescript/client) - [Temporal Client](/develop/typescript/client/temporal-client) - [Namespaces](/develop/typescript/client/namespaces) ## [Temporal Nexus](/develop/typescript/nexus) - [Quickstart](/develop/typescript/nexus/quickstart) - [Feature guide](/develop/typescript/nexus/feature-guide) - [Standalone Operations](/develop/typescript/nexus/standalone-operations) ## [Platform](/develop/typescript/platform) - [Observability](/develop/typescript/platform/observability) - [Enriching the UI](/develop/typescript/platform/enriching-ui) ## [Best practices](/develop/typescript/best-practices) - [Testing](/develop/typescript/best-practices/testing-suite) - [Debugging](/develop/typescript/best-practices/debugging) - [Converters and encryption](/develop/typescript/best-practices/data-handling) - [Entity pattern](/develop/typescript/best-practices/entity-pattern) ## [Integrations](/develop/typescript/integrations) - [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript) - [LangSmith integration](/develop/typescript/integrations/langsmith) - [Mastra integration](https://mastra.ai/guides/deployment/temporal) - [OpenAI Agents SDK integration](/develop/typescript/integrations/openai-agents) - [Parseable integration](https://github.com/parseablehq/temporal-plugin/blob/main/INTEGRATION.md) - [Strands Agents integration](/develop/typescript/integrations/strands-agents) - [Vercel AI SDK integration](/develop/typescript/integrations/ai-sdk) ## Temporal TypeScript technical resources - [TypeScript SDK Quickstart - Setup Guide](/develop/typescript/set-up-your-local-typescript) - [TypeScript API Documentation](https://typescript.temporal.io) - [TypeScript SDK Code Samples](https://github.com/temporalio/samples-typescript) - [TypeScript SDK GitHub](https://github.com/temporalio/sdk-typescript) - [Temporal 101 in TypeScript Free Course](https://learn.temporal.io/courses/temporal_101/typescript/) ## Get connected with the Temporal TypeScript community - [Temporal TypeScript Community Slack](https://temporalio.slack.com/archives/C01DKSMU94L) - [TypeScript SDK Forum](https://community.temporal.io/tag/typescript-sdk) ## Linting and types in TypeScript If you started your project with `@temporalio/create`, you already have our recommended TypeScript and ESLint configurations. If you incrementally added Temporal to an existing app, we do recommend setting up linting and types because they help catch bugs well before you ship them to production, and they improve your development feedback loop. Take a look at our recommended [.eslintrc](https://github.com/temporalio/samples-typescript/blob/main/.shared/.eslintrc.js) file and tweak to suit your needs. --- # Activities - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities > This section explains how to implement Activities with the TypeScript SDK ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Activities - [Activity basics](/develop/typescript/activities/basics) - [Activity execution](/develop/typescript/activities/execution) - [Standalone Activities Quickstart](/develop/typescript/activities/standalone-activities-quickstart) - [Standalone Activities Feature Guide](/develop/typescript/activities/standalone-activities) - [Timeouts](/develop/typescript/activities/timeouts) - [Asynchronous Activity](/develop/typescript/activities/asynchronous-activity) - [Benign exceptions](/develop/typescript/activities/benign-exceptions) --- # Asynchronous Activity - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities/asynchronous-activity > Asynchronously complete an Activity in Temporal by enabling the Activity Function to return before the Activity Execution finishes, using AsyncCompletionClient. ## How to asynchronously complete an Activity [Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To asynchronously complete an Activity, call [`AsyncCompletionClient.complete`](https://typescript.temporal.io/api/classes/client.AsyncCompletionClient#complete). [activities-examples/src/activities/async-completion.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/activities/async-completion.ts) ```ts import { CompleteAsyncError, activityInfo } from '@temporalio/activity'; import { Client } from '@temporalio/client'; export async function doSomethingAsync(): Promise { const taskToken = activityInfo().taskToken; setTimeout(() => doSomeWork(taskToken), 1000); throw new CompleteAsyncError(); } // this work could be done in a different process or on a different machine async function doSomeWork(taskToken: Uint8Array): Promise { const client = new Client(); // does some work... await client.activity.complete(taskToken, "Job's done!"); } ``` ## Local Activities To call [Local Activities](/local-activity) in TypeScript, use [`proxyLocalActivities`](https://typescript.temporal.io/api/namespaces/workflow/#proxylocalactivities). ```ts import * as workflow from '@temporalio/workflow'; const { getEnvVar } = workflow.proxyLocalActivities({ startToCloseTimeout: '2 seconds', }); export async function yourWorkflow(): Promise { const someSetting = await getEnvVar('SOME_SETTING'); // ... } ``` Local Activities must be registered with the Worker the same way non-local Activities are. --- # Activity basics - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities/basics > Shows how to create an Activity with the TypeScript SDK ## How to develop an Activity One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](/activity-definition). Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a Temporal Client. The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/typescript/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. See [Standalone Activities](/develop/typescript/activities/standalone-activities-quickstart). - Activities execute in the standard Node.js environment. - Activities cannot be in the same file as Workflows and must be separately registered. - Activities may be retried repeatedly, so you may need to use idempotency keys for critical side effects. Activities are _just functions_. The following is an Activity that accepts a string parameter and returns a string. [snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts) ```ts export async function greet(name: string): Promise { return `👋 Hello, ${name}!`; } ``` ## How to develop Activity Parameters There is no explicit limit to the total number of parameters that an [Activity Definition](/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](/tasks#workflow-task). Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. This Activity takes a single `name` parameter of type `string`. [snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts) ```ts export async function greet(name: string): Promise { return `👋 Hello, ${name}!`; } ``` ## How to define Activity return values All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](/cloud/limits#per-message-grpc-limit)). Keep in mind that all return values are recorded in a [Workflow Execution Event History](/workflow-execution/event#event-history). In TypeScript, the return value is always a Promise. In the following example, `Promise` is the return value. ```typescript export async function greet(name: string): Promise { return `👋 Hello, ${name}!`; } ``` ## How to customize your Activity Type Activities have a Type that are referred to as the Activity name. The following examples demonstrate how to set a custom name for your Activity Type. You can customize the name of the Activity when you register it with the Worker. In the following example, the Activity Name is `activityFoo`. [snippets/src/worker-activity-type-custom.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/worker-activity-type-custom.ts) ```ts import { Worker } from '@temporalio/worker'; import { greet } from './activities'; async function run() { const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'snippets', activities: { activityFoo: greet, }, }); await worker.run(); } ``` ## Important design patterns for Activities The following are some important (and frequently requested) patterns for using our Activities APIs. These patterns address common needs and use cases. ### Share dependencies in Activity functions (dependency injection) Because Activities are "just functions," you can also create functions that create Activities. This is a helpful pattern for using closures to do the following: - Store expensive dependencies for sharing, such as database connections. - Inject secret keys (such as environment variables) from the Worker to the Activity. [activities-dependency-injection/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/activities.ts) ```ts export interface DB { get(key: string): Promise; } export const createActivities = (db: DB) => ({ async greet(msg: string): Promise { const name = await db.get('name'); // simulate read from db return `${msg}: ${name}`; }, async greet_es(mensaje: string): Promise { const name = await db.get('name'); // simulate read from db return `${mensaje}: ${name}`; }, }); ``` #### See full example When you register these in the Worker, pass your shared dependencies accordingly: ```ts import { createActivities } from './activities'; async function run() { // Mock DB connection initialization in Worker const db = { async get(_key: string) { return 'Temporal'; }, }; const worker = await Worker.create({ taskQueue: 'dependency-injection', workflowsPath: require.resolve('./workflows'), activities: createActivities(db), }); await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); ``` Because Activities are always referenced by name, inside the Workflow they can be proxied as normal, although the types need some adjustment: [activities-dependency-injection/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/workflows.ts) ```ts import type { createActivities } from './activities'; // Note usage of ReturnType<> generic since createActivities is a factory function const { greet, greet_es } = proxyActivities>({ startToCloseTimeout: '30 seconds', }); ``` ### Import multiple Activities simultaneously You can proxy multiple Activities from the same `proxyActivities` call if you want them to share the same timeouts, retries, and options: ```ts export async function Workflow(name: string): Promise { // destructuring multiple activities with the same options const { act1, act2, act3 } = proxyActivities(); /* activityOptions */ await act1(); await Promise.all([act2, act3]); } ``` ### Dynamically reference Activities Because Activities are referenced only by their string names, you can reference them dynamically if needed: ```js export async function DynamicWorkflow(activityName, ...args) { const acts = proxyActivities(/* activityOptions */); // these are equivalent await acts.activity1(); await acts['activity1'](); // dynamic reference to activities using activityName let result = await acts[activityName](...args); } ``` Type safety is still supported here, but we encourage you to validate and handle mismatches in Activity names. An invalid Activity name leads to a `NotFoundError` with a message that looks like this: ``` ApplicationFailure: Activity function actC is not registered on this Worker, available activities: ["actA", "actB"] ``` --- # Benign exceptions - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities/benign-exceptions > Mark expected or non-severe Activity errors as benign to reduce noise in logs, metrics, and OpenTelemetry traces. When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the `category` field to `ApplicationFailureCategory.BENIGN` when creating an [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure). Benign errors: - Have Activity failure logs downgraded to DEBUG level - Do not emit Activity failure metrics - Do not set the OpenTelemetry failure status to ERROR ```typescript import { ApplicationFailure, ApplicationFailureCategory, } from '@temporalio/common'; export async function myActivity(): Promise { try { return await callExternalService(); } catch (err) { const message = err instanceof Error ? err.message : String(err); throw ApplicationFailure.create({ message, // Mark this error as benign since it's expected category: ApplicationFailureCategory.BENIGN, }); } } ``` Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Activity execution - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities/execution > Shows how to perform Activity execution with the TypeScript SDK ## How to start an Activity Execution Calls to spawn [Activity Executions](/activity-execution) are written within a [Workflow Definition](/workflow-definition). In TypeScript, you never call an Activity function directly. Instead, you pass in the _types_ of your Activities and Activity options to the `proxyActivities` function. This will give you an _Activity Handle_, a type-safe proxy object with the same function names and signatures as your real activities. From the Activity Handle, you can call your Activities as if they were normal async functions. ```typescript import { proxyActivities } from '@temporalio/workflow'; // Only import the activity types, not the functions themselves import type * as activities from './activities'; // Retrieve the Activity Handle by passing in the Activity types and options const activityHandle = proxyActivities({ startToCloseTimeout: '1 minute', }); // Deconstruct the individual Activity functions from the Activity Handle const { greet } = activityHandle; // A workflow that calls an activity export async function example(name: string): Promise { return await greet(name); } ``` When you call a proxied function, the Workflow does not execute the Activity code directly. Instead, it schedules an Activity Task. After the Activity Task is scheduled, it becomes available for a Worker to pick up and execute. This results in the set of three [Activity Task](/tasks#activity-task) related Events: [ActivityTaskScheduled](/references/events#activitytaskscheduled), [ActivityTaskStarted](/references/events#activitytaskstarted), and [ActivityTaskCompleted](/references/events#activitytaskcompleted) in your Workflow Execution Event History. The Worker may run many Activity executions at the same time, all using the same Activity function code. Temporal can also retry an Activity if it fails or times out. For this reason, you should write Activities to be [idempotent](/encyclopedia/activities/activity-definition.mdx#idempotency): calling them multiple times with the same input should have the same effect as calling them once. --- # Standalone Activities Feature Guide Source: https://docs.temporal.io/develop/typescript/activities/standalone-activities > Execute Activities independently without a Workflow using the Temporal TypeScript SDK. > **Public Preview** Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a [Temporal Client](/develop/typescript/client/temporal-client). The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/typescript/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **💡 Tip:** > > New to Standalone Activities? Start with the [Standalone Activities Quickstart](/develop/typescript/activities/standalone-activities-quickstart). > This page covers the following: - [Start a Standalone Activity without waiting for the result](#start-activity) - [Get a handle to an existing Standalone Activity](#get-activity-handle) - [Wait for the result of a Standalone Activity](#get-activity-result) - [List Standalone Activities](#list-activities) - [Count Standalone Activities](#count-activities) - [Run Standalone Activities with Temporal Cloud](#run-standalone-activities-temporal-cloud) > **📝 Note:** > > This documentation uses source code from the [standalone-activity](https://github.com/temporalio/samples-typescript/tree/main/standalone-activity) sample. > ## Start a Standalone Activity without waiting for the result Starting a Standalone Activity means sending a request to the Temporal Server to durably enqueue your Activity job, without waiting for it to be executed by your Worker. Use [`start`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#start) method of the client or the typed interface to start your Standalone Activity and get a handle: [standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts) ```ts const handle = await activitiesClient.start('greet', { ...activityOptions, id: activityId, args: ['Temporal'], }); ``` Or use the Temporal CLI: ```bash temporal activity start \ --type greet \ --activity-id my-standalone-activity-id \ --task-queue hello-standalone-activities \ --start-to-close-timeout 10s \ --input '"World"' ``` ## Get a handle to an existing Standalone Activity You can also use [`getHandle`](https://typescript.temporal.io/api/classes/client.ActivityClient#gethandle) to create a handle to a previously started Standalone Activity. Because the client doesn't know how the Activity was started, this method is not available in the typed interface. The method takes an optional type argument to constrain the Activity result type, but correctness of this argument is not verified. [standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts) ```ts const newHandle = client.activity.getHandle(activityId); ``` You can now use the handle to wait for the result, describe, cancel, or terminate the Activity. ## Wait for the result of a Standalone Activity Under the hood, calling [`execute`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#execute) is the same as calling [`start`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#start) to durably enqueue the Standalone Activity, and then calling `await handle.result()` to wait for the Activity to be executed and fetch the result: [standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts) ```ts console.log(await handle.result()); // Hello, Temporal! ``` Or use the Temporal CLI to wait for a result by Activity Id: ```bash temporal activity result --activity-id my-standalone-activity-id ``` ## List Standalone Activities Use [`list`](https://typescript.temporal.io/api/classes/client.ActivityClient#list) method of the client to list Standalone Activity Executions that match a [List Filter](/list-filter) query. The result is an `AsyncIterable` that yields [`ActivityExecutionInfo`](https://typescript.temporal.io/api/interfaces/client.ActivityExecutionInfo) entries. These APIs return only Standalone Activity Executions. Activities running inside Workflows are not included. ```typescript const query = 'TaskQueue="hello-standalone-activities"'; for await (const a of client.activity.list(query)) { console.log( `${a.activityId} | ${a.activityRunId} | ${a.activityType} | ${a.status} | ${a.closeTime?.toISOString()}`, ); } ``` The sample file [standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts) shows how to list and count activities. Run it: ```bash npm run list ``` Or use the Temporal CLI: ```bash temporal activity list ``` The query parameter accepts the same [List Filter](/list-filter) syntax used for [Workflow Visibility](/visibility). For example, "ActivityType = 'MyActivity' AND Status = 'Running'". ## Count Standalone Activities Use [`count`](https://typescript.temporal.io/api/classes/client.ActivityClient#count) method of the client to count Standalone Activity Executions that match a [List Filter](/list-filter) query. This returns the total count of executions (running, completed, failed, etc.) - not the number of queued tasks. It works the same way as counting Workflow Executions. The same query will work for both listing and counting. [standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts) ```ts const { count } = await client.activity.count(query); console.log(`Total activities: ${count}`); ``` The sample file [standalone-activity/src/list.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/list.ts) shows how to list and count activities. Run it: ```bash npm run list ``` Or use the Temporal CLI: ```bash temporal activity count ``` ## Run Standalone Activities with Temporal Cloud The Worker and Client code in the [Standalone Activities Quickstart](/develop/typescript/activities/standalone-activities-quickstart) use [`loadClientConnectConfig`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconnectconfig), so the same code works against Temporal Cloud - configure the connection via environment variables or a TOML profile. No code changes are needed. For a step-by-step guide on connecting to Temporal Cloud, including Namespace creation, certificate generation, and authentication setup in the Cloud UI, see [Connect to Temporal Cloud](/develop/typescript/client/temporal-client#connect-to-temporal-cloud). ### Connect with mTLS Set these environment variables with values from your Temporal Cloud Namespace settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_TLS_CLIENT_CERT_PATH='path/to/your/client.pem' export TEMPORAL_TLS_CLIENT_KEY_PATH='path/to/your/client.key' ``` ### Connect with an API key Set these environment variables with values from your Temporal Cloud API key settings: ``` export TEMPORAL_ADDRESS=..tmprl.cloud:7233 export TEMPORAL_NAMESPACE=. export TEMPORAL_API_KEY= ``` Then run the Worker and starter code as shown in the [Standalone Activities Quickstart](/develop/typescript/activities/standalone-activities-quickstart). --- # Standalone Activities TypeScript Quickstart Source: https://docs.temporal.io/develop/typescript/activities/standalone-activities-quickstart > Execute a Standalone Activity with the Temporal TypeScript SDK without writing a Workflow. # Quickstart Standalone Activities are Activities that run independently, without being orchestrated by a Workflow. Instead of starting an Activity from within a Workflow Definition, you start a Standalone Activity directly from a [Temporal Client](/develop/typescript/client/temporal-client). The way you write the Activity and register it with a Worker is identical to [Workflow Activities](/develop/typescript/activities/basics). The only difference is that you execute a Standalone Activity directly from your Temporal Client. > **📝 Note:** > > This documentation uses source code from the [standalone-activity](https://github.com/temporalio/samples-typescript/tree/main/standalone-activity) sample. > ## Get started with Standalone Activities Prerequisites: - **Temporal TypeScript SDK** (v1.17.0 or higher). See the [TypeScript Quickstart](/develop/typescript/set-up-your-local-typescript) for install instructions. - **Temporal CLI** v1.7.0 or higher. Install with Homebrew, or see the [Temporal CLI install guide](/cli/setup-cli) for other platforms. Verify the installation with `temporal --version`. Start the Temporal development server with `temporal server start-dev`. This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Server will now be available for client connections on `localhost:7233`, and the Temporal Web UI will now be accessible at [http://localhost:8233](http://localhost:8233). ```bash brew install temporal ``` ```bash temporal --version ``` ```bash temporal server start-dev ``` ## Clone the sample Clone the [samples-typescript](https://github.com/temporalio/samples-typescript) repository to follow along: ``` git clone https://github.com/temporalio/samples-typescript.git cd samples-typescript ``` The sample project is structured as follows: ``` standalone-activity/src/ ├── activities.ts ├── execute.ts ├── list.ts └── worker.ts ``` ## Write an Activity function The way you write a Standalone Activity is identical to how you write an Activity to be orchestrated by a Workflow. In fact, an Activity can be executed both as a Standalone Activity and as a Workflow Activity. [standalone-activity/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/activities.ts) ```typescript import { ApplicationFailure } from '@temporalio/activity'; export async function greet(name: string): Promise { if (typeof name !== 'string') { throw ApplicationFailure.create({ message: 'name must be a string', nonRetryable: true }); } return \`Hello, \${name}!\`; } ``` ## Run a Worker with the Activity registered Running a Worker for Standalone Activities is the same as running a Worker for Workflow Activities — you create a Worker, register the Activity, and run the Worker. The Worker doesn't need to know whether the Activity will be invoked from a Workflow or as a Standalone Activity. See [How to run a Worker](/develop/typescript/workers/run-worker-process#run-a-dev-worker) for more details on Worker setup and configuration options. [standalone-activity/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/worker.ts) Open a new terminal, navigate to the `samples-typescript/standalone-activity` directory, and run the Worker. Leave this terminal running - the Worker needs to stay up to process activities. ```typescript import { NativeConnection, Worker } from '@temporalio/worker'; import * as activities from './activities'; import { loadClientConnectConfig } from '@temporalio/envconfig'; async function run() { const config = loadClientConnectConfig(); const connection = await NativeConnection.connect(config.connectionOptions); try { const worker = await Worker.create({ connection, namespace: 'default', taskQueue: 'hello-standalone-activities', activities, }); await worker.run(); } finally { await connection.close(); } } run().catch((err) => { console.error(err); process.exit(1); }); ``` ```bash npm run start ``` ## Run the sample client The sample file [standalone-activity/src/execute.ts](https://github.com/temporalio/samples-typescript/blob/main/standalone-activity/src/execute.ts) contains a program demonstrating various ways to execute Standalone Activities and fetch results. The code in the following sections is copied from this file. To run it: 1. Make sure the Temporal Server is running (from the [Get Started](#get-started) step above). 2. Make sure the Worker is running (from the [Run a Worker](#run-worker) step above). 3. Open a new terminal, navigate to the `samples-typescript/standalone-activity` directory, and run the execute command. ```bash npm run execute ``` ## Execute a Standalone Activity with type checking Start by [creating a Temporal Client](/develop/typescript/client/temporal-client). Then call [`client.activity.typed()`](https://typescript.temporal.io/api/classes/client.ActivityClient#typed) to get a typed Activity Client interface. Any TypeScript type can be used as type argument as long as it has Activity functions as its methods. An easy way to provide such a type is to use the `typeof` operator on imported activities. Note that calling `typed` does not create a new Client object - it only adjusts the type annotation of the existing Client. `typed` can be called multiple times with different type arguments to use the same Client for multiple Activity interfaces. Afterwards, call the [`execute`](https://typescript.temporal.io/api/interfaces/client.TypedActivityClient#execute) method of the typed client to execute a Standalone Activity. Call this from your application code, not from inside a Workflow Definition. This durably enqueues your Standalone Activity in the Temporal Server, waits for it to be executed on your Worker, and then fetches the result. An unknown or mistyped Activity name, or wrong argument types will cause compilation to fail. ```typescript import { Connection, Client, ActivityExecutionFailedError } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import * as activities from './activities'; import { nanoid } from 'nanoid'; const config = loadClientConnectConfig(); const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection }); const activitiesClient = client.activity.typed(); ``` ```typescript const taskQueue = 'hello-standalone-activities'; const activityOptions = { taskQueue, startToCloseTimeout: '10s', }; // In practice, use a meaningful business identifier, like customer or transaction identifier const activityId = nanoid(); const result = await activitiesClient.execute('greet', { ...activityOptions, id: activityId, args: ['World'], }); ``` ## Execute a Standalone Activity without type checking Since Activity types are not always available, the [`start`](https://typescript.temporal.io/api/classes/client.ActivityClient#start) and [`execute`](https://typescript.temporal.io/api/classes/client.ActivityClient#execute) methods can be called on [`ActivityClient`](https://typescript.temporal.io/api/classes/client.ActivityClient) directly without using the typed interface. When called that way, neither the Activity name nor argument types are checked client-side. Or use the Temporal CLI. ```typescript await client.activity.execute('greet', { ...activityOptions, id: activityId, args: [1], }); ``` ```bash temporal activity execute \\ --type greet \\ --activity-id my-standalone-activity-id \\ --task-queue hello-standalone-activities \\ --start-to-close-timeout 10s \\ --input '"World"' ``` ## Run with Temporal Cloud All code samples on this page use [`loadClientConnectConfig()`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconnectconfig) to configure the Temporal Client connection. It responds to [environment variables](/references/client-environment-configuration) and [TOML configuration files](/references/client-environment-configuration), so the same code works against a local dev server and Temporal Cloud without changes. See [Run Standalone Activities with Temporal Cloud](/develop/typescript/activities/standalone-activities#run-standalone-activities-temporal-cloud) in the Feature Guide for mTLS and API key setup. ## Next steps - **[Standalone Activities Feature Guide](/develop/typescript/activities/standalone-activities)**: Start without waiting, get handles, list and count Activities, and connect to Temporal Cloud. - **[Activity basics](/develop/typescript/activities/basics)**: How to write and register Activities with the TypeScript SDK. --- # Activity Timeouts - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/activities/timeouts > Optimize Workflow Execution with Temporal TypeScript SDK - Set Activity Timeouts and Retry Policies efficiently. This page shows how to do the following: - [Activity Timeouts](#activity-timeouts) - [Activity Retry Policy](#activity-retries) - [Activity next Retry delay](#activity-next-retry-delay) - [Heartbeat an Activity](#activity-heartbeats) - [Activity Heartbeat Timeout](#activity-heartbeat-timeout) ## Activity Timeouts **How to set Activity Timeouts using the Temporal TypeScript SDK** Each Activity Timeout controls the maximum duration of a different aspect of an Activity Execution. The following Timeouts are available in the Activity Options: - **[Schedule-To-Close Timeout](/encyclopedia/detecting-activity-failures#schedule-to-close-timeout):** is the maximum amount of time allowed for the entire [Activity Execution](/activity-execution), from when the [Activity Task](/tasks#activity-task) is initially scheduled by the Workflow to when the server receives a successful completion for that Activity Task. - **[Start-To-Close Timeout](/encyclopedia/detecting-activity-failures#start-to-close-timeout):** is the maximum time allowed for a single [Activity Task Execution](/tasks#activity-task-execution), from when the Activity Task Execution gets polled by a [Worker](/workers#worker) to when the server receives a successful completion for that Activity Task. - **[Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout):** is the maximum amount of time that is allowed from when an [Activity Task](/tasks#activity-task) is initially scheduled by the Workflow to when a [Worker](/workers#worker) polls the Activity Task Execution. This timeout is non-retryable by design. An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set. The following properties can be set on the [`ActivityOptions`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions) when creating Activity proxy functions using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API: - [`scheduleToCloseTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#scheduletoclosetimeout) - [`startToCloseTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#starttoclosetimeout) - [`scheduleToStartTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions/#scheduletostarttimeout) ```typescript const { myActivity } = proxyActivities({ scheduleToCloseTimeout: '5m', // startToCloseTimeout: "30s", // recommended // scheduleToStartTimeout: "60s", }); ``` ## Activity Retry Policy **How to set an Activity Retry Policy using the Temporal TypeScript SDK** A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience. Activity Executions are automatically associated with a default [Retry Policy](/encyclopedia/retry-policies) if a custom one is not provided. To set an Activity's Retry Policy in TypeScript, assign the [`ActivityOptions.retry`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions#retry) property when creating the corresponding Activity proxy function using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API. ```typescript const { myActivity } = proxyActivities({ // ... retry: { initialInterval: '10s', maximumAttempts: 5, }, }); ``` ## Activity next Retry delay **How to override the next Retry delay following an Activity failure using the Temporal TypeScript SDK** The time to wait after a retryable Activity failure until the next retry is attempted is normally determined by that Activity's Retry Policy. However, an Activity may override that duration when explicitly failing with an [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure) by setting a next Retry delay. To override the next Retry delay for an `ApplicationFailure` thrown by an Activity in TypeScript, provide the [`nextRetryDelay`](https://typescript.temporal.io/api/interfaces/common.ApplicationFailureOptions#nextretrydelay) property on the object argument of the [`ApplicationFailure.create()`](https://typescript.temporal.io/api/classes/common.ApplicationFailure#create) factory method. ```typescript throw ApplicationFailure.create({ // ... nextRetryDelay: '15s', }); ``` ## Heartbeat an Activity **How to Heartbeat an Activity using the Temporal TypeScript SDK** An [Activity Heartbeat](/encyclopedia/detecting-activity-failures#activity-heartbeat) is a ping from the [Worker Process](/workers#worker-process) that is executing the Activity to the [Temporal Service](/temporal-service). Each Heartbeat informs the Temporal Service that the [Activity Execution](/activity-execution) is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) time period, the Activity will be considered as timed out and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't get notified of Cancellation requests. > **📝 Note:** > Handling Activity Cancellation > In the TypeScript SDK, Activity implementations must opt in to observe cancellation. > > Use one of the following in your Activity to be notified of cancellation: > - `await Context.current().cancelled` — rejects with `CancelledFailure`. > - `await Context.current().sleep(ms)` — rejects on cancellation. > - Pass `Context.current().cancellationSignal` (an `AbortSignal`) to libraries that support abort. > > > **Important:** Activities receive cancellation notifications when they heartbeat; if an Activity doesn’t heartbeat, delivery of the cancellation notification can be delayed. > > See [Activity namespace reference](https://typescript.temporal.io/api/namespaces/activity#cancellation) and [Context API](https://typescript.temporal.io/api/classes/activity.Context) for more information. Heartbeats may not always be sent to the Temporal Service—they may be [throttled](/encyclopedia/detecting-activity-failures#throttling) by the Worker. Heartbeat throttling may lead to Cancellation getting delivered later than expected. To Heartbeat an Activity Execution in TypeScript, call the [`heartbeat()`](https://typescript.temporal.io/api/namespaces/activity#heartbeat) function from the Activity implementation. ```typescript export async function myActivity(): Promise { for (let progress = 1; progress <= 1000; ++progress) { // Do something that takes time await sleep('1s'); heartbeat(); } } ``` An Activity may optionally checkpoint its progression, by providing a `details` argument to the [`heartbeat()`](https://typescript.temporal.io/api/namespaces/activity#heartbeat) function. Should the Activity Execution times out and gets retried, then the Temporal Server will provide the `details` from the last Heartbeat it received to the next Activity Execution. This can be used to allow the Activity to efficiently resume its work. ```typescript export async function myActivity(): Promise { // Resume work from latest heartbeat, if there's one, or start from 1 otherwise const startingPoint = activityInfo().heartbeatDetails?.progress ?? 1; for (let progress = startingPoint; progress <= 1000; ++progress) { // Do something that takes time await sleep('1s'); heartbeat({ progress }); } } ``` ## Activity Heartbeat Timeout **How to set a Heartbeat Timeout using the Temporal TypeScript SDK** A [Heartbeat Timeout](/encyclopedia/detecting-activity-failures#heartbeat-timeout) works in conjunction with [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat). If the Temporal Server doesn't receive a Heartbeat before expiration of the Heartbeat Timeout, the Activity is considered as timed out and another [Activity Task Execution](/tasks#activity-task-execution) may be scheduled according to the Retry Policy. To set an Activity's Heartbeat Timeout in TypeScript, set the [`ActivityOptions.heartbeatTimeout`](https://typescript.temporal.io/api/interfaces/common.ActivityOptions#heartbeattimeout) property when creating the corresponding Activity proxy functions using the [`proxyActivities()`](https://typescript.temporal.io/api/namespaces/workflow#proxyactivities) API. ```typescript const { myLongRunningActivity } = proxyActivities({ // ... heartbeatTimeout: '30s', }); ``` --- # Best practices - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Best practices - [Testing](/develop/typescript/best-practices/testing-suite) - [Debugging](/develop/typescript/best-practices/debugging) - [Converters and encryption](/develop/typescript/best-practices/data-handling) - [Entity pattern](/develop/typescript/best-practices/entity-pattern) --- # Data handling - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling All data sent to and from the Temporal Service passes through the **Data Converter**. The Data Converter has three layers that handle different concerns: ![The Flow of Data through a Data Converter](/diagrams/data-converter-flow-with-external-storage.svg) Of these three layers, only the PayloadConverter is required. Temporal uses a default PayloadConverter that handles JSON serialization. The PayloadCodec and ExternalStorage layers are optional. You only need to customize these layers when your application requires non-JSON types, encryption, or payload offloading. | | [PayloadConverter](/develop/typescript/best-practices/data-handling/data-conversion) | [PayloadCodec](/develop/typescript/best-practices/data-handling/data-encryption) | [ExternalStorage](/develop/typescript/best-practices/data-handling/external-storage) | | ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------- | | **Purpose** | Serialize application data to bytes | Transform encoded payloads (encrypt, compress) | Offload large payloads to external store | | **Default** | JSON serialization | None (passthrough) | None (all payloads are stored in Event History) | For a deeper conceptual explanation, see the [Data Conversion encyclopedia](/dataconversion) and [External Storage](/external-storage). --- # Payload conversion - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/data-conversion > Customize how Temporal serializes application objects using Payload Converters in the TypeScript SDK. Payload Converters serialize your application objects into a `Payload` and deserialize them back. A `Payload` is a binary form with metadata that Temporal uses to transport data. By default, Temporal uses a Payload Converter that handles `null`, byte arrays, protobuf messages, and anything JSON-serializable. You only need a custom Payload Converter when your application uses types that aren't natively supported. ## Default supported types The default Data Converter supports converting multiple types including: - `undefined` - `null` - `Uint8Array` - JSON serializables You can work with [protobufs](#protobufs) like `proto3-json-serializer` or `protobufjs` by using an instance of `DefaultPayloadConverterWithProtobufs`. ## Payload Codec > API documentation: [PayloadCodec](https://typescript.temporal.io/api/interfaces/common.PayloadCodec) The default `PayloadCodec` does nothing. To create a custom one, you can implement the following interface: ```ts interface PayloadCodec { /** * Encode an array of {@link Payload}s for sending over the wire. * @param payloads May have length 0. */ encode(payloads: Payload[]): Promise; /** * Decode an array of {@link Payload}s received from the wire. */ decode(payloads: Payload[]): Promise; } ``` ## Use custom payload conversion Temporal SDKs provide a [Payload Converter](/payload-converter) that can be customized to convert a custom data type to a [Payload](/dataconversion#payload) and back. The order in which your encoding Payload Converters are applied depends on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, the value gets passed through each Payload Converter in sequence until the converter that handles the data type does the conversion. ## Composite Data Converters Use a [Composite Data Converter](https://typescript.temporal.io/api/classes/common.CompositePayloadConverter) to apply custom, type-specific Payload Converters in a specified order. Defining a new Composite Data Converter is not always necessary to implement custom data handling. You can override the default Converter with a custom Codec, but a Composite Data Converter may be necessary for complex Workflow logic. A Composite Data Converter can include custom rules created, and it can also leverage the default Data Converters built into Temporal. In fact, the default Data Converter logic is implemented internally in the Temporal source as a Composite Data Converter. It defines these rules in this order: ```typescript export class DefaultPayloadConverter extends CompositePayloadConverter { constructor() { super( new UndefinedPayloadConverter(), new BinaryPayloadConverter(), new JsonPayloadConverter(), ); } } ``` The order of applying the Payload Converters is important. During serialization, the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-null Payload. To replace the default Data Converter with a custom `CompositeDataConverter`, use the following: ```typescript export const payloadConverter = new CompositePayloadConverter( new UndefinedPayloadConverter(), new EjsonPayloadConverter(), ); ``` You can do this in its own `payload-converter.ts` file for example. In the code snippet above, a converter is created that first attempts to handle `null` and `undefined` values. If the value isn't `null` or `undefined`, the EJSON serialization logic written in the `EjsonPayloadConverter` is then used. The Payload Converter is then provided to the Worker and Client. Here is the Worker code: ```typescript const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'ejson', dataConverter: { payloadConverterPath: require.resolve('./payload-converter'), }, }); ``` With this code, you now ensure that the Worker serializes and deserializes Workflow and Activity inputs and outputs using your EJSON-based logic, along with handling undefined values appropriately. Here is the Client: ```typescript const client = new Client({ dataConverter: { payloadConverterPath: require.resolve('./payload-converter'), }, }); ``` You can now use a variety of data types in arguments. ## How to use a custom payload converter in TypeScript To support custom Payload conversion, create a [custom Payload Converter](/payload-converter#composite-data-converters) and configure the Data Converter to use it in your Client options. You can use Custom Payload Converters to change how application objects get serialized to binary Payload. To handle custom data types that are not natively JSON-serializable (for example, `BigInt`, `Date`, or binary data), you can create a custom Payload Converter. A Custom Payload Converter is responsible for converting your custom data types to a payload format that Temporal can manage. To implement a Custom Payload Converter in TypeScript, you need to do the following steps: 1. **Implement `PayloadConverter` Interface**: Start by creating a class that implements Temporal's [`PayloadConverter`](https://typescript.temporal.io/api/interfaces/common.PayloadConverter) interface. ```typescript interface PayloadConverter { /** * Converts a value to a {@link Payload}. * @param value The value to convert. Example values include the Workflow args sent by the client and the values returned by a Workflow or Activity. */ toPayload(value: T): Payload; /** * Converts a {@link Payload} back to a value. */ fromPayload(payload: Payload): T; } ``` This custom converter should include logic for both serialization (`toPayload`) and deserialization (`fromPayload`), handling your specific data types or serialization format. The method `toPayload` returns a Payload object, which is used to manage and transport serialized data. The method `fromPayload` returns the deserialized data. This ensures that the data returned is in the same format as it was before serialization, allowing it to be used directly in the application. 2. Configure the Data Converter. To send values that are not JSON-serializable like a `BigInt` or `Date`, provide the custom Data Converter to the Client and Worker as described in the [Composite Data Converters](#composite-data-converters) section. #### Custom implementation Some example implementations are in the SDK itself: - [common/src/converter/payload-converter.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/common/src/converter/payload-converter.ts) - [common/src/converter/protobuf-payload-converters.ts](https://github.com/temporalio/sdk-typescript/blob/main/packages/common/src/converter/protobuf-payload-converters.ts) The sample project [samples-typescript/ejson](https://github.com/temporalio/samples-typescript/tree/main/ejson) creates an EJSON custom `PayloadConverter`. It implements `PayloadConverterWithEncoding` instead of `PayloadConverter` so that it could be used with [CompositePayloadConverter](https://typescript.temporal.io/api/classes/common.CompositePayloadConverter/): [ejson/src/ejson-payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/ejson-payload-converter.ts) ```ts import { EncodingType, METADATA_ENCODING_KEY, Payload, PayloadConverterWithEncoding, PayloadConverterError, } from '@temporalio/common'; import EJSON from 'ejson'; import { decode, encode } from '@temporalio/common/lib/encoding'; /** * Converts between values and [EJSON](https://docs.meteor.com/api/ejson.html) Payloads. */ export class EjsonPayloadConverter implements PayloadConverterWithEncoding { // Use 'json/plain' so that Payloads are displayed in the UI public encodingType = 'json/plain' as EncodingType; public toPayload(value: unknown): Payload | undefined { if (value === undefined) return undefined; let ejson; try { ejson = EJSON.stringify(value); } catch (e) { throw new UnsupportedEjsonTypeError( `Can't run EJSON.stringify on this value: ${value}. Either convert it (or its properties) to EJSON-serializable values (see https://docs.meteor.com/api/ejson.html ), or create a custom data converter. EJSON.stringify error message: ${errorMessage( e, )}`, e as Error, ); } return { metadata: { [METADATA_ENCODING_KEY]: encode('json/plain'), // Include an additional metadata field to indicate that this is an EJSON payload format: encode('extended'), }, data: encode(ejson), }; } public fromPayload(content: Payload): T { return content.data ? EJSON.parse(decode(content.data)) : content.data; } } export class UnsupportedEjsonTypeError extends PayloadConverterError { public readonly name: string = 'UnsupportedJsonTypeError'; constructor( message: string | undefined, public readonly cause?: Error, ) { super(message ?? undefined); } } ``` Then we instantiate one and export it: [ejson/src/payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/payload-converter.ts) ```ts import { CompositePayloadConverter, UndefinedPayloadConverter } from '@temporalio/common'; import { EjsonPayloadConverter } from './ejson-payload-converter'; export const payloadConverter = new CompositePayloadConverter( new UndefinedPayloadConverter(), new EjsonPayloadConverter(), ); ``` We provide it to the Worker and Client: [ejson/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/worker.ts) ```ts const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'ejson', dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }, }); ``` [ejson/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/client.ts) ```ts const client = new Client({ connection, dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }, }); ``` Then we can use supported data types in arguments: [ejson/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/client.ts) ```ts const user: User = { id: uuid(), // age: 1000n, BigInt isn't supported hp: Infinity, matcher: /.*Stormblessed/, token: Uint8Array.from([1, 2, 3]), createdAt: new Date(), }; const handle = await client.workflow.start(example, { args: [user], taskQueue: 'ejson', workflowId: `example-user-${user.id}`, }); ``` And they get parsed correctly for the Workflow: [ejson/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/ejson/src/workflows.ts) ```ts import type { Result, User } from './types'; export async function example(user: User): Promise { const success = user.createdAt.getTime() < Date.now() && user.hp > 50 && user.matcher.test('Kaladin Stormblessed') && user.token instanceof Uint8Array; return { success, at: new Date() }; } ``` #### Protobufs To serialize values as [Protocol Buffers](https://protobuf.dev/) (protobufs): - Use [protobufjs](https://protobufjs.github.io/protobuf.js/). - Use runtime-loaded messages (not generated classes) and `MessageClass.create` (not `new MessageClass()`). - Generate `json-module.js` with a command like the following: ```sh pbjs -t json-module --workflow-id commonjs -o protos/json-module.js protos/*.proto ``` - Patch `json-module.js`: [protobufs/protos/root.js](https://github.com/temporalio/samples-typescript/blob/main/protobufs/protos/root.js) ```js const { patchProtobufRoot } = require('@temporalio/common/lib/protobufs'); const unpatchedRoot = require('./json-module'); module.exports = patchProtobufRoot(unpatchedRoot); ``` - Generate `root.d.ts` with the following command: ```sh pbjs -t static-module protos/*.proto | pbts -o protos/root.d.ts - ``` - Create a [`DefaultPayloadConverterWithProtobufs`](https://typescript.temporal.io/api/classes/protobufs.DefaultPayloadConverterWithProtobufs/): [protobufs/src/payload-converter.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/payload-converter.ts) ```ts import { DefaultPayloadConverterWithProtobufs } from '@temporalio/common/lib/protobufs'; import root from '../protos/root'; export const payloadConverter = new DefaultPayloadConverterWithProtobufs({ protobufRoot: root }); ``` Alternatively, we can use Protobuf Payload Converters directly, or with other converters. If we know that we only use Protobuf objects, and we want them binary encoded (which saves space over proto3 JSON, but can't be viewed in the Web UI), we could do the following: ```ts import { ProtobufBinaryPayloadConverter } from '@temporalio/common/lib/protobufs'; import root from '../protos/root'; export const payloadConverter = new ProtobufBinaryPayloadConverter(root); ``` Similarly, if we wanted binary-encoded Protobufs in addition to the other default types, we could do the following: ```ts import { BinaryPayloadConverter, CompositePayloadConverter, JsonPayloadConverter, UndefinedPayloadConverter, } from '@temporalio/common'; import { ProtobufBinaryPayloadConverter } from '@temporalio/common/lib/protobufs'; import root from '../protos/root'; export const payloadConverter = new CompositePayloadConverter( new UndefinedPayloadConverter(), new BinaryPayloadConverter(), new ProtobufBinaryPayloadConverter(root), new JsonPayloadConverter(), ); ``` - Provide it to the Worker: [protobufs/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/worker.ts) ```ts const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), activities, taskQueue: 'protobufs', dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }, }); ``` [WorkerOptions.dataConverter](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#dataconverter) - Provide it to the Client: [protobufs/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/client.ts) ```ts import { Client, Connection } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { v4 as uuid } from 'uuid'; import { foo, ProtoResult } from '../protos/root'; import { example } from './workflows'; async function run() { const config = loadClientConnectConfig(); const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }, }); const handle = await client.workflow.start(example, { args: [foo.bar.ProtoInput.create({ name: 'Proto', age: 2 })], // can't do: // args: [new foo.bar.ProtoInput({ name: 'Proto', age: 2 })], taskQueue: 'protobufs', workflowId: 'my-business-id-' + uuid(), }); console.log(`Started workflow ${handle.workflowId}`); const result: ProtoResult = await handle.result(); console.log(result.toJSON()); } ``` - Use protobufs in your Workflows and Activities: [protobufs/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/workflows.ts) ```ts import { proxyActivities } from '@temporalio/workflow'; import { foo, ProtoResult } from '../protos/root'; import type * as activities from './activities'; const { protoActivity } = proxyActivities({ startToCloseTimeout: '1 minute', }); export async function example(input: foo.bar.ProtoInput): Promise { const result = await protoActivity(input); return result; } ``` [protobufs/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/protobufs/src/activities.ts) ```ts import { foo, ProtoResult } from '../protos/root'; export async function protoActivity(input: foo.bar.ProtoInput): Promise { return ProtoResult.create({ sentence: `${input.name} is ${input.age} years old.` }); } ``` --- # Payload encryption - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/data-encryption > Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the TypeScript SDK. Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. ## Custom Payload Codec > Background: [Encryption](/payload-codec#encryption) The following is an example class that implements the `PayloadCodec` interface: [encryption/src/encryption-codec.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/encryption-codec.ts) ```ts import { webcrypto as crypto } from 'node:crypto'; import { METADATA_ENCODING_KEY, Payload, PayloadCodec, ValueError } from '@temporalio/common'; import { temporal } from '@temporalio/proto'; import { decode, encode } from '@temporalio/common/lib/encoding'; import { decrypt, encrypt } from './crypto'; const ENCODING = 'binary/encrypted'; const METADATA_ENCRYPTION_KEY_ID = 'encryption-key-id'; export class EncryptionCodec implements PayloadCodec { constructor( protected readonly keys: Map, protected readonly defaultKeyId: string, ) {} static async create(keyId: string): Promise { const keys = new Map(); keys.set(keyId, await fetchKey(keyId)); return new this(keys, keyId); } async encode(payloads: Payload[]): Promise { return Promise.all( payloads.map(async (payload) => ({ metadata: { [METADATA_ENCODING_KEY]: encode(ENCODING), [METADATA_ENCRYPTION_KEY_ID]: encode(this.defaultKeyId), }, // Encrypt entire payload, preserving metadata data: await encrypt( temporal.api.common.v1.Payload.encode(payload).finish(), this.keys.get(this.defaultKeyId)!, // eslint-disable-line @typescript-eslint/no-non-null-assertion ), })), ); } async decode(payloads: Payload[]): Promise { return Promise.all( payloads.map(async (payload) => { if (!payload.metadata || decode(payload.metadata[METADATA_ENCODING_KEY]) !== ENCODING) { return payload; } if (!payload.data) { throw new ValueError('Payload data is missing'); } const keyIdBytes = payload.metadata[METADATA_ENCRYPTION_KEY_ID]; if (!keyIdBytes) { throw new ValueError('Unable to decrypt Payload without encryption key id'); } const keyId = decode(keyIdBytes); let key = this.keys.get(keyId); if (!key) { key = await fetchKey(keyId); this.keys.set(keyId, key); } const decryptedPayloadBytes = await decrypt(payload.data, key); console.log('Decrypting payload.data:', payload.data); return temporal.api.common.v1.Payload.decode(decryptedPayloadBytes); }), ); } } async function fetchKey(_keyId: string): Promise { // In production, fetch key from a key management system (KMS). You may want to memoize requests if you'll be decoding // Payloads that were encrypted using keys other than defaultKeyId. const key = Buffer.from('test-key-test-key-test-key-test!'); const cryptoKey = await crypto.subtle.importKey( 'raw', key, { name: 'AES-GCM', }, true, ['encrypt', 'decrypt'], ); return cryptoKey; } ``` The encryption and decryption code is in [src/crypto.ts](https://github.com/temporalio/samples-typescript/tree/main/encryption/src/crypto.ts). Because encryption is CPU intensive, and doing AES with the crypto module built into Node.js blocks the main thread, we use `@ronomon/crypto-async`, which uses the Node.js thread pool. As before, we provide a custom Data Converter to the Client and Worker: [encryption/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/client.ts) ```ts const client = new Client({ connection, dataConverter: await getDataConverter(), }); const handle = await client.workflow.start(example, { args: ['Alice: Private message for Bob.'], taskQueue: 'encryption', workflowId: `my-business-id-${uuid()}`, }); console.log(`Started workflow ${handle.workflowId}`); console.log(await handle.result()); ``` [encryption/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/worker.ts) ```ts const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'encryption', dataConverter: await getDataConverter(), }); ``` When the Client sends `'Alice: Private message for Bob.'` to the Workflow, it gets encrypted on the Client and decrypted in the Worker. The Workflow receives the decrypted message and appends another message. When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client. [encryption/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/workflows.ts) ```ts export async function example(message: string): Promise { return `${message}\nBob: Hi Alice, I'm Workflow Bob.`; } ``` --- # External Storage - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/data-handling/external-storage > Offload large payloads to external storage using the claim check pattern in the TypeScript SDK. > **Pre-release** > APIs and configuration may change before General Availability. Join the > [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for > help. The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how to set up External Storage with Amazon S3 or Google Cloud Storage, and how to implement a custom storage driver. For a conceptual overview of External Storage and its use cases, see [External Storage](/external-storage). ## Store and retrieve large payloads with Amazon S3 or Google Cloud Storage The TypeScript SDK includes storage drivers for Amazon S3 and Google Cloud Storage. Select your storage backend in the tabs that follow. Only the driver setup differs between the two. Everything after that is the same. ### Prerequisites - A bucket that you have read and write access to. Refer to [lifecycle management](/external-storage#lifecycle) to ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see [Durable External Storage](/external-storage#durable-external-storage). - Credentials with permission to write objects on components that store payloads, and to read objects on components that retrieve them. Components that only retrieve payloads do not need write access, and the reverse is also true. - Install the driver, the client adapter for your cloud provider's SDK, and that SDK: **Amazon S3** ```sh npm install @temporalio/external-storage-s3 \ @temporalio/external-storage-s3-aws-sdk \ @aws-sdk/client-s3 ``` **Google Cloud Storage** ```sh npm install @temporalio/external-storage-gcs \ @temporalio/external-storage-gcs-google-sdk \ @google-cloud/storage ``` ### Procedure 1. Create a storage client, wrap it in the matching driver client, and pass the result to the driver. Each cloud SDK picks up your standard credentials from the environment: **Amazon S3** The AWS SDK reads environment variables, an IAM role, or your AWS config file. [features/snippets/external_storage/s3_setup/s3_driver_create.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_driver_create.ts) ```ts import { S3Client } from '@aws-sdk/client-s3'; import { S3StorageDriver } from '@temporalio/external-storage-s3'; import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk'; const s3Client = new S3Client({ region: 'us-east-2' }); const driver = new S3StorageDriver({ client: new AwsSdkS3StorageDriverClient(s3Client), bucket: 'my-temporal-payloads', }); ``` **Google Cloud Storage** The Google Cloud SDK reads Application Default Credentials. [features/snippets/external_storage/gcs_setup/gcs_driver_create.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/gcs_setup/gcs_driver_create.ts) ```ts import { Storage } from '@google-cloud/storage'; import { GcsStorageDriver } from '@temporalio/external-storage-gcs'; import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk'; const storage = new Storage(); const driver = new GcsStorageDriver({ client: new GoogleCloudGcsStorageDriverClient(storage), bucket: 'my-temporal-payloads', }); ``` To route payloads to different buckets at runtime, pass a function as `bucket` instead of a string. The function receives the store context and the payload, and returns a bucket name. 2. Build an `ExternalStorage` instance from the driver, set it on your Data Converter, and pass the converter to your Client and Worker. This step is the same for every driver. External Storage runs outside the Workflow sandbox, so you can pass the driver object directly rather than referencing it by path the way a custom Payload Converter requires: [features/snippets/external_storage/s3_setup/s3_external_storage_setup.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/s3_setup/s3_external_storage_setup.ts) ```ts import { Client, Connection } from '@temporalio/client'; import { ExternalStorage } from '@temporalio/common'; import { Worker } from '@temporalio/worker'; async function createClientAndWorker() { const dataConverter = { externalStorage: new ExternalStorage({ drivers: [driver] }), }; const connection = await Connection.connect(); const client = new Client({ connection, dataConverter }); const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'my-task-queue', dataConverter, }); } ``` By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `payloadSizeThreshold` option, even setting it to `0` to externalize all payloads regardless of size. Refer to [Configure payload size threshold](#configure-payload-size-threshold) for more information. All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business logic. Both drivers upload and download payloads concurrently, address objects by a SHA-256 hash of their contents so identical payloads are stored once, and verify that hash on retrieve. Each rejects any single payload larger than `maxPayloadSize`, which defaults to 50 MiB, and includes diagnostic metadata in error messages to help troubleshoot storage failures. ## Implement a custom storage driver If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver. Refer to [Choose a storage system](/external-storage#choose-storage) for guidance on selecting a backing store and [Lifecycle management](/external-storage#lifecycle) for retention requirements. A shared filesystem works for local development and for Workers that mount the same volume. For anything else, use a storage system that every Client and Worker can reach. The following sections walk through the key parts of a custom driver. ### 1. Implement the StorageDriver interface A custom driver implements the `StorageDriver` interface, which has two readonly properties and two methods: - `name` is a unique string that identifies the driver instance. The SDK stores this name in the claim check reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. For example, two S3 drivers could be named `"s3-primary"` and `"s3-archive"`. - `type` is a string that identifies the driver implementation, and the Worker reports it in its heartbeat. Unlike `name`, `type` must be the same across all instances of the same driver type regardless of configuration. Two S3 drivers named `"s3-primary"` and `"s3-archive"` would both report `"aws.s3driver"` as their type, while the built-in GCS driver reports `"gcp.gcsdriver"`. - `store()` receives an array of payloads and returns one `StorageDriverClaim` per payload. A claim wraps a set of string key-value pairs that the driver uses to locate the payload later. - `retrieve()` receives the claims that `store()` produced and returns the original payloads. ### 2. Store payloads In `store()`, serialize each Payload protobuf message to bytes and write the bytes to your storage system. The application data has already been serialized by the [Payload Converter](/develop/typescript/best-practices/data-handling/data-conversion) and [Payload Codec](/develop/typescript/best-practices/data-handling/data-encryption) before it reaches the driver. See the [data conversion pipeline](/external-storage#data-pipeline) for more details. Return a `StorageDriverClaim` for each payload with enough information to retrieve it later. The `context.target` provides identity information and is a discriminated union: check the `kind` property to distinguish `"workflow"` from `"activity"`, then read `namespace`, `id`, `runId`, and `type`. Consider structuring your storage keys to include this information so that you can identify which Workflow owns each payload. Within that scope, content-addressable keys, such as a SHA-256 hash of the payload bytes, can help deduplicate identical payloads. The built-in S3 and GCS drivers use this approach. ### 3. Retrieve payloads In `retrieve()`, download the bytes using the claim data, then reconstruct the Payload protobuf message. The Payload Converter handles deserializing the application data after the driver returns the payload. ### 4. Configure the Data Converter Pass your driver to an `ExternalStorage` instance on the Data Converter, and use the same converter when creating your Client and Worker. Both sides need it: a Client without External Storage configured cannot read an offloaded result. You can also package your driver as a [plugin](/develop/plugins-guide) for easier reuse across services: ```ts export function createDataConverter(rootDir: string = STORAGE_ROOT): DataConverter { return { externalStorage: new ExternalStorage({ drivers: [new FileSystemStorageDriver({ rootDir })], payloadSizeThreshold: PAYLOAD_SIZE_THRESHOLD, // With one driver registered, every offloaded payload goes to it. Register more // than one and a `driverSelector` becomes required, letting you route per // payload: a cheap archive tier for a known-bulky Workflow type, a driver per // tenant, or a per-region bucket. Returning `null` from the selector keeps that // payload inline, which is how you exempt specific payloads from offloading. // // driverSelector: (context, _payload) => // context.target?.type === 'processDocument' ? coldDriver : hotDriver, }), }; } ``` ## Configure payload size threshold You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the `payloadSizeThreshold` option, or set it to `0` to externalize all payloads regardless of size. Payloads smaller than the threshold stay inline in Event History. The size compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data. [features/snippets/external_storage/threshold/threshold_config.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/threshold/threshold_config.ts) ```ts const dataConverter = { externalStorage: new ExternalStorage({ drivers: [driver], payloadSizeThreshold: 0, }), }; ``` ## Use multiple storage drivers When you register multiple drivers, you must provide a `driverSelector` function that chooses which driver stores each payload. `ExternalStorage` throws if you register more than one driver without a selector. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when migrating between storage backends. Return `null` from the selector to keep a specific payload inline in Event History. Multiple drivers are useful in scenarios such as: - Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old driver remains available for retrieving existing claims. - Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver based on the runtime environment. Every registered driver needs a distinct `name`. Because `S3StorageDriver` defaults its `name` to `"aws.s3driver"`, registering two S3 drivers requires setting the `driverName` option on at least one of them. The following example registers two drivers but always selects `preferredDriver` for new payloads. The `legacyDriver` is only registered so the Worker can retrieve payloads that were previously stored with it: [features/snippets/external_storage/multiple_drivers/multiple_drivers.ts](https://github.com/temporalio/features/blob/main/features/snippets/external_storage/multiple_drivers/multiple_drivers.ts) ```ts const preferredDriver = new S3StorageDriver({ client: new AwsSdkS3StorageDriverClient(s3Client), bucket: 'my-bucket', }); const legacyDriver = new LegacyStorageDriver(); const externalStorage = new ExternalStorage({ drivers: [preferredDriver, legacyDriver], driverSelector: () => preferredDriver, }); ``` ## Multi-region durability with Amazon S3 To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) and an [S3 Multi-Region Access Point (MRAP)](https://aws.amazon.com/s3/features/multi-region-access-points/), then point the driver at the MRAP ARN instead of a bucket name. See [Durable External Storage](/external-storage#durable-external-storage) for the full pattern and trade-offs. MRAP requests are signed with [SigV4A](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html). The AWS SDK for JavaScript does not bundle a SigV4A signer, so install one alongside your existing dependencies: ```sh npm install @aws-sdk/signature-v4a ``` Import the signer once during application startup. The import registers the signer with the AWS SDK, so you do not reference it directly: ```ts import '@aws-sdk/signature-v4a'; ``` `@aws-sdk/signature-v4-crt` is an alternative implementation backed by the AWS Common Runtime. When both are installed, the AWS SDK prefers the CRT implementation. With the signer registered, the only remaining change is the value you pass as `bucket`: ```ts const driver = new S3StorageDriver({ client: new AwsSdkS3StorageDriverClient(s3Client), bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap', }); ``` --- # Debugging - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/debugging > The Temporal TypeScript SDK Debugging guide provides tools and tips for debugging Workflows and Workers in development and production environments. Troubleshoot common issues using the Web UI, Temporal CLI, and more. The Debugging section of the Temporal TypeScript SDK developer's guide covers tools for debugging and how to troubleshoot common issues. ## How to debug in a development environment In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](/web-ui) or [Temporal CLI](/cli). ## How to debug in a production environment You can debug production Workflows using: - [Web UI](/web-ui) - [Temporal CLI](/cli) - [Replay](/develop/typescript/best-practices/testing-suite#replay) - [Tracing](/develop/typescript/platform/observability#tracing) - [Logging](/develop/typescript/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](/develop/worker-performance). For information on setting up SDK metrics, see [Metrics](/develop/typescript/platform/observability#metrics) in the Observability section of the TypeScript SDK developer's guide. Debug Server performance with [Cloud metrics](/cloud/metrics/) or [self-hosted Server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics). ## How to troubleshoot common issues in the TypeScript SDK ### Two locations to watch - Workflow Errors are reflected in Temporal Web. - Worker errors and logs are reflected in the terminal. If something isn't behaving the way you expect, make sure to check both locations for helpful error messages. ### Stale Workflows If you are developing Workflows and finding that code isn't executing as expected, the first place to look is whether old Workflows are still running. If those old Workflows have the same name and are on the same Task Queue, Temporal will try to continue executing them on your new code by design. You may get errors that make no sense to you because: - Temporal is trying to execute old Workflow code that no longer exists in your codebase. - Your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about. Stale Workflows are usually a non-issue because the errors generated are just noise from code you no longer want to run. If you need to terminate old stale Workflows, you can do so with Temporal Web or the Temporal CLI. ### Workflow/Activity registration errors **If your Workflows or Activities are not imported or spelled correctly**, here are some errors we've seen: - `ApplicationFailure: 'MyFunction' is not a function` - `Workflow did not register a handler for MyQuery` Double check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues. **If you are running Temporal in a monorepo**, then your `node_modules` may be in a different location than where Temporal expects to find it by default, which results in errors like: ```bash [ERROR] Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js' in '/src' ``` Our [Next.js tutorial](https://learn.temporal.io/tutorials/typescript/nextjs) is written for people setting up Temporal **within an existing monorepo**, which may be of use here. When you pass a `workflowsPath`, our Webpack config expects to find `node_modules` in the same or a parent/ancestor directory. **If you are custom bundling your own Workflows** you may get errors like these: ```bash [ERROR] Failed to activate workflow { runId: 'aaf84a83-51ce-462a-9ab7-6a641a703bff', error: ReferenceError: exports is not defined, workflowExists: false } ``` Temporal Workflow Bundles need to [export a set of methods that fit the compiled `worker-interface.ts` from `@temporalio/workflow`](https://github.com/temporalio/sdk-typescript/blob/803d95ada755736c17c5af719fa6c447e4cd75ac/packages/worker/src/workflow/bundler.ts#L180) as an entry point. We do offer a `bundleWorkflowCode` method to assist you with this, though it uses our Webpack settings. For more information, see the [Register types](/develop/typescript/workers/run-worker-process#register-types) section. ### Webpack errors The TypeScript SDK's Worker bundles Workflows based on `workflowsPath` with [Webpack](https://webpack.js.org/) and run them inside v8 isolates. If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's [logger](/develop/typescript/platform/observability#logging). If you do not see Webpack output in your terminal make sure that you have not disabled SDK logging (see reference to `Runtime.install()` in the link above). **A common mistake for newcomers to the TypeScript SDK is trying to use Node.js built-ins and modules in their Workflow code.** Usually, the best thing to do is move that code to an Activity. Some common examples that will **not** work in the Workflow isolate: Importing node built-in modules > **🚨 Danger:** > Antipattern > > ```ts > import fs from 'fs'; > > const config = fs.readFileSync('config.json', 'utf8'); > ``` > This is invalid because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed. You'll typically see an error in this form in the Webpack output: ``` 2021-10-14T19:22:00.606Z [INFO] Module not found: Error: Can't resolve 'fs' in '/Users/you/your-project/src' 2021-10-14T19:22:00.606Z [INFO] resolve 'fs' in '/Users/you/your-project/src' 2021-10-14T19:22:00.606Z [INFO] Parsed request is a module 2021-10-14T19:22:00.606Z [INFO] using description file: /Users/you/your-project/package.json (relative path: ./src) 2021-10-14T19:22:00.606Z [INFO] Field 'browser' doesn't contain a valid alias configuration ``` Importing and calling Activities directly from Workflow code > **🚨 Danger:** > Antipattern > > ```ts > import { makeHTTPRequest } from './activities'; > > export async function yourWorkflow(): Promise { > return await makeHTTPRequest('https://temporal.io'); > } > ``` > This is invalid because activity implementations should not be directly referenced by Workflow code. Activities are used by Workflows in order to make network calls and read from the filesystem, operations which are non-deterministic by nature because they rely on external state. Temporal records Activity results in the Event history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow. You'll typically see an error in this form in the Webpack output: ``` 2021-10-14T19:46:52.731Z [INFO] ERROR in ./src/activities.ts 8:31-46 2021-10-14T19:46:52.731Z [INFO] Module not found: Error: Can't resolve 'http' in '/Users/you/your-project/src' 2021-10-14T19:46:52.731Z [INFO] 2021-10-14T19:46:52.731Z [INFO] BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default. 2021-10-14T19:46:52.731Z [INFO] This is no longer the case. Verify if you need this module and configure a polyfill for it. 2021-10-14T19:46:52.731Z [INFO] 2021-10-14T19:46:52.731Z [INFO] If you want to include a polyfill, you need to: 2021-10-14T19:46:52.731Z [INFO] - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }' 2021-10-14T19:46:52.731Z [INFO] - install 'stream-http' 2021-10-14T19:46:52.731Z [INFO] If you don't want to include a polyfill, you can use an empty module like this: 2021-10-14T19:46:52.731Z [INFO] resolve.fallback: { "http": false } ``` To properly call your Activities from Workflow code use `proxyActivities` and make sure to only import the Activity types. ```ts import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { makeHTTPRequest } = proxyActivities(); export async function yourWorkflow(): Promise { return await makeHTTPRequest('https://temporal.io'); } ``` ### Works in Dev but not in Prod The two main sources of dev-prod discrepancies are in bundling and connecting. #### Production bundling You may experience your Client sending stripped names as the Workflow "Type" when scheduling a Workflow. Webpack can change the Workflow's function name to something shorter. Temporal won't know how to handle the mismatch between the shorter name and the expected Workflow type. You may experience errors like this: ``` Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request. ``` Or you may see shorter names in the Temporal Service's Web UI when Webpack changed the Workflow's function name to something shorter, in this case the single letter 's': ![Temporal Web UI showing stripped ](/img/webui/stripped_workflow_types_in_webui.png) This issue can happen when your bundler strips out Workflow function names. Temporal relies on those names to set the "Workflow Type" in the Service Web UI. To prevent the build process from shortening Workflow function names, modify the webpack configuration file ( `webpack.config.js`) to set the Boolean that retains the original names in the `TerserPlugin` configuration section. Setting the option (`keep_fnames`) to `true` prevents name stripping. **webpackterser** ```js // webpack.config.js module.exports = { optimization: { minimize: true, minimizer: [ new TerserPlugin({ terserOptions: { keep_fnames: true, // don't strip function names in production }, }), ], }, }; ``` **esbuild** ```js require('esbuild').buildSync({ entryPoints: ['app.js'], minify: true, keepNames: true, outfile: 'out.js', }); ``` See the [esbuild docs](https://esbuild.github.io/api/#keep-names) for more information. #### Connecting to Temporal Server If you are trying to connect in production and getting this: ```bash [TransportError: transport error] ``` It is a sign that something is wrong with your Cert/Key pair. Log it out and make sure it is an exact match with what is expected (often, the issue can be whitespace when injecting from your production secrets management environment). ### Resetting Workflows to deal with logical bugs You can "rewind time" using the Temporal CLI, resetting Event History to some previous point in time. You can read the Temporal CLI docs on: - [Restarting and resetting Workflows by ID](/cli) - [Resetting all Workflows by binary checksum identifier](/cli) If you need to reset programmatically, the TS SDK does not have any high level APIs for this, but you can make raw gRPC calls to [resetWorkflowExecution](https://typescript.temporal.io/api/classes/proto.temporal.api.workflowservice.v1.WorkflowService-1/#resetworkflowexecution). Resetting should only be used to deal with serious logical bugs in your code: it's not for handling transient failures, like a downstream service being unreachable. It should not be used in the course of normal application flows. ### gRPC call timeouts (context deadline exceeded) The opaque `context deadline exceeded` error comes from `gRPC`: ``` Error: 4 DEADLINE_EXCEEDED: context deadline exceeded at Object.callErrorFromStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client.js:179:52) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181) at /Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78 at processTicksAndRejections (node:internal/process/task_queues:78:11) { code: 4, details: 'context deadline exceeded', metadata: Metadata { internalRepr: Map(1) { 'content-type' => [Array] }, options: {} }, page: '/api/getBuyState' } ``` Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server. Querying a Workflow Execution whose query handler causes an error can result in the query call timing out. Some troubleshooting actions you can take: - Verify the connection from your Worker to the Temporal Server is working and doesn't have unusually high latency. - If you are running Temporal Server yourself, check your [server metrics](/self-hosted-guide/production-checklist#scaling-and-metrics) to ensure it's not overloaded. - If what's timing out is a query, check the logs of your Workers to see if they are having issues handling the query. If none of the preceding actions help you discover why timeouts are occurring, please try to produce a minimal repro and we'll be glad to help. --- # Entity pattern - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/entity-pattern > Implement the Single-Entity Design Pattern in TypeScript to manage Workflow iterations and handle Signals, ensuring efficient Workflow Execution with updates. ### Single-entity design pattern in TypeScript The following is a simple pattern that represents a single entity. It tracks the number of iterations regardless of frequency, and calls `continueAsNew` while properly handling pending updates from Signals. ```ts interface Input { /* Define your Workflow input type here */ } interface Update { /* Define your Workflow update type here */ } const MAX_ITERATIONS = 1; export async function entityWorkflow( input: Input, isNew = true, ): Promise { try { const pendingUpdates = Array(); setHandler(updateSignal, (updateCommand) => { pendingUpdates.push(updateCommand); }); if (isNew) { await setup(input); } for (let iteration = 1; iteration <= MAX_ITERATIONS; ++iteration) { // Ensure that we don't block the Workflow Execution forever waiting // for updates, which means that it will eventually Continue-As-New // even if it does not receive updates. await condition(() => pendingUpdates.length > 0, '1 day'); while (pendingUpdates.length) { const update = pendingUpdates.shift(); await runAnActivityOrChildWorkflow(update); } } } catch (err) { if (isCancellation(err)) { await CancellationScope.nonCancellable(async () => { await cleanup(); }); } throw err; } await continueAsNew(input, false); } ``` --- # Testing - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/best-practices/testing-suite > The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, unit testing, and time-skipping functionalities. The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: - **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. - **Integration:** Anything between end-to-end and unit testing. - Running Activities with mocked Context and other SDK imports (and usually network requests). - Running Workers with mock Activities, and using a Client to start Workflows. - Running Workflows with mocked SDK imports. - **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. ## Test frameworks Some SDKs have support or examples for popular test frameworks, runners, or libraries. TypeScript has sample tests for [Jest](https://jestjs.io/) and [Mocha](https://mochajs.org/). **Jest** - Minimum Jest version: `27.0.0` - [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/workflows.test.ts) - [`jest.config.js`](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/jest.config.js) (must use [`testEnvironment: 'node'`](https://jestjs.io/docs/configuration#testenvironment-string); `testEnvironment: 'jsdom'` is not supported) **Mocha** - [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/mocha/workflows.test.ts) - Test coverage library: [`@temporalio/nyc-test-coverage`](https://github.com/temporalio/sdk-typescript/tree/main/packages/nyc-test-coverage) ## Testing Activities An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. ### Run an Activity If an Activity references its context, you need to mock that context when testing in isolation. First, create a [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment). The constructor accepts an optional partial Activity [`Info`](https://typescript.temporal.io/api/interfaces/activity.Info) object in case any info fields are needed for the test. Then use [`MockActivityEnvironment.run()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#run) to run a function in an Activity [Context](https://typescript.temporal.io/api/classes/activity.Context). ```ts import { activityInfo } from '@temporalio/activity'; import { MockActivityEnvironment } from '@temporalio/testing'; import assert from 'assert'; // A function that takes two numbers and returns a promise that resolves to the sum of the two numbers // and the current attempt. async function activityFoo(a: number, b: number): Promise { return a + b + activityInfo().attempt; } // Create a MockActivityEnvironment with attempt set to 2. Run the activityFoo // function with parameters 5 and 35. Assert that the result is 42. const env = new MockActivityEnvironment({ attempt: 2 }); const result = await env.run(activityFoo, 5, 35); assert.equal(result, 42); ``` ### Listen to Heartbeats When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them. [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) is an [`EventEmitter`](https://nodejs.org/api/events.html#class-eventemitter) that emits a `heartbeat` event that you can use to listen for Heartbeats emitted by the Activity. When an Activity is run by a Worker, Heartbeats are throttled to avoid overloading the server. `MockActivityEnvironment`, however, does not throttle Heartbeats. ```ts import { heartbeat } from '@temporalio/activity'; import assert from 'assert'; async function activityFoo(): Promise { heartbeat(6); } const env = new MockActivityEnvironment(); env.on('heartbeat', (d: unknown) => { assert(d === 6); }); await env.run(activityFoo); ``` ### Cancel an Activity If an Activity is supposed to react to a Cancellation, you can test whether it reacts correctly by canceling it. [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) exposes a [`.cancel()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#cancel) method that cancels the Activity Context. ```ts import { CancelledFailure, heatbeat, sleep } from '@temporalio/activity'; import { MockActivityEnvironment } from '@temporalio/testing'; import assert from 'assert'; async function activityFoo(): Promise { heartbeat(6); // @temporalio/activity's sleep() is Cancellation-aware, which means that on Cancellation, // CancelledFailure will be thrown from it. await sleep(100); } const env = new MockActivityEnvironment(); env.on('heartbeat', (d: unknown) => { assert(d === 6); }); await assert.rejects(env.run(activityFoo), (err) => { assert.ok(err instanceof CancelledFailure); }); ``` ## Testing Workflows ### How to mock Activities Mock the Activity invocation when unit testing your Workflows. When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker. Implement only the relevant Activities for the Workflow being tested. ```ts import type * as activities from './activities'; // Creating a mock object of the activities. const mockActivities: Partial = { makeHTTPRequest: async () => '99', }; // Creating a worker with the mocked activities. const worker = await Worker.create({ activities: mockActivities, // ... }); ``` ### How to skip time Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Set up time skipping Make sure you have the `testing` package installed for the TypeScript SDK: ```bash npm install @temporalio/testing ``` The `@temporalio/testing` package downloads the test server and exports [`TestWorkflowEnvironment`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment), which you use to connect the Client and Worker to the test server and interact with the test server. [`TestWorkflowEnvironment.createTimeSkipping`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#createtimeskipping) starts the test server. A typical test suite should set up a single instance of the test environment to be reused in all tests (for example, in a [Jest](https://jestjs.io/) `beforeAll` hook or a [Mocha](https://mochajs.org/) `before()` hook). ```typescript import { TestWorkflowEnvironment } from '@temporalio/testing'; let testEnv: TestWorkflowEnvironment; // beforeAll and afterAll are injected by Jest beforeAll(async () => { testEnv = await TestWorkflowEnvironment.createTimeSkipping(); }); afterAll(async () => { await testEnv?.teardown(); }); ``` `TestWorkflowEnvironment` has [`client`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#client) and [`nativeConnection`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#nativeconnection) for creating Workers: ```typescript import { Worker } from '@temporalio/worker'; import { v4 as uuid4 } from 'uuid'; import { workflowFoo } from './workflows'; test('workflowFoo', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', ... }); const result = await worker.runUntil( testEnv.client.workflow.execute(workflowFoo, { workflowId: uuid4(), taskQueue: 'test', }) ); expect(result).toEqual('foo'); }); ``` This test uses the test connection to create a Worker, runs the Worker until the Workflow is complete, and then makes an assertion about the Workflow's result. The Workflow is executed using `testEnv.client.workflow`, which is connected to the test server. #### Skip time automatically Start a test server process that skips time as needed. For example, in the time-skipping mode, Timers, which include sleeps and conditional timeouts, are fast-forwarded except when Activities are running. The test server starts in "normal" time. When you use `TestWorkflowEnvironment.client.workflow.execute()` or `.result()`, the test server switches to "skipped" time mode until the Workflow completes. In "skipped" mode, timers (`sleep()` calls and `condition()` timeouts) are fast-forwarded except when Activities are running. `workflows.ts` ```ts import { sleep } from '@temporalio/workflow'; export async function sleeperWorkflow() { await sleep('1 day'); } ``` `test.ts` ```ts import { sleeperWorkflow } from './workflows'; test('sleep completes almost immediately', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('./workflows'), }); // Does not wait an entire day await worker.runUntil( testEnv.client.workflow.execute(sleeperWorkflow, { workflowId: uuid(), taskQueue: 'test', }), ); }); ``` #### Skip time manually You can call `testEnv.sleep()` from your test code to advance the test server's time. This is useful for testing intermediate states or indefinitely long-running Workflows. However, to use `testEnv.sleep()`, you need to avoid automatic time skipping by starting the Workflow with `.start()` instead of `.execute()` (and not calling `.result()`). `workflow.ts` ```ts import { sleep } from '@temporalio/workflow'; import { defineQuery, setHandler } from '@temporalio/workflow'; export const daysQuery = defineQuery('days'); export async function sleeperWorkflow() { let numDays = 0; setHandler(daysQuery, () => numDays); for (let i = 0; i < 100; i++) { await sleep('1 day'); numDays++; } } ``` `test.ts` ```ts test('sleeperWorkflow counts days correctly', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('./workflows'), }); // `start()` starts the test server in "normal" mode, not skipped time mode. // If you don't advance time using `testEnv.sleep()`, then `sleeperWorkflow()` // will run for days. handle = await testEnv.client.workflow.start(sleeperWorkflow, { workflowId: uuid4(), taskQueue, }); worker.run(); let numDays = await handle.query(daysQuery); assert.equal(numDays, 0); // Advance the test server's time by 25 hours await testEnv.sleep('25 hours'); numDays = await handle.query(daysQuery); assert.equal(numDays, 1); await testEnv.sleep('25 hours'); numDays = await handle.query(daysQuery); assert.equal(numDays, 2); }); ``` #### Skip time in Activities Call [`TestWorkflowEnvironment.sleep`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#sleep) from the mock Activity. In the following test, `processOrderWorkflow` sends a notification to the user after one day. The `processOrder` mocked Activity calls `testEnv.sleep(‘2 days')`, during which the Workflow sends email (by calling the `sendNotificationEmail` Activity). Then, after the Workflow completes, we assert that `sendNotificationEmail` was called. Workflow implementation [timer-examples/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/workflows.ts) ```ts export async function processOrderWorkflow({ orderProcessingMS, sendDelayedEmailTimeoutMS, }: ProcessOrderOptions): Promise { let processing = true; // Dynamically define the timeout based on given input const { processOrder } = proxyActivities>({ startToCloseTimeout: orderProcessingMS, }); const processOrderPromise = processOrder().then(() => { processing = false; }); await Promise.race([processOrderPromise, sleep(sendDelayedEmailTimeoutMS)]); if (processing) { await sendNotificationEmail(); await processOrderPromise; } return 'Order completed!'; } ``` [timer-examples/src/test/workflows.test.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/test/workflows.test.ts) ```ts it('sends reminder email if processOrder does not complete in time', async () => { // This test doesn't actually take days to complete: the TestWorkflowEnvironment starts the // Test Server, which automatically skips time when there are no running Activities. let emailSent = false; const mockActivities: ReturnType = { async processOrder() { // Test server switches to "normal" time while an Activity is executing. // Call `env.sleep` to skip ahead 2 days, by which time sendNotificationEmail // should have been called. await env.sleep('2 days'); }, async sendNotificationEmail() { emailSent = true; }, }; const worker = await Worker.create({ connection: env.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('../workflows'), activities: mockActivities, }); await worker.runUntil( env.client.workflow.execute(processOrderWorkflow, { workflowId: uuid(), taskQueue: 'test', args: [{ orderProcessingMS: ms('3 days'), sendDelayedEmailTimeoutMS: ms('1 day') }], }), ); assert.ok(emailSent); }); ``` ### Test functions in Workflow context For a function or method to run in the Workflow context (where it's possible to get the current Workflow info, or running inside the sandbox in the case of TypeScript or Python), it needs to be run by the Worker as if it were a Workflow. > **📝 Note:** > > This section is applicable in Python and TypeScript. > In Python, we allow testing of Workflows only and not generic Workflow-related code. > To test a function in your Workflow code that isn't a Workflow, put the file it's exported from in [WorkerOptions.workflowsPath](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#workflowspath). Then execute the function as if it were a Workflow: `workflows/file-with-workflow-function-to-test.ts` ```ts import { sleep } from '@temporalio/workflow'; export async function functionToTest(): Promise { await sleep('1 day'); return 42; } ``` `test.ts` ```ts const worker = await Worker.create({ connection: testEnv.nativeConnection, workflowsPath: require.resolve( './workflows/file-with-workflow-function-to-test', ), }); const result = await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions), ); assert.equal(result, 42); ``` If `functionToTest` starts a Child Workflow, that Workflow must be exported from the same file (so that the Worker knows about it): ```ts import { sleep } from '@temporalio/workflow'; import { someWorkflowToRunAsChild } from './some-workflow'; export { someWorkflowToRunAsChild }; export async function functionToTest(): Promise { const result = await wf.executeChild(someWorkflowToRunAsChild); return result + 42; } ``` ### Assert in Workflow The `assert` statement is a convenient way to insert debugging assertions into the Workflow context. The `assert` method is available in Python and TypeScript. The Node.js [`assert`](https://nodejs.org/api/assert.html) module is included in Workflow bundles. By default, a failed `assert` statement throws `AssertionError`, which causes a [Workflow Task](/tasks#workflow-task) to fail and be indefinitely retried. To prevent this behavior, use [`workflowInterceptorModules`](https://typescript.temporal.io/api/namespaces/testing/#workflowinterceptormodules) from `@temporalio/testing`. These interceptors catch an `AssertionError` and turn it into an `ApplicationFailure` that fails the entire Workflow Execution (not just the Workflow Task). `workflows/file-with-workflow-function-to-test.ts` ```ts import assert from 'assert'; export async function functionToTest() { assert.ok(false); } ``` `test.ts` ```ts import { TestWorkflowEnvironment, workflowInterceptorModules, } from '@temporalio/testing'; const worker = await Worker.create({ connection: testEnv.nativeConnection, interceptors: { workflowModules: workflowInterceptorModules, }, workflowsPath: require.resolve( './workflows/file-with-workflow-function-to-test', ), }); await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions), // throws WorkflowFailedError ); ``` ## How to Replay a Workflow Execution Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: To replay a single Event History, use [worker.runReplayHistory](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistory). When an Event History is replayed and non-determinism is detected (that is, the Workflow code is incompatible with the History), [DeterminismViolationError](https://typescript.temporal.io/api/classes/workflow.DeterminismViolationError) is thrown. If replay fails for any other reason, [ReplayError](https://typescript.temporal.io/api/classes/worker.ReplayError) is thrown. In the following example, a single Event History is loaded from a JSON file on disk (as obtained from the [Web UI](/web-ui) or the [Temporal CLI](/cli/command-reference/workflow#show)): ```ts const filePath = './history_file.json'; const history = await JSON.parse(fs.promises.readFile(filePath, 'utf8')); await Worker.runReplayHistory( { workflowsPath: require.resolve('./your/workflows'), }, history, ); ``` Alternatively, we can download the Event History programmatically using a Client: ```ts const connection = await Connection.connect({ address }); const client = new Client({ connection, namespace: 'your-namespace' }); const handle = client.workflow.getHandle('your-workflow-id'); const history = await handle.fetchHistory(); await Worker.runReplayHistory( { workflowsPath: require.resolve('./your/workflows'), }, history, ); ``` To gain confidence that changes to a Workflow are safe to deploy, we recommend that you obtain Event Histories from the relevant Task Queue and replay them in bulk. You can do so by combining the [Client.workflow.list()](https://typescript.temporal.io/api/classes/client.WorkflowClient#list) and [worker.runReplayHistories()](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistories) APIs. In the following example (which, as of server 1.18, requires [Advanced Visibility](/visibility#advanced-visibility) to be enabled), Event Histories are downloaded from the server and then replayed by passing in a client and a set of Workflows Executions. The [results](https://typescript.temporal.io/api/interfaces/worker.ReplayResult) returned by the async iterator contain information about the Workflow Execution and whether an error occurred during replay. ```ts const executions = client.workflow.list({ query: 'TaskQueue=foo and StartTime > "2022-01-01T12:00:00"', }); const histories = executions.intoHistories(); const results = Worker.runReplayHistories( { workflowsPath: require.resolve('./your/workflows'), }, histories, ); for await (const result of results) { if (result.error) { console.error('Replay failed', result); } } ``` --- # Client - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/client ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Temporal Client - [Temporal Client](/develop/typescript/client/temporal-client) - [Namespaces](/develop/typescript/client/namespaces) --- # Manage Namespaces - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/client/namespaces > Efficiently create and manage Namespaces on Temporal using CLI or SDK APIs. Isolate Workflow Executions, control access with custom Authorizers, and manage via Temporal Cloud UI or CLI. ## How to create and manage Namespaces You can create, update, deprecate or delete your [Namespaces](/namespaces) using either the Temporal CLI or SDK APIs. Use Namespaces to isolate your Workflow Executions according to your needs. For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces. You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces from the command line. On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. ### How to register Namespaces Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to create Namespaces. On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. ### How to manage Namespaces You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces. On Temporal Cloud, use the [Temporal Cloud UI](/cloud/namespaces#create-a-namespace) or [`temporal cloud namespace` commands](/cli/command-reference/cloud/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and `temporal operator namespace` commands will not work with Temporal Cloud. To manage Namespaces from the command line on Temporal Cloud, use the [Temporal Cloud extension](/cli/cloud). Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. --- # Temporal Client - Typescript SDK Source: https://docs.temporal.io/develop/typescript/client/temporal-client A [Temporal Client](/encyclopedia/temporal-client) enables you to communicate with the Temporal Service. Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and Queries to Workflow Executions, getting Workflow results, and more. You cannot initialize a Temporal Client inside a Workflow. However, they're commonly initialized inside an Activity to communicate with a Temporal Service. For [Standalone Activities](/standalone-activity), a Temporal Client can also start and manage Standalone Activities directly, without involving a Workflow. This page shows you how to do the following using the TypeScript SDK with the Temporal Client: - [Connect to a local development Temporal Service](#connect-to-development-service) - [Connect to Temporal Cloud](#connect-to-temporal-cloud) - [Connect to Temporal Service from a Worker](#connect-to-temporal-service-from-a-worker) - [Start a Workflow Execution](#start-workflow-execution) - [Get Workflow results](#get-workflow-results) In the TypeScript SDK, connecting to Temporal Service from a Temporal Application and from within an Activity rely on a different type of connection than connecting from a Worker. The sections [Connect to a local development Temporal Service](#connect-to-development-service) and [Connect to Temporal Cloud](#connect-to-temporal-cloud) apply to connecting from a Temporal Application or from within an Activity. See [Connect to Temporal Service from a Worker](#connect-to-temporal-service-from-a-worker) for details on connecting from a Worker. ## Connect to development Temporal Service To connect to a development Temporal service from a Temporal Application or from within an Activity, import the [`Connection` class](https://typescript.temporal.io/api/classes/client.Connection) from `@temporalio/client` and use [`Connection.connect`](https://typescript.temporal.io/api/classes/client.Connection#connect) to create a Connection object to connect to the Temporal Service. Then pass in that connection when you create a new `Client` instance. If you leave the connection options empty, the SDK defaults to connecting to `127.0.0.1:7233` in the `default` Namespace. ```ts import { Connection, Client } from '@temporalio/client'; async function run() { const connection = await Connection.connect(); // your code goes here const client = new Client({ connection }); } run().catch((err) => { console.error(err); process.exit(1); }); ``` If you need to connect to a Temporal Service with custom options, you can provide connection options directly in code, load them from **environment variables**, or a **TOML configuration file** using the `@temporalio/envconfig` helpers. We recommend environment variables or a configuration file for secure, repeatable configuration. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to [Environment Configuration](/develop/environment-configuration) for more details about configuration files and profiles. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. ```toml title="config.toml" # Default profile for local development [profile.default] address = "localhost:7233" namespace = "default" # Optional: Add custom gRPC headers [profile.default.grpc_meta] my-custom-header = "development-value" trace-id = "dev-trace-123" # Production profile for Temporal Cloud [profile.prod] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" # TLS configuration for production [profile.prod.tls] # TLS auto-enables when TLS config or an API key is present # disabled = false client_cert_path = "/etc/temporal/certs/client.pem" client_key_path = "/etc/temporal/certs/client.key" # Custom headers for production [profile.prod.grpc_meta] environment = "production" service-version = "v1.2.3" ``` You can create a Temporal Client using a profile from the configuration file as follows. In this example, you load the `default` profile for local development: [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { console.log('--- Loading default profile from config.toml ---'); // For this sample to be self-contained, we explicitly provide the path to // the config.toml file included in this directory. // By default though, the config.toml file will be loaded from // ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). const configFile = resolve(__dirname, '../config.toml'); // loadClientConnectConfig is a helper that loads a profile and prepares // the configuration for Connection.connect and Client. By default, it loads the // "default" profile. const config = loadClientConnectConfig({ configSource: { path: configFile }, }); console.log(`Loaded 'default' profile from ${configFile}.`); console.log(` Address: ${config.connectionOptions.address}`); console.log(` Namespace: ${config.namespace}`); console.log(` gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Environment Variables** Use the `@temporalio/envconfig` module to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. Set the following environment variables before running your application. Replace the placeholder values with your actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the `default` Namespace. You may omit these variables entirely since they're the defaults. ```bash export TEMPORAL_NAMESPACE="default" export TEMPORAL_ADDRESS="localhost:7233" ``` After setting the environment variables, use the following code to create the Temporal Client. Since the environment variables take precedence, they will override any values set in the configuration file. Therefore, you may leave `loadClientConnectConfig`'s arguments empty: [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) ```ts {7,17-18} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { // ... const config = loadClientConnectConfig({ // ... }); // ... console.log(` Address: ${config.connectionOptions.address}`); console.log(` Namespace: ${config.namespace}`); console.log(` gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Code** If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. ```ts const connection = await Connection.connect({ address: , tls: true, apiKey: , }); const client = new Client({ connection, namespace: ., }); ``` ## Connect to Temporal Cloud You can connect to Temporal Cloud using either an [API key](/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: - Your credentials for authentication. - If you are using an API key, provide the API key value. - If you are using mTLS, provide the mTLS CA certificate and mTLS private key. - Your [_Namespace Id_](/cloud/namespaces#temporal-cloud-namespace-id) and _Account Id_ combination, which follows the format `.`. - The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](/cloud/high-availability). See [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](/cloud/certificates). You can provide these connection options using environment variables, a configuration file, or directly in code. **Configuration File** You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](/references/client-environment-configuration). You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. > **ℹ️ Info:** > > The connection options set in configuration files have lower precedence than environment variables. This means that if > you set the same option in both the configuration file and as an environment variable, the environment variable value > overrides the option set in the configuration file. > For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" tls_client_cert_data = "your-tls-client-cert-data" tls_client_key_path = "your-tls-client-key-path" ``` With the connections options defined in the configuration file, use the `loadClientConnectConfig` helper from `@temporalio/envconfig` to load the `staging` profile from the configuration file. You can then pass the resulting configuration to the `Connection.connect` method. After that, you then pass the `connection` object and the Namespace to the `Client` constructor to create a Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. [env-config/src/load-profile.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-profile.ts) ```ts {15-18,30-31} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { console.log("--- Loading 'staging' profile with programmatic overrides ---"); const configFile = resolve(__dirname, '../config.toml'); const profileName = 'staging'; // The 'staging' profile in config.toml has an incorrect address (localhost:9999) // We'll programmatically override it to the correct address // Load the 'staging' profile. const config = loadClientConnectConfig({ profile: profileName, configSource: { path: configFile }, }); // Override the target host to the correct address. // This is the recommended way to override configuration values. config.connectionOptions.address = 'localhost:7233'; console.log(`\nLoaded '${profileName}' profile from ${configFile} with overrides.`); console.log(` Address: ${config.connectionOptions.address} (overridden from localhost:9999)`); console.log(` Namespace: ${config.namespace}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Environment Variables** The following environment variables are required to connect to Temporal Cloud: - `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. - `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. - `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. - `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. - `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your application. Import the `@temporalio/envconfig` package to set connection options for the Temporal Client using environment variables. The `loadClientConnectConfig` function will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](/references/client-environment-configuration). For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { // ... const config = loadClientConnectConfig({ // ... }); // ... console.log(` Address: ${config.connectionOptions.address}`); console.log(` Namespace: ${config.namespace}`); console.log(` gRPC Metadata: ${JSON.stringify(config.connectionOptions.metadata)}`); console.log('\nAttempting to connect to client...'); try { const connection = await Connection.connect(config.connectionOptions); const client = new Client({ connection, namespace: config.namespace }); console.log('✅ Client connected successfully!'); await connection.close(); } catch (err) { console.log(`❌ Failed to connect: ${err}`); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` **Code** You can also provide connections options in your code directly. To create an initial connection, provide the Namespace and API key values to the `Connection.connect` method. ```ts const connection = await Connection.connect({ address: , tls: true, apiKey: , }); const client = new Client({ connection, namespace: ., }); ``` To update an API key, use the `setApiKey` method on the Connection object: ```ts connection.setApiKey(); ``` ## Connect to Temporal Service from a Worker Connecting to Temporal Service from a Worker requires the same set of connections options as connecting from a Temporal Application or from within an Activity, but the connection type is different. When connecting from a Worker, you create a `NativeConnection` object instead of a `Connection` object. The [`NativeConnection` class](https://typescript.temporal.io/api/classes/worker.NativeConnection) is imported from `@temporalio/worker` instead of `@temporalio/client`. After you create the `NativeConnection` object, you pass it to `Worker.create()` when creating the Worker. To provide connection options to the `NativeConnection`, you can use environment variables, a configuration file, or directly in code. The following code snippets show how to create a `NativeConnection` object using each method. Refer to [Connect to a local development Temporal Service](#connect-to-development-service) and [Connect to Temporal Cloud](#connect-to-temporal-cloud) for details on how to provide connection options using each method. **Configuration File** Ensure you have a TOML configuration file with the necessary connection options defined. For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an API key: ```toml # Cloud profile for Temporal Cloud [profile.staging] address = "your-namespace.a1b2c.tmprl.cloud:7233" namespace = "your-namespace" api_key = "your-api-key-here" ``` Use the `loadClientConnectConfig` helper from `@temporalio/envconfig` to load the `staging` profile from the configuration file and create a `NativeConnection` object as follows: ```ts {1,15,17} import { NativeConnection } from '@temporalio/worker'; import { loadClientConnectConfig } from '@temporalio/envconfig'; import { resolve } from 'path'; async function main() { const configFile = resolve(__dirname, '../config.toml'); const profileName = 'staging' // Load the 'staging' profile. const config = loadClientConnectConfig({ profile: profileName, configSource: { path: configFile }, }); const connection = await NativeConnection.connect(config.connectionOptions); const worker = await Worker.create({ connection, namespace: ., // ... }); } ``` **Environment Variables** Ensure you have set the necessary environment variables to connect to Temporal Cloud. For example: ```bash export TEMPORAL_NAMESPACE="your-namespace.your-account-id" export TEMPORAL_ADDRESS="your-namespace.a1b2c.tmprl.cloud:7233" export TEMPORAL_TLS_CLIENT_CERT_PATH="/path/to/your/client/cert.pem" export TEMPORAL_TLS_CLIENT_KEY_PATH="/path/to/your/client/key.pem" ``` After setting the environment variables, use the following code to create a `NativeConnection` object using the `loadClientConnectConfig` helper from `@temporalio/envconfig`: ```ts {1,5} import { NativeConnection } from '@temporalio/worker'; import { loadClientConnectConfig } from '@temporalio/envconfig'; async function main() { const config = loadClientConnectConfig(); const connection = await NativeConnection.connect(config.connectionOptions); const worker = await Worker.create({ connection, namespace: process.env.TEMPORAL_NAMESPACE, // ... }); } ``` **Code** You can also provide connections options in your TypeScript code directly. To create an initial connection, provide the connections to the ` NativeConnection.connect` method, and then pass the resulting `NativeConnection` object to `Worker.create()` when creating the Worker: ```ts {1,4,9} import { NativeConnection } from '@temporalio/worker'; import { loadClientConnectConfig } from '@temporalio/envconfig'; const connection = await NativeConnection.connect({ address: , tls: true, apiKey: , }); const worker = await Worker.create({ connection, namespace: ., // ... }); ``` `@temporalio/worker` v1.15.0 and later support replacing a running Worker's connection, which lets you rotate an mTLS client certificate without restarting the Worker. gRPC's TLS credentials don't support dynamic certs, so instead of updating the existing `NativeConnection` you create a new one with the new certificate and assign it to `worker.connection`: ```ts import { readFileSync } from 'fs'; const newConnection = await NativeConnection.connect({ address: , tls: { clientCertPair: { crt: readFileSync('client-cert-new.pem'), key: readFileSync('client-key-new.pem'), }, }, }); worker.connection = newConnection; ``` The Worker starts using `newConnection` for subsequent calls to the Temporal Service; calls already in flight on the old connection finish normally. ## NativeConnection, Connection, and Client `NativeConnection`, `Connection`, and `Client` are all classes provided by the TypeScript SDK to facilitate communication with the Temporal Service. This section explains the differences between these classes and their respective use cases. For detailed information about each class, refer to the [Temporal TypeScript API documentation](https://typescript.temporal.io/api/namespaces/client). ### NativeConnection vs. Connection The TypeScript SDK provides two types of connection classes to connect to the Temporal Service: `NativeConnection` and `Connection`. The `NativeConnection` class is used to connect from a Worker, while the `Connection` class is used to connect from a Temporal Application or from within an Activity, typically through a `Client` object. Both connection classes accept the same set of connection options. ### Connection vs. Client A `Client` object is a high-level, lightweight abstraction that simplifies interaction with the Temporal Service. It internally manages a `Connection` object to handle the low-level communication details. The `Client` class provides convenient methods for common operations such as starting Workflow Executions, sending Signals and Queries, and retrieving Workflow results. A `Connection` object is a lower-level and expensive object that represents a direct connection to the Temporal Service. You pass in a `Connection` object to the `Client` constructor to create a `Client` instance. Since a `Connection` is expensive to create, create a single `Connection` object and reuse it across your application whenever possible. When instantiating a `Connection`, you specify most connection options except for the Namespace, such as the Temporal Service endpoint, TLS settings, and authentication credentials. When instantiating a `Client`, you provide the `Connection` object and the Namespace you want to connect to, along with other client options. ## Start Workflow Execution **How to start a Workflow Execution using the Typescript SDK** [Workflow Execution](/workflow-execution) semantics rely on several parameters—that is, to start a Workflow Execution you must supply a Task Queue that will be used for the Tasks (one that a Worker is polling), the Workflow Type, language-specific contextual data, and Workflow Function parameters. In the examples below, all Workflow Executions are started using a Temporal Client. To spawn Workflow Executions from within another Workflow Execution, use either the Child Workflow or External Workflow APIs. See the [Customize Workflow Type](/develop/typescript/workflows/basics#workflow-type) section to see how to customize the name of the Workflow Type. A request to spawn a Workflow Execution causes the Temporal Service to create the first Event ([WorkflowExecutionStarted](/references/events#workflowexecutionstarted)) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first [WorkflowTaskScheduled](/references/events#workflowtaskscheduled) Event. When you have a Client, you can schedule the start of a Workflow with `client.workflow.start()`, specifying `workflowId`, `taskQueue`, and `args` and returning a Workflow handle immediately after the Server acknowledges the receipt. ```typescript const handle = await client.workflow.start(example, { workflowId: 'your-workflow-id', taskQueue: 'your-task-queue', args: ['argument01', 'argument02', 'argument03'], // this is typechecked against workflowFn's args }); const handle = client.getHandle(workflowId); const result = await handle.result(); ``` Calling `client.workflow.start()` and `client.workflow.execute()` send a command to Temporal Server to schedule a new Workflow Execution on the specified Task Queue. It does not actually start until a Worker that has a matching Workflow Type, polling that Task Queue, picks it up. You can test this by executing a Client command without a matching Worker. Temporal Server records the command in Event History, but does not make progress with the Workflow Execution until a Worker starts polling with a matching Task Queue and Workflow Definition. Workflow Execution run in a separate V8 isolate context in order to provide a [deterministic runtime](/workflow-definition#deterministic-constraints). ### Set a Workflow's Task Queue In most SDKs, the only Workflow Option that must be set is the name of the [Task Queue](/task-queue). For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task Queue name. A Task Queue is a dynamic queue in Temporal polled by one or more Workers. Workers bundle Workflow code and node modules using Webpack v5 and execute them inside V8 isolates. Activities are directly required and run by Workers in the Node.js environment. Workers are flexible. You can host any or all of your Workflows and Activities on a Worker, and you can host multiple Workers on a single machine. The Worker needs three main things: - `taskQueue`: The Task Queue to poll. This is the only required argument. - `activities`: Optional. Imported and supplied directly to the Worker. - Workflow bundle. Choose one of the following options: - Specify `workflowsPath` pointing to your `workflows.ts` file to pass to Webpack; for example, `require.resolve('./workflows')`. Workflows are bundled with their dependencies. - If you prefer to handle the bundling yourself, pass a prebuilt bundle to `workflowBundle`. ```ts import { Worker } from '@temporalio/worker'; import * as activities from './activities'; async function run() { // Step 1: Register Workflows and Activities with the Worker and connect to // the Temporal server. const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), activities, taskQueue: 'hello-world', }); // Worker connects to localhost by default and uses console.error for logging. // Customize the Worker by passing more options to create(): // https://typescript.temporal.io/api/classes/worker.Worker // If you need to configure server connection parameters, see docs: // /typescript/security#encryption-in-transit-with-mtls // Step 2: Start accepting tasks on the `tutorial` queue await worker.run(); } run().catch((err) => { console.error(err); process.exit(1); }); ``` `taskQueue` is the only required option; however, use `workflowsPath` and `activities` to register Workflows and Activities with the Worker. When scheduling a Workflow, you must specify `taskQueue`. ```ts import { Client, Connection } from '@temporalio/client'; // This is the code that is used to start a Workflow. const connection = await Connection.create(); const client = new Client({ connection }); const result = await client.workflow.execute(yourWorkflow, { // required taskQueue: 'your-task-queue', // required workflowId: 'your-workflow-id', }); ``` When creating a Worker, you must pass the `taskQueue` option to the `Worker.create()` function. ```ts const worker = await Worker.create({ // imported elsewhere activities, taskQueue: 'your-task-queue', }); ``` Optionally, in Workflow code, when calling an Activity, you can specify the Task Queue by passing the `taskQueue` option to `proxyActivities()`, `startChild()`, or `executeChild()`. If you do not specify `taskQueue`, the TypeScript SDK places Activity and Child Workflow Tasks in the same Task Queue as the Workflow Task Queue. ### Set a Workflow Id Although it is not required, we recommend providing your own [Workflow Id](/workflow-execution/workflowid-runid#workflow-id) that maps to a business process or business entity identifier, such as an order identifier or customer identifier. Connect to a Client with `client.workflow.start()` and any arguments. Then specify your `taskQueue` and set your `workflowId` to a meaningful business identifier. ```typescript const handle = await client.workflow.start(example, { workflowId: 'yourWorkflowId', taskQueue: 'yourTaskQueue', args: ['your', 'arg', 'uments'], }); ``` This starts a new Client with the given Workflow Id, Task Queue name, and an argument. ### Get the results of a Workflow Execution If the call to start a Workflow Execution is successful, you will gain access to the Workflow Execution's Run Id. The Workflow Id, Run Id, and Namespace may be used to uniquely identify a Workflow Execution in the system and get its result. It's possible to both block progress on the result (synchronous execution) or get the result at some other point in time (asynchronous execution). In the Temporal Platform, it's also acceptable to use Queries as the preferred method for accessing the state and results of Workflow Executions. To return the results of a Workflow Execution: ```typescript return 'Completed ' + wf.workflowInfo().workflowId + ', Total Charged: ' + totalCharged; ``` `totalCharged` is just a function declared in your code. For a full example, see [subscription-workflow-project-template-typescript/src/workflows.ts](https://github.com/temporalio/subscription-workflow-project-template-typescript/blob/main/src/workflows.ts). A Workflow function may return a result. If it doesn’t (in which case the return type is `Promise`), the result will be `undefined`. If you started a Workflow with `client.workflow.start()`, you can choose to wait for the result anytime with `handle.result()`. ```typescript const handle = client.getHandle(workflowId); const result = await handle.result(); ``` Using a Workflow Handle isn't necessary with `client.workflow.execute()`. Workflows that prematurely end will throw a `WorkflowFailedError` if you call `result()`. If you call `result()` on a Workflow that prematurely ended for some reason, it throws a [`WorkflowFailedError` error](https://typescript.temporal.io/api/classes/client.WorkflowFailedError/) that reflects the reason. For that reason, it is recommended to catch that error. ```typescript const handle = client.getHandle(workflowId); try { const result = await handle.result(); } catch (err) { if (err instanceof WorkflowFailedError) { throw new Error('Temporal workflow failed: ' + workflowId, { cause: err, }); } else { throw new Error('error from Temporal workflow ' + workflowId, { cause: err, }); } } ``` --- # Install the TypeScript SDK Source: https://docs.temporal.io/develop/typescript/install-typescript-sdk > Scaffold a new TypeScript project or add Temporal packages to an existing one with npm, plus where to find the API reference. ## How to install a Temporal SDK A [Temporal SDK](/encyclopedia/architecture/temporal-sdks) provides a framework for [Temporal Application](/temporal#temporal-application) development. An SDK provides you with the following: - A [Temporal Client](/encyclopedia/temporal-client) to communicate with a [Temporal Service](/temporal-service). - APIs to develop [Workflows](/workflows). - APIs to create and manage [Worker Processes](/workers#worker). - APIs to author [Activities](/activity-definition). [![NPM](https://img.shields.io/npm/v/temporalio.svg?style=for-the-badge)](https://www.npmjs.com/search?q=author%3Atemporal-sdk-team) This project requires Node.js 18 or later. **Create a project** ```bash npx @temporalio/create@latest ./your-app ``` **Add to an existing project** ```bash npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity @temporalio/common ``` > **📝 Note:** > > The TypeScript SDK is designed with TypeScript-first developer experience in mind, but it works equally well with > JavaScript. > ### How to find the TypeScript SDK API reference The Temporal TypeScript SDK API reference is published to [typescript.temporal.io](https://typescript.temporal.io). ### Where are SDK-specific code examples? You can find a complete list of executable code samples in [Temporal's GitHub repository](https://github.com/temporalio?q=samples-&type=all&language=&sort=). Additionally, several of the [Tutorials](https://learn.temporal.io) are backed by a fully executable template application. Use the [TypeScript samples library](https://github.com/temporalio/samples-typescript) stored on GitHub to demonstrate various capabilities of Temporal. **Where can I find video demos?** [Temporal TypeScript YouTube playlist](https://www.youtube.com/playlist?list=PLl9kRkvFJrlTavecydpk9r6cF7qBmQJvb). ### How to import an ECMAScript module The JavaScript ecosystem is quickly moving toward publishing ECMAScript modules (ESM) instead of CommonJS modules. For example, `node-fetch@3` is ESM, but `node-fetch@2` is CommonJS. For more information about importing a pure ESM dependency, see our [Fetch ESM](https://github.com/temporalio/samples-typescript/tree/main/fetch-esm) sample for the necessary configuration changes: - `package.json` must have include the `"type": "module"` attribute. - `tsconfig.json` should output in `esnext` format. - Imports must include the `.js` file extension. ## Linting and types in TypeScript If you started your project with `@temporalio/create`, you already have our recommended TypeScript and ESLint configurations. If you incrementally added Temporal to an existing app, we do recommend setting up linting and types because they help catch bugs well before you ship them to production, and they improve your development feedback loop. Take a look at our recommended [.eslintrc](https://github.com/temporalio/samples-typescript/blob/main/.shared/.eslintrc.js) file and tweak to suit your needs. --- # Integrations Source: https://docs.temporal.io/develop/typescript/integrations > Integrations with other tools and services. The following integrations are available for the Temporal TypeScript SDK. - [AI SDK by Vercel](/develop/typescript/integrations/ai-sdk) — Build AI-powered applications with Durable Execution using the Vercel AI SDK. _(TypeScript · Agent framework)_ - [Braintrust](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript) — Monitor and evaluate AI application performance with Braintrust observability. _(TypeScript · Agent observability)_ - [LangSmith](/develop/typescript/integrations/langsmith) — Trace and debug LLM calls in Temporal Workflows with LangSmith. _(TypeScript · Agent observability)_ - [Mastra](https://mastra.ai/guides/deployment/temporal) — Build durable AI agents and workflows with the Mastra TypeScript framework. _(TypeScript · Agent framework)_ - [OpenAI Agents SDK](/develop/typescript/integrations/openai-agents) — Run OpenAI Agents with Durable Execution using Temporal. _(TypeScript · Agent framework)_ - [Parseable](https://github.com/parseablehq/temporal-plugin/blob/main/INTEGRATION.md) — Stream Temporal Workflow and Activity execution events to Parseable for observability and analysis. _(TypeScript · Agent observability)_ - [Strands Agents](/develop/typescript/integrations/strands-agents) — Orchestrate AWS Strands Agents with durable Temporal Workflows. _(TypeScript · Agent framework)_ --- # AI SDK by Vercel integration Source: https://docs.temporal.io/develop/typescript/integrations/ai-sdk > Implement AI applications in TypeScript using the Temporal TypeScript SDK and the AI SDK. Temporal's integration with [Vercel's AI SDK](https://ai-sdk.dev/) lets you use the AI SDK's API directly in Workflow code while Temporal handles Durable Execution. Like all API calls, LLM API calls are non-deterministic. In a [Temporal Application](/glossary#temporal-application), that means you cannot make LLM calls directly from a [Workflow](/glossary#workflow); they must run as [Activities](/glossary#activity). The AI SDK plugin handles this automatically: when you call methods in the AI SDK such as `generateText()`, `streamText()`, or `streamObject()`, the plugin wraps those calls in Activities behind the scenes. This preserves the Vercel AI SDK's developer experience that you are already familiar with while Temporal handles Durable Execution for you. > **📝 Note:** > > Import the plugin (`AiSdkPlugin`) from `@temporalio/ai-sdk` in your Worker setup, but import Workflow-side helpers such > as `temporalProvider` and `TemporalMCPClient` from the `@temporalio/ai-sdk/workflow` subpath. The `/workflow` subpath is > safe to use in the deterministic Workflow sandbox. > All code snippets in this guide are taken from the TypeScript SDK [ai-sdk samples](https://github.com/temporalio/samples-typescript/tree/main/ai-sdk). Refer to the samples for the complete code and run them locally. > **Public Preview** ## Prerequisites - This guide assumes you are already familiar with the Vercel AI SDK. If you aren't, refer to the [Vercel AI SDK documentation](https://ai-sdk.dev/) for more details. - If you are new to Temporal, we also recommend you read the [Understanding Temporal](/evaluate/understanding-temporal) document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics of Temporal. - Ensure you have set up your local development environment by following the [Set up your local with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. When you are done, leave the Temporal Development Server running if you want to test your code locally. ## Configure Workers to use the AI SDK Workers are the compute layer of a Temporal Application. They are responsible for executing the code that defines your [Workflows](/glossary#workflow) and [Activities](/glossary#activity). Before you can execute a Workflow or Activity with the Vercel AI SDK, you need to create a Worker and configure it to use the AI SDK plugin. Follow the steps below to configure your Worker. 1. Install the `@temporalio/ai-sdk` package. ```bash npm install @temporalio/ai-sdk ``` 2. Create a `worker.ts` file and configure the Worker to use the AI SDK plugin. ```ts {9-11} import { openai } from '@ai-sdk/openai'; import { AiSdkPlugin } from '@temporalio/ai-sdk'; //... other import statements, initializing a connection // to the Temporal Service to be used by the Worker const worker = await Worker.create({ plugins: [ new AiSdkPlugin({ modelProvider: openai, }), ], connection, namespace: 'default', taskQueue: 'ai-sdk', workflowsPath: require.resolve('./workflows'), activities, }); // ... code that runs the worker ``` The `modelProvider` specifies which AI provider to use when creating models. Choose the provider that best suits your needs. In the Worker options, you are also specifying that the Worker polls the `ai-sdk` Task Queue for work in the `default` Namespace. Make sure that you configure your Client application to use the same Task Queue and Namespace. 3. Run the Worker. This Worker will now poll the Temporal Service for work on the `ai-sdk` Task Queue in the `default` Namespace until you stop it. ```bash nodemon worker.ts ``` You must ensure the Worker process has access to your API credentials. Most provider SDKs read credentials from environment variables. Refer to the [Vercel AI SDK documentation](https://ai-sdk.dev/providers/ai-sdk-providers) for instructions on how to set up your environment variables for the provider you chose. > **💡 Tip:** > > You only need to give provider credentials to the Worker process. The client application, meaning the application > that sends requests to the Temporal Service to start Workflow Executions, doesn't need to know about the credentials. > See the full example at [ai-sdk samples](https://github.com/temporalio/samples-typescript/tree/main/ai-sdk). ## Develop a Simple Haiku Agent To help you get started, you can develop a simple Haiku Agent that generates haikus based on a prompt. If you weren't using Temporal, you would write code like this to generate a haiku: ```ts import { generateText } from 'ai'; import { openai } from '@ai-sdk/openai'; async function haikuAgent(prompt: string): Promise { const result = await generateText({ model: openai('gpt-4o-mini'), prompt, system: 'You only respond in haikus.', }); return result.text; } ``` To add Durable Execution to your agent, implement the agent as a Temporal Workflow. Use the AI SDK as you normally would, but pass `temporalProvider.languageModel()` as the model. The string you provide (like `'gpt-4o-mini'`) is passed to your configured `modelProvider` to create the model. ```ts {2,6} import { generateText } from 'ai'; import { temporalProvider } from '@temporalio/ai-sdk/workflow'; export async function haikuAgent(prompt: string): Promise { const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt, system: 'You only respond in haikus.', }); return result.text; } ``` With only two line changes, you have added Durable Execution to your agent. Your agent now gets automatic retries, timeouts, and the ability to run for extended periods without losing state if the process crashes. ## Provide your durable agent with tools The Vercel AI SDK lets you provide tools to your agents, and when the model calls them, they execute in the Workflow. Since tool functions run in Workflow context, they must follow Workflow rules. That means they must call Activities or Child Workflows to perform non-deterministic operations like API calls. For example, if you want to call an external API to get the weather, you would implement it as an Activity and call it from the tool function. The following is an example of an Activity that gets the weather for a given location: [ai-sdk/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/activities.ts) ```ts export async function getWeather(input: { location: string; }): Promise<{ city: string; temperatureRange: string; conditions: string }> { return { city: input.location, temperatureRange: '14-20C', conditions: 'Sunny with wind.', }; } ``` Then in your agent implementation, provide the tool to the model using the `tools` option and instruct the model to use the tool when needed. ```ts {15-23} import { proxyActivities } from '@temporalio/workflow'; import { generateText, tool } from 'ai'; import { temporalProvider } from '@temporalio/ai-sdk/workflow'; import { z } from 'zod'; const { getWeather } = proxyActivities({ startToCloseTimeout: '1 minute', }); export async function toolsAgent(question: string): Promise { const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt: question, system: 'You are a helpful agent.', tools: { getWeather: tool({ description: 'Get the weather for a given city', inputSchema: z.object({ location: z.string().describe('The location to get the weather for'), }), execute: getWeather, }), }, stopWhen: stepCountIs(5), }); return result.text; } ``` ## Integrate with Model Context Protocol (MCP) servers [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that lets AI applications connect to external tools and data sources. Calls to MCP servers, being calls to external APIs, are non-deterministic and would usually need to be implemented as Activities. The Temporal AI SDK integration handles this for you and provides a built-in implementation of a stateless MCP client that you can use inside Workflows. Follow the steps below to integrate your agent with an MCP server. 1. Create a connection to the MCP servers using the `experimental_createMCPClient` function from the `@ai-sdk/mcp` package. You can register multiple MCP servers by providing multiple factory functions in `mcpClientFactories`. ```ts import { experimental_createMCPClient as createMCPClient } from '@ai-sdk/mcp'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; const mcpClientFactories = { testServer: () => createMCPClient({ transport: new StdioClientTransport({ command: 'node', args: ['lib/mcp-server.js'], }), }), }; ``` The example uses `StdioClientTransport` as the transport mechanisms for client-server communication. Each time the Worker processes a Task that requires communication with the MCP server, it will start the server process and connect to it as required by the Task. 2. Configure the Worker to use the MCP client factories. ```ts {5} const worker = await Worker.create({ plugins: [ new AiSdkPlugin({ modelProvider: openai, mcpClientFactories }), ]}, ... ); ``` 3. In your agent Workflow, use `TemporalMCPClient` to get tools from the MCP server by referencing it by name: ```ts {4-5,9} import { TemporalMCPClient, temporalProvider } from '@temporalio/ai-sdk/workflow'; export async function mcpAgent(prompt: string): Promise { const mcpClient = new TemporalMCPClient({ name: 'testServer' }); const tools = await mcpClient.tools(); const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt, tools, system: 'You are a helpful agent, You always use your tools when needed.', stopWhen: stepCountIs(5), }); return result.text; } ``` Both listing tools and calling them run as Activities behind the scenes, giving you automatic retries, timeouts, and full observability. ## Stream model output from a Workflow The AI SDK's `streamText()` and `streamObject()` functions stream a response incrementally instead of returning it all at once. The plugin runs the model call in an Activity and publishes each delta onto a [Workflow Stream](/develop/typescript/workflows/workflow-streams) topic. External consumers can subscribe to that topic by Workflow ID to render tokens live as they arrive, while the Workflow durably reassembles the final result. > **📝 Note:** > > Inside the Workflow, the streamed result is reassembled after the Activity completes, so the Workflow's own `textStream` > / `partialObjectStream` is not incremental. Live, token-by-token deltas are delivered to **external** consumers through > a `WorkflowStreamClient`. This is inherent to Durable Execution: the Activity must run to completion before the Workflow > can observe a deterministic, replayable result. > Streaming uses the [`@temporalio/workflow-streams`](/develop/typescript/workflows/workflow-streams) package. Install it alongside the AI SDK plugin: ```bash npm install @temporalio/workflow-streams ``` To stream text, configure a provider with a `streamingTopic` (this enables streaming and names the topic that deltas are published to), host a `WorkflowStream` in your Workflow, and consume the AI SDK stream as usual: ```ts import { streamText } from 'ai'; import { TemporalProvider } from '@temporalio/ai-sdk/workflow'; import { WorkflowStream } from '@temporalio/workflow-streams/workflow'; export const STREAM_TOPIC = 'text-stream'; const streamingProvider = new TemporalProvider({ languageModel: { streamingTopic: STREAM_TOPIC }, }); ``` [ai-sdk/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/workflows.ts) ```ts export async function streamingAgent(prompt: string): Promise { // Host the WorkflowStream as the first statement of the workflow so its // publish-signal handler is registered before the streaming activity starts // publishing deltas to it. new WorkflowStream(); // A subscriber flips this once it has consumed the final delta (see below). let consumerDone = false; setHandler(consumerDoneSignal, () => { consumerDone = true; }); const result = streamText({ model: streamingProvider.languageModel('gpt-4o-mini'), prompt, system: 'You only respond in haikus.', }); // The model call runs in an activity that publishes each delta to // STREAM_TOPIC for external subscribers. Inside the workflow the deltas are // replayed after the activity completes, so this loop durably reassembles // the full text. let text = ''; for await (const delta of result.textStream) { text += delta; } // The workflow's stream log lives in memory and is discarded once the run // completes, which can race a subscriber's final poll. Rather than guess at a // fixed delay, wait for the subscriber to signal that it received the last // delta. The timeout is a fallback for when nothing is subscribed, so the run // can't hang forever. await condition(() => consumerDone, '10 seconds'); return text; } ``` The consumer subscribes to the topic by Workflow ID and renders each delta as it arrives. Each item's payload is the JSON-encoded AI SDK stream part: [ai-sdk/src/client.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/client.ts) ```ts // Subscribe to a Workflow's stream topic and render each text delta live as it // is published by the streaming activity. Each item's payload is the // JSON-encoded AI SDK stream part; `resultType: true` decodes it to raw bytes. async function renderStream(client: Client, workflowId: string, topic: string): Promise { const streamClient = WorkflowStreamClient.create(client, workflowId); for await (const item of streamClient.subscribe(topic, 0, { resultType: true })) { const part = JSON.parse(new TextDecoder().decode(item.data)); if (part.type === 'text-delta') process.stdout.write(part.delta); if (part.type === 'finish') break; } process.stdout.write('\n'); // Acknowledge receipt so the Workflow can complete without racing this final // poll against its in-memory stream log being discarded. await client.workflow.getHandle(workflowId).signal(consumerDoneSignal); } ``` ### Stream structured output `streamObject()` streams a structured object as its JSON is generated. It flows through the same model path as `streamText()`, so no extra wiring is needed beyond choosing a distinct `streamingTopic` so concurrent streams stay separable. External subscribers see the object build up incrementally, and the Workflow durably resolves the final, validated object. [ai-sdk/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/workflows.ts) ```ts export async function streamObjectAgent(prompt: string): Promise { new WorkflowStream(); const result = streamObject({ model: objectStreamingProvider.languageModel('gpt-4o-mini'), schema: z.object({ recipe: z.object({ name: z.string(), ingredients: z.array(z.object({ name: z.string(), amount: z.string() })), steps: z.array(z.string()), }), }), prompt, }); // External subscribers see the object build up incrementally via the partial // JSON deltas published to OBJECT_STREAM_TOPIC. The workflow drains the // partial stream and durably resolves the final, validated object. for await (const _partial of result.partialObjectStream) { // Draining drives the stream; the consumer renders the live partials. } const object = await result.object; await sleep('500 milliseconds'); return object.recipe.name; } ``` See the full example at [ai-sdk samples](https://github.com/temporalio/samples-typescript/tree/main/ai-sdk). --- # LangSmith integration Source: https://docs.temporal.io/develop/typescript/integrations/langsmith > Add LangSmith tracing to TypeScript Workflows using the Temporal TypeScript SDK. > **Public Preview** Temporal's LangSmith integration lets you trace AI agent Workflows in [LangSmith](https://smith.langchain.com/) alongside every LLM call, tool execution, and Temporal operation. Temporal gives your agent code [durable execution](/temporal#durable-execution). LangSmith adds the observability side, so you can inspect LLM inputs and outputs, follow a request from the Client through to the model, and compare runs over time. The `LangSmithPlugin` is what connects the two. It propagates trace context across Temporal boundaries (Workflow → Activity → Child Workflow) so that runs started on the Client nest correctly under Workflow and Activity runs on the Worker. It can also emit LangSmith runs for the Temporal operations themselves: Workflow executions, Activity executions, Signals, and Updates. All code snippets in this guide are taken from the [LangSmith samples](https://github.com/temporalio/samples-typescript/tree/main/langsmith). Refer to the samples for complete code. ## Prerequisites - This guide assumes you are already familiar with LangSmith. If you aren't, refer to the [LangSmith documentation](https://docs.smith.langchain.com/) for more details. - If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Ensure you have set up your local development environment by following the [Set up your local with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. When you are done, leave the Temporal Development Server running if you want to test your code locally. ## Configure Workers to use LangSmith Workers execute the code that defines your Workflows and Activities. To trace Workflow and Activity execution in LangSmith, add the `LangSmithPlugin` to your Worker. Follow the steps below to configure your Worker. 1. Install the `@temporalio/langsmith` package along with `langsmith`, which is a peer dependency. ```bash npm install @temporalio/langsmith langsmith ``` 2. Construct a LangSmith Client and a single shared `LangSmithPlugin`, then pass the plugin to your Worker. ```ts {16} import { NativeConnection, Worker } from '@temporalio/worker'; import { Client as LangSmithClient } from 'langsmith'; import { LangSmithPlugin } from '@temporalio/langsmith'; import * as activities from './activities'; const connection = await NativeConnection.connect({ address: 'localhost:7233' }); const langsmith = new LangSmithClient(); // reads LANGSMITH_API_KEY from the env const plugin = new LangSmithPlugin({ client: langsmith, addTemporalRuns: true }); const worker = await Worker.create({ connection, taskQueue: 'langsmith', workflowsPath: require.resolve('./workflows'), activities, plugins: [plugin], }); await worker.run(); ``` Reuse this one instance on both the Worker and Client — see [Compose with other plugins](#compose-with-other-plugins). 3. Run the Worker with LangSmith tracing enabled. Ensure the Worker process has access to your LangSmith API key. ```bash export LANGSMITH_TRACING=true export LANGSMITH_API_KEY="your-api-key" ``` Tracing is **off by default**, matching the `langsmith` library. With no tracing flag set to `true`, the plugin emits nothing. The plugin reads the same flags `langsmith` itself uses — `LANGSMITH_TRACING` (or `LANGSMITH_TRACING_V2`) and their `LANGCHAIN_` aliases. ## Configure Clients to use LangSmith In TypeScript the Client and the Worker are configured independently, so add a `LangSmithPlugin` to your `Client` too. This links client-side operations, like starting a Workflow, to the Workflows they trigger. ```ts {4} import { Connection, Client } from '@temporalio/client'; const connection = await Connection.connect(); const client = new Client({ connection, plugins: [plugin] }); ``` Wrap the call that starts your Workflow in a `traceable` so the rest of the trace nests under your own run. ```ts {3-11} import { traceable } from 'langsmith/traceable'; const pipeline = traceable( async () => { return client.workflow.execute(GreetingWorkflow, { taskQueue: 'langsmith', workflowId: 'greeting-1', args: ['hello'], }); }, { name: 'user_pipeline' } ); await pipeline(); ``` ## Trace Activities Any non-deterministic work in a Temporal Workflow (LLM calls, tool executions, database queries, external API calls, and so on) must run inside an Activity. That makes Activities an important place to add LangSmith runs. A `traceable` from `langsmith/traceable` works unchanged inside an Activity body: the run shows up in LangSmith nested under `RunActivity:` for the Activity that scheduled it. [langsmith/activity-tracing/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/activity-tracing/src/activities.ts) ```ts import { traceable } from 'langsmith/traceable'; const callModel = traceable( async (prompt: string): Promise => { return `answer to: ${prompt}`; }, { name: 'inner_llm_call' }, ); export async function answer(prompt: string): Promise { return callModel(prompt); } ``` ## Trace Workflows A `traceable` also works inside a Workflow body, and the plugin keeps it replay-safe: each run is emitted exactly once and is never duplicated when the Workflow replays its history. [langsmith/workflow-tracing/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/workflow-tracing/src/workflows.ts) ```ts import { traceable } from 'langsmith/traceable'; const extractKeyPoints = traceable(async (text: string): Promise => `points:${text}`, { name: 'extract_key_points', }); const summarize = traceable(async (points: string): Promise => `summary:${points}`, { name: 'summarize', }); export async function SummarizeWorkflow(text: string): Promise { const points = await extractKeyPoints(text); return summarize(points); } ``` > **📝 Note:** > > Inside a Workflow body, sequential `await inner(...)` nesting is exact. Under `Promise.all(...)` fan-out, or for > `traceable` calls made after an `await` in the same scope, parenting falls back to the Workflow run. This affects only > the visual shape of the trace, never Workflow history or control flow. Activity-side and client-side `traceable` are > unaffected. > ## Trace Signal and Update handlers A `traceable` inside a Signal or Update handler nests under that handler's run, following the same Workflow-body semantics. Temporal-internal Queries (`__temporal*`, `__stack_trace`) are never traced. [langsmith/message-handlers/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/message-handlers/src/workflows.ts) ```ts import { traceable } from 'langsmith/traceable'; import { allHandlersFinished, condition, defineSignal, defineUpdate, setHandler } from '@temporalio/workflow'; const classifyMessage = traceable(async (text: string): Promise => `intent:${text}`, { name: 'classify_intent', }); const draftReply = traceable(async (text: string): Promise => `reply:${text}`, { name: 'draft_reply', }); export const handleMessage = defineSignal<[string]>('handle_message'); export const composeReply = defineUpdate('compose_reply'); export const complete = defineSignal('complete'); export async function ConversationWorkflow(): Promise { const log: string[] = []; let done = false; setHandler(handleMessage, async (text: string) => { log.push(await classifyMessage(text)); }); setHandler(composeReply, async (text: string) => { const reply = await draftReply(text); log.push(reply); return reply; }); setHandler(complete, () => { done = true; }); await condition(() => done && allHandlersFinished()); return log; } ``` ## Trace multi-step agents across Activities and Child Workflows A single trace threads through a multi-step agent that calls Activities and a Child Workflow. The parent Workflow gathers facts and writes a report in Activities, then delegates review to a Child Workflow, and every run nests under the same trace. [langsmith/agent-pipeline/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/langsmith/agent-pipeline/src/workflows.ts) ```ts import { executeChild, proxyActivities, workflowInfo } from '@temporalio/workflow'; import type * as activities from './activities'; const { gatherFacts, writeReport, reviewReport } = proxyActivities({ startToCloseTimeout: '1 minute', }); export async function ReviewWorkflow(report: string): Promise { return reviewReport(report); } export async function ResearchWorkflow(topic: string): Promise { const facts = await gatherFacts(topic); const report = await writeReport(facts); return executeChild(ReviewWorkflow, { args: [report], workflowId: `${workflowInfo().workflowId}-review`, }); } ``` ## Include Temporal operations as runs By default (`addTemporalRuns: false`), the plugin only propagates LangSmith context so that your `traceable` runs nest correctly. It does not create its own runs. Set `addTemporalRuns: true` if you want first-class runs for the Temporal operations themselves: Workflow executions, Activity executions, Signals, Updates, and so on (`StartWorkflow:`, `RunWorkflow:`, `StartActivity:`, `RunActivity:`, `HandleSignal:`, `HandleUpdate:`). `Start*` and `Run*` pairs appear as siblings: the `Start*` run is emitted by the side scheduling the operation (for example, the Client), and the `Run*` run is emitted by the side executing it (for example, the Worker). With the plugin configured on both the Client and the Worker, and `addTemporalRuns: true`, a trace for a simple LLM call looks like this: ``` user_pipeline StartWorkflow:GreetingWorkflow RunWorkflow:GreetingWorkflow StartActivity:answer RunActivity:answer inner_llm_call ``` Without `addTemporalRuns` (the default), only your `traceable` runs appear. Context still propagates, so they nest correctly under the client-side run: ``` user_pipeline inner_llm_call ``` ## Plugin options | Option | Default | Meaning | | ----------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `client` | `new Client()` | The LangSmith `Client` that runs are emitted to. | | `addTemporalRuns` | `false` | Emit first-class runs for Temporal operations (`StartWorkflow:`, `RunActivity:`, `HandleSignal:`, …) in addition to your `traceable` runs. | | `projectName` | LangSmith default | Target LangSmith project for emitted runs. | | `tags` | — | Tags attached to every run the plugin emits. | | `metadata` | — | Metadata merged into every run the plugin emits. Credential-looking keys are scrubbed before emission. | The LangSmith API key is never accepted as a plugin option and never crosses a Temporal boundary. Supply a pre-constructed `Client` (which reads `LANGSMITH_API_KEY` from the process environment) or let the plugin build a default client from the environment. ## Compose with other plugins Register observability **first** (outermost) so it observes everything beneath it, then governance, then agent-framework plugins. ```ts const worker = await Worker.create({ connection, taskQueue: 'langsmith', workflowsPath: require.resolve('./workflows'), activities, plugins: [ new LangSmithPlugin({ client: langsmith }), // observability — first // new GovernancePlugin(...), // new AgentFrameworkPlugin(...), ], }); ``` The plugin de-duplicates its own instrumentation, so a Worker built from a plugin-configured Client will not double-instrument. ## Samples | Sample | Demonstrates | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | [activity-tracing](https://github.com/temporalio/samples-typescript/tree/main/langsmith/activity-tracing) | A `traceable` model call inside an Activity, nested under the Workflow and Activity runs. | | [workflow-tracing](https://github.com/temporalio/samples-typescript/tree/main/langsmith/workflow-tracing) | Replay-safe `traceable` calls in a Workflow body — emitted once, never duplicated on replay. | | [agent-pipeline](https://github.com/temporalio/samples-typescript/tree/main/langsmith/agent-pipeline) | A multi-step agent whose trace threads through Activities and a Child Workflow. | | [message-handlers](https://github.com/temporalio/samples-typescript/tree/main/langsmith/message-handlers) | `traceable` calls inside Signal and Update handlers, nested under each handler's run. | --- # OpenAI Agents SDK integration Source: https://docs.temporal.io/develop/typescript/integrations/openai-agents > Run OpenAI Agents SDK agents as durable Temporal Workflows in TypeScript, with model calls executed as Activities. Temporal's integration with the [OpenAI Agents SDK for JavaScript/TypeScript](https://openai.github.io/openai-agents-js/) lets you run agents as Temporal Workflows. Agent orchestration—the agent loop, tool selection, and handoffs—runs inside the Workflow, while model calls run as [Activities](/glossary#activity). Like with other types of API calls, in a [Temporal Application](/glossary#temporal-application), you make LLM calls in your Activities. This integration handles that for you: model calls are executed as Activities, so they retry durably and are not repeated during Workflow replay. Your agents survive Worker restarts and can run for extended periods without losing state. > **Pre-release** ## Prerequisites - This guide assumes you are already familiar with the OpenAI Agents SDK. If you aren't, refer to the [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-js/) for more details. - If you are new to Temporal, we recommend you read the [Understanding Temporal](/evaluate/understanding-temporal) document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics of Temporal. - Ensure you have set up your local development environment by following the [Set up your local with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. When you are done, leave the Temporal Development Server running if you want to test your code locally. ## Install ```bash # Or `pnpm add`/`yarn add` npm install @temporalio/openai-agents @openai/agents-core @openai/agents-openai openai ``` `@openai/agents-core`, `@openai/agents-openai`, and `openai` are peer dependencies. ### Import paths Most applications use two import paths: `@temporalio/openai-agents` in Worker and Client code, and `@temporalio/openai-agents/workflow` in Workflow code. The other subpaths are for tracing setup or manual Worker wiring. | Import path | Import from | Use for | | :----------------------------------------------- | :--------------- | :---------------------------------------------------------- | | `@temporalio/openai-agents` | Worker or Client | Plugin setup, MCP providers, model option types | | `@temporalio/openai-agents/workflow` | Workflow | Runner, Workflow-safe tools, sessions, MCP handles | | `@temporalio/openai-agents/otel` | Worker or Client | Replay-safe OpenTelemetry setup | | `@temporalio/openai-agents/workflow-interceptor` | Worker bundling | Manual `workflowInterceptorModules` wiring without a plugin | ## Create a Hello World Workflow A Temporal-backed agent needs three pieces: a Workflow that runs the agent, a Worker configured with the integration plugin, and a Client configured with the same plugin. ### Write the Workflow Use `TemporalOpenAIRunner` instead of the upstream `Runner`. The runner runs the agent loop inside the Workflow and dispatches each model call to an Activity. [openai-agents/src/basic/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/workflows.ts) ```ts export async function helloWorld(prompt: string): Promise { const agent = new Agent({ name: 'HelloAgent', instructions: 'You are a helpful assistant.' }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` `TemporalOpenAIRunner` mirrors the OpenAI Agents SDK `Runner`, with familiar options such as `maxTurns`, `context`, and `session`. A few differences apply for Workflow-safe execution: - `runConfig.model` must be a model name string. The Worker's `modelProvider` resolves it inside the model Activity. - `signal` is not supported. Use Temporal cancellation APIs, such as `CancellationScope`, to cancel Workflow work. ### Configure the Worker Register `OpenAIAgentsPlugin` on the Worker. The plugin registers the model Activity, adds the trace-propagation interceptors, installs the Workflow-bundle polyfills the OpenAI Agents SDK needs, and registers any configured MCP server providers. [openai-agents/src/basic/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/worker.ts) ```ts const worker = await Worker.create({ connection, taskQueue: 'openai-agents-basic', workflowsPath: require.resolve('./workflows'), activities, plugins: [ new OpenAIAgentsPlugin({ modelProvider: new OpenAIProvider({ apiKey }), modelParams: { useLocalActivity: true }, }), ], bundlerOptions: { webpackConfigHook: (config) => ({ ...config, resolve: { ...config.resolve, conditionNames: ['require', 'browser', 'default'], }, }), }, }); await worker.run(); ``` `modelParams` controls scheduling for the model Activity—including `startToCloseTimeout`, `retry`, and `useLocalActivity`. See `ModelActivityOptions` for the public field list. The Worker above sets `useLocalActivity: true`, which runs model calls as Local Activities to keep the event history smaller. You must ensure the Worker process has access to your model-provider credentials. Most provider SDKs read credentials from environment variables. ### Configure the Client Register the same plugin type on the Client so model parameters and tracing options propagate to new Workflows. Attach one `OpenAIAgentsPlugin` instance per Client or Connection configuration. [openai-agents/src/basic/client.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/client.ts) ```ts const connection = await Connection.connect(); const client = new Client({ connection, plugins: [new OpenAIAgentsPlugin({ modelProvider: new OpenAIProvider({ apiKey }) })], }); const taskQueue = 'openai-agents-basic'; const workflowId = 'openai-agents-' + nanoid(); ``` From there, start or execute Workflows as you normally would. The plugin does not change the Client API. ## Tools Inline function tools, hosted tools, Activity-backed tools, Nexus operation tools, and nested agent tools can all be used from a Temporal-backed agent. Any tool that performs I/O must run outside the Workflow sandbox, usually through an Activity or a Nexus Operation. ### Activity-backed tools Use `activityAsTool` for HTTP calls, database access, file system work, or other I/O. The tool name must match a registered Activity. [openai-agents/src/basic/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/workflows.ts) ```ts export async function tools(prompt: string): Promise { const weatherTool = activityAsTool( { name: 'getWeather', description: 'Get the current weather for a city.', parameters: { type: 'object', properties: { city: { type: 'string', description: 'The city name' } }, required: ['city'], additionalProperties: false, }, }, { startToCloseTimeout: '1 minute' }, ); const agent = new Agent({ name: 'WeatherAgent', instructions: 'You are a helpful weather assistant. Always use the getWeather tool to answer weather questions.', tools: [weatherTool], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` That type parameter is only used at compile time. At runtime, the Activity is invoked by name through `proxyActivities`. ### Inline and hosted tools For deterministic computation, use `tool()` from `@openai/agents-core` directly. Inline tools run in the Workflow sandbox and must not perform non-deterministic activities like, I/O or reading wall-clock time beyond Temporal's replacements. Hosted tools from `@openai/agents-openai`, such as `webSearchTool()`, run server-side through the model provider during the model Activity. [openai-agents/src/basic/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/basic/workflows.ts) ```ts export async function inlineTool(prompt: string): Promise { const addTool = tool({ name: 'add', description: 'Add two numbers together.', parameters: z.object({ a: z.number().describe('First number'), b: z.number().describe('Second number') }), execute: async ({ a, b }) => String(a + b), }); const agent = new Agent({ name: 'MathAgent', instructions: 'You are a math assistant. Use the add tool to compute sums.', tools: [addTool], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` A hosted tool is declared the same way, and the model provider runs it during the model Activity: [openai-agents/src/tools/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/tools/workflows.ts) ```ts export async function webSearch(prompt: string): Promise { const agent = new Agent({ name: 'WebSearchAgent', instructions: 'Use the web search tool to find current information, then answer concisely.', tools: [webSearchTool()], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` ### Nexus operation tools Use `nexusOperationAsTool` to expose a [Nexus](/nexus) Operation as an agent tool. The Workflow starts the Operation through a Nexus client and feeds the stringified result back to the agent. Define the service and its Operations: [openai-agents/src/nexus-tools/api.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/nexus-tools/api.ts) ```ts export interface GetWeatherInput { city: string; } export interface GetWeatherOutput { city: string; temperatureC: number; conditions: string; } export const weatherService = nexus.service('weather', { getWeather: nexus.operation(), }); ``` Then turn the Operation into a tool: [openai-agents/src/nexus-tools/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/nexus-tools/workflows.ts) ```ts export async function nexusToolWorkflow(prompt: string): Promise { const weatherTool = nexusOperationAsTool( weatherService.operations.getWeather, { name: 'getWeather', description: 'Get the current weather for a city.', parameters: { type: 'object', properties: { city: { type: 'string', description: 'The city name' } }, required: ['city'], additionalProperties: false, }, }, { service: weatherService, endpoint: WEATHER_ENDPOINT, scheduleToCloseTimeout: '1 minute' }, ); const agent = new Agent({ name: 'WeatherAgent', instructions: 'You are a weather assistant. Always use the getWeather tool to answer weather questions.', tools: [weatherTool], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` ### Nested agent tools Use `agentAsTool` to expose another `Agent` as a tool while keeping nested model calls durable: [openai-agents/src/agent-patterns/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/agent-patterns/workflows.ts) ```ts export async function agentsAsTools(prompt: string): Promise { const specialistAgent = new Agent({ name: 'SpecialistAgent', instructions: 'You are a specialist. Answer questions concisely.', }); const specialistTool = agentAsTool(specialistAgent, { toolName: 'ask_specialist', toolDescription: 'Ask the specialist agent a question and get a concise answer.', }); const orchestratorAgent = new Agent({ name: 'OrchestratorAgent', instructions: 'You orchestrate tasks. Use the ask_specialist tool to get answers, then synthesize a final response.', tools: [specialistTool], }); const runner = new TemporalOpenAIRunner(); const result = await runner.run(orchestratorAgent, prompt); return result.finalOutput ?? ''; } ``` Nested approval interruptions are not supported. If a nested run pauses for approval, the tool invocation fails with an `ApplicationFailure` of type `NestedAgentInterruption`. ## MCP servers The integration supports stateless and stateful [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) servers. Register a provider for each server on the Worker. Both kinds go in the same `mcpServerProviders` list; a stateful provider additionally takes the `NativeConnection` it should run its dedicated Worker on. [openai-agents/src/mcp/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/mcp/worker.ts) ```ts // A stateless provider reconnects per operation, so each tool call stands alone. const statelessProviders = [ new StatelessMCPServerProvider( 'filesystem', () => new MCPServerStdio({ command: 'npx', args: ['ts-node', filesystemServerPath], name: 'filesystem', }), ), new StatelessMCPServerProvider( 'streamableHttp', () => new MCPServerStreamableHttp({ url: toolsHttp.url, name: 'streamableHttp' }), ), new StatelessMCPServerProvider('sse', () => new MCPServerSSE({ url: toolsSse.url, name: 'sse' })), ]; // A stateful provider also takes the connection, which the plugin uses to run a // dedicated Worker holding the MCP session open for the life of the Workflow run. const statefulProviders = [new StatefulMCPServerProvider('memory', () => createNotesServer(), connection)]; const worker = await Worker.create({ connection, taskQueue: 'openai-agents-mcp', workflowsPath: require.resolve('./workflows'), activities, plugins: [ new OpenAIAgentsPlugin({ modelProvider: new OpenAIProvider({ apiKey }), modelParams: { useLocalActivity: true }, mcpServerProviders: [...statelessProviders, ...statefulProviders], }), ], bundlerOptions: { webpackConfigHook: (config) => ({ ...config, resolve: { ...config.resolve, conditionNames: ['require', 'browser', 'default'], }, }), }, }); ``` ### Stateless MCP servers Use stateless servers when each tool call is independent. Reference the provider name from Workflow code with `statelessMcpServer`: [openai-agents/src/mcp/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/mcp/workflows.ts) ```ts export async function filesystem(prompt: string): Promise { const agent = new Agent({ name: 'FilesystemAgent', instructions: 'You are a helpful assistant with access to a filesystem.', mcpServers: [statelessMcpServer('filesystem')], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } ``` ### Stateful MCP servers Use stateful servers when a persistent connection or session is required. The plugin starts a dedicated in-process Worker pinned to a per-run Task Queue and routes MCP operations to it. In the Workflow, call `connect()` before use and `cleanup()` in a `finally` block: [openai-agents/src/mcp/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/mcp/workflows.ts) ```ts export async function statefulMemory(prompt: string): Promise { const server = statefulMcpServer('memory'); await server.connect(); try { const agent = new Agent({ name: 'MemoryAgent', instructions: 'You are a helpful assistant with access to a persistent notes store.', mcpServers: [server], }); const result = await new TemporalOpenAIRunner().run(agent, prompt); return result.finalOutput ?? ''; } finally { await server.cleanup(); } } ``` Dedicated Worker startup and heartbeat failures surface as an `ApplicationFailure` whose type is exported as `DEDICATED_WORKER_FAILURE_TYPE`. ## Sessions and human-in-the-loop Because the agent loop runs inside a Workflow, conversation history and pending approvals must be replay safe. ### Replay-safe sessions Use `WorkflowSafeMemorySession` for conversation history. It replaces the upstream `MemorySession`, which is not replay safe because it depends on host process state. [openai-agents/src/sessions/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/sessions/workflows.ts) ```ts export async function multiTurnChat(prompts: string[]): Promise { const agent = new Agent({ name: 'ChatAgent', instructions: 'You are a helpful assistant.' }); const session = new WorkflowSafeMemorySession(); const runner = new TemporalOpenAIRunner(); const replies: string[] = []; for (const prompt of prompts) { const result = await runner.run(agent, prompt, { session }); replies.push(result.finalOutput ?? ''); } return replies; } ``` Session history lives on the Workflow heap and is rebuilt by replay within a single run. It does **not** automatically survive `continueAsNew`—a continued run starts with an empty session. To carry history across a Continue-As-New boundary, capture the items and re-seed the new run's session through the constructor's `initialItems`: [openai-agents/src/sessions/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/sessions/workflows.ts) ```ts export async function carryoverChat(input: CarryoverChatInput): Promise { const agent = new Agent({ name: 'ChatAgent', instructions: 'You are a helpful assistant.' }); const session = new WorkflowSafeMemorySession({ initialItems: input.initialItems }); const runner = new TemporalOpenAIRunner(); const accumulated = input.accumulated ?? []; const [prompt, ...remaining] = input.prompts; if (prompt === undefined) { return accumulated; } const result = await runner.run(agent, prompt, { session }); accumulated.push(result.finalOutput ?? ''); if (remaining.length === 0) { return accumulated; } const items = await session.getItems(); await continueAsNew({ prompts: remaining, initialItems: items, accumulated, }); } ``` ### Run state and approvals `TemporalOpenAIRunner.run` accepts a `RunState` as its second argument, matching the upstream runner. This supports human-approval flows that pause, wait for a Signal or Update, then Continue-As-New for as long as the approval takes. [openai-agents/src/human-approval/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/human-approval/workflows.ts) ```ts export async function approvalWorkflow(input: ApprovalInput = {}): Promise { const action = tool({ name: 'dangerousAction', description: 'Performs a dangerous action that requires human approval before execution.', parameters: { type: 'object', properties: { reason: { type: 'string', description: 'The reason for performing the dangerous action.' }, }, required: ['reason'], additionalProperties: false, } as const, needsApproval: true, execute: async (args) => `did: ${(args as { reason: string }).reason}`, }); const agent = new Agent({ name: 'Approver', instructions: "You carry out the user's request using the dangerousAction tool.", tools: [action], modelSettings: { toolChoice: 'required' }, }); const runner = new TemporalOpenAIRunner(); if (input.resumeFromRunState !== undefined) { const state = await RunState.fromString(agent, input.resumeFromRunState); for (const interruption of state.getInterruptions()) { state.approve(interruption); } const resumed = await runner.run(agent, state); return resumed.finalOutput ?? ''; } let approved = false; setHandler(approveSignal, () => { approved = true; }); const result = await runner.run(agent, 'Delete the old backup files.'); if (result.interruptions.length === 0) { return result.finalOutput ?? ''; } await condition(() => approved); await continueAsNew({ resumeFromRunState: result.state.toString() }); throw new Error('unreachable'); } ``` The agent passed to `RunState.fromString` must define the same tool names, handoff graph, and MCP servers as the run that produced the serialized state. ## Streaming `run` supports streaming with `{ stream: true }`. The streaming model Activity publishes each model event to a [Workflow Stream](/workflow-streams) topic as the model produces it, so an external client can observe a run live while it stays durable. Streaming is experimental. Set the topic name in `modelParams.streamingTopic` on the Client's `OpenAIAgentsPlugin`, not the Worker's; `run` fails with a `StreamingTopicNotConfigured` error if no topic is configured. Streaming requires the `@temporalio/workflow-streams` package: [openai-agents/src/streaming/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/openai-agents/src/streaming/workflows.ts) ```ts import { Agent } from '@openai/agents-core'; import { TemporalOpenAIRunner } from '@temporalio/openai-agents/workflow'; import { WorkflowStream } from '@temporalio/workflow-streams/workflow'; import { condition, defineSignal, setHandler } from '@temporalio/workflow'; export const streamingTopic = 'model-stream'; export const consumerDoneSignal = defineSignal('consumer-done'); export async function streamingChat(prompt: string): Promise { new WorkflowStream(); let consumerDone = false; setHandler(consumerDoneSignal, () => { consumerDone = true; }); const agent = new Agent({ name: 'StreamingAgent', instructions: 'You are a helpful assistant.' }); const result = await new TemporalOpenAIRunner().run(agent, prompt, { stream: true }); // The external client is the event consumer; the Workflow only drives the run to completion. for await (const _event of result); await result.completed; // Completing discards the stream log, racing a subscriber's final poll; the timeout covers no subscriber. await condition(() => consumerDone, '10 seconds'); return result.finalOutput ?? ''; } ``` Streaming from within the Workflow is replay-safe. Subscribe from an external client with `WorkflowStreamClient` from `@temporalio/workflow-streams/client`; see the [`@temporalio/workflow-streams`](https://github.com/temporalio/sdk-typescript/tree/main/contrib/workflow-streams) docs for the subscriber API. ## Tracing OpenAI Agents SDK tracing works across Client, Workflow, Activity, Nexus, and MCP boundaries. ### OpenAI hosted traces Enable the upstream hosted exporter before constructing the plugin, in the Worker process (not inside Workflow code): ```typescript import { OpenAITracingExporter } from '@openai/agents-openai'; import { addTraceProcessor, BatchTraceProcessor } from '@openai/agents-core'; addTraceProcessor(new BatchTraceProcessor(new OpenAITracingExporter())); ``` > **⚠️ Warning:** > > We don't recommend calling `setDefaultOpenAITracingExporter()`. If you do need to call it, be aware that it overwrites > internal state on any `OpenAIAgentsPlugin` instances you've already constructed. Set up hosted tracing with > `addTraceProcessor` instead, as shown above. > ### OpenTelemetry If you already collect traces with OpenTelemetry, the integration can emit the agent's spans through your OpenTelemetry pipeline. Model calls, tools, and orchestration then land in the same backend as the rest of your application's traces, instead of living only in the OpenAI dashboard. To turn this on, install the optional `@opentelemetry/sdk-trace-base` peer dependency: ```bash # Or `pnpm add`/`yarn add` npm install @opentelemetry/sdk-trace-base ``` Then register the tracer provider and enable OpenTelemetry instrumentation in the plugin options: ```typescript import { trace } from '@opentelemetry/api'; import { createTracerProvider } from '@temporalio/openai-agents/otel'; // NOTE: TracerProvider must be declared before plugin creation trace.setGlobalTracerProvider(createTracerProvider()); ``` Then set `useOtelInstrumentation` to `true` in the plugin's `interceptorOptions`: ```ts plugins: [ new OpenAIAgentsPlugin({ modelProvider: new OpenAIProvider({ apiKey }), modelParams: { useLocalActivity: true }, interceptorOptions: { useOtelInstrumentation: true, addTemporalSpans: true }, }), ], ``` If you need a different provider class, configure it with `TemporalIdGenerator` and mark it with `markReplaySafeTracerProvider` before registering it. ### Temporal orchestration spans Set `addTemporalSpans: true` to emit `temporal:*` agent-SDK spans for orchestration operations such as Workflow starts, Signals, Queries, Updates, Activities, child Workflows, Nexus Operations, and Continue-As-New. It sits alongside `useOtelInstrumentation` in `interceptorOptions`, as shown in the Worker above. These are agent-SDK spans, so they reach the hosted OpenAI dashboard, custom `TracingProcessor`s, and OpenTelemetry when enabled. ## Resources - [OpenAI Agents SDK samples](https://github.com/temporalio/samples-typescript/tree/main/openai-agents) — runnable examples for the patterns in this guide. - [`@temporalio/openai-agents` README](https://github.com/temporalio/sdk-typescript/blob/main/contrib/openai-agents/README.md) — the full plugin reference, including pre-built Workflow bundles and the complete feature-support matrix. - [OpenAI Agents SDK for JavaScript/TypeScript](https://openai.github.io/openai-agents-js/) - [Temporal Plugins guide](/develop/plugins-guide) — the Plugin system this integration is built on, which you can also use to build your own integrations. --- # Strands Agents integration Source: https://docs.temporal.io/develop/typescript/integrations/strands-agents > Run Strands Agents AI Workflows with Durable Execution using the Temporal TypeScript SDK and Strands plugin. Temporal's integration with [Strands Agents](https://strandsagents.com/) is an [SDK Plugin](/develop/plugins-guide) that gives your Strands agents [Durable Execution](/temporal#durable-execution) via the Temporal platform. The plugin routes model invocations, tool calls, MCP tool calls, and hooks through Temporal Activities, so every step your agent takes is recorded in Workflow history and can survive crashes, restarts, and infrastructure failures. > **ℹ️ Info:** > > The Temporal TypeScript SDK integration with Strands Agents is currently at an experimental release stage. The API may > change in future versions. > Code snippets in this guide are taken from the [Strands Agents plugin samples](https://github.com/temporalio/samples-typescript/tree/main/strands-agents). Refer to the samples for the complete code. ## Get started Install the plugin, then run a minimal Strands agent inside a Temporal Workflow. ### Prerequisites - This guide assumes you are already familiar with Strands Agents. If you are not, refer to the [Strands Agents documentation](https://strandsagents.com/) for more details. - If you are new to Temporal, read [Understanding Temporal](/evaluate/understanding-temporal) or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. - Set up your local development environment by following the [Set up your local with the TypeScript SDK](/develop/typescript/set-up-your-local-typescript) guide. Leave the Temporal development server running if you want to test your code locally. ### Install the plugin Install the Strands Agents plugin alongside the Strands Agents SDK: ```bash npm install @temporalio/strands-agents @strands-agents/sdk ``` ### Run a Strands agent with Durable Execution The following example runs a Strands agent inside a Temporal Workflow. Model calls execute as Temporal Activities, which means they get automatic retries, timeouts, and Durable Execution. If the Worker process crashes mid-conversation, Temporal replays the Workflow and resumes from the last completed Activity. **1. Define the Workflow** Create a Workflow that constructs a `TemporalAgent` and invokes it with a prompt. The `startToCloseTimeout` in `activityOptions` sets the maximum time each model call Activity can run: [strands-agents/src/workflows/hello-world.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/hello-world.ts) ```ts import { TemporalAgent } from '@temporalio/strands-agents'; export async function helloWorld(prompt: string): Promise { const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, }); const result = await agent.invoke(prompt); return result.toString(); } ``` **2. Start a Worker** Create a Worker that registers your Workflows and the `StrandsPlugin`. The plugin automatically registers the Activities that handle model calls. The same Worker serves every example in this guide; the `mcpClients` wiring is explained in [Connect to MCP servers](#connect-to-mcp-servers): [strands-agents/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/worker.ts) ```ts import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; import { McpClient } from '@strands-agents/sdk'; import { StrandsPlugin } from '@temporalio/strands-agents'; import { NativeConnection, Worker } from '@temporalio/worker'; import * as activities from './activities'; const ext = path.extname(fileURLToPath(import.meta.url)); const ECHO_SERVER = fileURLToPath(new URL(`./mcp-server${ext}`, import.meta.url)); function makeEchoClient(): McpClient { return new McpClient({ transport: new StdioClientTransport({ command: 'npx', args: ['tsx', ECHO_SERVER], }), }); } async function run() { const connection = await NativeConnection.connect({ address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233', }); try { const worker = await Worker.create({ connection, taskQueue: 'strands-agents', workflowsPath: fileURLToPath(new URL(`./workflows${ext}`, import.meta.url)), activities, // Omit `models:` so the plugin registers its default `BedrockModel` under // the name `bedrock`. To use a different provider or pin a model ID, // pass e.g. `models: { bedrock: () => new BedrockModel({ modelId: '...' }) }`. plugins: [new StrandsPlugin({ mcpClients: { echo: makeEchoClient } })], }); console.log('Worker started. Ctrl+C to exit.'); await worker.run(); } finally { await connection.close(); } } run().catch((err) => { console.error(err); process.exit(1); }); ``` **3. Run the Workflow** Start the Workflow from a separate client script. This example sends the prompt "Write a haiku about durable execution" and prints the agent's response: [strands-agents/src/hello-world.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/hello-world.ts) ```ts import { Client, Connection } from '@temporalio/client'; import { helloWorld } from './workflows'; async function run() { const connection = await Connection.connect({ address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233', }); const client = new Client({ connection }); const result = await client.workflow.execute(helloWorld, { args: ['Write a haiku about durable execution.'], taskQueue: 'strands-agents', workflowId: 'strands-hello-world', }); console.log(`Result: ${result}`); } run().catch((err) => { console.error(err); process.exit(1); }); ``` ## Build the agent Customize which model provider your agent uses, add tools that run as Activities, subscribe to lifecycle events with hooks, and connect to MCP servers. ### Choose and configure models `new StrandsPlugin({ models })` takes a mapping of `name` to factory function. Each factory is called lazily on first use (on the Worker, outside the Workflow sandbox) and the constructed model is cached for the Worker's lifetime. If you omit `models`, the plugin registers a single `BedrockModel` factory under the name `"bedrock"`, matching Strands' own implicit default. When you provide a custom `models` mapping, each `TemporalAgent` selects which factory to invoke by name with the `model` option: ```ts import { BedrockModel } from '@strands-agents/sdk/models/bedrock'; import { AnthropicModel } from '@strands-agents/sdk/models/anthropic'; import { TemporalAgent, StrandsPlugin } from '@temporalio/strands-agents'; // workflow export async function multiModelWorkflow(prompt: string): Promise { const a = new TemporalAgent({ model: 'claude', activityOptions: { startToCloseTimeout: '60 seconds' }, }); const b = new TemporalAgent({ model: 'bedrock', activityOptions: { startToCloseTimeout: '60 seconds' }, }); // ... } // worker new StrandsPlugin({ models: { claude: () => new AnthropicModel({ apiKey: '...' }), bedrock: () => new BedrockModel({}), }, }); ``` Each `TemporalAgent` carries its own Activity options (timeouts, Retry Policy, Task Queue, streaming topic) and dispatches to a shared model Activity, which resolves the model name against the registered factories at runtime. A model name not present in the `models` mapping throws inside the Activity. ### Run non-deterministic tools as Activities Strands tools that perform I/O, access external services, or produce non-deterministic results need to run as Temporal Activities rather than inline in the Workflow. Register the tool as an Activity on the Worker, and pass it to the agent using `workflow.activityAsTool`. Deterministic tools can run directly in the Workflow as a plain Strands `tool()`. Define an Activity for the tool: [strands-agents/src/activities/tools.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/tools.ts) ```ts export async function fetchWeather(input: { city: string; }): Promise<{ city: string; temperatureF: number; conditions: string }> { return { city: input.city, temperatureF: 72, conditions: 'sunny', }; } ``` Pass the Activity to the agent in the Workflow using `workflow.activityAsTool` (imported here as `strandsWorkflow`). The `inputSchema` is a JSON Schema (or a Zod schema) that tells the model how to call the tool: [strands-agents/src/workflows/tools.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/tools.ts) ```ts import { tool } from '@strands-agents/sdk'; import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents'; import { z } from 'zod'; const letterCounter = tool({ name: 'letterCounter', description: 'Count how many times `letter` appears in `word` (case-insensitive).', inputSchema: z.object({ word: z.string(), letter: z.string(), }), callback: ({ word, letter }) => word.toLowerCase().split(letter.toLowerCase()).length - 1, }); export async function toolsWorkflow(prompt: string): Promise { const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, tools: [ letterCounter, strandsWorkflow.activityAsTool('fetchWeather', { description: 'Fetch the current weather for a city.', inputSchema: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'], }, activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } }, }), ], }); const result = await agent.invoke(prompt); return result.toString(); } ``` Register the Activity functions on the Worker by passing them in the `activities` option, as shown in [Start a Worker](#run-a-strands-agent-with-durable-execution). The `activityName` passed to `activityAsTool` must match the name the Activity is registered under. ### React to agent lifecycle events Strands' [hook system](https://strandsagents.com/) lets you subscribe callbacks to events in the agent lifecycle, such as invocation start/end, model call before/after, tool call before/after, and message added. Use hooks to add logging, metrics, or custom logic at each stage. Register callbacks with `agent.addHook(EventClass, callback)`. Hook callbacks fire in Workflow context, so deterministic callbacks work without any extra setup. For callbacks that need I/O (writing audit logs, metrics, alerting), use `workflow.activityAsHook` to dispatch the work as a Temporal Activity. The following example shows both patterns. The first callback mutates Workflow state (deterministic), while `persistToolCall` runs as an Activity (I/O-safe): [strands-agents/src/workflows/hooks.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/hooks.ts) ```ts import { AfterToolCallEvent, tool } from '@strands-agents/sdk'; import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents'; import { z } from 'zod'; const echo = tool({ name: 'echo', description: 'Echo back the input text.', inputSchema: z.object({ text: z.string() }), callback: ({ text }) => text, }); export async function hooksWorkflow(prompt: string): Promise { const fired: string[] = []; const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, tools: [echo], }); // Callback 1: in-workflow, deterministic state mutation. agent.addHook(AfterToolCallEvent, (event) => { fired.push(event.toolUse.name); }); // Callback 2: dispatch to a Temporal activity for I/O. agent.addHook( AfterToolCallEvent, strandsWorkflow.activityAsHook('persistToolCall', { activityInput: (event) => event.toolUse.name, activityOptions: { startToCloseTimeout: '15 seconds', retry: { maximumAttempts: 3 } }, }), ); await agent.invoke(prompt); return fired; } ``` The Activity dispatched by the hook is a normal Temporal Activity registered on the Worker: [strands-agents/src/activities/hooks.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/hooks.ts) ```ts import { log } from '@temporalio/activity'; export async function persistToolCall(toolName: string): Promise { // In production, write to a database / S3 / your audit pipeline. log.info(`audit: tool ${toolName} completed`); } ``` > **⚠️ Caution:** > > Hook callbacks run in Workflow context, so they must be > [deterministic](/develop/typescript/workflows/basics#workflow-logic-requirements). Do not use `Date.now()`, > `randomUUID()`, or I/O inside hook callbacks. Use `workflow.activityAsHook` for anything that requires I/O. > The `activityInput` function extracts serializable values from the event to pass as the Activity's input. This is needed because hook events hold references to the `Agent`, `Tool` instances, and other objects that cannot cross the Activity boundary. ### Connect to MCP servers If your agent needs access to tools provided by an [MCP](https://modelcontextprotocol.io/) server, configure the MCP clients on the Worker and reference them by name in the Workflow. `new StrandsPlugin({ mcpClients })` takes a mapping of `name` to `McpClient` factory, mirroring the `models` pattern. The plugin registers per-server `{name}-listTools` and `{name}-callTool` Activities. In the Workflow, `new TemporalMCPClient({ server: 'name' })` is a thin handle that references the server by name and carries the per-call Activity options. Define the Workflow with a `TemporalMCPClient`: [strands-agents/src/workflows/mcp.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/mcp.ts) ```ts import { TemporalAgent, TemporalMCPClient } from '@temporalio/strands-agents'; export async function mcpWorkflow(prompt: string): Promise { const echo = new TemporalMCPClient({ server: 'echo', activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } }, }); const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, tools: [echo], }); const result = await agent.invoke(prompt); return result.toString(); } ``` Register the MCP client factory on the Worker via `mcpClients`, as shown in the [Worker](#run-a-strands-agent-with-durable-execution) above. Each factory returns a fully configured `McpClient`, so you can pass any options the `McpClient` constructor accepts (transport, URL, headers, and so on). By default, `TemporalMCPClient` re-lists the server's tools on every agent turn, so an MCP server that is restarted or redeployed mid-Workflow — with tools added, removed, or renamed — is picked up. To list the tools just once at the beginning of the Workflow and reuse that schema for the Workflow's lifetime (one fewer Activity per turn), set `cacheTools: true`: ```ts const echo = new TemporalMCPClient({ server: 'echo', cacheTools: true, activityOptions: { startToCloseTimeout: '30 seconds' }, }); ``` To amortize connection setup, the `{name}-listTools` and `{name}-callTool` Activities share one Worker-process MCP connection and reuse it across calls. The connection is disconnected after it sits idle for `mcpConnectionIdleTimeout` (default 5 minutes); the Timer resets on every reuse. `mcpConnectionIdleTimeout` accepts a millisecond number or a duration string (such as `'30 seconds'`), like `startToCloseTimeout`: ```ts new StrandsPlugin({ mcpClients: { echo: () => new McpClient({ url: 'http://localhost:8765/mcp' }) }, mcpConnectionIdleTimeout: '30 seconds', }); ``` ## Interact with the agent Control the shape of agent responses, stream output in real time, and pause the agent for human approval. ### Add human approval gates Some agent actions, such as deleting resources or sending messages, may require human approval before proceeding. Strands offers two ways to interrupt an agent and wait for a response. Both work with the plugin. In each case, `agent.invoke()` returns an `AgentResult` with `stopReason: 'interrupt'` and an `interrupts` array instead of throwing. Pair this with a Signal handler that supplies responses, then resume by calling `agent.invoke(responses)`. #### Interrupt from a hook A hook on an interruptible event such as `BeforeToolCallEvent` can pause the agent by calling `event.interrupt(...)`. The hook runs in Workflow context, so it must be deterministic. The Workflow waits for a Signal carrying the approval response, then resumes the agent: [strands-agents/src/workflows/human-in-the-loop.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/human-in-the-loop.ts) ```ts import { BeforeToolCallEvent, tool, type InterruptResponseContent, type InterruptResponseContentData, } from '@strands-agents/sdk'; import { TemporalAgent } from '@temporalio/strands-agents'; import { condition, defineQuery, defineSignal, setHandler } from '@temporalio/workflow'; import { z } from 'zod'; export const hitlApproveSignal = defineSignal<[string]>('hitlApprove'); export const hitlPendingApprovalQuery = defineQuery('hitlPendingApproval'); const deleteFile = tool({ name: 'deleteFile', description: 'Delete a file at the given path.', inputSchema: z.object({ path: z.string() }), callback: ({ path }) => `deleted ${path}`, }); export async function humanInTheLoop(prompt: string): Promise { let approval: string | null = null; let pendingReason: string | null = null; setHandler(hitlApproveSignal, (response) => { approval = response; }); setHandler(hitlPendingApprovalQuery, () => pendingReason); const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, tools: [deleteFile], }); agent.addHook(BeforeToolCallEvent, (event) => { if (event.toolUse.name !== 'deleteFile') return; const path = (event.toolUse.input as { path?: string }).path; const response = event.interrupt({ name: 'approval', reason: `approve delete of ${path}?`, }); if (response !== 'approve') { event.cancel = 'denied'; } }); let result = await agent.invoke(prompt); while (result.stopReason === 'interrupt') { const interrupts = result.interrupts ?? []; pendingReason = (interrupts[0]?.reason as string | undefined) ?? null; await condition(() => approval !== null); const response = approval!; approval = null; pendingReason = null; const responses: InterruptResponseContentData[] = interrupts.map((i) => ({ type: 'interruptResponse', interruptResponse: { interruptId: i.id, response }, })); result = await agent.invoke(responses as InterruptResponseContent[]); } return result.toString(); } ``` #### Interrupt from an Activity tool An `activityAsTool`-wrapped Activity can interrupt the agent by throwing an interrupt-shaped `ApplicationFailure`. The plugin's Failure Converter preserves the interrupt payload across the Activity boundary, so `AgentResult.interrupts` is populated the same way as for hooks. Define the Activity that raises the interrupt with the `STRANDS_INTERRUPT_TYPE` failure type: [strands-agents/src/activities/activity-interrupt.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/activities/activity-interrupt.ts) ```ts import { ApplicationFailure } from '@temporalio/common'; import { STRANDS_INTERRUPT_TYPE } from '@temporalio/strands-agents'; const APPROVED = new Set(); export async function deleteThing(input: { name: string }): Promise { if (!APPROVED.has(input.name)) { // First attempt: mark the name as approved on the way out (simulating the // human flipping a flag during the interrupt pause) and stop the agent by // raising an interrupt-shaped failure. The plugin's `StrandsFailureConverter` // would also recognize a thrown `{ interrupts: [{ toJSON: () => ... }] }`, // but throwing `ApplicationFailure` directly avoids any chance of the // converter being skipped (and keeps `nonRetryable: true` so the workflow // sees the interrupt instead of a retry-then-success). APPROVED.add(input.name); throw ApplicationFailure.create({ message: 'interrupt:approval', type: STRANDS_INTERRUPT_TYPE, nonRetryable: true, details: [ { id: `delete:${input.name}`, name: 'approval', reason: `approve delete of protected resource '${input.name}'?`, source: 'tool', }, ], }); } return `deleted ${input.name}`; } ``` The Workflow resumes the agent the same way as for a hook interrupt: [strands-agents/src/workflows/activity-interrupt.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/activity-interrupt.ts) ```ts import type { InterruptResponseContent, InterruptResponseContentData } from '@strands-agents/sdk'; import { TemporalAgent, workflow as strandsWorkflow } from '@temporalio/strands-agents'; import { condition, defineQuery, defineSignal, setHandler } from '@temporalio/workflow'; export const activityInterruptApproveSignal = defineSignal<[string]>('activityInterruptApprove'); export const activityInterruptPendingApprovalQuery = defineQuery('activityInterruptPendingApproval'); export async function activityInterrupt(prompt: string): Promise { let approval: string | null = null; let pendingReason: string | null = null; setHandler(activityInterruptApproveSignal, (response) => { approval = response; }); setHandler(activityInterruptPendingApprovalQuery, () => pendingReason); const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, tools: [ strandsWorkflow.activityAsTool('deleteThing', { description: 'Delete a thing by name.', inputSchema: { type: 'object', properties: { name: { type: 'string' } }, required: ['name'], }, activityOptions: { startToCloseTimeout: '30 seconds', retry: { maximumAttempts: 3 } }, }), ], }); let result = await agent.invoke(prompt); while (result.stopReason === 'interrupt') { const interrupts = result.interrupts ?? []; pendingReason = (interrupts[0]?.reason as string | undefined) ?? null; await condition(() => approval !== null); const response = approval!; approval = null; pendingReason = null; const responses: InterruptResponseContentData[] = interrupts.map((i) => ({ type: 'interruptResponse', interruptResponse: { interruptId: i.id, response }, })); result = await agent.invoke(responses as InterruptResponseContent[]); } return result.toString(); } ``` > **⚠️ Caution:** > > Activity-tool interrupts rely on the plugin's Failure Converter, which is installed via the client's Data Converter. > Attach `StrandsPlugin` to the **client** (not just the Worker) for Activity-tool interrupts to work. Workers built from > that client pick up the plugin automatically. > > ```ts > const client = new Client({ connection, plugins: [new StrandsPlugin({ models })] }); > ``` > ### Return structured data from an agent To have the agent return a typed object instead of free-form text, pass a `structuredOutputSchema` (any Zod schema) to `TemporalAgent`. The values flow through the model Activity unchanged, and the parsed object is available on `result.structuredOutput`: [strands-agents/src/workflows/structured-output.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/structured-output.ts) ```ts import { TemporalAgent } from '@temporalio/strands-agents'; import { z } from 'zod'; export const PersonInfo = z.object({ name: z.string().describe('Name of the person'), age: z.number().describe('Age of the person'), occupation: z.string().describe('Occupation of the person'), }); export type PersonInfo = z.infer; export async function structuredOutputWorkflow(prompt: string): Promise { const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, structuredOutputSchema: PersonInfo, }); const result = await agent.invoke(prompt); return result.structuredOutput as PersonInfo; } ``` ### Stream agent output to clients For long-running agent calls, you may want to forward model output chunks to an external consumer as they arrive rather than waiting for the full response. Pass `streamingTopic: '...'` to `TemporalAgent` and host a `WorkflowStream` on the Workflow via [`@temporalio/workflow-streams`](https://github.com/temporalio/sdk-typescript/tree/main/packages/workflow-streams). Each model stream event is published on the named topic from inside the model Activity. Subscribers read events through `WorkflowStreamClient`. Chunks are batched on `streamingBatchInterval` (default `'100 milliseconds'`). Define the Workflow with a `WorkflowStream` and a streaming topic: [strands-agents/src/workflows/streaming.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/streaming.ts) ```ts import { TemporalAgent } from '@temporalio/strands-agents'; import { WorkflowStream } from '@temporalio/workflow-streams/workflow'; export async function streamingWorkflow(prompt: string): Promise { // Constructing the stream installs the publish/poll handlers that // WorkflowStreamClient calls. Nothing in the workflow body reads from it. void new WorkflowStream(); const agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, streamingTopic: 'events', }); const result = await agent.invoke(prompt); return result.toString(); } ``` Subscribe to the stream from a client: [strands-agents/src/streaming.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/streaming.ts) ```ts import { Client, Connection } from '@temporalio/client'; import { WorkflowStreamClient } from '@temporalio/workflow-streams/client'; import { streamingWorkflow } from './workflows'; interface StreamEvent { type?: string; delta?: { type?: string; text?: string }; } async function run() { const connection = await Connection.connect({ address: process.env.TEMPORAL_ADDRESS ?? 'localhost:7233', }); const client = new Client({ connection }); const workflowId = 'strands-streaming'; const handle = await client.workflow.start(streamingWorkflow, { args: ['Count from 1 to 5, one number per sentence.'], taskQueue: 'strands-agents', workflowId, }); const stream = WorkflowStreamClient.create(client, workflowId); const consume = (async () => { for await (const item of stream.subscribe(['events'], 0, { pollCooldown: '50 milliseconds', resultType: true, })) { const event = item.data; if (event.type === 'modelContentBlockDeltaEvent' && event.delta?.type === 'textDelta' && event.delta.text) { process.stdout.write(event.delta.text); } else if (event.type === 'modelMessageStopEvent') { process.stdout.write('\n'); return; } } })(); const result = await handle.result(); await consume; console.log(`Final result: ${result}`); } run().catch((err) => { console.error(err); process.exit(1); }); ``` ## Run in production Configure retry policies, handle long-running chat sessions, and add distributed tracing. ### Configure retries `TemporalAgent` disables Strands' built-in `ModelRetryStrategy` so that retries are handled exclusively by Temporal. Configure retries with `activityOptions.retry` on `TemporalAgent` for model calls, and on the Activity options accepted by `workflow.activityAsTool`, `workflow.activityAsHook`, and `TemporalMCPClient` for their respective calls: ```ts new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 }, }, }); ``` Passing `retryStrategy` to `new TemporalAgent(...)` throws. Remove the argument (or pass `retryStrategy: null`) and use `activityOptions.retry` instead. ### Handle long-running chat sessions A chat-style Workflow accumulates message history with every turn. Over a long session, the Workflow's Event History can grow large enough to hit Temporal's per-Workflow history limit. To avoid this, use [Continue-as-New](/develop/typescript/workflows/continue-as-new) to start a fresh Workflow execution while carrying the agent's message history forward as input. In this example, each user turn arrives as a Workflow [Update](/develop/typescript/workflows/message-passing#updates), so the caller gets the agent's reply back from the same call. `workflowInfo().continueAsNewSuggested` flips to `true` once the server decides history has grown large enough; the Workflow checks it after each turn and hands off to a fresh run, carrying `agent.messages` as input: [strands-agents/src/workflows/continue-as-new.ts](https://github.com/temporalio/samples-typescript/blob/main/strands-agents/src/workflows/continue-as-new.ts) ```ts import type { Message } from '@strands-agents/sdk'; import { TemporalAgent } from '@temporalio/strands-agents'; import { allHandlersFinished, condition, continueAsNew, defineQuery, defineSignal, defineUpdate, setHandler, workflowInfo, } from '@temporalio/workflow'; export interface ChatInput { messages?: Message[]; } export const chatTurn = defineUpdate('turn'); export const chatEnd = defineSignal('endChat'); export const chatMessages = defineQuery('messages'); export async function chatWorkflow(input: ChatInput = {}): Promise { let done = false; let agent: TemporalAgent | null = null; // Serialize concurrent `turn` updates so they can't interleave on `agent.messages`. let pending: Promise = Promise.resolve(); setHandler(chatTurn, async (prompt) => { await condition(() => agent !== null); const prev = pending; let release!: () => void; pending = new Promise((resolve) => { release = resolve; }); try { await prev; const result = await agent!.invoke(prompt); return result.toString().trim(); } finally { release(); } }); setHandler(chatEnd, () => { done = true; }); setHandler(chatMessages, () => (agent ? [...agent.messages] : [])); agent = new TemporalAgent({ activityOptions: { startToCloseTimeout: '60 seconds', retry: { maximumAttempts: 3 } }, messages: input.messages ?? [], }); await condition(() => done || workflowInfo().continueAsNewSuggested); // Drain in-flight `turn` updates before exiting or handing off. await condition(allHandlersFinished); if (!done) { await continueAsNew({ messages: agent.messages }); } } ``` ### Add tracing with OpenTelemetry To get distributed traces across model, tool, and MCP Activities, combine `StrandsPlugin` with the [OpenTelemetry plugin](https://github.com/temporalio/sdk-typescript/tree/main/packages/interceptors-opentelemetry). Register `OpenTelemetryPlugin` on both the client and the Worker. You get OpenTelemetry spans around the model, tool, and MCP Activities the plugin schedules, plus any spans Strands itself emits inside `invoke`: ```ts import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; import { Resource } from '@opentelemetry/resources'; import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { StrandsPlugin } from '@temporalio/strands-agents'; const otel = new OpenTelemetryPlugin({ resource: new Resource({ 'service.name': 'strands-worker' }), spanProcessor: new SimpleSpanProcessor(new OTLPTraceExporter()), }); // client const client = new Client({ connection, plugins: [otel] }); // worker const worker = await Worker.create({ connection, taskQueue: 'strands-agents', workflowsPath: require.resolve('./workflows'), plugins: [otel, new StrandsPlugin({ models })], }); ``` ### Snapshots are not supported `TemporalAgent.takeSnapshot()` and `TemporalAgent.loadSnapshot()` throw. Temporal's Event History already persists Workflow state durably at a finer granularity than Strands snapshots, so snapshots are redundant inside a Workflow. ### Samples The [Strands Agents plugin samples](https://github.com/temporalio/samples-typescript/tree/main/strands-agents) demonstrate all supported patterns end-to-end. --- # Nexus - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/nexus ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Temporal Nexus - [Quickstart](/develop/typescript/nexus/quickstart) - [Feature guide](/develop/typescript/nexus/feature-guide) - [Standalone Operations](/develop/typescript/nexus/standalone-operations) --- # Feature guide - TypeScript SDK feature guide Source: https://docs.temporal.io/develop/typescript/nexus/feature-guide > Use Temporal Nexus within the TypeScript SDK to connect Durable Executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Use [Temporal Nexus](/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. > **💡 Tip:** > > New to Nexus? Start with the [Nexus TypeScript Quickstart](/develop/typescript/nexus/quickstart). > This page shows how to do the following: - [Run a development Temporal Service with Nexus enabled](#run-the-temporal-nexus-development-server) - [Create caller and handler Namespaces](#create-caller-handler-namespaces) - [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint) - [Define the Nexus Service contract](#define-nexus-service-contract) - [Develop a Nexus Service and Operation handlers](#develop-nexus-service-operation-handlers) - [Develop a caller Workflow that uses a Nexus Service](#develop-caller-workflow-nexus-service) - [Understand exceptions in Nexus Operations](#exceptions-in-nexus-operations) - [Cancel a Nexus Operation](#canceling-a-nexus-operation) - [Make Nexus calls across Namespaces in Temporal Cloud](#nexus-calls-across-namespaces-temporal-cloud)
> **📝 Note:** > > This documentation uses source code derived from the [TypeScript Nexus sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-hello). > ## Run the Temporal Development Server with Nexus enabled Prerequisites: - [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/typescript/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) (`v1.3.0` or higher recommended) - [Install the latest Temporal TypeScript SDK](https://learn.temporal.io/getting_started/typescript/dev_environment/#add-temporal-typescript-sdk-dependencies) (`v1.12.3` or higher) The first step in working with Temporal Nexus involves starting a Temporal Server with Nexus enabled. ``` temporal server start-dev ``` This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233), and the Temporal Server should now be available for client connections on `localhost:7233`. ## Create caller and handler Namespaces Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. ``` temporal operator namespace create --namespace my-target-namespace temporal operator namespace create --namespace my-caller-namespace ``` For this example, `my-target-namespace` will contain the Nexus Operation handler, and you will use a Workflow in `my-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. ## Create a Nexus Endpoint to route requests from caller to handler After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. ``` temporal operator nexus endpoint create \ --name my-nexus-endpoint-name \ --target-namespace my-target-namespace \ --target-task-queue my-handler-task-queue ``` You can also use the Web UI to create the Namespaces and Nexus endpoint. ## Define the Nexus Service contract Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](/dataconversion). The default data converter encodes payloads in the following order: Null, Byte array, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, JSON is a common choice. This example uses plain TypeScript objects, serialized into JSON. Note: By default, the TypeScript SDK [does not support Protobuf JSON encoding](https://typescript.temporal.io/api/interfaces/common.PayloadConverter). If passing Protobuf payloads use the [ProtobufJsonPayloadConverter](https://typescript.temporal.io/api/classes/protobufs.ProtobufJsonPayloadConverter) instead. [nexus-hello/src/api.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/api.ts) ```ts import * as nexus from 'nexus-rpc'; export const helloService = nexus.service('hello', { /** * Return the input message, unmodified. In the present sample, this Operation * will be implemented using the Synchronous Nexus Operation handler syntax. */ echo: nexus.operation(), /** * Return a salutation message, in the requested language. In the present sample, * this Operation will be implemented by starting the `helloWorkflow` Workflow. */ hello: nexus.operation(), }); export interface EchoInput { message: string; } export interface EchoOutput { message: string; } export interface HelloInput { name: string; language: LanguageCode; } export interface HelloOutput { message: string; } export type LanguageCode = 'en' | 'fr' | 'de' | 'es' | 'tr'; ``` ## Develop a Nexus Service handler and Operation handlers A Nexus Service handler is defined using the `nexus-rpc`'s [`serviceHandler`](https://nexus-rpc.github.io/sdk-typescript/functions/serviceHandler.html) function. Nexus Service handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. A Service handler must provide Operation handlers for each Operation declared by the Service. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Use a synchronous Nexus Operation only when its complete execution path is highly reliable, has predictably low latency, and finishes well within the [10-second handler deadline](/cloud/limits#nexus-operation-request-timeout). Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database. Handlers should be reliable since the [circuit breaker](/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint. The `@temporalio/nexus` package provides utilities to help create Nexus Operations that interact with a Temporal namespace: - `WorkflowRunOperationHandler` - Create an asynchronous operation handler that starts a Workflow. - `getClient()` - Get a Temporal Client connected using the same `NativeConnection` as the present Temporal Worker. It can be used to implement synchronous handlers backed by Temporal primitives such as Signals and Queries. ### Develop a Synchronous Nexus Operation handler Simple RPC handlers can be implemented as synchronous Nexus Operation handlers, which is defined in TypeScript as a simple async function. Use `getClient()` from `@temporalio/nexus` to get the Temporal Client for signaling, querying, and listing Workflows. Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](/nexus/operations#circuit-breaking). [nexus-hello/src/service/handler.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/handler.ts) ```ts import * as nexus from 'nexus-rpc'; // ... import { helloService, EchoInput, EchoOutput, HelloInput, HelloOutput } from '../api'; // ... export const helloServiceHandler = nexus.serviceHandler(helloService, { echo: async (ctx, input: EchoInput): Promise => { // A simple async function can be used to defined a Synchronous Nexus Operation. // This is often sufficient for Operations that simply make arbitrary short calls to // other services or databases, or that perform simple computations such as this one. // // You may also access a Temporal Client by calling `temporalNexus.getClient()`. // That Client can be used to make arbitrary calls, such as signaling, querying, // or listing workflows. return input; }, // ... }); ``` ### Use the Temporal Client for Signals, Queries, and Updates A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](/cloud/limits#nexus-operation-request-timeout). The handler receives an AbortSignal via `ctx.abortSignal` that is triggered when the deadline is exceeded — pass it to Temporal Client calls to ensure they are canceled if the timeout is reached. Updates should be short-lived to stay within this deadline. The handler context also exposes `ctx.requestDeadline` as an optional `Date`, representing the time by which the current request must complete. Note that this is the deadline for the current _request_, not the overall operation. Use it to make decisions about whether to start work that may not finish in time, or to set timeouts on downstream calls. The [nexus_messaging](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging) sample shows how to create a Nexus Service that uses synchronous operations to send Updates and Queries. Use the Nexus library, as shown below, to get the Client that the Worker was initialized with. In this example, the Workflow Id is derived from the client Id, with the "workflowIdForUser" method. This converts a given client Id (in this case, the client is passing in a user ID) into a Workflow Id. This way the client only needs the identifier it cares about. [nexus-messaging/src/callerpattern/service/handler.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-messaging/src/callerpattern/service/handler.ts) ```ts import * as temporalNexus from '@temporalio/nexus'; function workflowIdForUser(userId: string): string { return `GreetingWorkflow_for_${userId}`; } export const nexusGreetingServiceHandler = nexus.serviceHandler(nexusGreetingService, { getLanguages: async (ctx, input: GetLanguagesInput) => { const client = temporalNexus.getClient(); const handle = client.workflow.getHandle(workflowIdForUser(input.userId)); return await handle.query(getLanguagesQuery); }, ... ``` There are two examples of messaging through Nexus in the sample code, [caller pattern](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging/src/callerpattern) and [on-demand pattern](https://github.com/temporalio/samples-typescript/tree/main/nexus-messaging/src/ondemandpattern). The caller pattern shows how to send messages to an existing Workflow, while the on-demand pattern shows how to start a Workflow through Nexus and then send Signals to it. ### Develop an Asynchronous Nexus Operation handler to start a Workflow Use `@temporalio/nexus`'s `WorkflowRunOperationHandler` helper class to easily expose a Temporal Workflow as a Nexus Operation. Note that even though a Nexus operation can only take one input parameter, if you need to pass multiple arguments through to the workflow, you can do so by using multiple properties of the input object, and placing them in the array provided to the `args` option when calling `startWorkflow`. [nexus-hello/src/service/handler.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/handler.ts) ```ts import * as nexus from 'nexus-rpc'; import * as temporalNexus from '@temporalio/nexus'; import { helloService, EchoInput, EchoOutput, HelloInput, HelloOutput } from '../api'; import { helloWorkflow } from './workflows'; // ... export const helloServiceHandler = nexus.serviceHandler(helloService, { // ... hello: new temporalNexus.WorkflowRunOperationHandler( // WorkflowRunOperationHandler takes a function that receives the Operation's context and input. // That function can be used to validate and/or transform the input before passing it to // the Workflow, as well as to customize various Workflow start options as appropriate. // Call temporalNexus.startWorkflow() to actually start the Workflow from inside the // WorkflowRunOperationHandler's delegate function. async (ctx, input: HelloInput) => { return await temporalNexus.startWorkflow(ctx, helloWorkflow, { args: [input], // Workflow IDs should typically be business-meaningful IDs and are used to dedupe workflow starts. // For this example, the workflow handles the greeting request for a given person and language pair. workflowId: workflowIdForHello(input), // Task queue defaults to the task queue this Operation is handled on. }); }, ), }); ``` Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. In general, the ID should be passed in the Operation input as part of the Nexus Service contract. > **💡 Tip:** > RESOURCES > > [Attach multiple Nexus callers to a handler Workflow](/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing. > ### Register your Nexus Service handler in a Worker After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register your Nexus Service handler in a Worker. [nexus-hello/src/service/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/worker.ts) ```ts import { Worker, NativeConnection } from '@temporalio/worker'; import { helloServiceHandler } from './handler'; // ... const namespace = 'my-target-namespace'; const serviceTaskQueue = 'my-handler-task-queue'; const worker = await Worker.create({ connection, namespace, taskQueue: serviceTaskQueue, workflowsPath: require.resolve('./workflows'), nexusServices: [helloServiceHandler], }); ``` ## Develop a caller Workflow that uses the Nexus Service To execute a Nexus Operation from a Workflow, import the necessary service definition types, then use `@temporalio/workflow`'s `createNexusServiceClient` to create a Nexus client for that service. You will need to provide the Nexus Endpoint name, which you registered previously in [Create a Nexus Endpoint to route requests from caller to handler](#create-nexus-endpoint). [nexus-hello/src/caller/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/workflows.ts) ```ts import * as wf from "@temporalio/workflow"; import { helloService, LanguageCode } from "../service/api"; const HELLO_SERVICE_ENDPOINT = "hello-service-endpoint-name"; export async function helloCallerWorkflow(name: string, language: LanguageCode): Promise { const nexusClient = wf.createNexusServiceClient({ service: helloService, endpoint: HELLO_SERVICE_ENDPOINT, }); const helloResult = await nexusClient.executeOperation( "hello", { name, language }, { scheduleToCloseTimeout: "10s" } ); return helloResult.message; } ``` ### Register the caller Workflow in a Worker and start the caller Workflow This Workflow can be registered with a Worker and started using `client.startWorkflow()` or `client.executeWorkflow()`, as usual. Refer to the [complete TypeScript sample](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello) for reference. - [nexus-hello/src/caller/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/worker.ts) shows how to register the caller Workflow in a Worker and run the Worker. - [nexus-hello/src/starter.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/starter.ts) shows how to use a Temporal Client to execute the sample caller Workflow. ## Exceptions in Nexus operations Temporal provides general guidance on [Errors in Nexus operations](/references/failures#errors-in-nexus-operations). In TypeScript, there are three Nexus-specific exception classes: - `nexus-rpc`'s [`OperationError`](https://nexus-rpc.github.io/sdk-typescript/classes/OperationError.html): this is the exception type you should throw in a Nexus operation to indicate that it has failed according to its own application logic and should not be retried. - `nexus-rpc`'s [`HandlerError`](https://nexus-rpc.github.io/sdk-typescript/classes/HandlerError.html): you can throw this exception type in a Nexus operation with a specific [HandlerErrorType](https://nexus-rpc.github.io/sdk-typescript/types/HandlerErrorType.html). The error will be marked as either retryable or non-retryable according to the type, following the [Nexus spec](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors). The non-retryable handler error types are `BAD_REQUEST`, `UNAUTHENTICATED`, `UNAUTHORIZED`, `NOT_FOUND`, `NOT_IMPLEMENTED`; the retryable types are `RESOURCE_EXHAUSTED`, `INTERNAL`, `UNAVAILABLE`, `UPSTREAM_TIMEOUT`. - `@temporalio/nexus`'s [`NexusOperationFailure`](https://typescript.temporal.io/api/classes/common.NexusOperationFailure): this is the error thrown inside a Workflow when a Nexus operation fails for any reason. Use the `cause` attribute on the exception to access the cause chain. ## Canceling a Nexus Operation Nexus Operations, just like other cancellable APIs provided by the `@temporalio/workflow` package, execute within Cancellation Scopes. Requesting cancellation of a Cancellation Scope results in requesting cancellation for all cancellable operations owned by that scope. The Workflow itself defines the root Cancellation Scope. Requesting cancellation of the Workflow therefore propagates the cancellation request to all cancellable operations started by that workflow, including Nexus Operations. To provide more granular control over cancellation of a specific Nexus Operation, you may explicitly create a new Cancellation Scope, and start the Nexus Operation from within that scope. An example demonstrating this can be found at our [nexus cancellation sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-cancellation). Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancellation request. Once the caller Workflow completes, the caller's Nexus Machinery will not make any further attempts to cancel operations that are still running. It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending operations to finish before exiting the Workflow. ## Make Nexus calls across Namespaces in Temporal Cloud This section assumes you are already familiar with how to connect a Worker to Temporal Cloud. The Temporal Cloud CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install `tcld` and generate certificates Certificate generation is only available in `tcld`. To install the latest version of `tcld`, run the following command (on macOS): ``` brew install temporalio/brew/tcld ``` If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: ``` tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key ``` These certificates will be valid for one year. ### Create caller and handler Namespaces Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. **Temporal CLI** ``` temporal cloud login temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 temporal cloud namespace create \ --name \ --region aws-us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` **tcld** ``` tcld login tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 ``` Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces). ### Create a Nexus Endpoint to route requests from caller to handler To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. **Temporal CLI** ``` temporal cloud nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file description.md ``` **tcld** ``` tcld nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file description.md ``` The `--allow-namespace` is used to build an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in Runtime Access Control. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus). ## Observability ### Web UI A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Event history: ![Observability Sync](/img/cloud/nexus/go-sdk-observability-sync.png) An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Event history: ![Observability Async](/img/cloud/nexus/go-sdk-observability-async.png) ### Temporal CLI Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: ``` temporal workflow describe -w ``` Nexus events are included in the caller's Event history: ``` temporal workflow show -w ``` For **asynchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationStarted` - `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: - `NexusOperationScheduled` - `NexusOperationCompleted` > **📝 Note:** > > `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. > ### OpenTelemetry The `@temporalio/interceptors-opentelemetry` package supports Nexus Operations, providing automatic trace context propagation across Nexus boundaries from the caller Workflow to the handler. The easiest way to enable it is with the `OpenTelemetryPlugin`, which auto-registers Nexus interceptors alongside Activity and Workflow interceptors: ```ts import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; const plugin = new OpenTelemetryPlugin({ resource: myResource, spanProcessor: mySpanProcessor, }); const worker = await Worker.create({ // ... plugins: [plugin], nexusServices: [myServiceHandler], }); ``` The plugin creates the following spans: - **Caller side:** `StartNexusOperation:service/operation` — created when the caller Workflow starts a Nexus Operation. - **Handler side:** `RunStartNexusOperation:service/operation` and `RunCancelNexusOperation:service/operation` — created when the handler processes the operation. These spans are children of the caller span, linked via trace context propagated in Nexus request headers. See the [interceptors-opentelemetry sample](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) for a complete example. For custom interceptor logic beyond tracing (for example, logging, authorization), see [Nexus interceptor registration](/develop/typescript/workers/interceptors#nexus-interceptor-registration). ## Learn more - Read the high-level description of the [Temporal Nexus feature](/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082). - Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus). - Deploy Nexus Endpoints in production with [Temporal Cloud](/cloud/nexus). --- # Nexus TypeScript Quickstart Source: https://docs.temporal.io/develop/typescript/nexus/quickstart > Build a Nexus Service that wraps an existing Temporal Workflow using the TypeScript SDK [Temporal Nexus](/evaluate/nexus) connects Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. Build a Nexus Service that wraps an existing Temporal Workflow, then invoke it from a caller Workflow. > **ℹ️ Info:** > To evaluate whether Nexus fits your use case, see the [evaluation guide](/evaluate/nexus). To learn how Nexus works, see [Temporal Nexus](/nexus). **Prerequisites:** Complete the [TypeScript SDK Quickstart](/develop/typescript/set-up-your-local-typescript) first. You should have `activities.ts`, `workflows.ts`, `worker.ts`, and `client.ts` from that guide. ## What you'll build You have `example` running in the `default` Namespace. By the end of this guide: 1. A Nexus Service will expose `example` as an Operation. 2. A second Namespace will contain a Workflow that calls that Operation. 3. The caller Workflow will get back `"Hello, Temporal!"` — the same result, but across Namespaces. ## 1. Define the Nexus Service Create a file called `service.ts` that defines the Nexus Service contract. Creating a Nexus Service establishes the contract between your implementation and any callers. It provides type safety when invoking Nexus Operations and ensures that Operation Handlers fulfill the contract. `nexus.service()` declares a named service, and `nexus.operation()` defines a typed operation. The `example` Workflow returns `string`, so the operation output type is `string`. ```typescript import * as nexus from 'nexus-rpc'; export interface MyInput { name: string; } export const sayHelloService = nexus.service('say-hello', { sayHello: nexus.operation(), }); ``` ## 2. Define the Nexus Operation handlers Create a file called `handler.ts` that implements the Nexus Operation handler. Operation handlers contain the logic that runs when a caller invokes a Nexus Operation. `WorkflowRunOperationHandler` creates an asynchronous Nexus Operation that starts a Workflow. The handler bridges the Nexus `MyInput` interface to the `example` Workflow's `string` parameter by extracting `input.name`. ```typescript import { randomUUID } from 'crypto'; import * as nexus from 'nexus-rpc'; import * as temporalNexus from '@temporalio/nexus'; import { sayHelloService, MyInput } from './service'; import { example } from './workflows'; export const sayHelloHandler = nexus.serviceHandler(sayHelloService, { sayHello: new temporalNexus.WorkflowRunOperationHandler( async (ctx, input: MyInput) => { return await temporalNexus.startWorkflow(ctx, example, { args: [input.name], workflowId: "say-hello-nexus-" + randomUUID(), // Task queue defaults to the task queue this Operation is handled on. }); }, ), }); ``` ## 3. Register the Nexus Service handler in a Worker Update your existing `worker.ts` to register the Nexus Service Handler. A Worker will only poll for and process incoming Nexus requests if the Nexus Service Handlers are registered. This is the same Worker concept used for Workflows and Activities. The `nexusServices` parameter registers the handler so it can receive Nexus Operation requests. ```typescript import { NativeConnection, Worker } from '@temporalio/worker'; import * as activities from './activities'; import { sayHelloHandler } from './handler'; async function run() { const connection = await NativeConnection.connect({ address: 'localhost:7233', }); try { const worker = await Worker.create({ connection, namespace: 'default', taskQueue: 'hello-world', workflowsPath: require.resolve('./workflows'), activities, nexusServices: [sayHelloHandler], }); await worker.run(); } finally { await connection.close(); } } run().catch((err) => { console.error(err); process.exit(1); }); ``` ## 4. Develop the caller Workflow Update your existing `workflows.ts` file with a Workflow which invokes the Nexus Operation. The caller Workflow demonstrates the consumer side of Nexus. Instead of importing handler code directly, the caller only depends on the Service contract. This keeps the caller and handler decoupled so they can live in separate Namespaces, repositories, or even teams. The `wf.createNexusServiceClient()` method creates a client bound to your Nexus Service and Endpoint. `executeOperation` starts the operation and waits for the result. ```typescript import { proxyActivities, createNexusServiceClient } from '@temporalio/workflow'; // Only import the activity types import type * as activities from './activities'; import { sayHelloService } from './service'; const { greet } = proxyActivities({ startToCloseTimeout: '1 minute', }); /** A workflow that simply calls an activity */ export async function example(name: string): Promise { return await greet(name); } const NEXUS_ENDPOINT = 'my-nexus-endpoint-name'; export async function callerWorkflow(name: string): Promise { const nexusClient = createNexusServiceClient({ service: sayHelloService, endpoint: NEXUS_ENDPOINT, }); return await nexusClient.executeOperation('sayHello', { name }, { scheduleToCloseTimeout: '10s' }); } ``` ## 5. Create the caller Namespace and Nexus Endpoint Before running the application, create a caller Namespace and a Nexus Endpoint to route requests from the caller to the handler. The handler uses the `default` Namespace that was created when you started the dev server. Namespaces provide isolation between the caller and handler sides. The Nexus Endpoint acts as a routing layer that connects the caller Namespace to the handler's target Namespace and Task Queue. The endpoint name must match the variable defined in `caller.ts` from step 4. Make sure your local Temporal dev server is running (`temporal server start-dev`). ```bash temporal operator namespace create --namespace my-caller-namespace ``` ```bash temporal operator nexus endpoint create \\ --name my-nexus-endpoint-name \\ --target-namespace default \\ --target-task-queue hello-world ``` ## 6. Run and Verify Create a file called `caller-starter.ts` to start the caller Worker and execute the Workflow. This step brings everything together: the caller Worker hosts `callerWorkflow`, which uses the Nexus client to invoke `sayHello` on the handler side. The full request flows from the caller Workflow, through the Nexus Endpoint, to the handler Worker running the `example` Workflow, and back to the caller. **Run the application:** 1. Start the handler Worker in one terminal: ```bash npx ts-node src/worker.ts ``` 2. Run the caller in another terminal: ```bash npx ts-node src/caller-starter.ts ``` You should see: ``` Workflow result: Hello, Temporal! ``` Open the [Temporal Web UI](http://localhost:8233) and find the `callerWorkflow` execution. You should see `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted` events in the Event history. ```typescript import { randomUUID } from 'crypto'; import { Connection, Client } from '@temporalio/client'; import { NativeConnection, Worker } from '@temporalio/worker'; import { callerWorkflow } from './workflows'; const CALLER_TASK_QUEUE = 'my-caller-task-queue'; const NAMESPACE = 'my-caller-namespace'; async function main() { const clientConnection = await Connection.connect({ address: 'localhost:7233', }); const client = new Client({ connection: clientConnection, namespace: NAMESPACE, }); const workerConnection = await NativeConnection.connect({ address: 'localhost:7233', }); try { const worker = await Worker.create({ connection: workerConnection, namespace: NAMESPACE, taskQueue: CALLER_TASK_QUEUE, workflowsPath: require.resolve('./workflows'), }); await worker.runUntil(async () => { const result = await client.workflow.execute(callerWorkflow, { args: ['Temporal'], workflowId: \`caller-workflow-\${randomUUID()}\`, taskQueue: CALLER_TASK_QUEUE, }); console.log('Workflow result:', result); }); } finally { await workerConnection.close(); } } main().catch((err) => { console.error(err); process.exit(1); }); ``` ## Next Steps Now that you have a working Nexus Service, here are some resources to deepen your understanding: - **[TypeScript Nexus Feature Guide](/develop/typescript/nexus)**: Covers synchronous and asynchronous Operations, error handling, cancellation, and cross-Namespace calls. - **[Nexus Operations](/nexus/operations)**: The full Operation lifecycle, including retries, timeouts, and execution semantics. - **[Nexus Services](/nexus/services)**: Designing Service contracts and registering multiple Services per Worker. - **[Nexus Patterns](/nexus/patterns)**: Comparing the collocated and router-queue deployment patterns. - **[Error Handling in Nexus](/nexus/error-handling)**: Handling retryable and non-retryable errors across caller and handler boundaries. - **[Execution Debugging](/nexus/execution-debugging)**: Bi-directional linking and OpenTelemetry tracing for debugging Nexus calls. - **[Nexus Endpoints](/nexus/endpoints)**: Managing Endpoints and understanding how they route requests. - **[Temporal Nexus on Temporal Cloud](/cloud/nexus)**: Deploying Nexus in a production Temporal Cloud environment with built-in access controls and multi-region connectivity. --- # Standalone Nexus Operations - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/nexus/standalone-operations > Execute Nexus Operations independently without a Workflow using the Temporal TypeScript SDK. > **Pre-release** > Requires TypeScript SDK `v1.20.2` or above. All APIs are experimental and may be subject to backwards-incompatible changes. [Standalone Nexus Operations](/standalone-nexus-operation) let you run Nexus Operation Executions independently, without being orchestrated by a Workflow. Instead of calling a Nexus Operation from within a Workflow Definition using `@temporalio/workflow`'s `createNexusServiceClient()`, you execute a Standalone Nexus Operation directly from a Nexus service client created on the Temporal Client using `client.nexus.createServiceClient()`. Standalone Nexus Operations use the same Nexus Service contract, Operation handlers, and Worker setup as Workflow-driven Operations — only the execution path differs. See the [Nexus feature guide](/develop/typescript/nexus/feature-guide) for details on [defining a Service contract](/develop/typescript/nexus/feature-guide#define-nexus-service-contract) and [developing Operation handlers and registering a Service in a Worker](/develop/typescript/nexus/feature-guide#develop-nexus-service-operation-handlers). This page focuses on the client-side APIs that are unique to Standalone Nexus Operations: - [Execute a Standalone Nexus Operation](#execute-operation) - [Start a Standalone Nexus Operation and Wait for the Result](#get-operation-result) - [List Standalone Nexus Operations](#list-operations) - [Count Standalone Nexus Operations](#count-operations) - [Run Standalone Nexus Operations with Temporal Cloud](#run-standalone-nexus-operations-temporal-cloud) > **📝 Note:** > This documentation uses source code from the > [TypeScript Nexus Standalone sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-standalone-operations). > ## Prerequisites Standalone Nexus Operations are at Pre-release and require a special Temporal CLI build. ### 1. Install and verify the Pre-release Temporal CLI The `temporal nexus operation` commands require a Pre-release build of the Temporal CLI. See [Temporal CLI support](/standalone-nexus-operation#temporal-cli-support) for the platform downloads, then verify: ```bash ./temporal --version # temporal version 1.7.4-standalone-nexus-operations ``` Run it as `./temporal` from the directory where you extracted it. The standard `brew install temporal` build does not include Standalone Nexus Operation support during Pre-release. ### 2. Start a local dev server The Pre-release dev server enables Standalone Nexus Operations by default — no dynamic config is required. Start it with the caller and handler Namespaces pre-created: ```bash ./temporal server start-dev \ --namespace my-caller-namespace \ --namespace my-handler-namespace ``` The starter and Worker connect to two different Namespaces (a caller Namespace and a handler Namespace), mirroring how Nexus crosses Namespace boundaries. To run the examples on this page against the [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-standalone-operations), create a Nexus Endpoint that routes to the handler Namespace and the Worker's Task Queue: ```bash ./temporal operator nexus endpoint create \ --name my-nexus-endpoint \ --target-namespace my-handler-namespace \ --target-task-queue nexus-handler-queue ``` Start the sample Worker in the handler Namespace: ```bash TEMPORAL_NAMESPACE=my-handler-namespace npm run worker ``` Run the starter in the caller Namespace (from a separate terminal): ```bash TEMPORAL_NAMESPACE=my-caller-namespace npm run starter ``` ## Execute a Standalone Nexus Operation To execute a Standalone Nexus Operation, first create a [`NexusServiceClient`](https://typescript.temporal.io/api/interfaces/client.NexusServiceClient) using [`client.nexus.createServiceClient()`](https://typescript.temporal.io/api/classes/client.NexusClient#createserviceclient), bound to a specific Nexus Endpoint and Service. The endpoint must be pre-created on the server. Then call `startOperation()` or `executeOperation()` from application code (for example, a starter program), not from inside a Workflow Definition. `executeOperation` waits for the Operation to complete and returns the result. Both methods require `id`. `scheduleToCloseTimeout` is optional and defaults to the maximum allowed by the Temporal server. ```ts const nexusClient = client.nexus.createServiceClient({ endpoint: ENDPOINT_NAME, service: myNexusService, }); // Await the result of the operation immediately. const echoResult = await nexusClient.executeOperation( myNexusService.operations.echo, { message: 'hello' }, { id: `echo-${nanoid()}`, scheduleToCloseTimeout: '10s', }, ); ``` See the full [starter sample](https://github.com/temporalio/samples-typescript/blob/main/nexus-standalone-operations/src/starter.ts) for a complete example that executes both synchronous and asynchronous Operations, gets their results, and lists and counts Operations. Or use the Temporal CLI to execute a Standalone Nexus Operation: ```bash ./temporal nexus operation execute \ --namespace my-caller-namespace \ --endpoint my-nexus-endpoint \ --service myNexusService \ --operation echo \ --operation-id my-echo-op \ --input '{"message":"hello"}' ``` ## Start a Standalone Nexus Operation and Wait for the Result `startOperation` returns a [`NexusOperationHandle`](https://typescript.temporal.io/api/interfaces/client.NexusOperationHandle). Use `NexusOperationHandle.result()` to wait until the Operation completes and retrieve its result. This works for both synchronous and asynchronous Operations. ```ts // Start an operation and get a NexusOperationHandle const handle = await nexusClient.startOperation( myNexusService.operations.hello, { name: 'World' }, { id: `hello-${nanoid()}`, scheduleToCloseTimeout: '10s', }, ); // Await the result const helloResult = await handle.result(); console.log(helloResult.greeting); ``` If the Operation completed successfully, the result is returned. If the Operation failed, the failure is thrown as an error. Or use the Temporal CLI to wait for a result by Operation ID: ```bash ./temporal nexus operation result --namespace my-caller-namespace --operation-id my-echo-op ``` ## List Standalone Nexus Operations Use [`client.nexus.list()`](https://typescript.temporal.io/api/classes/client.NexusClient#list) to list Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. The result is an async iterator that yields operation metadata entries. Note that `client.nexus.list()` is called on the base `Client`, not on the `NexusServiceClient`. ```ts const query = `Endpoint = "${ENDPOINT_NAME}"`; for await (const op of client.nexus.list({ query })) { console.log( `OperationId: ${op.operationId},`, `Operation: ${op.operation},`, `Status: ${op.status}`, ); } ``` The `query` parameter accepts [List Filter](/list-filter) syntax. For example, `"Endpoint = 'my-endpoint' AND Status = 'Running'"`. Or use the Temporal CLI: ```bash ./temporal nexus operation list --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Count Standalone Nexus Operations Use [`client.nexus.count()`](https://typescript.temporal.io/api/classes/client.NexusClient#count) to count Standalone Nexus Operation Executions that match a [List Filter](/list-filter) query. Note that `client.nexus.count()` is called on the base `Client`, not on the Nexus service client. ```ts const query = `Endpoint = "${ENDPOINT_NAME}"`; const count = await client.nexus.count(query); console.log(`Total Nexus operations: ${count.count}`); ``` Or use the Temporal CLI: ```bash ./temporal nexus operation count --namespace my-caller-namespace --query 'Endpoint = "my-nexus-endpoint"' ``` ## Run Standalone Nexus Operations with Temporal Cloud The code samples referenced on this page use [`loadClientConnectConfig()`](https://typescript.temporal.io/api/namespaces/envconfig#loadclientconfig) from `@temporalio/envconfig`, so the same code works against Temporal Cloud — just configure the connection via environment variables or a TOML profile. No code changes are needed. For full details on connecting to Temporal Cloud, including Namespace creation, Nexus Endpoint setup, certificate generation, and authentication options, see [Make Nexus calls across Namespaces in Temporal Cloud](/develop/typescript/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) and [Connect to Temporal Cloud](/develop/typescript/client/temporal-client#connect-to-temporal-cloud). --- # Platform - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/platform > This section explains how to implement platform with the TypeScript SDK ![TypeScript SDK Banner](/img/assets/banner-typescript-temporal.png) ## Platform - [Observability](/develop/typescript/platform/observability) - [Enriching the UI](/develop/typescript/platform/enriching-ui) --- # Enriching the user interface - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/platform/enriching-ui > Add contextual information to workflows and events in the Temporal UI using the TypeScript SDK. Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. ## Adding Summary and Details to Workflows ### Starting a Workflow When starting a Workflow, you can provide a static summary and details to help identify the workflow in the UI: ```typescript import { Client } from '@temporalio/client'; const client = new Client(); // Start a workflow with static summary and details const handle = await client.workflow.start(yourWorkflow, { args: ['workflow input'], taskQueue: 'your-task-queue', workflowId: 'your-workflow-id', staticSummary: 'Order processing for customer #12345', staticDetails: 'Processing premium order with expedited shipping' }); ``` `staticSummary` is a single-line description that appears in the workflow list view, limited to 200 bytes. `staticDetails` can be multi-line and provides more comprehensive information that appears in the workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use the `execute` method with the same parameters: ```typescript const result = await client.workflow.execute(yourWorkflow, { args: ['workflow input'], taskQueue: 'your-task-queue', workflowId: 'your-workflow-id', staticSummary: 'Order processing for customer #12345', staticDetails: 'Processing premium order with expedited shipping' }); ``` ### Inside the Workflow Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. ```typescript import { getCurrentDetails, setCurrentDetails } from '@temporalio/workflow'; export async function yourWorkflow(input: string): Promise { // Get the current details const currentDetails = getCurrentDetails(); console.log(`Current details: ${currentDetails}`); // Set/update the current details setCurrentDetails('Updated workflow details with new status'); return 'Workflow completed'; } ``` ### Adding Summary to Activities and Timers You can attach a `summary` to activities by using `executeWithOptions` when calling them: ```typescript import { proxyActivities } from '@temporalio/workflow'; import type * as activities from './activities'; const { yourActivity } = proxyActivities({ startToCloseTimeout: '10 seconds' }); export async function yourWorkflow(input: string): Promise { // Execute an activity with a summary using executeWithOptions const result = await yourActivity.executeWithOptions( { staticSummary: 'Processing user data' }, [input] // Note: arguments must be passed as an array ); return result; } ``` Similarly, you can attach a `summary` to timers within a workflow: ```typescript import { sleep } from '@temporalio/workflow'; export async function yourWorkflow(input: string): Promise { // Create a timer with a summary await sleep('5 minutes', { summary: 'Waiting for payment confirmation' }); return 'Timer completed'; } ``` The input format for `summary` is a string, and limited to 200 bytes. ## Viewing Summary and Details in the UI Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in three key locations: ### Workflow Overview Section At the top of the workflow details page, you'll find the workflow-level metadata: - **Summary & Details** - Displays the static summary and static details set when starting the workflow - **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Timeline The **Timeline** tab on the Workflow details page renders each Activity and Timer as a horizontal bar. When you set a `Summary` on an Activity or Timer, the summary text is shown directly on the bar label, making it possible to distinguish individual instances of the same Activity Type at a glance. Labels longer than 120 characters are truncated with an ellipsis. Setting a distinct `Summary` per Activity is especially useful for **fan-out Workflows** that schedule many instances of the same Activity Type, where the Activity Type alone is not enough to tell each bar apart on the Timeline. Activity `Summary` support on the Timeline shipped in Temporal UI **v2.34.6** and is available on Temporal Cloud and on self-hosted UI builds at that version or later. ### Event History Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. --- # Observability - TypeScript SDK Source: https://docs.temporal.io/develop/typescript/platform/observability > Enhance the observability of your Temporal Application with metrics, tracing, logging, and visibility features. View Workflow state, set up OpenTelemetry, and customize logging for seamless monitoring and insights. The observability section of the TypeScript developer guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution. This section covers features related to viewing the state of the application, including: - [Emit metrics](#metrics) - [Set up tracing](#tracing) - [Log from a Workflow](#logging) - [Visibility APIs](#visibility) ## Emit metrics Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process. For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics). - For an overview of Prometheus and Grafana integration, refer to the [Monitoring](/self-hosted-guide/monitoring) guide. - For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics). - For an end-to-end example that exposes metrics with the TypeScript SDK, refer to the [samples-typescript](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) repo. Workers can emit metrics and traces. There are a few [telemetry options](https://typescript.temporal.io/api/interfaces/worker.TelemetryOptions) that can be provided to [`Runtime.install`](https://typescript.temporal.io/api/classes/worker.Runtime/#install). The common options are: - `metrics: { otel: { url } }`: The URL of a gRPC [OpenTelemetry collector](https://opentelemetry.io/docs/collector/). - `metrics: { prometheus: { bindAddress } }`: Address on the Worker host that will hav