Nexus Operations

SUPPORT, STABILITY, and DEPENDENCY INFO

Temporal Nexus is available in Public Preview for Temporal Cloud and self-hosted deployments.

Nexus Operations are arbitrary-duration operations that may be synchronous or asynchronous, short-lived or long-lived, and are used to connect Temporal Applications within and across Namespaces, clusters, regions, and clouds.

Unlike a traditional RPC, an asynchronous Nexus Operation has an operation identity that can be used to re-attach to a long-lived Nexus Operation, for example, one backed by a Temporal Workflow. Nexus Operations support a uniform interface to get the status of an operation or its result, receive a completion callback, or cancel the operation – all of which are fully integrated into the Temporal Platform.

SDK support

The Temporal SDK provides an integrated Temporal experience to build, run, and use Nexus Operations.

RESOURCES

• Go SDK - Nexus quick start and code sample
• Java SDK - Nexus quick start and code sample

Caller Workflows use the Temporal SDK to execute Nexus Operations.

Nexus Operation handlers are created with Temporal SDK builder functions such as:

• New-Workflow-Run-Operation
  • Start a Workflow as an asynchronous Operation.
• New-Sync-Operation
  • Invoke an underlying Query or Signal as a synchronous Operation.
  • Invoke an Update as a synchronous Operation.
  • Execute arbitrary code as a synchronous Operation.

Nexus Operation lifecycle

When you execute a Nexus Operation from a caller Workflow using the Temporal SDK, a command is sent to Temporal to schedule the Nexus Operation which is atomically handed off to the Nexus Machinery. The Nexus Machinery uses state-machine-based invocation and completion callbacks to ensure at-least-once execution of a Nexus Operation and reliable delivery of the result.

The Nexus Machinery is responsible for making the Nexus RPC calls on behalf of the caller Workflow, with automatic-retries. This means you don't have to use Nexus RPC directly, only the Temporal SDK along with the Temporal Service.

Synchronous Operation lifecycle

Nexus supports synchronous Operations that take less than 10 seconds to execute, as measured from the caller's Nexus Machinery.

The lifecycle for a synchronous Nexus Operation, for example to Signal or Query a Workflow:

1. Caller Workflow executes a Nexus Operation using the Temporal SDK.
2. Caller Worker issues a ScheduleNexusOperation command to it's Namespace gRPC endpoint.
3. Caller Namespace records a NexusOperationScheduled event in the caller's Workflow History.
4. Caller Nexus Machinery makes a Nexus call to start a Nexus Operation.
5. Handler Nexus Machinery receives the Nexus request and sync matches to a handler Worker.
6. Handler Worker gets a Nexus Task to start a Nexus Operation, by polling the Nexus Endpoint's target Task Queue.
7. Handler processes the Nexus Task, using the Temporal SDK New-Sync-Operation.
8. Handler responds to it's Namespace gRPC endpoint with the Operation result.
9. Caller Namespace records the result in the caller's Workflow history as a Nexus event, for example Completed or Failed.
10. Caller Worker polls for a Workflow Task on it's Workflow Task Queue.
11. Caller Workflow gets the Operation result, using the Temporal SDK.

tip

Stay within the remaining request deadline budget to avoid being timed out. If a Nexus handler times out, the Operation will be retried by the caller's Nexus Machinery until the Operation's Schedule-to-Close timeout has been exceeded.

Asynchronous Operation lifecycle

An asynchronous Nexus Operation may take up to 60 days to complete in Temporal Cloud, which is the maximum Schedule-to-Close-Timeout.

The lifecycle of an asynchronous Nexus Operation, with differences between the synchronous lifecycle in bold:

1. Caller Workflow executes a Nexus Operation using the Temporal SDK.
2. Caller Worker issues a ScheduleNexusOperation command to its Namespace gRPC endpoint.
3. Caller Namespace records a NexusOperationScheduled event in the caller Workflow's History.
4. Caller Nexus Machinery makes a Nexus RPC to start a Nexus Operation.
5. Handler Nexus Machinery receives the Nexus request and sync matches to a handler Worker.
6. Handler Worker gets a Nexus Task to start a Nexus Operation, by polling the Nexus Endpoint's target Task Queue.
7. Handler processes the Nexus Task, using the Temporal SDK New-Workflow-Run-Operation.
8. Handler responds to it's Namespace gRPC endpoint with the start Operation response.
9. Caller Namespace records the response in the caller's Workflow history as a Nexus event, for example NexusOperationStarted.
10. Handler Workflow completes and the Nexus Completion Callback is delivered to the caller's Nexus Machinery.
11. Caller Namespace records the result in the caller's Workflow history as a Nexus event, for example Completed or Failed.
12. Caller Worker polls for a Workflow Task on its Workflow Task Queue.
13. Caller Workflow gets the Operation result, using the Temporal SDK.

Executing arbitrary code from a synchronous Nexus Operation handler

Synchronous Nexus Operation handlers can execute arbitrary code, but unlike Activities they should be short-lived. As mentioned above, a synchronous Nexus Operation handler has less than 10 seconds to process a Nexus start Operation request and should stay within the remaining request deadline budget for the Nexus request. For example, this may be done by looking at the Request-Timeout header or hooking into cancelation that is triggered when the timeout is exceeded.

System interactions

Temporal Nexus Operations are requested and processed using the Temporal queue-based Worker architecture. Workers interact with their Namespace gRPC endpoint as before. Nexus Machinery on both sides handles the cross-namepace communication.

For example, when you execute a Nexus Operation in a caller Workflow the following Namespace gRPC calls are made:

1. **RespondWorkflowTaskCompleted (ScheduleNexusOperation command) is used by the caller Worker to schedule a Nexus Operation, which atomically hands off execution to the caller's Nexus Machinery.
2. PollNexusTaskQueue is used by the handler Worker to receive a Nexus Task to process, for example to start a Nexus Operation.
3. RespondNexusTaskCompleted or RespondNexusTaskFailed is used by the handler Worker to return the Nexus Task result. When asked to start a Nexus Operation, the Nexus handler decides if the Operation will be synchronous or asynchronous.
   1. This is typically a static decision based on the Temporal SDK builder function used.
   2. Asynchronous Nexus Operations, created with the New-Workflow-Run-Operation SDK helper, will return a Nexus Operation ID, that can be used to perform additional actions like canceling an Operation.
   3. Synchronous Nexus Operations, created with the New-Sync-Operation SDK helper, will return the Operation result directly.
   4. The caller's Nexus Machinery receives the result and records a NexusOperation event in the caller's Workflow History.
4. PollWorkflowTaskQueue is used by the caller Worker to receive a Workflow Task with the Nexus Operation event which may be Started, Completed, Failed, Canceled, or TimedOut.

Automatic Retries

Once the caller Workflow schedules an Operation with the caller's Temporal Service, the caller's Nexus Machinery keeps trying to start the Operation. If a retryable Nexus error is returned the Nexus Machinery will retry until the Nexus Operation's Start-to-Close-Timeout is exceeded.

For example, if a Nexus handler returns a retryable error, or an upstream timeout is encountered by the caller, the Nexus request will be retried up to the default Retry Policy's max attempts and expiration interval.

note

This differs from how Activities and Workflows handle errors and retries:

• Errors in Activities
• Non-retryable errors

See errors in Nexus handlers to control the retry behavior by returning a non-retryable Nexus error.

Execution semantics

At-least-once execution semantics and idempotency

The Nexus Machinery provides reliable execution with at-least-once execution semantics for a Nexus Operation, until the caller's Schedule-to-Close-Timeout is exceeded, at which time the overall Nexus Operation times out.

The Nexus Machinery breaks up the Nexus Operation lifecycle into one or more Nexus Tasks that a Nexus handler is responsible for processing. If a Nexus handler times out or returns a non-retryable Nexus error, then the Nexus Machinery will retry the Nexus request to provide at-least-once execution. This means it's possible for your Nexus handler to be invoked multiple times for a given Nexus Operation.

To deal with at-least-once execution, the Nexus Operation handler should be idempotent, like Activities should be idempotent. It's not required in all cases, but highly recommended in general.

Exactly-once execution semantics through an underlying WorkflowIDReusePolicy

To deduplicate work and get exactly-once execution semantics, a Nexus Operation can start a Workflow with a WorkflowIDReusePolicy of RejectDuplicates which only allows one Workflow Execution per Workflow ID within a Namespace for the Retention Period.

Cancelation

The request to cancel a caller Workflow is automatically propagated to all pending Nexus Operations, and in turn any underlying handler Workflows.

If an underlying handler Workflow is canceled, the Nexus Operation will report a Canceled Failure to the caller's Workflow Execution.

Termination

If the caller Workflow is Terminated, all pending Nexus Operations are abandoned. If possible, consider cancelation instead.

Versioning

Task Routing is the simplest way to version your service code.

If you have a new backward-incompatible Nexus Operation Handler, for example due to a wire-level incompatible breaking change, start by using a different Service and Task Queue. The version may be part of the service name, for example prod.payments.v2. Callers can then migrate to the new version in their normal deployment schedule.