Skip to main content

Early Return + Local Activities

View Markdown
TLDR

Combine Update-with-Start with Local Activities in the synchronous Phase 1 to reduce first-response latency from ~265 ms to ~160 ms. Phase 1 (initialization) runs as Local Activities with no server round-trips. Phase 2 (settlement) runs as regular Activities in the background. The client receives its response as soon as Phase 1 completes, so the in-process execution speed of Local Activities directly improves the time-to-first-byte.

Overview

The Early Return pattern uses Update-with-Start to send the client an early response after a fast Phase 1 completes, while slow Phase 2 work continues in the background. This pattern extends Early Return by running Phase 1 Activities as Local Activities, eliminating all server round-trips on the synchronous hot path.

Because the client waits only for Phase 1, and Phase 1 now runs entirely in-process, the end-to-end first-response time drops from approximately 265 ms (Early Return with regular Activities) to approximately 160 ms.

Numbered walkthrough:

  1. The client sends a single UpdateWithStart RPC, which atomically starts the Workflow and delivers the Update in one server call.
  2. The Worker picks up the first Workflow Task and executes Phase 1. Because all Phase 1 Activities are Local Activities, they run in-process with no additional server calls. The entire phase completes inside a single Workflow Task.
  3. When the Workflow Task completes, the server marks the Update as fulfilled and returns the result to the waiting client. This happens as soon as Phase 1 finishes—approximately 160 ms after the initial request.
  4. The Workflow continues in a new Workflow Task to execute Phase 2 using regular Activities. These run in the background. The client is not blocked by this work.

Problem

The plain Early Return pattern reduces first-response latency significantly compared to waiting for the full Workflow. However, if Phase 1 uses regular Activities, each Activity still incurs server scheduling overhead (~50 ms per call on Temporal Cloud). With two or three Phase 1 Activities, this overhead alone can account for 100–150 ms of the first-response time.

Solution

Run all Phase 1 Activities as Local Activities. They execute in-process within the first Workflow Task, so their results are available to the Update handler as soon as the task completes—with no additional server calls. Phase 2 Activities remain regular Activities, which is acceptable because Phase 2 runs in the background after the client has already received its response.

# workflows.py
from temporalio import workflow
from datetime import timedelta
from activities import validate_transaction, init_transaction, complete_transaction, cancel_transaction

LOCAL_TIMEOUT = timedelta(seconds=5)
ACTIVITY_TIMEOUT = timedelta(seconds=30)

@workflow.defn
class TransactionWorkflow:
def __init__(self) -> None:
self._tx: Transaction | None = None
self._phase1_done = False
self._phase1_error: Exception | None = None

@workflow.update
async def get_result(self, req: TransactionRequest) -> Transaction:
# Wait for Phase 1 to finish before returning to the caller.
await workflow.wait_condition(lambda: self._phase1_done)
if self._phase1_error:
raise self._phase1_error
return self._tx

@workflow.run
async def run(self, req: TransactionRequest) -> None:
try:
# Phase 1: Local Activities — zero server round-trips on the hot path.
tx = await workflow.execute_local_activity(
validate_transaction, req,
schedule_to_close_timeout=LOCAL_TIMEOUT,
)
self._tx = await workflow.execute_local_activity(
init_transaction, tx,
schedule_to_close_timeout=LOCAL_TIMEOUT,
)
except Exception as e:
self._phase1_error = e
finally:
self._phase1_done = True

if self._phase1_error:
if self._tx is not None:
await workflow.execute_activity(
cancel_transaction, self._tx,
start_to_close_timeout=ACTIVITY_TIMEOUT,
)
return

# Phase 2: Regular Activities — background settlement (client already has response).
await workflow.execute_activity(
complete_transaction, self._tx,
start_to_close_timeout=ACTIVITY_TIMEOUT,
)

When to use

Good fit:

  • User-facing workflows where the first response is more latency-critical than total execution time
  • Phase 1 consists of short, idempotent validation and initialization steps that fit naturally as Local Activities
  • Phase 2 is slow (network I/O, external systems) and does not need to be on the client's critical path
  • You already use or plan to use the Early Return pattern

Poor fit:

  • Phase 1 Activities are long-running or require heartbeating—Local Activities cannot heartbeat
  • The Workflow's total latency matters more than first-response latency
  • Phase 1 and Phase 2 cannot be cleanly separated

Benefits and trade-offs

PatternFirst ResponseTotal LatencyComplexity
Synchronous workflowSame as total~850 msLow
Early Return (regular activities)~265 ms~850 msMedium
Local Activities onlySame as total~275 msMedium
Early Return + Local Activities~160 ms~275 msMedium
Eager Workflow Start + Local Activities~160 ms~265 msHigh

Best practices

  • Keep Phase 1 Local Activities short. Each must complete well within the Workflow Task timeout (default 10 seconds). Aim for under 5 seconds total for all Phase 1 work.
  • Design Phase 1 for at-least-once execution. If the Workflow Task that runs Phase 1 fails and retries, all Phase 1 Local Activities re-execute. Phase 1 operations must be idempotent.
  • Separate Phase 1 and Phase 2 concerns cleanly. The Update handler should wait only on the Phase 1 sentinel flag, not on any Phase 2 state. Phase 2 should be independent enough to proceed without client involvement.
  • Set appropriate timeouts for Phase 2. Phase 2 regular Activities run in the background and should have a startToCloseTimeout that reflects the maximum acceptable settlement time.

Common pitfalls

  • Putting slow operations in Phase 1. If any Phase 1 Local Activity takes too long, the Workflow Task times out and retries. The client also waits longer for its early response, defeating the purpose of the pattern.
  • Non-idempotent Phase 1. A retried Workflow Task re-executes all Local Activities in that task. Ensure Phase 1 operations (e.g., creating a record in an external system) are safe to re-run.
  • Ignoring Phase 1 errors in Phase 2. Always check Phase 1 error state before proceeding to Phase 2. If Phase 1 failed, Phase 2 should run a compensating Activity (cancel, rollback) rather than complete.
  • Mixing Local and regular Activity stubs incorrectly. In Java, Workflow.newLocalActivityStub and Workflow.newActivityStub return distinct objects. Make sure Phase 1 uses the local stub and Phase 2 uses the regular stub.
  • Early Return — the baseline Update-with-Start pattern without Local Activity optimization
  • Local Activities — using Local Activities for full-workflow latency reduction without early return
  • Eager Workflow Start — eliminates the Matching step when starting the Workflow for additional total latency improvement