Skip to main content

Failure detection - .NET SDK

This page shows how to do the following:

Raise and Handle Exceptions​

In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language. Temporal uses several different error classes internally — for example, CancelledFailureException in the .NET SDK, to handle a Workflow cancellation. You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic.

The one Temporal error class that you will typically raise deliberately is ApplicationFailureException. In fact, any other exceptions that are raised from your C# code in a Temporal Activity will be converted to an ApplicationFailureException internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries.

In other words, these two code samples do the same thing:

[Serializable]
public class InvalidDepartmentException : Exception
{
    public InvalidDepartmentException() : base() { }
    public InvalidDepartmentException(string message) : base(message) { }
    public InvalidDepartmentException(string message, Exception inner) : base(message, inner) { }
}

[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new InvalidDepartmentException("Invalid department");
}
[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new ApplicationFailureException("Invalid department", errorType: "InvalidDepartmentException");
}

Depending on your implementation, you may decide to use either method. One reason to use the Temporal ApplicationFailureException class is because it allows you to set an additional non_retryable parameter. This way, you can decide whether an error should not be retried automatically by Temporal. This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse:

[Activity]
public Task<OrderConfirmation> SendBillAsync(Bill bill)
{
    throw new ApplicationFailureException("Invalid department", nonRetryable: true);
}

You can alternately specify a list of errors that are non-retryable in your Activity Retry Policy.

Failing Workflows​

One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately. The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold. Activities will not actually return a failure to your Workflow until this condition or another non-retryable condition is met. At this point, you can decide how to handle an error returned by your Activity the way you would in any other program. For example, you could implement a Saga Pattern that uses try/catch blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure.

You will only fail a Workflow by manually raising an ApplicationFailureException from the Workflow code. You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue:

try
{
    await Workflow.ExecuteActivityAsync(
        (Activities act) => act.ValidateCreditCardAsync(order.Customer.CreditCardNumber),
        options);
}
catch (ActivityFailureException err)
{
    logger.LogError("Unable to process credit card: {Message}", err.Message);
    throw new ApplicationFailureException(message: "Invalid credit card number error");
}

This works differently in a Workflow than raising exceptions from Activities. In an Activity, any C# exceptions or custom exceptions are converted to a Temporal ApplicationError. In a Workflow, any exceptions that are raised other than an explicit Temporal ApplicationError will only fail that particular Workflow Task and be retried. This includes any typical C# RuntimeErrors that are raised automatically. These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly.

Workflow timeouts​

How to set Workflow timeouts using the Temporal .NET SDK

Each Workflow timeout controls the maximum duration of a different aspect of a Workflow Execution.

Workflow timeouts are set when starting the Workflow Execution.

These values can be set in the WorkflowOptions when calling StartWorkflowAsync or ExecuteWorkflowAsync.

Available timeouts are:

  • ExecutionTimeout
  • RunTimeout
  • TaskTimeout
var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        WorkflowExecutionTimeout = TimeSpan.FromMinutes(5),
    });

Set Workflow retries​

How to set Workflow retries using the Temporal .NET SDK

A Retry Policy can work in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Use a Retry Policy to retry a Workflow Execution in the event of a failure.

Workflow Executions do not retry by default, and Retry Policies should be used with Workflow Executions only in certain situations.

The RetryPolicy can be set in the WorkflowOptions when calling StartWorkflowAsync or ExecuteWorkflowAsync.

var result = await client.ExecuteWorkflowAsync(
    (MyWorkflow wf) => wf.RunAsync(),
    new(id: "my-workflow-id", taskQueue: "my-task-queue")
    {
        RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) },
    });

Activity Timeouts​

How to set Activity Timeouts using the Temporal .NET SDK

Each Activity Timeout controls the maximum duration of a different aspect of an Activity Execution.

The following Timeouts are available in the Activity Options.

An Activity Execution must have either the Start-To-Close or the Schedule-To-Close Timeout set.

These values can be set in the ActivityOptions when calling ExecuteActivityAsync.

Available timeouts are:

  • ScheduleToCloseTimeout
  • ScheduleToStartTimeout
  • StartToCloseTimeout
return await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

Set an Activity Retry Policy​

How to an Activity Retry Policy using the Temporal .NET SDK

A Retry Policy works in cooperation with the timeouts to provide fine controls to optimize the execution experience.

Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided.

To create an Activity Retry Policy in .NET, set the RetryPolicy on the ActivityOptions when calling ExecuteActivityAsync.

return await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(5),
        RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) },
    });

Override the Retry interval with nextRetryDelay​

When you throw an Application Failure and assign the nextRetryDelay field, its value replaces and overrides the Retry interval defined in the active Retry Policy.

For example, you might scale the next Retry delay interval based on the current number of attempts. Here's how you'd do that in an Activity. In the following sample, the attempt count is retrieved from the Activity Execution context and used to set the number of seconds for the next Retry delay:

var attempt = ActivityExecutionContext.Current.Info.Attempt;

throw new ApplicationFailureException(
    $"Something bad happened on attempt {attempt}",
    errorType: "my_failure_type",
    nextRetryDelay: TimeSpan.FromSeconds(3 * attempt));

Heartbeat an Activity​

How to Heartbeat an Activity using the Temporal .NET SDK

An Activity Heartbeat is a ping from the Worker Process that is executing the Activity to the Temporal Service. Each Heartbeat informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed. If the Temporal Service does not receive a Heartbeat within a Heartbeat Timeout time period, the Activity will be considered failed and another Activity Task Execution may be scheduled according to the Retry Policy.

Heartbeats may not always be sent to the Temporal Service—they may be throttled by the Worker.

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected.

Heartbeats can contain a Details field describing the Activity's current progress. If an Activity gets retried, the Activity can access the Details from the last Heartbeat that was sent to the Temporal Service.

To Heartbeat an Activity Execution in .NET, use the Heartbeat() method on the ActivityExecutionContext.

[Activity]
public async Task MyActivityAsync()
{
    while (true)
    {
        // Send heartbeat
        ActivityExecutionContext.Current.Heartbeat();

        // Do some work, passing the cancellation token
        await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken);
    }
}

In addition to obtaining cancellation information, Heartbeats also support detail data that persists on the server for retrieval during Activity retry. If an Activity calls Heartbeat(123, 456) and then fails and is retried, HeartbeatDetails on the ActivityInfo returns an collection containing 123 and 456 on the next Run.

Set a Heartbeat Timeout​

How to set a Heartbeat Timeout using the Temporal .NET SDK

A Heartbeat Timeout works in conjunction with Activity Heartbeats.

HeartbeatTimeout is a property on ActivityOptions for ExecuteActivityAsync used to set the maximum time between Activity Heartbeats.

await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.MyActivity(param),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(5),
        HeartbeatTimeout = TimeSpan.FromSeconds(30),
    });