Skip to main content

Temporal Platform Visibility

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.

Standard Visibility#

Standard Visibility, within the Temporal Platform, is the subsystem and APIs that list Workflow Executions by a predefined set of filters.

Open Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, or Run Id.

Closed Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, Run Id, or Execution Status (Completed, Failed, Timed Out, Terminated, Cancelled, or Continued-As-New).

Advanced Visibility#

Advanced Visibility, within the Temporal Platform, is the subsystem and APIs that enable the listing, filtering, and sorting of Workflow Executions via a custom SQL-like List Filter.

To use Advanced Visibility, your Temporal Cluster must be integrated with Elasticsearch. We highly recommend operating a Temporal Cluster with Elasticsearch for any use case that spawns more than just a few Workflow Executions. Elasticsearch takes on the Visibility request load, relieving potential performance issues.

List Filters#

A List Filter is the SQL-like string that is provided as the parameter to an Advanced Visibility List API.

The following is an example List Filter:

WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00") order by StartTime desc

More example List Filters

A List Filter contains Search Attribute keys, Search Attribute values, and Operators.

  • The following operators are supported in List Filters:

    • AND, OR, ()
    • =, !=, >, >=, <, <=
    • IN
    • BETWEEN ... AND
    • ORDER BY
  • A List Filter applies to a single Namespace.

  • The range of a List Filter timestamp (StartTime, CloseTime, ExecutionTime) cannot exceed 9223372036854775807 (that is, maxInt64 - 1001).

  • A List Filter that uses a time range has a resolution of 1 ms on Elasticsearch 6 and 1 ns on Elasticsearch 7.

  • List Filter Search Attribute key names are case sensitive.

  • An Advanced List Filter API may take longer than expected if it is retrieving a large number of Workflow Executions (more than 10 million).

  • A ListWorkflow API supports pagination. Use the page token in the following call to retrieve the next page; continue until the page token is null/nil.

  • To paginate through a large number of Workflow Executions without skipping or duplicating them, use the ScanWorkflow API.

  • To efficiently count the number of Workflow Executions, use the CountWorkflow API.

Example List Filters#

WorkflowId = '<workflow-id>'
WorkflowId = '<workflow-id>' or WorkflowId = '<another-workflow-id>'
WorkflowId = '<workflow-id>' order by StartTime desc
WorkflowId = '<workflow-id>' and ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' or ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' and StartTime > '2021-08-22T15:04:05+00:00'
ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00'
ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00'
order by ExecutionTime
order by StartTime desc, CloseTime asc
order by CustomIntField asc

Search Attributes#

A Search Attribute is an indexed key used in a List Filter to filter a list of Workflow Executions that have the Search Attribute in their metadata.

note

If a Temporal Cluster does not have Elasticsearch integrated, but a Workflow Execution is spawned and tagged with Search Attributes, no errors occur. However, you won't be able to use Advanced Visibility List APIs and List Filters to find and list the Workflow Execution.

When using Continue-As-New or a Temporal Cron Job, Search Attributes are carried over to the new Run by default.

Default Search Attributes#

A Temporal Cluster that is integrated with Elasticsearch has a set of default Search Attributes already available. These Search Attributes are created when the initial index is created.

NAMETYPE
BatcherNamespaceKeyword
BatcherUserKeyword
BinaryChecksumsKeyword
CloseTimeDatetime
ExecutionDurationInt
ExecutionStatusKeyword
ExecutionTimeDatetime
HistoryLengthInt
RunIdKeyword
StartTimeDatetime
StateTransitionCountInt
TaskQueueKeyword
TemporalChangeVersionKeyword
WorkflowIdKeyword
WorkflowTypeKeyword
  • All default Search Attributes are reserved and read-only. (You cannot create a custom one with the same name or alter the existing one.)

  • ExecutionStatus values correspond to Workflow Execution Statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut.

  • StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in RFC3339Nano format (such as "2006-01-02T15:04:05.999999999Z07:00").

  • ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, Golang duration format, or "hh:mm:ss" format.

  • CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a Closed Workflow Execution.

  • ExecutionTime can differ from StartTime in retry and cron use cases.

Custom Search Attributes#

Custom Search Attribute keys must be added to a Temporal Cluster using tctl. Adding a Search Attribute key makes it available to use with Workflow Executions within that Cluster.

There is no hard limit on the number of attributes you can add. However, we recommend enforcing the following limits:

  • Number of keys: 100 per Workflow
  • Size of each value: 2 KB per value
  • Total size of keys and values: 40 KB per Workflow
note

Due to Elasticsearch limitations, you can only add Search Attributes. It is not possible to rename Search Attribute keys or remove them from the index schema.

Search Attributes must be one of the following types:

  • String
  • Keyword
  • Int
  • Double
  • Bool
  • Datetime

The temporalio/auto-setup Docker image uses a pre-defined set of custom Search Attributes that are handy for testing. Their names indicate their types:

  • CustomStringField
  • CustomKeywordField
  • CustomIntField
  • CustomDoubleField
  • CustomBoolField
  • CustomDatetimeField

Note:

  • Double is backed up by scaled_float Elasticsearch type with scale factor 10000 (4 decimal digits).
  • Datetime is backed up by date type with milliseconds precision in Elasticsearch 6 and date_nanos type with nanoseconds precision in Elasticsearch 7.
  • Int is 64-bit integer (long Elasticsearch type).
  • Keyword and String types are concepts taken from Elasticsearch. Each word in a String is considered a searchable keyword. For a UUID, that can be problematic because Elasticsearch indexes each portion of the UUID separately. To have the whole string considered as a searchable keyword, use the Keyword type. For example, if the key ProductId has the value of 2dd29ab7-2dd8-4668-83e0-89cae261cfb1:
    • As a Keyword it would be matched only by ProductId = "2dd29ab7-2dd8-4668-83e0-89cae261cfb1.
    • As a String it would be matched by ProductId = 2dd8, which could cause unwanted matches.
  • The String type cannot be used in the "Order By" clause.

Search Attributes as Workflow Execution metadata#

To actually have results from the use of a List Filter, Search Attributes must be added to a Workflow Execution as metadata. How to do this entirely depends on the method by which you spawn the Workflow Execution:

Get notified of updates