Skip to main content

What is the Temporal Visibility feature?

This guide provides a comprehensive overview of Temporal Visibility.

Support, stability, and dependency info
  • For Temporal Server v1.19 and earlier, all supported databases for Visibility provide standard Visibility features, and an Elasticsearch database is required for advanced Visibility features.
  • For Temporal Server v1.20 and later, advanced Visibility features are enabled on all supported SQL databases, in addition to Elasticsearch.
  • In Temporal Server v1.21 and later, standard Visibility is no longer in development, and we recommend migrating to a database that supports Advanced Visibility features. The Visibility configuration for the Temporal Service has been updated and Dual Visibility is enabled. For details, see Visibility store setup.

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service.

The Visibility store in your Temporal Service stores persisted Workflow Execution Event History data and is set up as a part of your Persistence store to enable listing and filtering details about Workflow Executions that exist on your Temporal Service.

With Temporal Server v1.21, you can set up Dual Visibility to migrate your Visibility store from one database to another.

Support for separate standard and advanced Visibility setups will be deprecated from Temporal Server v1.21 onwards. Check Supported databases for updates.

What is standard Visibility?​

Standard Visibility, within the Temporal Platform, is the subsystem and APIs that list Workflow Executions by a predefined set of filters.

Open Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, or Run Id.

Closed Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, Run Id, or Execution Status (Completed, Failed, Timed Out, Terminated, Canceled, or Continued-As-New).

Search Attributes are not supported with Standard Visibility.

Support for standard Visibility is deprecated beginning with Temporal Server v1.21. For updates, check Supported databases.

What is advanced Visibility?​

Visibility, within the Temporal Platform, is the subsystem and APIs that enable the listing, filtering, and sorting of Workflow Executions through a custom SQL-like List Filter.

  • In Temporal Service version 1.20 and later, advanced Visibility is available on SQL databases like MySQL (version 8.0.17 and later) and PostgreSQL (version 12 and later), in addition to support for Elasticsearch.
  • For Temporal Server versions 1.19.1 and earlier, you must integrate with ElasticSearch to use advanced Visibility. Elasticsearch takes on the Visibility request load, relieving potential performance issues. We highly recommend operating a Temporal Service with Elasticsearch for any use case that spawns more than just a few Workflow Executions.
  • On Temporal Cloud, advanced Visibility is enabled by default for all users.

What is Dual Visibility?​

Dual Visibility is a feature that lets you set a secondary Visibility store in addition to a primary store in your Temporal Service. Setting up Dual Visibility is optional and can be used to migrate your Visibility database or create a backup Visibility store.

For example, if you have Cassandra configured as your Visibility database, you can set up a supported SQL database as your secondary Visibility store and gradually migrate your data to the secondary store before deprecating your primary one.

A Dual Visibility setup requires two Visibility store configurations:

  • Primary Visibility: The primary Visibility store where Visibility data is written to and read from by default. The primary Visibility store is set with the visibilityStore configuration key in your Temporal Service.
  • Secondary Visibility: A secondary storage for your Visibility data. The secondary Visibility store is set with the secondaryVisibilityStore configuration key in your Temporal Service.

For configuration details, see Dual Visibility setup.

The following combinations are allowed in a Dual Visibility setting.

PrimarySecondary
Standard (Cassandra or SQL)Advanced (SQL or Elasticsearch)
Advanced (SQL)Advanced (SQL)
Advanced (Elasticsearch)Advanced (Elasticsearch)

With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to write to primary only, secondary only, or to both primary and secondary Visibility stores. When migrating from one Visibility store database to another, set up the database you want to migrate to as your secondary Visibility store.

You can plan your migration using specific dynamic configuration keys that help you transition your read and write operations from the primary to the secondary Visibility store. For details on migrating your Visibility store databases, see Dual Visibility.

What is a List Filter?​

The Visibility List API requires you to provide a List Filter as an SQL-like string parameter.

A List Filter includes Search Attribute names, Search Attribute values, and operators so that it can retrieve a filtered list of Workflow Executions from the Visibility Store.

List Filter Search Attribute names are case sensitive. A single Namespace scopes each List Filter.

A List Filter using a time range provides a resolution of 1 ns on Elasticsearch and 1 µs for SQL databases.

Supported operators​

List Filters support the following operators:

  • =, !=, >, >=, <, <=
  • AND, OR, ()
  • BETWEEN ... AND
  • IN
  • STARTS_WITH

The ORDER BY operator is currently not supported in Temporal Cloud.

Custom Search Attributes of the Text type cannot be used in ORDER BY clauses.

Partial string match​

There are different options for partial string matching when the type of the Search Attribute is Text versus Keyword.

Text​

Search Attributes of type Text are broken up into words that match with the = operator.

For example, if you have a custom Text Search Attribute named Description with either of the following values—

my-business-id-foobar
my business id foobar

—then the following List Filter matches—

Description = 'foobar'

—but a partial word does not:

// Doesn't match
Description = 'foo'

Keyword​

For Search Attributes of type Keyword like WorkflowId, the only kind of partial string matching that works is using BETWEEN for suffixes.

WorkflowId BETWEEN "order-" AND "order-~" matches WorkflowIds that have characters after order- with ASCII values lower than ~ (126, the highest-value printable character), such as the following:

order-
order-1234
order-abracadabra

It does not match order-~~.

Filter Composition Quick Reference

Composition

  • Data types:
    • String literals with single or double quotes
    • Numbers (Integer and Floating Point)
    • Booleans
  • Comparison: =, !=, >, >=, <, <=
  • Expressions/Operators:
    • IN array
    • BETWEEN value AND value
    • STARTS_WITH string
    • IS NULL, IS NOT NULL
    • expr AND expr, expr OR expr, ( expr )
  • Array: ( comma-separated-values )

Please note

  • Wrap attributes with backticks if it contains characters not in [a-zA-Z0-9].
  • STARTS_WITH is only available for Keyword search attributes.

Efficient API usage​

If the Advanced List Filter API retrieves a substantial number of Workflow Executions (more than 10,000), the response time might be longer.

Beginning with Temporal Server v1.20, you can employ the CountWorkflow API to efficiently count the number of Workflow Executions.

To paginate the results using the ListWorkflow API, use the page token to retrieve the next page. Continue until the page token becomes null/nil.

List Filter examples​

Here are examples of List Filters set with the Temporal CLI:

WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00")

When you use the preceding example, you receive a list of Workflows fulfilling the following criteria:

  • Workflow Type is main.YourWorkflowDefinition.
  • Workflow isn't in a running state.
  • Workflow either started after "2021-06-07T16:46:34.236-08:00" or closed after "2021-06-07T16:46:34-08:00".

The following are additional examples of List Filters.

WorkflowId = '<workflow-id>'
WorkflowId = '<workflow-id>' or WorkflowId = '<another-workflow-id>'
WorkflowId IN ('<workflow-id>', '<another-workflow-id>')
WorkflowId = '<workflow-id>' and ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' or ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' and StartTime > '2021-08-22T15:04:05+00:00'
ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00'
ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00'
WorkflowType STARTS_WITH '<workflow-type-prefix>'

What is a Search Attribute?​

A Search Attribute is an indexed field used in a List Filter to filter a list of Workflow Executions that have the Search Attribute in their metadata.

Each Search Attribute is a key-value pair metadata object included in a Workflow Execution's Visibility information. This information is available in the Visibility store.

note

Search Attribute values are not encrypted because the Temporal Server must be able to read these values from the Visibility store when retrieving Workflow Execution details.

Temporal provides some default Search Attributes, such as ExecutionStatus, the current state of your Workflow Executions. You can also create custom Search Attribute keys in your Visibility store and assign values when starting a Workflow Execution or in Workflow code.

When using Continue-As-New or a Temporal Cron Job, Search Attribute keys are carried over to the new Workflow Run by default. Search Attribute values are only available for as long as the Workflow is.

Search Attributes are most effective for search purposes or tasks requiring collection-based result sets. For business logic in which you need to get information about a Workflow Execution, consider one of the following:

  • Storing state in a local variable and exposing it with a Query.
  • Storing state in an external datastore through Activities and fetching it directly from the store.

If your business logic requires high throughput or low latency, store and fetch the data through Activities. You might experience lag due to time passing between the Workflow's state change and the Activity updating the Visibility store.

Default Search Attributes​

A Temporal Service has a set of default Search Attributes already available. Default Search Attributes are set globally in any Namespace. These Search Attributes are created when the initial index is created.

NAMETYPEDEFINITION
BatcherUserKeywordUsed by internal batcher Workflow that runs in TemporalBatcher Namespace division to indicate the user who started the batch operation.
BinaryChecksumsKeyword ListList of binary Ids of Workers that run the Workflow Execution. Deprecated since server version 1.21 in favor of the BuildIds search attribute.
BuildIdsKeyword ListList of Worker Build Ids that have processed the Workflow Execution, formatted as versioned:{BuildId} or unversioned:{BuildId}, or the sentinel unversioned value. Available from server version 1.21.
CloseTimeDatetimeThe time at which the Workflow Execution completed.
ExecutionDurationIntThe time needed to run the Workflow Execution (in nanoseconds). Available only for closed Workflows.
ExecutionStatusKeywordThe current state of the Workflow Execution.
ExecutionTimeDatetimeThe time at which the Workflow Execution actually begins running; same as StartTime for most cases but different for Cron Workflows and retried Workflows.
HistoryLengthIntThe number of events in the history of Workflow Execution. Available only for closed Workflows.
HistorySizeBytesLongThe size of the Event History.
RunIdKeywordIdentifies the current Workflow Execution Run.
StartTimeDatetimeThe time at which the Workflow Execution started.
StateTransitionCountIntThe number of times that Workflow Execution has persisted its state. Available only for closed Workflows.
TaskQueueKeywordTask Queue used by Workflow Execution.
TemporalChangeVersionKeyword ListStores change/version pairs if the GetVersion API is enabled.
TemporalScheduledStartTimeDatetimeThe time that the Workflow is schedule to start according to the Schedule Spec. Can be manually triggered. Set on Schedules.
TemporalScheduledByIdKeywordThe Id of the Schedule that started the Workflow.
TemporalSchedulePausedBooleanIndicates whether the Schedule has been paused. Set on Schedules.
WorkflowIdKeywordIdentifies the Workflow Execution.
WorkflowTypeKeywordThe type of Workflow.

  • All default Search Attributes are reserved and read-only. You cannot create a custom one with the same name or alter the existing one.

  • Search attributes are not encrypted in the system. Do not use PII as either the search attribute name or the value.

  • ExecutionStatus values correspond to Workflow Execution statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut.

  • StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in RFC3339Nano format (such as "2006-01-02T15:04:05.999999999Z07:00").

  • ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, Golang duration format, or "hh:mm:ss" format.

  • CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a closed Workflow Execution.

  • ExecutionTime can differ from StartTime in retry and cron use cases.

You can use the default Search Attributes in a List Filter, such as in the Temporal Web UI or with the temporal workflow list commands, under the following conditions:

  • Without advanced Visibility, you can only use the = operator with a single default Search Attribute in your List Filter. For example: temporal workflow list --query "ExecutionStatus = 'Completed'" or temporal workflow list --query "WorkflowType = 'YourWorkflow'".
  • With advanced Visibility, you can combine default Search Attributes in a List Filter to get a list of specific Workflow Executions. For example: temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and ExecutionStatus != 'Running' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"

Custom Search Attributes​

You can create custom Search Attributes with unique key names that are relevant to your business needs.

Use custom Search Attributes in a List Filter, such as in the Temporal Web UI or with the temporal workflow list commands, under the following conditions:

  • Without advanced Visibility, you cannot use a custom Search Attribute in your List Filter.
  • With advanced Visibility, you can create multiple custom Search Attributes and use them in combinations with List Filters to get a list of specific Workflow Executions. For example: temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and YourCustomSA = 'YourCustomSAValue' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"
    • With Temporal Server v1.19 and earlier, you must integrate Elasticsearch to use custom Search Attributes with List Filters.
    • With Temporal Server v1.20 and later, custom Search Attribute capabilities are available on MySQL (v8.0.17 or later), PostgreSQL (v12 and later), and SQLite (v3.31.0 and later), in addition to Elasticsearch.

If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the supported SQL databases with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.

See custom Search Attributes limits for limits on the number and size of custom Search Attributes you can create.

Supported types​

Custom Search Attributes must be one of the following types:

  • Bool
  • Datetime
  • Double
  • Int
  • Keyword
  • KeywordList
  • Text

Note:

  • Double is backed up by scaled_float Elasticsearch type with scale factor 10000 (4 decimal digits).
  • Datetime is backed up by date type with milliseconds precision in Elasticsearch 6 and date_nanos type with nanoseconds precision in Elasticsearch 7.
  • Int is 64-bit integer (long Elasticsearch type).
  • Keyword and Text types are concepts taken from Elasticsearch. Each word in a Text is considered a searchable keyword. For a UUID, that can be problematic because Elasticsearch indexes each portion of the UUID separately. To have the whole string considered as a searchable keyword, use the Keyword type. For example, if the key ProductId has the value of 2dd29ab7-2dd8-4668-83e0-89cae261cfb1:
    • As a Keyword it would be matched only by ProductId = "2dd29ab7-2dd8-4668-83e0-89cae261cfb1.
    • As a Text it would be matched by ProductId = 2dd8, which could cause unwanted matches.
  • With Temporal Server v1.19 and earlier, the Keyword type can store a list of values.
  • With Temporal Server v1.20 and later, the Keyword type supports only a single value. To store a list of values, use KeywordList.
  • The Text type cannot be used in the "Order By" clause.

Custom Search Attributes limits​

The following table lists the maximum number of custom Search Attributes you can create per Namespace by supported Visibility database.

Search Attribute typeMySQL (v8.0.17 and later)PostgreSQL (v12 and later)SQLite (v3.31.0 and later)Temporal Cloud
Bool33320
Datetime33320
Double33320
Int33320
Keyword10101040
KeywordList3335
Text3335

Temporal does not impose a limit on the number of custom Search Attributes you can create with Elasticsearch. However, Elasticsearch sets a default mapping limit that may apply. Custom Search Attributes are an advanced Visibility feature and are not supported on Cassandra.

Size limits for a custom Search Attribute:

  • The default single Search Attribute value size limit is 2 KB.
  • The maximum total Search Attribute size is 40 KB.
  • The maximum total characters per Search Attribute value is 255.

For Temporal Cloud specific configurations, see the Defaults, limits, and configurable settings -Temporal Cloud guide.

Usage​

Search Attributes available in your Visibility store can be used with Workflow Executions for the Temporal Service. To actually have results from the use of a List Filter, Search Attributes must be added to a Workflow Execution as metadata.

With Workflows you can do the following:

After you add and set your Search Attributes, use your default or custom Search Attributes in a List Filter.