This guide provides a comprehensive overview of Temporal Visibility.
- For Temporal Server v1.19 and earlier, all supported databases for Visibility provide standard Visibility features, and an Elasticsearch database is required for advanced Visibility features.
- For Temporal Server v1.20 and later, advanced Visibility features are enabled on all supported SQL databases, in addition to Elasticsearch.
- In Temporal Server v1.21 and later, standard Visibility is no longer in development, and we recommend migrating to a database that supports Advanced Visibility features. The Visibility configuration for Temporal Clusters has been updated and Dual Visibility is enabled. For details, see Visibility store setup.
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Cluster.
The Visibility store in your Temporal Cluster stores persisted Workflow Execution Event History data and is set up as a part of your Persistence store to enable listing and filtering details about Workflow Executions that exist on your Temporal Cluster.
With Temporal Server v1.21, you can set up Dual Visibility to migrate your Visibility store from one database to another.
What is standard Visibility?
Standard Visibility, within the Temporal Platform, is the subsystem and APIs that list Workflow Executions by a predefined set of filters.
Open Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, or Run Id.
Closed Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, Run Id, or Execution Status (Completed, Failed, Timed Out, Terminated, Canceled, or Continued-As-New).
Search Attributes are not supported with Standard Visibility.
Support for standard Visibility is deprecated beginning with Temporal Server v1.21. For updates, check Supported databases.
What is advanced Visibility?
- In Temporal Cluster version 1.20 and later, advanced Visibility is available on SQL databases like MySQL (version 8.0.17 and later) and PostgreSQL (version 12 and later), in addition to support for Elasticsearch.
- For Temporal Server versions 1.19.1 and earlier, you must integrate with ElasticSearch to use advanced Visibility. Elasticsearch takes on the Visibility request load, relieving potential performance issues. We highly recommend operating a Temporal Cluster with Elasticsearch for any use case that spawns more than just a few Workflow Executions.
- On Temporal Cloud, advanced Visibility is enabled by default for all users.
What is Dual Visibility?
Dual Visibility is a feature that lets you set a secondary Visibility store in addition to a primary store in your Temporal Cluster. Setting up Dual Visibility is optional and can be used to migrate your Visibility database or create a backup Visibility store.
For example, if you have Cassandra configured as your Visibility database, you can set up a supported SQL database as your secondary Visibility store and gradually migrate your data to the secondary store before deprecating your primary one.
A Dual Visibility setup requires two Visibility store configurations:
- Primary Visibility: The primary Visibility store where Visibility data is written to and read from by default. The primary Visibility store is set with the
visibilityStoreconfiguration key in your Temporal Cluster.
- Secondary Visibility: A secondary storage for your Visibility data. The secondary Visibility store is set with the
secondaryVisibilityStoreconfiguration key in your Temporal Cluster.
For configuration details, see Dual Visibility setup.
The following combinations are allowed in a Dual Visibility setting.
|Standard (Cassandra or SQL)||Advanced (SQL or Elasticsearch)|
|Advanced (SQL)||Advanced (SQL)|
|Advanced (Elasticsearch)||Advanced (Elasticsearch)|
With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Cluster to write to primary only, secondary only, or to both primary and secondary Visibility stores. When migrating from one Visibility store database to another, set up the database you want to migrate to as your secondary Visibility store.
You can plan your migration using specific dynamic configuration keys that help you transition your read and write operations from the primary to the secondary Visibility store. For details on migrating your Visibility store databases, see Dual Visibility.
What is a List Filter?
The Visibility List API requires you to provide a List Filter as an SQL-like string parameter.
List Filters support the following operators:
- =, !=, >, >=, <, <=
- AND, OR, ()
- BETWEEN ... AND
Partial string match
Search Attributes of type
Text are broken up into words that match with the
For example, if you have a custom
Text Search Attribute named
Description with either of the following values—
my business id foobar
—then the following List Filter matches—
Description = 'foobar'
—but a partial word does not:
// Doesn't match
Description = 'foo'
For Search Attributes of type
WorkflowId, the only kind of partial string matching that works is using BETWEEN for suffixes.
WorkflowId BETWEEN "order-" AND "order-~" matches WorkflowIds that have characters after
order- with ASCII values lower than
~ (126, the highest-value printable character), such as the following:
It does not match
Efficient API usage
If the Advanced List Filter API retrieves a substantial number of Workflow Executions (more than 10,000), the response time might be longer.
Beginning with Temporal Server v1.20, you can employ the
CountWorkflow API to efficiently count the number of Workflow Executions.
To paginate the results using the
ListWorkflow API, use the page token to retrieve the next page.
Continue until the page token becomes
List Filter examples
Here are examples of List Filters set with
WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00")
When you use the preceding example, you receive a list of Workflows fulfilling the following criteria:
- Workflow Type is
- Workflow isn't in a running state.
- Workflow either started after "2021-06-07T16:46:34.236-08:00" or closed after "2021-06-07T16:46:34-08:00".
The following are additional examples of List Filters.
WorkflowId = '<workflow-id>'
WorkflowId = '<workflow-id>' or WorkflowId = '<another-workflow-id>'
WorkflowId IN ('<workflow-id>', '<another-workflow-id>')
WorkflowId = '<workflow-id>' and ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' or ExecutionStatus = 'Running'
WorkflowId = '<workflow-id>' and StartTime > '2021-08-22T15:04:05+00:00'
ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00'
ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00'
What is a Search Attribute?
Each Search Attribute is a key-value pair metadata object included in a Workflow Execution's Visibility information. This information is available in the Visibility store.
Search Attribute values are not encrypted because the Temporal Server must be able to read these values from the Visibility store when retrieving Workflow Execution details.
Temporal provides some default Search Attributes, such as
ExecutionStatus, the current state of your Workflow Executions.
You can also create custom Search Attribute keys in your Visibility store and assign values when starting a Workflow Execution or in Workflow code.
When using Continue-As-New or a Temporal Cron Job, Search Attribute keys are carried over to the new Workflow Run by default. Search Attribute values are only available for as long as the Workflow is.
Search Attributes are most effective for search purposes or tasks requiring collection-based result sets. For business logic in which you need to get information about a Workflow Execution, consider one of the following:
- Storing state in a local variable and exposing it with a Query.
- Storing state in an external datastore through Activities and fetching it directly from the store.
If your business logic requires high throughput or low latency, store and fetch the data through Activities. You might experience lag due to time passing between the Workflow's state change and the Activity updating the Visibility store.
Default Search Attributes
A Temporal Cluster has a set of default Search Attributes already available. Default Search Attributes are set globally in any Namespace. These Search Attributes are created when the initial index is created.
|BatcherUser||Keyword||Used by internal batcher Workflow that runs in |
|BinaryChecksums||Keyword List||List of binary Ids of Workers that run the Workflow Execution. Deprecated since server version 1.21 in favor of the |
|BuildIds||Keyword List||List of Worker Build Ids that have processed the Workflow Execution, formatted as |
|CloseTime||Datetime||The time at which the Workflow Execution completed.|
|ExecutionDuration||Int||The time needed to run the Workflow Execution (in nanoseconds). Available only for closed Workflows.|
|ExecutionStatus||Keyword||The current state of the Workflow Execution.|
|ExecutionTime||Datetime||The time at which the Workflow Execution actually begins running; same as |
|HistoryLength||Int||The number of events in the history of Workflow Execution. Available only for closed Workflows.|
|HistorySizeBytes||Long||The size of the Event History.|
|RunId||Keyword||Identifies the current Workflow Execution Run.|
|StartTime||Datetime||The time at which the Workflow Execution started.|
|StateTransitionCount||Int||The number of times that Workflow Execution has persisted its state. Available only for closed Workflows.|
|TaskQueue||Keyword||Task Queue used by Workflow Execution.|
|TemporalChangeVersion||Keyword List||Stores change/version pairs if the GetVersion API is enabled.|
|TemporalScheduledStartTime||Datetime||The time that the Workflow is schedule to start according to the Schedule Spec. Can be manually triggered. Set on Schedules.|
|TemporalScheduledById||Keyword||The Id of the Schedule that started the Workflow.|
|TemporalSchedulePaused||Boolean||Indicates whether the Schedule has been paused. Set on Schedules.|
|WorkflowId||Keyword||Identifies the Workflow Execution.|
|WorkflowType||Keyword||The type of Workflow.|
All default Search Attributes are reserved and read-only. You cannot create a custom one with the same name or alter the existing one.
ExecutionStatus values correspond to Workflow Execution statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut.
StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in RFC3339Nano format (such as "2006-01-02T15:04:05.999999999Z07:00").
ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, Golang duration format, or "hh:mm:ss" format.
CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a closed Workflow Execution.
ExecutionTime can differ from StartTime in retry and cron use cases.
You can use the default Search Attributes in a List Filter, such as in the Temporal Web UI or with the
tctl workflow list commands, under the following conditions:
- Without advanced Visibility, you can only use the
=operator with a single default Search Attribute in your List Filter. For example:
tctl workflow list -q "ExecutionStatus = 'Completed'"or
tctl workflow list -q "WorkflowType = 'YourWorkflow'".
- With advanced Visibility, you can combine default Search Attributes in a List Filter to get a list of specific Workflow Executions.
tctl workflow list -q "WorkflowType = 'main.YourWorkflowDefinition' and ExecutionStatus != 'Running' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"
Custom Search Attributes
You can create custom Search Attributes with unique key names that are relevant to your business needs.
Use custom Search Attributes in a List Filter, such as in the Temporal Web UI or with the
tctl workflow list commands, under the following conditions:
- Without advanced Visibility, you cannot use a custom Search Attribute in your List Filter.
- With advanced Visibility, you can create multiple custom Search Attributes and use them in combinations with List Filters to get a list of specific Workflow Executions.
tctl workflow list -q "WorkflowType = 'main.YourWorkflowDefinition' and YourCustomSA = 'YourCustomSAValue' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"
- With Temporal Server v1.19 and earlier, you must integrate Elasticsearch to use custom Search Attributes with List Filters.
- With Temporal Server v1.20 and later, custom Search Attribute capabilities are available on MySQL (v8.0.17 or later), PostgreSQL (v12 and later), and SQLite (v3.31.0 and later), in addition to Elasticsearch.
If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the supported SQL databases with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.
See custom Search Attributes limits for limits on the number and size of custom Search Attributes you can create.
Custom Search Attributes must be one of the following types:
- KeywordList (not supported on Temporal Cloud)
- Double is backed up by
scaled_floatElasticsearch type with scale factor 10000 (4 decimal digits).
- Datetime is backed up by
datetype with milliseconds precision in Elasticsearch 6 and
date_nanostype with nanoseconds precision in Elasticsearch 7.
- Int is 64-bit integer (
- Keyword and Text types are concepts taken from Elasticsearch. Each word in a Text is considered a searchable keyword.
For a UUID, that can be problematic because Elasticsearch indexes each portion of the UUID separately.
To have the whole string considered as a searchable keyword, use the Keyword type.
For example, if the key
ProductIdhas the value of
- As a Keyword it would be matched only by
ProductId = "2dd29ab7-2dd8-4668-83e0-89cae261cfb1.
- As a Text it would be matched by
ProductId = 2dd8, which could cause unwanted matches.
- As a Keyword it would be matched only by
- With Temporal Server v1.19 and earlier, the Keyword type can store a list of values.
- With Temporal Server v1.20 and later, the Keyword type supports only a single value. To store a list of values, use KeywordList.
- The Text type cannot be used in the "Order By" clause.
Custom Search Attributes limits
The following table lists the maximum number of custom Search Attributes you can create per Namespace by supported Visibility database.
|Search Attribute type||MySQL (v8.0.17 and later)||PostgreSQL (v12 and later)||SQLite (v3.31.0 and later)||Temporal Cloud|
Temporal does not impose a limit on the number of custom Search Attributes you can create with Elasticsearch. However, Elasticsearch sets a default mapping limit that may apply. Custom Search Attributes are an advanced Visibility feature and are not supported on Cassandra.
Size limits for a custom Search Attribute:
- The default single Search Attribute value size limit is 2 KB.
- The maximum total Search Attribute size is 40 KB.
- The maximum total characters per Search Attribute value is 255.
For Temporal Cloud specific configurations, see the Default limits sheet.
Search Attributes available in your Visibility store can be used with Workflow Executions for that Cluster. To actually have results from the use of a List Filter, Search Attributes must be added to a Workflow Execution as metadata.
- To create custom Search Attributes in your Visibility store, see Create custom Search Attributes.
- To remove a custom Search Attribute from the Visbility store, see Remove custom Search Attributes. Removing custom Search Attributes is not supported on Temporal Cloud.
- To rename a custom Search Attribute on Temporal Cloud, see
tcld namespace search-attributes rename.
With Workflows you can do the following:
Set the value of Search Attributes in your Workflow
Update the value set for a Search Attribute from within the Workflow code
Remove the value set for a Search Attribute from within the Workflow code
To get a list of Search Attributes using
tctl, see How to view Search Attributes using tctl.
After you add and set your Search Attributes, use your default or custom Search Attributes in a List Filter.