Skip to main content

How to integrate Elasticsearch into a Temporal Cluster

To integrate Elasticsearch with your Temporal Cluster, edit the persistence section of your development.yaml configuration file and run the index schema setup commands.

Elasticsearch is required to support Advanced Visibility.

note

These steps are needed only if you have a "plain" Temporal Server Docker image.

If you operate a Temporal Cluster using our Helm charts or docker-compose, the Elasticsearch index schema and index are created automatically using the auto-setup Docker image.

Supported versions
  • Elasticsearch v7.10 is supported from Temporal version 1.7.0 onwards
  • Elasticsearch v6.8 is supported in all Temporal versions
  • Both versions are explicitly supported with AWS Elasticsearch

Edit persistence#

  1. Add the advancedVisibilityStore: es-visibility key-value pair to the persistence section. The development_es.yaml file in the temporalio/temporal repo is a working example. The configuration instructs the Temporal Cluster how and where to connect to Elasticsearch storage.
persistence:  ...  advancedVisibilityStore: es-visibility
  1. Define the Elasticsearch datastore connection information under the es-visibility key:
persistence:  ...  advancedVisibilityStore: es-visibility  datastores:    ...    es-visibility:      elasticsearch:        version: "v7"        url:          scheme: "http"          host: "127.0.0.1:9200"        indices:          visibility: temporal_visibility_v1_dev

Create index schema and index#

Run the following commands to create the index schema and index:

docker/auto-setup.sh

    # ES_SERVER is the URL of Elasticsearch server i.e. "http://localhost:9200".    SETTINGS_URL="${ES_SERVER}/_cluster/settings"    SETTINGS_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json    TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template"    SCHEMA_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json    INDEX_URL="${ES_SERVER}/${ES_VIS_INDEX}"    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n"    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n"    curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n"

Set Elasticsearch privileges#

Ensure that the following privileges are granted for the Elasticsearch Temporal index:

Add custom Search Attributes (optional)#

This step is optional and useful for testing custom Search Attributes.

Run the following tctl command to add six custom Search Attributes to the Elasticsearch Temporal index:

docker/auto-setup.sh

      tctl --auto_confirm admin cluster add-search-attributes \          --name CustomKeywordField --type Keyword \          --name CustomStringField --type Text \          --name CustomTextField --type Text \          --name CustomIntField --type Int \          --name CustomDatetimeField --type Datetime \          --name CustomDoubleField --type Double \          --name CustomBoolField --type Bool

Get notified of updates