Upgrade the Temporal Server
How to upgrade the Temporal Server version
If a newer version of the Temporal Server is available, a notification appears in the Temporal Web UI.
If you are using a version that is older than 1.0.0, reach out to us at community.temporal.io to ask how to upgrade.
First check to see if an upgrade to the database schema is required for the version you wish to upgrade to. If a database schema upgrade is required, it will be called out directly in the release notes. Some releases require changes to the schema, and some do not. We ensure that any consecutive versions are compatible in terms of database schema upgrades, features, and system behavior; however there is no guarantee that there is compatibility between any two non-consecutive versions.
Key considerations
When upgrading the Temporal Server, there are two key considerations to keep in mind:
-
Sequential Upgrades: Temporal Server should be upgraded sequentially. That is, if you're on version (v1.n.x), your next upgrade should be to (v1.n+1.x) or the closest available subsequent version. This sequence should be repeated until your desired version is reached.
-
Data Compatibility: During an upgrade, the Temporal Server either updates or restructures the existing version data to match the data format of the newer version. Temporal Server ensures backward compatibility only between two successive minor versions. Consequently, skipping versions during an upgrade may lead to older data formats becoming unreadable. If the previous data format cannot be interpreted and converted to the newer format, the upgrade process will be unsuccessful.
Step-by-Step Upgrade Procedure:
Upgrading the Temporal Server requires a methodical approach to ensure data integrity, compatibility, and seamless transition between versions. The following documentation outlines the step-by-step process to successfully upgrade your Temporal Server.
When upgrading your Temporal Server version, ensure that you upgrade sequentially.
- Upgrade Database Schema: Before initiating the Temporal Server upgrade, use one of the recommended upgrade tools to update your database schema. This ensures it is aligned with the version of Temporal Server you aim to upgrade to.
- Upgrade Temporal Server: Once the database schema is updated, proceed to upgrade the Temporal Server deployment to the next sequential version.
- Iterative Upgrades (optional): Continue this process (steps 1 and 2) iteratively until you reach the desired Temporal Server version.
By adhering to the above guidelines and following the step-by-step procedure, you can ensure a smooth and successful upgrade of your Temporal Server.
The Temporal Server upgrade updates or rewrites the old version data with the format introduced in the newer version. Because Temporal Server guarantees backward compatibility between two consecutive minor versions, and because older versions of the code are eventually removed from the code base, skipping versions when upgrading might cause older formats to become unrecognizable. If the old format of the data can't be read to be rewritten to the new format, the upgrades fail.
Check the Temporal Server releases and follow these releases in order. You can skip patch versions; use the latest patch of a minor version when upgrading.
Also, be aware that each upgrade requires the History Service to load all Shards and update the Shard metadata, so allow approximately 10 minutes on each version for these processes to complete before upgrading to the next version.
Use one of the upgrade tools to upgrade your database schema to be compatible with the Temporal Server version being upgraded to.
If you are using a schema tools version prior to Temporal Server v1.8.0, we strongly recommend never using the "dryrun" (-y, or --dryrun) options in any of your schema update commands.
Using this option might lead to potential loss of data, as when using it will create a new database and drop your
existing one.
This flag was removed in the 1.8.0 release.
Upgrade Cassandra schema
If you are using Cassandra for your Cluster's persistence, use the temporal-cassandra-tool to upgrade both the default Persistence and Visibility schemas.
Example default schema upgrade:
temporal_v1.2.1 $ temporal-cassandra-tool \
--tls \
--tls-ca-file <...> \
--user <cassandra-user> \
--password <cassandra-password> \
--endpoint <cassandra.example.com> \
--keyspace temporal \
--timeout 120 \
update \
--schema-dir ./schema/cassandra/temporal/versioned
Example visibility schema upgrade:
temporal_v1.2.1 $ temporal-cassandra-tool \
--tls \
--tls-ca-file <...> \
--user <cassandra-user> \
--password <cassandra-password> \
--endpoint <cassandra.example.com> \
--keyspace temporal_visibility \
--timeout 120 \
update \
--schema-dir ./schema/cassandra/visibility/versioned
Upgrade PostgreSQL or MySQL schema
If you are using MySQL or PostgreSQL use the temporal-sql-tool, which works similarly to the temporal-cassandra-tool.
Refer to this Makefile for context.
PostgreSQL
Example default schema upgrade:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal update-schema -d ./schema/postgresql/v96/temporal/versioned
Example visibility schema upgrade:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal_visibility update-schema -d ./schema/postgresql/v96/visibility/versioned
If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your PostgreSQL version first, and then run temporal-sql-tool with the postgres12 plugin, as shown in the following example:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned
MySQL
Example default schema upgrade:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 3036 -u root -pw root --pl mysql --db temporal update-schema -d ./schema/mysql/v57/temporal/versioned/
Example visibility schema upgrade:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 3036 -u root -pw root --pl mysql --db temporal_visibility update-schema -d ./schema/mysql/v57/visibility/versioned/
If you're upgrading MySQL to v8.0.17 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your MySQL version first, and then run temporal-sql-tool with the mysql8 plugin, as shown in the following example:
./temporal-sql-tool \
--tls \
--tls-enable-host-verification \
--tls-cert-file <path to your client cert> \
--tls-key-file <path to your client key> \
--tls-ca-file <path to your CA> \
--ep localhost -p 5432 -u temporal -pw temporal --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned.
Roll-out technique
We recommend preparing a staging Cluster and then do the following to verify the upgrade is successful:
- Create some simulation load on the staging cluster.
- Upgrade the database schema in the staging cluster.
- Wait and observe for a few minutes to verify that there is no unstable behavior from both the server and the simulation load logic.
- Upgrade the server.
- Now do the same to the live environment cluster.