Skip to main content

Codec Server - Temporal Platform feature guide

Temporal Server stores and persists the data handled in your Workflow Execution. Encrypting this data ensures that any sensitive application data is secure when handled by the Temporal Server.

For example, if you have sensitive information passed in the following objects that are persisted in the Workflow Execution Event History, use encryption to secure it:

Using encryption ensures that your sensitive data exists unencrypted only on the Client and the Worker Process that is executing the Workflows and Activities, on hosts that you control.

By default, your data is serialized to a Payload by a Data Converter. To encrypt your Payload, configure your custom encryption logic with a Payload Codec and set it with a custom Data Converter.

A Payload Codec does byte-to-byte conversion to transform your Payload (for example, by implementing compression and/or encryption and decryption) and is an optional step that happens between the Client and the Payload Converter:

Remote data encoding architecture

Remote data encoding architecture

You can run your Payload Codec with a Codec Server and use the Codec Server endpoints in the Web UI and CLI to decode your encrypted Payload locally. For details on how to set up a Codec Server, see Codec Server setup.

However, if you plan to set up remote data encoding for your data, ensure that you consider all security implications of running encryption remotely before implementing it.

When implementing a custom codec, it is recommended to perform your compression or encryption on the entire input Payload and store the result in the data field of a new Payload with a different encoding metadata field. This ensures that the input Payload's metadata is preserved. When the encoded Payload is sent to be decoded, you can verify the metadata field before applying the decryption. If your Payload is not encoded, it is recommended to pass the unencoded data to the decode function instead of failing the conversion.

Examples for implementing encryption:

Codec Server setup​

Use a Codec Server to programmatically decode your encoded payloads.

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.

The Codec Server is independent of the Temporal Server and decodes your encrypted payloads through endpoints. When you configure a Codec Server endpoint in the Temporal Web UI or CLI, the Web UI and CLI use the remote endpoint to receive decoded payloads from the Codec Server. See API contract requirements.

Decoded payloads can then be displayed in the Workflow Execution Event History on the Web UI. Note that when you use a Codec Server, the decoded payloads are decoded and returned on the client side only; payloads on the Temporal Server (whether on Temporal Cloud or a self-hosted Temporal Service) remain encrypted.

Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following:

  • When you register a Codec Server endpoint with your Web UI, expect the Codec Server to receive multiple requests per Workflow Execution.
  • Ensure that you secure access to your Codec Server. For details, see Authorization. You might need some form of Key management infrastructure for sharing your encryption keys between the Workers and your Codec Server.
  • You will need to enable CORS on the HTTP/HTTPS endpoints in your Codec Server to receive requests from the Temporal Web UI.
  • You may introduce latency in the Web UI when sending and receiving payloads to the Codec Server.

Your Codec Server should share logic with the custom Payload Codec used elsewhere in your application.

API contract specifications​

When you create your Codec Server to handle requests from the Web UI, the following requirements must be met.

Endpoints​

The Web UI and CLI send a POST to a /decode endpoint. In your Codec Server, create a /decode path and pass the incoming payload to the decode method in your Payload Codec.

For examples on how to create your Codec Server, see the following Codec Server implementation samples:

You can also add a verification step to check whether the incoming request has the required authorization to access the decode logic in your Payload Codec.

Headers​

Each request from the Web UI to your Codec Server includes the following headers:

  • Content-Type: application/json: Ensure that your Codec Server can accommodate this MIME type.

  • X-Namespace: {namespace}: This is a custom HTTP Header. Ensure that the CORS configuration in your Codec Server includes this header.

  • [Optional] Authorization: <credentials>: Include this in your CORS configuration when enabling authorization with your Codec Server.

For details on setting up authorization, see Authorization.

Request body​

The general specification for the POST request body contains payloads. By default, all field values in your payload are base64 encoded, regardless of whether they are encrypted by your custom codec implementation.

The following example shows a sample POST request body with base64 encoding.

{
  "payloads": [{
    "metadata": {
      "encoding": <base64EncodedEncodingHint>
    },
    "data": <encryptedPayloadData>
  }, ...]
}

CORS​

By default, in cross-origin Fetch/XHR invocations, browsers will not send credentials. Enable Cross-Origin Resource Sharing (CORS) requests on your Codec Server to receive HTTP/HTTPS requests from the Temporal Web UI.

At a minimum, enable the following responses from your Codec Server to allow requests coming from the Temporal Web UI:

  • Access-Control-Allow-Origin
  • Access-Control-Allow-Methods
  • Access-Control-Allow-Headers

For example, for Temporal Cloud Web UI hosted at https://cloud.temporal.io, enable the following in your Codec Server:

  • Access-Control-Allow-Origin: https://cloud.temporal.io
  • Access-Control-Allow-Methods: POST, GET, OPTIONS
  • Access-Control-Allow-Headers: X-Namespace, Content-Type

For details on what a sample request/response looks like from the Temporal Web UI, see Sample Request/Response. If setting authorization, include Authorization in your Access-Control-Allow-Headers. For details on setting up authorization, see Authorization.

Authorization​

It is important to establish how you will provide access to your Codec Server. Because it is designed to decode potentially sensitive data with a single API call, access to a production Codec Server should be restricted.

Depending on your infrastructure and risk levels, it might be sufficient to restrict HTTP ingress to your Codec Server (such as by using a VPN like WireGuard). The Temporal Web UI can communicate with a Codec Server that is only accessible on localhost, so this is a legitimate security pattern. However, if your Codec Server is exposed to the internet at all, you will likely need an authentication solution.

If you are already using an organization-wide authentication provider, you should integrate it with your Codec Server. Remember, a Codec Server is just a standalone HTTP server, so you can use existing libraries for OAuth, Auth0, or any other protocol. This repository contains an example of using Auth0 to handle browser-based auth to a Codec Server.

To enable authorization from the Web UI (for both a self-hosted Temporal Service and Temporal Cloud), your Codec Server must be an HTTPS Server.

Temporal Cloud

The Temporal Cloud UI provides an option to pass access tokens (JWT) to your Codec Server endpoints. Use the access tokens to validate access and then return decoded payloads from the Codec Server.

You can enable this by selecting Pass access token in your Codec Server endpoint interface where you add your endpoint. Enabling this option in the Temporal Cloud UI adds an authorization header to each request sent to the Codec Server endpoint that you set.

In your Codec Server implementation, verify the signature on this access token (in your authorization header) against our JWKS endpoint.

The token provided from Temporal Cloud UI contains the email identifier of the person requesting access to the payloads. Based on the permissions you have provided to the user in your access control systems, set conditions in your Codec Server whether to return decoded payloads or just return the original encoded payloads.

Self-hosted Temporal Service

On a self-hosted Temporal Service, configure authorization in the Web UI configuration in your Temporal Service setup.

With this enabled, you can pass access tokens to your Codec Server and validate the requests from the Web UI to the Codec Server endpoints that you set. Note that with a self-hosted Temporal Service, you must explicitly configure authorization specifications for the Web UI and CLI.

Sample request/response​

Consider the following sample request/response when creating and hosting a Codec Server with the following specifications:

  • Scheme: https
  • Host: dev.mydomain.com/codec
  • Path: /decode
HTTP/1.1 POST /decode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"anNvbi9wcm90b2J1Zg==","messageType":"dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="},"data":"eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"}]}

200 OK
Content-Type: application/json

{
  "payloads": [{
    "metadata":{
      "encoding": "json/protobuf",
      "messageType": "temporal_shop.orchestrations.v1.StartShoppingCartRequest"
    },
    "data":{
      "cartId":"example-cart",
      "shopperId":"your-shopper-id-example",
      "email":"your-email@domain.com"
    }}]
}

You can also perform remote encoding on an /encode endpoint, which looks the same in reverse:

  • Scheme: https
  • Host: dev.mydomain.com/codec
  • Path: /encode
HTTP/1.1 POST /encode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"json/protobuf","messageType":"temporal_shop.orchestrations.v1.StartShoppingCartRequest"},"data":{"cartId":"example-cart","shopperId":"your-shopper-id-example","email":"your-email@domain.com"}}]}

200 OK
Content-Type: application/json

{
  "payloads": [
    {
      "metadata": {
        "encoding": "anNvbi9wcm90b2J1Zg==",
        "messageType": "dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="
      },
      "data": "eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"
    }
  ]
}

Set your Codec Server endpoints with Web UI and CLI​

After you create your Codec Server and expose the requisite endpoints, set the endpoints in your Web UI and CLI.

Web UI​

On Temporal Cloud and self-hosted Temporal Service, you can configure a Codec Server endpoint to be used for a Namespace in the Web UI.

Codec Server endpoint Namespace setting

Codec Server endpoint Namespace setting

To set a Codec Server endpoint on a Namespace, do the following.

  1. In the Web UI, go to Namespaces, select the Namespace where you want to configure the Codec Server endpoint, and click Edit.
  2. In the Codec Server section on the Namespace configuration page, enter your Codec Server endpoint and port number.
  3. Optional: If your Codec Server is configured to authenticate requests from Temporal Web UI, enable Pass access token to send a JWT access token with the HTTPS requests.
  4. Optional: If your Codec Server is configured to verify origins of requests, enable Include cross-origin credentials.

On Temporal Cloud, you must have Namespace Admin privileges to add a Codec Server endpoint on the Namespace. Setting a Codec Server endpoint on a Cloud Namespace enables it for all users on the Namespace.

Setting a Codec Server endpoint on a self-hosted Temporal Service enables it for the entire Temporal Service. You can use a single Codec Server to handle different encoding and decoding routes for each namespace.

You can also override the global Codec Server setting at the browser level. This can be useful when developing, testing, or troubleshooting encoding functionality.

Codec Server endpoint browser setting

Codec Server endpoint browser setting

To set a browser override for the Namespace-level endpoint, do the following.

  1. Navigate to Workflows in your Namespace.
  2. In the top-right corner, select Configure Codec Server.
  3. Select whether you want to use the Namespace-level (or Temporal Service-level for self-hosted Temporal Service) or the browser-level Codec Endpoint setting as the default for your browser. In Temporal Cloud:
    • Use Namespace-level settings, where available. Otherwise, use my browser setting. Uses the Namespace-level Codec Server endpoint by default. If no endpoint is set on the Namespace, your browser setting is applied.
    • Use my browser setting and ignore Namespace-level setting. Applies your browser-level setting by default, overriding the Namespace-level Codec Server endpoint.
  4. Enter your Codec Server endpoint and port number.
  5. Optional: If your Codec Server is configured to authenticate requests from Temporal Web UI, enable Pass access token to send a JWT access token with the HTTPS requests.
  6. Optional: If your Codec Server is configured to verify origins of requests, enable Include cross-origin credentials.

In a self-hosted Temporal Service with dedicated UI Server configuration, you can also set the codec endpoint in the UI server configuration file:

codec:
    endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}}

CLI​

You can configure a Codec Server endpoint with the Temporal CLI using the --codec_endpoint flag.

For example, if you are running your Codec Server on http://localhost:8888, you can use env set to set the endpoint globally:

temporal env set --codec-endpoint "http://localhost:8888"

If your Codec Server endpoint is not set globally, provide the --codec-endpoint option with each command. For example, to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace", run:

temporal --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow"  --run-id "<yourRunId>" --output "table"

For details, see the CLI reference.

If your Codec Server requires authentication, the Temporal CLI will also accept a --codec-auth parameter to supply an authorization header:

temporal workflow show \
   --workflow-id converters_workflowID \
   --codec-endpoint 'http://localhost:8081/{namespace}' \
   --codec-auth 'auth-header'

Working with Large Payloads​

Codec Servers can be used for more than encryption and decryption of sensitive data. Codec Server behavior is left up to implementers -- they can also call external services or perform other tasks, as long as they hook in at the encoding and decoding stages of a Workflow payload.

By default, Temporal limits payload size to 4MB. If this limitation is problematic for your use case, you could implement a codec that persists your payloads to an object store outside of workflow histories. An example implementation is available from DataDog.

Temporal Nexus​

The Data Converter works the same for a Nexus Operation as it does for other payloads sent between a Worker and Temporal Cloud. Both the caller and handler Workers must use compatible Data Converters to pass operation inputs and results between them.

See Nexus Payload Encryption & Data Converter for details.