Skip to main content
Version: june-2022

What is a Data Converter?

A Data Converter is a Temporal SDK component that encodes and decodes data entering and exiting a Temporal Server.

Data Converter encodes and decodes data

Data is encoded before it is sent to a Temporal Server, and it is decoded when it is received from a Temporal Server.

The main pieces of data that run through the Data Converter are arguments and return values:

  • The Client:
    • Encodes Workflow, Signal, and Query arguments
    • Decodes Workflow and Query return values
  • The Worker:
    • Decodes Workflow, Signal, and Query arguments
    • Encodes Workflow and Query return values
    • Decodes and encodes Activity arguments and return values

Each piece of data (like a single argument or return value) is encoded as a Payload Protobuf message, which consists of binary data and key-value metadata.

Default Data Converter

Each Temporal SDK includes a default Data Converter. In most SDKs, the default converter supports binary, JSON, and Protobufs. (In SDKs that cannot determine parameter types at runtime—like TypeScript—Protobufs aren't included in the default converter.) It tries to encode values in the following order:

  • Null
  • Binary
  • Protobuf JSON
  • JSON

For example:

  • If a value is an instance of a Protobuf message, it will be encoded with proto3 JSON.
  • If a value isn't null, binary, or a Protobuf, it will be encoded as JSON. If any part of it is not serializable as JSON (for example, a Date—see JSON data types), an error will be thrown.

The default converter also supports decoding binary Protobufs.

Custom Data Converter

Applications can create their own custom Data Converters to alter the format (for example using MessagePack instead of JSON) or add compression or encryption.

To use a custom Data Converter, provide it to:

Custom Data Converters are not applied to all data:

Payload Codecs

In TypeScript and Go, data conversion happens in two stages:

  • First, a Payload Converter converts a value into a Payload
  • Then, a Payload Codec transforms an array of Payloads (for example, a list of Workflow arguments) into another array of Payloads

The Payload Codec is an optional step that happens between the wire and the Payload Converter:

Temporal Server <--> Wire <--> Payload Codec <--> Payload Converter <--> User code

Common Payload Codec transformations are compression and encryption.

In codec implementations, we recommended running the function (whether it be compressing, encrypting, etc) on the entire input Payload, and putting the result in a new Payload's data field. That way, the input Payload's headers are preserved. See, for example:

Encryption

Doing encryption in a custom Data Converter ensures that all application data is encrypted while:

  • Being sent to/from Temporal Server
  • Moving inside Temporal Server
  • Stored by Temporal Server

Then data only exists unencrypted in memory on the Client and in the Worker Process that is executing Workflows and Activities on hosts that the application developer controls.

Our encryption samples use AES GCM with 256-bit keys: