A Data Converter is a Temporal SDK component that encodes and decodes data entering and exiting a Temporal Server.
Data is encoded before it is sent to a Temporal Server, and it is decoded when it is received from a Temporal Server.
The main pieces of data that run through the Data Converter are arguments and return values:
- The Client:
- Encodes Workflow, Signal, and Query arguments
- Decodes Workflow and Query return values
- The Worker:
- Decodes Workflow, Signal, and Query arguments
- Encodes Workflow and Query return values
- Decodes and encodes Activity arguments and return values
Each piece of data (like a single argument or return value) is encoded as a
Payload Protobuf message, which consists of binary
data and key-value
Default Data Converter
Each Temporal SDK includes a default Data Converter. In most SDKs, the default converter supports binary, JSON, and Protobufs. (In SDKs that cannot determine parameter types at runtime—like TypeScript—Protobufs aren't included in the default converter.) It tries to encode values in the following order:
- Protobuf JSON
- If a value is an instance of a Protobuf message, it will be encoded with proto3 JSON.
- If a value isn't null, binary, or a Protobuf, it will be encoded as JSON. If any part of it is not serializable as JSON (for example, a Date—see JSON data types), an error will be thrown.
The default converter also supports decoding binary Protobufs.
Custom Data Converter
Applications can create their own custom Data Converters to alter the format (for example using MessagePack instead of JSON) or add compression or encryption.
To use a custom Data Converter, provide it to:
- The Client and Worker in the SDKs you use
- Temporal Web via
tctl data-converter web(for displaying decoded data in the Web UI)
--data-converter-plugin(for displaying decoded headers in
Custom Data Converters are not applied to all data:
searchAttributesare always encoded with JSON
- Headers are not encoded by the SDK (the one exception will be—once implemented—the SDK running OTel baggage through custom Codecs)
- First, a Payload Converter converts a value into a
- Then, a Payload Codec transforms an array of Payloads (for example, a list of Workflow arguments) into another array of Payloads
The Payload Codec is an optional step that happens between the wire and the Payload Converter:
Temporal Server <--> Wire <--> Payload Codec <--> Payload Converter <--> User code
Common Payload Codec transformations are compression and encryption.
In codec implementations, we recommended running the function (whether it be compressing, encrypting, etc) on the entire input Payload, and putting the result in a new Payload's
data field. That way, the input Payload's headers are preserved. See, for example:
Doing encryption in a custom Data Converter ensures that all application data is encrypted while:
- Being sent to/from Temporal Server
- Moving inside Temporal Server
- Stored by Temporal Server
Then data only exists unencrypted in memory on the Client and in the Worker Process that is executing Workflows and Activities on hosts that the application developer controls.
Our encryption samples use AES GCM with 256-bit keys: