Data Converters
Background reading: Data Converters in Temporal
Contents:
Default Data Converter
In TypeScript, the default Data Converter supports:
undefined
Uint8Array
- JSON
Custom Data Converter
API doc: DataConverter
To send values that are not JSON-serializable like BigInt
s or Date
s, provide a custom Data Converter to the Client and Worker:
Data Converters have two parts:
PayloadConverter
: sync methods that sometimes run inside the Workflow isolate (and are thus limited)PayloadCodec
: async methods that are run outside the isolate
- TypeScript
- JavaScript
interface DataConverter {
payloadConverterPath?: string;
payloadCodec?: PayloadCodec;
}
PayloadConverter
API doc: PayloadConverter
- TypeScript
- JavaScript
interface PayloadConverter {
/**
* Converts a value to a {@link Payload}.
* @param value The value to convert. Example values include the Workflow args sent by the client and the values returned by a Workflow or Activity.
*/
toPayload<T>(value: T): Payload;
/**
* Converts a {@link Payload} back to a value.
*/
fromPayload<T>(payload: Payload): T;
}
Custom implementation
Some example implementations are in the SDK itself:
There's also a sample project that creates an EJSON custom PayloadConverter
: samples-typescript/ejson
It implements PayloadConverterWithEncoding
instead of PayloadConverter
so that it could be used with CompositePayloadConverter
:
Then we instantiate one and export it:
We provide it to the Worker and Client:
Then we can use supported data types in arguments:
And they get parsed correctly for the Workflow:
Protobufs
To serialize values as Protocol Buffers:
Use
protobufjs
Use runtime-loaded messages (not generated classes) and
MessageClass.create
(notnew MessageClass()
)Generate
json-module.js
with a command like:pbjs -t json-module -w commonjs -o protos/json-module.js protos/*.proto
Patch
json-module.js
:
Generate
root.d.ts
with:pbjs -t static-module protos/*.proto | pbts -o protos/root.d.ts -
Create a
DefaultPayloadConverterWithProtobufs
:
Alternatively, we can use Protobuf Payload Converters directly, or with other converters. If we know that we only use Protobuf objects, and we want them binary encoded (which saves space over proto3 JSON, but can't be viewed in the Web UI), we could do:
- TypeScript
- JavaScript
import {ProtobufBinaryPayloadConverter} from "@temporalio/common/lib/protobufs";
import root from "../protos/root";
export const payloadConverter = new ProtobufBinaryPayloadConverter(root);
import { ProtobufBinaryPayloadConverter } from "@temporalio/common/lib/protobufs";
import root from "../protos/root";
export const payloadConverter = new ProtobufBinaryPayloadConverter(root);
Similarly, if we wanted binary encoded Protobufs in addition to the other default types, we could do:
- TypeScript
- JavaScript
import {
BinaryPayloadConverter,
CompositePayloadConverter,
JsonPayloadConverter,
UndefinedPayloadConverter,
} from "@temporalio/common";
import {ProtobufBinaryPayloadConverter} from "@temporalio/common/lib/protobufs";
import root from "../protos/root";
export const payloadConverter = new CompositePayloadConverter(
new UndefinedPayloadConverter(),
new BinaryPayloadConverter(),
new ProtobufBinaryPayloadConverter(root),
new JsonPayloadConverter()
);
import { BinaryPayloadConverter, CompositePayloadConverter, JsonPayloadConverter, UndefinedPayloadConverter, } from "@temporalio/common";
import { ProtobufBinaryPayloadConverter } from "@temporalio/common/lib/protobufs";
import root from "../protos/root";
export const payloadConverter = new CompositePayloadConverter(new UndefinedPayloadConverter(), new BinaryPayloadConverter(), new ProtobufBinaryPayloadConverter(root), new JsonPayloadConverter());
- Provide it to the Worker:
- Provide it to the Client:
- Use protobufs in our Workflows and Activities:
PayloadCodec
API doc: PayloadCodec
The default PayloadCodec
does nothing. To create a custom one, we implement this interface:
- TypeScript
- JavaScript
interface PayloadCodec {
/**
* Encode an array of {@link Payload}s for sending over the wire.
* @param payloads May have length 0.
*/
encode(payloads: Payload[]): Promise<Payload[]>;
/**
* Decode an array of {@link Payload}s received from the wire.
*/
decode(payloads: Payload[]): Promise<Payload[]>;
}
Encryption
Background: Data Converter ➡️ Encryption
Here's an example class that implements the PayloadCodec
interface:
The encryption and decryption code is in src/crypto.ts
. Since encryption is CPU-intensive, and doing AES with Node's built-in crypto module blocks the main thread, we use @ronomon/crypto-async
, which uses Node's threadpool.
As before, we provide a custom data converter to the Client and Worker:
When the Client sends 'Alice: Private message for Bob.'
to the Workflow, it gets encrypted on the Client and decrypted in the Worker. The Workflow receives the decrypted message and appends another message. When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client.