Data Converters

Background reading: Data Converters in Temporal

Contents:

• Default Data Converter
• Custom Data Converter
  • PayloadConverter
    • Custom implementation
    • Protobufs
  • PayloadCodec
    • Encryption

Default Data Converter

In TypeScript, the default Data Converter supports:

• undefined
• Uint8Array
• JSON

Custom Data Converter

API doc: DataConverter

To send values that are not JSON-serializable like BigInts or Dates, provide a custom Data Converter to the Client and Worker:

• new WorkflowClient({ ..., dataConverter })
• Worker.create({ ..., dataConverter })

Data Converters have two parts:

• PayloadConverter: sync methods that sometimes run inside the Workflow isolate (and are thus limited)
• PayloadCodec: async methods that are run outside the isolate

TypeScript

interface DataConverter {
  payloadConverterPath?: string;
  payloadCodec?: PayloadCodec;
}

JavaScript

PayloadConverter

API doc: PayloadConverter

TypeScript

interface PayloadConverter {
  /**
   * Converts a value to a {@link Payload}.
   * @param value The value to convert. Example values include the Workflow args sent by the client and the values returned by a Workflow or Activity.
   */
  toPayload<T>(value: T): Payload;

  /**
   * Converts a {@link Payload} back to a value.
   */
  fromPayload<T>(payload: Payload): T;
}

JavaScript

Custom implementation

Some example implementations are in the SDK itself:

• common/src/converter/payload-converters.ts
• common/src/converter/protobuf-payload-converters.ts

There's also a sample project that creates an EJSON custom PayloadConverter: samples-typescript/ejson

It implements PayloadConverterWithEncoding instead of PayloadConverter so that it could be used with CompositePayloadConverter:

Then we instantiate one and export it:

We provide it to the Worker and Client:

Then we can use supported data types in arguments:

And they get parsed correctly for the Workflow:

Protobufs

To serialize values as Protocol Buffers:

• Use protobufjs

• Use runtime-loaded messages (not generated classes) and MessageClass.create (not new MessageClass())

• Generate json-module.js with a command like:

  pbjs -t json-module -w commonjs -o protos/json-module.js protos/*.proto

• Patch json-module.js:

• Generate root.d.ts with:

  pbjs -t static-module protos/*.proto | pbts -o protos/root.d.ts -

• Create a DefaultPayloadConverterWithProtobufs:

Alternatively, we can use Protobuf Payload Converters directly, or with other converters. If we know that we only use Protobuf objects, and we want them binary encoded (which saves space over proto3 JSON, but can't be viewed in the Web UI), we could do:

TypeScript

import {ProtobufBinaryPayloadConverter} from "@temporalio/common/lib/protobufs";
import root from "../protos/root";

export const payloadConverter = new ProtobufBinaryPayloadConverter(root);

JavaScript

import { ProtobufBinaryPayloadConverter } from "@temporalio/common/lib/protobufs";
import root from "../protos/root";

export const payloadConverter = new ProtobufBinaryPayloadConverter(root);

Similarly, if we wanted binary encoded Protobufs in addition to the other default types, we could do:

TypeScript

import {
  BinaryPayloadConverter,
  CompositePayloadConverter,
  JsonPayloadConverter,
  UndefinedPayloadConverter,
} from "@temporalio/common";
import {ProtobufBinaryPayloadConverter} from "@temporalio/common/lib/protobufs";
import root from "../protos/root";

export const payloadConverter = new CompositePayloadConverter(
  new UndefinedPayloadConverter(),
  new BinaryPayloadConverter(),
  new ProtobufBinaryPayloadConverter(root),
  new JsonPayloadConverter()
);

JavaScript

import { BinaryPayloadConverter, CompositePayloadConverter, JsonPayloadConverter, UndefinedPayloadConverter, } from "@temporalio/common";
import { ProtobufBinaryPayloadConverter } from "@temporalio/common/lib/protobufs";
import root from "../protos/root";

export const payloadConverter = new CompositePayloadConverter(new UndefinedPayloadConverter(), new BinaryPayloadConverter(), new ProtobufBinaryPayloadConverter(root), new JsonPayloadConverter());

• Provide it to the Worker:

WorkerOptions.dataConverter

• Provide it to the Client:

• Use protobufs in our Workflows and Activities:

PayloadCodec

API doc: PayloadCodec

The default PayloadCodec does nothing. To create a custom one, we implement this interface:

TypeScript

interface PayloadCodec {
  /**
   * Encode an array of {@link Payload}s for sending over the wire.
   * @param payloads May have length 0.
   */
  encode(payloads: Payload[]): Promise<Payload[]>;

  /**
   * Decode an array of {@link Payload}s received from the wire.
   */
  decode(payloads: Payload[]): Promise<Payload[]>;
}

JavaScript

Encryption

Background: Data Converter ➡️ Encryption

Here's an example class that implements the PayloadCodec interface:

The encryption and decryption code is in src/crypto.ts. Since encryption is CPU-intensive, and doing AES with Node's built-in crypto module blocks the main thread, we use @ronomon/crypto-async, which uses Node's threadpool.

As before, we provide a custom data converter to the Client and Worker:

When the Client sends 'Alice: Private message for Bob.' to the Workflow, it gets encrypted on the Client and decrypted in the Worker. The Workflow receives the decrypted message and appends another message. When it returns that longer string, the string gets encrypted by the Worker and decrypted by the Client.