Skip to main content

Export - Temporal Cloud feature guide

Support, stability, and dependency info
  • Workflow History Export is in a Public Preview release status for Temporal Cloud.

The Workflow History Export feature allows users to export Closed Workflow Histories to a user's Cloud Storage Sink.

The Export feature in Temporal Cloud provides the following benefits:

  • Preserve complete set of Workflow History data in proto format for compliance and auditing purposes.
  • Enables feeding Workflow History into data warehouses for analytics.

For pricing information, see Temporal Cloud Pricing.

note

If you have enabled encryption for the data handled in your Workflow Execution, the exported data will also be encrypted.

Supported integrations for Workflow History Export​

The Workflow History Export feature allows users to export Closed Workflow Histories to an object storage as the destination.

You can enable Workflow History Export per Namespace in Temporal Cloud.

Prerequisites​

Before configuring the Export Sink, ensure you have the following:

  1. A cloud account in the cloud provider where your namespace is hosted.
  2. An object storage setup to receive the exported History.

Configure Workflow History Export​

You can configure your Workflow History Export in AWS.

Verify export setup​

How to verify that the Export feature is correctly set up

From the Export configuration page, select Verify. This action checks if Temporal can successfully write a test file to the sink.

If everything is configured correctly, you will see a Success status indicating Temporal has written to your sink.

Monitor export progress​

How to monitor the History export progress

Once you've finalized the setup, here's how to monitor the export progress:

  1. Export Job Execution:

    • Schedule: The Export job is scheduled to run on an hourly basis, starting at 10 minutes past each hour.
    • Duration: The time taken for the export process can vary, so it might not be instantaneous.

  2. Checking the Object Storage:

    • Files Arrival: Post the initial hour of setting up, inspect your object storage. You should see the exported Workflow History files. These files encapsulate the relevant data from the Closed Workflows.
    • Directory Structure: Your exported files will adhere to the following naming convention and path:
    //[bucket-name]/temporal-workflow-history/export/[Namespace]/[Year]/[Month]/[Day]/[Hour]/[Minute]/

  3. Delivery guarantee:

    • At least once delivery.
    • Each Closed Workflow is expected to be found in the exported file within one to four hours.

  4. UI insights:

    • Last Successful Export: This displays the timestamp of the most recent successful export. It's an essential indicator of the last time your export process completed without any hitches.
    • Last Status Check: This reflects the timestamp of the latest internal Workflow check. This internal check routinely evaluates the health and status of the Export mechanism, ensuring its uninterrupted functioning.

  5. Usage monitoring:

    • Actions from the Export Job are included in the Usage UI.
    • Metrics: Export related metrics are available in temporal_cloud_v0_total_action_count with the label is_background="true". For more information, see Cloud metrics.

For optimal results, review the S3 or GCS bucket for any new exported files and refer to the UI insights. This dual check ensures you remain abreast of the export progress and any potential issues.

Working with exported files​

The proto schema is defined in https://github.com/temporalio/api/blob/master/temporal/api/export/v1/message.proto. You could build your own custom deserializer for it. You can find sample go code to reference when you want to deserialize export file in the Appendix section of this document.