Skip to main content

Datadog Metrics

Exporting cloud metrics to Datadog provides enhanced observability, allowing you to monitor, alert, and visualize key performance indicators of your applications and infrastructure. Temporal's integration with Datadog extends the monitoring capabilities of your Temporal Cloud deployment.

What will you learn?​

You will set up your environment to export metrics from Temporal Cloud to Datadog, including:

  • Preparing your environment with the necessary prerequisites
  • Configuring certificates for secure communication
  • Deploying the integration using Helm and Minikube
  • Verifying the setup to ensure metrics are being exported correctly

Prerequisites​

Before you begin, ensure you have the following:

Step 1. Set up the GitHub project​

Temporal provides a script demonstrating the minimum work necessary to read recently generated metrics from a Temporal Cloud account using the Prometheus API and import them into Datadog while handling some common edge and error cases.

The following tutorial uses the Prometheus Querying Language to Datadog in Go (PromQL); however, you can use and customize this code and others to suit your needs.

Temporal also provides a scrape version which can be used by the Otel or Datadog agents.

Temporal also provides examples in other languages:

Clone the promql-to-dd-go directory from the Temporal samples server and navigate into it:

gh repo clone temporalio/samples-server
cd samples-server/cloud/observability/promql-to-dd-go
note

These examples are provided as-is, without support. They are intended as reference material only.

Metrics are exported on a per-account basis.

Step 2. Create your certificates​

Choose a method for generating certificates. You can use one of the methods provided, or if your organization has its own certification manager, use that.

These certificates are used to communicate with your Temporal Cloud account. For more information on certifications and Temporal Cloud, see Certificate management.

Using tcld​

Use the tcld to authenticate, generate, and add certificates to your Cloud account.

  1. Login: Authenticate with your Temporal Cloud account. 
    tcld login
  2. Create certs: Generate the necessary certificates for secure communication. 
    tcld generate-certificates certificate-authority-certificate --org ${ACCOUNT_ID} -d 1y --ca-cert ca.pem --ca-key ca.key
  3. Add certificates to Cloud account: Ensure your Temporal Cloud Namespace is configured with the generated certificates. 
    tcld namespace accepted-client-ca add --ca-certificate-file ca.pem

Replace ${ACCOUNT_ID} with your Temporal Cloud account ID. This is the assigned account identifier.

Using certstrap​

You can also use certification managers like certstrap.

  1. Follow the certstrap README to download and install certstrap.
  2. Create a new certificate authority with certstrap init --common-name CertAuth.
  3. Request a certificate key pair with certstrap request-cert --common-name metrics-cert.
  4. Sign the certificate request to generate the end-entity certificate with certstrap sign metrics-cert --CA CertAuth.
  5. Locate your newly created certificates in the out folder within the certstrap directory.
  6. Add the certificates to your Cloud Account, for more information see How to add, update, and remove certificates in a Temporal Cloud Namespace.
note

Certificate names must be unique per account.

Next, verify that your certificates are set up correctly.

Verifying certificates setup​

Now that you've created and set your certifications, test them to ensure they are working correctly.

Test your setup with the following command:

curl --cert ca.pem --key ca.key "https://<Organization_ID>.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count" | jq .

Provide the Organization ID of your Temporal Cloud account.

Step 3. Run Helm and Minikube​

Use the following commands to run Minikube and Helm in your environment. While this is using Minikube, you can deploy to any Kubernetes cluster, with your environment specific configuration.

  1. Start Minikube:

In one terminal, start your Minikube instance:

minikube start
  1. Deploy with Helm:

In a second terminal, start your Helm instance.

Set your environment variables for the Datadog API key and Temporal Account Id, then deploy using Helm:

helm install promqltodd . \
  --set prom_endpoint=https://${ACCOUNT_ID}.tmprl.cloud/prometheus \
  --set dd_api_key=${DD_API_KEY} \
  --set query_interval_seconds=15 \
  --set-file 'ca_cert=../out/metrics-cert.crt' \
  --set-file 'ca_key=../out/metrics-cert.key'

Update the path and the name of your certificate and key.

Use the helm status command to confirm the deployment or helm get all to view the deployment.

Deploy K9s​

Start your K9s instance to monitor the deployment:

k9s

Review the logs to confirm the integration is functioning as expected.

Use a pre-built dashboard to visualize your metrics in Datadog.

You've successfully set up the export of metrics from Temporal Cloud to Datadog.

Next steps​

This is just the start.

Customize and extend this code, dashboards, and deployment processes to meet your needs.

  • Explore Datadog dashboards to visualize your Temporal metrics.
  • Set up alerts in Datadog based on the metrics received from Temporal Cloud.
  • Consider integrating additional observability tools or exporting metrics to other platforms as needed.