Skip to main content

Debugging - TypeScript SDK feature guide

The Debugging section of the Temporal TypeScript SDK developer's guide covers tools for debugging and how to troubleshoot common issues.

How to debug in a development environment​

In addition to the normal development tools of logging and a debugger, you can also see what’s happening in your Workflow by using the Web UI or tctl.

How to debug in a production environment​

You can debug production Workflows using:

You can debug and tune Worker performance with metrics and the Worker performance guide. For information on setting up SDK metrics, see Metrics in the Observability section of the TypeScript SDK developer's guide.

Debug Server performance with Cloud metrics or self-hosted Server metrics.

How to troubleshoot common issues in the TypeScript SDK​

Two locations to watch​

  • Workflow Errors are reflected in Temporal Web.
  • Worker errors and logs are reflected in the terminal.

If something isn't behaving the way you expect, make sure to check both locations for helpful error messages.

Stale Workflows​

If you are developing Workflows and finding that code isn't executing as expected, the first place to look is whether old Workflows are still running.

If those old Workflows have the same name and are on the same task queue, Temporal will try to continue executing them on your new code by design. You may get errors that make no sense to you because

  • Temporal is trying to execute old Workflow code that no longer exists in your codebase, or
  • your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about.

The biggest sign that this is happening is if you notice Temporal is acting non-deterministically: running the same Workflow twice gets different results.

Stale workflows are usually a non-issue because the errors generated are just noise from code you no longer want to run. If you need to terminate old stale Workflows, you can do so with Temporal Web or tctl.

Workflow/Activity registration errors​

If your Workflows or Activities are not imported or spelled correctly, here are some errors we've seen:

  • ApplicationFailure: 'MyFunction' is not a function
  • Workflow did not register a handler for MyQuery

Double check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues.

If you are running Temporal in a monorepo, then your node_modules may be in a different location than where Temporal expects to find it by default, which results in errors like:

[ERROR] Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js' in '/src'

Our Next.js tutorial is written for people setting up Temporal within an existing monorepo, which may be of use here.

When you pass a workflowsPath, our Webpack config expects to find node_modules in the same or a parent/ancestor directory.

If you are custom bundling your own Workflows you may get errors like these:

[ERROR] Failed to activate workflow {
  runId: 'aaf84a83-51ce-462a-9ab7-6a641a703bff',
  error: ReferenceError: exports is not defined,
  workflowExists: false
}

Temporal Workflow Bundles need to export a set of methods that fit the compiled worker-interface.ts from @temporalio/workflow as an entry point. We do offer a bundleWorkflowCode method to assist you with this, though it uses our Webpack settings. For more information, see the Register types section.

Webpack errors​

The TypeScript SDK's Worker bundles Workflows based on workflowsPath with Webpack and run them inside v8 isolates.

If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's logger.

If you do not see Webpack output in your terminal make sure that you have not disabled SDK logging (see reference to Runtime.install() in the link above).

A common mistake for newcomers to the TypeScript SDK is trying to use Node.js built-ins and modules in their Workflow code. Usually, the best thing to do is move that code to an Activity.

Some common examples that will not work in the Workflow isolate:

Importing node built-in modules

Antipattern
import fs from "fs";

const config = fs.readFileSync("config.json", "utf8");

This is invalid because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed.

You'll typically see an error in this form in the Webpack output:

2021-10-14T19:22:00.606Z [INFO] Module not found: Error: Can't resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO] resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO]   Parsed request is a module
2021-10-14T19:22:00.606Z [INFO]   using description file: /Users/you/your-project/package.json (relative path: ./src)
2021-10-14T19:22:00.606Z [INFO]     Field 'browser' doesn't contain a valid alias configuration

Importing and calling Activities directly from Workflow code

Antipattern
import { makeHTTPRequest } from "./activities";

export async function yourWorkflow(): Promise<string> {
  return await makeHTTPRequest("https://temporal.io");
}

This is invalid because activity implementations should not be directly referenced by Workflow code. Activities are used by Workflows in order make network calls and reading from the filesystem, operations which are non-deterministic by nature because they rely on external state. Temporal records Activity results in the Workflow history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow.

You'll typically see an error in this form in the Webpack output:

2021-10-14T19:46:52.731Z [INFO] ERROR in ./src/activities.ts 8:31-46
2021-10-14T19:46:52.731Z [INFO] Module not found: Error: Can't resolve 'http' in '/Users/you/your-project/src'
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
2021-10-14T19:46:52.731Z [INFO] This is no longer the case. Verify if you need this module and configure a polyfill for it.
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] If you want to include a polyfill, you need to:
2021-10-14T19:46:52.731Z [INFO]         - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }'
2021-10-14T19:46:52.731Z [INFO]         - install 'stream-http'
2021-10-14T19:46:52.731Z [INFO] If you don't want to include a polyfill, you can use an empty module like this:
2021-10-14T19:46:52.731Z [INFO]         resolve.fallback: { "http": false }

To properly call your Activities from Workflow code use proxyActivities and make sure to only import the Activity types.

import { proxyActivities } from "@temporalio/workflow";

import type * as activities from "./activities";
const { makeHTTPRequest } = proxyActivities<typeof activities>();

export async function yourWorkflow(): Promise<string> {
  return await makeHTTPRequest("https://temporal.io");
}

Works in Dev but not in Prod​

The two main sources of dev-prod discrepancies are in bundling and connecting.

Production bundling​

If you are getting errors like this:

Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request.

This is due to your bundler stripping out Workflow function names, which we rely on to set the "Workflow Type" in Temporal. Turn it off and it should work.

// webpack.config.js
module.exports = {
  optimization: {
    minimize: true,
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          keep_fnames: true, // don't strip funciton names in production
        },
      }),
    ],
  },
};

Connecting to Temporal Server​

If you are trying to connect in production and getting this:

[TransportError: transport error]

It is a sign that something is wrong with your Cert/Key pair. Log it out and make sure it is an exact match with what is expected (often, the issue can be whitespace when injecting from your production secrets management environment).

Resetting Workflows to deal with logical bugs​

You can "rewind time" using the tctl CLI, resetting Workflow History to some previous point in time. You can read the CLI docs on:

If you need to reset programmatically, the TS SDK does not have any high level APIs for this, but you can make raw gRPC calls to resetWorkflowExecution.

Resetting should only be used to deal with serious logical bugs in your code: it's not for handling transient failures, like a downstream service being unreachable. It should not be used in the course of normal application flows.

gRPC call timeouts (context deadline exceeded)​

The opaque context deadline exceeded error comes from gRPC:

Error: 4 DEADLINE_EXCEEDED: context deadline exceeded
    at Object.callErrorFromStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client.js:179:52)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 4,
  details: 'context deadline exceeded',
  metadata: Metadata {
    internalRepr: Map(1) { 'content-type' => [Array] },
    options: {}
  },
  page: '/api/getBuyState'
}

Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server. Querying a Workflow Execution whose query handler causes an error can result in the query call timing out.

Some troubleshooting actions you can take:

  • Verify the connection from your Worker to the Temporal Server is working and doesn't have unusually high latency.
  • If you are running Temporal Server yourself, check your server metrics to ensure it's not overloaded.
  • If what's timing out is a query, check the logs of your Workers to see if they are having issues handling the query.

If none of the preceding actions help you discover why timeouts are occurring, please try to produce a minimal repro and we'll be glad to help.