Orchestrating Workflows with Step Functions

Relevant source files

The following files were used as context for generating this wiki page:

• packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts
• packages/@aws-cdk/aws-stepfunctions/lib/chain.ts
• packages/@aws-cdk/aws-stepfunctions-tasks/lib/lambda/invoke-function.ts
• packages/@aws-cdk/aws-stepfunctions-tasks/lib/ecs/run-ecs-fargate-task.ts
• packages/@aws-cdk/aws-stepfunctions/README.md
• packages/@aws-cdk/aws-codepipeline-actions/README.md

The @aws-cdk/aws-stepfunctions module provides constructs for building serverless workflows using AWS Step Functions. It allows developers to define complex, multi-step processes in code using high-level, object-oriented abstractions. These workflows, known as state machines, are composed of various states such as tasks, choices, and parallel executions. This library, in conjunction with @aws-cdk/aws-stepfunctions-tasks, enables seamless integration with other AWS services like AWS Lambda, Amazon ECS, and more.

The core idea is to represent the Amazon States Language as a tree of construct objects. A state machine is defined by its start state, and the entire workflow is composed by chaining states together. This approach provides a powerful and readable way to define, provision, and manage orchestration logic directly within a CDK application.

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:9-16, packages/@aws-cdk/aws-stepfunctions/README.md:50-58

Core Concepts

State Machine

The primary resource is the StateMachine, which takes a definition and configuration to create an executable workflow. The definition is constructed by linking various state objects together, starting with a single entry point.

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:251-252, packages/@aws-cdk/aws-stepfunctions/README.md:44-50

The following diagram shows the main components of a StateMachine construct.

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:186-249, packages/@aws-cdk/aws-stepfunctions/lib/chain.ts:17-23

Key properties for a StateMachine include:

Property	Type	Description
definition	IChainable	The definition of the state machine, starting with the first state.
timeout	Duration	The maximum execution time for the state machine.
logs	LogOptions	Defines the CloudWatch Logs configuration for execution history.
tracingEnabled	boolean	Enables X-Ray tracing for the state machine.
role	iam.IRole	The IAM role for the state machine execution. A role is created by default if not provided.

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:186-249, packages/@aws-cdk/aws-stepfunctions/README.md:50-55

State Definition and Chaining

A workflow definition is built by creating instances of state constructs and linking them together using the .next() method. This creates a Chain, which is an IChainable object representing a sequence of states. The Chain.start() static method can be used to begin a new chain explicitly.

import * as sfn from '@aws-cdk/aws-stepfunctions';

const step1 = new sfn.Pass(this, 'Step1');
const step2 = new sfn.Pass(this, 'Step2');
const step3 = new sfn.Pass(this, 'Step3');

// A chain can be started from the first state
const definition1 = step1.next(step2).next(step3);

// Or by using Chain.start()
const definition2 = sfn.Chain.start(step1).next(step2).next(step3);

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:328-349, packages/@aws-cdk/aws-stepfunctions/lib/chain.ts:32-41

The following diagram illustrates how states are chained together.

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:328-338

State Machine Fragments

For reusability, you can create "mini-state machines" by implementing the IChainable interface. The StateMachineFragment abstract class provides helpers for this, such as prefixStates() to avoid naming conflicts when a fragment is used multiple times in the same state machine. A fragment must define a startState and one or more endStates.

class MyJob extends sfn.StateMachineFragment {
    public readonly startState: sfn.State;
    public readonly endStates: sfn.INextable[];

    constructor(parent: cdk.Construct, id: string, props: MyJobProps) {
        super(parent, id);

        const first = new sfn.Task(this, 'First', { /* ... */ });
        const last = new sfn.Task(this, 'Last', { /* ... */ });

        this.startState = first;
        this.endStates = [last];
    }
}

// Usage
new sfn.Parallel(this, 'All jobs')
    .branch(new MyJob(this, 'Quick', { jobFlavor: 'quick' }).prefixStates())
    .branch(new MyJob(this, 'Slow', { jobFlavor: 'slow' }).prefixStates());

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:358-388, packages/@aws-cdk/aws-stepfunctions/lib/chain.ts:17-28

State Types

The library provides constructs for all standard Amazon States Language states.

State	Description
Task	Represents a single unit of work performed by a state machine. Integrates with various AWS services.
Pass	Passes its input to its output, potentially adding or filtering data. Useful for debugging and data transformation.
Wait	Delays the state machine from continuing for a specified time.
Choice	Adds branching logic to a state machine based on conditions.
Parallel	Begins parallel branches of execution.
Map	Executes a set of steps for each element of an input array.
Succeed	Stops an execution successfully.
Fail	Stops an execution with a failure.
CustomState	Allows defining a state with a raw JSON object for features not yet available in constructs.

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:61-71

Task States

Task states perform work by integrating with other AWS services. The @aws-cdk/aws-stepfunctions-tasks module provides a rich set of pre-built integrations.

AWS Lambda Integration

The LambdaInvoke task calls an AWS Lambda function. You can control the input passed to the function and how its output is processed.

Sources: packages/@aws-cdk/aws-stepfunctions-tasks/lib/lambda/invoke-function.ts

Key LambdaInvoke Properties:

Property	Type	Description
lambdaFunction	lambda.IFunction	The Lambda function to invoke.
inputPath	string	A JSONPath to select a portion of the state to pass as input to the function. Defaults to $ (the entire state).
outputPath	string	A JSONPath to specify where to place the Lambda's output in the state. Defaults to $ (replaces the entire state).
payload	sfn.TaskInput	The JSON payload to pass to the Lambda function. If specified, inputPath is ignored.
invocationType	InvocationType	The type of invocation (e.g., RequestResponse, Event, DryRun).

Sources: packages/@aws-cdk/aws-stepfunctions-tasks/lib/lambda/invoke-function.ts:114-158

Amazon ECS Integration

The RunEcsFargateTask task allows you to run a container task on an Amazon ECS cluster using the Fargate launch type.

Key RunEcsFargateTask Properties:

Property	Type	Description
cluster	ecs.ICluster	The ECS cluster to run the task on.
taskDefinition	ecs.TaskDefinition	The ECS task definition to run.
containerOverrides	ContainerOverride[]	Overrides for one or more containers in the task definition.
integrationPattern	sfn.ServiceIntegrationPattern	The integration pattern (e.g., RUN_JOB to wait for completion).
assignPublicIp	boolean	Assign a public IP to the task.
securityGroups	ec2.ISecurityGroup[]	Security groups to associate with the task.

Sources: packages/@aws-cdk/aws-stepfunctions-tasks/lib/ecs/run-ecs-fargate-task.ts:108-149

Example Workflow: Job Poller

The following example demonstrates a common pattern where a job is submitted, and the state machine polls for its status until it completes.

const submitJob = new tasks.LambdaInvoke(this, 'Submit Job', {
  lambdaFunction: submitLambda,
  outputPath: '$.Payload',
});

const waitX = new sfn.Wait(this, 'Wait X Seconds', {
    time: sfn.WaitTime.secondsPath('$.waitSeconds'),
});

const getStatus = new tasks.LambdaInvoke(this, 'Get Job Status', {
  lambdaFunction: getStatusLambda,
  inputPath: '$.guid',
  outputPath: '$.Payload',
});

const jobFailed = new sfn.Fail(this, 'Job Failed', {
    cause: 'AWS Batch Job Failed',
    error: 'DescribeJob returned FAILED',
});

const finalStatus = new tasks.LambdaInvoke(this, 'Get Final Job Status', {
  lambdaFunction: getStatusLambda,
  inputPath: '$.guid',
  outputPath: '$.Payload',
});

const definition = submitJob
    .next(waitX)
    .next(getStatus)
    .next(new sfn.Choice(this, 'Job Complete?')
        .when(sfn.Condition.stringEquals('$.status', 'FAILED'), jobFailed)
        .when(sfn.Condition.stringEquals('$.status', 'SUCCEEDED'), finalStatus)
        .otherwise(waitX));

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:19-41

This CDK code defines the following workflow:

Sources: packages/@aws-cdk/aws-stepfunctions/README.md:19-41

Permissions Management

The StateMachine construct automatically manages IAM permissions. When you add tasks, the state machine's execution role is granted the necessary permissions (e.g., lambda:InvokeFunction for a LambdaInvoke task). You can also grant permissions to other principals to interact with the state machine.

Method	Description	IAM Permissions Granted
grantStartExecution(principal)	Grants permission to start an execution.	states:StartExecution
grantRead(principal)	Grants read-only access to the state machine and its executions.	states:ListExecutions, states:DescribeExecution, etc.
grantTaskResponse(principal)	Grants permission to send task success, failure, or heartbeat signals.	states:SendTaskSuccess, states:SendTaskFailure, states:SendTaskHeartbeat
grant(principal, ...actions)	Grants custom state-machine-level permissions.	User-specified actions.

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:311-396, packages/@aws-cdk/aws-stepfunctions/README.md:468-524

Integration with Other Services

Invoking from AWS CodePipeline

A Step Functions state machine can be invoked as a step in an AWS CodePipeline pipeline using the StepFunctionsInvokeAction. This is useful for orchestrating complex deployment steps, tests, or other automated processes as part of a CI/CD workflow.

The input to the state machine can be provided as a literal object or from a file within a CodePipeline artifact.

import * as stepfunction from '@aws-cdk/aws-stepfunctions';
import * as codepipeline_actions from '@aws-cdk/aws-codepipeline-actions';

const simpleStateMachine  = new stepfunction.StateMachine(stack, 'SimpleStateMachine', {
  definition: startState,
});

// Example 1: Literal input
const stepFunctionAction1 = new codepipeline_actions.StepFunctionsInvokeAction({
  actionName: 'Invoke',
  stateMachine: simpleStateMachine,
  stateMachineInput: codepipeline_actions.StateMachineInput.literal({ IsHelloWorldExample: true }),
});

// Example 2: Input from a file in an artifact
const inputArtifact = new codepipeline.Artifact();
const stepFunctionAction2 = new codepipeline_actions.StepFunctionsInvokeAction({
  actionName: 'InvokeWithFile',
  stateMachine: simpleStateMachine,
  stateMachineInput: codepipeline_actions.StateMachineInput.filePath(inputArtifact.atPath('assets/input.json')),
});

Sources: packages/@aws-cdk/aws-codepipeline-actions/README.md:477-506

Monitoring and Logging

CloudWatch Logging

You can configure a state machine to send execution history to a CloudWatch Logs log group by specifying the logs property. You can control the level of detail logged, such as logging all events or only errors.

const logGroup = new logs.LogGroup(stack, 'MyLogGroup');

new sfn.StateMachine(stack, 'MyStateMachine', {
    definition: chain,
    logs: {
      destination: logGroup,
      level: sfn.LogLevel.ALL,
    }
});

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:228-235, packages/@aws-cdk/aws-stepfunctions/README.md:440-450

X-Ray Tracing

AWS X-Ray tracing can be enabled for a state machine by setting the tracingEnabled property to true. This provides detailed tracing information for the execution of the state machine and its integrated services.

new sfn.StateMachine(stack, 'MyStateMachine', {
    definition: chain,
    tracingEnabled: true
});

Sources: packages/@aws-cdk/aws-stepfunctions/lib/state-machine.ts:237, packages/@aws-cdk/aws-stepfunctions/README.md:452-459