Building a Real-World Use Case with AWS Step Functions and the Callback Pattern

AWS Step Functions is a powerful orchestration service that lets you coordinate distributed applications using visual workflows. One of its most powerful capabilities is the Callback Pattern (Wait for Callback), which allows you to pause a workflow until an external event occurs.

In this article, we'll explore how to use this pattern to build real-world scenarios, such as workflows requiring manual approval.

<div class="toc"> <ul> <li><a href="#introduction">Introduction</a></li> <li><a href="#what-is-callback-pattern">What is the Callback Pattern?</a></li> <li><a href="#real-world-scenario">Real-World Scenario: Content Moderation</a></li> <li><a href="#step-by-step-implementation">Step-by-Step Implementation</a></li> <li><a href="#faq">Frequently Asked Questions (FAQ)</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </div> <h2 id="introduction">Introduction</h2>

AWS Step Functions allows you to build complex workflows by stitching together different AWS services (Lambda, DynamoDB, SNS, etc.). In a standard flow, one step starts as soon as the previous one finishes. However, sometimes a step might take days or even months to complete (e.g., a manager's approval or shipping a product). This is where the Callback Pattern comes in.

<h2 id="what-is-callback-pattern">What is the Callback Pattern?</h2>

The callback pattern tells Step Functions to "wait" after starting a task. When used, Step Functions generates a unique Task Token and passes it to the relevant service (e.g., a Lambda function or SQS queue).

The workflow pauses as follows:

The task is started, and a token is generated.
The token is passed to an external system or person.
When the external system completes the task, it calls the SendTaskSuccess (or SendTaskFailure) API with this token to resume the workflow.

This mechanism is ideal for "human-in-the-loop" processes.

<h2 id="real-world-scenario">Real-World Scenario: Content Moderation</h2>

Imagine you run a blogging platform. When users submit a post, you don't want posts containing specific keywords (e.g., "forbidden") to be published automatically. Such posts should fall into a moderator's approval queue.

The flow would be:

User submits a post.
Step Function starts and analyzes the text.
If a forbidden word is found, the flow pauses and notifies a moderator.
The moderator clicks "Approve" or "Reject" on the admin dashboard.
The click action sends the token back to Step Functions, and the flow resumes.

<h2 id="step-by-step-implementation">Step-by-Step Implementation</h2>

Below, you can find the Step Function definition (ASL) and Lambda function example required to set up this architecture.

<h3 id="step-function-definition">Step Function Definition (ASL)</h3>

The following JSON definition implements the callback pattern using the .waitForTaskToken suffix:

{
  "StartAt": "CheckContent",
  "States": {
    "CheckContent": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.content",
          "StringMatches": "*forbidden*",
          "Next": "WaitForApproval"
        }
      ],
      "Default": "PublishContent"
    },
    "WaitForApproval": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "RequestModerationFunction",
        "Payload": {
          "taskToken.$": "$$.Task.Token",
          "content.$": "$.content",
          "author.$": "$.author"
        }
      },
      "Next": "ProcessDecision"
    },
    "ProcessDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "APPROVED",
          "Next": "PublishContent"
        }
      ],
      "Default": "RejectContent"
    },
    "PublishContent": {
      "Type": "Pass",
      "End": true
    },
    "RejectContent": {
      "Type": "Fail",
      "Cause": "Content Rejected by Moderator"
    }
  }
}

<h3 id="lambda-function">Lambda Function (Sending the Token)</h3>

The Lambda function called when the moderator approves (Node.js example):

const AWS = require('aws-sdk');
const stepfunctions = new AWS.StepFunctions();

exports.handler = async (event) => {
    const { taskToken, decision } = JSON.parse(event.body);
    
    const params = {
        taskToken: taskToken,
        output: JSON.stringify({ status: decision }) // 'APPROVED' or 'REJECTED'
    };

    try {
        await stepfunctions.sendTaskSuccess(params).promise();
        return { statusCode: 200, body: "Process successfully submitted." };
    } catch (error) {
        console.error(error);
        return { statusCode: 500, body: "An error occurred." };
    }
};

This code receives the moderator's decision and resumes the waiting Step Function workflow with the relevant token.

<h2 id="faq">Frequently Asked Questions (FAQ)</h2> <div itemscope itemtype="https://schema.org/FAQPage"> <div itemscope itemprop="mainEntity" itemtype="https://schema.org/Question"> <h3 itemprop="name">How long can Step Functions wait for a callback?</h3> <div itemscope itemprop="acceptedAnswer" itemtype="https://schema.org/Answer"> <div itemprop="text">Standard Step Functions workflows can wait for up to 1 year. Express workflows can only run for a maximum of 5 minutes, so Standard workflows are recommended for the callback pattern.</div> </div> </div> <div itemscope itemprop="mainEntity" itemtype="https://schema.org/Question"> <h3 itemprop="name">What happens if the Task Token is lost?</h3> <div itemscope itemprop="acceptedAnswer" itemtype="https://schema.org/Answer"> <div itemprop="text">If the token is lost and the callback cannot be made, the workflow waits until it times out (or is manually stopped). Therefore, it is always recommended to define a 'TimeoutSeconds' value.</div> </div> </div> <div itemscope itemprop="mainEntity" itemtype="https://schema.org/Question"> <h3 itemprop="name">Is this pattern expensive for serverless architectures?</h3> <div itemscope itemprop="acceptedAnswer" itemtype="https://schema.org/Answer"> <div itemprop="text">No, you do not pay for the wait time in Standard Step Functions. You are only charged for state transitions, making it very cost-effective.</div> </div> </div> </div> <h2 id="conclusion">Conclusion</h2>

The AWS Step Functions Callback pattern is an excellent solution for manual approval processes, external webhook integrations, or long-running asynchronous operations. With the "Wait for Callback" feature, you can manage your workflows without server management or complex poller mechanisms.

For more information, check out the <a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token" target="_blank" rel="noopener noreferrer">AWS Official Documentation</a>.

<p><em>Kaynak / Source: <a href="https://awsfundamentals.com/blog/building-a-real-world-use-case-with-step-functions-and-the-callback-pattern" target="_blank" rel="noopener noreferrer">https://awsfundamentals.com/blog/building-a-real-world-use-case-with-step-functions-and-the-callback-pattern</a></em></p>