Reviewing AWS Step Functions: The benefits and limitations

Introduction

We began to explore the role of AWS Step Functions during a complex data transfer project for a client. A key highlight of AWS Step Functions is its user-friendly drag-and-drop visual composer, a much-appreciated feature when dealing with extensive scripts. This blog post aims to discuss some benefits and limitations encountered while utilising the service.

Step Functions, a core component of AWS, provides a serverless orchestration service. It enables the coordination of multiple AWS services into adaptable workflows, referred to as ‘state machines’. This service facilitates the creation of complex business processes by integrating AWS Lambda, Amazon S3, and Amazon DynamoDB, among others.

Here are some example use cases where AWS Step Functions can be applied in a digital marketing context:

1. Marketing attribution – Creating a state machine that does the necessary data transforms by calling AWS Glue jobs, which then feeds into an AI notebook in SageMaker where the attribution model runs. Finally, the output can be directed to Amazon Relational Database Service (RDS) ready for consumption.
2. Server-side tagging – With Step Functions’ scalable properties, it’s possible to handle each request coming through from a website, making changes/additions using Lambda functions before pushing to a server-side tag management system. While many tag managers provide the ability to make changes on the fly, this setup offers the entire fleet of AWS tools for modifications.
3. Collating data from APIs – AWS Step Functions can be used to aggregate data from various external APIs. By creating a workflow that triggers Lambda functions to call APIs, parse responses, and store the results in a database like Amazon RDS or DynamoDB, data from different sources can be efficiently collated and processed. This is particularly useful for applications that rely on real-time data from multiple services.

The advantages of using AWS Step Functions are numerous:

• The serverless nature of Step Functions relieves users from managing infrastructure. AWS takes care of scaling, provisioning, and maintenance, allowing focus on workflow definition and optimisation.
• Its integration with the AWS ecosystem is remarkable. For instance, executing a Lambda function is as straightforward as identifying satellites in AWS Ground Station. The integration with the AWS API expands workflow possibilities, offering integrations across a wide range of AWS services.
• Each state in a Step Function workflow enhances error handling and retry capabilities. In the case of known errors, workflows can be extended to restart processes, maintaining continuity in the existing workflow. This feature is particularly useful for managing errors across various workflows.
• The visual representation of workflows, while not perfect, provides a clear overview of the process’s progress. Diagnostic information, such as Lambda function logs, are accessible within the workflow UI, aiding in troubleshooting. See the screenshot below which visually demonstrates where the error occurred in the execution.

However, there are some drawbacks to Step Functions in AWS:

• Integration with Step Functions may require alterations or complete rewrites of existing components. For instance, modifying Lambda functions to handle JSON formats for workflow compatibility can be time-consuming.
• Vendor lock-in is a significant concern. Few alternatives match Step Functions’ capabilities, making a switch to another cloud provider a daunting task. The introduction of third-party endpoint integration in Step Functions doesn’t fully mitigate this issue, as switching cloud providers often necessitates a complete overhaul of workflows.
• The drag-and-drop feature, while visually appealing, doesn’t guarantee logical correctness of workflows. The lack of intuitive syntactical checking can lead to a false sense of ease in workflow creation, overlooking potential limitations. See the screenshot below where a state machine can be created which creates a Lambda function, deletes the Lambda function, and then tries to invoke the very same Lambda function. Note that this state machine has been created but obviously would never work.

Conclusion

In conclusion, while the visual composer of AWS Step Functions serves primarily as a planning tool, it has proven beneficial in designing microservice architectures, akin to tools like Microsoft’s Azure Logic Apps or Google Cloud Workflows.
Despite initial disappointments, the process of planning and executing workflows within Step Functions can be rewarding.

Looking ahead, the hope is that AWS will evolve Step Functions into a more intuitive, drag-and-drop-centric solution, making cloud computing more accessible and reducing the technical complexity currently involved. The potential of Amazon Q, with its generative AI-driven insights and utilisation of previously built state machines, promises to take this efficiency to the next level, potentially inspiring similar innovations across the cloud computing landscape.