aws_stepfunction_tasks: CallAwsServiceCrossRegion's Lambda doesn't return service response correctly

[shatgupt]

Describe the bug

Lambda created by CallAwsServiceCrossRegion is returning the response from the service called (say Lambda) as a nested byte-array.

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "8f4f0076-f64e-42c5-a7bb-1b607a0a265f",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "ExecutedVersion": "$LATEST",
  "Payload": {
    "0": 123,
    "1": 34,
    "2": 115,
    "3": 116,
    "4": 97,
    "5": 116,
    "6": 117,
    "7": 115,
    "8": 67,
    "9": 111,
    "10": 100,
    "11": 101,
    "12": 34,
    "13": 58,
    "14": 32,
    "15": 50,
    "16": 48,
    "17": 48,
    "18": 44,
    "19": 32,
    "20": 34,
    "21": 98,
    "22": 111,
    "23": 100,
    "24": 121,
    "25": 34,
    "26": 58,
    "27": 32,
    "28": 123,
    "29": 34,
    "30": 115,
    "31": 116,
    "32": 97,
    "33": 116,
    "34": 117,
    "35": 115,
    "36": 34,
    "37": 58,
    "38": 32,
    "39": 34,
    "40": 115,
    "41": 117,
    "42": 99,
    "43": 99,
    "44": 101,
    "45": 115,
    "46": 115,
    "47": 34,
    "48": 125,
    "49": 125
  },
  "StatusCode": 200
}

The actual response from the Lambda it called was

{"statusCode": 200, "body": {"status": "success"}}

which I am able to obtain with the following code:

// Convert the payload object to a byte array and then to string
const payloadBytes = Object.values(response.Payload);
const decodedText = String.fromCharCode(...payloadBytes);
const parsedData = JSON.parse(decodedText);

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

I am expecting the Payload in response to be the actual response from the service called. In this case, something like following:

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "8f4f0076-f64e-42c5-a7bb-1b607a0a265f",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "ExecutedVersion": "$LATEST",
  "Payload": {
    "statusCode": 200,
    "body": {
       "status": "success"
    }
  },
  "StatusCode": 200
}

Current Behavior

See Describe the bug section above.

Reproduction Steps

Create a basic Lambda that returns a JSON. Example

{"statusCode": 200, "body": {"status": "success"}}

Create a cross region Lambda invoke task for the above Lambda using following code:

tasks.CallAwsServiceCrossRegion.jsonata(this, 'HandleTaskResponse', {
            service: 'lambda',
            action: 'invoke',
            region: `{% $orchestratorRegion %}`,
            parameters: {
                FunctionName: `{% $orchestratorTaskResponseHandlerArn %}`,
                Payload: {"hello": "world"},
            },
            iamResources: ['*'],
            iamAction: 'lambda:InvokeFunction', // Required because action above doesn't match the IAM action name for Lambda invoke
        });

Possible Solution

Provided the cross-region-aws-sdk-handler Lambda code to an LLM and it identified the issue and provided solutions:

The Bug

Line 81: return res;

The Lambda is directly returning the raw AWS SDK response object, which can contain:

• Buffers
• Uint8Arrays
• Binary data
• Non-JSON-serializable objects

When AWS Lambda serializes this response, these binary objects get converted to the byte array format you're seeing.

The Fix

The Lambda needs to properly serialize the response before returning it. Here's what needs to be changed:

Current problematic code:

const res = await client.send(command);
return res;  // ❌ This is the bug

Fixed code:

const res = await client.send(command);

// Properly serialize the response
return JSON.parse(JSON.stringify(res, (key, value) => {
  // Handle Buffer/Uint8Array objects
  if (value instanceof Buffer || value instanceof Uint8Array) {
    return Array.from(value);
  }
  // Handle other binary data types
  if (value && typeof value === 'object' && value.constructor && value.constructor.name === 'Uint8Array') {
    return Array.from(value);
  }
  return value;
}));

Or a simpler approach:

const res = await client.send(command);

// Convert to JSON string and back to ensure serialization
const serializedRes = JSON.stringify(res);
return JSON.parse(serializedRes);

Root Cause Explanation

AWS SDK v3 responses often contain binary data that can't be directly serialized to JSON. The Lambda runtime tries to serialize the response, but binary objects get converted to byte arrays, which is exactly what you're seeing in your Step Functions output.

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.189.1

AWS CDK CLI version

2.1012.0 (build e4c1f15)

Node.js Version

v22.15.0

OS

Mac

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @shatgupt,

Thank you for the detailed bug report and analysis! You've correctly identified the root cause of this issue in the cross-region-aws-sdk-handler.

The problem is indeed at line 81 where the handler directly returns the raw AWS SDK response object without proper serialization. AWS SDK v3 responses often contain binary data (Buffers, Uint8Arrays) that can't be directly serialized to JSON. When the Lambda runtime tries to serialize this response, binary objects get converted to the byte array format you're seeing.

Your proposed solution is spot-on. The handler needs to properly serialize the response before returning it. The simplest fix would be to use JSON.stringify/parse to ensure all data is properly serializable:

const res = await client.send(command);
// Convert to JSON string and back to ensure serialization
const serializedRes = JSON.stringify(res);
return JSON.parse(serializedRes);

This issue affects all AWS services that return binary data in their responses when called through CallAwsServiceCrossRegion, making it a significant usability problem.

While I am making it a p1, would you be interested in contributing a PR to fix this? The change would be minimal but would greatly improve the developer experience for anyone using cross-region AWS service calls in Step Functions. The fix should be applied to the handler file mentioned above.

Thanks again for the thorough investigation and clear reproduction steps!