Error: The state/task 'UpdateMembers' returned a result with a size exceeding the maximum number of bytes service limit.

[starwolf73]

Hi,
We have an large Organizations with more than 1300 accounts overs 17+ regions and this solution was a must for our Security Hub installation. Unfortunately, in DEV Org with less accounts and regions it was working fine until we start implementing it in PROD. If failed at the last iteration #1301 for a total of 9124 events.
We received the following error in our step-function:

States.DataLimitExceeded
The state/task 'UpdateMembers' returned a result with a size exceeding the maximum number of bytes service limit.

Anything we can do to increase this limit? Or it required a change in the code to process it differently?

[scottbward]

Thanks for reporting this. I am looking into this and will get back as soon as I can regarding what can be done to address this error.

[scottbward]

The error that you are reporting is from the AWS Step Functions service. Quotas for Step functions are outlined here: https://docs.aws.amazon.com/step-functions/latest/dg/limits-overview.html, specifically it would be the "Maximum input or output size for a task, state, or execution" quota. From a few searches it looks like the solve is to alter the way that information is being passed around for the tasks in your step function. I suspect that due to the large number of accounts you are accruing a large amount of responses from AWS API calls and that is accounting for the max size exceeded.

Here are a few resources that might help:
https://ncoughlin.com/posts/aws-step-functions-notes/#statesdatalimitexceeded. - This seems like it may align well to your situation.

https://stackoverflow.com/questions/55702571/step-function-exceeding-the-maximum-number-of-characters-service-limit
https://stackoverflow.com/questions/71833184/aws-step-functions-size-exceeding-the-maximum-number-of-bytes-service-limit?noredirect=1&lq=1

Let me know if this helps or if you have additional questions. I'll do my best to help you along on this.