Overview:
When using this serverless function to export to a SIEM, the code below typically fragments the messages and as a result you get fragmented incomplete events within your SIEM. If this is happening you can adjust the below code:
Existing code:
if JsonFormat == "SingleLine":
gzOutfile.write(('[' + ',\n'.join(json.dumps(i) for i in data) + ']').encode('UTF-8'))
New Code:
print ("Indent: " + JsonFormat)
if JsonFormat == "SingleLine":
# FIX: Remove brackets and commas, join with a newline.
ndjson_data = '\n'.join(json.dumps(i) for i in finalData)
gzOutfile.write(ndjson_data.encode('UTF-8'))
gzOutfile.close()
else:
gzOutfile.write(json.dumps(finalData, indent=2).encode("UTF-8"))
gzOutfile.close()
Ensure the environment variable JsonFormat for this Lambda function is set to SingleLine. If it is set to anything else, it will trigger the else block which uses indent=2, causing the exact same "fragmentation" problem.
Overview:
When using this serverless function to export to a SIEM, the code below typically fragments the messages and as a result you get fragmented incomplete events within your SIEM. If this is happening you can adjust the below code:
Existing code:
if JsonFormat == "SingleLine":
gzOutfile.write(('[' + ',\n'.join(json.dumps(i) for i in data) + ']').encode('UTF-8'))
New Code:
print ("Indent: " + JsonFormat)
if JsonFormat == "SingleLine":
# FIX: Remove brackets and commas, join with a newline.
ndjson_data = '\n'.join(json.dumps(i) for i in finalData)
gzOutfile.write(ndjson_data.encode('UTF-8'))
gzOutfile.close()
else:
gzOutfile.write(json.dumps(finalData, indent=2).encode("UTF-8"))
gzOutfile.close()
Ensure the environment variable JsonFormat for this Lambda function is set to SingleLine. If it is set to anything else, it will trigger the else block which uses indent=2, causing the exact same "fragmentation" problem.