[Question/Discussion] Design docs and rationale which AWS service is used

[ordnungswidrig]

The rational behing using S3 and SQS and the event flow should be documented. I think this can be a major contribution to get crux running quickly and easily when you're on aws anyways. I had a take at implementing the event log and kv store using dynamodb which has different challenges. The pros and cons of each AWS service should be discussed.

[csm]

Nods.

This is an experiment, and nothing more, at the moment. I'm starting with S3 at the core because it's the absolute cheapest storage option available, and if one's use case can deal with the high, unpredictable latency in S3, then it's a great option for storing lots of data.

My first cut at this was to leverage the "standalone" config of crux into AWS: store the TX log on an S3 KV store, and refine the polling event consumer machinery to use SQS to block the system from polling S3 for new data constantly -- only poll when SQS tells us new objects have been added.

I'm working on another approach (I'll probably push this here sometime today, but it's far from usable) that combines DynamoDB, S3, and SQS as follows, which was inspired by how the JDBC TxLog is written:

• Use a single key in DDB for an atomic counter, to generate increasing event IDs (analogous to AUTOINCREMENT primary keys in SQL).
• Store the event log in S3 with sortable keys based on the counter above (I'm storing each event in key l/<id>, and also storing objects k/<event-key>/<id> to track what event keys map to what event IDs). This is nice, because since S3 returns object listings in sorted order, you can traverse the event IDs in order.
• Use SQS to block the polling event consumer from constantly querying S3 for new items.

These approaches are naive! They are certainly not reasonable for production use with large data sets, but I'm just working through ideas at this point, and exploring how crux works internally.

(Please feel free to add issues like this or other comments; that's going to be a better way to prompt me to explain myself than waiting for me to write some documentation)