Compare library performance against AWS-SDK

[jpbourgeon]

Performance has not been tested - it's possible that our use of https or fetch is slower than whatever magic the AWS SDK is doing.

As a first step, I suggest using a single one-shot performance benchmark against any standard service. This would test give a fast answer about how itty-aws compares to AWS-SDK. Then we can decide if performance should be systematically tested throughout API and services.

mcollina/autocannon is a maintained benchmarking tool that could be use for this purpose.
bestiejs/benchmark.js is more widely used but has been unmaintained for years now.

[jpbourgeon]

@sam-goodwin in #21
I set up some very basic benchmarking here: https://github.com/sam-goodwin/itty-aws/tree/main/benchmark but it only really tests for cold start testing, which obviously this will do well at. A more important test would be sustained use - to test if performance degrades with usage due to poor connection management. I think that's likely the biggest risk right now - how well does the vanilla https/fetch libraries hold up against whatever AWS is doing?

Do you agree with the following use case ?

As a user I want to benchmark Itty-aws against AWS-SDK v2 and v3 to guarantee that it performs at least as well as the tools provided by AWS.

Using mcollina/autocannon, the test will compare the performance of 1000 calls of 3 simple lambdas written respectively using AWS-SDKv2, AWS-SDKv3 and Itty-AWS.

• Test setup: create a dynamodb table
• Test teardown: delete the dynamodb table

The lambda will perform a set of predefined CRUD operations on DynamoDB with its respective package:

• write a document
• read the document
• update the document
• delete the document

[jpbourgeon]

Sleep on it, they say.

As a user I want to benchmark Itty-AWS in various runtimes and against AWS-SDK v2 and v3 to guarantee that it performs at least as well as the tools provided by AWS.

Forked from the maxday/lambda-perf project, we will setup a cloud stack that will compare the performance of basic lambdas in various runtimes. We will gather the following metrics : Avg Cold Start duration, Avg Memory Used, Avg duration (excluding cold starts).
The benchmark will be repeatable using various versions of Itty-AWS to show the impact of code changes on the perfs.

Each function will be deployed 10 times and will run 10 times. We will have 10 coldstarts and 90 executions per runtime to calculate our metrics.

The tested function will do simple API calls: write, read and delete a basic DynamoDB document.

Targeted runtimes, to start with:

• AWS Node.js 16.x / AWS-SDKv2
• AWS Node.js 18.x / AWS-SDKv3
• AWS Node.js 16.x / Itty-AWS (build from source based on branch, tag or PR)
• AWS Node.js 18.x / Itty-AWS (build from source based on branch, tag or PR)

I'll go with that unless you have another strategy in mind.

[sam-goodwin]

Sorry, been busy with my upcoming release.

Avg Cold Start duration, Avg Memory Used, Avg duration (excluding cold starts).

Average is good but might be too coarse. Can we collect the raw metrics and plot them on a histogram. I'm interested in the variance as much as the average performance amortized over many calls.

[jpbourgeon]

Yes, I will log detailed data from each call, from which we will build useful metrics: average, histogram, variance and standard deviation, etc.

About TTFB to measure latency, that would be a very useful addition to our performance metrics.

Also, it is especially important to measure latency from inside the AWS cloud to remove as many network variables as possible. Which validates the approach to measuring SDK calls with a Lambda function.

I need to investigate how to measure the latency of SDK functions. This is non-trivial since HTTP calls are wrapped in SDK functions and cannot be monitored directly. I will start with Node.js performance measurement apis to see if we can gather enough data observing the async SDK functions calls.

[jpbourgeon]

Just to keep you posted. I forked the repo and started working on it. I have grasped the global architecture of the benchmarking and it is progressing. However, I only have a few free hours each day to code, so there is still time before the PR.