[:httpd] Fix HTTP large request + response handling

[harmon25]

• buffering support and improved state management when parsing requests
• chunked sending for large responses
• auto computes content-length header

Copilot assisted - might not be cleanest implementation of a fix for the issue, but it does the trick.

[UncleGrumpy]

Thanks for the submission! I did a few problems with this, but we are going to want @bettio or @pguyot to review this as well.

[UncleGrumpy]

Also, we don't have any CI checks for this repo yet, but would you please signoff all of your commits?

[harmon25]

I could squash + sign when we are done with review? i can also open another PR with the commits signed

[UncleGrumpy]

I could squash + sign when we are done with review? i can also open another PR with the commits signed

Since this PR is focused on just the one bug fix, squash and sign will be great. No need to open a separate PR. I would have helped you get previous commits signed if that was necessary. ;-)

[harmon25]

OK i signed off with a force push!

[UncleGrumpy]

I wonder if it would be better to set this a bit smaller, like 1024, so that it is less than most network MTU size (typically 1350-1500, depending on network). Exceeding the MTU will cause packet fragmentation, which isn't much of a problem for unix machines, but may not be handled as gracefully by microcontrollers.

[harmon25]

in my independent fork i landed on 1460
https://github.com/harmon25/atomvm_httpd/blob/diagnostics/src/gen_tcp_server.erl#L49-L51

i will try and port those changes here also

[UncleGrumpy]

That makes sense. 1500 is the most common MTU, and I think there is some tcp/ip overhead in each packet on top of what httpd is sending. If a VPN is used the MTU will be lower. I think 2048 is too high. We should actually expose this as a configuration option, with some lower default (like 1460 that you are using).

[schnittchen]

This breaks iolist response bodies: My response tuple contains a body which is a list of 19 elements, and I'm getting Content-Length: 19. I'm getting a 19 bytes response body which should be much larger.

[harmon25]

All of the changes here I have mirrored in my standalone fork - which adds a bunch of tests via ex_unit - i have also ported over a change that allows for websocket frames that span multiple packets, as well as the ability to pass socket options through a new httpd.start/4

Sorry this PR got a bit out of hand...I wont add anything else but suggested fixes - will also test with the examples here to validate

[harmon25]

untill i can validate the original example app i will keep this in draft - still seem to have issues...