pgduck_server: cap s3_uploader_thread_limit by host memory

[sfc-gh-mslot]

Problem

DuckDB's httpfs S3 uploader keeps up to s3_uploader_thread_limit multipart-upload part buffers in flight per open file (default 50), each ~80MB and allocated against DuckDB's memory_limit (MemoryTag::EXTENSION). The default part size is ~80MB because it derives from s3_uploader_max_filesize / s3_uploader_max_parts_per_file (800GB / 10000). On a small host (memory_limit 2GB) a single large snapshot writing to S3 can pin up to ~4GB in part buffers alone, hit the limit, and crash pgduck_server with an out-of-memory error. systemd restarts it and the snapshot retries into the same failure, so one oversized table can crash-loop the server for hours, aborting concurrent queries for every other mirror each time.

Solution

Set s3_uploader_thread_limit at startup instead of leaving it at the upstream default of 50. The part buffers are charged against memory_limit, so the cap is sized against that budget rather than physical RAM (the two can differ, e.g. a 2GB limit on a 4GB host). We read the effective memory_limit back from DuckDB and use ~2 threads per GiB, clamped to [2, 50] (never above the upstream default). Uploads are network-bound and run on detached threads independent of DuckDB's threads setting, so CPU count is not the constraint.

memory_limit	s3_uploader_thread_limit	Worst-case S3 buffers/file	(default was 50)
≤1 GiB	2 (floor)	~160 MB	~4 GB
2 GiB	4	~320 MB	~4 GB
4 GiB	8	~640 MB	~4 GB
8 GiB	16	~1.3 GB	~4 GB
16 GiB	32	~2.6 GB	~4 GB
24 GiB	48	~3.8 GB	~4 GB
≥25 GiB	50 (cap)	~4.0 GB	~4 GB

The SET GLOBAL runs after memory_limit is configured and before the init file is applied, so it reflects the effective limit and an operator can still override it:

# init_file_path: raise the cap on a memory-rich host
SET GLOBAL s3_uploader_thread_limit TO 64;

Test plan

• Single TU and full pgduck_server build clean under the tree's -Wall/-Werror=… flags.
• Started the built binary at several limits; init logs s3_uploader_thread_limit is set to: N and SELECT current_setting('s3_uploader_thread_limit') returns it: --memory_limit=2GB → 4, 512MB → 2 (floor), a large limit → 50 (cap).

[sfc-gh-mslot]

useful to pass in memoryLimit here?