Skip to content

Treat missing cgroups files as a quiet fallback, not a stack-traced error - #52

Open
auxten wants to merge 2 commits into
mainfrom
fix/cgroups-stack-trace-restricted-containers
Open

Treat missing cgroups files as a quiet fallback, not a stack-traced error#52
auxten wants to merge 2 commits into
mainfrom
fix/cgroups-stack-trace-restricted-containers

Conversation

@auxten

@auxten auxten commented May 14, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #51.

In strictly restricted containers (k8s pods with hidden /sys/fs/cgroup, distroless images, FaaS sandboxes, …) neither cgroups v1 memory/memory.stat nor cgroups v2 memory.current is accessible. The previous code modeled this expected condition as a FILE_DOESNT_EXIST exception, which ended up dumping a full C++ stack trace via tryLogCurrentException on every embedded server construction — i.e. essentially every chDB query in short-lived sessions:

stderr:
Cannot use cgroups reader: Code: 107. DB::Exception: Cannot find cgroups v1 or v2 current memory file. (FILE_DOESNT_EXIST), Stack trace ...
 0. ? @ 0x000000001cf32cf3
 1. ? @ 0x000000000fbd529e
 ...

The query still completed (Jemalloc fallback) but the noise on stderr was alarming, broke log monitoring, and made users think their queries had failed.

Approach

Reshape ICgroupsReader::getCgroupsPath() to express what it really is: a probe that may or may not find a usable cgroup hierarchy. The new tryGetCgroupsPath() returns std::optional<std::pair<std::string, CgroupsVersion>>, so callers can distinguish:

  • Cgroups absent (the normal case in restricted containers) — take the empty branch and emit a single LOG_DEBUG line. No stack trace, no exception machinery.
  • Cgroups present but reader construction fails (file vanished, permissions, parsing error, …) — still log the full diagnostic via tryLogCurrentException. Genuinely unexpected errors are preserved.

Both call sites in MemoryWorker and AsynchronousMetrics are updated. The now-unused FILE_DOESNT_EXIST ErrorCodes declaration in MemoryWorker.cpp is removed.

Why this shape

Two alternatives were considered:

  1. Catch FILE_DOESNT_EXIST inside the existing catch and downgrade to LOG_DEBUG. Smallest diff, but still uses exceptions for control flow on a hot path that runs on every embedded session start. Doesn't fix the underlying "this isn't actually an error" mismodeling.
  2. Make tryGetCgroupsPath() non-throwing (this PR). Same diff size at call sites, semantically correct, no exception machinery for the expected case, and existing createCgroupsReader failures still surface with their full diagnostic.

The existing gtest_cgroups_reader.cpp only exercises createCgroupsReader (which is unchanged), so no test surgery is needed.

Test plan

  • Build on Linux and run gtest_cgroups_reader — should still pass unchanged.
  • Run a chDB query in a normal Linux host — LOG_DEBUG from MemoryWorker shows it picked the cgroups path; behaviour unchanged.
  • Run a chDB query in a container with /sys/fs/cgroup not mounted (or --security-opt=apparmor=unconfined style hardening that hides it) — stderr should be clean (no stack trace), and MemoryWorker should fall back to Jemalloc as before.
  • Build on macOS — OS_LINUX-guarded code is excluded, no API breakage.

auxten added 2 commits May 14, 2026 23:42
…rror

In strictly restricted containers (k8s pods with hidden cgroupfs, distroless
images, FaaS sandboxes, ...) neither cgroups v1's `memory/memory.stat` nor
cgroups v2's `memory.current` is accessible. The previous code modeled this
expected condition as a `FILE_DOESNT_EXIST` exception, which ended up dumping
a full C++ stack trace via `tryLogCurrentException` on every embedded server
construction (i.e. essentially every chDB query in short-lived sessions).

The query still completed (Jemalloc was used as the fallback memory source)
but the noise on stderr was alarming, broke log monitoring, and made users
think their queries had failed.

This change reshapes `ICgroupsReader::getCgroupsPath()` to express its
real semantics: it is a probe that may or may not find a usable cgroup
hierarchy. The new `tryGetCgroupsPath()` returns `std::optional`, so:

  * Cgroups absent (the expected case in restricted containers): callers
    take the empty branch and emit a single `LOG_DEBUG` line.
  * Cgroups present but reader construction fails (file vanished,
    permissions, etc.): callers still log the full diagnostic via
    `tryLogCurrentException`. Truly unexpected errors are preserved.

Both call sites in `MemoryWorker` and `AsynchronousMetrics` are updated
accordingly. The unused `FILE_DOESNT_EXIST` ErrorCodes declaration in
`MemoryWorker.cpp` is removed.

Fixes #51
The "Test chdb DataStore tests against upstream chdb (latest tag)" step
fetches the latest chdb release tag via an unauthenticated curl to
api.github.com, which is subject to a 60 req/hr per-IP rate limit.
On busy CI runners (especially shared GitHub-hosted macOS arm64 boxes)
this regularly returned HTTP 403, the empty body crashed the downstream
`json.load`, and the whole job failed before a single test ran.

This change:

* Sends `Authorization: Bearer $GITHUB_TOKEN` when the variable is set,
  raising the rate limit from 60 req/hr to 5000 req/hr.
* Retries the request up to 3 times with backoff to absorb transient
  network/rate-limit hiccups.
* Emits a clear actionable error (suggesting `CHDB_TAG=` as a workaround)
  when the lookup ultimately fails, instead of an opaque JSONDecodeError.
* Wires `GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}` through to the step
  in all four wheel-build workflows so the auth path is exercised in CI.

The script keeps working unauthenticated (e.g. for local runs) when no
GITHUB_TOKEN is set; the retry/auth machinery is purely additive.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cgroups stack trace dumped to stderr on every query in restricted k8s containers

1 participant