You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The PHP Valgrind Tests (EngineVersion 9.0) CI job intermittently fails with possibly lost memory reported against glide-core global allocations, even when
all tests pass. The failure signature and error count fluctuate run-to-run
(e.g. ERROR SUMMARY: 2 errors from 2 contexts on one run, 3 errors from 3 contexts on the next), which points at timing-dependent thread-teardown
behavior rather than a real leak.
These allocations are process-lifetime globals in the Rust glide-core that
are freed at process teardown, not per-connection. They are not caused by the
PHP/C extension code. As a stopgap, valgrind.supp suppressions were added in
PR #321 so the flaky gate stops blocking unrelated PRs. This issue tracks the
proper fix.
glide-core client registry — glide_core::scope::register_client
inserting into DashMap<u64, glide_core::client::Client>; the hashbrown RawTable::reserve_rehash grows the backing table and valgrind marks the
pre-rehash table possibly lost. Reached via the normal connect() path for
both standalone (create_glide_client) and cluster
(create_glide_cluster_client) clients. ~2.2–2.6 KB across 15–16 blocks.
Why the existing suppressions don't catch these
valgrind.supp already has opentelemetry_resource_from_detectors, but it only
matches the hashbrown ... insert → from_detectors frame chain. The observed
record takes the Value::clone path instead, so it isn't matched.
Investigate whether glide_core::scope::register_client / GlideOpenTelemetry::initialise can be given deterministic teardown (or
pre-sized to avoid rehash) so valgrind no longer reports possibly lost.
Reassess the CI gate: possibly lost > 0 on process-lifetime globals is a
weak signal; consider gating only on definitely lost / indirectly lost,
or on the delta vs a baseline, to reduce false positives.
Once addressed upstream, remove the stopgap suppressions.
Acceptance criteria
The Valgrind 9.0 job passes deterministically across repeated runs.
Root cause fixed in glide-core (or CI gate hardened) rather than relying
solely on suppressions.
Stopgap suppressions removed if no longer needed.
Confirmed pre-existing on main
To rule out PR #321 (mTLS) as the cause, the same PHP CI workflow was run via workflow_dispatch against main (which contains none of the mTLS changes):
Result on main: the Valgrind 9.0 job also fails with no test failure — possibly lost: 2,506 bytes in 16 blocks, ERROR SUMMARY: 3 errors from 3 contexts. The three possibly lost records are identical in origin to those
seen on the PR:
glide_core::scope::register_client DashMap rehash via create_glide_client
(standalone, ~2.2 KB)
glide_core::scope::register_client DashMap rehash via create_glide_cluster_client (cluster, ~280 bytes)
Byte/block counts jitter slightly run-to-run (e.g. 2,506 vs 2,522 bytes; 14 vs
15 blocks), which is the DashMap-rehash timing variance that also makes the error
count flicker (2 ↔ 3) and the gate flaky. This confirms the issue is pre-existing
in glide-core and independent of PR #321.
Important
Blocked by / fix first: #330. The Valgrind job is currently not triggered
on valkey-glide submodule bumps (path-filter gap in php.yml), which is how
this regression landed unexercised. #330 must be fixed first so CI actually
runs the Valgrind job on glide-core bumps; otherwise regressions like this will
continue to slip through regardless of the work here.
Summary
The PHP Valgrind Tests (EngineVersion 9.0) CI job intermittently fails with
possibly lostmemory reported against glide-core global allocations, even whenall tests pass. The failure signature and error count fluctuate run-to-run
(e.g.
ERROR SUMMARY: 2 errors from 2 contextson one run,3 errors from 3 contextson the next), which points at timing-dependent thread-teardownbehavior rather than a real leak.
These allocations are process-lifetime globals in the Rust glide-core that
are freed at process teardown, not per-connection. They are not caused by the
PHP/C extension code. As a stopgap,
valgrind.suppsuppressions were added inPR #321 so the flaky gate stops blocking unrelated PRs. This issue tracks the
proper fix.
Affected allocations (observed loss records)
OpenTelemetry resource init —
opentelemetry::common::Value::clone(
Box<[u8]>) →opentelemetry_sdk::resource::Resource::from_detectors,during one-time
GlideOpenTelemetry::initialise(once_cell). ~22 bytes.glide-core client registry —
glide_core::scope::register_clientinserting into
DashMap<u64, glide_core::client::Client>; thehashbrownRawTable::reserve_rehashgrows the backing table and valgrind marks thepre-rehash table
possibly lost. Reached via the normalconnect()path forboth standalone (
create_glide_client) and cluster(
create_glide_cluster_client) clients. ~2.2–2.6 KB across 15–16 blocks.Why the existing suppressions don't catch these
valgrind.suppalready hasopentelemetry_resource_from_detectors, but it onlymatches the
hashbrown ... insert → from_detectorsframe chain. The observedrecord takes the
Value::clonepath instead, so it isn't matched.Stopgap applied in PR #321
Added two tolerant suppressions to
valgrind.supp:opentelemetry_resource_value_clone_from_detectorsglide_core_scope_register_client_dashmap_rehashProper fix (this issue)
glide_core::scope::register_client/GlideOpenTelemetry::initialisecan be given deterministic teardown (orpre-sized to avoid rehash) so valgrind no longer reports
possibly lost.possibly lost > 0on process-lifetime globals is aweak signal; consider gating only on
definitely lost/indirectly lost,or on the delta vs a baseline, to reduce false positives.
Acceptance criteria
solely on suppressions.
Confirmed pre-existing on
mainTo rule out PR #321 (mTLS) as the cause, the same
PHP CIworkflow was run viaworkflow_dispatchagainstmain(which contains none of the mTLS changes):Result on
main: the Valgrind 9.0 job also fails with no test failure —possibly lost: 2,506 bytes in 16 blocks,ERROR SUMMARY: 3 errors from 3 contexts. The threepossibly lostrecords are identical in origin to thoseseen on the PR:
Resource::from_detectorsinit (~22 bytes)glide_core::scope::register_clientDashMap rehash viacreate_glide_client(standalone, ~2.2 KB)
glide_core::scope::register_clientDashMap rehash viacreate_glide_cluster_client(cluster, ~280 bytes)Byte/block counts jitter slightly run-to-run (e.g. 2,506 vs 2,522 bytes; 14 vs
15 blocks), which is the DashMap-rehash timing variance that also makes the error
count flicker (2 ↔ 3) and the gate flaky. This confirms the issue is pre-existing
in glide-core and independent of PR #321.
Important
Blocked by / fix first: #330. The Valgrind job is currently not triggered
on
valkey-glidesubmodule bumps (path-filter gap inphp.yml), which is howthis regression landed unexercised. #330 must be fixed first so CI actually
runs the Valgrind job on glide-core bumps; otherwise regressions like this will
continue to slip through regardless of the work here.