feat(metric): add APICallMutex wait duration and TryLock skip metrics - #76
Merged
Merged
Conversation
Add metrics to observe mutex contention on LXD API calls: - mutex_wait_duration_seconds histogram for Lock() wait time - mutex_skipped_total counter for TryLock() skips prompt: Implement the following plan: APICallMutex のロック待ち時間メトリクス追加 ---- Added LXDAPIMutexWaitDuration histogram and LXDAPIMutexSkippedTotal counter metrics. Instrumented all Lock() call sites (allocateInstance, isExistInstanceWithTimeout, GetResourceFromLXD) and the TryLock site in scrape_lxd.go. Used observer pattern to avoid circular dependency from lxdclient package.
WithContext was called before APICallMutex.Lock(), causing concurrent goroutines to overwrite each other's context on the shared client. Additionally, context was not reset on error paths, leaving the client with a canceled context for all subsequent calls. Changes: - Move WithContext after Mutex acquisition in all call sites - Add defer reset to context.Background() on all paths - Add Mutex to setLXDHostResourceCache (was missing) - Remove xsemaphore from GetResourceFromLXDWithClient (callers now hold the Mutex and set context) - Add regression tests for Mutex-based concurrency control
fix(lxdclient): fix context race condition in ProtocolLXD WithContext
3 tasks
whywaita
added a commit
that referenced
this pull request
Mar 10, 2026
revert(metric): remove APICallMutex mutex wait metrics from #76
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
APICallMutexcontention: lock wait duration histogram and TryLock skip counterLock()call sites and theTryLock()site in scrape_lxd.golxdclientpackageChanges
server/pkg/metric/lxd_api.go: AddLXDAPIMutexWaitDuration(histogram) andLXDAPIMutexSkippedTotal(counter) metrics, plusObserveMutexWaithelperserver/pkg/lxdclient/metrics_hook.go: AddMutexWaitObservertype andSetMutexWaitObserver/observeMutexWaitfunctionsserver/main.go: Register the two new metrics with Prometheusserver/pkg/api/pool.go: Measure mutex wait inallocateInstanceserver/pkg/api/server_host.go: Measure mutex wait inisExistInstanceWithTimeoutserver/pkg/lxdclient/resource.go: Measure mutex wait inGetResourceFromLXDserver/pkg/metric/scrape_lxd.go: Increment skip counter onTryLockfailureserver/pkg/metric/lxd_api_test.go: Tests forObserveMutexWaitandLXDAPIMutexSkippedTotalMetrics
Test Plan
go test ./pkg/metric/...passesgo vet ./...passesgo build ./...passes