fix(catalog): watch YAML data files for live reload in MCP and agent catalogs - #2997
Merged
google-oss-prow[bot] merged 3 commits intoJul 28, 2026
Merged
Conversation
pboyd
force-pushed
the
fix/catalog-yaml-file-watching
branch
from
July 21, 2026 14:51
2567140 to
7e3f389
Compare
pboyd
force-pushed
the
fix/catalog-yaml-file-watching
branch
3 times, most recently
from
July 22, 2026 17:05
b4fa893 to
edadfa7
Compare
| ch, err := basecatalog.GetMonitor().Path(ctx, path) | ||
| if err != nil { | ||
| glog.Errorf("unable to watch MCP catalog file %s: %v", path, err) | ||
| watchFailed = true |
Contributor
There was a problem hiding this comment.
one wrong path, skips all the live updates even in correct path, what do you think about skipping only the bad ones?
Member
Author
There was a problem hiding this comment.
This is fixed in the last commit.
| // cross-contamination when cleaning up shared CatalogSource records. | ||
| // This is called by the unified loader when becoming leader. | ||
| func (l *ModelLoader) PerformLeaderOperations(ctx context.Context, allKnownSourceIDs mapset.Set[string]) error { | ||
| return l.performLeaderWrites(ctx, allKnownSourceIDs) |
Contributor
There was a problem hiding this comment.
shouldn't we add also here .state.WaitForInflightWrites(30 * time.Second) like we have done in the other plugins?
Member
Author
There was a problem hiding this comment.
Also fixed in the last commit.
…catalogs Previously, only sources.yaml was watched for file changes. YAML data files referenced via yamlCatalogPath required a pod restart or a sources.yaml touch to pick up edits. MCP catalog: make yamlMCPProvider.Servers() long-lived. After emitting the initial batch it sends a zero-value sentinel record, registers a file watcher on each data file via basecatalog.GetMonitor().Path(), and re-emits + re-sends the sentinel on every detected change. The loader's PerformLeaderOperations now returns immediately after launching per-source goroutines; loadServersFromProvider uses sync.OnceFunc to release the pre-registered TrackWrite slot on the first sentinel so that WaitForInflightWrites unblocks correctly. Subsequent batches (file reloads) perform orphan cleanup and status updates the same way as the initial batch. Agent catalog: add watchAndLoadFromYAML that performs the initial load, releases its TrackWrite slot, then loops on GetMonitor().Path() events to reload on change. PerformLeaderOperations is restructured to launch goroutines and return immediately, matching the MCP and model loader patterns. Tests updated to use t.Context() so background goroutines are cancelled when each test ends, and to break on sentinel records rather than waiting for channel close. Assisted-by: Claude Sonnet 4.6 Signed-off-by: Paul Boyd <paul@pboyd.io>
…e leak - Set up file watcher before initial read to avoid missing changes that arrive during the read - Emit error + sentinel on YAML read failure so the consumer updates source status to "error" instead of silently continuing - Skip orphan cleanup when all models fail to load, preserving stale data until the error is fixed - Reset failedModels between batch cycles so partial-failure state does not leak into subsequent batches Assisted-by: Claude Sonnet 4.6 Signed-off-by: Paul Boyd <paul@pboyd.io>
- MCP provider: skip individual bad file watcher paths instead of failing all live updates. Changed break to continue on error and replaced watchFailed boolean with watchCount integer. - Model loader: add missing WaitForInflightWrites drain step to PerformLeaderOperations, matching MCP and agent loader patterns. This prevents a race where old goroutine writes could re-insert data during orphan cleanup. Assisted-by: Claude Sonnet 4.5 Signed-off-by: Paul Boyd <paul@pboyd.io>
pboyd
force-pushed
the
fix/catalog-yaml-file-watching
branch
from
July 28, 2026 13:46
edadfa7 to
bac0260
Compare
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Al-Pragliola The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Previously, only sources.yaml was watched for file changes. YAML data files referenced via yamlCatalogPath required a pod restart or a sources.yaml touch to pick up edits.
MCP catalog: make yamlMCPProvider.Servers() long-lived. After emitting the initial batch it sends a zero-value sentinel record, registers a file watcher on each data file via basecatalog.GetMonitor().Path(), and re-emits + re-sends the sentinel on every detected change. The loader's PerformLeaderOperations now returns immediately after launching per-source goroutines; loadServersFromProvider uses sync.OnceFunc to release the pre-registered TrackWrite slot on the first sentinel so that WaitForInflightWrites unblocks correctly. Subsequent batches (file reloads) perform orphan cleanup and status updates the same way as the initial batch.
Agent catalog: add watchAndLoadFromYAML that performs the initial load, releases its TrackWrite slot, then loops on GetMonitor().Path() events to reload on change. PerformLeaderOperations is restructured to launch goroutines and return immediately, matching the MCP and model loader patterns.
Tests updated to use t.Context() so background goroutines are cancelled when each test ends, and to break on sentinel records rather than waiting for channel close.
How Has This Been Tested?
Verified locally on a dev cluster.
Merge criteria:
DCOcheck)