refactor: eliminate duplication and fix regressions - #16
ClaudiaFang wants to merge 7 commits into
Conversation
- Add last_commit_id to GitLabFileResponse - Use last_commit_id as sha in GitLabService.getFile - Add rename detection to SyncManager.processSingleBatchPush
…rvice and fix getFullPath regression
There was a problem hiding this comment.
Code Review
This pull request refactors the codebase to improve modularity and maintainability, notably by introducing a BaseGitService to unify GitHub and GitLab logic and decomposing large methods in SyncStatusView and SyncManager. However, the refactoring of listFiles in both Git services introduced regressions where the rootPath prefix is no longer stripped, which will break path matching in the UI. Furthermore, the status refresh logic and batch push operations are currently inefficient, performing excessive sequential network requests that could lead to API rate limiting and performance issues in larger vaults.
| return data.tree | ||
| .filter(item => item.type === 'blob') | ||
| .map(item => item.path); | ||
|
|
||
| // Filter by rootPath if set | ||
| if (this.rootPath) { | ||
| const prefix = this.rootPath + '/'; | ||
| return allFiles | ||
| .filter(file => file.startsWith(prefix)) | ||
| .map(file => file.substring(prefix.length)); | ||
| } | ||
|
|
||
| return allFiles; | ||
| .map(item => item.path) | ||
| .filter(p => !this.rootPath || p.startsWith(this.rootPath)); |
There was a problem hiding this comment.
The listFiles implementation contains a regression and a logic error:
- Regression: It no longer strips the
rootPathprefix from the returned paths. Since the rest of the application (e.g.,SyncStatusViewandGitignoreManager) expects paths relative to the configured root, this will break path matching and status detection for any user with a non-emptyrootPath. - Imprecise Filter: Using
startsWith(this.rootPath)can match unintended directories (e.g., ifrootPathisdocs, it will incorrectly matchdocs-internal/). It should check for a proper path boundary.
const prefix = this.rootPath ? this.rootPath.replace(/\/+$/, '') + '/' : '';
return data.tree
.filter(item => item.type === 'blob')
.map(item => item.path)
.filter(p => !prefix || p.startsWith(prefix))
.map(p => prefix ? p.substring(prefix.length) : p);| const url = `${this.baseUrl}/api/v4/projects/${encodedProjectId}/repository/tree?ref=${branch}&recursive=true&per_page=100`; | ||
| const response = await this.safeRequest(url, 'GET'); | ||
| const data = response.json as GitLabTreeItem[]; | ||
|
|
||
| return data | ||
| .filter(item => item.type === 'blob') | ||
| .map(item => item.path); | ||
|
|
||
| // Filter by rootPath if set | ||
| if (this.rootPath) { | ||
| const prefix = this.rootPath + '/'; | ||
| return allFiles | ||
| .filter(file => file.startsWith(prefix)) | ||
| .map(file => file.substring(prefix.length)); | ||
| } | ||
|
|
||
| return allFiles; | ||
| .map(item => item.path) | ||
| .filter(p => !this.rootPath || p.startsWith(this.rootPath)); |
There was a problem hiding this comment.
Similar to the GitHub service, listFiles here has significant regressions:
- It no longer uses the
pathparameter in the GitLab API request, which is inefficient as it fetches the entire repository tree instead of just the relevant subtree. - It does not strip the
rootPathprefix from the results, which breaks path matching in the UI for subfolder-mapped vaults. - The
startsWithfilter is imprecise and can match sibling directories with similar names.
| const url = `${this.baseUrl}/api/v4/projects/${encodedProjectId}/repository/tree?ref=${branch}&recursive=true&per_page=100`; | |
| const response = await this.safeRequest(url, 'GET'); | |
| const data = response.json as GitLabTreeItem[]; | |
| return data | |
| .filter(item => item.type === 'blob') | |
| .map(item => item.path); | |
| // Filter by rootPath if set | |
| if (this.rootPath) { | |
| const prefix = this.rootPath + '/'; | |
| return allFiles | |
| .filter(file => file.startsWith(prefix)) | |
| .map(file => file.substring(prefix.length)); | |
| } | |
| return allFiles; | |
| .map(item => item.path) | |
| .filter(p => !this.rootPath || p.startsWith(this.rootPath)); | |
| const searchPath = this.rootPath ? this.rootPath.replace(/\/+$/, '') : ''; | |
| const url = `${this.baseUrl}/api/v4/projects/${encodedProjectId}/repository/tree?ref=${branch}&recursive=true&per_page=100${searchPath ? `&path=${encodeURIComponent(searchPath)}` : ''}`; | |
| const response = await this.safeRequest(url, 'GET'); | |
| const data = response.json as GitLabTreeItem[]; | |
| const prefix = searchPath ? searchPath + '/' : ''; | |
| return data | |
| .filter(item => item.type === 'blob') | |
| .map(item => item.path) | |
| .filter(p => !prefix || p.startsWith(prefix)) | |
| .map(p => prefix ? p.substring(prefix.length) : p); |
| private async performStatusCheck(filesToCheck: Array<TFile | string>): Promise<void> { | ||
| const total = filesToCheck.length; | ||
| for (let i = 0; i < total; i++) { | ||
| const file = filesToCheck[i]; | ||
| if (file) { | ||
| await this.refreshFileStatus(file); | ||
| checked++; | ||
| const c = this.containerEl.children[1]; | ||
| if (c) { | ||
| const fill = c.querySelector('.ssv-progress-fill'); | ||
| const text = c.querySelector('.ssv-progress-text'); | ||
| if (fill && text) { | ||
| const pct = Math.round((checked / total) * 100); | ||
| fill.setAttr('style', `width: ${pct}%`); | ||
| text.textContent = `Checking files… ${checked}/${total} (${pct}%)`; | ||
| } | ||
| } | ||
| } | ||
| this.updateRefreshProgress(i + 1, total); | ||
| } | ||
| } |
There was a problem hiding this comment.
This loop performs a sequential network request (getFile) for every file in the vault during a refresh. For vaults with many files, this will be extremely slow and will likely trigger API rate limits on GitHub/GitLab, leading to a poor user experience.
Recommendation: Optimize the status check by leveraging the blob SHAs already provided by the Git provider's tree API (e.g., in listFiles). Compare these remote SHAs with local metadata or calculated hashes to identify changed files. Full content should only be fetched when a diff is explicitly requested or during the actual sync operation.
| const remote = await this.gitService.getFile(path, this.settings.branch); | ||
| await this.gitService.pushFile(path, content, this.settings.branch, `Update ${name} from Obsidian`, remote.sha || undefined); | ||
| const newRemote = await this.gitService.getFile(path, this.settings.branch); | ||
| this.settings.syncMetadata[path] = { lastSyncedSha: newRemote.sha, lastSyncedAt: Date.now(), lastKnownPath: path }; |
There was a problem hiding this comment.
In a batch push operation, performing two extra getFile calls per file (one before to get the SHA and one after to update metadata) is inefficient and doubles the network overhead.
- The
existingShais often already known from the preceding conflict check or local metadata. - The
pushFileAPI response from GitHub (and commit info from GitLab) can be used to update metadata without an additional fetch. Consider updating thepushFileinterface to return the new SHA.
This PR incorporates additional fixes and refactoring after the merge of #15.