Merge users/sanvern/c2cstage-mergesfrom-stage2 into current branch#3408
Draft
Copilot wants to merge 121 commits into
Draft
Merge users/sanvern/c2cstage-mergesfrom-stage2 into current branch#3408Copilot wants to merge 121 commits into
Copilot wants to merge 121 commits into
Conversation
…y. This is to avoid spamming the azcopy codeowners.
…zCopy with environment variables and XDM custom paths
- Add new moverDependencies.go with property getters/setters for copy and sync args - Expose internal sync functionality for external mover integration - Add RawMoverSyncCmdArgs struct for mover-specific sync operations - Create utility functions for cooking sync arguments and client options - Update enumerator initialization to work with mover dependencies
- Add buildmode package to distinguish between default and mover builds - Create conditional compilation using 'mover' build tag
- Add TraverserErrorItemInfo interface for standardized error reporting - Enable error channel support across sync, local, and remote traversers - Implement ErrorFileInfo struct for local file enumeration errors - Add SyncEnumeratorOptions to configure error handling during sync - Provide structured error information including file path, size, and location
- Add GetMemAvailable() and GetTotalPhysicalMemory() functions - Implement Windows version using GlobalMemoryStatusEx API - Implement Linux/macOS version using /proc/meminfo parsing - Support platform-specific memory retrieval with build tags - Return memory values in bytes for consistent interface - Add error handling for system call failures and parsing errors
…e pacer target rates - Add validation loop in RequestTrafficAllocation to wait for positive rate limits - Adding UpdateTargetBandwidth in JobsAdmin interface
- Add ChangeLogLevel method to jobLogger and sysLogger implementations - Enable runtime log level changes for individual jobs via JobsAdmin - Add ChangeLogLevel support in JobMgr to update job-specific loggers
- Properties to track source and destination folders - Properties to track source files and folders that don't require transfer
…ure file level error message for XDM consumption
…me and ChangeTime for local traverser consumption
- [XDM][Az Files support] Don't lose data on retry of a failed additive copy, due to newer LastWriteTime Setting LastWriteTime to minimal timestamp in prologue - Setting LastWriteTime for directories - Setting ChangeTime for files and directories whenever its available
…ing metadata only transfer
…for non-folder-aware target
- Adding ChangeTime and LastWriteTime properties - Func to update those fields from AzFile file\directory properties and blob metadata - Updated file and blob traversers to populate the fields
…n AzCopy - In the first version, it will be tailored for XDM Mover consumption - Includes initial integration changes - Includes throttling mechansim - Include new and updates to existing models NOTE: This is expected to build successfully but will not work. Subsequent integration changes will enable it properly
- Updated local, S3, blob traversers to handle directory and prefixes conditionally - Updated error channel writes for few failure scenarios - Fixed a counter increment bug in local traverser non-recursive flow - Added ancestor check for symlink loop detection in local traverser - Refactored WalkWithSymlinks to take in a struct instead of individual arguments
…ing final job status message for mover build mode
…averser, Adding a prefix to errors
…s for NFS file share target in error channel
…r precision time compare for nfs file share target
- Introduced sync scan counters for enumeration failure - Resolving the traverser root path when we follow symlinks. This handles the scenarios where local traverser is executed for a symlink to a folder - Incrementing scanning errors for file and folder during enumeration - Sending enumeration failure in the provided error channel for different error scenarios
* Mover: Added all blob/blobfs to blob/blobfs pairs to use destCleaner function, as we want folders to be deleted if not present on the source * mover: added mover check for adding root to be processed pre-emptively for hns blob container as source
…ms-copy-source-authorization header is populated in PutBlockFromURL requests. This is needed for trusted access to work (#3267)
* Users/sanvern/c2cprivate networking 09122025 (#3234) * Contains following changes to enable private networking for C2C workflow. 1. Update all references of Minio to latest V7 version. 2. Private networking changes enabled behind PrivateNetworkConfig.Enabled flag. 3. Implementation of custom roundtrip for Minio client to route the traffic through private endpoints. 4. Distributing the HTTP requests among private endpoints configured. 5. Maintain private endpoint status for error reporting. 6. S3chunkReader for Putblob implementation which will be merged SingleChunkReader. 7. Handling and reporting HTTP and S3Errors. --------- Co-authored-by: Akash C <akchidan@microsoft.com> * remove debug logs (#3325) * BugFix for data integrity - Buffer returned too early to buffer pool even before it is used, resulting into buffer overwrite. * Users/pbanakar/s3 chunk reader fix (#3330) * Deadlock bugfix during retryBlockingPrefetch * connection pooling and racecondition bugfix * changing cooldown period to 10mins and making private network check generic * remove connection pooling from global client and use default transport * Avoiding flooding of http 404 errors * Racecondition BugFix for traverser and STE s3 client sharing the same provider instance overwriting each other. * Racecondition in acessing the memory mapped file after unmapping * Removing redundant logging * Bugfix for cancel operation on sync job. * Improving error message (#3340) * update error message * update * Reverting commits a7c444f, 21f20f7 * Bugfix for crashes on cancel operation for sync job (#3342) * Addressing PR review comments. --------- Co-authored-by: sgv-msft <105514088+sgv-msft@users.noreply.github.com> Co-authored-by: Parmananda Banakar <pbanakar@microsoft.com> Co-authored-by: pbanakar-microsoft <162348738+pbanakar-microsoft@users.noreply.github.com> Co-authored-by: Santosh Gopalakrishna Vernekar <sanvern@microsoft.com>
* adding blob type to sync * adding sync from CLI * fixing build error with sync blobtype
* Skip Archive/Glacier objects during enumeration for S3 * using new counter variable for archive objects skipped * removing GetSkippedSpecialFileCount() * changing counter to uint64 * Review point fixed for incrmeentEnumCounter func * MoverDependencies: Add init and close utils for scanning logger
…-azcopy into users/sanvern/c2cstage-mergesfrom-stage2
…m-stage2' into copilot/get-latest-commit-info # Conflicts: # .github/CODEOWNERS # azcopy/syncComparator.go # azcopy/syncProcessor.go # cmd/copy.go # cmd/copyEnumeratorInit.go # cmd/credentialUtil.go # cmd/jobsResume.go # cmd/login.go # cmd/root.go # cmd/sync.go # cmd/syncEnumerator.go # cmd/zc_processor.go # cmd/zt_generic_traverser_test.go # cmd/zt_sync_file_file_test.go # common/credentialFactory.go # common/lifecyleMgr.go # common/rpc-models.go # e2etest/newe2e_resource_managers_blob.go # e2etest/newe2e_resource_managers_blobfs.go # e2etest/newe2e_resource_managers_local.go # e2etest/newe2e_task_validation.go # go.mod # go.sum # jobsAdmin/JobsAdmin.go # jobsAdmin/init.go # ste/folderCreationTracker.go # ste/folderCreationTracker_test.go # ste/mgr-JobMgr.go # ste/sender-azureFile.go # ste/sender-azureFileFromLocal.go # ste/sender-blobFS.go # ste/xfer-anyToRemote-fileProperties.go # traverser/syncIndexer.go # traverser/zc_enumerator.go # traverser/zc_traverser_blob.go # traverser/zc_traverser_file.go # traverser/zc_traverser_local.go # traverser/zc_traverser_s3.go
…lation errors in common, ste, traverser packages Co-authored-by: varshaj-msft <262910439+varshaj-msft@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Get latest commit on branch c2cstage-mergesfrom-stage2
Merge users/sanvern/c2cstage-mergesfrom-stage2 into current branch
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the latest commit from
users/sanvern/c2cstage-mergesfrom-stage2and resolves resulting compilation conflicts caused by architectural divergence between the two branches (lifecycle manager refactor in HEAD vs. source branch additions).Description
Feature / Bug Fix: Merge latest source branch commits and resolve compilation errors from conflicting architectural changes (minio v6→v7 upgrade, lifecycle manager move from
common/tocmd/, new NFS/SMB cross-protocol support, credential factory signature changes).Related Links:
Key conflict resolutions
ste/sender-azureFile.go/sender-azureFile-helper.go: Removed duplicate method declarations (addNFSPropertiesToHeaders,addNFSPermissionsToHeaders,addPermissionsToHeaders,addSMBPropertiesToHeaders) now canonical in helper file; fixedShouldTransferLastWriteTime()calls to passFromToarg per updated signature.ste/mgr-JobPartMgr.go: Added missingcredInfo common.CredentialInfofield; fixedRegisterPropertiesTransferto 2-arg call; simplifiedCpkInfo()to match new single-returnGetCpkInfo.ste/mgr-JobMgr.go: Re-addedGetJobErrorHandler()toIJobMgrinterface withjobMgrErrorHandlerimpl delegating to the job's logger (replaces oldjobErrorHandlerfield removed by source branch).ste/sender-blobFS.go: Wrappedcommon.Metadataincommon.SafeMetadataforAddStatToBlobMetadatacalls (new 3-arg signature requiresMetadataStore).ste/sourceInfoProvider-File.go: ImplementedReadLink()via raw HTTPGET ?restype=symboliclinksincefile.Client.GetSymbolicLink()exists only in the internal generated SDK client, not the public API.common/util.go: AddedIsNFSCopy()/SetNFSFlag()global helpers (source branch addition not present in HEAD).traverser/zc_traverser_s3.go: Updated minio v6→v7 API calls (StatObjectnow requirescontext.Context;ListObjectsV2replaced byListObjectswithListObjectsOptions;CreateS3Clientsignature updated).jobsAdmin/,cmd/: Restored HEAD~1 versions to avoidcommon.LifecycleMgrcircular-import conflict (source branch kept full LCM incommon/; HEAD moved it tocmd/); removed deprecatedjobErrorHandlerparam fromNewJobMgrcallers.Outstanding
azcopy/package still fails to build:SkippedHardlinkCountmissing fromListJobSummaryResponse, andPosixPropertiesStyle/JobErrorHandlermissing fromCopyJobPartOrderRequest/ResumeJobRequest. These fields need to be added to the relevant common structs.Type of Change
How Has This Been Tested?
Incremental
go build ./...used to validate each fix. Thecommon/,ste/,traverser/,jobsAdmin/packages now compile cleanly. Full end-to-end and integration testing pending resolution of remainingazcopy/build errors.💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.