[pull] master from rclone:master#238
Open
pull[bot] wants to merge 4410 commits into
Open
Conversation
In this commit 0db3e7a vfs: fix slow nfs serve by adding --vfs-handle-caching We added --vfs-handle-caching but unfortunately forgot to disable it for the TestRWCacheUpdate test.
The Linkbox open API (/api/open/file_search) no longer returns download URLs, breaking all downloads. This switches to using the web API (/api/file/my_file_list/web) which requires email+password authentication but returns working download URLs. This will unfortunately require changing your existing rclone config. - Add email, password, and web_token config options - Add web API login via /api/user/login_email with token caching and retry - Create separate CDN HTTP client with HTTP/2 disabled and browser User-Agent to avoid CDN fingerprint blocking - Remove searchOK regex and name-filtering (web API doesn't support it)
…entStream Decoder Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.96.0 to 1.97.3. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.96.0...service/s3/v1.97.3) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.97.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Use textContent instead of innerHTML when inserting the dropdown label to prevent DOM text being reinterpreted as HTML. Fixes https://github.com/rclone/rclone/security/code-scanning/176
This also streamlines the update process in RELEASE.md and the Makefile
This notes in particular not to use `--links` but to use `--local-links`. Fixes #9317
… cache file Before this change when a cache item was in its grace period (with HandleCaching) and the file is reopened, _checkObject runs before the grace timer recovery check. If the remote object's fingerprint changed _checkObject removes the cache file from disk. However the grace recovery path still reused the now-stale fd pointing to a deleted inode, skipping _createFile entirely. This left no cache file on disk, causing cache.Exists() to return false and breaking rename-while-writing logic. Fix this by checking the cache file still exists before reusing the fd in grace recovery. If the file was removed, close the stale fd and downloaders and fall through to _createFile. Also update the fingerprint in item.rename after setting the new object, preventing unnecessary cache invalidation when a file is reopened after a rename. This was discovered in the integration tests on backends that update modtime on rename (like mailru).
This restricts the GITHUB_TOKEN to least-privilege in all workflows, fixing CodeQL code-scanning alerts for actions/missing-workflow-permissions.
Enable HTTP/2 cleartext support for non-TLS listeners.
Previously ssh.InsecureIgnoreHostKey() was set unconditionally as the default HostKeyCallback with no indication to the user. This logs a warning pointing users to the documentation on how to enable host key validation. See: https://github.com/rclone/rclone/security/code-scanning/167
Replace racy assert.False/Sleep/assert.True pattern with assert.Eventually which polls until the async download completes.
…rsioning fails When a user has --s3-versions set but lacks the s3:GetBucketVersioning permission, GetBucketVersioning returns an error and isVersioned() caches the result as false. This caused CleanUpHidden (backend cleanup-hidden) to silently exit with "bucket is not versioned so not removing old versions", ignoring the user's explicit --s3-versions flag. Fix this by trusting the explicit --s3-versions flag in purge(), bypassing the GetBucketVersioning check when the user has explicitly declared the bucket is versioned.
The Global Acceleration Endpoint (cos.accelerate.myqcloud.com) of Tencent COS does not seem to support "CreateBucket" (maybe also other bucket management operations). Since the acceleration functionality must be enabled per-bucket in the Tencent Cloud console, the bucket will always exist before this endpoint is used, so this check can be safely skipped. Now, "no_check_bucket = true" will be auto set when using this endpoint. Why "NewFs()": on-the-fly remotes (connection string remotes), for example, ":s3,provider=TencentCOS,...:..." will also be fixed. Why no unit test: I can't find a good way to test "NewFs()" without leveraging live endpoints. I think we can extract all existing mutations for different providers (e.g., AWS, Fastly, and Rabata) from "NewFs()" to a new function in the future. Some Tencent docs about this CDN endpoint: - English: Global Acceleration Endpoint | https://www.tencentcloud.com/pt/document/product/436/40700 - Chinese: 对象存储 全球加速概述_腾讯云 | https://cloud.tencent.com/document/product/436/38866 Assisted-By: OpenCode
This adds a new gui command which runs an embedded copy of the GUI at https://github.com/rclone/rclone-web/ The GUI release is fetched as part of the CI build.
…9291 These Debugf calls in NewFilter() ran during GlobalOptionsInit(), before InitLogging() configured the JSON log format. This caused plain-text debug lines to leak to stderr when --use-json-log was set, breaking tooling that expected only JSON output. The resolved time values are already available via --dump filters so this commit removes the debug messages.
`json:"entry_permissions"` is known to be either empty [] or of
structure {string: boolean}. This may have been a breaking API change on
Drime's side. Because EntryPermissions is not used, the type was changed
to `any` to capture both cases, otherwise we could implement custom
unmarshalling for that type.
The fetch-gui-dist.sh script calls the GitHub releases API unauthenticated, which is limited to 60 requests/hour per source IP. GitHub Actions runners share outbound IPs, so this quota is regularly exhausted. Pass GITHUB_TOKEN (or GH_TOKEN) as an Authorization header when present, raising the limit to 1000/hour, and wire secrets.GITHUB_TOKEN into the workflow step. Local unauthenticated runs still work.
- Fail gracefully if `make fetch-gui` hasn't been run - Return errors instead of panic or fatal errrors - Don't run `make fetch-gui` on every make since we have it in the workflow
ResetCounters unconditionally restarted the average loop, spawning a ticker goroutine that pinned the StatsInfo even when no loop had been running before. statsGroups.delete calls ResetCounters on every removed group, so deleting N stats groups leaked N goroutines and prevented GC of the underlying StatsInfo objects. Only restart the loop if it was active before the reset.
…kend Before this change, if the user changed their password or public-key and the auth proxy script returned updated config parameters for the backend (eg a rotated api_key) rclone would continue to re-use the old backend with the old config parameters out of the fscache. This was because both the VFS cache and the fs/cache key were derived from the user name only, so a change in the user's password or public-key did not invalidate the cached backend. Fix this by deriving the cache key from the user plus a hash of the password/public-key, so a credential change forces a fresh backend. The hash uses a per-process random HMAC key so the fragment that appears in logs cannot be brute-forced offline.
Added information about an alternative, easier way to access folders inside "Computers" using rclone. Expanded details about folder behavior in "Computers".
…king Run still uses a fresh local directory as the backing Fs that the server wraps. RunWithBackend takes an extra remote name (e.g. "TestS3Minio:") and uses a random subdirectory of that remote instead, starting the matching fstest/testserver/init.d script on the way in and tearing it down on the way out. AuthProxy is only run for the local backend.
…g in memory Previously serve s3 buffered every part of a multipart upload in memory (in the gofakes3 S3 library) and concatenated them when the upload completed, so memory use grew with the size of the upload. serve s3 now streams the parts, in part-number order, into a single PutStream upload to the underlying remote, which performs its own upload with bounded memory. The whole file is never held in memory - memory use is bounded by the parts in flight. This works for any remote that supports PutStream (nearly all, including crypt) and for any part size, so clients that don't produce uniform-sized parts (for example PostgreSQL backup tools such as pgBarman and pgBackRest) work too. Parts must arrive in ascending, contiguous part-number order; parts uploaded out of order are buffered until their turn, and there is no per-part retry (a failure aborts the whole upload). These trade-offs are documented. Passing --disable-multipart-streaming, or using a remote without PutStream, reverts to buffering the parts in memory (the previous behaviour); a one-off NOTICE is logged the first time this happens. Fixes #7453
TestS3Minio brings up a minio container via the fstest/testserver framework, which exec's bash init.d scripts that shell out to docker. This is not available on all platforms - Windows has no POSIX shell to run the scripts, and macOS CI runners have no docker daemon - which caused the build to fail there. Add testy.SkipUnlessDocker to detect whether the framework can run and skip the test when it cannot.
#9471 The new "errors" dump flag makes the HTTP dump conditional on the transaction failing with a retryable error (a transport error, HTTP 429 or HTTP 5xx), so first-failure diagnostics can be captured without the noise of dumping every transaction. The existing dump flags continue to control what is dumped, for example --dump errors,bodies, and on its own --dump errors dumps the headers.
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to each HTTP transaction and logs the connection level events - DNS resolution, TCP connect, TLS handshake (including the negotiated TLS version, cipher, ALPN protocol and server certificate), connection reuse, request write and time to first response byte. Each line is tagged with the time elapsed since the start of the transaction and the request pointer so it can be correlated with the other dumps. This is complementary to the existing dump flags: it shows how the connection behaved rather than what was sent, which is useful for debugging connectivity, DNS, TLS, proxy and keep-alive problems.
Convert the bug report and feature request templates to GitHub issue forms. Rewrite the pull request template to require that non-trivial changes are discussed in an issue first, and to spell out that backend changes need a clean test_all run and a test account for the integration tester before they can be merged. Document the latter requirement in CONTRIBUTING.md. Remove the legacy ISSUE_TEMPLATE.md which is superseded by the forms.
#9412 The Cobra generated PowerShell completion script captures rclone's output through a pipeline with Invoke-Expression. PowerShell decodes that output using [Console]::OutputEncoding, which on non-UTF-8 hosts (for example PowerShell 5.1 on a Windows install with an OEM code page such as CP852) misinterprets the UTF-8 bytes rclone emits and corrupts remote and path names containing non-ASCII characters, so tab completion produces a path that does not exist. Inject "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" into the generated script immediately before the Invoke-Expression call. This is safe on PowerShell 7+, where UTF-8 is already the default. If the expected line is not present (for example after a Cobra template change) the script is emitted unmodified so we never produce a corrupted completion script.
Since v1.74.0, ncdu doubled every keystroke on Windows: pressing Enter entered a directory and immediately descended a level deeper, "n" sorted by name and then reversed the order, and so on. Arrow keys were unaffected. The cause is an upstream regression in github.com/gdamore/tcell/v2 pulled in by the v2.9.0 -> v2.13.8 bump in commit 23917555. The same regression affects lazygit and micro. There is no fixed tcell release, so pin tcell back to v2.9.0, the last version before that commit, until the input handling is fixed upstream. See: gdamore/tcell#1124 See: jesseduffield/lazygit#5344 Fixes #9539
Add a .github/CODEOWNERS file mapping backends and core subsystems to their maintainers so GitHub auto-requests reviews. Move the per-area responsibility mapping out of MAINTAINERS.md into CODEOWNERS, leaving MAINTAINERS.md as the maintainer list with a pointer to CODEOWNERS.
…ixes #8239 Option help strings are also used to generate the website documentation, so some contain markdown links with root-relative targets such as [encoding section in the overview](/overview/#encoding). These render correctly on rclone.org but are confusing in the interactive config prompt, where the user sees the raw markdown and the link has no reachable root. Rewrite such links to text (https://rclone.org/path) when showing an option's help in the interactive config. The raw help is left unchanged so documentation generation is unaffected.
Before this change, with --files-from and --no-traverse, a single file that could not be read (for example permission denied) stopped all the other files in the list being copied. This happened because the error was returned from the listing, which caused the whole source listing to be discarded. This change counts and logs such per-file errors and carries on, so the readable files are still copied and rclone exits with a non-zero error code. Fixes #9115
…folders This adds two new Dropbox backend flags: --dropbox-skip-shared-folders skips all shared folder mount points regardless of ownership. --dropbox-skip-unowned-folders only skips shared folders that are not owned by the current user. These help avoid backing up the same shared folder multiple times when backing up multiple Dropbox accounts. Fixes #9514
The HeadObject path translates a 404 (Not Found) and a 405 (Method Not Allowed) response into fs.ErrorObjectNotFound, but the GetObject path returns the raw AWS SDK error instead. With --s3-no-head-object set, Object.Open() is the first request to hit the source for a GET operation, so callers that rely on fs.ErrorObjectNotFound see an opaque "operation error S3: GetObject, ... StatusCode: 404" string when --s3-no-head-object is set. Mirror the error handling of HeadObject. The 405 (Method Not Allowed) case applies to GetObject for the same reason it applies to HeadObject: A request that specifies the versionId of a delete marker returns 405 instead of 404 (commit 8470bdf, AWS S3 delete-marker docs). Co-authored-by: Jan Schlien <rclone-github@jan-o-sch.net>
…8188 After an upload (notably multipart) to a slow backing remote, the file lives in the VFS and is returned by ListBucket, but node.DirEntry() stays nil until the --vfs-write-back writeback completes. HeadObject and GetObject returned gofakes3.KeyNotFound while it was nil, so a HEAD/GET in that window 404'd even though the object existed. getFileHashByte already falls back to hashing the VFS cache when the backing object isn't available yet. Drop the early nil return, pass the node (not the fs.Object) to getFileHashByte, and take the Content-Type from fs.MimeTypeFromName when the backing object isn't there yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )