Local filesystem writes can retain stale tail data when overwriting shorter objects

[LuisLin17]

Problem

The Local FS storage backend opens existing objects with os.O_RDWR|os.O_CREATE but without os.O_TRUNC:

handle, err := os.OpenFile(o.path, os.O_RDWR|os.O_CREATE, 0o644)

fsObject.Put writes from offset zero, so overwriting an existing object with shorter content leaves the old tail in the file. The same behavior affects the uncompressed StoreFile path that uses getHandle(false).

Example:

existing object: abcdefgh
new object:      xy
read result:     xycdefgh

This makes the Local FS backend inconsistent with object-storage backends, where a put replaces the complete object. It can corrupt local snapshots, cache files, or metadata when an object is rewritten with a smaller payload.

Proposed fix

Use truncation for complete-object writes, either by adding os.O_TRUNC to the write handle or by explicitly truncating and resetting the offset before copying:

handle, err := os.OpenFile(o.path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0o644)
if err != nil {
    return nil, err
}

The truncating mode should only be used for complete replacement writes. Read, range-read, append, and multipart/random-offset paths should retain their existing semantics.

Tests

Please add two separate regression tests:

1. Put path:

   • Write a longer payload to an object.
   • Overwrite it with a shorter payload using Put.
   • Verify Size equals the new payload length.
   • Verify reading returns exactly the new payload, with no stale suffix.

2. Direct, uncompressed StoreFile path:

   • Store a longer source file into an object.
   • Overwrite it with a shorter source file using uncompressed StoreFile.
   • Verify Size equals the new source file length.
   • Verify reading returns exactly the new source content, with no stale suffix.

The compressed StoreFile path is a separate implementation and should not be conflated with this regression.

Relevant code:

• packages/shared/pkg/storage/storage_fs.go
• fsObject.Put
• fsObject.StoreFile
• fsObject.getHandle

[tttboy123]

I verified this is still present on current main at a00be47: both Put and the uncompressed StoreFile path reuse getHandle(false), which opens an existing object without O_TRUNC. That leaves the previous suffix behind when a shorter payload replaces a longer object.

I would like to take this issue. I will work test-first at the public storage interface:

• add a Put regression that writes a longer payload, overwrites it with a shorter one, then verifies both Size and the exact bytes read back;
• add the equivalent regression for the uncompressed StoreFile path;
• make complete-replacement writes truncate the destination while leaving range reads, compressed storage, and any non-replacement semantics unchanged;
• run the focused shared/storage tests, race-enabled package tests, and repository lint relevant to the change.

Could a maintainer assign #3642 to @tttboy123 if this scope looks right?

[tttboy123]

Implemented the agreed scope in #3649.

The fix adds O_TRUNC only when getHandle(false) is used for complete-object replacement writes. Both current callers—Put and the uncompressed StoreFile path—now replace the destination fully; read/range-read and compressed storage paths are unchanged.

I added separate regressions for both public paths. Each writes a longer value, replaces it with a shorter one, and checks both the reported size and exact bytes. Before the fix both tests failed with size 24 instead of 5; after the fix they pass.

Validation completed:

• focused Local FS suite passed;
• race-enabled Local FS suite passed;
• all non-Docker storage tests passed;
• package compile check passed;
• shared-module lint passed with 0 issues.

The only unavailable local coverage is the existing testcontainers integration group because rootless Docker is not installed; I recorded that limitation in the PR. Please review and assign #3642 to @tttboy123 when convenient.