Release v1.8.3: upload tokens, duplicate rejection, and an nginx auth fix - #748
Merged
Merged
Conversation
Every upload is hashed once it lands on disk. When that hash matches a video the library already holds, the new copy is removed and the request answers 409 with the existing video's id, title, and watch URL instead of creating a second entry for the same content. A row whose own file is missing from disk does not count as a duplicate: that upload is a restore, and scan-video flips the row back to available. Nor does the row whose path is the file just written, which is the same restore landing exactly where the missing original used to live. Applies to all four video upload routes — public and authenticated, single-shot and chunked. The upload card tracks a 'duplicate' status alongside done and error, shows it in amber as "Already exists", and the batch summary names how many were already in the library with links to open them. That alert stays up for ten seconds rather than the usual three and a half, since a message with links in it needs long enough to click; SnackbarAlert now takes autoHideDuration to allow it.
Adds a token-authenticated upload route so an external tool can push videos and images into Fireshare without a password or a browser session, and gives every account that may upload a place to mint, rename, regenerate and delete its own tokens. POST /api/upload/token takes one video or image per request, deciding which by extension, and accepts the same title, folder, game and tag options the browser routes take. It reuses the helpers behind /api/upload and /api/upload/image rather than reimplementing them, so filename sanitising, the extension allowlist, folder containment, the demo size cap, duplicate rejection and uploader attribution are the same code paths the UI already exercises. A `game` name is accepted alongside `game_id`, resolved against existing games only, since a machine rarely knows a row id. GET on the same route validates a token and reports the owner and accepted types, so a tool can check its configuration without uploading anything. A token carries no permissions of its own. Every request re-reads the owner's account, so revoking upload, disabling the account, or deleting it stops the token on the next call. Only the sha256 of the secret is stored — the raw value exists in exactly one response, the one that mints it — and the credential is read from a header only, never a query string, which would land in proxy access logs and shell history. An unknown token, a revoked permission and a disabled account are all reported identically. Repeated bad tokens from one address are throttled, deliberately in a separate bucket from login_throttle so a tool looping on a stale token cannot lock people out of signing in. Tokens never expire; they end when deleted or regenerated. Deleting a user deletes their tokens with them. The management UI lives in Settings > Security, shown only to accounts holding the upload permission, and reveals each new secret once with a copy button and a ready-to-paste curl example.
Adds the two things an unattended uploader needs beyond a single POST: a way to send a large file in pieces, and a way to find out what folders and games it may name. POST /api/upload/token/chunked takes one chunk per request, in any order, grouped by a caller-chosen checkSum. Every request but the last answers 202; whichever completes the set reassembles the file, verifies it against the declared fileSize, and answers exactly as the single-shot route does — including the 409 when the finished video is already in the library. Both video and image uploads can be chunked, since the finished file's extension decides which it is. Both upload routes now funnel through _prepare_upload / _finish_upload, so they cannot drift apart in what they accept, where they file it, or how they attribute it. Unlike the browser-facing chunked routes, malformed chunkPart, totalChunks or fileSize values answer 400 rather than raising through to a 500: a token route is driven by somebody else's code, so bad input is expected. totalChunks is capped, because every chunk is a request and a file on disk. An abandoned chunked image upload leaves its parts under the image root rather than the video root, so the startup sweep that removes orphaned .partNNNN files now covers both. GET /api/upload/token/options returns the video and image folders, the default folder, and every game in the library. Games are listed in full rather than through /api/games, which hides games with nothing linked to them yet: those are exactly the ones an upload might be the first to use, and `game` name resolution already accepts them.
Reject uploads of videos already in the library
Upload tokens for scripts and other tools
Upload tokens for scripts and other tools
Three failures a long-running uploader hits and one thing it could not ask.
Reassembly wrote straight to the final filename, so a process killed partway
through the copy left a truncated file under a real media extension. Nothing
sweeps that -- the glob only matched .partNNNN -- while the scheduled
bulk-import picks it up within minutes and ingests it as a genuine video. It
now assembles under `{checkSum}.assembling` and renames into place once the
size check passes, which is atomic within a filesystem; the staging file sits
in the destination directory precisely so that holds.
The 202 said only "not done yet", which is indistinguishable from "I have
none of your parts". A client whose set had been swept would send its
remaining chunks, get 202 for each, run out, and hang forever with no error.
It now reports how many parts are actually held, so a caller that has sent
more than that knows to start the file over.
The startup sweep removed every part file unconditionally, which meant any
restart during an upload destroyed it. Parts survive a restart perfectly well
on their own, so the sweep is now keyed on mtime with a 24h cut -- still
collecting sets abandoned by a tool that never came back, since each part
carries the mtime of the moment it was written and an upload in progress is
nowhere near the threshold. `.assembling` files are swept on the same terms.
Adds GET /api/upload/token/exists?video_id=<hex>. Duplicate rejection only
fires once the file is on disk, so a chunked upload of something already in
the library crosses the network in full before the 409. A tool that hashes
its own file can now ask first. Matches _reject_duplicate's semantics: a row
whose file is missing from disk does not count, because that upload is a
restore.
Also corrects the docs, which told tool authors to send `folder` "with
whichever chunk you like". That is the one field where it breaks --
_prepare_upload derives the parts directory from it, so a chunk that omits it
lands in the default folder and the completing request never finds a full
set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lity Make chunked token uploads survive a restart
Fireshare already maps folders to games: anything scanned in a folder carries that folder's game. The mapping was readable only with a session, so a tool holding an upload token could not see it — and a tool deciding where to *put* a file needs exactly that mapping, read the other way round. Filing a clip in the folder its game already owns is how it ends up tagged without the upload having to name a game at all. Added to /api/upload/token/options rather than given a route of its own: that endpoint already answers "what may I choose", folder rules are the same kind of answer, and a caller that wants them wants the folder list in the same breath. Image folder rules are listed separately, matching how the folders themselves are. Rules whose game has gone are left out, since there is nothing a caller could do with one.
Tell tools which folder each game's media belongs in
The Videos page had no route of its own: it lived at "/", which is also the
landing route that forwards a visitor to whichever page sits at the top of the
sidebar. Home ships at the top of that order, so the sidebar's Videos link
pointed at "/" and was immediately bounced to /home. There was no way to reach
Videos from the sidebar at all.
Videos now lives at /videos and "/" is purely the landing redirect, so a link to
a page never passes back through the question of which page comes first.
LandingRoute loses its children along with the special case that rendered them.
The fallback landingHref returns when every page has been hidden moves from "/"
to /videos for the same reason: pointing the landing redirect at itself would
now loop.
"/" keeps forwarding with the query string attached, so existing links and
bookmarks still land somewhere sensible. The two navigate('/') calls left — the
logo and the redirect after login — mean "go to the landing page" and are
unchanged.
fix: give Videos a path of its own so its sidebar link stops redirecting
/api/upload/token/exists took only video_id. Images needed it more: a video at least gets a 409 once its bytes are on disk, but an image is never rejected at all — the upload is accepted and scan_image quietly folds it into the existing row, so a client pays for the whole transfer and is told it succeeded. Pass exactly one of video_id or image_id; both or neither is a 400. An image_id on an instance with images turned off is a 503. The video response shape is unchanged. The docs claimed images were not deduplicated. They are — scan_image has always matched on image_id — it just never said so to the caller.
Let a tool ask about an image before uploading it, too
A token on its own does nothing: someone who wants their clips uploaded automatically still has to go and write the thing that does the uploading, and writing it yourself reads as the only option when nothing says otherwise. Firesync is that thing, already built, and this is the page a person is looking at when the question occurs to them. The card sits between the explanation and the token list rather than below it, so it is also there for the account that has no tokens yet — the one most likely to be deciding how to approach this at all. It links to the releases page, since what they want from here is the download. Trimmed the pane's opening copy while nearby. The permission semantics it spelled out are in docs/UploadTokens.md, which the paragraph already links to, and saying it twice cost more attention than it earned.
feat: point people at Firesync from the Upload Tokens pane
The /_content/derived/ location ran auth_request without capturing an id from its regex, so $video_id was unset when the auth subrequest built its X-Fireshare-Video-Id header. Every derived asset request logged "using uninitialized video_id" and the gate fell back to re-parsing X-Original-URI. Capture the first path segment like the video locations already do, so the gate checks the same id nginx serves the file from. Also initialise the variable in the dev config's prefix location, which had the same warning.
ShaneIsrael
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything on
developsince v1.8.2 (PRs #740–#747).Upload tokens for scripts and other tools
upload_tokentable (migrations4n5o6p7q8r9)docs/UploadTokens.mdDuplicate uploads
Fixes
using uninitialized "video_id"warnings on every thumbnail request