Skip to content

persist OneDrive thumbnails to ImageKit CDN to fix expiring URLs#165

Open
RsbhThakur wants to merge 1 commit intodevfrom
RsbhThakur
Open

persist OneDrive thumbnails to ImageKit CDN to fix expiring URLs#165
RsbhThakur wants to merge 1 commit intodevfrom
RsbhThakur

Conversation

@RsbhThakur
Copy link
Copy Markdown

Changelog

All changes in this PR are documented here.


Summary

Two independent concerns are addressed in this PR:

  1. Permanent thumbnail storage — OneDrive / Microsoft Graph thumbnail URLs expire after ~30 minutes, causing every page load to re-hit the Graph API. Thumbnails are now uploaded to ImageKit CDN and stored permanently in MongoDB at two points:

    • At upload time (new files) — the ImageKit upload runs in the background after the file is saved, so the client response is not blocked. The DB is updated with the permanent URL once the background task completes.
    • On first broken link (legacy files) — the existing <img onError> refresh path uploads to ImageKit and permanently updates MongoDB, so the file is never fetched from Graph API again.
      Deleting a file also deletes its ImageKit thumbnail.
  2. Node.js v25 compatibilityjsonwebtoken v8 used the long-removed SlowBuffer API and crashed on startup under Node 25. Upgraded to v9, which resolves this without any breaking API changes. The underlying buffer-equal-constant-time package is also patched to guard against SlowBuffer being undefined.


New dependency — server/

Package Version Purpose
@imagekit/nodejs latest ImageKit SDK — permanent CDN thumbnail storage
jsonwebtoken v9 (upgraded from v8) Fix SlowBuffer crash on Node 25

New files

server/services/imagekit.js

Lazy-initialised ImageKit singleton (deferred until first use so env vars are loaded).

  • uploadThumbnail(oneDriveFileId, imageBuffer) — wraps the buffer with toFile(), uploads to ImageKit under /thumbnails/{fileId}.webp with useUniqueFileName: false (overwrites on re-upload), applies WebP conversion at the CDN edge. Returns { url, fileId } where fileId is ImageKit's internal ID needed for deletion.
  • deleteThumbnail(imagekitFileId) — deletes the thumbnail from ImageKit by its internal file ID.
  • isImageKitUrl(url) — returns true when a URL already points at the configured ImageKit endpoint; used to skip re-uploading on already-migrated files.

server/scripts/thumbnail_migration.mongodb

mongosh queries for one-time migration of legacy files:

  1. Unique index on files.fileId — speeds up FileModel.findOne({ fileId }) lookups.
  2. Sparse index on files.thumbnail — enables fast scans for unmigrated files.
  3. Aggregate status report — breaks down files into imagekit_permanent / onedrive_expiring / no_thumbnail.
  4. Bulk updateMany — clears legacy OneDrive URLs so each file self-migrates lazily on next client request.
    5–7. Verification and progress-monitoring queries.

Modified files

server/modules/course/course.model.js

Added imagekitFileId: { type: String } to FileSchema.
Stores ImageKit's internal file ID so thumbnails can be deleted from ImageKit when a file is removed.

server/services/UploadFile.js

UploadFile() — After a successful OneDrive upload, createLink and the thumbnail fetch from Graph API now run in parallel via Promise.all (previously sequential — saves one full round-trip). The file is saved to MongoDB immediately with the temporary OneDrive URL, then a fire-and-forget background task:

  1. Downloads the raw image bytes from the temporary thumbnail URL.
  2. Uploads to ImageKit (uploadThumbnail).
  3. Updates MongoDB with the permanent URL and imagekitFileId.

The client response is not blocked by the ImageKit upload at any point.

DeleteFile() — Looks up imagekitFileId from MongoDB and calls deleteThumbnail() in a fire-and-forget task before proceeding with the OneDrive deletion. The client response is not blocked.

server/modules/onedrive/onedrive.controller.js

thumbnail() controller — rewritten. Handles legacy files whose stored OneDrive URLs have expired. New behaviour implements a 2-tier lookup:

Tier 1 — MongoDB
  ImageKit URL already stored (new upload or previously migrated)
  → return URL immediately

Tier 2 — Microsoft Graph API (legacy files, first broken-link trigger only)
  → fetch temporary OneDrive thumbnail URL
  → download raw image bytes
  → upload to ImageKit (permanent CDN URL)
  → write permanent URL + imagekitFileId back to MongoDB
  → return URL

Legacy files hit Tier 2 exactly once, then permanently move to Tier 1.

server/package.json

  • @imagekit/nodejs added.
  • jsonwebtoken upgraded ^8.5.1^9.0.3.

server/.env

Three new environment variables required:

IMAGEKIT_PUBLIC_KEY=<your_imagekit_public_key>
IMAGEKIT_PRIVATE_KEY=<your_imagekit_private_key>
IMAGEKIT_URL_ENDPOINT=<your_imagekit_url_endpoint>

Get these from the ImageKit dashboard under Developer Options.

server/jsconfig.json (new)

Added to prevent VS Code from scanning into node_modules and surfacing TypeScript warnings from third-party SDK source files.


Migration path for existing deployments

  1. Add the three IMAGEKIT_* variables to the server environment.
  2. Restart the server — new uploads will store to ImageKit automatically.
  3. To migrate legacy files, run the bulk-clear query (UI components (Landing, Dashboard) #4) from server/scripts/thumbnail_migration.mongodb in mongosh. This strips old OneDrive URLs so files self-migrate on next client browse.

No data loss occurs. Files without a thumbnail show no preview until a client triggers the refresh endpoint, at which point they are permanently stored.


What does NOT change

  • The thumbnail endpoint path and response shape are unchanged.
  • No client-side code is modified.
  • The thumbnail field on FileModel continues to hold a string URL — now a permanent ImageKit URL instead of an expiring OneDrive one.
  • All other server routes and modules are untouched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant