New: Run adapt-migrations on framework update

[taylortom]

New

Run adapt-migrations directly via its JS API (no grunt) for framework update, course import, and plugin update.

Replaces the grunt-based approach attempted in adapt-security/adapt-authoring-contentplugin#29

Changes

• Add adapt-migrations as a direct npm dependency of adaptframework
• Create shared migration utilities that call the adapt-migrations JS API
• After framework update, migrate all existing courses in the DB
• Refactor course import to use the same mechanism instead of spawning grunt
• After plugin update, migrate courses using that plugin

Trigger Points

Trigger	Scope	Module
Framework update (--update-framework / POST /adapt/update)	All courses	adaptframework
Course import with migrateContent: true	Single imported course	adaptframework
Plugin update (updatePlugin())	Courses using that plugin	contentplugin

Shared Utilities (in adaptframework/lib/utils/)

readFrameworkPluginVersions.js

• async readFrameworkPluginVersions(frameworkDir) -> [{name, version}]
• Globs src/core/bower.json + src/{components,extensions,menu,theme}/*/bower.json
• Reads each with readJson, returns [{name, version}]

collectMigrationScripts.js

• async collectMigrationScripts(frameworkDir) -> [string]
• Globs src/core/migrations/**/*.js + src/*/*/migrations/**/*.js
• Returns absolute paths

runContentMigration.js

• async runContentMigration({ content, fromPlugins, toPlugins, scripts, cachePath }) -> content[]
• Core function shared by all three trigger points
• Calls load() with migration scripts
• Creates Journal with { content, fromPlugins, originalFromPlugins, toPlugins }
• Calls migrate({ journal, logger })
• Returns journal.data.content (mutated in-place by Proxy)

Flow 1: Framework Update

lib/utils/migrateExistingCourses.js

• async migrateExistingCourses({ fromPlugins, toPlugins, frameworkDir }) -> { migrated, failed, errors[] }
• Collects migration scripts via collectMigrationScripts()
• Queries all courses via content.find({ _type: 'course' })
• For each course (sequentially):
  • Fetches content: course + config + content.find({ _courseId })
  • Snapshots originals via JSON.parse(JSON.stringify(...))
  • Calls runContentMigration()
  • Compares each item with util.isDeepStrictEqual, updates changed items via content.update()
  • Catches per-course errors, logs, continues

Changes to AdaptFrameworkModule.js — updateFramework()

async updateFramework (version) {
  const fromPlugins = await readFrameworkPluginVersions(this.path)  // BEFORE update
  // ... existing CLI update ...
  const toPlugins = await readFrameworkPluginVersions(this.path)    // AFTER update
  await migrateExistingCourses({ fromPlugins, toPlugins, frameworkDir: this.path })
  await this.postUpdateHook.invoke()  // fix: add missing await
}

Changes to handlers.js — postUpdateHandler()

• Include migration results in the JSON response

Flow 2: Course Import (replaces grunt)

Changes to AdaptFrameworkImport.js

Replace migrateCourseData() (currently spawns grunt capture+migrate):

1. Keep patchThemeName() and patchCustomStyle() (patch JSON files on disk)
2. Call loadCourseData() to read import's JSON into memory
3. Flatten this.contentJson into array format for adapt-migrations
4. Build fromPlugins from this.usedContentPlugins (import's bower.json versions)
5. Build toPlugins from readFrameworkPluginVersions(this.framework.path)
6. Call runContentMigration({ content, fromPlugins, toPlugins, scripts })
7. Write migrated content back into this.contentJson structure

Update task pipeline:

[this.loadCourseData, importContent],                   // always load first
[this.migrateCourseData, !isDryRun && migrateContent],  // in-memory migration
[this.importCourseData, !isDryRun && importContent],    // no reload needed

Remove runGruntMigration() method.

Flow 3: Plugin Update

New method on AdaptFrameworkModule — migrateCourses()

Public method that contentplugin can call via this.framework:

async migrateCourses ({ fromPlugins, toPlugins, courseIds }) -> { migrated, failed, errors[] }

• Collects migration scripts from this.path
• For each courseId: fetch content, run migration, diff, update DB
• Reuses runContentMigration internally
• migrateExistingCourses becomes a thin wrapper that queries all courses then calls this

Changes to contentplugin/lib/ContentPluginModule.js — updatePlugin()

async updatePlugin (_id) {
  const [{ name }] = await this.find({ _id })
  const fromPlugins = await readFrameworkPluginVersions(this.framework.path)  // BEFORE
  const [pluginData] = await this.framework.runCliCommand('updatePlugins', { plugins: [name] })
  const p = await this.update({ name }, pluginData._sourceInfo)
  await this.processPluginSchemas(pluginData)
  const toPlugins = await readFrameworkPluginVersions(this.framework.path)    // AFTER
  const courses = await this.getPluginUses(_id)
  if (courses.length) {
    await this.framework.migrateCourses({
      fromPlugins,
      toPlugins,
      courseIds: courses.map(c => c._id)
    })
  }
  this.log('info', `successfully updated plugin ${p.name}@${p.version}`)
  return p
}

Migration Applicability & Tracking

No per-course migration tracking is needed. Each migration script gates itself via whereFromPlugin version checks (e.g. { name: 'adapt-contrib-core', version: '<6.24.2' }). Only migrations matching the fromPlugins→toPlugins range will run; others are skipped. After each migration, updatePlugin bumps the version in fromPlugins so subsequent migrations in the same run chain correctly.

All courses are assumed to be at the same framework/plugin version because:

• New courses are created with current schemas
• Imported courses are migrated at import time
• Framework/plugin updates migrate all relevant courses

This means on each update, every relevant course is processed, but only migrations for the specific version delta actually execute. The version gating is load-bearing — migrations are not all independently idempotent, so correct fromPlugins is essential. Per-course version tracking could be added later as a performance optimisation for large-scale deployments.

Files Summary

New files (adaptframework/lib/utils/)

• readFrameworkPluginVersions.js
• collectMigrationScripts.js
• runContentMigration.js
• migrateExistingCourses.js

Modified files

File	Changes
adaptframework/lib/AdaptFrameworkModule.js	Add migrateCourses() method; update updateFramework()
adaptframework/lib/AdaptFrameworkImport.js	Replace grunt with in-memory migration; update task pipeline
adaptframework/lib/handlers.js	Include migration results in update response
adaptframework/lib/utils.js	Add barrel exports
adaptframework/errors/errors.json	Add FW_UPDATE_MIGRATION_FAILED
adaptframework/package.json	Add adapt-migrations dependency
contentplugin/lib/ContentPluginModule.js	Call migrateCourses() from updatePlugin()

Tests

File	Tests
tests/utils-readFrameworkPluginVersions.spec.js	Parses bower.json, handles missing dirs
tests/utils-collectMigrationScripts.spec.js	Finds core + plugin migration scripts
tests/utils-runContentMigration.spec.js	Loads scripts, creates journal, runs migrate
tests/utils-migrateExistingCourses.spec.js	Mocks content module, verifies DB updates, error isolation

Key Design Decisions

1. adapt-migrations as direct dependency of adaptframework
2. Shared runContentMigration — single core function used by all three triggers
3. Public migrateCourses() on AdaptFrameworkModule — allows contentplugin to trigger targeted migrations via this.framework
4. Plugin versions from bower.json — read before/after any update to determine fromPlugins/toPlugins
5. Import: in-memory migration — load content first, migrate in memory, skip disk round-trip
6. Sequential per-course — avoids memory pressure
7. Error isolation — one course failing doesn't block others
8. Minimal DB writes — deep-compare original vs migrated, only update changed items
9. No per-course tracking — version gating in migration scripts is sufficient; tracking can be added later for performance

Relevant Files (for reference)

• grunt/tasks/migration.js — Current grunt task wrapper (being replaced)
• src/core/migrations/ — Core migration scripts
• adapt-migrations — JS API library (load, migrate, Journal, Logger)

Verification

1. Unit tests: node --test --experimental-test-module-mocks 'tests/utils-*.spec.js'
2. Lint: npx standard
3. Integration (update): trigger framework update with courses in DB
4. Integration (import): import course from older framework version
5. Integration (plugin): update a plugin with courses using it; verify content migrated