Implement reverse converter + differential snapshot generation + IG parity tests

CLAUDE.md

@@ -29,6 +29,8 @@ Please be concise and to the point.
 * When you create or update typescript file, run `bun tsc --noEmit` to check for errors and fix them.
 * Create tests for new functionality. Put test file as ./test/<filename>.test.ts by convention.
 * Use `import {describe, it, expect} from 'bun:test'` and `bun run test` to run tests.
+* Conversion rule: for `StructureDefinition -> FHIRSchema`, always use `differential.element` as source of truth.
+* Do not treat `snapshot.element` as primary conversion input. If needed, derive/prepare differential first.
 
 
 ## Tasks
@@ -39,4 +41,3 @@ When task finished move files to ./tasks/done/<filename>.md and write what was d
 
 
 
-

README.md

@@ -39,6 +39,31 @@ const structureDefinition = {
 const fhirSchema = translate(structureDefinition);
 ```
 
+Important:
+- `fhirschema` conversion treats `StructureDefinition.differential.element` as the source of truth.
+- Conversion input must provide differential content; snapshot is not the canonical conversion input model.
+- If only snapshot is available, expand/prepare differential first before conversion.
+
+### Converting FHIRSchema back to StructureDefinition
+
+```typescript
+import { toStructureDefinition } from '@atomic-ehr/fhirschema';
+
+const structureDefinition = toStructureDefinition(fhirSchema);
+```
+
+Corner cases and lossy roundtrip details are documented in `docs/reverse-converter-corner-cases.md`.
+
+### Generating Snapshot from Differential via FHIRSchema Merge
+
+```typescript
+import { generateSnapshot } from '@atomic-ehr/fhirschema';
+
+const withSnapshot = await generateSnapshot(profileSD, {
+  resolver: (canonicalUrl) => baseDefinitionsByUrl[canonicalUrl],
+});
+```
+
 ### Validating FHIR Resources
 
 ```typescript
@@ -112,4 +137,4 @@ This project uses:
 
 ## License
 
-[License information to be added]
+[License information to be added]

docs/reverse-converter-corner-cases.md

@@ -0,0 +1,34 @@
+# FHIRSchema -> StructureDefinition: Corner Cases
+
+This project now supports a backward converter (`toStructureDefinition`) from FHIRSchema into a FHIR `StructureDefinition` differential.
+
+The conversion is best-effort. Some SD details are irreversibly lost in `SD -> FHIRSchema`, so exact reconstruction is impossible in specific cases.
+
+## Irreversible / lossy areas
+
+1. Metadata not represented in FHIRSchema
+Only fields carried in FHIRSchema header can be restored. Fields like `id`, `title`, `date`, publisher/contact metadata, mappings, examples, aliases, requirements, comments, and other authoring-time details are not recoverable.
+
+2. Differential vs snapshot provenance
+FHIRSchema is a normalized representation and does not preserve whether a rule originally came from differential or snapshot context. The backward converter always emits a differential list.
+
+3. Element-level authoring annotations
+`ElementDefinition.id`, `alias`, `mapping`, `example`, `condition`, `definition`, and `requirements` are intentionally stripped by the forward converter, so they cannot be rebuilt.
+
+4. Slicing semantics beyond retained shape
+FHIRSchema stores normalized slice structures (`slicing`, `slices`, `match`, optional `schema`). Some source SD authoring patterns (adjacency/order nuances, exact discriminator derivation shape, or equivalent-but-different slice encodings) cannot be reconstructed exactly.
+
+5. Choice element reconstruction
+FHIRSchema encodes choices as `choices` + `choiceOf`. The backward converter emits `[x]` declarations and typed variants when needed, but the exact original representation (single multi-type element vs fully expanded typed differential entries) may differ while remaining semantically equivalent.
+
+6. Content references
+`contentReference` is converted to `elementReference` in FHIRSchema. Backward conversion restores only local references that still match the current schema URL/path convention.
+
+7. Fixed/pattern normalization
+Forward conversion normalizes both `fixed[x]` and `pattern[x]` into `pattern` (plus optional inferred `type`). This means original distinction (`fixed` vs `pattern`) may be unrecoverable.
+
+## Practical guidance
+
+1. Treat roundtrip as semantic compatibility, not byte-for-byte SD equality.
+2. For strict publishing workflows, keep original StructureDefinitions as source-of-truth artifacts.
+3. Use tests that validate key constraints/cardinality/types rather than full JSON identity where lossy areas are involved.

docs/sd-to-fs-converter-analysis.md

@@ -0,0 +1,65 @@
+# SD -> FS Converter Analysis
+
+This is a technical analysis of the existing `translate` converter in `src/converter/index.ts`.
+
+## Pipeline summary
+
+1. Header mapping (`buildResourceHeader`)
+Maps SD metadata to FS and computes `class` (`profile`, `extension`, or `kind`).
+
+2. Differential selection (`getDifferential`)
+Uses only `differential.element` and skips root path entries.
+
+3. Iterative stack algorithm
+For each differential element:
+- parse path (`parsePath`)
+- enrich against previous path (`enrichPath`)
+- compute enter/exit operations (`calculateActions`)
+- transform element fields (`transformElement`)
+- apply actions to stack (`applyActions`)
+
+4. Choice expansion
+`choice-handler.ts` expands `value[x]`-style definitions into parent `choices` plus typed children with `choiceOf`.
+
+5. Final unwind + normalization
+Remaining exits are applied, then schema is normalized (required sorting, stable nested element ordering by index).
+
+## Strengths
+
+1. Deterministic output shape
+The stack algorithm plus normalization yields stable conversion output.
+
+2. Good slicing support
+Handles sliced transitions and nested slices with discriminator-derived matching.
+
+3. Practical FHIR mappings
+Covers cardinality, type, Reference target profiles, pattern/fixed normalization, binding extraction, and constraints.
+
+4. Robust test coverage
+Unit and golden tests validate algorithm behavior and many real-world profiles.
+
+## Known limitations / design tradeoffs
+
+1. Differential-only approach
+Snapshot content is mostly ignored (except one binding lookup for choice declarations), so some information can be missed when differential is sparse.
+
+2. Information loss by design
+Many ElementDefinition authoring fields are dropped (`id`, `alias`, mapping/example blocks, etc.) for validator-oriented schema simplification.
+
+3. Fixed vs pattern collapsed
+Both are normalized into `pattern`, which helps runtime validation but loses original authoring intent.
+
+4. Extension and slicing normalization
+The generated FS is semantically focused; exact original SD authoring layout/order is not preserved.
+
+## Complexity profile
+
+- Time: approximately O(N * P), where N is number of differential elements and P is average path depth.
+- Memory: O(N) for stack and resulting schema.
+- Practical behavior: linear in profile size, with additional cost in deep slicing/choice-heavy profiles.
+
+## Suggested improvements
+
+1. Add optional strict mode preserving more ElementDefinition fields.
+2. Expand roundtrip tests (SD -> FS -> SD -> FS) for representable subsets.
+3. Add diagnostics hooks (e.g., list of dropped fields) to improve transparency for profile authors.

scripts/compare-ig-snapshots.ts

@@ -0,0 +1,237 @@
+import { mkdirSync } from 'node:fs';
+import { writeFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { gunzipSync } from 'node:zlib';
+import { generateSnapshot } from '../src/index';
+import type { StructureDefinition, StructureDefinitionElement } from '../src/converter/types';
+
+type PackageRef = { id: string; version: string };
+
+type ProfileReport = {
+  url: string;
+  version?: string;
+  name: string;
+  generated: boolean;
+  error?: string;
+  originalCount: number;
+  generatedCount: number;
+  commonCount: number;
+  missingCount: number;
+  extraCount: number;
+  keySetEqual: boolean;
+  precision: number;
+  recall: number;
+};
+
+type FinalReport = {
+  generatedAt: string;
+  target: PackageRef;
+  dependencies: PackageRef[];
+  totals: {
+    profiles: number;
+    generated: number;
+    failed: number;
+    exactKeySetMatches: number;
+    avgPrecision: number;
+    avgRecall: number;
+  };
+  failures: Array<{ url: string; error: string }>;
+  profiles: ProfileReport[];
+};
+
+function parsePackageRef(raw: string): PackageRef {
+  const parts = raw.split('@');
+  if (parts.length !== 2 || !parts[0] || !parts[1]) {
+    throw new Error(`Invalid package ref: ${raw}. Expected packageId@version`);
+  }
+  return { id: parts[0], version: parts[1] };
+}
+
+function parseArgs() {
+  const args = Bun.argv.slice(2);
+  const map = new Map<string, string>();
+  for (let i = 0; i < args.length; i += 2) {
+    map.set(args[i], args[i + 1]);
+  }
+
+  const targetRaw = map.get('--target');
+  if (!targetRaw) throw new Error('Missing --target packageId@version');
+  const depsRaw = map.get('--deps') || '';
+  const tmpDir = map.get('--tmp') || '/tmp/ig-snapshot-compare';
+
+  const target = parsePackageRef(targetRaw);
+  const dependencies = depsRaw
+    .split(',')
+    .map((x) => x.trim())
+    .filter(Boolean)
+    .map(parsePackageRef);
+
+  return { target, dependencies, tmpDir };
+}
+
+function packageUrl(pkg: PackageRef): string {
+  return `https://fs.get-ig.org/rs/${pkg.id}-${pkg.version}.ndjson.gz`;
+}
+
+function packageFile(tmpDir: string, pkg: PackageRef): string {
+  return join(tmpDir, `${pkg.id}-${pkg.version}.ndjson.gz`);
+}
+
+async function downloadPackage(tmpDir: string, pkg: PackageRef): Promise<string> {
+  const url = packageUrl(pkg);
+  const filePath = packageFile(tmpDir, pkg);
+  const file = Bun.file(filePath);
+  if (await file.exists()) {
+    return filePath;
+  }
+
+  const res = await fetch(url);
+  if (!res.ok) {
+    throw new Error(`Failed to download ${url}: ${res.status} ${res.statusText}`);
+  }
+
+  await Bun.write(filePath, new Uint8Array(await res.arrayBuffer()));
+  return filePath;
+}
+
+async function loadNdjsonGz(path: string): Promise<unknown[]> {
+  const bytes = await Bun.file(path).bytes();
+  const text = gunzipSync(bytes).toString('utf8');
+  return text
+    .split('\n')
+    .map((line) => line.trim())
+    .filter((line) => line.length > 0)
+    .map((line) => JSON.parse(line));
+}
+
+function isStructureDefinition(resource: unknown): resource is StructureDefinition {
+  if (!resource || typeof resource !== 'object') return false;
+  const r = resource as Record<string, unknown>;
+  return r.resourceType === 'StructureDefinition' && typeof r.url === 'string';
+}
+
+function keyForElement(el: StructureDefinitionElement): string {
+  return `${el.path}|${el.sliceName ?? ''}`;
+}
+
+function byKey(elements: StructureDefinitionElement[]): Set<string> {
+  return new Set(elements.map((e) => keyForElement(e)));
+}
+
+function round(value: number): number {
+  return Math.round(value * 10000) / 10000;
+}
+
+async function run() {
+  const { target, dependencies, tmpDir } = parseArgs();
+  mkdirSync(tmpDir, { recursive: true });
+
+  const allPackages = [target, ...dependencies];
+  const paths = await Promise.all(allPackages.map((p) => downloadPackage(tmpDir, p)));
+  const resourcesByPackage = await Promise.all(paths.map((p) => loadNdjsonGz(p)));
+
+  const allSds = resourcesByPackage.flatMap((resources) => resources.filter(isStructureDefinition));
+  const targetSds = resourcesByPackage[0].filter(isStructureDefinition);
+
+  const resolverMap = new Map<string, StructureDefinition>();
+  for (const sd of allSds) {
+    resolverMap.set(sd.url, sd);
+    if (sd.version) resolverMap.set(`${sd.url}|${sd.version}`, sd);
+  }
+
+  const reports: ProfileReport[] = [];
+
+  for (const sd of targetSds) {
+    if (!sd.snapshot?.element || sd.snapshot.element.length === 0) {
+      continue;
+    }
+
+    try {
+      const generated = await generateSnapshot(sd, {
+        resolver: {
+          async resolve(canonical, options) {
+            const direct = resolverMap.get(canonical);
+            if (direct) return direct;
+            if (options?.version) return resolverMap.get(`${canonical}|${options.version}`);
+            return undefined;
+          },
+        },
+      });
+
+      const originalSet = byKey(sd.snapshot.element);
+      const generatedSet = byKey(generated.snapshot?.element || []);
+
+      const common = [...originalSet].filter((k) => generatedSet.has(k));
+      const missing = [...originalSet].filter((k) => !generatedSet.has(k));
+      const extra = [...generatedSet].filter((k) => !originalSet.has(k));
+
+      const precision = generatedSet.size === 0 ? 0 : common.length / generatedSet.size;
+      const recall = originalSet.size === 0 ? 0 : common.length / originalSet.size;
+
+      reports.push({
+        url: sd.url,
+        version: sd.version,
+        name: sd.name || sd.id || sd.url,
+        generated: true,
+        originalCount: originalSet.size,
+        generatedCount: generatedSet.size,
+        commonCount: common.length,
+        missingCount: missing.length,
+        extraCount: extra.length,
+        keySetEqual: missing.length === 0 && extra.length === 0,
+        precision: round(precision),
+        recall: round(recall),
+      });
+    } catch (error) {
+      reports.push({
+        url: sd.url,
+        version: sd.version,
+        name: sd.name || sd.id || sd.url,
+        generated: false,
+        error: error instanceof Error ? error.message : String(error),
+        originalCount: sd.snapshot.element.length,
+        generatedCount: 0,
+        commonCount: 0,
+        missingCount: sd.snapshot.element.length,
+        extraCount: 0,
+        keySetEqual: false,
+        precision: 0,
+        recall: 0,
+      });
+    }
+  }
+
+  const generated = reports.filter((r) => r.generated);
+  const failed = reports.filter((r) => !r.generated);
+  const avgPrecision = generated.length
+    ? generated.reduce((sum, r) => sum + r.precision, 0) / generated.length
+    : 0;
+  const avgRecall = generated.length
+    ? generated.reduce((sum, r) => sum + r.recall, 0) / generated.length
+    : 0;
+
+  const finalReport: FinalReport = {
+    generatedAt: new Date().toISOString(),
+    target,
+    dependencies,
+    totals: {
+      profiles: reports.length,
+      generated: generated.length,
+      failed: failed.length,
+      exactKeySetMatches: generated.filter((r) => r.keySetEqual).length,
+      avgPrecision: round(avgPrecision),
+      avgRecall: round(avgRecall),
+    },
+    failures: failed.map((f) => ({ url: f.url, error: f.error || 'unknown error' })),
+    profiles: reports,
+  };
+
+  const prefix = `${target.id}-${target.version}`;
+  const jsonPath = join(tmpDir, `${prefix}-snapshot-compare.json`);
+  await writeFile(jsonPath, JSON.stringify(finalReport, null, 2), 'utf8');
+
+  console.log(`Report: ${jsonPath}`);
+  console.log(JSON.stringify(finalReport.totals, null, 2));
+}
+
+await run();