Data Parser Engine (DPE)

A headless, schema-driven JavaScript library that takes a raw file payload and a declarative schema, and returns mapped JSON records. Handles legacy print-formatted text, modern spreadsheets, and structured XML/JSON — all through a single entry point.

Heavy lifting is delegated to battle-tested upstream libraries (PapaParse, SheetJS, Mammoth, fast-xml-parser, JSZip). DPE is the thin coordination layer that picks the right parser, strips cruft, and shapes the output.

Try it live: Schema Studio — interactive schema-tuning tool with examples for every format.

Supported formats

Format	Layout choice	Notes
`csv`	implicit delimited	RFC 4180-ish; PapaParse
`prn`	required delimited / fixed	PRN files — operator picks layout in setup
`txt`	required delimited / fixed	TXT files — operator picks layout in setup
`fixed`	implicit fixed-width	For text files with arbitrary extensions
`xml`	n/a	fast-xml-parser, with dot-path extraction
`json`	n/a	Native JSON, with dot-path extraction
`passthrough`	n/a	Line-by-line `[{line, text}]` for text files DPE doesn't structurally parse (e.g. raw EDIFACT/X12)
`xls`	n/a	Legacy Excel via SheetJS
`xlsx`	n/a	Modern Excel via SheetJS
`ods`	n/a	OpenDocument Spreadsheet via SheetJS
`docx`	n/a	Word document text extraction via Mammoth
`odt`	n/a	OpenDocument Text — paragraphs and tables

The format key declares the file type. File extension is irrelevant — DPE parses what the operator tells it to.

Obtaining DPE

DPE is open source (MIT) on GitHub. Clone the repo (or download a source zip from the GitHub "Code" button / a release), then vendor the built artifact you need from dist/:

git clone https://github.com/grimblefritz/data-parser-engine.git

Browser, plain <script> integration: copy dist/dpe.iife.js (and optionally dist/version.js if you want the engine version available as window.DPE_ENGINE_VERSION for diagnostics).
Bundler / ESM integration: copy dist/dpe.esm.js into your project tree and import it by relative path. DPE's dependencies (PapaParse, SheetJS, Mammoth, fast-xml-parser, JSZip) are declared as imports and your bundler is expected to resolve them — install each in your own package.json.

DPE is not published to npm — vendor the built file from dist/. Questions? dpe@smisco.biz.

Quick start

Browser (drop-in)

Load the five upstream libraries first, then the IIFE build of DPE. The class is exposed as window.DataParser.

Required window globals for the IIFE build (the engine reads these by name; if they aren't present at load time, parsing the relevant format will fail):

Library	Global the IIFE reads
PapaParse	`Papa`
SheetJS (xlsx)	`XLSX`
Mammoth	`mammoth`
fast-xml-parser	`XMLParser`
JSZip	`JSZip`

Most CDN-packaged builds expose these names. Notable nuance: fast-xml-parser's cdnjs build (fxparser.min.js) attaches XMLParser directly to window; some other CDN packagings of the same library use a different global. The CDN URLs below are known-good as of writing.

<script src="https://cdnjs.cloudflare.com/ajax/libs/PapaParse/5.4.1/papaparse.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/xlsx/0.18.5/xlsx.full.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mammoth/1.6.0/mammoth.browser.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/fast-xml-parser/4.3.6/fxparser.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jszip/3.10.1/jszip.min.js"></script>
<script src="dist/dpe.iife.js"></script>

<script>
async function go(file) {
    const result = await DataParser.parse(file, {
        format: 'csv',
        delimiter: '|',
        mapping: { id: 'sku', buy: 'buy_price' }
    });
    console.log(result.data);     // mapped rows
    console.log(result.errors);   // any parse warnings/errors
    console.log(result.meta);     // diagnostics
}
</script>

The exact upstream-library version pinning above is illustrative, not required — any reasonably current version that exposes the documented globals is fine.

Wiring it to a file input (the most common entry point — File extends Blob, so it's accepted directly):

<input type="file" id="dataFile">

<script>
document.getElementById('dataFile').addEventListener('change', async (e) => {
    const file = e.target.files[0];
    if (!file) return;
    const result = await DataParser.parse(file, { format: 'csv' });
    console.log(result.data);
});
</script>

ESM (with bundler)

After vendoring dist/dpe.esm.js into your project (see Obtaining DPE above):

import DataParser from './vendor/dpe.esm.js';   // adjust path to where you placed it

const result = await DataParser.parse(blob, { format: 'xlsx' });

Your bundler needs to resolve DPE's runtime dependencies, so install them as direct dependencies of your own project:

npm install papaparse mammoth fast-xml-parser jszip https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz

SheetJS note: The xlsx package on npm is stale (0.18.5). SheetJS distributes current releases from their own CDN as tarballs. The URL above installs 0.20.3 — the same version DPE is tested against. Check docs.sheetjs.com for newer releases.

The schema

A single object describes what the file is and how to extract from it.

Global keys

Key	Type	Default	Notes
`format`	string	—	Required. From the format table above.
`layout`	string	—	Required for `prn` and `txt`: `'delimited'` or `'fixed'`. Not accepted on any other format.
`encoding`	string	`'utf-8'`	Text formats only. e.g., `'windows-1252'`, `'iso-8859-1'`, `'cp437'`.
`dropRegex`	string or string[]	—	Pre-filter. Strips matching lines before parsing. Text formats only.
`mapping`	object	—	Output shaping. See Mapping below.
`mappingMode`	string	`'replace'`	Only meaningful when `mapping` is supplied. `'replace'` (default): output contains only mapped target keys. `'extend'`: output contains all source fields with mapped target keys overlaid.
`strict`	boolean	`false`	If true, any error rejects the promise.
`studioVersion`	string	—	Optional provenance stamp. The Schema Studio stamps every schema it produces with its own build identifier (e.g. `'b42.260524.113043'` — build number, `YYMMDD`, `HHMMSS`). The engine accepts and ignores it; useful for tracing which Studio build authored a saved schema.

Format-specific keys

Delimited parser (csv, prn+layout:'delimited', txt+layout:'delimited'):

Key	Default	Notes
`delimiter`	`','`	Or `'auto'` for PapaParse autodetect. Common: `'\t'`, `'
`quoteChar`	`'"'`
`escapeChar`	`'\\'`
`hasHeaders`	`true`	When `false`, PapaParse returns each row as an array (not an object with auto-named keys). Pair with positional integer mapping (see Mapping below) to extract by 1-indexed slot.

Fixed-width parser (fixed, prn+layout:'fixed', txt+layout:'fixed'):

Key	Notes
`fieldDefinitions`	Required. Array of `{ name, start, end, trim }`. `start` inclusive, `end` exclusive, 0-based. `trim` default `true`.

XML / JSON parsers (xml, json):

Key	Notes
`rootPath`	Dot notation. Path to the array (or single record) to extract from the parsed tree.

Spreadsheet parser (xls, xlsx, ods):

Key	Notes
`sheetName`	Defaults to first sheet in the workbook.

Document parser (docx, odt): no format-specific keys.

Passthrough parser (passthrough): no format-specific keys. Output is [{line, text}], one record per line, 1-indexed. dropRegex still applies. Useful for inspecting (and mapping by position from) files DPE doesn't yet structurally parse — for example raw EDIFACT or X12 transaction dumps.

Return shape

{
    data:         [ /* post-mapping records */ ],
    raw:          [ /* post-parser, pre-mapping records */ ],
    inputText:    string | null,   // text formats: original text as received; null for binary
    filteredText: string | null,   // text formats: text after dropRegex stripping; null for binary
    errors: [ { severity: 'error'|'warning', message, row? } ],
    meta: {
        format,         // schema.format echoed
        layout,         // 'delimited' | 'fixed' | null
        rowsIn,         // rows read pre-mapping
        rowsOut,        // rows in data array
        droppedLines,   // lines killed by dropRegex
        encoding,       // encoding used (text formats only)
        mappingMode,    // 'replace' | 'extend' | null (null when no mapping was supplied)
        durationMs
    }
}

The pipeline stages — inputText → filteredText → raw → data — are all exposed so the caller can inspect each transformation. (This is what powers the RAW input / Post-filter / Pre-mapping / Data tabs in Schema Studio.)

The promise rejects only on unrecoverable failures: unsupported format, schema validation failure, file unreadable, format-specific catastrophic error. Row-level issues populate errors and resolve normally.

Error handling

Two distinct failure surfaces:

1. Row-level issues (the common case). The promise resolves; inspect result.errors. Each entry is { severity: 'error' | 'warning', message: string, row? }. Today: PapaParse delimited parse problems are emitted as severity: 'error'; unknown schema keys and dropRegex-on-binary-format are emitted as severity: 'warning'.

2. The promise rejects. Three exception shapes, distinguished by which extra property is attached:

try {
    const result = await DataParser.parse(file, schema);
    // ...
} catch (err) {
    if (err.validationErrors) {
        // Schema didn't validate. err.validationErrors is a string[]
        // of human-readable problems. err.message is "Schema validation: <joined>".
        console.error('Bad schema:', err.validationErrors);
    } else if (err.errors) {
        // strict mode caught row-level errors that would normally be collected.
        // err.errors is the same shape as result.errors above.
        // err.message is "Strict mode: N parse error(s)".
        console.error('Parse failed under strict mode:', err.errors);
    } else if (err.cause) {
        // The format-specific parser threw (corrupt XML root, unreadable ZIP, etc.).
        // err.message is "Parse failed (<format>): <underlying message>".
        // err.cause is the original Error from the upstream library.
        console.error('Parser blew up:', err.message, err.cause);
    } else {
        // Bare Error — usually a payload normalization issue
        // (e.g., "payload must be Blob, File, ArrayBuffer, or string").
        console.error(err.message);
    }
}

strict: true upgrades any collected severity: 'error' entry to a thrown rejection at the end of parsing. warning-severity entries are never upgraded. The default (strict: false) collects everything into result.errors and resolves.

Engine version

The IIFE build's companion dist/version.js sets window.DPE_ENGINE_VERSION to a string of the form bN.YYMMDD.HHMMSS (build number, date, time) — useful for diagnostics, audit logs, or surfacing in a UI footer. If you don't load version.js, the global is simply absent; the engine doesn't require it.

The ESM build does not auto-load the version stamp. Import it as a module side-effect if you want the same global, or read dist/version.js's contents directly at build time.

Examples

Messy delimited file with header/footer cruft

const result = await DataParser.parse(blob, {
    format: 'csv',
    delimiter: '|',
    dropRegex: ['^=', '^Generated:', '^-{3,}', '^END OF'],
    mapping: { id: 'sku', name: 'description', buy: 'buy_price' }
});

Fixed-width PRN with page headers

const result = await DataParser.parse(blob, {
    format: 'prn',
    layout: 'fixed',
    encoding: 'windows-1252',
    dropRegex: [
        '^Page \\d+ of \\d+',
        '^Report Date:',
        '^=+$',
        '^SKU\\s'             // strip column-header line
    ],
    fieldDefinitions: [
        { name: 'sku',  start:  0, end:  9 },
        { name: 'desc', start:  9, end: 34 },
        { name: 'buy',  start: 34, end: 43 },
        { name: 'sell', start: 43, end: 51 }
    ]
});

XML with deep extraction + mapping

const result = await DataParser.parse(blob, {
    format: 'xml',
    rootPath: 'Envelope.Body.Records.Record',
    mapping: {
        id:       '@_id',                   // XML attribute
        category: 'Material.Category',      // nested element
        buyPrice: 'Pricing.Buy.#text'       // text node of nested element
    }
});

JSON with rootPath

const result = await DataParser.parse(blob, {
    format: 'json',
    rootPath: 'data.prices',
    mapping: { id: 'sku', price: 'buy' }
});

Spreadsheet (xls / xlsx / ods)

const result = await DataParser.parse(file, {
    format: 'xlsx',
    sheetName: 'Prices',            // optional; defaults to first sheet
    mapping: { id: 'sku', buy: 'buy_price' }
});

Word / OpenDocument text

// DOCX: emits [{index, text}] per non-empty paragraph
const docResult = await DataParser.parse(file, { format: 'docx' });

// ODT: emits paragraphs as {index, text} AND table rows as {tableIndex, rowIndex, col0, col1, …}
const odtResult = await DataParser.parse(file, { format: 'odt' });

Passthrough (raw EDIFACT / X12 / unsupported text)

const result = await DataParser.parse(blob, {
    format: 'passthrough',
    dropRegex: ['^UNA', '^UNB', '^UNZ'],   // strip envelope segments
    mapping: { line: 1, content: 2 }        // positional: field 1 = line number, field 2 = text
});
// result.raw  → [{line: 1, text: "UNH+1+PRICAT:..."}, {line: 2, text: "LIN+1++..."}, ...]
// result.data → [{line: 1, content: "UNH+1+PRICAT:..."}, {line: 2, content: "LIN+1++..."}, ...]

Headerless CSV with positional mapping

const result = await DataParser.parse(blob, {
    format: 'csv',
    delimiter: ',',
    hasHeaders: false,
    mapping: { sku: 1, description: 2, price: 4 }   // 1-indexed column positions
});
// Each row arrives as an array — mapping extracts by position
// result.data → [{sku: "ABC-100", description: "Copper wire", price: "3.42"}, ...]

Pre-filter (`dropRegex`)

For real-world print-format dumps (.prn, .txt), the page headers, dates, banners, and decorative lines often outnumber the actual data rows. dropRegex strips them out before the parser sees the file:

raw text → [dropRegex strips matching lines] → clean text → parser

Accepts a single string or an array of strings.
Each is compiled as a RegExp with the m flag (multi-line anchors).
Any line matching any pattern is dropped.
Count of dropped lines is reported in meta.droppedLines.

dropRegex applies to every text format — csv, prn, txt, fixed, xml, json, and passthrough. For binary formats (xlsx, xls, ods, docx, odt) it is ignored and a warning is added to errors.

Mapping

If mapping is absent, records pass through unchanged.

mapping: {
    targetKey: 'sourceField',        // direct (string source)
    fullName:  'Name.Last',          // dot path on nested objects
    customId:  '@_id',               // XML attribute (fast-xml-parser prefix)
    price:     3                     // positional integer source (1-indexed)
}

Source value types:

String — named lookup. Literal key first; if undefined and the source contains ., fall back to dot-path resolution.
Positive integer ≥ 1 — 1-indexed positional access. For array rows: row[n - 1]. For object rows: Object.values(row)[n - 1]. Useful for header-less CSV, passthrough records, and fixed-width with auto-named fields.

Missing values become null (not undefined).

Mode (`mappingMode`)

Only meaningful when mapping is supplied.

Value	Behavior
`'replace'`	(default) Output object contains only the mapped target keys.
`'extend'`	Output object contains all source fields, with mapped target keys overlaid on top.

Engine API takes a JSON object

Always. The output = input textarea syntax in Schema Studio is a UI-only convenience that serializes to / parses from this same JSON shape before reaching the engine.

Payload types

Payload	Text formats	Binary formats
`Blob` / `File`	`FileReader.readAsText(p, encoding)`	`FileReader.readAsArrayBuffer(p)`
`ArrayBuffer`	`new TextDecoder(encoding).decode(p)`	passthrough
`string`	passthrough	rejected

Dependencies

Library	Used by
PapaParse	delimited parser
fast-xml-parser	xml parser, odt parser
SheetJS (`xlsx`)	xls, xlsx, ods
Mammoth	docx
JSZip	odt

ESM build (dist/dpe.esm.js): dependencies are imported normally; your bundler resolves them.
IIFE build (dist/dpe.iife.js): dependencies are expected as window globals (Papa, XLSX, mammoth, XMLParser, JSZip). Load them from a CDN before loading DPE.

Version note: The CDN URLs in the Quick Start examples use fast-xml-parser 4.3.6 and SheetJS 0.18.5. The ESM build (package.json) pins fast-xml-parser ^5.8.0 and SheetJS 0.20.3. DPE uses a narrow slice of each library's API and works with both generations, but if you're bundling the ESM build, your resolved versions will differ from the CDN examples.

Building from source

Only needed if you're modifying the engine itself. Integrators consuming a vendored dist/ artifact don't need this.

npm install
npm run build        # produces dist/dpe.{esm,iife}.js plus dist/{version,studio-version}.js
npm run build:engine # engine artifacts only
npm test             # runs the Node test suite against samples/

Scope (v1)

Browser-first. The engine targets the browser. Node works for many formats (the test suite runs in Node — 46/46 passing), but it is not an officially supported runtime and edge cases in binary-format handling may surface.
Whole-file-into-memory. No streaming.
No type coercion (mapping returns raw values; convert downstream).
Target: ES2020 — runs in any evergreen browser. No transpile shim shipped for older runtimes.
TypeScript: no .d.ts ships with the build. Consumers using TS can either declare a minimal ambient module (declare module './vendor/dpe.esm.js' { ... }) or treat the return value as any at the call site.
License: DPE is released under the MIT License (see LICENSE). It bundles third-party libraries under their own permissive licenses — see THIRD-PARTY-NOTICES.md.

Status

Pre-release. Tested against synthesized samples for every format. Not yet hardened against the full diversity of real-world legacy file output. Bug reports welcome.

Name		Name	Last commit message	Last commit date
.build		.build
assets		assets
dist		dist
samples		samples
src		src
tests		tests
.gitignore		.gitignore
GETTING_STARTED.txt		GETTING_STARTED.txt
LICENSE		LICENSE
README.md		README.md
THIRD-PARTY-NOTICES.md		THIRD-PARTY-NOTICES.md
build-engine.mjs		build-engine.mjs
build-studio.mjs		build-studio.mjs
index.html		index.html
package-lock.json		package-lock.json
package.json		package.json
studio.html		studio.html

Navigation Menu

Repository files navigation

Data Parser Engine (DPE)

Supported formats

Obtaining DPE

Quick start

Browser (drop-in)

ESM (with bundler)

The schema

Global keys

Format-specific keys

Return shape

Error handling

Engine version

Examples

Messy delimited file with header/footer cruft

Fixed-width PRN with page headers

XML with deep extraction + mapping

JSON with rootPath

Spreadsheet (xls / xlsx / ods)

Word / OpenDocument text

Passthrough (raw EDIFACT / X12 / unsupported text)

Headerless CSV with positional mapping

Pre-filter (`dropRegex`)

Mapping

Mode (`mappingMode`)

Engine API takes a JSON object

Payload types

Dependencies

Building from source

Scope (v1)

Status

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Data Parser Engine (DPE)

Supported formats

Obtaining DPE

Quick start

Browser (drop-in)

ESM (with bundler)

The schema

Global keys

Format-specific keys

Return shape

Error handling

Engine version

Examples

Messy delimited file with header/footer cruft

Fixed-width PRN with page headers

XML with deep extraction + mapping

JSON with rootPath

Spreadsheet (xls / xlsx / ods)

Word / OpenDocument text

Passthrough (raw EDIFACT / X12 / unsupported text)

Headerless CSV with positional mapping

Pre-filter (dropRegex)

Mapping

Mode (mappingMode)

Engine API takes a JSON object

Payload types

Dependencies

Building from source

Scope (v1)

Status

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Pre-filter (`dropRegex`)

Mode (`mappingMode`)