Welcome to the repository for 'The Art of Literary Modeling' (AoLM), a PhD project and dissertation by Jonathan Armoza that develops a framework for measuring and assessing the data quality of corpora of digital literature in digital humanities research.
This repository accompanies the dissertation The Art of Literary Modeling and contains the code used for the experiments and tutorial demonstrations described in the text.
AoLM's code can be divided into three functional parts: (1) reading in and processing digital texts and metadata into comparable components, (2) making data quality measurements on those inputs, and (3) assessing those measurement outputs together to determine the overall quality of the dataset.
The core of AoLM is its data quality metrics, of which there are six working examples. The repository's assessment code consists of a set of workflow scripts dubbed 'experiments' that run these data quality metrics over sample corpora of digital texts and metadata and then output all of the metrics' measurements to file and/or plot select metric measurements for visual comparison.
Accompanying and aiding that reading/measurement/assessment code is a small library of utility scripts for text processing and data visualization. Text processing code used by the final version of the project can be found in the aolm_code/objects and aolm_code/utilities folders, and visualization code can be found within the 'experiment' and separate plotting scripts in the experiments/chapter1 folder.
Below, you will find documentation on all three functional parts of AoLM including a tutorial demonstrating how to work with data quality metrics. Additional, file-by-file documentation for code used in the final version of AoLM – as well as prototypical code that went unused for it – can be found in the 'Appendix' section of the written portion of 'The Art of Literary Modeling'. (Watch this space for a link to that writing once it is made public.)
NOTE: Any script in the AoLM repository requires the installation and activation of the 'aolm' conda environment. See the setup section of the data quality metric tutorial for installing this environment on your system from the command line.
Several public domain datasets of literature are also in the repository, including 14 digital editions of Mark Twain's Adventures of Huckleberry Finn from Project Gutenberg, the Internet Archive, and Mark Twain Project Online, all 3 volumes of Twain's autobiography from Mark Twain Project Online, 9 digital editions of novels by Herman Melville from Project Gutenberg, and digital editions of all of Emily Dickinson's poems and their variants (over 4800 files) from Emily Dickinson Archive. Each digital edition (novel/autobiography/poem) has been downloaded directly from their source archive online and processed to a minimal degree to aid their use by AoLM's data quality metrics while maintaining as much of their rawness as possible. Each digital edition is also supplemented by a set of metadata about it provided by those source archives.
- Quickstart
- Text and Metadata Ingestion
- Data Quality Metrics
- Data Quality Metric Tutorial
- Workflow Scripts and Overall Data Quality Assessment
- Command Line Tool
- Reproducibility
- Glossary
- Clone the repository and create the environment:
git clone https://github.com/jarmoza/aolm_full
cd aolm_full
conda env create -f environment.yml
conda activate aolm- Install spaCy models:
python -m spacy download en_core_web_lg
python -m spacy download en_core_web_sm- Run the (solved) tutorial:
python tutorial/tutorial_solution.pyIn order to measure the data quality of digital text files and metadata about them, AoLM processes those files into generalized components in order to compare them. AoLM's archive-specific reader objects make this possible. Below is a description of the JSON file formats for text and metadata as well as a description of those reader objects – which may be extended and modified using class inheritance for your own projects.
With AoLM's ethos of only using minimal intervention during the processing of digital texts in order to read them into memory for data quality measurement, the suggestion is to locate beginning and ends of the primary components of your raw digital source text. If there are subcomponents of the body text that you wish to isolate, inserting a consistent demarcation marker (i.e. "CHAPTER <roman numeral>.") is recommended. The reader objects and text JSON format have implementations that accommodate both begin and end lines as well as text component demarcation markers. You can observe an example of chapter markers in this 'raw' text edition of Adventures of Huckleberry Finn.
AoLM's text JSON file format has two sections: keys and component. The keys section contains values to help a text processing script navigate the raw input txt file of the work and values to describe the output from text processing that will appear in the components section. AoLM uses the IAHuckFinnWriter class object to do this work - where the JSON file's keys section is filled out by the user and then the paths to the raw txt and that JSON file are given to IAHuckFinnWriter to produce the components section of the JSON file. (The resulting completed JSON file can then be read into memory by a reader object. See below.) See this example of a completed JSON version of the February 2021 edition of Adventures of Huckleberry Finn on Project Gutenberg.
The keys section contains order, input, and output subsections. order defines the order of the primary components of the text (i.e. [header, frontmatter, body, footer]). A startline and endline are defined for each of those text components in the input subsection. The output subsection specifies the key that will be used in the file's component section. As you will see in the example file, body is a special case in that it contains subcomponents (i.e. chapters for a novel). NOTE: The prefix to be used to find those subcomponents in the raw text is specified in body in the input subsection under the key subcomponent_input_prefix. A prefix for those subcomponents (chapters) as they will appear in the component section is specified in body in the output subsection.
The metadata JSON files used by the metadata sufficiency metric, AoLM's only metadata metric, is more straightforward. See this example of metadata from the Internet Archive. Here key-value pairs from the source archive's metadata are simply listed in the file. (Note that the key strings themselves have been normalized for the sake of the metric's comparsion.) However, in cases where metadata must be manually extracted (i.e. from within the provided raw text like Project Gutenberg editions), AoLM's metadata file format (and metadata metric) also allows for an unkeyed_fields key where such key-value pairs can be placed separately from metadata that was specifically provided separately by the source archive. See this metadata file from the February 2021 Project Gutenberg edition of Adventures of Huckleberry Finn for an example of that scenario.
AoLM uses reader objects to load texts from their JSON file format into memory as a set of generalized components that its metric objects can use to make tallies and evaluative measurements from.
AoLM's scheme for doing this uses object-oriented programming class inheritance beginning with a AOLMText object which is used in the base reader class, AOLMTextReader. Child classes that derive from AOLMTextReader are created for texts coming from specific source archives so that the reading/ingestion functionality can be tailored to inputs coming from those sources. See for example, the Internet Archive reader for Huckleberry Finn editions as opposed to the Mark Twain Project Online reader for Huckleberry Finn editions.
Each data quality metric is part of what is called a "data quality assessment framework" (DQAF) in Information Science. AoLM uses as its foundation many of the concepts concerning data quality from the field and specifically from information scientist Laura Sebastian-Coleman in her book, Measuring Data Quality for Ongoing Improvement (2012).
The metrics implemented for the project reflect several core categories for data quality provided by Sebastian-Coleman, and are listed in the table below. In order to begin creating your own metrics, have a look at dq_metric.py. This script defines the base class to derive from for your own metrics, and contains the core functionality needed for a metric object. Those core functions include the constructor, compute, evaluate, and output methods. Your derived child class can and likely will include more helper functionality. Take a look at the code for my own metrics in the aolm_code/data_quality/core/dq_metrics folder to see examples of derived/expanded metric functionality.
| name | category | primary dimensions | secondary / inferred dimensions | function |
|---|---|---|---|---|
| lexical validity | intrinsic | - accuracy | - believability | evaluates token-level correctness by comparing text against external lexical resources |
| record consensus | representational | - representational consistency | - interpretability | measures cross-edition consistency by comparing shared textual features across versions |
| metadata sufficiency | contextual | - completeness - value-added |
- relevancy | assesses completeness and consistency of metadata across records and repositories |
| record counts to control records | contextual | - completeness | - accuracy | evaluates dataset completeness by comparing structural counts to a reference edition |
| authorial signature | representational | - representational consistency | - interpretability | quantifies stylistic variation by comparing normalized word-frequency distributions across texts |
| legomena | representational | - interpretability | - representational consistency | analyzes low-frequency lexical distributions to identify anomalies and distinctive patterns |
The lexical validity metric utilizes three external dictionaries (spaCy, the Corpus of Historical American English, and WordNet) to check if tokens in a digital text are valid/known English words. It evaluates this measurement at the chapter, edition, and corpus level.
The record consensus metric takes a set of digital editions and sees how much of a work's words and sentences match above a given percent match threshold across all editions. It does this at the chapter, edition, and corpus level.
The metadata sufficiency metric looks at metadata among digital editions that source from the same digital archive and considers the percent coverage of metadata keys as well as the percent of mismatch between (potentially) identical, but differently keyed values for those metadata keys.
The record counts to control records metric compares a set of editions against a (presumed) master edition to understand how much each of those compared editions' words and sentences and chapter counts match with those of the master edition. It evaluates these measurements at the chapter, edition, and corpus level.
The authorial signature metric determines the average (document-length normalized) term frequency vector for a collection of digital texts and then measures the distance between each edition's term frequency vector (also normalized) to determine the distance each digital edition is from that "authorial signature" average vector. (NOTE: This produces a data quality measure in different terms than the percentages of the previous metrics.)
The legomena metric looks to the n-legomena featured in a set of digital texts – where 'n' is specified by the user of the metric. (Hapax legomena are words used just once in a body of text, dis legemona are words used twice, etc.) Vectors are computed consisting of the average counts for each legomena word for each chapter and edition. The average legomena total of those counts is calculated for the whole set of digital texts, and the average legomena count per work is considered as the final metric value. (This follows with the authorial signature's metric data quality measure as something other than a percent.)
This tutorial will guide you through a sample exercise in (1) reading a dataset of digital texts, (2) running the record counts to control records data quality metric over them, and (3) producing output files from the metric. The example dataset for the tutorial will be a set of editions of Mark Twain's Adventures of Huckleberry sourced from the Internet Archive, Project Gutenberg, and Mark Twain Project Online at University of California, Berkeley.
NOTE: All files and folders necessary for this tutorial exist in the project's tutorial folder. You will need to use your operating system's terminal/command prompt program as well as your favorite text editor program. Keep in mind that this will need to be an editor capable of editing plain text like 'Notepad', 'TextEdit', or a software development environment like 'Visual Studio Code' rather than a word processor like 'Microsoft Word'. This is because the AoLM's code is in Python, a programming language which requires the use of tab characters to create code block hierarchy (a.k.a. 'scope'). Here are the files and folders you will make use of for the tutorial:
tutorial/aolm_tutorial.py- The script file you will edit for the tutorial. Contains comments and pseudocode to help your work.tutorial/tutorial_solution.py- The script file containing the tutorial solutiontutorial/data- Thesmall_setandfull_setfolders in here are where digital editions for the tutorial exist. Note theeditionsandmetadatasubfolders in each. You will point your code in the tutorial script to theeditionsfolder withinsmall_setorfullset.tutorial/output- A folder that has been provided for you to point your code in the tutorial script to place your output files
The aolm_tutorial.py in the tutorial folder acts as a workspace for you to add code to as you follow along with the tutorial. Follow along with the steps outlined below and place your code in that script file in the tutorial_workspace function. Once you have run the script, the output files for this tutorial will be found in the tutorial/output folder. If you are satisfied with the results, you may choose to perform your own exercises by running different data quality metrics from the aolm_code/data_quality/core/dq_metrics folder. Note that you will need some beginner-level Python proficiency to follow along with the tutorial. The full tutorial solution along with some extra commentary on it can be found in tutorial_solution.py.
Functionality to enable further exercises and explorations of AoLM's metrics and datasets can be found in cli_lib.py as well as via a script to run AoLM's data quality metrics at the command line in aolm_cli.py. (NOTE: aolm_cli.py functions as a command-line tool for AoLM and has still to be fully tested. See the command line tool section below.)
In your terminal, run the git clone command below in a location on your hard drive where you would like the 'Art of Literary Modeling' code repository to live. (You can use the cd command to do this, e.g. cd /UserDirectory/code if such a folder exists. mkdir can be used to create new folders in your present directory as well, e.g. mkdir code.)
git clone https://github.com/jarmoza/aolm_full
First, check whether conda is already installed. Open your terminal (Command Prompt, PowerShell, or Terminal) and run:
conda --versionIf you see a version number (for example, conda 24.11.0), you can skip this step.
If not, follow the instructions for your operating system below.
Windows
- Open Command Prompt or PowerShell
curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Windows-x86_64.exestart "" /wait Miniforge3-Windows-x86_64.exe /InstallationType=JustMe /AddToPath=1 /S- Close your terminal, then open a new one
conda --version
macOS, Apple silicon
- Open Terminal
curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.shbash Miniforge3-MacOSX-arm64.sh -bconda init- Close your Terminal, then open a new one
conda --version
macOS, Intel
- Open Terminal
curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-x86_64.shbash Miniforge3-MacOSX-x86_64.sh -bconda init- Close your Terminal, then open a new one
conda --version
Linux
- Open a terminal
curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.shbash Miniforge3-Linux-x86_64.sh -bconda init- Close your terminal, then open a new one
conda --version
Troubleshooting: conda not found
If conda --version does not return a version number:
Windows
- Close and reopen your terminal
%USERPROFILE%\miniforge3\Scripts\activate- Try
conda --versionagain
macOS/Linux
source ~/miniforge3/bin/activateconda init- Close and reopen your terminal
- Try
conda --versionagain
In the aolm_full folder in your terminal, create the conda environment by running:
conda env create -f environment.yml
Next, run the following command. Repeat only this last command when you want to re-run
the tutorial script in the terminal. (To exit the 'aolm' conda environment, run conda deactivate.)
conda activate aolm
AoLM's metrics require the use two of spaCy's language models. Run the following commands in your terminal to download them:
python -m spacy download en_core_web_lg
python -m spacy download en_core_web_smRun the following command to ensure the models installed successfully:
python -c "import spacy; spacy.load('en_core_web_lg'); print('spaCy model installed correctly')"
Places to put code corresponding to the numbered and lettered steps below can be found in the tutorial_workspace function found in tutorial/aolm_tutorial.py. (Recall that Python requires the use of tab indentation for code lines. In this case, since code is being defined in tutorial_workspace, each line requires one tab indentation.)
The first thing you will do is define string variables for IDs and folder paths that will be used for reading digital texts/metadata and then later, running a data quality metric(s) over those texts/metadata. (This step can be skipped if you don't mind repeatedly entering raw string values into function/constructor calls.) These are used by AoLM's reader objects and data quality metric objects. A set of pre-defined values for these parameters have been placed at the top of the tutorial script for your convenience.
Helpful Notes:
- The
pwdcommand in your terminal/power shell program will show you the current, full directory path you are located in. (cdis the command in the Windows command prompt.) - Copy and paste the output into a string variable definition (like in step A below)
- The path must end in a folder separator character -
/for macOS and Linux,\for Windows
Building the input path via your terminal:
>pwd
/Users/myusername/aolm_full/tutorial/Copy and add /data/small_set/editions/ or /data/small_set/metadata/ (\ if in Windows):
/Users/myusername/aolm_full/tutorial/data/small_set/editions/
Define one (or two) variables for the locations of your digital editions. As mentioned, pre-defined dataset and metadata location variables have been placed at the top of the tutorial file, including a TUTORIAL_DATASET_LOCATION.
EDITIONS_LOCATION = '/Users/myusername/aolm_full/tutorial/data/small_set/editions/'
METADATA_LOCATION = '/Users/myusername/aolm_full/tutorial/data/small_set/metadata/'Define variables for IDs for the digital source of the compared editions and any baseline (e.g. 'master') edition In this case, the 'record counts to control records' metric requires a baseline edition. Other metrics will not require either ID if they are not comparing across digital text source archives like this metric does.
TUTORIAL_BASELINE_SOURCE_ID = 'mark_twain_project'
TUTORIAL_SOURCE_ID = 'internet_archive'
TUTORIAL_METRIC_ID = 'my_unique_metric_id'
TUTORIAL_COLLECTION_TITLE = 'Internet Archive'
TUTORIAL_WORK_TITLE = 'Adventures of Huckleberry Finn'The next step is to read in the digital edition files and/or metadata files you wish to use for your metric(s). AoLM uses custom JSON file formats for both editions and metadata. (The composition of these files are mentioned above in the 'Text and Metadata Ingestion' section.) For this tutorial, two convenience functions read_huckfinn_dataset_files_by_source and read_metadata_files_by_source have been provided for your use. You only need to specify the editions/metadata location and the string ID that represents the subfolder they are stored in (i.e. 'internet_archive'). Two differently-sized datasets have been provided for your use in the tutorial/data folder. The tutorial script defaults to using a dataset with just a few editions for comparison via the small_set folder, but a full_set folder also exists. (NOTE: metadata_files is defined below as an example, but is not used for this tutorial.)
edition_readers = read_huckfinn_dataset_files_by_source(
EDITIONS_LOCATION, TUTORIAL_SOURCE_ID)
edition_readers.update(read_huckfinn_dataset_files_by_source(
EDITIONS_LOCATION, TUTORIAL_BASELINE_SOURCE_ID))B. Reading Metadata into Memory (not for this tutorial; shown here for your future experimentation with the metadata metric)
metadata_files = read_metadata_files_by_source(METADATA_LOCATION, TUTORIAL_SOURCE_ID)This next step is where you will run a data quality metric on your editions or metadata. The available AoLM metrics include: 'record consensus', 'record counts to control records', 'lexical validity', 'metadata suffiency', 'authorial signature', and 'legomena'. Each class name you will need for these metrics can be found at the top of the tutorial script file in the section that lists all of the imports. This tutorial uses the 'record counts to control records' metric. (Its class name is DatasetCompleteness_RecordCountsToControlRecords.)
First, create the metric object and give it the parameters its constructor requires. (To see where these metric class constructors are defined look at the Python files in aolm_code/data_quality/core/dq_metrics. In the class section you will see a constructor definition that begins with def __init__(...):. This is where you will find the class object's required parameters. However, there is also a helper create_metric function in cli_lib.py you may utilize.)
metric = DatasetCompleteness_RecordCountsToControlRecords(
TUTORIAL_METRIC_ID,
edition_readers,
TUTORIAL_SOURCE_ID,
TUTORIAL_WORK_TITLE,
TUTORIAL_COLLECTION_TITLE,
EDITIONS_LOCATION,
TUTORIAL_BASELINE_SOURCE_ID
)This line creates a data quality metric object.
Each metric object performs two steps to produce its metric and sub-metric values. The metric object's compute function tallies the values of the editions or metadata it uses for producing the final metric values.
metric.compute()Then the metric object's evaluate function calculates statistics based on those tallies to produce the metric and sub-metric values.
metric.evaluate()In this final step, you will output both the tallies and the metric and sub-metric values of the data quality metric you just ran. Since the implementation of outputting these values has yet to be standardized across AoLM metrics, two convenience functions, output_metric_tallies and output_metric_values have been provided for your use. The former outputs a CSV file and the latter, a JSON file for the 'record counts to control records' metric. There has been an output folder provided in the tutorial folder for your convenience. (NOTE: Most other metrics produce JSON file for their tallies.) You will want to replace the folder path for the output files below with one for your computer. That path can be found by using the pwd command in your terminal to find the current directory path and adding the output/ folder name to its end. A script_run_time variable has also been provided in the tutorial_workspace function of aolm_tutorial.py if useful for timestamping your outputs.
Building the output path via your terminal:
>pwd
/Users/myusername/aolm_full/tutorialCopy and add /output/ (\ if in Windows):
/Users/myusername/aolm_full/tutorial/output/
output_metric_tallies(metric, "/Users/myusername/aolm_full/tutorial/output/" + "metric_tallies_" + script_run_time + '.csv')output_metric_values(metric, "/Users/myusername/aolm_full/tutorial/output/" + "metric_values_" + script_run_time + '.json')Once your edits are made in aolm_tutorial.py,
- Make sure you are located in the tutorial directory in your terminal program (i.e. using
cd /UserDirectory/code/aolm_full/tutorial/if that is the path where you placed theaolm_fullproject directory) - Run the following command:
python aolm_tutorial.pyCongratulations! You have just run a data quality metric over a number of editions of Adventures of Huckleberry Finn and output its measurements.
Look inside the tutorial/output folder for the csv and json files containing the record counts to control record metric values.
Metric Tallies File
The first stage of a metric (called compute) tallies the components of a corpus of digital texts it intends to evaluate for data quality. In this case with the 'record counts to control records' metric, the tallies output CSV file will contain percent match of the words and sentences of each chapter of each Huckleberry Finn edition against the Mark Twain Project Online's (MTPO) edition of the novel.
The CSV's column headers include: edition_name, chapter_name, count_type, and percent. Each row of values found in the CSV file depict the amount of words or amount of sentences that match when comparing a particular chapter in an edition against the same chapter in the MTPO edition.
Helpful Note: CSV files are best viewed in your preferred spreadsheet program (e.g. 'Microsoft Excel').
Metric Values File
The second stage of a metric (called evaluate) takes the tallies from the first stage and performs some light statistical calculations over them in order to determine the data quality of the corpus. This includes multiple levels of sub-metrics that are all used to calculate the final metric's overall data quality value. All of this information is output in hierarchical JSON form with descriptive key names.
Output values include:
- the overall metric
value(the mean of the corpus' percent match for chapter, word, and sentence counts) sub-metricvalues for the corpus- the percent match for the corpus for chapter count
- the percent match for the corpus for word count
- the percent match for the corpus for sentence count
- the total percent coverage concerning the presence of chapters in each edition (currently unusued in the overall metric value calculation)
sub-sub-metricvalues for each edition- the percent matches of chapters
- the percent matches of words
- the percent matches of sentences
Helpful Note: JSON files are best viewed in the plain text editor you used to edit the tutorial code.
The concept of a data quality assessment framework (DQAF) includes the notion that after one uses a set of the metrics to measure and evaluate aspects of a dataset, those metric outputs should then be used together to make an overall assessment of that dataset's quality.
Data quality assessment is meant to be iterative work, performed at intervals, depending on how often a dataset is updated. It sums up all of the data quality work performed, and ideally assigns a timestamp to that assessment. In essence, freezing the assessment in time states, "This was the determined quality of the dataset using these measures at this day/time."
One simple form of assessment calculation could be a weighted average of each data quality metric's topline value – with weights assigned according to one's own judgment of the importance of a particular metric in determining a dataset's overall quality rating. (However, assessment calculations can be as complicated as one deems fit.)
AoLM's data quality measuring workflow scripts in the experiments/chapter1 folder apply one or more data quality metrics to a selected corpus of digital texts. For instance, huckfinn_dataquality.py and huckfinn_dataquality_experiment2.py in that folder provide illustrative examples of data quality metrics being applied individually (huckfinn_dataquality_experiment2.py) or in combination (huckfinn_dataquality.py) over a dataset. Measurement tallies and evaluative data quality ratings are output via the metrics' output and eval_output class methods. huckfinn_dataquality.py provides a good example of this in its output_results function. With those metric outputs (typically CSV or JSON file form – see the exact metric's output functions) a full data quality assessment calculation can then be made programmatically via script or manually calculated by the person performing the measurements and assessment.
Those two workflow script files – along with several others in the experiments/chapter1 folder – also contain examples of visualizing the metric outputs, notably the bar and heatmap plots featured in the dissertation draft (see the plot_results, plot_results2, and plot_heatmap functions in either script file).
The Art of Literary Modeling (AoLM) provides a command-line tool for running data quality metrics on digital texts and their associated metadata. The tool currently only supports evaluating the data quality of the multiple digital editions of Adventures of Huckleberry Finn found in the project's datasets (e.g. from the Internet Archive, Project Gutenberg, and Mark Twain Project Online) by running any of the project's 6 data quality metrics over them. The tool produces tallies of aspects of the edition (i.e. words, sentences, chapters) and produces data quality scores from the respective metric(s). Metrics can be executed individually or in combination, and results are exported as CSV and JSON files for downstream analysis and visualization.
NOTE: The command line tool is a work-in-progress and as of early 2026 is undergoing debugging and testing.
The command-line tool, tutorial/aolm_cli.py, can be run via the Python interpreter in your terminal.
python aolm_cli.py [FLAGS]` OR `python3 aolm_cli.py [FLAGS]| Flag | Description |
|---|---|
--input-folder |
Path to the folder containing edition dataset files used by most text-based metrics. |
--metadata-folder |
Path to the folder containing metadata files used by the Metadata Sufficiency metric. |
--source-id |
Identifier for the primary edition source being evaluated (e.g., internet_archive, project_gutenberg). Defaults to the "small_set" dataset but can be adjusted in aolm_cli.py |
--baseline-source-id |
Identifier for a baseline edition source used in metrics that compare against a baseline, (e.g. 'master' edition) [default value: mark_twain_project]. |
--work-title |
Full title of the literary work being analyzed. |
--collection-title |
Name of the collection of editions being evaluated. |
--metrics |
String of metric codes to execute. Multiple metrics can be combined (e.g., mv, acr). |
--output-folder |
Destination folder for generated metric output files. |
Some primary flags also utilize their own set of arguments to help the tool understand exactly what is being requested to run. For example, the arguments for --metrics would be placed to the right of it and include the following:
| Code | Metric |
|---|---|
m |
Metadata Sufficiency |
v |
Lexical Validity |
c |
Record Consensus |
a |
Authorial Signature |
l |
Legomena |
r |
Record Counts to Control Records |
| Source ID | Description |
|---|---|
internet_archive |
Editions sourced from the Internet Archive JSON datasets |
project_gutenberg |
Editions sourced from Project Gutenberg JSON datasets |
mark_twain_project |
Scholarly XML edition from the Mark Twain Project (used as the baseline/master edition) |
The following command runs the metadata sufficiency and lexical validity metrics over the editions of Huckleberry Finn from the Internet Archive. (Recall that it would be python3 if that's your interpreter.)
python aolm_cli.py --metrics mv --source-id internet_archiveThis repository provides multiple environment specifications for reproducing AoLM's dissertation work and for further use of its code:
environment.yml— recommended for most users; easier to install and sufficient for running the tutorialenvironment.lock.yml— a more tightly specified environment derived from the original development setupconda_spec.txt— an explicit, fully pinned environment for maximum reproducibility
Choose the method below based on your needs.
For most users, the environment can be created using the provided environment.yml file:
conda env create -f environment.yml
conda activate aolmIf dependency resolution fails on some systems, the libmamba solver may help:
conda env create -f environment.yml --solver=libmambaFor a closer approximation of the original development environment:
conda env create -f environment.lock.yml
conda activate aolmThe file conda_spec.txt contains an explicit list of packages and build hashes exported from the original working environment. This allows the environment used during the project work for Art of Literary Modeling to be reproduced as closely as possible.
conda create --name aolm --file conda_spec.txt
conda activate aolmThe dissertation analyses were run using the following spaCy model versions:
en_core_web_lg==3.8.0
en_core_web_sm==3.8.0
When using the standard environment (environment.yml), spaCy will install the latest compatible model version. For exact reproduction of the research environment, install the versions listed above.
The spaCy language models used in the project are installed separately:
python -m pip install en_core_web_lg==3.8.0
python -m pip install en_core_web_sm==3.8.0These models were used for all NLP processing in the tutorial and dissertation analyses.
The degree to which data meets the expectations of data consumers for a specific intended use.
A structured, iterative system for evaluating, measuring, and improving data quality through profiling, measurement, and assessment cycles.
A specific aspect of data quality (e.g., accuracy, completeness) used to define and organize measurements.
A higher-level grouping of dimensions:
- Intrinsic
- Contextual
- Representational
- Accessibility
The extent to which data correctly represents real-world values or trusted reference sources.
Correctness of data values relative to a real-world object or reference standard.
Degree to which data is generated without bias or subjective distortion.
Degree to which data is regarded as credible by users.
Trustworthiness of the data source or provenance.
The extent to which data is appropriate for a specific task or use case.
Applicability of data to the task at hand.
Currency and availability of data relative to its use.
Degree to which required data is present.
Sufficiency (not excess or deficiency) of data for a task.
Contribution of data to achieving a specific goal or outcome.
The extent to which data is presented clearly and is interpretable.
Clarity of meaning, units, and definitions.
Degree to which data can be readily comprehended.
Uniformity of format and structure across data.
Compactness of representation without loss of meaning.
The extent to which data can be obtained and used.
Ease and speed with which data can be retrieved.
Controls governing who can access data and under what conditions.
Examination of a dataset’s structure, content, and distributions to understand its condition.
Analysis of individual fields (type, format, range, validity).
Analysis of relationships across data (dependencies, hierarchies, cardinality).
Interpretive comparison of measured data against expectations to determine quality.
A specific procedure used to evaluate a data quality dimension.