Data Commons

In the context of the community package, a data commons simply refers to a collection of data repositories, which contain data files that can be combined to form a unified dataset.

Data Repositories

Data repositories can be initialized with the init_repository() function, but minimally these are GitHub repositories containing data files in an expected format (as handled by data_reformat_sdad()).

The Social Data Commons: Education repository is an example.

Data Commons Projects

A data commons project can be initialized with the init_datacommons() function, which creates a basic directory structure and some standard files.

The Social Data Commons repository is an example of such a project.

The init_datacommons function also creates a monitor site to display information about the data commons (e.g., the Social Data Commons Monitor).

Two pieces of information define a particular data commons: repositories and views.

Repositories

The commons.json file contains a list of the data repositories that make up the data commons, in the form of GitHub username and repository name strings ({username}/{repository-name}).

This list is also kept in scripts/repos.txt, which is used to clone, pull, or push the repositories with the included Shell scripts (which may be necessary if you are accessing private repositories).

The datacommons_refresh() function keeps these lists aligned, and uses them to clone repositories to the repos directory, or pull them if they already exist.

As repositories are established/updated, they are also checked with the check_repository() function (if run_checks is TRUE), and the results are stored along with file information in manifest/repos.json.

graph TB
  classDef file fill:#ffffff40
  refresh("datacommons_refresh()")
  commons["commons.json"]:::file --> refresh
  repos["scripts/repos.txt"]:::file --> refresh
  refresh -- reconcile lists --> list["reconciled lists"]
  list --> commons
  list --> repos
  refresh --> update("update repos")
  update -- for each --> exists{{exists?}}
  exists -- No --> clone("git clone")
  clone --> output
  exists -- Yes --> pull("git pull")
  pull --> output["repos/repo"]:::file
  output -- add file metadata --> manifest["manifest/repos.json"]:::file
  check("check_repository(#quot;repos/repo#quot;)") -- add check results --> manifest
  output --> run_checks{{run_checks?}}
  run_checks --> check

Loading

Once repositories are established, the datacommons_map_files() function searches for and maps data files, resulting in two mapping files in the cache directory:

graph TB
  classDef file fill:#ffffff40
  files:::file -- searched for in ./repos --> mapper("datacommons_map_files()")
  mapper -- for each file --> valid{{"has expected variable and ID columns?"}}
  valid -- No --> Ignore
  valid -- Yes --> record("extract information")
  record -- add to --> map_ids["cache/id_map.json"]:::file
  record -- add to --> map_vars["cache/variable_map.csv"]:::file
  mapper -- record state of valid files --> manifest["manifest/files.json"]:::file

Loading

The datacommons_find_variables() function uses the resulting cache/variable_map.csv file, and can be used to find variables within a data commons.

Views

Views are lists of variables and IDs to be extracted from data commons files. These can be created and updated with the datacommons_view() function.

Running the view function will trigger the file mapper if needed, then use those ID and variable maps to select files. Once files are selected, they are passed to the data_reformat_sdad() function, which processes the files to create unified files:

graph TB
  classDef file fill:#ffffff40
  def["views/{name}/view.json"]:::file --> view("datacommons_view()")
  view --> select("selection process")
  map_ids["cache/id_map.json"]:::file --> select
  map_vars["cache/variable_map.csv"]:::file --> select
  select -- after selection --> manifest["{output}/manifest.json"]:::file
  select --> reformat("data_reformat_sdad(files)")
  files["repos/../data.csv"]:::file -- read files --> reformat
  reformat -- "for each dataset (region type)" --> unified["{output}/{type}.csv.xz"]:::file
  reformat --> coverage["{output}/coverage.csv"]:::file

Loading

If a run_after script is specified, this will be run after the new unified files are made, which can be used to rebuild a data site in the view's output directory.