global-electrification-platform · danielfdsilva · Jun 6, 2019 · Sep 23, 2019 · Nov 5, 2019 · Nov 5, 2019
diff --git a/_config.js b/_config.js
@@ -4,7 +4,7 @@ module.exports = {
   description: 'Data ingest documentation',
   sectionsMeta: {
     's0-introduction': 'Introduction',
-    's1-preparing-the-data': 'Preparing the data',
+    's1-guides': 'Guides',
     's2-ingest-tool': 'Using the ingest tool'
   }
 };
diff --git a/app/posts/s0-introduction/0-0-about-the-platform/index.md b/app/posts/s0-introduction/0-0-about-the-platform/index.md
@@ -5,4 +5,6 @@ layout: post.html
 permalink: /
 ---
 
-Explore least cost electrification strategies around the world, interacting with country context and different investment scenarios by visualizing data provided by academic and policy organizations in an open format specification.
+A platform to explore least cost electrification strategies around the world, interacting with country context and different investment scenarios by visualizing data provided by academic and policy organizations in an open format specification.
+
+GEP is [open source](https://en.wikipedia.org/wiki/Open-source_software). 
diff --git a/app/posts/s0-introduction/0-1-architecture/index.md b/app/posts/s0-introduction/0-1-architecture/index.md
@@ -0,0 +1,12 @@
+---
+title: Architecture overview
+date: 2019-01-20
+layout: post.html
+---
+
+The components of GEP:
+
+- **Explorer**: a[single-page application](https://en.wikipedia.org/wiki/Single-page_application) to visualize electrification scenarios via web browser;
+- **Data Service**: [a server-side web API](https://en.wikipedia.org/wiki/Web_API) used by the Explorer to query scenarios data;
+- **Relational database**: PostgreSQL instance that store scenarios results and visualization options;
+- **Map service**: [vector tile server](https://en.wikipedia.org/wiki/Vector_tiles) used by the Explorer to fetch geospatial data for styling the maps;
diff --git a/app/posts/s0-introduction/0-1-data-overview/index.md b/app/posts/s0-introduction/0-1-data-overview/index.md
diff --git a/app/posts/s0-introduction/0-2-the-ingest-process/index.md b/app/posts/s0-introduction/0-2-the-ingest-process/index.md
diff --git a/app/posts/s1-guides/1-0-setup/index.md b/app/posts/s1-guides/1-0-setup/index.md
@@ -0,0 +1,11 @@
+---
+title: Application setup
+date: 2012-08-23
+layout: post.html
+---
+To run GEP locally it is necessary to install the Data Service, which provides an API to access ingested data, and the Explorer, a web interface to visualize models. Please visit the code repositories to learn how to setup in a local machine:
+
+- <a href="https://github.com/global-electrification-platform/data-service/" target="_blank">GEP Data Service</a>
+- <a href="https://github.com/global-electrification-platform/explorer/" target="_blank">GEP Explorer</a>
+
+Please refer to the [Architecture overview](/introduction/architecture-overview) to learn GEP components.
diff --git a/...paring-the-data/1-1-vector-tiles/index.md → ...posts/s1-guides/1-1-vector-tiles/index.md b/...paring-the-data/1-1-vector-tiles/index.md → ...posts/s1-guides/1-1-vector-tiles/index.md
@@ -4,14 +4,15 @@ date: 2012-09-28
 layout: post.html
 ---
 
-GEP Explorer consumes geo data from vector tilesets in order to render map of scenarios in a efficient way. Every model should have a vector tileset defined in its configuration. At the moment, the GEP Explorer only supports polygon geometries (clusters), not point data.
+GEP Explorer consumes geo data from vector tilesets in order to render a map of scenarios in an efficient way. Every model should have a vector tileset defined in its configuration. At the moment, the GEP Explorer only supports polygon geometries (clusters), not point data.
 
-Each geometry in the dataset should have a single property `id`, of type integer. Other properties should be discarded as they will not be used by the web client and will increase tile size. 
+Each geometry in the dataset should have a single property `id`, of type integer. Other properties should be discarded as they will not be used by the web client and will increase tile size.
 
-A recommended way to generate and host a vector tileset is to use the command-line tool tippecanoe and upload the resulting MBTiles file to Mapbox Studio. 
+A recommended way to generate and host a vector tileset is to use [the command-line tool tippecanoe](https://www.npmjs.com/package/tippecanoe) and upload the resulting MBTiles file to Mapbox Studio.
 
 An example command to generate MBTiles from GeoJSON:
 
+
     tippecanoe -o gep-features.mbtiles gep-features.geojson
 
 The option `--drop-densest-as-needed` should be avoided, if possible, as it limits the number of features displayed on the map at national level, affecting visual analysis.
diff --git a/app/posts/s1-guides/1-2-model-results/index.md b/app/posts/s1-guides/1-2-model-results/index.md
@@ -0,0 +1,17 @@
+---
+title: Formatting Model Results
+date: 2012-08-23
+layout: post.html
+---
+Electrification model’s datasets are composed of three parts:
+
+
+- **Geodata**: geographic features corresponding to each analyzed area;
+- **Model configuration**: includes general metadata about the model, like title, description and version. Also includes applicable filters, lever combinations and map display configuration;
+- **Scenarios**: series of datasets, each one is the output of a model using a specific lever combination.
+
+Each model is ingested in the platform separately, being composed by one [YAML](https://en.wikipedia.org/wiki/YAML) and several [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) files. The naming convention described below should be followed to allow the CLI tool to identify the model and scenario data correctly:
+
+
+- `{model_id}.yaml`: model configuration file;
+- `{model_id}-{levers_id}.csv`: scenario’s outputs, where `levers_id` is the level combination that generated the results available in each file.
diff --git a/...ta/1-0-model-config/1-1-0-levers/index.md → ...es/1-3-model-config/1-3-1-levers/index.md b/...ta/1-0-model-config/1-1-0-levers/index.md → ...es/1-3-model-config/1-3-1-levers/index.md
diff --git a/...a/1-0-model-config/1-1-1-filters/index.md → ...s/1-3-model-config/1-3-2-filters/index.md b/...a/1-0-model-config/1-1-1-filters/index.md → ...s/1-3-model-config/1-3-2-filters/index.md
diff --git a/...nfig/1-1-2-1-external-map-layers/index.md → ...config/1-3-3-external-map-layers/index.md b/...nfig/1-1-2-1-external-map-layers/index.md → ...config/1-3-3-external-map-layers/index.md
diff --git a/...config/1-1-2-2-tech-layers-style/index.md → ...l-config/1-3-4-tech-layers-style/index.md b/...config/1-1-2-2-tech-layers-style/index.md → ...l-config/1-3-4-tech-layers-style/index.md
diff --git a/...-model-config/1-1-2-full-example/index.md → ...-model-config/1-3-5-full-example/index.md b/...-model-config/1-1-2-full-example/index.md → ...-model-config/1-3-5-full-example/index.md
diff --git a/...paring-the-data/1-0-model-config/index.md → ...posts/s1-guides/1-3-model-config/index.md b/...paring-the-data/1-0-model-config/index.md → ...posts/s1-guides/1-3-model-config/index.md
@@ -18,6 +18,9 @@ A YAML file describes the model configuration and is used to set up the GEP Expl
 - `attribution`: object describing model provider:
   - `author`: name of model provider;
   - `url`: URL to provider website;
+- `sourceData`: object with urls for the model source data
+  - `clusters`: URL to scenario clusters for download
+  - `scenarios`: URL to full scenario data for download
 - `timesteps`:  an array containing the intermediate and final years used on the model, in respective order;
 - `levers`: array of [lever objects]({{baseurl}}/preparing-the-data/levers) describing possible combinations of input variables used on the model. The order of the objects in the array determines the order in the interface.
 - `filters`: array of [filter objects]({{baseurl}}/preparing-the-data/filters) describing filters that can be applied to scenario data. The order of the objects in the array determines the order in the interface.

diff --git a/...g-the-data/1-2-scenarios-results/index.md → .../s1-guides/1-4-scenarios-results/index.md b/...g-the-data/1-2-scenarios-results/index.md → .../s1-guides/1-4-scenarios-results/index.md
diff --git a/app/posts/s2-ingest-tool/2-0-prepare-the-environment/index.md b/app/posts/s2-ingest-tool/2-0-prepare-the-environment/index.md
diff --git a/app/posts/s2-ingest-tool/2-0-the-ingest-process/index.md b/app/posts/s2-ingest-tool/2-0-the-ingest-process/index.md
@@ -0,0 +1,25 @@
+---
+title: The ingest process
+date: 2019-01-20
+layout: post.html
+---
+
+#### Hardware requirements
+As models outputs might have gigabytes of data, the ingest process can take several hours to complete, even for small countries. A dedicated computer is needed, with the following requirements:
+
+- Stable network connection
+- Sleep/hibernation mode disabled
+- 4 GB of RAM or more
+- Total free space two times bigger the size of uncompressed scenarios (approx. 250GB for large countries)
+- In AWS the minimum instance is `t2.medium`
+
+#### Install the command-line tool
+The [gep-data-service](https://github.com/developmentseed/gep-data-service) comes with a CLI (command-line interface) tool to import, delete and update the data in the platform. To install the CLI, clone the repository locally and follow instructions on "Install Dependencies" section. Once done, the CLI the command can be executed at repository root folder with:
+
+    node cli/ [command]
+
+The CLI will need to know the database to which data must be imported. Define a target database by exporting a PostgreSQL connection string to environment variable `PG_CONNECTION_STRING`. Example:
+
+    $ export PG_CONNECTION_STRING=postgresql://user:password@customhost:5432/gep
+
+Please refer to PostgreSQL manual to learn [how to write a connection string](https://www.postgresql.org/docs/9.6/libpq-connect.html#LIBPQ-CONNSTRING).
diff --git a/app/posts/s2-ingest-tool/2-1-available-commands/index.md b/app/posts/s2-ingest-tool/2-1-available-commands/index.md
@@ -4,57 +4,38 @@ date: 2019-01-20
 layout: post.html
 ---
 
-The [gep-data-service](https://github.com/developmentseed/gep-data-service) comes with a CLI (command-line interface) tool to import, delete and update the data in the platform.
-To install the CLI, clone the repository locally and follow instructions on "Install Dependencies" section. Once done, the CLI the command can be executed at repository root folder with:
-
-
-    yarn gep [command]
-
-The CLI will interact with the database through a connection string defined through the `PG_CONNECTION_STRING` environment variable.
-
-
-    export PG_CONNECTION_STRING=<connection-string>
-
-## list
+#### List existing models
 Outputs the existing models in the database. Example:
 
+    node cli/ list
 
-    yarn gep list
-
-## delete <ids…>
+#### Delete existing models
 Deletes the models and respective data that match the given ids. Example:
 
+    node cli/ delete mw-1 mw-2
 
-    yarn gep delete mw-1 mw-2
-
-## validate <path>
+#### Validate model output before ingest
 Validates the model and data at the given `path`. The path should point to the directory where the model and data are stored, not to the model itself. Example:
 
-    yarn gep validate ./data/mw-1/
+    node cli/ validate ./data/mw-1/
 
-## ingest [options] <path>
+#### Ingest a model
 Ingests the model and data at the given `path`. The path should point to the directory where the model and data are stored, not to the model itself. Example:
 
-
-    yarn gep ingest ./data/mw-1/
+    node cli/ ingest ./data/mw-1/
 
 If there is a model with the same id in the database the command will fail. If you want to re-ingest the model and override the existent one use the `--override` flag. Example:
 
+    node cli/ ingest --override ./data/mw-1/
 
-    yarn gep ingest --override ./data/mw-1/
-
-## ingest --config-only <path>
-
+#### Update a existing model
 Updates the non-data parts of model found at the given `path`. The path should point to the directory where the model and data are stored, not to the model itself.
-
 While it is not possible to update data related configurations (like levers and filters) without performing a full ingest, it is allowed to update metadata information, like descriptions and labels. Example:
 
-
-    yarn gep ingest --config-only ./data/mw-1/
+    node cli/ ingest --config-only ./data/mw-1/
 
 The following model properties can only be modified when doing a full ingest and the script will error if they’re changed:
 
-
     id
     country
     timesteps

diff --git a/package.json b/package.json
@@ -66,7 +66,7 @@
     "gulp-if": "^2.0.2",
     "gulp-imagemin": "^4.1.0",
     "gulp-load-plugins": "^1.5.0",
-    "gulp-markdown-to-json": "^1.0.3",
+    "gulp-markdown-to-json": "^1.1.0",
     "gulp-plumber": "^1.1.0",
     "gulp-rev": "^8.1.0",
     "gulp-rev-replace": "^0.4.3",