From c581f6ff1af0390ae6eadfe1472f7edc5e68b2ce Mon Sep 17 00:00:00 2001 From: Nell Shamrell Date: Sat, 20 Feb 2021 18:13:58 -0800 Subject: [PATCH 1/2] updates set up docs Signed-off-by: Nell Shamrell --- docs/getting-started.md | 51 +++++++++++--------- import_export/README.md | 100 ++++++++++++++++++++++++++-------------- 2 files changed, 96 insertions(+), 55 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 52cf298..9aefc6b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,15 +1,20 @@ -# Background +# Getting Started + +## Background For an overview of the project, see the main [README](https://github.com/nyc-cto/tms/blob/master/README.md) -# Developing Local with Docker +## Running Locally with Docker This doc outlines how we test ELSA locally, using Docker to create a standardized development environment and manage dependencies. -### TODO: Document AWS cloud setup +### Pre-requisites +- [Install Docker](https://www.docker.com/) on your local workstation +- [Install Docker Compose](https://docs.docker.com/compose/) on your local workstation +- (recommended if you are using Windows or a Mac) [Install Docker Desktop on your workstation](https://docs.docker.com/desktop/) -# Docker Setup +### Docker Notes At a high-level you need to get familiarized with two docker functions- 1. `docker build` @@ -20,32 +25,35 @@ You will not be running this very often. Typically you'll be building a new imag 2. `docker-compose` This command is responsible to "spin-up" the container that is defined in `docker-compose.yml`. This file essentially tells Docker to run an instance of the image that was previously built. At this step, we will specify the environment-specific configurations as well as "mount" volumes that are shared by both your local machine as well as the docker container. This is useful because you may want to change some code in your local machine and quickly test those changes inside the docker container to see if it had the intended effect. +### Set Up + +**Clone this Git repository** -### Pre-requisites -- Create a local git repository ``` -mkdir your-dir-name -cd your-dir-name -git init -git remote add origin git@github.com:nyc-cto/tms.git -git pull origin master +git clone git@github.com:nyc-cto/tms.git +cd tms ``` -- Docker Desktop (https://www.docker.com/get-started) +**Set up env.template** + - Copy over `.env.template` into `.env` with any other environment variables the application needs access to. Whenever you update `.env`, don't forget to `source .env` -- Setup Secrets by looking at the following templates and re-creating them: - - `git_key.template` -> `git_key` - - `IngestionGoogleKey.json.template` -> `IngestionGoogleKey.json` - - `TranslationGoogleKey.json.template` -> `TranslationGoogleKey.json` + +**Set up secrets files** + +```bash +cp secrets/git_key_template git_key +cp secrets/IngestionGoogleKey.json.template IngestionGoogleKey.json +cp TranslationGoogleKey.json.template TranslationGoogleKey.json +``` Your google keys may be the same as the existing ones, but your git_key will certainly need to be created. For SSH keys, see [connecting-to-github-with-ssh](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh). Currently the image will generate keys during image build process. If you wish to use those, once you are inside the Docker container (as documented further down this doc), you can `cat /var/.ssh_keys/github_deploy_key.pub` and enter that value into the [Github Deploy Keys page](https://github.com/settings/keys). +### Building -## Build Process -1. Open/Run Docker Desktop +1. Have Docker running on your local workstation (If you are on Windows or a Mac, the easiest way to do this is to open/run Docker Desktop) 2. Build the docker image locally ``` cd ~/tms @@ -62,6 +70,7 @@ serge la ``` ## Container Up/Down + 1. Now that the image exists and is built-- you're ready to spin up the container. For this you will need two terminal windows/tabs. You can do this in one if you prefer. ``` docker-compose up @@ -91,8 +100,7 @@ boot dev home lib media opt root sbin shared_directory sys tm 7. When you are done, to tear down just run `exit` inside of the shell session. And you can ctrl+c to stop the container from running. - -# Running Serge with Docker +## Running Serge with Docker Serge is currently setup to run in a Docker container that is also setup to run any Python 3.8.5 applications. It's based on the Debian:buster image. @@ -123,7 +131,7 @@ serge push sampleconfigs.serge ``` -# Running Translation Service w/ Docker +## Running Translation Service w/ Docker These are documented in [translation_service/README](https://github.com/nyc-cto/tms/blob/master/translation_service/README.md) @@ -149,3 +157,4 @@ Exit code: -128; last error: No such file or directory The fix for this is to run your command with `sudo`. `sudo serge pull sampleconfig.serge --initialize` which will give your environment the right privileges to create/destroy directories. +### TODO: Document AWS cloud setup diff --git a/import_export/README.md b/import_export/README.md index 02846de..d56a6df 100644 --- a/import_export/README.md +++ b/import_export/README.md @@ -12,7 +12,7 @@ To properly import/export Google docs with Serge, you need to set up the followi 2. OAuth redentials allowing you to access the Google Drive/Docs API via your Gmail account. To set up these credentials: a. Go to the [Google API Console](https://console.developers.google.com/apis/credentials) b. Create a project, if you have not already - c. Go to `Create Credentials --> OAuth client ID` + c. Go to `Create Credentials --> OAuth client ID` (You should choose "Desktop app" as the Application Type and whatever you like as the Name) d. Create the credentials, download then, and save 2 copies, one as `credentials_drive.json` and another as `credentials_docs.json`, in a credentials folder called `secrets` in the root level of the git project. 3. A Google drive folder which will contain all of the original and translated documents in folders. The original documents lie in the `en` folder, and each language you are translating into will have documents in ISO 639-1 standard language code-named folders. For example, if in your Google Drive you create a root folder named `Translations` and want to translate into Spanish (es) and Mandarin (zh), the directory structure would be as follows: @@ -39,36 +39,68 @@ language_folders: ``` Where abc123 and def456 are the folder IDs for the en and es folders, respectively (example strings in this case). - -## Wordpress - -### Importing from Wordpress - -When a wordpress website changes, import-wordpress.py will download the new contents of the site via wordpress’s built-in JSON REST API at the endpoint `wp-json/wp/v2/posts`. It needs you to set the site’s login credentials WP_IMPORT_URL, WP_IMPORT_USER, and WP_IMPORT_PASSWORD in the .env file. Then you can run the script via `python import-wordpress.py` - -To activate it, import-wordpress needs to be notified each time the website changes. It runs a flask server which listens on port 5000 (the default flask port) for requests to the /wp-updates endpoint. If you are testing it locally, you may find it easiest to trigger the webhook manually by just loading http://localhost:5000/wp-updates in your browser. - -#### Using WP Webhooks to Listen for Changes - -When deployed to production, we suggest you use a wordpress plugin to automatically trigger the flask endpoint when the site is changed. We used WP Webhooks. Once this plugin is installed, on the wp-admin page, go to Settings->WP Webhooks. To make debugging easier, you may temporarily turn off the global setting "Deactivate Post Trigger Delay". - -Go to Send Data->Send Data on Post Update. Add a new webhook. - -Webhook Name = Your name or something - -Webhook Url = http://IP.ADDRESS.OF.FLASK:5000/wp-updates - -Set  "allow unsafe looking urls". - -Click orange "Add" button - -Click "Send Demo". This should cause it to ping your flask endpoint. - -From the main menu on the left hand side of the page, go to Posts -> All Posts and click "Edit" under "Hello World". -Make a change to the text of the page and click "Update". -This should cause it to ping your flask endpoint. - -### Exporting to Wordpress - -TODO - + +### Example + +Make sure that the service is running somewhere (for example, in Docker, as detailed in [these setup instructions](../docs/getting-started.md)) + +Now that you have the Google Drive folder structure for translations, let's try running one. + +In your Translations/en folder, create a new doc called "translation1". + +In the doc, add this text: + +``` +Hello, world! +``` + +Next, head to your command line on your local workstation and enter your running Docker container (if you are using Docker to run the service): + +```bash +(your-local-workstation) $ docker-compose exec serge bash +``` + +Once you are in the container, change to this directory: + +```bash +(docker-container) $ cd /var/tms +``` + +And run this command: + +```bash +(docker-container) $ sh run_ingest_sync_simple.sh +``` + +## Wordpress + +### Importing from Wordpress + +When a wordpress website changes, import-wordpress.py will download the new contents of the site via wordpress’s built-in JSON REST API at the endpoint `wp-json/wp/v2/posts`. It needs you to set the site’s login credentials WP_IMPORT_URL, WP_IMPORT_USER, and WP_IMPORT_PASSWORD in the .env file. Then you can run the script via `python import-wordpress.py` + +To activate it, import-wordpress needs to be notified each time the website changes. It runs a flask server which listens on port 5000 (the default flask port) for requests to the /wp-updates endpoint. If you are testing it locally, you may find it easiest to trigger the webhook manually by just loading http://localhost:5000/wp-updates in your browser. + +#### Using WP Webhooks to Listen for Changes + +When deployed to production, we suggest you use a wordpress plugin to automatically trigger the flask endpoint when the site is changed. We used WP Webhooks. Once this plugin is installed, on the wp-admin page, go to Settings->WP Webhooks. To make debugging easier, you may temporarily turn off the global setting "Deactivate Post Trigger Delay". + +Go to Send Data->Send Data on Post Update. Add a new webhook. + +Webhook Name = Your name or something + +Webhook Url = http://IP.ADDRESS.OF.FLASK:5000/wp-updates + +Set  "allow unsafe looking urls". + +Click orange "Add" button + +Click "Send Demo". This should cause it to ping your flask endpoint. + +From the main menu on the left hand side of the page, go to Posts -> All Posts and click "Edit" under "Hello World". +Make a change to the text of the page and click "Update". +This should cause it to ping your flask endpoint. + +### Exporting to Wordpress + +TODO + From 61b329aab20591fe48bf8654b9519e01625a550f Mon Sep 17 00:00:00 2001 From: Nell Shamrell Date: Sat, 20 Feb 2021 18:17:00 -0800 Subject: [PATCH 2/2] add question for what time of Google OAuth application to choose Signed-off-by: Nell Shamrell --- import_export/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/import_export/README.md b/import_export/README.md index d56a6df..ba10118 100644 --- a/import_export/README.md +++ b/import_export/README.md @@ -12,7 +12,7 @@ To properly import/export Google docs with Serge, you need to set up the followi 2. OAuth redentials allowing you to access the Google Drive/Docs API via your Gmail account. To set up these credentials: a. Go to the [Google API Console](https://console.developers.google.com/apis/credentials) b. Create a project, if you have not already - c. Go to `Create Credentials --> OAuth client ID` (You should choose "Desktop app" as the Application Type and whatever you like as the Name) + c. Go to `Create Credentials --> OAuth client ID` (TODO: What Application Type should you choose?) d. Create the credentials, download then, and save 2 copies, one as `credentials_drive.json` and another as `credentials_docs.json`, in a credentials folder called `secrets` in the root level of the git project. 3. A Google drive folder which will contain all of the original and translated documents in folders. The original documents lie in the `en` folder, and each language you are translating into will have documents in ISO 639-1 standard language code-named folders. For example, if in your Google Drive you create a root folder named `Translations` and want to translate into Spanish (es) and Mandarin (zh), the directory structure would be as follows: