-
Notifications
You must be signed in to change notification settings - Fork 0
Instructions
This tutorial assumes that the user has python 2.7 installed. This tutorial also uses anaconda to create a python virtual envirnoment but virtualenv should be able to work in its place as well.
1. Create conda virtual environment:
conda create -n toil_test python=2.7
2. Activate conda environment:
source activate toil_test
4. Install toil to virtual environment:
pip install toil[all]
This tutorial is assuming the user already has an AWS set up. Also, it is only needed once, but AWS requires that users “subscribe” to use the Container Linux by CoreOS AMI. You will encounter errors if this is not done.
1. Generate key pair for AWS:
ssh-keygen -t rsa
2. This should prompt you to save your key. Save it in:
~/.ssh/id_rsa
3. Now move this to where it can be seen as an authorized key:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
eval `ssh-agent -s`
ssh-add
4. You’ll also need to chmod your private key (good practice but also enforced by AWS):
chmod 400 ~/.ssh/id_rsa
6. Next, you’ll need to add the key to AWS via the browser. For example, on us-east-1, this address would accessible here. For other regions, simply replace the region in the url.
7. Click the "Import Key Pair" button and add the .pub file from above. The easiest way to do this is to:
cat ~/.ssh/id_rsa.pub
and paste the results in the text box that comes up when you click the "Import Key Pair" button.
8. You'll next need an AWS access key. For those in the Griffith lab, this has already been done. For others,
On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security >Credentials. Expand the Access keys (access key ID and secret access key) section. Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to >a file on your computer. After you close the dialog box, you can’t retrieve this secret access key >again.
9. Now you should have a “AWS Access Key ID” and “AWS Secret Access Key”. We can now install the AWS CLI and make sure that it has the proper credentials. First install the AWS client:
pip install awscli --upgrade
10. Now configure your AWS credentials with:
aws configure
11. Add your “AWS Access Key ID” and “AWS Secret Access Key” from earlier and your region and output format:
" AWS Access Key ID [****************Q65Q]: "
" AWS Secret Access Key [****************G0ys]: "
" Default region name [us-east-1]: "
" Default output format [json]: "
12. Toil also relies on boto, and you’ll need to create a boto file containing your credentials as well. To do this, run:
pip install --upgrade boto
Then add your AWS access key credentials to the file ~/.boto
nano ~/.boto
Paste in the following (with your actual “AWS Access Key ID” and “AWS Secret Access Key”):
[Credentials]
aws_access_key_id = ****************Q65Q
aws_secret_access_key = ****************G0ys
1. Now everything should be setup to launch a toil instance. To launch an instance with just a t2.medium leader node, the following command can be used. If not working in aws region us-east-1, the TOIL_AWS_AMI parameter will have to be changed by finding the appropriate AMI id here.
TOIL_AWS_AMI=ami-3f061b45 TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:latest toil launch-cluster my-cluster --leaderNodeType t2.medium --zone us-east-1a --keyPairName id_rsa
2. You can ssh into the leader node by using the following command.
toil ssh-cluster --zone us-east-1a my-cluster
Exiting out of the instance just requires typing exit
3. In order to shut down the cluster correctly, use the following command. This will terminate the instance, clean up any s3 buckets that were created, and delete the security group that was created for the cluster.
toil destroy-cluster --zone us-east-1a my-cluster