grpc

[ZidongLiu]

Summary

• Implement grpc server, server receives instruction, and client has infinite loop to ask to do work
• protobuf and generated file
• server
• client
• eval client
• unit test
• test with small models
• test with large models
• environment update
• add new port args
• docker image
• lxc image

How to

Shortcut: At root folder, run python run_grpc.py --dataset=sst2 --eval-iterations=25 --large-model=opt-125m --model-dtype=float16 --seed=365 --iterations=2000 --train-batch-size=32 --test-batch-size=64 --num-clients=3 --num-sample-clients=2 --local-update-steps=1 --num-pert=5 --lr=0.000005 --momentum=0 --grad-estimate-method=rge-forward --mu=0.001.

This is equivalent to running following 3 steps.

At root folder, run server first, then spin up clients and eval client, for opt-125m model

1. in 1 terminal, run python grpc_server.py --dataset=sst2 --eval-iterations=25 --large-model=opt-125m --model-dtype=float16 --seed=365 --iterations=2000 --train-batch-size=32 --test-batch-size=64 --num-clients=3 --num-sample-clients=2 --local-update-steps=1 --num-pert=5 --lr=0.000005 --momentum=0 --grad-estimate-method=rge-forward --mu=0.001
2. in another 3(num_clients) terminals, run python grpc_client.py --dataset=sst2 --eval-iterations=25 --large-model=opt-125m --model-dtype=float16 --seed=365 --iterations=2000 --train-batch-size=32 --test-batch-size=64 --num-clients=3 --num-sample-clients=2 --local-update-steps=1 --num-pert=5 --lr=0.000005 --momentum=0 --grad-estimate-method=rge-forward --mu=0.001
3. in a final terminal, run python grpc_eval_client.py --dataset=sst2 --eval-iterations=25 --large-model=opt-125m --model-dtype=float16 --seed=365 --iterations=2000 --train-batch-size=32 --test-batch-size=64 --num-clients=3 --num-sample-clients=2 --local-update-steps=1 --num-pert=5 --lr=0.000005 --momentum=0 --grad-estimate-method=rge-forward --mu=0.001

To change server port address, use --rpc_master_addr, --rpc_master_port. This config works for server, training client and eval clients.

[ZidongLiu]

questions:

1. image we uploaded is a extension of the base image. It should have everything installed and repo cloned?
2. time it takes for image to popup in the website
3. number of GPUs
4. how many images we need to worry about. -> 2 images, 1 docker for GPU, 1 lxc image for message
5. Manual operations.
6. Order of these manual operations