Split up the main function and add separate wait time script - #3
Split up the main function and add separate wait time script#3isurugamage37 wants to merge 3 commits into
Conversation
| return float('inf') | ||
|
|
||
| def get_wait(): | ||
| channel = grpc.insecure_channel("149.165.151.41:8079") |
There was a problem hiding this comment.
Please use a global config variable/file.
| print(f"[{datetime.now()}] Failed to estimate for {model_name}: {e.details()}") | ||
| return float('inf') | ||
|
|
||
| def get_wait(): |
There was a problem hiding this comment.
The estimated wait time is instantaneous per model. Doing it for all models is not necessary. This function can be removed. I suggest making /one/ get_estimated_wait for the smallest model, and adding the different service times for the other models. This reduces the number of rpcs.
Also please update the "if total_est > deadline /1000 :" clause below to avoid skipping requests, and invoking the fastest model (and maintaining a separate counter for these cases)
There was a problem hiding this comment.
Prof. @prateek-s ,
If we call mobilenet_v3_small to get the wait time, that time already includes mobilenet’s own service time. Should we subtract mobilenet’s service time first to get just the queue wait time, and then add each model’s own service time when estimating for other models?
| return {m: await t for m, t in wait_tasks.items()} | ||
|
|
||
|
|
||
| def build_model_set(policy, wait_results, deadline_ms): |
There was a problem hiding this comment.
This should take the "max model-set size" as a parameter. Is it "K" ?
|
|
||
|
|
||
|
|
||
| async def QoED_test(transaction_id: str, deadline: int) -> dict: |
There was a problem hiding this comment.
rename function to something like "main_ensemble_invoke"
|
2 participants
No description provided.