Skip to content

hll memory estimate in MB instead of Bytes #312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jshencode wants to merge 1 commit into
base: master
Choose a base branch
from
+2 −2
Open

hll memory estimate in MB instead of Bytes #312

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions query/aql_processor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import (
)

const (
hllQueryRequiredMemoryInMB = 10 * 1024
hllQueryRequiredMemoryInBytes = 10 * (1 << 30)

@voyager-dw voyager-dw Sep 30, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so originally we only reserved 10K for HLL? and now 10GB, isn't a little too bigger?

@jshencode jshencode Sep 30, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah we have been reserved 10K.
10GB is to make sure each card only run one hll at a time since we cannot estimate hll effectively. We have agreed on 10G at the first place, but i think at some point the code got mixed up, MB changed to bytes. In production we have enough machines to spread out the large contract queries so that we don't see the problem. but in staging we can actually see the problem

@voyager-dw voyager-dw Sep 30, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, we better do 2 observation:

  1. how many real gpu memory allocated when running weekly hll contractor in production/staging
  2. how it impacts production query when hll runs with 10GB reserved. Now as we under-reserve the memory, so other queries may still able to run. After we bump to 10G, other queries in the same machine will be totally blocked even the GPU might have spare space.

)

// batchTransferExecutor defines the type of the functor to transfer a live batch or a archive batch
Expand Down Expand Up @@ -974,7 +974,7 @@ func (qc *AQLQueryContext) calculateMemoryRequirement(memStore memstore.MemStore
//we can track memory usage
//based on table, dimensions, duration to do estimation
if qc.OOPK.IsHLL() {
return hllQueryRequiredMemoryInMB
return hllQueryRequiredMemoryInBytes
}

for _, shardID := range qc.TableScanners[0].Shards {
Expand Down