Documenting issues that generic trace splitter is not able to solve yet.
- Trace has an annotation family for every model layer. In this case, since annotation's take precedence over tree traversal, the annotation family will be chosen as the splitting group, creating a split for every model layer (too fine a granularity).
- Trace has irregular pattern in events.
Ex:
get_next_batch (gpu_path with very few kernels)
run_batch (main gpu path)
get_next_batch
run_batch
get_next_batch
idle_step (non gpu path)
get_next_batch
run_batch
idle_step breaks pattern. Ideally, the trace splitter should have some tolerance for this and still consider this to be a relevant pattern.
3. Trace contains one major root with most of the GPU work and two minor roots with some GPU work. The call stack of the major root is a superset of the call stack of the two minor roots - it just has more wrapper nodes at the start. Sibling root detection would ideally traverse down each root and find a recurring pattern of events connecting the roots, but because the two minor roots start at a lower level than the main root, sibling root detection fails. Need some way of recognizing that the two minor roots start at a lower level in call stack than the major root.
4. Split window extension extends into next iters cpu ops.
Documenting issues that generic trace splitter is not able to solve yet.
Ex:
idle_step breaks pattern. Ideally, the trace splitter should have some tolerance for this and still consider this to be a relevant pattern.
3. Trace contains one major root with most of the GPU work and two minor roots with some GPU work. The call stack of the major root is a superset of the call stack of the two minor roots - it just has more wrapper nodes at the start. Sibling root detection would ideally traverse down each root and find a recurring pattern of events connecting the roots, but because the two minor roots start at a lower level than the main root, sibling root detection fails. Need some way of recognizing that the two minor roots start at a lower level in call stack than the major root.
4. Split window extension extends into next iters cpu ops.