Hi authors @BinLee26 ,
Thank you for open-sourcing the InterAgent project. I am currently studying the architecture and have a question regarding the design choice of the Transformer Block, especially the ordering of different conditioning modules.
From my understanding, within each Transformer Block, the model performs multiple condition fusions. The input contains:
- motion1 feature
- motion2 feature
- proprioception recent
- proprioception faraway
- exteroception recent
- exteroception faraway
The processing of motion1 feature can be roughly summarized as:
motion1_feature_1 = self_attention(motion1_feature)
motion1_feature_2 = cross_attention(motion1_feature_1, proprioception_recent)
motion1_feature_3 = cross_attention(motion1_feature_2, proprioception_faraway)
motion1_feature_4 = cross_attention(motion1_feature_3, exteroception_recent)
motion1_feature_5 = cross_attention(motion1_feature_4, exteroception_faraway)
motion1_feature_6 = cross_attention(motion1_feature, motion2_feature) # NOTE this is original motion1_feature rather than motion_feature_5
motion1_feature_final = FFN(motion1_feature_5 + motion1_feature_6)
Therefore, the architecture seems to contain two parallel branches:
1. A self-motion/context processing branch, where motion1 progressively integrates proprioceptive and exteroceptive information.
2. A motion interaction branch, where motion1 directly attends to motion2 using the original motion1 feature.
The outputs of these two branches are then fused before the FFN. However, compared with InterGen, where the conditioning and interaction fusion appear to be more sequential (i.e., the output of one attention module is used as the input of the next attention module), InterAgent adopts this parallel design.
I am wondering about the motivation behind this design choice. Are there any empirical findings or ablation studies comparing parallel fusion vs. sequential fusion?
I would appreciate any insights into the reasoning behind this module ordering design.
Thank you very much!
Hi authors @BinLee26 ,
Thank you for open-sourcing the InterAgent project. I am currently studying the architecture and have a question regarding the design choice of the Transformer Block, especially the ordering of different conditioning modules.
From my understanding, within each Transformer Block, the model performs multiple condition fusions. The input contains:
- motion1 feature
- motion2 feature
- proprioception recent
- proprioception faraway
- exteroception recent
- exteroception faraway
The processing of motion1 feature can be roughly summarized as:
motion1_feature_1 = self_attention(motion1_feature)
motion1_feature_2 = cross_attention(motion1_feature_1, proprioception_recent)
motion1_feature_3 = cross_attention(motion1_feature_2, proprioception_faraway)
motion1_feature_4 = cross_attention(motion1_feature_3, exteroception_recent)
motion1_feature_5 = cross_attention(motion1_feature_4, exteroception_faraway)
motion1_feature_6 = cross_attention(motion1_feature, motion2_feature) # NOTE this is original motion1_feature rather than motion_feature_5
motion1_feature_final = FFN(motion1_feature_5 + motion1_feature_6)
Therefore, the architecture seems to contain two parallel branches:
1. A self-motion/context processing branch, where motion1 progressively integrates proprioceptive and exteroceptive information.
2. A motion interaction branch, where motion1 directly attends to motion2 using the original motion1 feature.
The outputs of these two branches are then fused before the FFN. However, compared with InterGen, where the conditioning and interaction fusion appear to be more sequential (i.e., the output of one attention module is used as the input of the next attention module), InterAgent adopts this parallel design.
I am wondering about the motivation behind this design choice. Are there any empirical findings or ablation studies comparing parallel fusion vs. sequential fusion?
I would appreciate any insights into the reasoning behind this module ordering design.
Thank you very much!