aggregation model refactoring

[iejMac]

move normalization + logit_scale into each model or maybe perhaps some wrapper that does it for all models, think about this more

[iejMac]

open_clip has it nice that their model is kind of already defined by the paper, we still need to find our model so we need to be pretty flexible

[iejMac]

I think maybe a wrapper would be nice that would have features like logit_scale, normalization, etc. and you parameterize it with some sort of 'aggregation module' that does the actual aggregation

[iejMac]

class VideoCLIP
def __init__(self, aggregator, pre_norm)
    self.aggregator = aggregator
    self.pre_norm = nn.Identity() if not pre_norm else nn.Normalize())
def forward(self, x, normalize=True)
    x = self.pre_norm(x)
    emb = self.aggregator(x)
    emb = F.normalize(emb) if normalize else emb
    return emb

[iejMac]

idk maybe something like that