Question about boundary

[ybjiaang]

I don't quite understand how the model enforces boundary probability to be close to 1 or 0. It is possible for the model to be less confident during the EMA phase so that it can accumulate more information from previous steps? I have done some training and observe that boundaries have probabilities close to 0.5. Is it a data issue or there's some trick to let the model learn more confident probabilities? I am doing one layer hnet by the way.

[brwa-cartesia]

it might be data-dependent, we've observed probabilities farther from 0/1 on other modalities (e.g. DNA). one thing you can try is removing the STE (i.e. just use upsample = c * z instead of upsample = ste(c) * z). empirically, this increases the router confidence. it is also possible the model needs more training, we typically observe the probability dropping to ~0.53 and then going back up. our hypothesis is that as the main network produces more high quality outputs, the EMA learns to become more confident.