Problem
The DSpark backward loss uses global denominators, but the logged train/loss metric was computed from locally normalized rank losses. Averaging those rank-local losses gives each rank equal weight, regardless of how many supervised tokens or objective terms each rank contributed.
Impact
The logged loss can diverge from the actual global training objective. For example, a rank with one supervised token can affect the logged scalar as much as a rank with many supervised tokens.
Fix
Fixed in PR #30 by logging the globally scaled backward_loss.detach() with dp_mean, so the reported loss matches the global objective used for gradients.
Problem
The DSpark backward loss uses global denominators, but the logged
train/lossmetric was computed from locally normalized rank losses. Averaging those rank-local losses gives each rank equal weight, regardless of how many supervised tokens or objective terms each rank contributed.Impact
The logged loss can diverge from the actual global training objective. For example, a rank with one supervised token can affect the logged scalar as much as a rank with many supervised tokens.
Fix
Fixed in PR #30 by logging the globally scaled
backward_loss.detach()withdp_mean, so the reported loss matches the global objective used for gradients.