Skip to content

Dampening + warnings for preconditioner rank stability #92

@luciaquirke

Description

@luciaquirke
  • Report stable rank (froebenius norm ^2 / spectral norm ^2)

  • Histogram of eigenvalues, potentially using the same code as lovely_tensors provides in the tensor.plt call https://github.com/xl0/lovely-tensors

  • If preconditioning fails due to instability add dampening using the heuristic in the EK-FAC paper. This happens less often with TrackStar due to the down projection but it would be nice to make it generally available for all our methods.

https://arxiv.org/html/2407.21594v1
https://arxiv.org/abs/2308.03296

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions