fix(training): retry with backoff on 429 when creating RLOR trainer job

[hershalb]

Summary

• Adds exponential backoff retry (up to 5 attempts, 10s–120s delays with jitter) around rlor_mgr.create() in create_trainer_job() so that HTTP 429 (Too Many Requests) errors during RLOR trainer creation don't immediately fail the entire SFT job.
• Non-retryable errors (400, 401, etc.) still fail immediately.
• Retry count configurable via FW_TRAINER_CREATE_MAX_RETRIES env var (default: 5).

Context

accounts/youfychenbc5000-70d70d/supervisedFineTuningJobs/prcvvafl failed because the trainer creation POST got a 429 while B300 capacity was occupied by another job. The entire SFT job died instead of waiting and retrying.

Test plan

• Added unit tests: 429 retried then succeeds, 429 exhausts retries, non-retryable error not retried, status callback emits retry messages
• CI passes

Made with Cursor