Skip to content

add IPO preference optimization - #457

Open
yuetongz417 wants to merge 2 commits into
inclusionAI:mainfrom
yuetongz417:feat/add-ipo
Open

add IPO preference optimization#457
yuetongz417 wants to merge 2 commits into
inclusionAI:mainfrom
yuetongz417:feat/add-ipo

Conversation

@yuetongz417

Copy link
Copy Markdown

Summary

Adds Identity Preference Optimization (IPO) as a built-in offline preference-training algorithm.

Changes

  • Add a sequence-level IPO loss following Eq. 17 of the IPO paper using summed response log-probabilities.
  • Reuse the existing DPO preference trainer and reference-model scoring workflow.
  • Add IPOTrainerConfig and the --ipo-beta option.
  • Register ipo in the algorithm registry.
  • Add IPO reference-checkpoint resolution.
  • Add CPU tests for the loss, configuration, registry integration, and model-reference resolution.
  • Update CLI help and documentation.

The implementation currently uses sequence-summed log-probabilities rather than length-normalized averages.

@xsuler

xsuler commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

@yuetongz417 Hi, it would be better if you add an example in the examples folder and add an evaluation report of this implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants