Skip to content

Fix Backward loop issue on pytorch 2.5#25

Open
ybh1998 wants to merge 1 commit into
aliutkus:masterfrom
ybh1998:master
Open

Fix Backward loop issue on pytorch 2.5#25
ybh1998 wants to merge 1 commit into
aliutkus:masterfrom
ybh1998:master

Conversation

@ybh1998

@ybh1998 ybh1998 commented Feb 24, 2025

Copy link
Copy Markdown

When I tried to back propagate on PyTorch 2.5.1, the program stuck at loss.backward(). After some investigation, I found the following issue:

When I add a breakpoint at

gradients = torch.autograd.grad(

The grad_fn of ctx.saved_tensors[0] is <Interp1dBackward>, causing a loop backward call.

When I add a breakpoint at

ctx.save_for_backward(ynew, *saved_inputs)

The grad_fn of ynew is <AddBackward0>, which means the grad_fn of ctx.saved_tensors[0] is touched between forward and backward. I tried to add .detach() after calling interp1d, but the results were the same.

I suspect that the PyTorch auto_grad engine modifies the grad_fn of ynew in-place after some version. Adding a detach() in the forward function resolves this problem.

@Nikhil0504

Copy link
Copy Markdown

@aliutkus could you please push this since this was the error I was finding as well!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants