Skip to content

fuse: fix fuse_uring_abort when uring is already cancelled #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hbirth merged 2 commits into from
Apr 9, 2026
+114 −8
Merged

fuse: fix fuse_uring_abort when uring is already cancelled #138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f4a68d
fuse: debug print requests when we hang in fuse_wait_aborted()
hbirth Mar 26, 2026
1f7eb8f
fuse: fix io_uring connection abort leaving requests stuck
hbirth Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 111 additions & 2 deletions fs/fuse/dev.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,8 @@ static void request_wait_answer(struct fuse_req *req)
* Either request is already in userspace, or it was forced.
* Wait it out.
*/
wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags));
wait_event(req->waitq,
test_bit(FR_FINISHED, &req->flags));
}

static void __fuse_request_send(struct fuse_req *req)
Expand Down Expand Up @@ -2260,11 +2261,119 @@ void fuse_abort_conn(struct fuse_conn *fc)
}
EXPORT_SYMBOL_GPL(fuse_abort_conn);

static void fuse_debug_print_outstanding_reqs(struct fuse_conn *fc)
{
struct fuse_dev *fud;

pr_warn("FUSE: fuse_wait_aborted: num_waiting=%d (should be 0)\n",
atomic_read(&fc->num_waiting));

#ifdef CONFIG_FUSE_IO_URING
/* Print io_uring state if enabled */
if (fc->ring) {
struct fuse_ring *ring = fc->ring;

pr_warn("FUSE: io_uring enabled - queue_refs=%d ready=%d\n",
atomic_read(&ring->queue_refs), ring->ready);
}
#endif

/* Print all outstanding requests - lockless for debug */
list_for_each_entry(fud, &fc->devices, entry) {
struct fuse_pqueue *fpq = &fud->pq;
struct fuse_req *req;
int i;

/* Print all requests on fpq->io */
if (!list_empty(&fpq->io)) {
pr_warn("FUSE: Outstanding requests on fpq->io:\n");
list_for_each_entry(req, &fpq->io, list) {
#ifdef CONFIG_FUSE_IO_URING
if (test_bit(FR_URING, &req->flags) &&
req->ring_entry) {
struct fuse_ring_ent *ent = req->ring_entry;

pr_warn(" req %p: opcode=%u unique=%llu flags=0x%lx FR_WAITING=%d FR_LOCKED=%d FR_FORCE=%d FR_ABORTED=%d FR_URING=%d ring_ent=%p state=%d\n",
req, req->in.h.opcode,
req->in.h.unique, req->flags,
test_bit(FR_WAITING, &req->flags),
test_bit(FR_LOCKED, &req->flags),
test_bit(FR_FORCE, &req->flags),
test_bit(FR_ABORTED, &req->flags),
test_bit(FR_URING, &req->flags),
ent, ent->state);
} else {
#endif
pr_warn(" req %p: opcode=%u unique=%llu flags=0x%lx FR_WAITING=%d FR_LOCKED=%d FR_FORCE=%d FR_ABORTED=%d FR_URING=%d\n",
req, req->in.h.opcode,
req->in.h.unique, req->flags,
test_bit(FR_WAITING, &req->flags),
test_bit(FR_LOCKED, &req->flags),
test_bit(FR_FORCE, &req->flags),
test_bit(FR_ABORTED, &req->flags),
test_bit(FR_URING, &req->flags));
#ifdef CONFIG_FUSE_IO_URING
}
#endif
}
}

/* Print all requests on fpq->processing */
for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) {
if (list_empty(&fpq->processing[i]))
continue;

pr_warn("FUSE: Outstanding requests on fpq->processing[%d]:\n",
i);
list_for_each_entry(req, &fpq->processing[i], list) {
#ifdef CONFIG_FUSE_IO_URING
if (test_bit(FR_URING, &req->flags) &&
req->ring_entry) {
struct fuse_ring_ent *ent = req->ring_entry;

pr_warn(" req %p: opcode=%u unique=%llu flags=0x%lx FR_WAITING=%d FR_LOCKED=%d FR_FORCE=%d FR_ABORTED=%d FR_URING=%d ring_ent=%p state=%d\n",
req, req->in.h.opcode,
req->in.h.unique, req->flags,
test_bit(FR_WAITING, &req->flags),
test_bit(FR_LOCKED, &req->flags),
test_bit(FR_FORCE, &req->flags),
test_bit(FR_ABORTED, &req->flags),
test_bit(FR_URING, &req->flags),
ent, ent->state);
} else {
#endif
pr_warn(" req %p: opcode=%u unique=%llu flags=0x%lx FR_WAITING=%d FR_LOCKED=%d FR_FORCE=%d FR_ABORTED=%d FR_URING=%d\n",
req, req->in.h.opcode,
req->in.h.unique, req->flags,
test_bit(FR_WAITING, &req->flags),
test_bit(FR_LOCKED, &req->flags),
test_bit(FR_FORCE, &req->flags),
test_bit(FR_ABORTED, &req->flags),
test_bit(FR_URING, &req->flags));
#ifdef CONFIG_FUSE_IO_URING
}
#endif
}
}
}
}

void fuse_wait_aborted(struct fuse_conn *fc)
{
unsigned int timeout = 20;

/* matches implicit memory barrier in fuse_drop_waiting() */
smp_mb();
wait_event(fc->blocked_waitq, atomic_read(&fc->num_waiting) == 0);

wait:
wait_event_timeout(fc->blocked_waitq, atomic_read(&fc->num_waiting) == 0, HZ * timeout);

/* Debug: print info if we're waiting */
if (atomic_read(&fc->num_waiting) > 0) {
fuse_debug_print_outstanding_reqs(fc);
timeout *= 3;
goto wait;
}

fuse_uring_wait_stopped_queues(fc);
}
Expand Down
3 changes: 1 addition & 2 deletions fs/fuse/dev_uring.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,10 @@ void fuse_uring_flush_bg(struct fuse_conn *fc)
if (!queue)
Copy link
Copy Markdown
Collaborator

@bsbernd bsbernd Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't see why two runs are required. With one single lock, let's say we have two competing tasks, task-1 tries to add the request, task-2 tries to abort the queue.

Scenario1:
task-1 wins and is in the process to add to the queue -> takes the lock, task-2 (abort) is blocked. Once task-2 continues, It blocks the queue and flushes requests. After that another task cannot add new requests, because queue->stopped` is set.

Scenario2:
task-2 (abort) wins, takes the lock, set queue->lock and flushes the queue. Once it releases the lock, task-1 wakes up, sees that the queue is stopped and fails the request with -ENOTCONN.

Copy link
Copy Markdown
Collaborator Author

@hbirth hbirth Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it really that bad? this only gets called when the connection is destroyed.

Copy link
Copy Markdown
Collaborator

@bsbernd bsbernd Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somehow my previous comment disappeared, I had added that condition during development because issues came up. Just checked the code - it should be safe in current code.

continue;

queue->stopped = true;

WARN_ON_ONCE(ring->fc->max_background != UINT_MAX);
spin_lock(&queue->lock);
spin_lock(&fc->bg_lock);
queue->stopped = true;
fuse_uring_flush_queue_bg(queue);
spin_unlock(&fc->bg_lock);
spin_unlock(&queue->lock);
Expand Down
6 changes: 2 additions & 4 deletions fs/fuse/dev_uring_i.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,10 +178,8 @@ static inline void fuse_uring_abort(struct fuse_conn *fc)
if (ring == NULL)
return;

if (atomic_read(&ring->queue_refs) > 0) {
fuse_uring_flush_bg(fc);
fuse_uring_stop_queues(ring);
}
fuse_uring_flush_bg(fc);
fuse_uring_stop_queues(ring);
Copy link
Copy Markdown
Collaborator

@bsbernd bsbernd Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is not right, queue_refs == 0 means there are no ring_ent on any queue - an impossible condition. And with queue_refs == 0, ring->ready will not be true.

Copy link
Copy Markdown
Collaborator Author

@hbirth hbirth Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is that an impossible condition? I have seen it many times in debug prints

Copy link
Copy Markdown
Collaborator

@bsbernd bsbernd Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At startup time - > ring gets created, first queue gets created, then a ring-entry gets created (increases queue_refs), then ring->ready is set.

teardown:

fuse_uring_teardown_entries
fuse_uring_stop_list_entries() decreases queue_refs

Btw, if there would be a problem with FRRS_FUSE_REQ and FRRS_COMMIT, queue refs would never go to 0 - exactly here.

Anyway, fuse_uring_teardown_entries() is called by fuse_uring_stop_queues, which is supposed to be called in fuse_uring_abort() after fuse_uring_abort_end_requests(). Which means it is after queue->stopped = true

Copy link
Copy Markdown
Collaborator Author

@hbirth hbirth Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes ... that's why I had done those above (maybe in a bit of dispair) ... we have queue_refs == 0, and queue ready, and still requests active that will never terminate.

Copy link
Copy Markdown
Collaborator Author

@hbirth hbirth Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, all I want here is to call fuse_flush_bg() and fuse_uring_stop_queues() no matter what. What is wrong with that?

}

static inline void fuse_uring_wait_stopped_queues(struct fuse_conn *fc)
Expand Down
Loading