syzbot


possible deadlock in io_worker_cancel_cb

Status: fixed on 2022/03/08 16:11
Subsystems: fs io-uring
[Documentation on labels]
Reported-by: syzbot+b18b8be69df33a3918e9@syzkaller.appspotmail.com
Fix commit: d800c65c2d4e io-wq: drop wqe lock before creating new worker
First crash: 1035d, last: 1030d
Cause bisection: introduced by (bisect log) :
commit 71a85387546e50b1a37b0fa45dadcae3bfb35cf6
Author: Jens Axboe <axboe@kernel.dk>
Date: Fri Dec 10 15:29:30 2021 +0000

  io-wq: check for wq exit after adding new worker task_work

Crash: possible deadlock in io_worker_cancel_cb (log)
Repro: C syz .config
  
Discussions (3)
Title Replies (including bot) Last reply
[PATCH 5.15 000/177] 5.15.11-rc1 review 183 (183) 2021/12/21 23:14
[PATCH] io-wq: drop wqe lock before creating new worker 1 (1) 2021/12/13 16:46
[syzbot] possible deadlock in io_worker_cancel_cb 1 (3) 2021/12/13 16:40
Last patch testing requests (1)
Created Duration User Patch Repo Result
2021/12/13 16:06 12m axboe@kernel.dk git://git.kernel.dk/linux-block io_uring-5.16 OK

Sample crash report:
============================================
WARNING: possible recursive locking detected
5.16.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
iou-wrk-6468/6471 is trying to acquire lock:
ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_worker_cancel_cb+0xb7/0x210 fs/io-wq.c:187

but task is already holding lock:
ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0xb6/0x140 fs/io-wq.c:700

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&wqe->lock);
  lock(&wqe->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

1 lock held by iou-wrk-6468/6471:
 #0: ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0xb6/0x140 fs/io-wq.c:700

stack backtrace:
CPU: 1 PID: 6471 Comm: iou-wrk-6468 Not tainted 5.16.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1dc/0x2d8 lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2956 [inline]
 check_deadlock kernel/locking/lockdep.c:2999 [inline]
 validate_chain+0x5984/0x8240 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1382/0x2b00 kernel/locking/lockdep.c:5027
 lock_acquire+0x19f/0x4d0 kernel/locking/lockdep.c:5637
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 io_worker_cancel_cb+0xb7/0x210 fs/io-wq.c:187
 io_wq_cancel_tw_create fs/io-wq.c:1220 [inline]
 io_queue_worker_create+0x3cf/0x4c0 fs/io-wq.c:372
 io_wq_worker_sleeping+0xbe/0x140 fs/io-wq.c:701
 sched_submit_work kernel/sched/core.c:6295 [inline]
 schedule+0x67/0x1f0 kernel/sched/core.c:6323
 schedule_timeout+0xac/0x300 kernel/time/timer.c:1857
 wait_woken+0xca/0x1b0 kernel/sched/wait.c:460
 unix_msg_wait_data net/unix/unix_bpf.c:32 [inline]
 unix_bpf_recvmsg+0x7f9/0xe20 net/unix/unix_bpf.c:77
 unix_stream_recvmsg+0x214/0x2c0 net/unix/af_unix.c:2832
 sock_recvmsg_nosec net/socket.c:944 [inline]
 sock_recvmsg net/socket.c:962 [inline]
 sock_read_iter+0x3a7/0x4d0 net/socket.c:1035
 call_read_iter include/linux/fs.h:2156 [inline]
 io_iter_do_read fs/io_uring.c:3501 [inline]
 io_read fs/io_uring.c:3558 [inline]
 io_issue_sqe+0x144c/0x9590 fs/io_uring.c:6671
 io_wq_submit_work+0x2d8/0x790 fs/io_uring.c:6836
 io_worker_handle_work+0x808/0xdd0 fs/io-wq.c:574
 io_wqe_worker+0x395/0x870 fs/io-wq.c:630
 ret_from_fork+0x1f/0x30
 </TASK>

Crashes (9):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/12/12 10:36 upstream a763d5a5abd6 49ca1f59 .config console log report syz C ci-upstream-kasan-gce-smack-root possible deadlock in io_worker_cancel_cb
2021/12/17 14:53 upstream fa36bbe6d43f 44068e19 .config console log report info ci-upstream-kasan-gce-root possible deadlock in io_worker_cancel_cb
2021/12/17 07:45 upstream fa36bbe6d43f 44068e19 .config console log report info ci-upstream-kasan-gce possible deadlock in io_worker_cancel_cb
2021/12/16 19:42 upstream fa36bbe6d43f 8dd6a5e3 .config console log report info ci-upstream-kasan-gce-smack-root possible deadlock in io_worker_cancel_cb
2021/12/15 06:50 upstream 5472f14a3742 f752fb53 .config console log report info ci-upstream-kasan-gce possible deadlock in io_worker_cancel_cb
2021/12/14 13:31 upstream 5472f14a3742 d018dd31 .config console log report info ci-upstream-kasan-gce-root possible deadlock in io_worker_cancel_cb
2021/12/12 19:27 upstream a763d5a5abd6 49ca1f59 .config console log report info ci-upstream-kasan-gce possible deadlock in io_worker_cancel_cb
2021/12/12 07:28 upstream a763d5a5abd6 49ca1f59 .config console log report info ci-upstream-kasan-gce-smack-root possible deadlock in io_worker_cancel_cb
2021/12/16 21:20 upstream fa36bbe6d43f 8dd6a5e3 .config console log report info ci-qemu-upstream-386 possible deadlock in io_worker_cancel_cb
* Struck through repros no longer work on HEAD.