possible deadlock in io_worker_cancel

possible deadlock in io_worker_cancel_cb
Status: upstream: reported C repro on 2021/12/13 06:08
Reported-by: syzbot+b18b8be69df33a3918e9@syzkaller.appspotmail.com
Fix commit: d800c65c2d4e io-wq: drop wqe lock before creating new worker
Patched on: [ci-qemu-upstream ci-qemu-upstream-386 ci-qemu2-arm32 ci-qemu2-arm64 ci-qemu2-arm64-compat ci-qemu2-arm64-mte ci-upstream-bpf-kasan-gce ci-upstream-bpf-next-kasan-gce ci-upstream-gce-leak ci-upstream-kasan-gce ci-upstream-kasan-gce-386 ci-upstream-kasan-gce-root ci-upstream-kasan-gce-selinux-root ci-upstream-kasan-gce-smack-root ci-upstream-linux-next-kasan-gce-root ci-upstream-net-kasan-gce ci-upstream-net-this-kasan-gce ci2-upstream-kcsan-gce ci2-upstream-usb], missing on: [ci-qemu2-riscv64 ci-upstream-kmsan-gce ci-upstream-kmsan-gce-386]
First crash: 46d, last: 41d

Cause bisection: introduced by (bisect log) :
commit 71a85387546e50b1a37b0fa45dadcae3bfb35cf6
Author: Jens Axboe <axboe@kernel.dk>
Date: Fri Dec 10 15:29:30 2021 +0000

io-wq: check for wq exit after adding new worker task_work

Crash: possible deadlock in io_worker_cancel_cb (log)
Repro: C syz .config

Patch testing requests:
Created	Duration	User	Patch	Repo	Result
2021/12/13 16:06	12m	axboe@kernel.dk		git://git.kernel.dk/linux-block io_uring-5.16	OK

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.16.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
iou-wrk-6468/6471 is trying to acquire lock:
ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_worker_cancel_cb+0xb7/0x210 fs/io-wq.c:187

but task is already holding lock:
ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0xb6/0x140 fs/io-wq.c:700

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&wqe->lock);
  lock(&wqe->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

1 lock held by iou-wrk-6468/6471:
 #0: ffff88801aa98018 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0xb6/0x140 fs/io-wq.c:700

stack backtrace:
CPU: 1 PID: 6471 Comm: iou-wrk-6468 Not tainted 5.16.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1dc/0x2d8 lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2956 [inline]
 check_deadlock kernel/locking/lockdep.c:2999 [inline]
 validate_chain+0x5984/0x8240 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1382/0x2b00 kernel/locking/lockdep.c:5027
 lock_acquire+0x19f/0x4d0 kernel/locking/lockdep.c:5637
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 io_worker_cancel_cb+0xb7/0x210 fs/io-wq.c:187
 io_wq_cancel_tw_create fs/io-wq.c:1220 [inline]
 io_queue_worker_create+0x3cf/0x4c0 fs/io-wq.c:372
 io_wq_worker_sleeping+0xbe/0x140 fs/io-wq.c:701
 sched_submit_work kernel/sched/core.c:6295 [inline]
 schedule+0x67/0x1f0 kernel/sched/core.c:6323
 schedule_timeout+0xac/0x300 kernel/time/timer.c:1857
 wait_woken+0xca/0x1b0 kernel/sched/wait.c:460
 unix_msg_wait_data net/unix/unix_bpf.c:32 [inline]
 unix_bpf_recvmsg+0x7f9/0xe20 net/unix/unix_bpf.c:77
 unix_stream_recvmsg+0x214/0x2c0 net/unix/af_unix.c:2832
 sock_recvmsg_nosec net/socket.c:944 [inline]
 sock_recvmsg net/socket.c:962 [inline]
 sock_read_iter+0x3a7/0x4d0 net/socket.c:1035
 call_read_iter include/linux/fs.h:2156 [inline]
 io_iter_do_read fs/io_uring.c:3501 [inline]
 io_read fs/io_uring.c:3558 [inline]
 io_issue_sqe+0x144c/0x9590 fs/io_uring.c:6671
 io_wq_submit_work+0x2d8/0x790 fs/io_uring.c:6836
 io_worker_handle_work+0x808/0xdd0 fs/io-wq.c:574
 io_wqe_worker+0x395/0x870 fs/io-wq.c:630
 ret_from_fork+0x1f/0x30
 </TASK>

Crashes (9):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce-smack-root	2021/12/12 10:36	upstream	a763d5a5abd6	49ca1f59	.config	log	report	syz	C		possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce-root	2021/12/17 14:53	upstream	fa36bbe6d43f	44068e19	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce	2021/12/17 07:45	upstream	fa36bbe6d43f	44068e19	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce-smack-root	2021/12/16 19:42	upstream	fa36bbe6d43f	8dd6a5e3	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce	2021/12/15 06:50	upstream	5472f14a3742	f752fb53	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce-root	2021/12/14 13:31	upstream	5472f14a3742	d018dd31	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce	2021/12/12 19:27	upstream	a763d5a5abd6	49ca1f59	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-upstream-kasan-gce-smack-root	2021/12/12 07:28	upstream	a763d5a5abd6	49ca1f59	.config	log	report			info	possible deadlock in io_worker_cancel_cb
ci-qemu-upstream-386	2021/12/16 21:20	upstream	fa36bbe6d43f	8dd6a5e3	.config	log	report			info	possible deadlock in io_worker_cancel_cb