bisecting cause commit starting from a763d5a5abd65797aec3dd1bf01fe2ccbec32967 building syzkaller on 49ca1f59e37fcf63dc38a6bd2b60fcc47a0a708e testing commit a763d5a5abd65797aec3dd1bf01fe2ccbec32967 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 01dabffe3e7f9d67a9d994be29cc2572d299f5c95d59c2df4b94cd7d9fab6975 all runs: crashed: possible deadlock in io_worker_cancel_cb testing release v5.15 testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f022453079e48b0d1b235afa1df77c4f350bb0819401752f5af42ca0fa10a066 all runs: OK # git bisect start a763d5a5abd65797aec3dd1bf01fe2ccbec32967 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 7359 revisions left to test after this (roughly 13 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [ff0700f03609b9f0defacd4ce96d9519d721e0a2] Merge tag 'sound-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit ff0700f03609b9f0defacd4ce96d9519d721e0a2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 03e30864857a100b480cbd06b59ff8bb683c6a44087a564d3a46f542711fd4d6 all runs: OK # git bisect good ff0700f03609b9f0defacd4ce96d9519d721e0a2 Bisecting: 3722 revisions left to test after this (roughly 12 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [512b7931ad0561ffe14265f9ff554a3c081b476b] Merge branch 'akpm' (patches from Andrew) testing commit 512b7931ad0561ffe14265f9ff554a3c081b476b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 150f5b3da2380b4629e28d01fbd6c176c23461dd0d94e73859fe5e1b0b32b0fb all runs: OK # git bisect good 512b7931ad0561ffe14265f9ff554a3c081b476b Bisecting: 1845 revisions left to test after this (roughly 11 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [6cbcc7ab2147d721700029a78558dc0ea4207153] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit 6cbcc7ab2147d721700029a78558dc0ea4207153 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c71403d5ef08fc1410cfd6a6d40a5a7dde47a6b3196f221892870b3177982511 all runs: OK # git bisect good 6cbcc7ab2147d721700029a78558dc0ea4207153 Bisecting: 918 revisions left to test after this (roughly 10 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [6b54698aec0b59943f7e8a88151bdf208de990d0] Merge tag 'for-linus-5.16c-rc3-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip testing commit 6b54698aec0b59943f7e8a88151bdf208de990d0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d6b8690ce973b86f97dad2a0498b34767d3c0dd0528009e5142b655cee9fc856 all runs: OK # git bisect good 6b54698aec0b59943f7e8a88151bdf208de990d0 Bisecting: 459 revisions left to test after this (roughly 9 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [56a271be062afb007675e199e1e25b22d28cdd0a] Merge branch 'net-tls-cover-all-ciphers-with-tests' testing commit 56a271be062afb007675e199e1e25b22d28cdd0a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c947e46933300ab3fa5090a233bd9b806b013f4c13471604f5557621be07b168 all runs: OK # git bisect good 56a271be062afb007675e199e1e25b22d28cdd0a Bisecting: 254 revisions left to test after this (roughly 8 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [27698cd2a3c07d124fad6bfc8dd93fd1c5d46879] Merge tag 'mtd/fixes-for-5.16-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux testing commit 27698cd2a3c07d124fad6bfc8dd93fd1c5d46879 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1fbe4737bc7c64157c26633e94cba5ac86e4bac7c0874d37f4df460c26379c70 all runs: OK # git bisect good 27698cd2a3c07d124fad6bfc8dd93fd1c5d46879 Bisecting: 127 revisions left to test after this (roughly 7 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [b8a98b6bf66ae35361e987333233d07241642909] Merge tag 'pci-v5.16-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci testing commit b8a98b6bf66ae35361e987333233d07241642909 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 11702b62ca9be107156c5a0f0cd81b524643e5416571e62df03b1bb50d440c12 all runs: OK # git bisect good b8a98b6bf66ae35361e987333233d07241642909 Bisecting: 68 revisions left to test after this (roughly 6 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [9937e8daab29d9e20de6b7bc56c76db7a4eeda69] perf python: Fix NULL vs IS_ERR_OR_NULL() checking testing commit 9937e8daab29d9e20de6b7bc56c76db7a4eeda69 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a53ad9d52b393896d2f0b6b838be9a7d0ed08da33fc7fd1c1d8a0e6c1a0c55f7 all runs: OK # git bisect good 9937e8daab29d9e20de6b7bc56c76db7a4eeda69 Bisecting: 32 revisions left to test after this (roughly 5 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [2acdaf59e5958a96500b035830384168dbbe099e] Merge tag 'clk-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux testing commit 2acdaf59e5958a96500b035830384168dbbe099e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 697e95970a983194b239a159f3e9cd443a1357d2e38d6ad0926313f63e23e372 all runs: OK # git bisect good 2acdaf59e5958a96500b035830384168dbbe099e Bisecting: 18 revisions left to test after this (roughly 4 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [5eff363838654790f67f4bd564c5782967f67bcc] Revert "mtd_blkdevs: don't scan partitions for plain mtdblock" testing commit 5eff363838654790f67f4bd564c5782967f67bcc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bf5340366d599e7716445ec7ead520fab24db49ba3ba7c790736d61f6622a444 all runs: OK # git bisect good 5eff363838654790f67f4bd564c5782967f67bcc Bisecting: 9 revisions left to test after this (roughly 3 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [bbdff6d583be718935b613ab2a966cddaadf661f] Merge tag 'perf-tools-fixes-for-v5.16-2021-12-11' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux testing commit bbdff6d583be718935b613ab2a966cddaadf661f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 72a5b0da48a3e97002b8d084f5a4437ea0f1559b66d8b71ca44ad1010bd5b3cf run #0: crashed: possible deadlock in io_worker_cancel_cb run #1: crashed: possible deadlock in io_worker_cancel_cb run #2: crashed: possible deadlock in io_worker_cancel_cb run #3: crashed: possible deadlock in io_worker_cancel_cb run #4: crashed: possible deadlock in io_worker_cancel_cb run #5: crashed: possible deadlock in io_worker_cancel_cb run #6: crashed: possible deadlock in io_worker_cancel_cb run #7: crashed: possible deadlock in io_worker_cancel_cb run #8: OK run #9: OK # git bisect bad bbdff6d583be718935b613ab2a966cddaadf661f Bisecting: 5 revisions left to test after this (roughly 2 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [71a85387546e50b1a37b0fa45dadcae3bfb35cf6] io-wq: check for wq exit after adding new worker task_work testing commit 71a85387546e50b1a37b0fa45dadcae3bfb35cf6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2a9b822c3937b6083baa4e52fcf491c5db746908321399efb4ded93d6fbce88c run #0: crashed: possible deadlock in io_worker_cancel_cb run #1: crashed: possible deadlock in io_worker_cancel_cb run #2: crashed: possible deadlock in io_worker_cancel_cb run #3: crashed: possible deadlock in io_worker_cancel_cb run #4: crashed: possible deadlock in io_worker_cancel_cb run #5: crashed: possible deadlock in io_worker_cancel_cb run #6: crashed: possible deadlock in io_worker_cancel_cb run #7: crashed: possible deadlock in io_worker_cancel_cb run #8: crashed: possible deadlock in io_worker_cancel_cb run #9: OK # git bisect bad 71a85387546e50b1a37b0fa45dadcae3bfb35cf6 Bisecting: 0 revisions left to test after this (roughly 1 step) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [78a780602075d8b00c98070fa26e389b3b3efa72] io_uring: ensure task_work gets run as part of cancelations testing commit 78a780602075d8b00c98070fa26e389b3b3efa72 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bb1d5bdf59df25ef0124c52a40512d557ffe5937c4d6f792329b001a63552a37 all runs: OK # git bisect good 78a780602075d8b00c98070fa26e389b3b3efa72 warning: unable to access '/syzkaller/.config/git/attributes': Permission denied 71a85387546e50b1a37b0fa45dadcae3bfb35cf6 is the first bad commit commit 71a85387546e50b1a37b0fa45dadcae3bfb35cf6 Author: Jens Axboe Date: Fri Dec 10 08:29:30 2021 -0700 io-wq: check for wq exit after adding new worker task_work We check IO_WQ_BIT_EXIT before attempting to create a new worker, and wq exit cancels pending work if we have any. But it's possible to have a race between the two, where creation checks exit finding it not set, but we're in the process of exiting. The exit side will cancel pending creation task_work, but there's a gap where we add task_work after we've canceled existing creations at exit time. Fix this by checking the EXIT bit post adding the creation task_work. If it's set, run the same cancelation that exit does. Reported-and-tested-by: syzbot+b60c982cb0efc5e05a47@syzkaller.appspotmail.com Reviewed-by: Hao Xu Signed-off-by: Jens Axboe fs/io-wq.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) culprit signature: 2a9b822c3937b6083baa4e52fcf491c5db746908321399efb4ded93d6fbce88c parent signature: bb1d5bdf59df25ef0124c52a40512d557ffe5937c4d6f792329b001a63552a37 revisions tested: 15, total time: 2h54m50.844007543s (build: 1h34m35.661410496s, test: 1h18m45.563555944s) first bad commit: 71a85387546e50b1a37b0fa45dadcae3bfb35cf6 io-wq: check for wq exit after adding new worker task_work recipients (to): ["axboe@kernel.dk" "haoxu@linux.alibaba.com" "syzbot+b60c982cb0efc5e05a47@syzkaller.appspotmail.com"] recipients (cc): [] crash: possible deadlock in io_worker_cancel_cb ============================================ WARNING: possible recursive locking detected 5.16.0-rc1-syzkaller #0 Not tainted -------------------------------------------- iou-wrk-16647/16648 is trying to acquire lock: ffff88807e1a1818 (&wqe->lock){+.+.}-{2:2}, at: io_worker_cancel_cb+0xe2/0x1d0 fs/io-wq.c:187 but task is already holding lock: ffff88807e1a1818 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0x97/0xf0 fs/io-wq.c:700 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&wqe->lock); lock(&wqe->lock); *** DEADLOCK *** May be due to missing lock nesting notation 1 lock held by iou-wrk-16647/16648: #0: ffff88807e1a1818 (&wqe->lock){+.+.}-{2:2}, at: io_wq_worker_sleeping+0x97/0xf0 fs/io-wq.c:700 stack backtrace: CPU: 0 PID: 16648 Comm: iou-wrk-16647 Not tainted 5.16.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 print_deadlock_bug kernel/locking/lockdep.c:2956 [inline] check_deadlock kernel/locking/lockdep.c:2999 [inline] validate_chain kernel/locking/lockdep.c:3788 [inline] __lock_acquire.cold+0x229/0x3ab kernel/locking/lockdep.c:5027 lock_acquire kernel/locking/lockdep.c:5637 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5602 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154 io_worker_cancel_cb+0xe2/0x1d0 fs/io-wq.c:187 io_wq_cancel_tw_create fs/io-wq.c:1220 [inline] io_queue_worker_create+0x42a/0x4b0 fs/io-wq.c:372 io_wq_worker_sleeping+0x9f/0xf0 fs/io-wq.c:701 sched_submit_work kernel/sched/core.c:6295 [inline] schedule+0x1d0/0x260 kernel/sched/core.c:6323 schedule_timeout+0x19d/0x250 kernel/time/timer.c:1857 wait_woken+0x18b/0x1f0 kernel/sched/wait.c:453 unix_msg_wait_data net/unix/unix_bpf.c:32 [inline] unix_bpf_recvmsg+0x91a/0xc20 net/unix/unix_bpf.c:77 unix_stream_recvmsg+0x113/0x1e0 net/unix/af_unix.c:2832 sock_recvmsg_nosec net/socket.c:944 [inline] sock_recvmsg net/socket.c:962 [inline] sock_recvmsg net/socket.c:958 [inline] sock_read_iter+0x2b4/0x3f0 net/socket.c:1035 call_read_iter include/linux/fs.h:2156 [inline] io_iter_do_read fs/io_uring.c:3501 [inline] io_read+0x2ab/0xea0 fs/io_uring.c:3558 io_issue_sqe+0x68c/0x65d0 fs/io_uring.c:6671 io_wq_submit_work+0x154/0x4a0 fs/io_uring.c:6836 io_worker_handle_work+0x60b/0x1690 fs/io-wq.c:574 io_wqe_worker+0x479/0xb10 fs/io-wq.c:630 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295