bisecting cause commit starting from 634de1db0e9bbeb90d7b01020e59ec3dab4d38a1 building syzkaller on c334415ef5e147ea13e0f70dce3f665dea3e4de9 testing commit 634de1db0e9bbeb90d7b01020e59ec3dab4d38a1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 55b46a88e5e25fca2ec274994d745e70accfdca0a3b0faa00db000996e93581e all runs: crashed: possible deadlock in io_disarm_next testing release v5.17 testing commit f443e374ae131c168a065ea1748feac6b2e76613 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4ce70306d16aae853a35b5e70f632603fbf10828f61362e1889e25122f09a420 all runs: OK # git bisect start 634de1db0e9bbeb90d7b01020e59ec3dab4d38a1 f443e374ae131c168a065ea1748feac6b2e76613 Bisecting: 9804 revisions left to test after this (roughly 13 steps) [9a8b3d5f71eb74b1b95927bd320b1070866a119a] Merge tag 'mips_5.18' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit 9a8b3d5f71eb74b1b95927bd320b1070866a119a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1b1d33fba97e535c772412ac2892e55d1c1f453322c10b259e94eebd3846af2b all runs: OK # git bisect good 9a8b3d5f71eb74b1b95927bd320b1070866a119a Bisecting: 4899 revisions left to test after this (roughly 12 steps) [9cff4a31dd1fbcfb5082dd59c5293aa7a045f3ec] Merge branch 'for-next-next-v5.18-20220412' into for-next-20220412 testing commit 9cff4a31dd1fbcfb5082dd59c5293aa7a045f3ec compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 141f49b5b4a41b743b2587ae2ae867069e2c7c43cf9c6f56696e2b8aba118467 all runs: OK # git bisect good 9cff4a31dd1fbcfb5082dd59c5293aa7a045f3ec Bisecting: 2598 revisions left to test after this (roughly 11 steps) [14f495d442d8007dc4b03795ad46e1fcba82858c] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git testing commit 14f495d442d8007dc4b03795ad46e1fcba82858c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3a1ee0bfc1e43b51fa85c2423dc63e11cf742931014c9d5aa910a2d5ff0a60e2 all runs: OK # git bisect good 14f495d442d8007dc4b03795ad46e1fcba82858c Bisecting: 1370 revisions left to test after this (roughly 10 steps) [5becc74463a244b19b009f2ec46dbda486c48671] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git testing commit 5becc74463a244b19b009f2ec46dbda486c48671 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 317ec64901f57d996a1e029dea378339983ca12a56addafc8797c31ed482fb55 all runs: crashed: possible deadlock in io_disarm_next # git bisect bad 5becc74463a244b19b009f2ec46dbda486c48671 Bisecting: 677 revisions left to test after this (roughly 9 steps) [24a567e1aa53d4fa56e52141d48e0578ab3be861] Merge branch 'for-linux-next' of git://anongit.freedesktop.org/drm/drm-misc testing commit 24a567e1aa53d4fa56e52141d48e0578ab3be861 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4b22f0cd1f84938ddc6b5a681e763341df886b8551ea13d19a1c715e42018e38 all runs: OK # git bisect good 24a567e1aa53d4fa56e52141d48e0578ab3be861 Bisecting: 319 revisions left to test after this (roughly 8 steps) [a0cc7c3cd14d0d44d83b9157c999f4fc0765813c] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git testing commit a0cc7c3cd14d0d44d83b9157c999f4fc0765813c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4aac4bed851963a956154401bf51259de57d42a20ca09bfbaa092b3ceac87f1e all runs: OK # git bisect good a0cc7c3cd14d0d44d83b9157c999f4fc0765813c Bisecting: 153 revisions left to test after this (roughly 7 steps) [0bd9251def4ecc81fd0e435bcd116235a78baec0] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git testing commit 0bd9251def4ecc81fd0e435bcd116235a78baec0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8dc818e0c685fe060862b123b21e6d58a56bd6fcafb36ead02737820cd8f50d3 all runs: crashed: possible deadlock in io_disarm_next # git bisect bad 0bd9251def4ecc81fd0e435bcd116235a78baec0 Bisecting: 66 revisions left to test after this (roughly 6 steps) [dbfdd71836605da5ba8ff879c2a949a3f264b2f1] Merge branch 'for-5.19/io_uring' into for-next testing commit dbfdd71836605da5ba8ff879c2a949a3f264b2f1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 79ad33b6bef5e0d5912357bded7693e12bb7ac49fcd1a611af673aa5ec6555ad all runs: crashed: possible deadlock in io_disarm_next # git bisect bad dbfdd71836605da5ba8ff879c2a949a3f264b2f1 Bisecting: 49 revisions left to test after this (roughly 6 steps) [90c6c291453922362ae026f1049843368111bdfe] drdb: Switch to kvfree_rcu() API testing commit 90c6c291453922362ae026f1049843368111bdfe compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5b617cf1e2160224cb92747c9f4514542543f37350a3b50a007d4525b4743ffb all runs: OK # git bisect good 90c6c291453922362ae026f1049843368111bdfe Bisecting: 24 revisions left to test after this (roughly 5 steps) [40d8dfba8df01ece25da3751202524b2948a7850] io_uring: helper for empty req cache checks testing commit 40d8dfba8df01ece25da3751202524b2948a7850 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 85553bcf9a8a3613af525208b324719809d60b3dc207e1f24650975c9ab0c296 all runs: OK # git bisect good 40d8dfba8df01ece25da3751202524b2948a7850 Bisecting: 12 revisions left to test after this (roughly 4 steps) [ceba3567006f5e932521b93d327d8626a0078be1] io_uring: refactor io_queue_sqe() testing commit ceba3567006f5e932521b93d327d8626a0078be1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7b3924619ef61687a5bf92d62f8f73b8ff6afa0ea264331a8bffcf90786f1811 all runs: crashed: possible deadlock in io_disarm_next # git bisect bad ceba3567006f5e932521b93d327d8626a0078be1 Bisecting: 5 revisions left to test after this (roughly 3 steps) [b03080f869e11b96ca080dac354c0bf6b361a30c] io_uring: minor refactoring for some tw handlers testing commit b03080f869e11b96ca080dac354c0bf6b361a30c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 468f3122527eec66dacc507dd1be1316a77f2c1b39f557703fe67d9e4563db2b all runs: OK # git bisect good b03080f869e11b96ca080dac354c0bf6b361a30c Bisecting: 2 revisions left to test after this (roughly 2 steps) [65e46eb620ad7fa187415b25638a7b3fb1bc0be2] io_uring: helper for prep+queuing linked timeouts testing commit 65e46eb620ad7fa187415b25638a7b3fb1bc0be2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 10d46161f0639631ee169196c1f8aaddb952c3e5cd6ce92f46f544e246661f70 all runs: crashed: possible deadlock in io_disarm_next # git bisect bad 65e46eb620ad7fa187415b25638a7b3fb1bc0be2 Bisecting: 0 revisions left to test after this (roughly 1 step) [aeedb0f3f9938b2084fe8c912782b031a37161fa] io_uring: inline io_free_req() testing commit aeedb0f3f9938b2084fe8c912782b031a37161fa compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c2085fd1fe101da17b0c821b0443e1887a8b9ca050e98bbb0259ed8600b7cba3 all runs: crashed: possible deadlock in io_disarm_next # git bisect bad aeedb0f3f9938b2084fe8c912782b031a37161fa Bisecting: 0 revisions left to test after this (roughly 0 steps) [78bfbdd1a4977df1dded20f9783a6ec174e67ef8] io_uring: kill io_put_req_deferred() testing commit 78bfbdd1a4977df1dded20f9783a6ec174e67ef8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6ca26a33de0a3bd74299875092ff88c7ccf0f0a32831cb126c0b0df3c31da03f all runs: crashed: possible deadlock in io_disarm_next # git bisect bad 78bfbdd1a4977df1dded20f9783a6ec174e67ef8 78bfbdd1a4977df1dded20f9783a6ec174e67ef8 is the first bad commit commit 78bfbdd1a4977df1dded20f9783a6ec174e67ef8 Author: Pavel Begunkov Date: Fri Apr 15 22:08:23 2022 +0100 io_uring: kill io_put_req_deferred() We have several spots where a call to io_fill_cqe_req() is immediately followed by io_put_req_deferred(). Replace them with __io_req_complete_post() and get rid of io_put_req_deferred() and io_fill_cqe_req(). > size ./fs/io_uring.o text data bss dec hex filename 86942 13734 8 100684 1894c ./fs/io_uring.o > size ./fs/io_uring.o text data bss dec hex filename 86438 13654 8 100100 18704 ./fs/io_uring.o Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/10672a538774ac8986bee6468d960527af59169d.1650056133.git.asml.silence@gmail.com Signed-off-by: Jens Axboe fs/io_uring.c | 42 ++++++++---------------------------------- 1 file changed, 8 insertions(+), 34 deletions(-) culprit signature: 6ca26a33de0a3bd74299875092ff88c7ccf0f0a32831cb126c0b0df3c31da03f parent signature: 468f3122527eec66dacc507dd1be1316a77f2c1b39f557703fe67d9e4563db2b revisions tested: 17, total time: 3h42m31.035706942s (build: 1h42m32.368813184s, test: 1h58m23.213946533s) first bad commit: 78bfbdd1a4977df1dded20f9783a6ec174e67ef8 io_uring: kill io_put_req_deferred() recipients (to): ["asml.silence@gmail.com" "axboe@kernel.dk" "axboe@kernel.dk" "io-uring@vger.kernel.org"] recipients (cc): ["asml.silence@gmail.com" "linux-kernel@vger.kernel.org"] crash: possible deadlock in io_disarm_next ============================================ WARNING: possible recursive locking detected 5.18.0-rc3-syzkaller #0 Not tainted -------------------------------------------- syz-executor.2/4044 is trying to acquire lock: ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:374 [inline] ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: io_disarm_next+0x3c6/0x870 fs/io_uring.c:2417 but task is already holding lock: ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:374 [inline] ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x37/0x1d2 fs/io_uring.c:10054 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&ctx->timeout_lock); lock(&ctx->timeout_lock); *** DEADLOCK *** May be due to missing lock nesting notation 2 locks held by syz-executor.2/4044: #0: ffff888015c1d398 (&ctx->completion_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:349 [inline] #0: ffff888015c1d398 (&ctx->completion_lock){+.+.}-{2:2}, at: io_kill_timeouts+0x2f/0x1d2 fs/io_uring.c:10053 #1: ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:374 [inline] #1: ffff888015c1d3d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x37/0x1d2 fs/io_uring.c:10054 stack backtrace: CPU: 0 PID: 4044 Comm: syz-executor.2 Not tainted 5.18.0-rc3-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 print_deadlock_bug kernel/locking/lockdep.c:2958 [inline] check_deadlock kernel/locking/lockdep.c:3001 [inline] validate_chain kernel/locking/lockdep.c:3790 [inline] __lock_acquire.cold+0x149/0x399 kernel/locking/lockdep.c:5029 lock_acquire kernel/locking/lockdep.c:5641 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5606 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline] _raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170 spin_lock_irq include/linux/spinlock.h:374 [inline] io_disarm_next+0x3c6/0x870 fs/io_uring.c:2417 __io_req_complete_post+0x6d6/0xd00 fs/io_uring.c:2154 io_kill_timeouts+0xc1/0x1d2 fs/io_uring.c:10057 io_ring_ctx_wait_and_kill+0x180/0x2f0 fs/io_uring.c:10084 io_uring_release+0x3d/0x41 fs/io_uring.c:10105 __fput+0x1f5/0x8c0 fs/file_table.c:317 task_work_run+0xc0/0x160 kernel/task_work.c:164 exit_task_work include/linux/task_work.h:37 [inline] do_exit+0x986/0x2470 kernel/exit.c:795 do_group_exit+0xb2/0x2a0 kernel/exit.c:925 get_signal+0x1c12/0x1e50 kernel/signal.c:2864 arch_do_signal_or_restart+0x82/0x20f0 arch/x86/kernel/signal.c:867 exit_to_user_mode_loop kernel/entry/common.c:166 [inline] exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:201 __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline] syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:294 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7fe5f22890e9 Code: Unable to access opcode bytes at RIP 0x7fe5f22890bf. RSP: 002b:00007fe5f3376218 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007fe5f239bf68 RCX: 00007fe5f22890e9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007fe5f239bf6c RBP: 00007fe5f239bf60 R08: 00007ffca395d080 R09: 0000000000000000 R10: 0000000000000002 R11: 0000000000000246 R12: 00007fe5f239bf6c R13: 00007ffca389921f R14: 00007fe5f3376300 R15: 0000000000022000