bisecting cause commit starting from 673b41e04a035d760bc0aff83fa9ee24fd9c2779 building syzkaller on c8d1cc20df5ca5d9ea437054720fa3cfdfa1f578 testing commit 673b41e04a035d760bc0aff83fa9ee24fd9c2779 with gcc (GCC) 8.1.0 kernel signature: 682733d37d18d0c3fabb878ced7406ddbfb2c53503acea775b908a70016e6787 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted testing release v5.6 testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0 kernel signature: 582bc8e0339d045c988bfe9b94a4cdd460825331e816c4217843e50baf0b76c9 all runs: OK # git bisect start 673b41e04a035d760bc0aff83fa9ee24fd9c2779 7111951b8d4973bda27ff663f2cf18b663d15b48 Bisecting: 1116 revisions left to test after this (roughly 10 steps) [59838093be51ee9447f6ad05483d697b6fa0368d] Merge tag 'driver-core-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core testing commit 59838093be51ee9447f6ad05483d697b6fa0368d with gcc (GCC) 8.1.0 kernel signature: bec635739e39b2fa12fb4011d5e1b18088853599b608a2efe61c4f8d5e056ece run #0: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #1: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #2: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #3: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #4: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #5: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #6: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #7: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #8: crashed: INFO: trying to register non-static key in io_cqring_ev_posted run #9: boot failed: can't ssh into the instance # git bisect bad 59838093be51ee9447f6ad05483d697b6fa0368d Bisecting: 533 revisions left to test after this (roughly 9 steps) [47acac8cae28b36668bf89400c56b7fdebca3e75] Merge tag 'hwmon-for-v5.7' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging testing commit 47acac8cae28b36668bf89400c56b7fdebca3e75 with gcc (GCC) 8.1.0 kernel signature: ea1a6befde944c8f8ca6027f914329923a6eb814c4f1e25c7542c60160edd216 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 47acac8cae28b36668bf89400c56b7fdebca3e75 Bisecting: 247 revisions left to test after this (roughly 8 steps) [1592614838cb52f4313ceff64894e2ca78591498] Merge tag 'for-5.7/drivers-2020-03-29' of git://git.kernel.dk/linux-block testing commit 1592614838cb52f4313ceff64894e2ca78591498 with gcc (GCC) 8.1.0 kernel signature: ade671a99610788a46293abf3cb89ff9b96771dff03c5384f0d2201955ed7fae all runs: OK # git bisect good 1592614838cb52f4313ceff64894e2ca78591498 Bisecting: 123 revisions left to test after this (roughly 7 steps) [7af51678b6d367ee93dc3d21e72ecf15be50fcb1] docs: deprecated.rst: Add BUG()-family testing commit 7af51678b6d367ee93dc3d21e72ecf15be50fcb1 with gcc (GCC) 8.1.0 kernel signature: bf70f674cbaf9116e5d3ae78d67f2b13627fccb686b1fc731e8d22638e3329bc all runs: OK # git bisect good 7af51678b6d367ee93dc3d21e72ecf15be50fcb1 Bisecting: 59 revisions left to test after this (roughly 6 steps) [377ad0c28c1df7b0634e697f34bdea8325f39a66] Merge tag 'erofs-for-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs testing commit 377ad0c28c1df7b0634e697f34bdea8325f39a66 with gcc (GCC) 8.1.0 kernel signature: 42b6ee8f461cbadff3341d53bcdc350ebf19c36881e7212790ba91828dde1907 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 377ad0c28c1df7b0634e697f34bdea8325f39a66 Bisecting: 31 revisions left to test after this (roughly 5 steps) [bcda7baaa3f15c7a95db3c024bb046d6e298f76b] io_uring: support buffer selection for OP_READ and OP_RECV testing commit bcda7baaa3f15c7a95db3c024bb046d6e298f76b with gcc (GCC) 8.1.0 kernel signature: a23d647ab2234ecca5f57c0ab55f8f96faa8d73444adaa0004f28da00b1fbf0b all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad bcda7baaa3f15c7a95db3c024bb046d6e298f76b Bisecting: 15 revisions left to test after this (roughly 4 steps) [b41e98524e424d104aa7851d54fd65820759875a] io_uring: add per-task callback handler testing commit b41e98524e424d104aa7851d54fd65820759875a with gcc (GCC) 8.1.0 kernel signature: 19728b9d7220ff2ba784c9abf06e0aad2290396a0b20d4bd8cd9d13f62a86348 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad b41e98524e424d104aa7851d54fd65820759875a Bisecting: 7 revisions left to test after this (roughly 3 steps) [7d67af2c013402537385dae343a2d0f6a4cb3bfd] io_uring: add splice(2) support testing commit 7d67af2c013402537385dae343a2d0f6a4cb3bfd with gcc (GCC) 8.1.0 kernel signature: 853b81e4e7154ca2038708e93839df6a5177f0e75aa15d9ed477301436c3278a all runs: OK # git bisect good 7d67af2c013402537385dae343a2d0f6a4cb3bfd Bisecting: 3 revisions left to test after this (roughly 2 steps) [5eae8619907a1389dbd1b4a1049caf52782c0916] io_uring: remove IO_WQ_WORK_CB testing commit 5eae8619907a1389dbd1b4a1049caf52782c0916 with gcc (GCC) 8.1.0 kernel signature: 91982e0b6e8be805e1b563f2bc738586d97aa99a5a12515cb3b57aecc23bc569 all runs: OK # git bisect good 5eae8619907a1389dbd1b4a1049caf52782c0916 Bisecting: 1 revision left to test after this (roughly 1 step) [6fb614920b38bbf3c1c7fcd944c6d9b5d746103d] task_work_run: don't take ->pi_lock unconditionally testing commit 6fb614920b38bbf3c1c7fcd944c6d9b5d746103d with gcc (GCC) 8.1.0 kernel signature: 345117da53dd75ab6c2ae18bfd6d76b6bab8bf87f5d3b6f1838a349e0f7c0c57 all runs: OK # git bisect good 6fb614920b38bbf3c1c7fcd944c6d9b5d746103d Bisecting: 0 revisions left to test after this (roughly 0 steps) [c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14] io_uring: store io_kiocb in wait->private testing commit c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14 with gcc (GCC) 8.1.0 kernel signature: ad4be9b462dc605958edd76c4a3a886553d0eebe60e85d41b5829cf9c71df5d7 all runs: OK # git bisect good c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14 b41e98524e424d104aa7851d54fd65820759875a is the first bad commit commit b41e98524e424d104aa7851d54fd65820759875a Author: Jens Axboe Date: Mon Feb 17 09:52:41 2020 -0700 io_uring: add per-task callback handler For poll requests, it's not uncommon to link a read (or write) after the poll to execute immediately after the file is marked as ready. Since the poll completion is called inside the waitqueue wake up handler, we have to punt that linked request to async context. This slows down the processing, and actually means it's faster to not use a link for this use case. We also run into problems if the completion_lock is contended, as we're doing a different lock ordering than the issue side is. Hence we have to do trylock for completion, and if that fails, go async. Poll removal needs to go async as well, for the same reason. eventfd notification needs special case as well, to avoid stack blowing recursion or deadlocks. These are all deficiencies that were inherited from the aio poll implementation, but I think we can do better. When a poll completes, simply queue it up in the task poll list. When the task completes the list, we can run dependent links inline as well. This means we never have to go async, and we can remove a bunch of code associated with that, and optimizations to try and make that run faster. The diffstat speaks for itself. Signed-off-by: Jens Axboe fs/io_uring.c | 218 ++++++++++++++++++++-------------------------------------- 1 file changed, 76 insertions(+), 142 deletions(-) culprit signature: 19728b9d7220ff2ba784c9abf06e0aad2290396a0b20d4bd8cd9d13f62a86348 parent signature: ad4be9b462dc605958edd76c4a3a886553d0eebe60e85d41b5829cf9c71df5d7 revisions tested: 13, total time: 3h13m14.986977408s (build: 1h24m53.865682205s, test: 1h47m34.388508773s) first bad commit: b41e98524e424d104aa7851d54fd65820759875a io_uring: add per-task callback handler cc: ["axboe@kernel.dk" "io-uring@vger.kernel.org" "linux-fsdevel@vger.kernel.org" "linux-kernel@vger.kernel.org" "viro@zeniv.linux.org.uk"] crash: INFO: trying to register non-static key in io_cqring_ev_posted RAX: ffffffffffffffda RBX: 00007f47273a16d4 RCX: 000000000045c849 RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000910 RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000202 R14: 00000000004c4244 R15: 0000000000000004 INFO: trying to register non-static key. the code is fine but needs lockdep annotation. turning off the locking correctness validator. CPU: 1 PID: 8562 Comm: syz-executor.2 Not tainted 5.6.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 assign_lock_key kernel/locking/lockdep.c:880 [inline] register_lock_class+0x18a0/0x1a40 kernel/locking/lockdep.c:1189 __lock_acquire+0x101/0x4370 kernel/locking/lockdep.c:3836 lock_acquire+0x19b/0x420 kernel/locking/lockdep.c:4484 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x95/0xc0 kernel/locking/spinlock.c:159 __wake_up_common_lock+0xa8/0x120 kernel/sched/wait.c:122 io_cqring_ev_posted+0x91/0x180 fs/io_uring.c:1097 io_poll_remove_all fs/io_uring.c:3590 [inline] io_ring_ctx_wait_and_kill+0x2b0/0x990 fs/io_uring.c:6470 io_uring_create fs/io_uring.c:6990 [inline] io_uring_setup+0x184b/0x1e10 fs/io_uring.c:7017 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45c849 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f47273a0c78 EFLAGS: 00000246 ORIG_RAX: 00000000000001a9 RAX: ffffffffffffffda RBX: 00007f47273a16d4 RCX: 000000000045c849 RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000910 RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000202 R14: 00000000004c4244 R15: 0000000000000004 general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 1 PID: 8562 Comm: syz-executor.2 Not tainted 5.6.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__wake_up_common+0xd6/0x610 kernel/sched/wait.c:86 Code: 14 05 00 00 4c 8b 43 38 49 83 e8 18 49 8d 78 18 48 39 fd 0f 84 50 02 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 <80> 3c 01 00 0f 85 0b 05 00 00 49 8b 40 18 89 54 24 10 31 db 49 bd RSP: 0018:ffffc900043c7c30 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880a8428120 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000003 RDI: 0000000000000000 RBP: ffff8880a8428158 R08: ffffffffffffffe8 R09: ffffc900043c7cb8 R10: fffff52000878f80 R11: 0000000000000003 R12: ffffc900043c7cb8 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 FS: 00007f47273a1700(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f71346c7010 CR3: 00000000969df000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __wake_up_common_lock+0xc3/0x120 kernel/sched/wait.c:123 io_cqring_ev_posted+0x91/0x180 fs/io_uring.c:1097 io_poll_remove_all fs/io_uring.c:3590 [inline] io_ring_ctx_wait_and_kill+0x2b0/0x990 fs/io_uring.c:6470 io_uring_create fs/io_uring.c:6990 [inline] io_uring_setup+0x184b/0x1e10 fs/io_uring.c:7017 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45c849 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f47273a0c78 EFLAGS: 00000246 ORIG_RAX: 00000000000001a9 RAX: ffffffffffffffda RBX: 00007f47273a16d4 RCX: 000000000045c849 RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000910 RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000202 R14: 00000000004c4244 R15: 0000000000000004 Modules linked in: ---[ end trace e9dd66c588740021 ]--- RIP: 0010:__wake_up_common+0xd6/0x610 kernel/sched/wait.c:86 Code: 14 05 00 00 4c 8b 43 38 49 83 e8 18 49 8d 78 18 48 39 fd 0f 84 50 02 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 <80> 3c 01 00 0f 85 0b 05 00 00 49 8b 40 18 89 54 24 10 31 db 49 bd RSP: 0018:ffffc900043c7c30 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880a8428120 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000003 RDI: 0000000000000000 RBP: ffff8880a8428158 R08: ffffffffffffffe8 R09: ffffc900043c7cb8 R10: fffff52000878f80 R11: 0000000000000003 R12: ffffc900043c7cb8 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 FS: 00007f47273a1700(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f71346c7010 CR3: 00000000969df000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400