bisecting cause commit starting from ac935d22736659be734251247dcc6f444fb98972 building syzkaller on 37bccd4ed9e71025cd84e857f9ffca4ec8451c6b testing commit ac935d22736659be734251247dcc6f444fb98972 with gcc (GCC) 8.1.0 kernel signature: ed5ab3002bece4f617cf9a5fe5b1082cad8d6013780f0b758b8ba23a9d42d482 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted testing release v5.6 testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0 kernel signature: 39dc565ad5b241d2dadd4ffe1bcf233eeee2d26c1603e6e3e958c8100bedf019 all runs: OK # git bisect start ac935d22736659be734251247dcc6f444fb98972 7111951b8d4973bda27ff663f2cf18b663d15b48 Bisecting: 7631 revisions left to test after this (roughly 13 steps) [50a5de895dbe5df947b3a695777db5b2c313e065] Merge tag 'for-linus-hmm' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma testing commit 50a5de895dbe5df947b3a695777db5b2c313e065 with gcc (GCC) 8.1.0 kernel signature: f7df392984f395d094ac40be134187972d0dd3b93e7ce395154ce35c13d50884 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 50a5de895dbe5df947b3a695777db5b2c313e065 Bisecting: 4204 revisions left to test after this (roughly 12 steps) [56a451b780676bc1cdac011735fe2869fa2e9abf] Merge tag 'ntb-5.7' of git://github.com/jonmason/ntb testing commit 56a451b780676bc1cdac011735fe2869fa2e9abf with gcc (GCC) 8.1.0 kernel signature: 44b5cab9994a7828d791b63573ebfd7bdf63ce601b64639dfb3fa191d3bc8861 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 56a451b780676bc1cdac011735fe2869fa2e9abf Bisecting: 1643 revisions left to test after this (roughly 11 steps) [49835c15a55225e9b3ff9cc9317135b334ea2d49] Merge tag 'pm-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm testing commit 49835c15a55225e9b3ff9cc9317135b334ea2d49 with gcc (GCC) 8.1.0 kernel signature: 821070491f7302443c829cd539cb88d46c5e5b1a1bfa1b9c4fbc1541c1b67057 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 49835c15a55225e9b3ff9cc9317135b334ea2d49 Bisecting: 934 revisions left to test after this (roughly 10 steps) [063d1942247668eb0bb800aef5afbbef337344be] Merge tag 'media/v5.7-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media testing commit 063d1942247668eb0bb800aef5afbbef337344be with gcc (GCC) 8.1.0 kernel signature: bbee721e7c3acfa6dad395fcf2d9a8c25532a0827534fe4dbf5fbc3a52293588 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 063d1942247668eb0bb800aef5afbbef337344be Bisecting: 302 revisions left to test after this (roughly 9 steps) [481ed297d900af0ce395f6ca8975903b76a5a59e] Merge tag 'docs-5.7' of git://git.lwn.net/linux testing commit 481ed297d900af0ce395f6ca8975903b76a5a59e with gcc (GCC) 8.1.0 kernel signature: 53ac1aa176f3c12f4fe42a60c7841729bfe943bf07fe76830b250ee0885e86f1 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 481ed297d900af0ce395f6ca8975903b76a5a59e Bisecting: 181 revisions left to test after this (roughly 8 steps) [1592614838cb52f4313ceff64894e2ca78591498] Merge tag 'for-5.7/drivers-2020-03-29' of git://git.kernel.dk/linux-block testing commit 1592614838cb52f4313ceff64894e2ca78591498 with gcc (GCC) 8.1.0 kernel signature: b0ace9dfe7669f50cc13ae00561112e50a4ac96aae082bd4b6688af83a5ea05d run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: boot failed: can't ssh into the instance # git bisect good 1592614838cb52f4313ceff64894e2ca78591498 Bisecting: 90 revisions left to test after this (roughly 7 steps) [c9817ad5d82f04fbc66278eda27bff094dcb3119] docs: filesystems: convert udf.txt to ReST testing commit c9817ad5d82f04fbc66278eda27bff094dcb3119 with gcc (GCC) 8.1.0 kernel signature: dba3cfb2528fd7a4f01b52b4cd1419d4c6220e63a63ad3e7335350a6cb6a80f3 all runs: OK # git bisect good c9817ad5d82f04fbc66278eda27bff094dcb3119 Bisecting: 45 revisions left to test after this (roughly 6 steps) [60cf46ae605446feb0c43c472c0fd1af4cd96231] io-wq: hash dependent work testing commit 60cf46ae605446feb0c43c472c0fd1af4cd96231 with gcc (GCC) 8.1.0 kernel signature: bf4f125c5a238adde1cf639add9a60acd7e602475c2ac92d03d2f62aa1ff5c60 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad 60cf46ae605446feb0c43c472c0fd1af4cd96231 Bisecting: 22 revisions left to test after this (roughly 5 steps) [a2100672f3b2afdd55ccc2e640d1a8bd99ff6338] io_uring: clean up io_close testing commit a2100672f3b2afdd55ccc2e640d1a8bd99ff6338 with gcc (GCC) 8.1.0 kernel signature: 066ede88c4fbf31589384be4fdd8c781f917e9ac44da659809be4da45c00f145 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad a2100672f3b2afdd55ccc2e640d1a8bd99ff6338 Bisecting: 10 revisions left to test after this (roughly 4 steps) [e85530ddda4f08d4f9ed6506d4a1f42e086e3b21] io-wq: remove unused IO_WQ_WORK_HAS_MM testing commit e85530ddda4f08d4f9ed6506d4a1f42e086e3b21 with gcc (GCC) 8.1.0 kernel signature: 931a5da3d793ab206e1ad4fa4f34766bcb1d8cae7633d3dfada5d4f14424563f all runs: OK # git bisect good e85530ddda4f08d4f9ed6506d4a1f42e086e3b21 Bisecting: 5 revisions left to test after this (roughly 3 steps) [b41e98524e424d104aa7851d54fd65820759875a] io_uring: add per-task callback handler testing commit b41e98524e424d104aa7851d54fd65820759875a with gcc (GCC) 8.1.0 kernel signature: e00511cc84d1c8fd38ede9aa7816a80fc55eb5630aad0d0d1804268ce6a43676 all runs: crashed: INFO: trying to register non-static key in io_cqring_ev_posted # git bisect bad b41e98524e424d104aa7851d54fd65820759875a Bisecting: 2 revisions left to test after this (roughly 1 step) [3684f24653534c71c7dc9f44d7281a838f4e4979] io-wq: use BIT for ulong hash testing commit 3684f24653534c71c7dc9f44d7281a838f4e4979 with gcc (GCC) 8.1.0 kernel signature: 31a570277295a4f3b19edc88ece5312cb4027963218e6117c2fe3c7ae29ae1cd all runs: OK # git bisect good 3684f24653534c71c7dc9f44d7281a838f4e4979 Bisecting: 0 revisions left to test after this (roughly 1 step) [c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14] io_uring: store io_kiocb in wait->private testing commit c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14 with gcc (GCC) 8.1.0 kernel signature: 48dce13f43d3c49f7da64b38d5f50683cb99843010e41c94c51e04cf20dea161 all runs: OK # git bisect good c2f2eb7d2c1cdc37fa9633bae96f381d33ee7a14 b41e98524e424d104aa7851d54fd65820759875a is the first bad commit commit b41e98524e424d104aa7851d54fd65820759875a Author: Jens Axboe Date: Mon Feb 17 09:52:41 2020 -0700 io_uring: add per-task callback handler For poll requests, it's not uncommon to link a read (or write) after the poll to execute immediately after the file is marked as ready. Since the poll completion is called inside the waitqueue wake up handler, we have to punt that linked request to async context. This slows down the processing, and actually means it's faster to not use a link for this use case. We also run into problems if the completion_lock is contended, as we're doing a different lock ordering than the issue side is. Hence we have to do trylock for completion, and if that fails, go async. Poll removal needs to go async as well, for the same reason. eventfd notification needs special case as well, to avoid stack blowing recursion or deadlocks. These are all deficiencies that were inherited from the aio poll implementation, but I think we can do better. When a poll completes, simply queue it up in the task poll list. When the task completes the list, we can run dependent links inline as well. This means we never have to go async, and we can remove a bunch of code associated with that, and optimizations to try and make that run faster. The diffstat speaks for itself. Signed-off-by: Jens Axboe fs/io_uring.c | 218 ++++++++++++++++++++-------------------------------------- 1 file changed, 76 insertions(+), 142 deletions(-) culprit signature: e00511cc84d1c8fd38ede9aa7816a80fc55eb5630aad0d0d1804268ce6a43676 parent signature: 48dce13f43d3c49f7da64b38d5f50683cb99843010e41c94c51e04cf20dea161 revisions tested: 15, total time: 3h10m18.966131482s (build: 1h35m37.430339644s, test: 1h33m12.93304034s) first bad commit: b41e98524e424d104aa7851d54fd65820759875a io_uring: add per-task callback handler cc: ["axboe@kernel.dk" "io-uring@vger.kernel.org" "linux-fsdevel@vger.kernel.org" "linux-kernel@vger.kernel.org" "viro@zeniv.linux.org.uk"] crash: INFO: trying to register non-static key in io_cqring_ev_posted RAX: ffffffffffffffda RBX: 00000000004e0f40 RCX: 000000000045ca29 RDX: 0000000000000000 RSI: 0000000020000140 RDI: 000000000000047b RBP: 000000000078bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000204 R14: 00000000004c445f R15: 00007f820d7bd6d4 INFO: trying to register non-static key. the code is fine but needs lockdep annotation. turning off the locking correctness validator. CPU: 0 PID: 8481 Comm: syz-executor.2 Not tainted 5.6.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 assign_lock_key kernel/locking/lockdep.c:880 [inline] register_lock_class+0x18a0/0x1a40 kernel/locking/lockdep.c:1189 __lock_acquire+0x101/0x4370 kernel/locking/lockdep.c:3836 lock_acquire+0x19b/0x420 kernel/locking/lockdep.c:4484 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x95/0xc0 kernel/locking/spinlock.c:159 __wake_up_common_lock+0xa8/0x120 kernel/sched/wait.c:122 io_cqring_ev_posted+0x91/0x180 fs/io_uring.c:1097 io_poll_remove_all fs/io_uring.c:3590 [inline] io_ring_ctx_wait_and_kill+0x2b0/0x990 fs/io_uring.c:6470 io_uring_create fs/io_uring.c:6990 [inline] io_uring_setup+0x184b/0x1e10 fs/io_uring.c:7017 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45ca29 Code: 0d b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 db b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f820d7bcc78 EFLAGS: 00000246 ORIG_RAX: 00000000000001a9 RAX: ffffffffffffffda RBX: 00000000004e0f40 RCX: 000000000045ca29 RDX: 0000000000000000 RSI: 0000000020000140 RDI: 000000000000047b RBP: 000000000078bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000204 R14: 00000000004c445f R15: 00007f820d7bd6d4 general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 0 PID: 8481 Comm: syz-executor.2 Not tainted 5.6.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__wake_up_common+0xd6/0x610 kernel/sched/wait.c:86 Code: 14 05 00 00 4c 8b 43 38 49 83 e8 18 49 8d 78 18 48 39 fd 0f 84 50 02 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 <80> 3c 01 00 0f 85 0b 05 00 00 49 8b 40 18 89 54 24 10 31 db 49 bd RSP: 0018:ffffc90009417c30 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880a9360120 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000003 RDI: 0000000000000000 RBP: ffff8880a9360158 R08: ffffffffffffffe8 R09: ffffc90009417cb8 R10: fffff52001282f80 R11: 0000000000000003 R12: ffffc90009417cb8 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 FS: 00007f820d7bd700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000004e0f50 CR3: 000000009fb17000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __wake_up_common_lock+0xc3/0x120 kernel/sched/wait.c:123 io_cqring_ev_posted+0x91/0x180 fs/io_uring.c:1097 io_poll_remove_all fs/io_uring.c:3590 [inline] io_ring_ctx_wait_and_kill+0x2b0/0x990 fs/io_uring.c:6470 io_uring_create fs/io_uring.c:6990 [inline] io_uring_setup+0x184b/0x1e10 fs/io_uring.c:7017 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45ca29 Code: 0d b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 db b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f820d7bcc78 EFLAGS: 00000246 ORIG_RAX: 00000000000001a9 RAX: ffffffffffffffda RBX: 00000000004e0f40 RCX: 000000000045ca29 RDX: 0000000000000000 RSI: 0000000020000140 RDI: 000000000000047b RBP: 000000000078bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003 R13: 0000000000000204 R14: 00000000004c445f R15: 00007f820d7bd6d4 Modules linked in: ---[ end trace 4ed6a65e398b42a6 ]--- RIP: 0010:__wake_up_common+0xd6/0x610 kernel/sched/wait.c:86 Code: 14 05 00 00 4c 8b 43 38 49 83 e8 18 49 8d 78 18 48 39 fd 0f 84 50 02 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 <80> 3c 01 00 0f 85 0b 05 00 00 49 8b 40 18 89 54 24 10 31 db 49 bd RSP: 0018:ffffc90009417c30 EFLAGS: 00010046 RAX: dffffc0000000000 RBX: ffff8880a9360120 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000003 RDI: 0000000000000000 RBP: ffff8880a9360158 R08: ffffffffffffffe8 R09: ffffc90009417cb8 R10: fffff52001282f80 R11: 0000000000000003 R12: ffffc90009417cb8 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 FS: 00007f820d7bd700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000004e0f50 CR3: 000000009fb17000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400