bisecting cause commit starting from 4b93c544e90e2b28326182d31ee008eb80e02074 building syzkaller on 6ca601483d056968f63fd4735fc54073f4fe3c75 testing commit 4b93c544e90e2b28326182d31ee008eb80e02074 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: bd3c5739fc576408f59c0db282769de575944de125119bc3e0959d67e804219c run #0: crashed: WARNING in io_wq_submit_work run #1: crashed: WARNING in io_wq_submit_work run #2: crashed: WARNING in io_wq_submit_work run #3: crashed: BUG: corrupted list in __io_free_req run #4: crashed: WARNING in io_wq_submit_work run #5: crashed: WARNING in io_wq_submit_work run #6: crashed: WARNING in io_wq_submit_work run #7: crashed: WARNING in io_wq_submit_work run #8: crashed: WARNING in io_wq_submit_work run #9: crashed: WARNING in io_wq_submit_work run #10: crashed: WARNING in corrupted run #11: crashed: WARNING in io_wq_submit_work run #12: crashed: WARNING in corrupted run #13: crashed: WARNING in io_wq_submit_work run #14: crashed: WARNING in io_wq_submit_work run #15: crashed: WARNING in io_wq_submit_work run #16: crashed: WARNING in io_wq_submit_work run #17: crashed: WARNING in io_wq_submit_work run #18: crashed: WARNING in io_wq_submit_work run #19: crashed: WARNING in io_wq_submit_work testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 1bc25c310b773f41f60536ec01fad7eee81fbd08982b6b34e2bd4f1aafdef1ea all runs: OK # git bisect start 4b93c544e90e2b28326182d31ee008eb80e02074 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 4950 revisions left to test after this (roughly 12 steps) [ea7b4244b3656ca33b19a950f092b5bbc718b40c] x86/setup: Explicitly include acpi.h testing commit ea7b4244b3656ca33b19a950f092b5bbc718b40c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: b49a06514e8dee28c6643b746ab3321b6372b4894f3709a70efc11651ff087bb all runs: OK # git bisect good ea7b4244b3656ca33b19a950f092b5bbc718b40c Bisecting: 2653 revisions left to test after this (roughly 11 steps) [32b47072f319bb65e9afad59e78153d83496f1f5] Merge tag 'defconfig-5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 32b47072f319bb65e9afad59e78153d83496f1f5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: e1b7f1cb89f35b969ac45148c075d266398296c613e4c8bf0e1bd787bbf6add9 all runs: OK # git bisect good 32b47072f319bb65e9afad59e78153d83496f1f5 Bisecting: 1359 revisions left to test after this (roughly 10 steps) [83ec91697412ae64d25dcca74597ed03029aa00d] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/hid/hid testing commit 83ec91697412ae64d25dcca74597ed03029aa00d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 3f8b0da8f0220a44681a97953406ea676d69a0d8a6160f82b489bcf54e3696ce all runs: OK # git bisect good 83ec91697412ae64d25dcca74597ed03029aa00d Bisecting: 681 revisions left to test after this (roughly 9 steps) [50ddcdb2635c82e195a2557341d759c5b9419bf1] Merge tag 'livepatching-for-5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching testing commit 50ddcdb2635c82e195a2557341d759c5b9419bf1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: d22dd194035640b14011e174efe181151b3f28225951fae353e7f85ec20ab3bf all runs: OK # git bisect good 50ddcdb2635c82e195a2557341d759c5b9419bf1 Bisecting: 346 revisions left to test after this (roughly 8 steps) [6abaa83c7352b31450d7e8c173f674324c16b02b] Merge tag 'f2fs-for-5.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs testing commit 6abaa83c7352b31450d7e8c173f674324c16b02b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 2cc13ab9292ebecfed669e0d265e91819989e16b472b02e70bcaa86ee7d76c21 all runs: OK # git bisect good 6abaa83c7352b31450d7e8c173f674324c16b02b Bisecting: 163 revisions left to test after this (roughly 8 steps) [58ca24158758f1784400d32743373d7d6227d018] Merge tag 'trace-v5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace testing commit 58ca24158758f1784400d32743373d7d6227d018 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 16641971386cc7c580a8fadf8a57435c1b4d52fe2d8533bda740de4f5505355e run #0: basic kernel testing failed: failed to copy test binary to VM: timedout ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "/syzkaller/jobs/linux/gopath/src/github.com/google/syzkaller/bin/linux_amd64/syz-fuzzer" "root@10.128.0.166:./syz-fuzzer"] Warning: Permanently added '10.128.0.166' (ECDSA) to the list of known hosts. run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 58ca24158758f1784400d32743373d7d6227d018 Bisecting: 81 revisions left to test after this (roughly 6 steps) [ce73af80876dba7b855fe36d1ec473f77300c214] perf tools: Add missing newline at the end of header file testing commit ce73af80876dba7b855fe36d1ec473f77300c214 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 61890e0a2dd2851c1a321317674763cdbbd2ddb0ddd496d837ac2233a9942ef9 all runs: OK # git bisect good ce73af80876dba7b855fe36d1ec473f77300c214 Bisecting: 40 revisions left to test after this (roughly 5 steps) [050a0fc4edc75a97b4dba7b834fd6bf243bf06ed] perf cs-etm: Fix typo testing commit 050a0fc4edc75a97b4dba7b834fd6bf243bf06ed compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 1bc25c310b773f41f60536ec01fad7eee81fbd08982b6b34e2bd4f1aafdef1ea all runs: OK # git bisect good 050a0fc4edc75a97b4dba7b834fd6bf243bf06ed Bisecting: 24 revisions left to test after this (roughly 4 steps) [2fc2a7a62eb58650e71b4550cf6fa6cc0a75b2d2] io_uring: io_uring_complete() trace should take an integer testing commit 2fc2a7a62eb58650e71b4550cf6fa6cc0a75b2d2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 0be437103b07c554eb4d14abdd9024ce85325c1db91fccbeff0da138e72753fe run #0: crashed: WARNING in io_wq_submit_work run #1: crashed: WARNING in corrupted run #2: crashed: WARNING in io_wq_submit_work run #3: crashed: WARNING in io_wq_submit_work run #4: crashed: WARNING in io_wq_submit_work run #5: crashed: WARNING in io_wq_submit_work run #6: crashed: WARNING in io_wq_submit_work run #7: crashed: WARNING in io_wq_submit_work run #8: crashed: WARNING in io_wq_submit_work run #9: crashed: WARNING in corrupted # git bisect bad 2fc2a7a62eb58650e71b4550cf6fa6cc0a75b2d2 Bisecting: 7 revisions left to test after this (roughly 3 steps) [f95dc207b93da9c88ddbb7741ec3730c6657b88e] io-wq: split bounded and unbounded work into separate lists testing commit f95dc207b93da9c88ddbb7741ec3730c6657b88e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: d2add9641bd9a5f14ec1d59d0b971c63d0393bf56b087b9b1ce8c936b7f3f80c all runs: OK # git bisect good f95dc207b93da9c88ddbb7741ec3730c6657b88e Bisecting: 3 revisions left to test after this (roughly 2 steps) [fa84693b3c896460831fe0750554121121a23da8] io_uring: ensure IORING_REGISTER_IOWQ_MAX_WORKERS works with SQPOLL testing commit fa84693b3c896460831fe0750554121121a23da8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: fa2be741693ffda5a768d0a1a83c2862cb5a05ea62303b7b7f6913fc140e2217 all runs: crashed: WARNING in io_wq_submit_work # git bisect bad fa84693b3c896460831fe0750554121121a23da8 Bisecting: 1 revision left to test after this (roughly 1 step) [05c5f4ee4da7086cceacc78bf3a080e314c241fa] io-wq: get rid of FIXED worker flag testing commit 05c5f4ee4da7086cceacc78bf3a080e314c241fa compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: a8e5f8c1f19073d700f7e56d0a73c59c2482e674229337d0255c6665da99cecd all runs: OK # git bisect good 05c5f4ee4da7086cceacc78bf3a080e314c241fa Bisecting: 0 revisions left to test after this (roughly 0 steps) [3146cba99aa284b1d4a10fbd923df953f1d18035] io-wq: make worker creation resilient against signals testing commit 3146cba99aa284b1d4a10fbd923df953f1d18035 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 3f2394ef63f10b4eb421b8a432b50582e0bc645d67abc7a4795943f1a871713f all runs: crashed: WARNING in io_wq_submit_work # git bisect bad 3146cba99aa284b1d4a10fbd923df953f1d18035 3146cba99aa284b1d4a10fbd923df953f1d18035 is the first bad commit commit 3146cba99aa284b1d4a10fbd923df953f1d18035 Author: Jens Axboe Date: Wed Sep 1 11:20:10 2021 -0600 io-wq: make worker creation resilient against signals If a task is queueing async work and also handling signals, then we can run into the case where create_io_thread() is interrupted and returns failure because of that. If this happens for creating the first worker in a group, then that worker will never get created and we can hang the ring. If we do get a fork failure, retry from task_work. With signals we have to be a bit careful as we cannot simply queue as task_work, as we'll still have signals pending at that point. Punt over a normal workqueue first and then create from task_work after that. Lastly, ensure that we handle fatal worker creations. Worker creation failures are normally not fatal, only if we fail to create one in an empty worker group can we not make progress. Right now that is ignored, ensure that we handle that and run cancel on the work item. There are two paths that create new workers - one is the "existing worker going to sleep", and the other is "no workers found for this work, create one". The former is never fatal, as workers do exist in the group. Only the latter needs to be carefully handled. Signed-off-by: Jens Axboe fs/io-wq.c | 223 +++++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 166 insertions(+), 57 deletions(-) culprit signature: 3f2394ef63f10b4eb421b8a432b50582e0bc645d67abc7a4795943f1a871713f parent signature: a8e5f8c1f19073d700f7e56d0a73c59c2482e674229337d0255c6665da99cecd revisions tested: 15, total time: 3h37m1.659149273s (build: 1h45m31.051430929s, test: 1h49m56.189217458s) first bad commit: 3146cba99aa284b1d4a10fbd923df953f1d18035 io-wq: make worker creation resilient against signals recipients (to): ["axboe@kernel.dk" "axboe@kernel.dk" "io-uring@vger.kernel.org"] recipients (cc): ["asml.silence@gmail.com" "linux-kernel@vger.kernel.org"] crash: WARNING in io_wq_submit_work ------------[ cut here ]------------ WARNING: CPU: 1 PID: 8 at fs/io_uring.c:1135 req_ref_get fs/io_uring.c:1137 [inline] WARNING: CPU: 1 PID: 8 at fs/io_uring.c:1135 io_wq_submit_work+0x1e8/0x270 fs/io_uring.c:6520 Modules linked in: CPU: 1 PID: 8 Comm: kworker/u4:0 Not tainted 5.14.0-rc7-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events_unbound io_ring_exit_work RIP: 0010:req_ref_get fs/io_uring.c:1135 [inline] RIP: 0010:io_wq_submit_work+0x1e8/0x270 fs/io_uring.c:6520 Code: 03 38 d0 7c 04 84 d2 75 50 8b 43 a4 83 c0 7f 83 f8 7f 76 16 be 04 00 00 00 4c 89 e7 e8 91 27 de ff f0 ff 43 a4 e9 a9 fe ff ff <0f> 0b eb e6 48 89 ef e8 5c 6d fc ff 48 85 c0 48 89 c7 0f 84 bc fe RSP: 0018:ffffc90000cd78e0 EFLAGS: 00010246 RAX: 000000000000007f RBX: ffff888045a0f978 RCX: ffffffff81c82cb1 RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff888045a0f91c RBP: ffff888045a0f8c0 R08: 0000000000000000 R09: ffff888045a0f91f R10: ffffed1008b41f23 R11: 0000000000000002 R12: ffff888045a0f91c R13: ffff888045a0f918 R14: ffffed1008b41802 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f9c2b3f0000 CR3: 000000000a88e000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: io_run_cancel fs/io-wq.c:809 [inline] io_acct_cancel_pending_work.isra.0+0x23f/0x5c0 fs/io-wq.c:950 io_wqe_cancel_pending_work+0x73/0xe0 fs/io-wq.c:968 io_wq_cancel_cb+0x185/0x410 fs/io-wq.c:1003 io_uring_try_cancel_iowq fs/io_uring.c:9260 [inline] io_uring_try_cancel_requests+0x725/0xb80 fs/io_uring.c:9280 io_ring_exit_work+0x117/0x1610 fs/io_uring.c:9080 process_one_work+0x84c/0x13d0 kernel/workqueue.c:2276 worker_thread+0x598/0x1040 kernel/workqueue.c:2422 kthread+0x38b/0x460 kernel/kthread.c:319 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295