bisecting cause commit starting from 8d4b477da1a807199ca60e0829357ce7aa6758d5 building syzkaller on d2d6e680bf6f6e94be547a6c6f01864e74b3e29f testing commit 8d4b477da1a807199ca60e0829357ce7aa6758d5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 4a493c7a3ce1126665b136afde2ed48f9fa249bee7f6fb91ea72f17b1f84635b run #0: OK run #1: crashed: INFO: task hung in css_free_rwork_fn run #2: crashed: INFO: task hung in call_usermodehelper_exec run #3: crashed: INFO: task hung in css_free_rwork_fn run #4: crashed: INFO: task hung in css_free_rwork_fn run #5: OK run #6: crashed: INFO: task hung in css_free_rwork_fn run #7: crashed: INFO: task hung in css_free_rwork_fn run #8: OK run #9: crashed: INFO: task hung in call_usermodehelper_exec run #10: OK run #11: crashed: INFO: task hung in css_free_rwork_fn run #12: crashed: INFO: task hung in css_free_rwork_fn run #13: crashed: INFO: task hung in call_usermodehelper_exec run #14: crashed: INFO: task hung in css_free_rwork_fn run #15: crashed: INFO: task hung in call_usermodehelper_exec run #16: OK run #17: OK run #18: crashed: WARNING in io_ring_exit_work run #19: crashed: WARNING in io_ring_exit_work testing release v5.13 testing commit 62fb9874f5da54fdb243003b386128037319b219 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 1cb5111c0f95c1fc7e3896d55db5b53fbce5858f143c657e735c92f7019c7d10 run #0: OK run #1: crashed: INFO: task hung in call_usermodehelper_exec run #2: crashed: INFO: task hung in call_usermodehelper_exec run #3: OK run #4: crashed: INFO: task hung in css_free_rwork_fn run #5: crashed: INFO: task hung in call_usermodehelper_exec run #6: crashed: INFO: task hung in css_free_rwork_fn run #7: crashed: INFO: task hung in css_free_rwork_fn run #8: OK run #9: crashed: WARNING in io_ring_exit_work testing release v5.12 testing commit 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: f49397d19055b9442e2c2b720108afe8474cea2e340222187afff379ccf6ade1 all runs: OK # git bisect start 62fb9874f5da54fdb243003b386128037319b219 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 Bisecting: 8739 revisions left to test after this (roughly 13 steps) [85f3f17b5db2dd9f8a094a0ddc665555135afd22] Merge branch 'md-fixes' of https://git.kernel.org/pub/scm/linux/kernel/git/song/md into block-5.13 testing commit 85f3f17b5db2dd9f8a094a0ddc665555135afd22 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 714cf9b6fe7a618558f7d423d77c11e1f7356ff667a019d2f409379192a0b248 run #0: crashed: WARNING in io_rsrc_node_switch run #1: OK run #2: OK run #3: crashed: INFO: task hung in call_usermodehelper_exec run #4: crashed: INFO: task hung in css_free_rwork_fn run #5: crashed: INFO: task hung in css_free_rwork_fn run #6: crashed: INFO: task hung in css_free_rwork_fn run #7: crashed: INFO: task hung in css_free_rwork_fn run #8: crashed: INFO: task hung in call_usermodehelper_exec run #9: crashed: INFO: task hung in call_usermodehelper_exec # git bisect bad 85f3f17b5db2dd9f8a094a0ddc665555135afd22 Bisecting: 4263 revisions left to test after this (roughly 12 steps) [ca62e9090d229926f43f20291bb44d67897baab7] Merge tag 'regulator-v5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator testing commit ca62e9090d229926f43f20291bb44d67897baab7 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 7acaedd741eb1f2bf8986647e3ea9b22a76db6ed18246b721426aaee187cc1d2 all runs: OK # git bisect good ca62e9090d229926f43f20291bb44d67897baab7 Bisecting: 1907 revisions left to test after this (roughly 11 steps) [68a32ba14177d4a21c4a9a941cf1d7aea86d436f] Merge tag 'drm-next-2021-04-28' of git://anongit.freedesktop.org/drm/drm testing commit 68a32ba14177d4a21c4a9a941cf1d7aea86d436f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 4282333eaab1bba9872e0229b1dfe7bb3c24b7594fd6d38d20e7c0c71eba2f45 all runs: OK # git bisect good 68a32ba14177d4a21c4a9a941cf1d7aea86d436f Bisecting: 968 revisions left to test after this (roughly 10 steps) [be18cd1fcae2ed7db58d92d20733dfa8aa0a5173] Merge tag 'mmc-v5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc testing commit be18cd1fcae2ed7db58d92d20733dfa8aa0a5173 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 37807bd3a0a15369dade57caded56f5b06ce904f22283fcdf78e7fd061f5c326 run #0: crashed: WARNING in io_rsrc_node_switch run #1: OK run #2: OK run #3: crashed: INFO: task hung in call_usermodehelper_exec run #4: crashed: INFO: task hung in call_usermodehelper_exec run #5: crashed: INFO: task hung in call_usermodehelper_exec run #6: crashed: INFO: task hung in call_usermodehelper_exec run #7: crashed: INFO: task hung in call_usermodehelper_exec run #8: OK run #9: crashed: INFO: task hung in call_usermodehelper_exec # git bisect bad be18cd1fcae2ed7db58d92d20733dfa8aa0a5173 Bisecting: 473 revisions left to test after this (roughly 9 steps) [fc0586062816559defb14c947319ef8c4c326fb3] Merge tag 'for-5.13/drivers-2021-04-27' of git://git.kernel.dk/linux-block testing commit fc0586062816559defb14c947319ef8c4c326fb3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: bdcad53c2b9b0bf70184f88b21df9241db1d39267fb4ed0951612987f86f82b4 all runs: OK # git bisect good fc0586062816559defb14c947319ef8c4c326fb3 Bisecting: 259 revisions left to test after this (roughly 8 steps) [a8b5e037d8a00d396377a97f08f5fd2a410b96a1] Merge tag 'hsi-for-5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi testing commit a8b5e037d8a00d396377a97f08f5fd2a410b96a1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: eb7befe293a6618de3b89df982c0421a398f30a09040022abbea3c55ba66eba8 run #0: OK run #1: crashed: INFO: task hung in css_free_rwork_fn run #2: crashed: INFO: task hung in css_free_rwork_fn run #3: crashed: INFO: task hung in css_free_rwork_fn run #4: crashed: INFO: task hung in call_usermodehelper_exec run #5: OK run #6: OK run #7: crashed: INFO: task hung in css_free_rwork_fn run #8: crashed: INFO: task hung in call_usermodehelper_exec run #9: OK # git bisect bad a8b5e037d8a00d396377a97f08f5fd2a410b96a1 Bisecting: 106 revisions left to test after this (roughly 7 steps) [f70865db5ff35f5ed0c7e9ef63e7cca3d4947f04] io_uring: return back safer resurrect testing commit f70865db5ff35f5ed0c7e9ef63e7cca3d4947f04 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 63f769e103e4318936f6b4439b100be68660a683720933ef69474549b2bf9520 run #0: OK run #1: OK run #2: OK run #3: crashed: INFO: task hung in call_usermodehelper_exec run #4: crashed: INFO: task hung in css_free_rwork_fn run #5: OK run #6: OK run #7: crashed: INFO: task hung in call_usermodehelper_exec run #8: crashed: INFO: task hung in call_usermodehelper_exec run #9: crashed: INFO: task hung in call_usermodehelper_exec # git bisect bad f70865db5ff35f5ed0c7e9ef63e7cca3d4947f04 Bisecting: 53 revisions left to test after this (roughly 6 steps) [66ae0d1e2d9fe6ec70e73fcfdcf4b390e271c1ac] kernel: allow fork with TIF_NOTIFY_SIGNAL pending testing commit 66ae0d1e2d9fe6ec70e73fcfdcf4b390e271c1ac compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: bce0e7a1741a84beaaf001cd50dbd824f797afb90d8f4ab883eb154e365f1034 run #0: crashed: INFO: task hung in call_usermodehelper_exec run #1: crashed: INFO: task hung in css_free_rwork_fn run #2: crashed: INFO: task hung in call_usermodehelper_exec run #3: crashed: INFO: task hung in call_usermodehelper_exec run #4: crashed: INFO: task hung in css_free_rwork_fn run #5: OK run #6: crashed: INFO: task hung in css_free_rwork_fn run #7: crashed: INFO: task hung in css_free_rwork_fn run #8: crashed: INFO: task hung in call_usermodehelper_exec run #9: crashed: INFO: task hung in css_free_rwork_fn # git bisect bad 66ae0d1e2d9fe6ec70e73fcfdcf4b390e271c1ac Bisecting: 26 revisions left to test after this (roughly 5 steps) [8dd03afe611d371b8c8a2ebeec2720de662a21dc] io_uring: refactor rsrc refnode allocation testing commit 8dd03afe611d371b8c8a2ebeec2720de662a21dc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: aae0800dd7f7d2b6e4f0ead9f16a5506c7d61bb37bb28a2dc212aeedf651d842 all runs: OK # git bisect good 8dd03afe611d371b8c8a2ebeec2720de662a21dc Bisecting: 13 revisions left to test after this (roughly 4 steps) [e1d675df1a36e33e43c614e01d9f714618ac121e] io_uring: don't init req->work fully in advance testing commit e1d675df1a36e33e43c614e01d9f714618ac121e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 210614294900b0bdf19c805518e94e7216c726a42616fcce66735bb45ee9a095 all runs: OK # git bisect good e1d675df1a36e33e43c614e01d9f714618ac121e Bisecting: 6 revisions left to test after this (roughly 3 steps) [6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916] io_uring: allocate memory for overflowed CQEs testing commit 6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 00ddd30e72b9f7762a059e80cdf2f51969e2fb727dfa62074cd5462f28fdb0d0 run #0: crashed: WARNING: ODEBUG bug in netdev_run_todo run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK reproducer seems to be flaky # git bisect bad 6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916 Bisecting: 3 revisions left to test after this (roughly 2 steps) [8c130827f417da791edb919df8cac56af30a1489] io_uring: don't alter iopoll reissue fail ret code testing commit 8c130827f417da791edb919df8cac56af30a1489 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 78969e3d3b3ba95f0978bfd1bc057c17a08ee95b25a7bf5b4201f3d6b116f4fc all runs: OK # git bisect good 8c130827f417da791edb919df8cac56af30a1489 Bisecting: 1 revision left to test after this (roughly 1 step) [9532b99bd9ca3f8f2f17b38500a8901ac1e7baee] io_uring: optimise rw complete error handling testing commit 9532b99bd9ca3f8f2f17b38500a8901ac1e7baee compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 7735cfa59724152b44294516d09d2413f9b3b9f27a21f7839a724519b570de7f all runs: OK # git bisect good 9532b99bd9ca3f8f2f17b38500a8901ac1e7baee Bisecting: 0 revisions left to test after this (roughly 0 steps) [464dca612bc6bceceafadfb4bf28f1a27ccc4632] io_uring: mask in error/nval/hangup consistently for poll testing commit 464dca612bc6bceceafadfb4bf28f1a27ccc4632 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1 kernel signature: 1413a9861525ec90f42a488f18d927dd4cda2322988679a71ec3c10bb10ad12a all runs: OK # git bisect good 464dca612bc6bceceafadfb4bf28f1a27ccc4632 6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916 is the first bad commit commit 6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916 Author: Pavel Begunkov Date: Tue Feb 23 12:40:22 2021 +0000 io_uring: allocate memory for overflowed CQEs Instead of using a request itself for overflowed CQE stashing, allocate a separate entry. The disadvantage is that the allocation may fail and it will be accounted as lost (see rings->cq_overflow), so we lose reliability in case of memory pressure if the application is driving the CQ ring into overflow. However, it opens a way for for multiple CQEs per an SQE and even generating SQE-less CQEs. Signed-off-by: Pavel Begunkov [axboe: use GFP_ATOMIC | __GFP_ACCOUNT] Signed-off-by: Jens Axboe fs/io_uring.c | 101 ++++++++++++++++++++++++++-------------------------------- 1 file changed, 46 insertions(+), 55 deletions(-) culprit signature: 00ddd30e72b9f7762a059e80cdf2f51969e2fb727dfa62074cd5462f28fdb0d0 parent signature: 1413a9861525ec90f42a488f18d927dd4cda2322988679a71ec3c10bb10ad12a Reproducer flagged being flaky revisions tested: 17, total time: 4h45m35.034521026s (build: 1h54m44.932351224s, test: 2h49m4.986268529s) first bad commit: 6c2450ae55656f6b0370bfd4cb52ec8a4ecd0916 io_uring: allocate memory for overflowed CQEs recipients (to): ["asml.silence@gmail.com" "axboe@kernel.dk" "axboe@kernel.dk" "io-uring@vger.kernel.org"] recipients (cc): ["asml.silence@gmail.com" "linux-kernel@vger.kernel.org"] crash: WARNING: ODEBUG bug in netdev_run_todo ------------[ cut here ]------------ ODEBUG: free active (active state 0) object type: timer_list hint: delayed_work_timer_fn+0x0/0x90 kernel/workqueue.c:1601 WARNING: CPU: 0 PID: 25 at lib/debugobjects.c:505 debug_print_object+0x16e/0x250 lib/debugobjects.c:505 Modules linked in: CPU: 0 PID: 25 Comm: kworker/u4:1 Not tainted 5.12.0-rc7-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: netns cleanup_net RIP: 0010:debug_print_object+0x16e/0x250 lib/debugobjects.c:505 Code: ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 af 00 00 00 48 8b 14 dd 60 80 fe 88 4c 89 ee 48 c7 c7 60 74 fe 88 e8 8a 18 84 04 <0f> 0b 83 05 a5 f3 da 08 01 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 RSP: 0018:ffffc90000dff860 EFLAGS: 00010282 RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffffffff88fe7040 RDI: fffff520001bfefe RBP: 0000000000000001 R08: 0000000000000001 R09: ffff8880ba04c1a7 R10: ffffed1017409834 R11: 0000000000000001 R12: ffffffff88acc680 R13: ffffffff88fe7aa0 R14: ffffffff8159e0c0 R15: dffffc0000000000 FS: 0000000000000000(0000) GS:ffff8880ba000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8635fdf010 CR3: 000000000a68e000 CR4: 00000000001506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __debug_check_no_obj_freed lib/debugobjects.c:987 [inline] debug_check_no_obj_freed+0x301/0x420 lib/debugobjects.c:1018 slab_free_hook mm/slub.c:1554 [inline] slab_free_freelist_hook+0x147/0x210 mm/slub.c:1600 slab_free mm/slub.c:3161 [inline] kfree+0xe5/0x7f0 mm/slub.c:4213 device_release+0x93/0x200 drivers/base/core.c:2108 kobject_cleanup lib/kobject.c:705 [inline] kobject_release lib/kobject.c:736 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x139/0x410 lib/kobject.c:753 netdev_run_todo+0x810/0xcd0 net/core/dev.c:10505 default_device_exit_batch+0x2aa/0x360 net/core/dev.c:11456 cleanup_net+0x423/0x990 net/core/net_namespace.c:595 process_one_work+0x84c/0x13b0 kernel/workqueue.c:2275 worker_thread+0x598/0xf80 kernel/workqueue.c:2421 kthread+0x36f/0x450 kernel/kthread.c:292 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294