bisecting cause commit starting from fb0155a09b0224a7147cb07a4ce6034c8d29667f building syzkaller on 1b88c6d5c8477f1d4fb3b389443b200acc32e9a8 testing commit fb0155a09b0224a7147cb07a4ce6034c8d29667f with gcc (GCC) 8.1.0 kernel signature: 11c2edb6d0b21ae177e544419404d48ed572eea722b27fb78dddce8c493939d6 all runs: crashed: INFO: task hung in nbd_ioctl testing release v5.8 testing commit bcf876870b95592b52519ed4aafcf9d95999bc9c with gcc (GCC) 8.1.0 kernel signature: 8df03f06cc3ef2e94a58b919dc9d639dd3065e38e09170e39bfc8ee0e1cd93a1 all runs: crashed: INFO: task hung in nbd_ioctl testing release v5.7 testing commit 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 with gcc (GCC) 8.1.0 kernel signature: 64d193db4088d82799745081c193ed1e1659396d231bc535c0d2717e1067f71b all runs: crashed: INFO: task hung in nbd_ioctl testing release v5.6 testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0 kernel signature: 58bf3a6e3dd52bb683f9908fc8a10828062cc7fc5ef9773d4c03180f0fe795be all runs: crashed: INFO: task hung in nbd_ioctl testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: b2b00b41c5ac757661bfee1c5b84db5ca3008997da69abbe248a45c7ff07f620 all runs: crashed: INFO: task hung in nbd_ioctl testing release v5.4 testing commit 219d54332a09e8d8741c1e1982f5eae56099de85 with gcc (GCC) 8.1.0 kernel signature: dd7c1d597d10289a61f2b462610505bff7871a3a55a2bc1c1e9fc3a4d1826b68 run #0: crashed: possible deadlock in flush_workqueue run #1: crashed: possible deadlock in worker_thread run #2: crashed: possible deadlock in flush_workqueue run #3: crashed: possible deadlock in worker_thread run #4: crashed: possible deadlock in flush_workqueue run #5: crashed: possible deadlock in flush_workqueue run #6: crashed: possible deadlock in worker_thread run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl testing release v5.3 testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0 kernel signature: 32e39714f791e92cd7124117bfe7ee9a71ccaee4152726c827a7ff5a572558e8 all runs: OK # git bisect start 219d54332a09e8d8741c1e1982f5eae56099de85 4d856f72c10ecb060868ed10ff1b1453943fc6c8 Bisecting: 7882 revisions left to test after this (roughly 13 steps) [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi testing commit a9f8b38a071b468276a243ea3ea5a0636e848cf2 with gcc (GCC) 8.1.0 kernel signature: e76726033ec36977e8044fab8e379b0afa386a97edb11c66e8bd8c5bc3d6f831 run #0: crashed: possible deadlock in nbd_config_put run #1: crashed: possible deadlock in nbd_config_put run #2: crashed: possible deadlock in flush_workqueue run #3: crashed: possible deadlock in nbd_config_put run #4: crashed: possible deadlock in worker_thread run #5: crashed: possible deadlock in nbd_config_put run #6: crashed: possible deadlock in flush_workqueue run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl # git bisect bad a9f8b38a071b468276a243ea3ea5a0636e848cf2 Bisecting: 3920 revisions left to test after this (roughly 12 steps) [fe38bd6862074c0a2b9be7f31f043aaa70b2af5f] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit fe38bd6862074c0a2b9be7f31f043aaa70b2af5f with gcc (GCC) 8.1.0 kernel signature: 42d43f33a14b49e50b88c04af85ebef12580c9723744ab19d4e27fc4b62b31ee all runs: crashed: INFO: task hung in nbd_ioctl # git bisect bad fe38bd6862074c0a2b9be7f31f043aaa70b2af5f Bisecting: 1970 revisions left to test after this (roughly 11 steps) [fc6fd1392a8f3d5f3d722ad9c92314477c1a2a35] Merge branch 'x86-build-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit fc6fd1392a8f3d5f3d722ad9c92314477c1a2a35 with gcc (GCC) 8.1.0 kernel signature: 17db3b0ffebb860b917b4bcc809c8f3fd873afdd630f7d5abb1f7c6dd3480875 all runs: OK # git bisect good fc6fd1392a8f3d5f3d722ad9c92314477c1a2a35 Bisecting: 1254 revisions left to test after this (roughly 10 steps) [ea982ba7f79141d86eb7a440fcba6796ed718b9b] Merge tag 'mmc-v5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc testing commit ea982ba7f79141d86eb7a440fcba6796ed718b9b with gcc (GCC) 8.1.0 kernel signature: e309b139209a4250570fe694e501ad8a8cade2cb38775d041d9e3e92359fc479 run #0: crashed: INFO: task hung in nbd_ioctl run #1: crashed: INFO: task hung in nbd_ioctl run #2: crashed: INFO: task hung in nbd_ioctl run #3: crashed: INFO: task hung in nbd_config_put run #4: crashed: INFO: task hung in nbd_ioctl run #5: crashed: INFO: task hung in nbd_config_put run #6: crashed: INFO: task hung in nbd_ioctl run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl # git bisect bad ea982ba7f79141d86eb7a440fcba6796ed718b9b Bisecting: 358 revisions left to test after this (roughly 9 steps) [6dec8c15e5faa2a3c02d2e1d1b03b926b545ec0a] Merge tag 'xtensa-20190917' of git://github.com/jcmvbkbc/linux-xtensa testing commit 6dec8c15e5faa2a3c02d2e1d1b03b926b545ec0a with gcc (GCC) 8.1.0 kernel signature: e8d0ca140ff4b2ea9aff0fad1932a5a118cd1ce0145334d6cdf3d7e98e7bbe80 all runs: OK # git bisect good 6dec8c15e5faa2a3c02d2e1d1b03b926b545ec0a Bisecting: 179 revisions left to test after this (roughly 8 steps) [21fa1004ff5d749c90cef77525b73a49ef5583dc] Merge branch 'nvme-5.4' of git://git.infradead.org/nvme into for-5.4/block testing commit 21fa1004ff5d749c90cef77525b73a49ef5583dc with gcc (GCC) 8.1.0 kernel signature: f0ae1847f5e3c0d29b4014b9d3ddf208c57f1859bd2ddbe73f5891d38bbe545a all runs: crashed: INFO: task hung in nbd_ioctl # git bisect bad 21fa1004ff5d749c90cef77525b73a49ef5583dc Bisecting: 89 revisions left to test after this (roughly 7 steps) [10407ec9b42d30a6ebc49f7f84e2bb2131438699] nvme-tcp: Use protocol specific operations while reading socket testing commit 10407ec9b42d30a6ebc49f7f84e2bb2131438699 with gcc (GCC) 8.1.0 kernel signature: 709b777e18fc8952d270ff797f3ccdbe5f62e31cb31017e25ee91ce3680b3947 run #0: crashed: INFO: task hung in nbd_config_put run #1: crashed: INFO: task hung in nbd_ioctl run #2: crashed: INFO: task hung in nbd_config_put run #3: crashed: INFO: task hung in nbd_ioctl run #4: crashed: INFO: task hung in nbd_ioctl run #5: crashed: INFO: task hung in nbd_ioctl run #6: crashed: INFO: task hung in nbd_ioctl run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl # git bisect bad 10407ec9b42d30a6ebc49f7f84e2bb2131438699 Bisecting: 44 revisions left to test after this (roughly 6 steps) [5cc23ed75b629dfb0f8f7a7d0c80e0bab36b3960] block: sed-opal: Add/remove spaces testing commit 5cc23ed75b629dfb0f8f7a7d0c80e0bab36b3960 with gcc (GCC) 8.1.0 kernel signature: f897454fc83c85b276675888abbbceea4e2d355ad9b1ee03b49e2a223d728311 all runs: OK # git bisect good 5cc23ed75b629dfb0f8f7a7d0c80e0bab36b3960 Bisecting: 22 revisions left to test after this (roughly 5 steps) [97b27821b4854ca744946dae32a3f2fd55bcd5bc] writeback, memcg: Implement foreign dirty flushing testing commit 97b27821b4854ca744946dae32a3f2fd55bcd5bc with gcc (GCC) 8.1.0 kernel signature: 2260a36bd5e30b819fc99c50a7b49bf546e1fa14f7e9ce01850968840b0daef4 all runs: crashed: INFO: task hung in nbd_ioctl # git bisect bad 97b27821b4854ca744946dae32a3f2fd55bcd5bc Bisecting: 10 revisions left to test after this (roughly 4 steps) [d4b186ed227b80334abf1fe2c918c0ddc4374f38] null_blk: move duplicate code to callers testing commit d4b186ed227b80334abf1fe2c918c0ddc4374f38 with gcc (GCC) 8.1.0 kernel signature: 794d1cc0a5c14909af93ba3bee99a25a5442664ad3f34f2957acd6c00cb361a7 run #0: crashed: INFO: task hung in nbd_ioctl run #1: crashed: INFO: task hung in nbd_ioctl run #2: crashed: INFO: task hung in nbd_ioctl run #3: crashed: INFO: task hung in nbd_ioctl run #4: crashed: INFO: task hung in nbd_ioctl run #5: crashed: INFO: task hung in nbd_config_put run #6: crashed: INFO: task hung in nbd_ioctl run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl # git bisect bad d4b186ed227b80334abf1fe2c918c0ddc4374f38 Bisecting: 5 revisions left to test after this (roughly 3 steps) [887e975c4172d0d5670c39ead2f18ba1e4ec8133] nbd: add missing config put testing commit 887e975c4172d0d5670c39ead2f18ba1e4ec8133 with gcc (GCC) 8.1.0 kernel signature: 7ced696c67faa1076d969b2ac19877a5327abf0ee5773f8224dd6993074278b9 all runs: OK # git bisect good 887e975c4172d0d5670c39ead2f18ba1e4ec8133 Bisecting: 2 revisions left to test after this (roughly 2 steps) [320ea869a12cec206756207c6ca5f817ec45c7f2] block: improve the gap check in __bio_add_pc_page testing commit 320ea869a12cec206756207c6ca5f817ec45c7f2 with gcc (GCC) 8.1.0 kernel signature: fc17692615802fa314edd0409411ec2376c53d47410032934e4918b08ad8fddf all runs: crashed: INFO: task hung in nbd_ioctl # git bisect bad 320ea869a12cec206756207c6ca5f817ec45c7f2 Bisecting: 0 revisions left to test after this (roughly 1 step) [e9e006f5fcf2bab59149cb38a48a4817c1b538b4] nbd: fix max number of supported devs testing commit e9e006f5fcf2bab59149cb38a48a4817c1b538b4 with gcc (GCC) 8.1.0 kernel signature: 7d8503f23760d801bc9c0297af736e49ee9d9123b8cfdd34745f6ce4b8c10f39 run #0: crashed: INFO: task hung in nbd_ioctl run #1: crashed: INFO: task hung in nbd_ioctl run #2: crashed: INFO: task hung in nbd_config_put run #3: crashed: INFO: task hung in nbd_ioctl run #4: crashed: INFO: task hung in nbd_ioctl run #5: crashed: INFO: task hung in nbd_ioctl run #6: crashed: INFO: task hung in nbd_ioctl run #7: crashed: INFO: task hung in nbd_ioctl run #8: crashed: INFO: task hung in nbd_ioctl run #9: crashed: INFO: task hung in nbd_ioctl # git bisect bad e9e006f5fcf2bab59149cb38a48a4817c1b538b4 Bisecting: 0 revisions left to test after this (roughly 0 steps) [2da22da573481cc4837e246d0eee4d518b3f715e] nbd: fix zero cmd timeout handling v2 testing commit 2da22da573481cc4837e246d0eee4d518b3f715e with gcc (GCC) 8.1.0 kernel signature: 9bf1c698eaad81998a9be12f4882b0356c181e88438007e3a9a661f058e42b0e all runs: OK # git bisect good 2da22da573481cc4837e246d0eee4d518b3f715e e9e006f5fcf2bab59149cb38a48a4817c1b538b4 is the first bad commit commit e9e006f5fcf2bab59149cb38a48a4817c1b538b4 Author: Mike Christie Date: Sun Aug 4 14:10:06 2019 -0500 nbd: fix max number of supported devs This fixes a bug added in 4.10 with commit: commit 9561a7ade0c205bc2ee035a2ac880478dcc1a024 Author: Josef Bacik Date: Tue Nov 22 14:04:40 2016 -0500 nbd: add multi-connection support that limited the number of devices to 256. Before the patch we could create 1000s of devices, but the patch switched us from using our own thread to using a work queue which has a default limit of 256 active works. The problem is that our recv_work function sits in a loop until disconnection but only handles IO for one connection. The work is started when the connection is started/restarted, but if we end up creating 257 or more connections, the queue_work call just queues connection257+'s recv_work and that waits for connection 1 - 256's recv_work to be disconnected and that work instance completing. Instead of reverting back to kthreads, this has us allocate a workqueue_struct per device, so we can block in the work. Cc: stable@vger.kernel.org Reviewed-by: Josef Bacik Signed-off-by: Mike Christie Signed-off-by: Jens Axboe drivers/block/nbd.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) culprit signature: 7d8503f23760d801bc9c0297af736e49ee9d9123b8cfdd34745f6ce4b8c10f39 parent signature: 9bf1c698eaad81998a9be12f4882b0356c181e88438007e3a9a661f058e42b0e revisions tested: 21, total time: 4h50m56.98572831s (build: 2h18m12.279686922s, test: 2h30m30.143689431s) first bad commit: e9e006f5fcf2bab59149cb38a48a4817c1b538b4 nbd: fix max number of supported devs recipients (to): ["axboe@kernel.dk" "josef@toxicpanda.com" "mchristi@redhat.com"] recipients (cc): [] crash: INFO: task hung in nbd_ioctl INFO: task syz-executor.1:12364 blocked for more than 143 seconds. Not tainted 5.3.0-rc3-syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. syz-executor.1 D28232 12364 6824 0x00004004 Call Trace: context_switch kernel/sched/core.c:3254 [inline] __schedule+0x82a/0x18e0 kernel/sched/core.c:3880 schedule+0x8f/0x260 kernel/sched/core.c:3944 schedule_timeout+0x5a7/0x9d0 kernel/time/timer.c:1783 do_wait_for_common kernel/sched/completion.c:83 [inline] __wait_for_common kernel/sched/completion.c:104 [inline] wait_for_common+0x3ff/0x560 kernel/sched/completion.c:115 wait_for_completion+0x18/0x20 kernel/sched/completion.c:136 flush_workqueue+0x35d/0x1350 kernel/workqueue.c:2826 nbd_start_device_ioctl drivers/block/nbd.c:1254 [inline] __nbd_ioctl drivers/block/nbd.c:1329 [inline] nbd_ioctl+0x975/0xb40 drivers/block/nbd.c:1369 __blkdev_driver_ioctl block/ioctl.c:304 [inline] blkdev_ioctl+0x998/0x1820 block/ioctl.c:606 block_ioctl+0xd7/0x130 fs/block_dev.c:1994 vfs_ioctl fs/ioctl.c:46 [inline] file_ioctl fs/ioctl.c:509 [inline] do_vfs_ioctl+0x196/0x1150 fs/ioctl.c:696 ksys_ioctl+0x62/0x90 fs/ioctl.c:713 __do_sys_ioctl fs/ioctl.c:720 [inline] __se_sys_ioctl fs/ioctl.c:718 [inline] __x64_sys_ioctl+0x6e/0xb0 fs/ioctl.c:718 do_syscall_64+0xd0/0x540 arch/x86/entry/common.c:296 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45dd99 Code: 05 fd 1a 24 00 48 89 04 24 48 c7 44 24 08 10 00 00 00 e8 8a 47 fd ff 90 e8 f4 21 00 00 eb c2 cc cc 64 48 8b 0c 25 f8 ff ff ff <48> 3b 61 10 0f 86 af 01 00 00 48 83 ec 70 48 89 6c 24 68 48 8d 6c RSP: 002b:00007fb2b14d1c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000012e80 RCX: 000000000045dd99 RDX: 0000000000000000 RSI: 000000000000ab03 RDI: 0000000000000003 RBP: 000000000118c000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000118bfd4 R13: 00007ffd55e555af R14: 00007fb2b14d29c0 R15: 000000000118bfd4 INFO: lockdep is turned off. NMI backtrace for cpu 0 CPU: 0 PID: 1106 Comm: khungtaskd Not tainted 5.3.0-rc3-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x165/0x21a lib/dump_stack.c:113 nmi_cpu_backtrace.cold.0+0x49/0x86 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x194/0x1bd lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:205 [inline] watchdog+0x6a3/0xb60 kernel/hung_task.c:289 kthread+0x331/0x3f0 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x12/0x20 arch/x86/include/asm/irqflags.h:60