syzbot


possible deadlock in flush_workqueue (2)

Status: fixed on 2020/01/31 18:49
Reported-by: syzbot+a50c7541a4a55cd49b02@syzkaller.appspotmail.com
Fix commit: e7c58097793e hugetlbfs: revert "Use i_mmap_rwsem to fix page fault/truncate race"
First crash: 2002d, last: 1567d
Cause bisection: introduced by (bisect log) :
commit f547fac624be53ad8b07e9ebca7654a7827ba61b
Author: Sabrina Dubroca <sd@queasysnail.net>
Date: Fri Oct 12 14:22:47 2018 +0000

  ipv6: rate-limit probes for neighbourless routes

Crash: possible deadlock in __generic_file_fsync (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit e7c58097793ef15d58fadf190ee58738fbf447cd
Author: Mike Kravetz <mike.kravetz@oracle.com>
Date: Tue Jan 8 23:23:32 2019 +0000

  hugetlbfs: revert "Use i_mmap_rwsem to fix page fault/truncate race"

  
Discussions (2)
Title Replies (including bot) Last reply
[PATCH] fs/direct-io.c: avoid workqueue allocation race 5 (5) 2020/03/10 22:22
possible deadlock in flush_workqueue (2) 2 (6) 2019/11/07 13:42
Similar bugs (7)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in flush_workqueue 1 367d 367d 0/3 auto-obsoleted due to no activity on 2023/08/09 18:38
android-414 possible deadlock in flush_workqueue 1 1614d 1614d 0/1 auto-closed as invalid on 2020/03/10 11:29
linux-5.15 possible deadlock in flush_workqueue (2) 5 3d00h 30d 0/3 upstream: reported on 2024/03/13 19:04
linux-4.14 possible deadlock in flush_workqueue C done 15 1607d 1700d 1/1 fixed on 2019/12/18 17:48
linux-4.14 possible deadlock in flush_workqueue (2) 3 1570d 1575d 0/1 auto-closed as invalid on 2020/04/22 20:54
upstream possible deadlock in flush_workqueue net C 73762 2017d 2059d 11/26 fixed on 2018/10/11 14:33
linux-4.19 possible deadlock in flush_workqueue 3 1628d 1639d 0/1 auto-closed as invalid on 2020/02/25 05:02

Sample crash report:
block nbd0: Receive control failed (result -107)
block nbd0: shutting down sockets
============================================
WARNING: possible recursive locking detected
5.5.0-rc1-syzkaller #0 Not tainted
--------------------------------------------
kworker/u5:0/1600 is trying to acquire lock:
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: flush_workqueue+0xf7/0x14c0 kernel/workqueue.c:2772

but task is already holding lock:
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: set_work_data kernel/workqueue.c:615 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:642 [inline]
ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2235

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock((wq_completion)knbd0-recv);
  lock((wq_completion)knbd0-recv);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by kworker/u5:0/1600:
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: set_work_data kernel/workqueue.c:615 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:642 [inline]
 #0: ffff8880a7e93d28 ((wq_completion)knbd0-recv){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2235
 #1: ffffc900057c7dc0 ((work_completion)(&args->work)){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2239
 #2: ffff8880a1984978 (&nbd->config_lock){+.+.}, at: refcount_dec_and_mutex_lock lib/refcount.c:118 [inline]
 #2: ffff8880a1984978 (&nbd->config_lock){+.+.}, at: refcount_dec_and_mutex_lock+0x55/0xe0 lib/refcount.c:113

stack backtrace:
CPU: 0 PID: 1600 Comm: kworker/u5:0 Not tainted 5.5.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: knbd0-recv recv_work
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 print_deadlock_bug kernel/locking/lockdep.c:2371 [inline]
 check_deadlock kernel/locking/lockdep.c:2412 [inline]
 validate_chain kernel/locking/lockdep.c:2955 [inline]
 __lock_acquire.cold+0x15d/0x385 kernel/locking/lockdep.c:3955
 lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4485
 flush_workqueue+0x126/0x14c0 kernel/workqueue.c:2775
 drain_workqueue+0x1b4/0x3d0 kernel/workqueue.c:2940
 destroy_workqueue+0x80/0x700 kernel/workqueue.c:4352
 nbd_config_put+0x3dd/0x870 drivers/block/nbd.c:1210
 recv_work+0x19b/0x200 drivers/block/nbd.c:792
 process_one_work+0x9af/0x1740 kernel/workqueue.c:2264
 worker_thread+0x98/0xe40 kernel/workqueue.c:2410
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Crashes (256):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/12/15 07:18 upstream 07c4b9e9f71a eef6e580 .config console log report syz C ci-upstream-kasan-gce-root
2019/10/22 15:10 upstream 3b7c59a1950c c59a7cd8 .config console log report syz C ci-upstream-kasan-gce-selinux-root
2019/10/17 04:36 upstream bc88f85c6c09 8c88c9c1 .config console log report syz C ci-upstream-kasan-gce-root
2019/10/16 16:20 upstream 3b1f00aceb7a d4ea592f .config console log report syz C ci-upstream-kasan-gce-smack-root
2019/10/12 05:20 upstream 9e208aa06c21 426631dd .config console log report syz C ci-upstream-kasan-gce-selinux-root
2019/10/08 19:43 upstream eda57a0e4299 b1ebbfef .config console log report syz C ci-upstream-kasan-gce-root
2019/10/02 15:17 upstream 54ecb8f7028c 2e29b534 .config console log report syz C ci-upstream-kasan-gce-root
2019/09/22 22:42 upstream f7c3bf8fa7e5 d96e88f3 .config console log report syz C ci-upstream-kasan-gce-selinux-root
2019/09/22 18:22 upstream f7c3bf8fa7e5 d96e88f3 .config console log report syz C ci-upstream-kasan-gce-root
2019/09/21 21:21 upstream 227c3e9eb5cf d96e88f3 .config console log report syz C ci-upstream-kasan-gce-smack-root
2018/10/23 09:41 upstream ca9eb48fe01f ecb386fe .config console log report syz C ci-upstream-kasan-gce-root
2019/10/20 12:49 linux-next c4b9850b3676 8c88c9c1 .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2019/09/26 14:40 linux-next d47175169c28 24d405a3 .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2019/09/21 05:07 linux-next b5b3bd898ba9 d96e88f3 .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2018/11/10 14:00 linux-next 442b8cea2477 f9815aaf .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2019/09/23 01:58 upstream f7c3bf8fa7e5 d96e88f3 .config console log report syz ci-upstream-kasan-gce-root
2018/11/21 03:45 upstream 06e68fed3282 9aca6b52 .config console log report syz ci-upstream-kasan-gce-selinux-root
2018/10/24 16:21 upstream 638820d8da8e a8292de9 .config console log report syz ci-upstream-kasan-gce-smack-root
2018/10/24 11:23 upstream 44786880df19 a8292de9 .config console log report syz ci-upstream-kasan-gce-root
2018/10/19 16:13 upstream 91b15613ce7f 9aba67b5 .config console log report syz ci-upstream-kasan-gce-root
2019/10/19 19:32 linux-next c4b9850b3676 8c88c9c1 .config console log report syz ci-upstream-linux-next-kasan-gce-root
2018/11/20 23:50 linux-next 442b8cea2477 9aca6b52 .config console log report syz ci-upstream-linux-next-kasan-gce-root
2019/12/16 11:41 upstream 07c4b9e9f71a eef6e580 .config console log report ci-upstream-kasan-gce-selinux-root
2019/12/15 04:29 upstream 07c4b9e9f71a eef6e580 .config console log report ci-upstream-kasan-gce-selinux-root
2019/12/07 13:56 upstream eea2d5da29e3 85f26751 .config console log report ci-upstream-kasan-gce-root
2019/11/27 09:24 upstream 89d57dddd7d3 1048481f .config console log report ci-upstream-kasan-gce-root
2019/11/26 05:54 upstream 0be0ee71816b f746151a .config console log report ci-upstream-kasan-gce-smack-root
2019/11/25 20:06 upstream 219d54332a09 371caf77 .config console log report ci-upstream-kasan-gce-root
2019/11/24 21:41 upstream 6b8a79467876 598ca6c8 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/22 14:54 upstream 81429eb8d9ca 598ca6c8 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/20 21:08 upstream c74386d50fba 8098ea0f .config console log report ci-upstream-kasan-gce-smack-root
2019/11/18 13:58 upstream af42d3466bdc 1daed50a .config console log report ci-upstream-kasan-gce-smack-root
2019/11/17 04:16 upstream 6c9594bdd474 d5696d51 .config console log report ci-upstream-kasan-gce-root
2019/11/16 14:41 upstream 6c9594bdd474 d5696d51 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/13 14:36 upstream 0e3f1ad80fc8 048f2d49 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/13 12:06 upstream 0e3f1ad80fc8 048f2d49 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/12 20:34 upstream 100d46bd72ec 048f2d49 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/12 16:23 upstream de620fb99ef2 048f2d49 .config console log report ci-upstream-kasan-gce-selinux-root
2019/11/10 16:10 upstream 00aff6836241 dc438b91 .config console log report ci-upstream-kasan-gce-root
2019/11/10 10:58 upstream 00aff6836241 dc438b91 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/06 04:34 upstream 26bc67213424 bc2c6e45 .config console log report ci-upstream-kasan-gce-smack-root
2019/11/01 12:51 upstream e472c64aa4fa a41ca8fa .config console log report ci-upstream-kasan-gce-selinux-root
2019/11/01 00:32 upstream e472c64aa4fa a41ca8fa .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/31 11:55 upstream e472c64aa4fa a41ca8fa .config console log report ci-upstream-kasan-gce-smack-root
2019/10/28 02:47 upstream d6d5df1db6e9 25bb509e .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/27 12:52 upstream 5a1e843c66fa 25bb509e .config console log report ci-upstream-kasan-gce-smack-root
2019/10/27 06:33 upstream f877bee5ea0b 25bb509e .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/26 18:49 upstream f877bee5ea0b 25bb509e .config console log report ci-upstream-kasan-gce-smack-root
2019/10/26 13:07 upstream 8caacaad78b6 413926c5 .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/23 21:12 upstream 13b86bc4cd64 b602d64b .config console log report ci-upstream-kasan-gce-root
2019/10/22 05:30 upstream 7d194c2100ad c59a7cd8 .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/19 01:53 upstream b9959c7a347d 8c88c9c1 .config console log report ci-upstream-kasan-gce-root
2019/10/18 10:59 upstream 0e2adab6cf28 8c88c9c1 .config console log report ci-upstream-kasan-gce-root
2019/10/16 22:34 upstream bc88f85c6c09 8c88c9c1 .config console log report ci-upstream-kasan-gce-smack-root
2019/10/16 01:38 upstream 3b1f00aceb7a d4ea592f .config console log report ci-upstream-kasan-gce-root
2019/10/14 05:49 upstream d4615e5a4680 2f661ec4 .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/11 19:48 upstream 9e208aa06c21 426631dd .config console log report ci-upstream-kasan-gce-smack-root
2019/10/11 13:20 upstream fb20da6af705 1a3bad90 .config console log report ci-upstream-kasan-gce-smack-root
2019/10/10 21:30 upstream fb20da6af705 a4efa8c0 .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/08 22:24 upstream d5001955c281 b1ebbfef .config console log report ci-upstream-kasan-gce-selinux-root
2018/12/25 20:03 upstream 8fe28cb58bcb 8a41a0ad .config console log report ci-upstream-kasan-gce
2018/12/21 03:08 upstream 9097a058d49e 2b497001 .config console log report ci-upstream-kasan-gce-386
2019/12/28 02:11 linux-next 7ddd09fc4b74 be5c2c81 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/12/20 08:36 linux-next 7ddd09fc4b74 e30cbdae .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/26 11:21 linux-next 131b7b67e6c2 f746151a .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/12 11:22 linux-next fc6d6db1df2c 048f2d49 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/04 18:25 linux-next 49afce6d47fe 76630fc9 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/03 13:14 linux-next 49afce6d47fe c9610487 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/10/27 19:27 linux-next 139c2d13c258 25bb509e .config console log report ci-upstream-linux-next-kasan-gce-root
2019/10/22 04:10 linux-next a6fcdcd94927 c59a7cd8 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/10/12 17:28 linux-next 8ada228ac7ed 426631dd .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.