syzbot


possible deadlock in p9_write_work

Status: closed as invalid on 2022/05/26 10:41
Subsystems: v9fs
[Documentation on labels]
Reported-by: syzbot+bde0f89deacca7c765b8@syzkaller.appspotmail.com
First crash: 750d, last: 722d
Duplicate bugs (2)
duplicates (2):
Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
possible deadlock in iterate_supers ext4 42 723d 747d 0/26 closed as dup on 2022/03/30 22:17
possible deadlock in deactivate_super ext4 C unreliable 9 727d 747d 0/26 closed as dup on 2022/03/30 22:17
Discussions (5)
Title Replies (including bot) Last reply
How to convert drivers/gpu/drm/i915/ to use local workqueue? 10 (10) 2022/06/30 15:25
[PATCH v5] workqueue: Wrap flush_workqueue() using a macro 3 (3) 2022/06/07 17:08
[PATCH] checkpatch: warn about flushing system-wide workqueues 23 (23) 2022/05/27 06:21
Re: [syzbot] possible deadlock in p9_write_work 1 (1) 2022/03/30 23:43
[syzbot] possible deadlock in p9_write_work 6 (7) 2022/03/30 02:49

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.18.0-rc3-next-20220420-syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:16/7700 is trying to acquire lock:
ffff88807e874460 (sb_writers#3){.+.+}-{0:0}, at: p9_fd_write net/9p/trans_fd.c:428 [inline]
ffff88807e874460 (sb_writers#3){.+.+}-{0:0}, at: p9_write_work+0x25e/0xca0 net/9p/trans_fd.c:479

but task is already holding lock:
ffffc90004bffda8 ((work_completion)(&m->wq)){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 ((work_completion)(&m->wq)){+.+.}-{0:0}:
       process_one_work+0x905/0x1610 kernel/workqueue.c:2265
       worker_thread+0x665/0x1080 kernel/workqueue.c:2436
       kthread+0x2e9/0x3a0 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

-> #2 ((wq_completion)events){+.+.}-{0:0}:
       flush_workqueue+0x164/0x1440 kernel/workqueue.c:2831
       flush_scheduled_work include/linux/workqueue.h:583 [inline]
       ext4_put_super+0x99/0x1150 fs/ext4/super.c:1202
       generic_shutdown_super+0x14c/0x400 fs/super.c:462
       kill_block_super+0x97/0xf0 fs/super.c:1394
       deactivate_locked_super+0x94/0x160 fs/super.c:332
       deactivate_super+0xad/0xd0 fs/super.c:363
       cleanup_mnt+0x3a2/0x540 fs/namespace.c:1186
       task_work_run+0xdd/0x1a0 kernel/task_work.c:164
       resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:169 [inline]
       exit_to_user_mode_prepare+0x23c/0x250 kernel/entry/common.c:201
       __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
       syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294
       do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #1 (&type->s_umount_key#32){++++}-{3:3}:
       down_read+0x98/0x440 kernel/locking/rwsem.c:1489
       iterate_supers+0xdb/0x290 fs/super.c:692
       drop_caches_sysctl_handler+0xdb/0x110 fs/drop_caches.c:62
       proc_sys_call_handler+0x4a1/0x6e0 fs/proc/proc_sysctl.c:604
       call_write_iter include/linux/fs.h:2059 [inline]
       do_iter_readv_writev+0x3d1/0x640 fs/read_write.c:726
       do_iter_write+0x182/0x700 fs/read_write.c:852
       vfs_iter_write+0x70/0xa0 fs/read_write.c:893
       iter_file_splice_write+0x723/0xc70 fs/splice.c:689
       do_splice_from fs/splice.c:767 [inline]
       direct_splice_actor+0x110/0x180 fs/splice.c:936
       splice_direct_to_actor+0x34b/0x8c0 fs/splice.c:891
       do_splice_direct+0x1a7/0x270 fs/splice.c:979
       do_sendfile+0xae0/0x1240 fs/read_write.c:1246
       __do_sys_sendfile64 fs/read_write.c:1305 [inline]
       __se_sys_sendfile64 fs/read_write.c:1297 [inline]
       __x64_sys_sendfile64+0x149/0x210 fs/read_write.c:1297
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #0 (sb_writers#3){.+.+}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3095 [inline]
       check_prevs_add kernel/locking/lockdep.c:3214 [inline]
       validate_chain kernel/locking/lockdep.c:3829 [inline]
       __lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
       lock_acquire kernel/locking/lockdep.c:5665 [inline]
       lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
       percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
       __sb_start_write include/linux/fs.h:1702 [inline]
       sb_start_write include/linux/fs.h:1777 [inline]
       file_start_write include/linux/fs.h:2783 [inline]
       kernel_write fs/read_write.c:564 [inline]
       kernel_write+0x2ac/0x540 fs/read_write.c:555
       p9_fd_write net/9p/trans_fd.c:428 [inline]
       p9_write_work+0x25e/0xca0 net/9p/trans_fd.c:479
       process_one_work+0x996/0x1610 kernel/workqueue.c:2289
       worker_thread+0x665/0x1080 kernel/workqueue.c:2436
       kthread+0x2e9/0x3a0 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

other info that might help us debug this:

Chain exists of:
  sb_writers#3 --> (wq_completion)events --> (work_completion)(&m->wq)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&m->wq));
                               lock((wq_completion)events);
                               lock((work_completion)(&m->wq));
  lock(sb_writers#3);

 *** DEADLOCK ***

2 locks held by kworker/0:16/7700:
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:636 [inline]
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:663 [inline]
 #0: ffff888010c64d38 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x87a/0x1610 kernel/workqueue.c:2260
 #1: ffffc90004bffda8 ((work_completion)(&m->wq)){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264

stack backtrace:
CPU: 0 PID: 7700 Comm: kworker/0:16 Not tainted 5.18.0-rc3-next-20220420-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events p9_write_work
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2175
 check_prev_add kernel/locking/lockdep.c:3095 [inline]
 check_prevs_add kernel/locking/lockdep.c:3214 [inline]
 validate_chain kernel/locking/lockdep.c:3829 [inline]
 __lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
 lock_acquire kernel/locking/lockdep.c:5665 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
 percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
 __sb_start_write include/linux/fs.h:1702 [inline]
 sb_start_write include/linux/fs.h:1777 [inline]
 file_start_write include/linux/fs.h:2783 [inline]
 kernel_write fs/read_write.c:564 [inline]
 kernel_write+0x2ac/0x540 fs/read_write.c:555
 p9_fd_write net/9p/trans_fd.c:428 [inline]
 p9_write_work+0x25e/0xca0 net/9p/trans_fd.c:479
 process_one_work+0x996/0x1610 kernel/workqueue.c:2289
 worker_thread+0x665/0x1080 kernel/workqueue.c:2436
 kthread+0x2e9/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>

Crashes (22):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/04/25 11:09 linux-next f1244c81da13 c889aef9 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/22 17:53 linux-next f1244c81da13 131df97d .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/20 09:39 linux-next f1244c81da13 7d7bc738 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/15 12:42 linux-next 40354149f4d7 8bcc32a6 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/15 11:01 linux-next 40354149f4d7 8bcc32a6 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/13 19:29 linux-next c97e430ab553 b17b2923 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/13 18:09 linux-next c97e430ab553 b17b2923 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/12 19:53 linux-next d0c745e7b2d6 dacb3f1c .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/11 15:44 linux-next d12d7e1cfe38 af01ee7d .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/11 13:30 linux-next d12d7e1cfe38 af01ee7d .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/07 10:32 linux-next 2e9a9857569e c6ff3e05 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/06 20:54 linux-next 109f6d10ec17 97582466 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/06 08:03 linux-next 109f6d10ec17 0127c10f .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/05 17:52 linux-next 3ccc91681259 0127c10f .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/05 16:14 linux-next 3ccc91681259 0127c10f .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/04 16:04 linux-next 696206280c5e 5915c2cb .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/04 03:41 linux-next 696206280c5e 79a2a8fc .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/02 02:26 linux-next e5071887cd22 79a2a8fc .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/04/01 04:19 linux-next e5071887cd22 68fc921a .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/03/31 02:19 linux-next a67ba3cf9551 9d49f3a7 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/03/31 00:09 linux-next a67ba3cf9551 9d49f3a7 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
2022/03/28 08:57 linux-next 8515d05bf6bc 89bc8608 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in p9_write_work
* Struck through repros no longer work on HEAD.