syzbot


possible deadlock in deactivate_super

Status: closed as dup on 2022/03/30 22:17
Subsystems: ext4
[Documentation on labels]
Reported-by: syzbot+df9d057e38be100ccbbe@syzkaller.appspotmail.com
First crash: 756d, last: 736d
Cause bisection: introduced by (bisect log) [merge commit]:
commit 6a711b79aeed5e82c7b60450a8036702972cc48c
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Tue Mar 29 22:43:45 2022 +0000

  Merge branch 'for-next' of git://git.kernel.dk/linux-block.git

Crash: possible deadlock in deactivate_super (log)
Repro: C syz .config
  
Duplicate of
Title Repro Cause bisect Fix bisect Count Last Reported
possible deadlock in p9_write_work v9fs 22 730d 757d
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] possible deadlock in deactivate_super 0 (1) 2022/03/30 17:36

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.18.0-rc1-next-20220408-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor343/3615 is trying to acquire lock:
ffff888010c64d38 (
(wq_completion)events){+.+.}-{0:0}
, at: flush_workqueue+0x135/0x1440 kernel/workqueue.c:2828

but task is already holding lock:
ffff88801dfd40e0
 (&type->s_umount_key
#31){++++}-{3:3}
, at: deactivate_super+0xa5/0xd0 fs/super.c:362

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (
&type->s_umount_key#31
){++++}-{3:3}
:
       down_read+0x98/0x440 kernel/locking/rwsem.c:1489
       iterate_supers+0xdb/0x290 fs/super.c:692
       drop_caches_sysctl_handler+0xdb/0x110 fs/drop_caches.c:62
       proc_sys_call_handler+0x4a1/0x6e0 fs/proc/proc_sysctl.c:604
       call_write_iter include/linux/fs.h:2067 [inline]
       do_iter_readv_writev+0x3d1/0x640 fs/read_write.c:726
       do_iter_write+0x182/0x700 fs/read_write.c:852
       vfs_iter_write+0x70/0xa0 fs/read_write.c:893
       iter_file_splice_write+0x723/0xc70 fs/splice.c:689
       do_splice_from fs/splice.c:767 [inline]
       direct_splice_actor+0x110/0x180 fs/splice.c:936
       splice_direct_to_actor+0x34b/0x8c0 fs/splice.c:891
       do_splice_direct+0x1a7/0x270 fs/splice.c:979
       do_sendfile+0xae0/0x1240 fs/read_write.c:1246
       __do_sys_sendfile64 fs/read_write.c:1305 [inline]
       __se_sys_sendfile64 fs/read_write.c:1297 [inline]
       __x64_sys_sendfile64+0x149/0x210 fs/read_write.c:1297
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #2 (
sb_writers#3
){.+.+}-{0:0}
:
       percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
       __sb_start_write include/linux/fs.h:1710 [inline]
       sb_start_write include/linux/fs.h:1785 [inline]
       file_start_write include/linux/fs.h:2791 [inline]
       kernel_write fs/read_write.c:564 [inline]
       kernel_write+0x2ac/0x540 fs/read_write.c:555
       p9_fd_write net/9p/trans_fd.c:428 [inline]
       p9_write_work+0x25e/0xca0 net/9p/trans_fd.c:479
       process_one_work+0x996/0x1610 kernel/workqueue.c:2289
       worker_thread+0x665/0x1080 kernel/workqueue.c:2436
       kthread+0x2e9/0x3a0 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

-> #1 ((work_completion)(&m->wq)){+.+.}-{0:0}:
       process_one_work+0x905/0x1610 kernel/workqueue.c:2265
       worker_thread+0x665/0x1080 kernel/workqueue.c:2436
       kthread+0x2e9/0x3a0 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298

-> #0 ((wq_completion)events){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3095 [inline]
       check_prevs_add kernel/locking/lockdep.c:3214 [inline]
       validate_chain kernel/locking/lockdep.c:3829 [inline]
       __lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
       lock_acquire kernel/locking/lockdep.c:5665 [inline]
       lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
       flush_workqueue+0x164/0x1440 kernel/workqueue.c:2831
       flush_scheduled_work include/linux/workqueue.h:583 [inline]
       ext4_put_super+0x99/0x1150 fs/ext4/super.c:1202
       generic_shutdown_super+0x14c/0x400 fs/super.c:462
       kill_block_super+0x97/0xf0 fs/super.c:1394
       deactivate_locked_super+0x94/0x160 fs/super.c:332
       deactivate_super+0xad/0xd0 fs/super.c:363
       cleanup_mnt+0x3a2/0x540 fs/namespace.c:1186
       task_work_run+0xdd/0x1a0 kernel/task_work.c:164
       resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:169 [inline]
       exit_to_user_mode_prepare+0x23c/0x250 kernel/entry/common.c:201
       __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
       syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294
       do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x44/0xae

other info that might help us debug this:

Chain exists of:
  (wq_completion)events --> sb_writers#3 --> &type->s_umount_key#31

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&type->s_umount_key#31);
                               lock(sb_writers#3);
                               lock(&type->s_umount_key#31);
  lock((wq_completion)events);

 *** DEADLOCK ***

1 lock held by syz-executor343/3615:
 #0: ffff88801dfd40e0 (&type->s_umount_key#31){++++}-{3:3}, at: deactivate_super+0xa5/0xd0 fs/super.c:362

stack backtrace:
CPU: 0 PID: 3615 Comm: syz-executor343 Not tainted 5.18.0-rc1-next-20220408-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2175
 check_prev_add kernel/locking/lockdep.c:3095 [inline]
 check_prevs_add kernel/locking/lockdep.c:3214 [inline]
 validate_chain kernel/locking/lockdep.c:3829 [inline]
 __lock_acquire+0x2abe/0x5660 kernel/locking/lockdep.c:5053
 lock_acquire kernel/locking/lockdep.c:5665 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
 flush_workqueue+0x164/0x1440 kernel/workqueue.c:2831
 flush_scheduled_work include/linux/workqueue.h:583 [inline]
 ext4_put_super+0x99/0x1150 fs/ext4/super.c:1202
 generic_shutdown_super+0x14c/0x400 fs/super.c:462
 kill_block_super+0x97/0xf0 fs/super.c:1394
 deactivate_locked_super+0x94/0x160 fs/super.c:332
 deactivate_super+0xad/0xd0 fs/super.c:363
 cleanup_mnt+0x3a2/0x540 fs/namespace.c:1186
 task_work_run+0xdd/0x1a0 kernel/task_work.c:164
 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:169 [inline]
 exit_to_user_mode_prepare+0x23c/0x250 kernel/entry/common.c:201
 __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
 syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:294
 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7ff53b621407
Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe485f4d68 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007ff53b621407
RDX: 00007ffe485f4e29 RSI: 000000000000000a RDI: 00007ffe485f4e20
RBP: 00007ffe485f4e20 R08: 00000000ffffffff R09: 00007ffe485f4c00
R10: 0000555556fed653 R11: 0000000000000202 R12: 00007ffe485f5ee0
R13: 0000555556fed5f0 R14: 00007ffe485f4d90 R15: 00007ffe485f5f00
 </TASK>

Crashes (9):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/04/09 07:34 linux-next ff511c1c68a5 e22c3da3 .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/02 05:59 linux-next e5071887cd22 79a2a8fc .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/03/30 08:37 linux-next a67ba3cf9551 6bdac766 .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/19 21:03 linux-next 634de1db0e9b c334415e .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/14 13:55 linux-next 40354149f4d7 b17b2923 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/13 09:46 linux-next c97e430ab553 faabdb86 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/09 20:37 linux-next ff511c1c68a5 e22c3da3 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/04/02 03:52 linux-next e5071887cd22 79a2a8fc .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
2022/03/31 14:26 linux-next fdcbcd1348f4 c4c32d8c .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in deactivate_super
* Struck through repros no longer work on HEAD.