possible deadlock in freeze

possible deadlock in freeze_super
Status: upstream: reported on 2020/11/22 21:15
Reported-by: syzbot+f60646ed0fee22ac695d@syzkaller.appspotmail.com
First crash: 147d, last: 12d

Sample crash report:

======================================================
WARNING: possible circular locking dependency detected
5.12.0-rc5-syzkaller #0 Not tainted
------------------------------------------------------
kworker/1:1H/2006 is trying to acquire lock:
ffff88806e28e0e0 (&type->s_umount_key#76){+.+.}-{3:3}, at: freeze_super+0x44/0x3c0 fs/super.c:1664

but task is already holding lock:
ffffc90006f3fd78 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at: process_one_work+0x733/0xfd0 kernel/workqueue.c:2250

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}:
       lock_acquire+0x126/0x650 kernel/locking/lockdep.c:5510
       process_one_work+0x75d/0xfd0 kernel/workqueue.c:2251
       worker_thread+0xac1/0x1300 kernel/workqueue.c:2421
       kthread+0x39a/0x3c0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #1 ((wq_completion)glock_workqueue){+.+.}-{0:0}:
       lock_acquire+0x126/0x650 kernel/locking/lockdep.c:5510
       flush_workqueue+0x120/0x1620 kernel/workqueue.c:2786
       gfs2_gl_hash_clear+0x53/0x270 fs/gfs2/glock.c:1996
       gfs2_put_super+0x7b3/0x820 fs/gfs2/super.c:688
       generic_shutdown_super+0x120/0x2a0 fs/super.c:464
       kill_block_super+0x79/0xd0 fs/super.c:1394
       deactivate_locked_super+0xa7/0xf0 fs/super.c:335
       cleanup_mnt+0x462/0x510 fs/namespace.c:1136
       task_work_run+0x146/0x1c0 kernel/task_work.c:140
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:174 [inline]
       exit_to_user_mode_prepare+0x10b/0x1e0 kernel/entry/common.c:208
       __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
       syscall_exit_to_user_mode+0x26/0x70 kernel/entry/common.c:301
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #0 (&type->s_umount_key#76){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:2936 [inline]
       check_prevs_add+0x339/0x5c20 kernel/locking/lockdep.c:3059
       validate_chain kernel/locking/lockdep.c:3674 [inline]
       __lock_acquire+0x41f7/0x5e60 kernel/locking/lockdep.c:4900
       lock_acquire+0x126/0x650 kernel/locking/lockdep.c:5510
       down_write+0x5b/0x120 kernel/locking/rwsem.c:1406
       freeze_super+0x44/0x3c0 fs/super.c:1664
       freeze_go_sync+0x17b/0x3a0 fs/gfs2/glops.c:574
       do_xmote+0x6b7/0x14c0 fs/gfs2/glock.c:630
       glock_work_func+0x2c3/0x450 fs/gfs2/glock.c:934
       process_one_work+0x789/0xfd0 kernel/workqueue.c:2275
       worker_thread+0xac1/0x1300 kernel/workqueue.c:2421
       kthread+0x39a/0x3c0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#76 --> (wq_completion)glock_workqueue --> (work_completion)(&(&gl->gl_work)->work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&(&gl->gl_work)->work));
                               lock((wq_completion)glock_workqueue);
                               lock((work_completion)(&(&gl->gl_work)->work));
  lock(&type->s_umount_key#76);

 *** DEADLOCK ***

2 locks held by kworker/1:1H/2006:
 #0: ffff888142b53d38 ((wq_completion)glock_workqueue){+.+.}-{0:0}, at: process_one_work+0x6f4/0xfd0 kernel/workqueue.c:2248
 #1: ffffc90006f3fd78 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at: process_one_work+0x733/0xfd0 kernel/workqueue.c:2250

stack backtrace:
CPU: 1 PID: 2006 Comm: kworker/1:1H Not tainted 5.12.0-rc5-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: glock_workqueue glock_work_func
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x176/0x24e lib/dump_stack.c:120
 print_circular_bug+0xadf/0xd80 kernel/locking/lockdep.c:2005
 check_noncircular+0x221/0x2d0 kernel/locking/lockdep.c:2127
 check_prev_add kernel/locking/lockdep.c:2936 [inline]
 check_prevs_add+0x339/0x5c20 kernel/locking/lockdep.c:3059
 validate_chain kernel/locking/lockdep.c:3674 [inline]
 __lock_acquire+0x41f7/0x5e60 kernel/locking/lockdep.c:4900
 lock_acquire+0x126/0x650 kernel/locking/lockdep.c:5510
 down_write+0x5b/0x120 kernel/locking/rwsem.c:1406
 freeze_super+0x44/0x3c0 fs/super.c:1664
 freeze_go_sync+0x17b/0x3a0 fs/gfs2/glops.c:574
 do_xmote+0x6b7/0x14c0 fs/gfs2/glock.c:630
 glock_work_func+0x2c3/0x450 fs/gfs2/glock.c:934
 process_one_work+0x789/0xfd0 kernel/workqueue.c:2275
 worker_thread+0xac1/0x1300 kernel/workqueue.c:2421
 kthread+0x39a/0x3c0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Crashes (938):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Title
ci-upstream-kasan-gce-smack-root	2021/04/03 04:08	upstream	d93a0d43	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/04/03 02:59	upstream	d93a0d43	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/04/02 13:58	upstream	1678e493	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/04/02 12:53	upstream	1678e493	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/04/01 20:54	upstream	ffd9fb54	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/04/01 08:15	upstream	d19cc4bf	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/31 13:07	upstream	5e46d1b7	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/30 23:32	upstream	2bb25b3a	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream	2021/03/30 00:18	upstream	1e43c377	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/29 23:02	upstream	1e43c377	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/03/29 21:30	upstream	1e43c377	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/29 14:28	upstream	a5e13c6d	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/03/29 13:17	upstream	a5e13c6d	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream	2021/03/29 02:05	upstream	36a14638	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/28 22:12	upstream	81b1d39f	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/27 08:55	upstream	0f4498ce	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/26 18:18	upstream	db24726b	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/03/26 16:31	upstream	db24726b	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/26 09:42	upstream	db24726b	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/26 08:24	upstream	db24726b	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/03/25 18:09	upstream	e1381380	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/25 16:30	upstream	e1381380	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/24 22:48	upstream	4ee998b0	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-selinux-root	2021/03/24 21:43	upstream	4ee998b0	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream	2021/03/24 18:52	upstream	7acac4b3	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream	2021/03/24 16:16	upstream	7acac4b3	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/24 14:28	upstream	7acac4b3	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/23 21:49	upstream	7acac4b3	e613994b	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/23 20:22	upstream	7acac4b3	e613994b	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/23 15:36	upstream	84196390	e613994b	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream	2021/03/23 01:29	upstream	84196390	8092f30d	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/22 21:43	upstream	84196390	8092f30d	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/22 20:22	upstream	84196390	8092f30d	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/22 19:14	upstream	0d02ec6b	8092f30d	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-smack-root	2021/03/22 12:49	upstream	0d02ec6b	bea32f74	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2021/03/22 11:13	upstream	0d02ec6b	bea32f74	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/30 01:25	upstream	1e43c377	6a81331a	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/29 16:34	upstream	a5e13c6d	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/27 03:39	upstream	0f4498ce	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/26 13:44	upstream	db24726b	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/26 11:14	upstream	db24726b	a8529b82	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/26 01:03	upstream	00232240	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/25 12:14	upstream	e1381380	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/25 10:42	upstream	e1381380	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/25 08:51	upstream	e1381380	6a383ecf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/24 20:01	upstream	4ee998b0	607e3baf	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/23 22:00	upstream	7acac4b3	e613994b	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/22 23:22	upstream	84196390	8092f30d	.config	log	report	info	possible deadlock in freeze_super
ci-qemu-upstream-386	2021/03/22 07:20	upstream	0d02ec6b	bea32f74	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-linux-next-kasan-gce-root	2021/03/13 19:36	linux-next	d98f554b	4a003785	.config	log	report	info	possible deadlock in freeze_super
ci-upstream-kasan-gce-root	2020/11/22 19:00	upstream	a349e4c6	0d27f508	.config	log	report	info
ci-upstream-kasan-gce-smack-root	2020/11/18 21:12	upstream	c2e7554e	0767f13f	.config	log	report	info
ci-upstream-linux-next-kasan-gce-root	2021/01/17 14:00	linux-next	b3a3cbde	813be542	.config	log	report	info