syzbot


possible deadlock in freeze_super (2)

Status: upstream: reported C repro on 2024/12/05 16:25
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+6a15c01fcc9470b7e8f3@syzkaller.appspotmail.com
First crash: 15d, last: 14d
Bug presence (2)
Date Name Commit Repro Result
2024/12/07 linux-5.15.y (ToT) 0a51d2d4527b C [report] possible deadlock in freeze_super
2024/12/07 upstream (ToT) b5f217084ab3 C Didn't crash
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in freeze_super (2) gfs2 C error done 15 586d 803d 22/28 fixed on 2023/06/08 14:41
linux-5.15 possible deadlock in freeze_super 1 133d 133d 0/3 auto-obsoleted due to no activity on 2024/11/18 15:18
upstream possible deadlock in freeze_super gfs2 938 1358d 1489d 0/28 auto-closed as invalid on 2021/07/02 04:50

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.173-syzkaller #0 Not tainted
------------------------------------------------------
kworker/1:1H/226 is trying to acquire lock:
ffff0000c7f120e0 (&type->s_umount_key#41){+.+.}-{3:3}, at: freeze_super+0x5c/0x388 fs/super.c:1696

but task is already holding lock:
ffff80001df67c00 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at: process_one_work+0x6ac/0x11b8 kernel/workqueue.c:2285

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}:
       process_one_work+0x6d4/0x11b8 kernel/workqueue.c:2286
       worker_thread+0x910/0x1034 kernel/workqueue.c:2457
       kthread+0x37c/0x45c kernel/kthread.c:334
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870

-> #1 ((wq_completion)glock_workqueue){+.+.}-{0:0}:
       flush_workqueue+0x14c/0x11c4 kernel/workqueue.c:2830
       gfs2_gl_hash_clear+0xbc/0x2f4 fs/gfs2/glock.c:2180
       gfs2_put_super+0x5e4/0x684 fs/gfs2/super.c:624
       generic_shutdown_super+0x130/0x2f0 fs/super.c:475
       kill_block_super+0x70/0xdc fs/super.c:1427
       gfs2_kill_sb+0xc0/0xd4
       deactivate_locked_super+0xb8/0x13c fs/super.c:335
       deactivate_super+0x108/0x128 fs/super.c:366
       cleanup_mnt+0x3c0/0x474 fs/namespace.c:1143
       __cleanup_mnt+0x20/0x30 fs/namespace.c:1150
       task_work_run+0x130/0x1e4 kernel/task_work.c:188
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       do_notify_resume+0x262c/0x32b8 arch/arm64/kernel/signal.c:946
       prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
       exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
       el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:609
       el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:626
       el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #0 (&type->s_umount_key#41){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain kernel/locking/lockdep.c:3788 [inline]
       __lock_acquire+0x32d4/0x7638 kernel/locking/lockdep.c:5012
       lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5623
       down_write+0xbc/0x12c kernel/locking/rwsem.c:1551
       freeze_super+0x5c/0x388 fs/super.c:1696
       freeze_go_sync+0x128/0x31c fs/gfs2/glops.c:587
       do_xmote+0x304/0x1054 fs/gfs2/glock.c:742
       run_queue+0x3f8/0x6bc fs/gfs2/glock.c:872
       glock_work_func+0x27c/0x470 fs/gfs2/glock.c:1039
       process_one_work+0x790/0x11b8 kernel/workqueue.c:2310
       worker_thread+0x910/0x1034 kernel/workqueue.c:2457
       kthread+0x37c/0x45c kernel/kthread.c:334
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#41 --> (wq_completion)glock_workqueue --> (work_completion)(&(&gl->gl_work)->work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&(&gl->gl_work)->work));
                               lock((wq_completion)glock_workqueue);
                               lock((work_completion)(&(&gl->gl_work)->work)
);
  lock(
&type->s_umount_key
#41
);

 *** DEADLOCK ***

2 locks held by kworker/1:1H/226:
 #0: 
ffff0000c682c938
 (
(wq_completion)glock_workqueue
){+.+.}-{0:0}
, at: process_one_work+0x66c/0x11b8 kernel/workqueue.c:2283
 #1: 
ffff80001df67c00
 ((work_completion)(&(&gl->gl_work)->work)
){+.+.}-{0:0}
, at: process_one_work+0x6ac/0x11b8 kernel/workqueue.c:2285

stack backtrace:
CPU: 1 PID: 226 Comm: kworker/1:1H Not tainted 5.15.173-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: glock_workqueue glock_work_func
Call trace:
 dump_backtrace+0x0/0x530 arch/arm64/kernel/stacktrace.c:152
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:216
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
 dump_stack+0x1c/0x58 lib/dump_stack.c:113
 print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2011
 check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain kernel/locking/lockdep.c:3788 [inline]
 __lock_acquire+0x32d4/0x7638 kernel/locking/lockdep.c:5012
 lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5623
 down_write+0xbc/0x12c kernel/locking/rwsem.c:1551
 freeze_super+0x5c/0x388 fs/super.c:1696
 freeze_go_sync+0x128/0x31c fs/gfs2/glops.c:587
 do_xmote+0x304/0x1054 fs/gfs2/glock.c:742
 run_queue+0x3f8/0x6bc fs/gfs2/glock.c:872
 glock_work_func+0x27c/0x470 fs/gfs2/glock.c:1039
 process_one_work+0x790/0x11b8 kernel/workqueue.c:2310
 worker_thread+0x910/0x1034 kernel/workqueue.c:2457
 kthread+0x37c/0x45c kernel/kthread.c:334
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870

Crashes (8):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/06 03:05 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/06 01:42 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/06 00:14 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/05 22:20 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/05 21:34 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/05 18:34 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
2024/12/06 00:38 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in freeze_super
2024/12/05 16:24 linux-5.15.y 0a51d2d4527b 29f61fce .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in freeze_super
* Struck through repros no longer work on HEAD.