syzbot


possible deadlock in freeze_super (2)

Status: fixed on 2023/06/08 14:41
Subsystems: gfs2
[Documentation on labels]
Reported-by: syzbot+be899d4f10b2a9522dce@syzkaller.appspotmail.com
Fix commit: b66f723bb552 gfs2: Improve gfs2_make_fs_rw error handling
First crash: 773d, last: 556d
Cause bisection: failed (error log, bisect log)
  
Fix bisection: fixed by (bisect log) :
commit b66f723bb552ad59c2acb5d45ea45c890f84498b
Author: Andreas Gruenbacher <agruenba@redhat.com>
Date: Tue Jan 31 14:06:53 2023 +0000

  gfs2: Improve gfs2_make_fs_rw error handling

  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] possible deadlock in freeze_super (2) 1 (4) 2023/04/03 06:23
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in freeze_super 1 102d 102d 0/3 auto-obsoleted due to no activity on 2024/11/18 15:18
upstream possible deadlock in freeze_super gfs2 938 1328d 1459d 0/28 auto-closed as invalid on 2021/07/02 04:50

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.2.0-rc1-syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:1H/52 is trying to acquire lock:
ffff8880277440e0 (&type->s_umount_key#44){+.+.}-{3:3}, at: freeze_super+0x45/0x420 fs/super.c:1655

but task is already holding lock:
ffffc90000bd7d00 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at: process_one_work+0x831/0xdb0 kernel/workqueue.c:2264

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}:
       lock_acquire+0x182/0x3c0 kernel/locking/lockdep.c:5668
       process_one_work+0x852/0xdb0 kernel/workqueue.c:2265
       worker_thread+0xb14/0x1330 kernel/workqueue.c:2436
       kthread+0x266/0x300 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308

-> #1 ((wq_completion)glock_workqueue){+.+.}-{0:0}:
       lock_acquire+0x182/0x3c0 kernel/locking/lockdep.c:5668
       __flush_workqueue+0x178/0x1680 kernel/workqueue.c:2809
       gfs2_gl_hash_clear+0xa3/0x300 fs/gfs2/glock.c:2191
       gfs2_put_super+0x862/0x8d0 fs/gfs2/super.c:627
       generic_shutdown_super+0x130/0x310 fs/super.c:492
       kill_block_super+0x79/0xd0 fs/super.c:1386
       deactivate_locked_super+0xa7/0xf0 fs/super.c:332
       cleanup_mnt+0x494/0x520 fs/namespace.c:1291
       task_work_run+0x243/0x300 kernel/task_work.c:179
       ptrace_notify+0x29a/0x340 kernel/signal.c:2354
       ptrace_report_syscall include/linux/ptrace.h:411 [inline]
       ptrace_report_syscall_exit include/linux/ptrace.h:473 [inline]
       syscall_exit_work+0x8c/0xe0 kernel/entry/common.c:251
       syscall_exit_to_user_mode_prepare+0x63/0xc0 kernel/entry/common.c:278
       __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
       syscall_exit_to_user_mode+0xa/0x60 kernel/entry/common.c:296
       do_syscall_64+0x49/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x63/0xcd

-> #0 (&type->s_umount_key#44){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3097 [inline]
       check_prevs_add kernel/locking/lockdep.c:3216 [inline]
       validate_chain+0x1898/0x6ae0 kernel/locking/lockdep.c:3831
       __lock_acquire+0x1292/0x1f60 kernel/locking/lockdep.c:5055
       lock_acquire+0x182/0x3c0 kernel/locking/lockdep.c:5668
       down_write+0x9c/0x270 kernel/locking/rwsem.c:1562
       freeze_super+0x45/0x420 fs/super.c:1655
       freeze_go_sync+0x178/0x340 fs/gfs2/glops.c:577
       do_xmote+0x34d/0x13d0 fs/gfs2/glock.c:708
       glock_work_func+0x2c2/0x450 fs/gfs2/glock.c:1056
       process_one_work+0x877/0xdb0 kernel/workqueue.c:2289
       worker_thread+0xb14/0x1330 kernel/workqueue.c:2436
       kthread+0x266/0x300 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#44 --> (wq_completion)glock_workqueue --> (work_completion)(&(&gl->gl_work)->work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&(&gl->gl_work)->work));
                               lock((wq_completion)glock_workqueue);
                               lock((work_completion)(&(&gl->gl_work)->work));
  lock(&type->s_umount_key#44);

 *** DEADLOCK ***

2 locks held by kworker/0:1H/52:
 #0: ffff888018293938 ((wq_completion)glock_workqueue){+.+.}-{0:0}, at: process_one_work+0x7f2/0xdb0
 #1: ffffc90000bd7d00 ((work_completion)(&(&gl->gl_work)->work)
){+.+.}-{0:0}
, at: process_one_work+0x831/0xdb0 kernel/workqueue.c:2264

stack backtrace:
CPU: 0 PID: 52 Comm: kworker/0:1H Not tainted 6.2.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: glock_workqueue glock_work_func
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1b1/0x290 lib/dump_stack.c:106
 check_noncircular+0x2cc/0x390 kernel/locking/lockdep.c:2177
 check_prev_add kernel/locking/lockdep.c:3097 [inline]
 check_prevs_add kernel/locking/lockdep.c:3216 [inline]
 validate_chain+0x1898/0x6ae0 kernel/locking/lockdep.c:3831
 __lock_acquire+0x1292/0x1f60 kernel/locking/lockdep.c:5055
 lock_acquire+0x182/0x3c0 kernel/locking/lockdep.c:5668
 down_write+0x9c/0x270 kernel/locking/rwsem.c:1562
 freeze_super+0x45/0x420 fs/super.c:1655
 freeze_go_sync+0x178/0x340 fs/gfs2/glops.c:577
 do_xmote+0x34d/0x13d0 fs/gfs2/glock.c:708
 glock_work_func+0x2c2/0x450 fs/gfs2/glock.c:1056
 process_one_work+0x877/0xdb0 kernel/workqueue.c:2289
 worker_thread+0xb14/0x1330 kernel/workqueue.c:2436
 kthread+0x266/0x300 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>

Crashes (15):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/12/29 15:58 upstream 1b929c02afd3 44712fbc .config strace log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-upstream-fs possible deadlock in freeze_super
2023/02/22 09:52 upstream 4a7d37e824f5 42a4d508 .config console log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] ci-upstream-kasan-gce-root possible deadlock in freeze_super
2023/05/14 17:01 upstream bb7c241fae62 2b9ba477 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root possible deadlock in freeze_super
2023/02/12 05:55 upstream d12aca5c0cee 93e26d60 .config console log report info ci2-upstream-fs possible deadlock in freeze_super
2023/02/10 05:17 upstream 35674e787518 07980f9d .config console log report info ci2-upstream-fs possible deadlock in freeze_super
2023/02/01 18:33 upstream c0b67534c95c 9a6f477c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2023/01/20 05:30 upstream 081edded9b38 71197f3a .config console log report info ci2-upstream-fs possible deadlock in freeze_super
2023/01/19 06:27 upstream c1649ec55708 4620c2d9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2023/01/09 20:46 upstream 1fe4fd6f5cad 1dac8c7a .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2022/12/29 20:58 upstream 1b929c02afd3 44712fbc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2022/12/29 15:22 upstream 1b929c02afd3 44712fbc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2022/12/25 19:03 upstream 72a85e2b0a1e 9da18ae8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2022/12/09 02:47 upstream f3e8416619ce 1034e5fa .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in freeze_super
2022/11/27 21:39 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 6d464646530f 74a66371 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in freeze_super
2022/10/10 07:24 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci bbed346d5a96 aea5da89 .config console log report info [disk image] [vmlinux] ci-upstream-gce-arm64 possible deadlock in freeze_super
* Struck through repros no longer work on HEAD.