syzbot


possible deadlock in ocfs2_finish_quota_recovery

Status: upstream: reported on 2025/01/30 02:34
Reported-by: syzbot+9d13e0bd9eb62200af15@syzkaller.appspotmail.com
First crash: 6d06h, last: 3d14h
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in ocfs2_finish_quota_recovery C 2 4d02h 4d16h 0/3 upstream: reported C repro on 2025/01/31 16:27
upstream possible deadlock in ocfs2_finish_quota_recovery ocfs2 14 6h10m 2d23h 0/28 upstream: reported on 2025/02/02 09:01

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.178-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/239 is trying to acquire lock:
ffff88807b6bc0e0 (&type->s_umount_key#60){++++}-{3:3}, at: ocfs2_finish_quota_recovery+0x15a/0x2260 fs/ocfs2/quota_local.c:600

but task is already holding lock:
ffffc90002047d20 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2285

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       process_one_work+0x7f1/0x10c0 kernel/workqueue.c:2286
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #1 ((wq_completion)ocfs2_wq){+.+.}-{0:0}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       flush_workqueue+0x170/0x1610 kernel/workqueue.c:2830
       ocfs2_shutdown_local_alloc+0x105/0xa90 fs/ocfs2/localalloc.c:379
       ocfs2_dismount_volume+0x1db/0x8b0 fs/ocfs2/super.c:1882
       generic_shutdown_super+0x130/0x310 fs/super.c:475
       kill_block_super+0x7a/0xe0 fs/super.c:1427
       deactivate_locked_super+0xa0/0x110 fs/super.c:335
       cleanup_mnt+0x44e/0x500 fs/namespace.c:1143
       task_work_run+0x129/0x1a0 kernel/task_work.c:188
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       exit_to_user_mode_loop+0x106/0x130 kernel/entry/common.c:181
       exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:214
       __syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
       syscall_exit_to_user_mode+0x5d/0x240 kernel/entry/common.c:307
       do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #0 (&type->s_umount_key#60){++++}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       down_read+0x45/0x2e0 kernel/locking/rwsem.c:1498
       ocfs2_finish_quota_recovery+0x15a/0x2260 fs/ocfs2/quota_local.c:600
       ocfs2_complete_recovery+0x173c/0x24a0 fs/ocfs2/journal.c:1295
       process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#60 --> (wq_completion)ocfs2_wq --> (work_completion)(&journal->j_recovery_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&journal->j_recovery_work));
                               lock((wq_completion)ocfs2_wq);
                               lock((work_completion)(&journal->j_recovery_work));
  lock(&type->s_umount_key#60);

 *** DEADLOCK ***

2 locks held by kworker/u4:3/239:
 #0: ffff88807a92c938 ((wq_completion)ocfs2_wq#2){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2283
 #1: ffffc90002047d20 ((work_completion)(&journal->j_recovery_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2285

stack backtrace:
CPU: 1 PID: 239 Comm: kworker/u4:3 Not tainted 5.15.178-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024
Workqueue: ocfs2_wq ocfs2_complete_recovery
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 down_read+0x45/0x2e0 kernel/locking/rwsem.c:1498
 ocfs2_finish_quota_recovery+0x15a/0x2260 fs/ocfs2/quota_local.c:600
 ocfs2_complete_recovery+0x173c/0x24a0 fs/ocfs2/journal.c:1295
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/02/01 18:43 linux-5.15.y c16c81c81336 aa47157c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in ocfs2_finish_quota_recovery
2025/01/30 02:33 linux-5.15.y 003148680b79 afe4eff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in ocfs2_finish_quota_recovery
* Struck through repros no longer work on HEAD.