syzbot


possible deadlock in xfs_qm_dqget_cache_insert

Status: fixed on 2024/04/10 03:59
Subsystems: xfs
[Documentation on labels]
Reported-by: syzbot+8fdff861a781522bda4d@syzkaller.appspotmail.com
Fix commit: 0c6ca06aad84 xfs: quota radix tree allocations need to be NOFS on insert
First crash: 68d, last: 67d
Duplicate bugs (3)
duplicates (3):
Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
possible deadlock in xfs_qm_dqget xfs 3 56d 56d 0/26 closed as dup on 2024/03/25 21:16
possible deadlock in xfs_icwalk_ag xfs 14 42d 59d 0/26 closed as dup on 2024/03/23 08:37
possible deadlock in xfs_qm_dqfree_one xfs 5 60d 63d 0/26 closed as dup on 2024/03/19 03:45
Discussions (2)
Title Replies (including bot) Last reply
[syzbot] Monthly xfs report (Mar 2024) 0 (1) 2024/03/21 15:10
[syzbot] [xfs?] possible deadlock in xfs_qm_dqget_cache_insert 0 (1) 2024/03/14 09:31

Sample crash report:
XFS (loop0): Quotacheck needed: Please wait.
XFS (loop0): Quotacheck: Done.
======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
------------------------------------------------------
syz-executor.0/15543 is trying to acquire lock:
ffffffff8d928ce0 (fs_reclaim){+.+.}-{0:0}, at: might_alloc include/linux/sched/mm.h:303 [inline]
ffffffff8d928ce0 (fs_reclaim){+.+.}-{0:0}, at: slab_pre_alloc_hook mm/slub.c:3746 [inline]
ffffffff8d928ce0 (fs_reclaim){+.+.}-{0:0}, at: slab_alloc_node mm/slub.c:3827 [inline]
ffffffff8d928ce0 (fs_reclaim){+.+.}-{0:0}, at: kmem_cache_alloc+0x4f/0x320 mm/slub.c:3852

but task is already holding lock:
ffff88802600a958 (&qinf->qi_tree_lock){+.+.}-{3:3}, at: xfs_qm_dqget_cache_insert.constprop.0+0x2a/0x2c0 fs/xfs/xfs_dquot.c:825

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&qinf->qi_tree_lock){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
       __mutex_lock+0x175/0x9c0 kernel/locking/mutex.c:752
       xfs_qm_dqget_cache_lookup+0x66/0x820 fs/xfs/xfs_dquot.c:784
       xfs_qm_dqget_inode+0x1e7/0x6d0 fs/xfs/xfs_dquot.c:986
       xfs_qm_dqattach_one+0x26f/0x530 fs/xfs/xfs_qm.c:278
       xfs_qm_dqattach_locked+0x1a8/0x2d0 fs/xfs/xfs_qm.c:345
       xfs_qm_vop_dqalloc+0x344/0xe10 fs/xfs/xfs_qm.c:1710
       xfs_create+0x422/0x1140 fs/xfs/xfs_inode.c:1041
       xfs_generic_create+0x631/0x7c0 fs/xfs/xfs_iops.c:199
       lookup_open.isra.0+0x10a1/0x13c0 fs/namei.c:3497
       open_last_lookups fs/namei.c:3566 [inline]
       path_openat+0x92f/0x2990 fs/namei.c:3796
       do_filp_open+0x1dc/0x430 fs/namei.c:3826
       do_sys_openat2+0x17a/0x1e0 fs/open.c:1406
       do_sys_open fs/open.c:1421 [inline]
       __do_sys_open fs/open.c:1429 [inline]
       __se_sys_open fs/open.c:1425 [inline]
       __x64_sys_open+0x154/0x1e0 fs/open.c:1425
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #1 (&xfs_dir_ilock_class){++++}-{3:3}:
       down_write_nested+0x3d/0x50 kernel/locking/rwsem.c:1695
       xfs_ilock+0x2ef/0x3e0 fs/xfs/xfs_inode.c:206
       xfs_reclaim_inode fs/xfs/xfs_icache.c:945 [inline]
       xfs_icwalk_process_inode fs/xfs/xfs_icache.c:1631 [inline]
       xfs_icwalk_ag+0xca6/0x1740 fs/xfs/xfs_icache.c:1713
       xfs_icwalk+0x57/0x100 fs/xfs/xfs_icache.c:1762
       xfs_reclaim_inodes_nr+0x182/0x250 fs/xfs/xfs_icache.c:1011
       super_cache_scan+0x409/0x550 fs/super.c:227
       do_shrink_slab+0x44f/0x1160 mm/shrinker.c:435
       shrink_slab+0x18a/0x1310 mm/shrinker.c:662
       shrink_one+0x493/0x7b0 mm/vmscan.c:4767
       shrink_many mm/vmscan.c:4828 [inline]
       lru_gen_shrink_node mm/vmscan.c:4929 [inline]
       shrink_node+0x2123/0x3750 mm/vmscan.c:5888
       kswapd_shrink_node mm/vmscan.c:6696 [inline]
       balance_pgdat+0x9d0/0x1a90 mm/vmscan.c:6886
       kswapd+0x5c1/0xc10 mm/vmscan.c:7146
       kthread+0x2c1/0x3a0 kernel/kthread.c:388
       ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243

-> #0 (fs_reclaim){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
       validate_chain kernel/locking/lockdep.c:3869 [inline]
       __lock_acquire+0x2478/0x3b30 kernel/locking/lockdep.c:5137
       lock_acquire kernel/locking/lockdep.c:5754 [inline]
       lock_acquire+0x1b1/0x540 kernel/locking/lockdep.c:5719
       __fs_reclaim_acquire mm/page_alloc.c:3692 [inline]
       fs_reclaim_acquire+0x102/0x150 mm/page_alloc.c:3706
       might_alloc include/linux/sched/mm.h:303 [inline]
       slab_pre_alloc_hook mm/slub.c:3746 [inline]
       slab_alloc_node mm/slub.c:3827 [inline]
       kmem_cache_alloc+0x4f/0x320 mm/slub.c:3852
       radix_tree_node_alloc.constprop.0+0x7c/0x350 lib/radix-tree.c:276
       radix_tree_extend+0x1a2/0x4d0 lib/radix-tree.c:425
       __radix_tree_create lib/radix-tree.c:613 [inline]
       radix_tree_insert+0x499/0x630 lib/radix-tree.c:712
       xfs_qm_dqget_cache_insert.constprop.0+0x38/0x2c0 fs/xfs/xfs_dquot.c:826
       xfs_qm_dqget+0x182/0x4a0 fs/xfs/xfs_dquot.c:901
       xfs_qm_vop_dqalloc+0x627/0xe10 fs/xfs/xfs_qm.c:1773
       xfs_fileattr_set+0x7e8/0x2590 fs/xfs/xfs_ioctl.c:1347
       vfs_fileattr_set+0x7cc/0xc00 fs/ioctl.c:697
       ioctl_fssetxattr fs/ioctl.c:759 [inline]
       do_vfs_ioctl+0x24f/0x1ac0 fs/ioctl.c:873
       __do_sys_ioctl fs/ioctl.c:902 [inline]
       __se_sys_ioctl fs/ioctl.c:890 [inline]
       __x64_sys_ioctl+0x116/0x220 fs/ioctl.c:890
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x6d/0x75

other info that might help us debug this:

Chain exists of:
  fs_reclaim --> &xfs_dir_ilock_class --> &qinf->qi_tree_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&qinf->qi_tree_lock);
                               lock(&xfs_dir_ilock_class);
                               lock(&qinf->qi_tree_lock);
  lock(fs_reclaim);

 *** DEADLOCK ***

3 locks held by syz-executor.0/15543:
 #0: ffff8880656ec420 (sb_writers#16){.+.+}-{0:0}, at: ioctl_fssetxattr fs/ioctl.c:757 [inline]
 #0: ffff8880656ec420 (sb_writers#16){.+.+}-{0:0}, at: do_vfs_ioctl+0x14ef/0x1ac0 fs/ioctl.c:873
 #1: ffff88802e6429f8 (&sb->s_type->i_mutex_key#24){++++}-{3:3}, at: inode_lock include/linux/fs.h:793 [inline]
 #1: ffff88802e6429f8 (&sb->s_type->i_mutex_key#24){++++}-{3:3}, at: vfs_fileattr_set+0x153/0xc00 fs/ioctl.c:682
 #2: ffff88802600a958 (&qinf->qi_tree_lock){+.+.}-{3:3}, at: xfs_qm_dqget_cache_insert.constprop.0+0x2a/0x2c0 fs/xfs/xfs_dquot.c:825

stack backtrace:
CPU: 1 PID: 15543 Comm: syz-executor.0 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:114
 check_noncircular+0x31a/0x400 kernel/locking/lockdep.c:2187
 check_prev_add kernel/locking/lockdep.c:3134 [inline]
 check_prevs_add kernel/locking/lockdep.c:3253 [inline]
 validate_chain kernel/locking/lockdep.c:3869 [inline]
 __lock_acquire+0x2478/0x3b30 kernel/locking/lockdep.c:5137
 lock_acquire kernel/locking/lockdep.c:5754 [inline]
 lock_acquire+0x1b1/0x540 kernel/locking/lockdep.c:5719
 __fs_reclaim_acquire mm/page_alloc.c:3692 [inline]
 fs_reclaim_acquire+0x102/0x150 mm/page_alloc.c:3706
 might_alloc include/linux/sched/mm.h:303 [inline]
 slab_pre_alloc_hook mm/slub.c:3746 [inline]
 slab_alloc_node mm/slub.c:3827 [inline]
 kmem_cache_alloc+0x4f/0x320 mm/slub.c:3852
 radix_tree_node_alloc.constprop.0+0x7c/0x350 lib/radix-tree.c:276
 radix_tree_extend+0x1a2/0x4d0 lib/radix-tree.c:425
 __radix_tree_create lib/radix-tree.c:613 [inline]
 radix_tree_insert+0x499/0x630 lib/radix-tree.c:712
 xfs_qm_dqget_cache_insert.constprop.0+0x38/0x2c0 fs/xfs/xfs_dquot.c:826
 xfs_qm_dqget+0x182/0x4a0 fs/xfs/xfs_dquot.c:901
 xfs_qm_vop_dqalloc+0x627/0xe10 fs/xfs/xfs_qm.c:1773
 xfs_fileattr_set+0x7e8/0x2590 fs/xfs/xfs_ioctl.c:1347
 vfs_fileattr_set+0x7cc/0xc00 fs/ioctl.c:697
 ioctl_fssetxattr fs/ioctl.c:759 [inline]
 do_vfs_ioctl+0x24f/0x1ac0 fs/ioctl.c:873
 __do_sys_ioctl fs/ioctl.c:902 [inline]
 __se_sys_ioctl fs/ioctl.c:890 [inline]
 __x64_sys_ioctl+0x116/0x220 fs/ioctl.c:890
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7f98b547dda9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f98b61000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f98b55abf80 RCX: 00007f98b547dda9
RDX: 0000000020000000 RSI: 00000000401c5820 RDI: 0000000000000005
RBP: 00007f98b54ca47a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f98b55abf80 R15: 00007ffc3d4bc918
 </TASK>

Crashes (8):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/03/15 05:28 upstream fe46a7dd189e d615901c .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in xfs_qm_dqget_cache_insert
2024/03/15 03:36 upstream fe46a7dd189e d615901c .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 22:39 upstream fe46a7dd189e d615901c .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 22:02 upstream fe46a7dd189e d615901c .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 19:41 upstream e5e038b7ae9d 8d8ee116 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 03:44 upstream e5e038b7ae9d f919f202 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 03:07 upstream e5e038b7ae9d f919f202 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
2024/03/14 02:25 upstream e5e038b7ae9d f919f202 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in xfs_qm_dqget_cache_insert
* Struck through repros no longer work on HEAD.