syzbot


possible deadlock in bch2_btree_cache_scan

Status: upstream: reported on 2024/10/24 17:03
Subsystems: bcachefs
[Documentation on labels]
Reported-by: syzbot+3d89e46a004eafb88bc6@syzkaller.appspotmail.com
First crash: 191d, last: 62d
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [bcachefs?] possible deadlock in bch2_btree_cache_scan 0 (1) 2024/10/24 17:03

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.13.0-rc4-syzkaller-00071-gfd0584d220fe #0 Not tainted
------------------------------------------------------
kswapd1/89 is trying to acquire lock:
ffff888059b01c50 (&bc->lock){+.+.}-{4:4}, at: bch2_btree_cache_scan+0x184/0xec0 fs/bcachefs/btree_cache.c:480

but task is already holding lock:
ffffffff8ea36e80 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6864 [inline]
ffffffff8ea36e80 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xbf1/0x36f0 mm/vmscan.c:7246

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (fs_reclaim){+.+.}-{0:0}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       __fs_reclaim_acquire mm/page_alloc.c:3853 [inline]
       fs_reclaim_acquire+0x88/0x130 mm/page_alloc.c:3867
       might_alloc include/linux/sched/mm.h:318 [inline]
       slab_pre_alloc_hook mm/slub.c:4070 [inline]
       slab_alloc_node mm/slub.c:4148 [inline]
       __do_kmalloc_node mm/slub.c:4297 [inline]
       __kmalloc_noprof+0xae/0x4c0 mm/slub.c:4310
       kmalloc_noprof include/linux/slab.h:905 [inline]
       kzalloc_noprof include/linux/slab.h:1037 [inline]
       pcpu_mem_zalloc mm/percpu.c:510 [inline]
       pcpu_alloc_chunk mm/percpu.c:1443 [inline]
       pcpu_create_chunk+0x57/0xbc0 mm/percpu-vm.c:338
       pcpu_balance_populated mm/percpu.c:2076 [inline]
       pcpu_balance_workfn+0xc4d/0xd40 mm/percpu.c:2213
       process_one_work kernel/workqueue.c:3229 [inline]
       process_scheduled_works+0xa68/0x1840 kernel/workqueue.c:3310
       worker_thread+0x870/0xd30 kernel/workqueue.c:3391
       kthread+0x2f2/0x390 kernel/kthread.c:389
       ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

-> #1 (pcpu_alloc_mutex){+.+.}-{4:4}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       __mutex_lock_common kernel/locking/mutex.c:585 [inline]
       __mutex_lock+0x1ac/0xee0 kernel/locking/mutex.c:735
       pcpu_alloc_noprof+0x293/0x1760 mm/percpu.c:1795
       __six_lock_init+0x104/0x150 fs/bcachefs/six.c:869
       bch2_btree_lock_init+0x38/0x100 fs/bcachefs/btree_locking.c:12
       bch2_btree_node_mem_alloc+0x565/0x16f0 fs/bcachefs/btree_cache.c:805
       __bch2_btree_node_alloc fs/bcachefs/btree_update_interior.c:321 [inline]
       bch2_btree_reserve_get+0x2df/0x1890 fs/bcachefs/btree_update_interior.c:549
       bch2_btree_update_start+0x115d/0x14e0 fs/bcachefs/btree_update_interior.c:1266
       bch2_btree_split_leaf+0x123/0x840 fs/bcachefs/btree_update_interior.c:1856
       bch2_trans_commit_error+0x212/0x1380 fs/bcachefs/btree_trans_commit.c:942
       __bch2_trans_commit+0x7ead/0x93c0 fs/bcachefs/btree_trans_commit.c:1140
       bch2_trans_commit fs/bcachefs/btree_update.h:184 [inline]
       bch2_journal_replay+0x1a3a/0x2a40 fs/bcachefs/recovery.c:317
       bch2_run_recovery_pass+0xf2/0x1e0 fs/bcachefs/recovery_passes.c:191
       bch2_run_recovery_passes+0x3a7/0x880 fs/bcachefs/recovery_passes.c:244
       bch2_fs_recovery+0x25cc/0x39d0 fs/bcachefs/recovery.c:861
       bch2_fs_start+0x356/0x5b0 fs/bcachefs/super.c:1037
       bch2_fs_get_tree+0xd68/0x1710 fs/bcachefs/fs.c:2170
       vfs_get_tree+0x92/0x2b0 fs/super.c:1814
       do_new_mount+0x2be/0xb40 fs/namespace.c:3507
       do_mount fs/namespace.c:3847 [inline]
       __do_sys_mount fs/namespace.c:4057 [inline]
       __se_sys_mount+0x2d6/0x3c0 fs/namespace.c:4034
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&bc->lock){+.+.}-{4:4}:
       check_prev_add kernel/locking/lockdep.c:3161 [inline]
       check_prevs_add kernel/locking/lockdep.c:3280 [inline]
       validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
       __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5226
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       __mutex_lock_common kernel/locking/mutex.c:585 [inline]
       __mutex_lock+0x1ac/0xee0 kernel/locking/mutex.c:735
       bch2_btree_cache_scan+0x184/0xec0 fs/bcachefs/btree_cache.c:480
       do_shrink_slab+0x72f/0x1160 mm/shrinker.c:437
       shrink_slab+0x1093/0x14d0 mm/shrinker.c:664
       shrink_one+0x43b/0x850 mm/vmscan.c:4836
       shrink_many mm/vmscan.c:4897 [inline]
       lru_gen_shrink_node mm/vmscan.c:4975 [inline]
       shrink_node+0x37c5/0x3e50 mm/vmscan.c:5956
       kswapd_shrink_node mm/vmscan.c:6785 [inline]
       balance_pgdat mm/vmscan.c:6977 [inline]
       kswapd+0x1ca9/0x36f0 mm/vmscan.c:7246
       kthread+0x2f2/0x390 kernel/kthread.c:389
       ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

other info that might help us debug this:

Chain exists of:
  &bc->lock --> pcpu_alloc_mutex --> fs_reclaim

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(fs_reclaim);
                               lock(pcpu_alloc_mutex);
                               lock(fs_reclaim);
  lock(&bc->lock);

 *** DEADLOCK ***

1 lock held by kswapd1/89:
 #0: ffffffff8ea36e80 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6864 [inline]
 #0: ffffffff8ea36e80 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xbf1/0x36f0 mm/vmscan.c:7246

stack backtrace:
CPU: 0 UID: 0 PID: 89 Comm: kswapd1 Not tainted 6.13.0-rc4-syzkaller-00071-gfd0584d220fe #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 print_circular_bug+0x13a/0x1b0 kernel/locking/lockdep.c:2074
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2206
 check_prev_add kernel/locking/lockdep.c:3161 [inline]
 check_prevs_add kernel/locking/lockdep.c:3280 [inline]
 validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
 __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5226
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
 __mutex_lock_common kernel/locking/mutex.c:585 [inline]
 __mutex_lock+0x1ac/0xee0 kernel/locking/mutex.c:735
 bch2_btree_cache_scan+0x184/0xec0 fs/bcachefs/btree_cache.c:480
 do_shrink_slab+0x72f/0x1160 mm/shrinker.c:437
 shrink_slab+0x1093/0x14d0 mm/shrinker.c:664
 shrink_one+0x43b/0x850 mm/vmscan.c:4836
 shrink_many mm/vmscan.c:4897 [inline]
 lru_gen_shrink_node mm/vmscan.c:4975 [inline]
 shrink_node+0x37c5/0x3e50 mm/vmscan.c:5956
 kswapd_shrink_node mm/vmscan.c:6785 [inline]
 balance_pgdat mm/vmscan.c:6977 [inline]
 kswapd+0x1ca9/0x36f0 mm/vmscan.c:7246
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (152):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/28 13:49 upstream fd0584d220fe d3ccff63 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root possible deadlock in bch2_btree_cache_scan
2025/02/27 01:59 upstream 5394eea10651 6a8fcbc4 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/26 15:12 upstream ac9c34d1e45a d34966d1 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/26 07:10 upstream 2a1944bff549 d34966d1 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/26 04:09 upstream 2a1944bff549 d34966d1 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/26 00:26 upstream 2a1944bff549 d34966d1 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/21 22:28 upstream 8a61cb6e150e d34966d1 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/21 13:21 upstream 334426094588 0808a665 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/21 11:22 upstream 334426094588 0808a665 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/21 10:09 upstream 334426094588 0808a665 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/19 14:46 upstream 6537cfb395f3 cbd8edab .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/17 18:42 upstream 0ad2507d5d93 4121cf9d .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/14 19:13 upstream 128c8f96eb86 fe17639f .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/14 13:33 upstream 128c8f96eb86 fe17639f .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/14 09:49 upstream ab68d7eb7b1a d9a046cf .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/14 07:36 upstream ab68d7eb7b1a d9a046cf .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/14 01:05 upstream ab68d7eb7b1a d9a046cf .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/13 11:15 upstream 4dc1d1bec898 7320a65f .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/13 09:28 upstream 4dc1d1bec898 2afad16e .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/13 02:22 upstream 4dc1d1bec898 2afad16e .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/12 14:02 upstream 09fbf3d50205 b27c2402 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/12 12:30 upstream 09fbf3d50205 b27c2402 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/12 08:18 upstream 09fbf3d50205 f2baddf5 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/12 03:24 upstream 09fbf3d50205 f2baddf5 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/12 02:11 upstream 09fbf3d50205 f2baddf5 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/11 23:43 upstream 09fbf3d50205 f2baddf5 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/10 07:16 upstream 69b54314c975 ef44b750 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/09 22:05 upstream 69b54314c975 ef44b750 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/08 09:55 upstream 7ee983c850b4 ef44b750 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/07 03:03 upstream bb066fe812d6 53657d1b .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/04 06:07 upstream 0de63bb7d919 8f267cef .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/03 13:37 upstream 2014c95afece a21a8419 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/03 05:09 upstream 69e858e0b8b2 568559e4 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/03 03:11 upstream 69e858e0b8b2 568559e4 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/02 20:14 upstream 69e858e0b8b2 568559e4 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/02 14:15 upstream 69e858e0b8b2 568559e4 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 22:10 upstream 69e858e0b8b2 0dff8567 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 16:04 upstream 69e858e0b8b2 0dff8567 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 14:23 upstream 69e858e0b8b2 0dff8567 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 11:46 upstream 69e858e0b8b2 aa47157c .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 10:01 upstream 69e858e0b8b2 aa47157c .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 08:41 upstream 69e858e0b8b2 aa47157c .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 07:03 upstream 69e858e0b8b2 aa47157c .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2025/02/01 04:30 upstream 69e858e0b8b2 aa47157c .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2024/10/20 19:34 upstream 715ca9dd687f cd6fc0a3 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
2024/10/20 16:51 upstream 715ca9dd687f cd6fc0a3 .config console log report [disk image (non-bootable)] [vmlinux] [kernel image] ci-snapshot-upstream-root possible deadlock in bch2_btree_cache_scan
* Struck through repros no longer work on HEAD.