syzbot


possible deadlock in start_this_handle (4)

Status: upstream: reported on 2023/03/01 00:02
Subsystems: fscrypt ext4
[Documentation on labels]
Reported-by: syzbot+cf0b4280f19be4031cf2@syzkaller.appspotmail.com
First crash: 388d, last: 5d01h
Duplicate bugs (2)
duplicates (2):
Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
possible deadlock in find_and_lock_process_key fscrypt 6 13d 113d 0/26 closed as dup on 2023/11/27 22:16
possible deadlock in fscrypt_initialize (2) fscrypt 63 327d 383d 0/26 closed as dup on 2023/11/27 21:45
Discussions (4)
Title Replies (including bot) Last reply
[syzbot] Monthly ext4 report (Mar 2024) 0 (1) 2024/03/12 09:29
[syzbot] Monthly ext4 report (Feb 2024) 0 (1) 2024/02/09 20:57
[syzbot] Monthly ext4 report (Nov 2023) 0 (1) 2023/11/08 23:44
[syzbot] [ext4?] possible deadlock in start_this_handle (4) 1 (2) 2023/03/01 00:19
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in start_this_handle (3) ext4 8 425d 613d 22/26 fixed on 2023/02/24 13:50
upstream possible deadlock in start_this_handle (2) ext4 8 1099d 1132d 0/26 auto-closed as invalid on 2021/07/13 16:11
upstream possible deadlock in start_this_handle ext4 8 1981d 2019d 0/26 auto-closed as invalid on 2019/04/13 16:27

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.8.0-rc7-syzkaller #0 Not tainted
------------------------------------------------------
kswapd0/88 is trying to acquire lock:
ffff88802fd12950 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0x203c/0x22a0 fs/jbd2/transaction.c:463

but task is already holding lock:
ffffffff8e219320 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6771 [inline]
ffffffff8e219320 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xb3e/0x36d0 mm/vmscan.c:7143

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (fs_reclaim){+.+.}-{0:0}:
       lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
       __fs_reclaim_acquire mm/page_alloc.c:3692 [inline]
       fs_reclaim_acquire+0x87/0x130 mm/page_alloc.c:3706
       might_alloc include/linux/sched/mm.h:303 [inline]
       slab_pre_alloc_hook mm/slub.c:3761 [inline]
       slab_alloc_node mm/slub.c:3842 [inline]
       kmem_cache_alloc+0x48/0x340 mm/slub.c:3867
       ext4_es_insert_delayed_block+0x2d9/0xa10 fs/ext4/extents_status.c:2090
       ext4_insert_delayed_block fs/ext4/inode.c:1676 [inline]
       ext4_da_map_blocks fs/ext4/inode.c:1777 [inline]
       ext4_da_get_block_prep+0xa67/0x1420 fs/ext4/inode.c:1817
       ext4_block_write_begin+0x53d/0x1850 fs/ext4/inode.c:1055
       ext4_da_write_begin+0x5e8/0xa50 fs/ext4/inode.c:2894
       generic_perform_write+0x323/0x640 mm/filemap.c:3930
       ext4_buffered_write_iter+0xc6/0x350 fs/ext4/file.c:299
       ext4_file_write_iter+0x1dc/0x19c0
       do_iter_readv_writev+0x46e/0x640
       vfs_writev+0x395/0xbb0 fs/read_write.c:971
       do_pwritev fs/read_write.c:1072 [inline]
       __do_sys_pwritev2 fs/read_write.c:1131 [inline]
       __se_sys_pwritev2+0x1ca/0x2d0 fs/read_write.c:1122
       do_syscall_64+0xfb/0x240
       entry_SYSCALL_64_after_hwframe+0x6f/0x77

-> #1 (&ei->i_data_sem){++++}-{3:3}:
       lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
       down_write+0x3a/0x50 kernel/locking/rwsem.c:1579
       ext4_map_blocks+0x988/0x1d20 fs/ext4/inode.c:616
       mpage_map_one_extent fs/ext4/inode.c:2163 [inline]
       mpage_map_and_submit_extent fs/ext4/inode.c:2216 [inline]
       ext4_do_writepages+0x15e7/0x3ca0 fs/ext4/inode.c:2679
       ext4_writepages+0x203/0x3e0 fs/ext4/inode.c:2768
       do_writepages+0x3ac/0x680 mm/page-writeback.c:2553
       __writeback_single_inode+0x155/0xfd0 fs/fs-writeback.c:1625
       writeback_sb_inodes+0x8e3/0x1220 fs/fs-writeback.c:1916
       __writeback_inodes_wb+0x11b/0x260 fs/fs-writeback.c:1987
       wb_writeback+0x461/0xc80 fs/fs-writeback.c:2094
       wb_check_background_flush fs/fs-writeback.c:2164 [inline]
       wb_do_writeback fs/fs-writeback.c:2252 [inline]
       wb_workfn+0xc33/0xfb0 fs/fs-writeback.c:2279
       process_one_work kernel/workqueue.c:2633 [inline]
       process_scheduled_works+0x915/0x1420 kernel/workqueue.c:2706
       worker_thread+0xa5f/0x1000 kernel/workqueue.c:2787
       kthread+0x2f1/0x390 kernel/kthread.c:388
       ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:243

-> #0 (jbd2_handle){++++}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
       validate_chain+0x18ca/0x58e0 kernel/locking/lockdep.c:3869
       __lock_acquire+0x1345/0x1fd0 kernel/locking/lockdep.c:5137
       lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
       start_this_handle+0x205c/0x22a0 fs/jbd2/transaction.c:463
       jbd2__journal_start+0x2d9/0x5d0 fs/jbd2/transaction.c:520
       __ext4_journal_start_sb+0x215/0x5b0 fs/ext4/ext4_jbd2.c:112
       __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
       ext4_dirty_inode+0x92/0x110 fs/ext4/inode.c:5939
       __mark_inode_dirty+0x30b/0xdb0 fs/fs-writeback.c:2452
       mark_inode_dirty_sync include/linux/fs.h:2387 [inline]
       iput+0x1f2/0x900 fs/inode.c:1762
       __dentry_kill+0x20d/0x630 fs/dcache.c:603
       shrink_kill+0xa9/0x2c0 fs/dcache.c:1048
       shrink_dentry_list+0x2c0/0x5b0 fs/dcache.c:1075
       prune_dcache_sb+0x10f/0x180 fs/dcache.c:1156
       super_cache_scan+0x34f/0x4b0 fs/super.c:221
       do_shrink_slab+0x6d2/0x1140 mm/shrinker.c:435
       shrink_slab+0x1092/0x14d0 mm/shrinker.c:662
       shrink_one+0x423/0x7e0 mm/vmscan.c:4767
       shrink_many mm/vmscan.c:4828 [inline]
       lru_gen_shrink_node mm/vmscan.c:4929 [inline]
       shrink_node+0x375e/0x3e20 mm/vmscan.c:5888
       kswapd_shrink_node mm/vmscan.c:6693 [inline]
       balance_pgdat mm/vmscan.c:6883 [inline]
       kswapd+0x17d0/0x36d0 mm/vmscan.c:7143
       kthread+0x2f1/0x390 kernel/kthread.c:388
       ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:243

other info that might help us debug this:

Chain exists of:
  jbd2_handle --> &ei->i_data_sem --> fs_reclaim

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(fs_reclaim);
                               lock(&ei->i_data_sem);
                               lock(fs_reclaim);
  rlock(jbd2_handle);

 *** DEADLOCK ***

2 locks held by kswapd0/88:
 #0: ffffffff8e219320 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6771 [inline]
 #0: ffffffff8e219320 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xb3e/0x36d0 mm/vmscan.c:7143
 #1: ffff88802ff760e0 (&type->s_umount_key#31){++++}-{3:3}, at: super_trylock_shared fs/super.c:561 [inline]
 #1: ffff88802ff760e0 (&type->s_umount_key#31){++++}-{3:3}, at: super_cache_scan+0x94/0x4b0 fs/super.c:196

stack backtrace:
CPU: 1 PID: 88 Comm: kswapd0 Not tainted 6.8.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/25/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e7/0x2e0 lib/dump_stack.c:106
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
 check_prev_add kernel/locking/lockdep.c:3134 [inline]
 check_prevs_add kernel/locking/lockdep.c:3253 [inline]
 validate_chain+0x18ca/0x58e0 kernel/locking/lockdep.c:3869
 __lock_acquire+0x1345/0x1fd0 kernel/locking/lockdep.c:5137
 lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754
 start_this_handle+0x205c/0x22a0 fs/jbd2/transaction.c:463
 jbd2__journal_start+0x2d9/0x5d0 fs/jbd2/transaction.c:520
 __ext4_journal_start_sb+0x215/0x5b0 fs/ext4/ext4_jbd2.c:112
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_dirty_inode+0x92/0x110 fs/ext4/inode.c:5939
 __mark_inode_dirty+0x30b/0xdb0 fs/fs-writeback.c:2452
 mark_inode_dirty_sync include/linux/fs.h:2387 [inline]
 iput+0x1f2/0x900 fs/inode.c:1762
 __dentry_kill+0x20d/0x630 fs/dcache.c:603
 shrink_kill+0xa9/0x2c0 fs/dcache.c:1048
 shrink_dentry_list+0x2c0/0x5b0 fs/dcache.c:1075
 prune_dcache_sb+0x10f/0x180 fs/dcache.c:1156
 super_cache_scan+0x34f/0x4b0 fs/super.c:221
 do_shrink_slab+0x6d2/0x1140 mm/shrinker.c:435
 shrink_slab+0x1092/0x14d0 mm/shrinker.c:662
 shrink_one+0x423/0x7e0 mm/vmscan.c:4767
 shrink_many mm/vmscan.c:4828 [inline]
 lru_gen_shrink_node mm/vmscan.c:4929 [inline]
 shrink_node+0x375e/0x3e20 mm/vmscan.c:5888
 kswapd_shrink_node mm/vmscan.c:6693 [inline]
 balance_pgdat mm/vmscan.c:6883 [inline]
 kswapd+0x17d0/0x36d0 mm/vmscan.c:7143
 kthread+0x2f1/0x390 kernel/kthread.c:388
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:243
 </TASK>

Crashes (23):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/03/04 16:27 upstream 90d35da658da 3717835d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root possible deadlock in start_this_handle
2024/03/14 06:32 upstream e5e038b7ae9d f919f202 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in start_this_handle
2024/03/01 03:09 upstream 87adedeba51a 352ab904 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in start_this_handle
2024/02/28 13:11 upstream cf1182944c7c 55d6f11d .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in start_this_handle
2024/01/29 19:54 upstream 41bccc98fb79 991a98f4 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in start_this_handle
2023/11/09 05:23 upstream 34f763262743 4862372a .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/11/08 10:54 upstream 305230142ae0 b93f63e8 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/10/18 20:01 upstream dd72f9c7e512 342b9c55 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/10/01 01:37 upstream 3b517966c561 8e26a358 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/09/18 10:39 upstream ce9ecca0238b 0b6a67ac .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/07/01 09:17 upstream a507db1d8fdc af3053d2 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in start_this_handle
2023/08/16 11:18 upstream 4853c74bd7ab 39990d51 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/08/10 11:15 upstream 374a7f47bf40 da3c3ef8 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/07/13 01:33 upstream eb26cbb1a754 86081196 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/07/10 23:09 upstream 3f01e9fed845 52ae002a .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/06/30 21:08 upstream 533925cb7604 af3053d2 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/06/26 19:56 upstream 6995e2de6891 4cd5bb25 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/03/18 13:47 upstream 478a351ce0d6 7939252e .config console log report info ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/03/17 10:16 upstream 38e04b3e4240 18b58603 .config console log report info ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/03/12 22:41 upstream 134231664868 5205ef30 .config console log report info ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/03/05 16:18 upstream b01fe98d34f3 f8902b57 .config console log report info ci-qemu-upstream-386 possible deadlock in start_this_handle
2023/02/24 23:50 upstream a93e884edf61 630c6bc9 .config console log report info ci-qemu-upstream-386 possible deadlock in start_this_handle
2024/01/29 16:10 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 0802e17d9aca 991a98f4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in start_this_handle
* Struck through repros no longer work on HEAD.