syzbot


possible deadlock in __jbd2_log_wait_for_space

Status: upstream: reported C repro on 2023/03/07 17:44
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+385d1b32404207ed55d6@syzkaller.appspotmail.com
First crash: 421d, last: 16h41m
Bug presence (2)
Date Name Commit Repro Result
2023/12/09 linux-5.15.y (ToT) 8a1d809b0545 C [report] possible deadlock in __jbd2_log_wait_for_space
2023/12/09 upstream (ToT) f2e8a57ee903 C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in __jbd2_log_wait_for_space ext4 C error 34993 384d 633d 22/26 fixed on 2023/06/08 14:41
linux-6.1 possible deadlock in __jbd2_log_wait_for_space origin:lts-only C done 1036 3h14m 421d 0/3 upstream: reported C repro on 2023/03/07 18:21
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/01/02 17:25 1h00m fix candidate upstream job log (0)

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.157-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor950/3559 is trying to acquire lock:
ffff88807ef123f8 (&journal->j_checkpoint_mutex){+.+.}-{3:3}, at: __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:71

but task is already holding lock:
ffff888071a2dda8 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:789 [inline]
ffff888071a2dda8 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xa0/0x360 fs/ext4/file.c:262

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&sb->s_type->i_mutex_key#9){++++}-{3:3}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       down_read+0x45/0x2e0 kernel/locking/rwsem.c:1498
       inode_lock_shared include/linux/fs.h:799 [inline]
       ext4_bmap+0x4b/0x410 fs/ext4/inode.c:3152
       bmap+0xa1/0xd0 fs/inode.c:1756
       jbd2_journal_bmap fs/jbd2/journal.c:980 [inline]
       __jbd2_journal_erase fs/jbd2/journal.c:1790 [inline]
       jbd2_journal_flush+0x7a2/0xc90 fs/jbd2/journal.c:2496
       ext4_ioctl_checkpoint fs/ext4/ioctl.c:849 [inline]
       __ext4_ioctl fs/ext4/ioctl.c:1267 [inline]
       ext4_ioctl+0x3249/0x5b80 fs/ext4/ioctl.c:1276
       vfs_ioctl fs/ioctl.c:51 [inline]
       __do_sys_ioctl fs/ioctl.c:874 [inline]
       __se_sys_ioctl+0xf1/0x160 fs/ioctl.c:860
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #0 (&journal->j_checkpoint_mutex){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __mutex_lock_common+0x1da/0x25a0 kernel/locking/mutex.c:596
       mutex_lock_io_nested+0x45/0x60 kernel/locking/mutex.c:777
       __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:71
       add_transaction_credits+0x950/0xc00 fs/jbd2/transaction.c:299
       start_this_handle+0x747/0x1570 fs/jbd2/transaction.c:423
       jbd2__journal_start+0x2d1/0x5c0 fs/jbd2/transaction.c:521
       __ext4_journal_start_sb+0x175/0x370 fs/ext4/ext4_jbd2.c:105
       __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
       ext4_dirty_inode+0x8b/0x100 fs/ext4/inode.c:5990
       __mark_inode_dirty+0x2fd/0xd60 fs/fs-writeback.c:2464
       generic_update_time+0x1cb/0x1e0 fs/inode.c:1817
       inode_update_time fs/inode.c:1830 [inline]
       file_update_time+0x3eb/0x460 fs/inode.c:2019
       ext4_write_checks+0x24a/0x2c0 fs/ext4/file.c:247
       ext4_buffered_write_iter+0xac/0x360 fs/ext4/file.c:263
       ext4_file_write_iter+0x87c/0x1990
       __kernel_write+0x5b1/0xa60 fs/read_write.c:539
       __dump_emit+0x264/0x3a0 fs/coredump.c:875
       dump_user_range+0x91/0x320 fs/coredump.c:949
       elf_core_dump+0x3c7d/0x4570 fs/binfmt_elf.c:2285
       do_coredump+0x1852/0x31e0 fs/coredump.c:826
       get_signal+0xc06/0x14e0 kernel/signal.c:2875
       arch_do_signal_or_restart+0xc3/0x1890 arch/x86/kernel/signal.c:867
       handle_signal_work kernel/entry/common.c:154 [inline]
       exit_to_user_mode_loop+0x97/0x130 kernel/entry/common.c:178
       exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:214
       irqentry_exit_to_user_mode+0x5/0x30 kernel/entry/common.c:320
       exc_page_fault+0x342/0x740 arch/x86/mm/fault.c:1535
       asm_exc_page_fault+0x22/0x30 arch/x86/include/asm/idtentry.h:568

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&sb->s_type->i_mutex_key#9);
                               lock(&journal->j_checkpoint_mutex);
                               lock(&sb->s_type->i_mutex_key#9);
  lock(&journal->j_checkpoint_mutex);

 *** DEADLOCK ***

2 locks held by syz-executor950/3559:
 #0: ffff88807ef0e460
 (sb_writers#5){.+.+}-{0:0}, at: do_coredump+0x1825/0x31e0 fs/coredump.c:825
 #1: ffff888071a2dda8 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:789 [inline]
 #1: ffff888071a2dda8 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xa0/0x360 fs/ext4/file.c:262

stack backtrace:
CPU: 0 PID: 3559 Comm: syz-executor950 Not tainted 5.15.157-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __mutex_lock_common+0x1da/0x25a0 kernel/locking/mutex.c:596
 mutex_lock_io_nested+0x45/0x60 kernel/locking/mutex.c:777
 __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:71
 add_transaction_credits+0x950/0xc00 fs/jbd2/transaction.c:299
 start_this_handle+0x747/0x1570 fs/jbd2/transaction.c:423
 jbd2__journal_start+0x2d1/0x5c0 fs/jbd2/transaction.c:521
 __ext4_journal_start_sb+0x175/0x370 fs/ext4/ext4_jbd2.c:105
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_dirty_inode+0x8b/0x100 fs/ext4/inode.c:5990
 __mark_inode_dirty+0x2fd/0xd60 fs/fs-writeback.c:2464
 generic_update_time+0x1cb/0x1e0 fs/inode.c:1817
 inode_update_time fs/inode.c:1830 [inline]
 file_update_time+0x3eb/0x460 fs/inode.c:2019
 ext4_write_checks+0x24a/0x2c0 fs/ext4/file.c:247
 ext4_buffered_write_iter+0xac/0x360 fs/ext4/file.c:263
 ext4_file_write_iter+0x87c/0x1990
 __kernel_write+0x5b1/0xa60 fs/read_write.c:539
 __dump_emit+0x264/0x3a0 fs/coredump.c:875
 dump_user_range+0x91/0x320 fs/coredump.c:949
 elf_core_dump+0x3c7d/0x4570 fs/binfmt_elf.c:2285
 do_coredump+0x1852/0x31e0 fs/coredump.c:826
 get_signal+0xc06/0x14e0 kernel/signal.c:2875
 arch_do_signal_or_restart+0xc3/0x1890 arch/x86/kernel/signal.c:867
 handle_signal_work kernel/entry/common.c:154 [inline]
 exit_to_user_mode_loop+0x97/0x130 kernel/entry/common.c:178
 exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:214
 irqentry_exit_to_user_mode+0x5/0x30 kernel/entry/common.c:320
 exc_page_fault+0x342/0x740 arch/x86/mm/fault.c:1535
 asm_exc_page_fault+0x22/0x30 arch/x86/include/asm/idtentry.h:568
RIP: 0033:0x7fc1a195e89e
Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 d2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f
RSP: 002b:00007fc1a1901108 EFLAGS: 00010287
RAX: 00007fc1a1901110 RBX: 00007fc1a1901110 RCX: 0000000000000000
RDX: 00000000000000e0 RSI: 0000000000000000 RDI: 00007fc1a1901110
RBP: 0000000000000001 R08: 00000000000000e0 R09: 00007fc1a19016c0
R10: 00007ffc66c52457 R11: 0000000000000202 R12: 00007fc1a19b91dc
R13: 00007fc1a19ec31c R14: 0030656c69662f2e R15: 00007ffc66c52458
 </TASK>

Crashes (1178):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/05/01 20:15 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2023/12/09 06:48 linux-5.15.y 8a1d809b0545 28b24332 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/03/22 18:35 linux-5.15.y b95c01af2113 4b6cdce6 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/08 00:42 linux-5.15.y 574362648507 2b789849 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/02 00:03 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/01 08:11 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/27 03:57 linux-5.15.y c52b9710c83d 07b455f9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/26 20:32 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/26 15:30 linux-5.15.y c52b9710c83d 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/24 06:48 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/17 15:40 linux-5.15.y c52b9710c83d acc528cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/17 13:38 linux-5.15.y c52b9710c83d 18f6e127 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/16 05:44 linux-5.15.y fa3df276cd36 0d592ce4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/15 07:10 linux-5.15.y fa3df276cd36 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/14 22:49 linux-5.15.y fa3df276cd36 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/13 11:01 linux-5.15.y cdfd0a7f0139 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/12 17:22 linux-5.15.y cdfd0a7f0139 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/12 05:16 linux-5.15.y cdfd0a7f0139 27de0a5c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/11 16:39 linux-5.15.y cdfd0a7f0139 95ed9ece .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/10 17:29 linux-5.15.y cdfd0a7f0139 4320ec32 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/09 19:51 linux-5.15.y 9465fef4ae35 171ec371 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/08 15:17 linux-5.15.y 9465fef4ae35 53df08b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/07 10:10 linux-5.15.y 9465fef4ae35 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/06 22:52 linux-5.15.y 9465fef4ae35 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/02 13:27 linux-5.15.y 9465fef4ae35 eb2966c4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/02 07:36 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/02 03:22 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/02 00:19 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/01 16:21 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/03/28 18:04 linux-5.15.y 9465fef4ae35 e91187ee .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/03/27 11:37 linux-5.15.y 9465fef4ae35 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/03/26 20:14 linux-5.15.y b95c01af2113 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/03/25 07:16 linux-5.15.y b95c01af2113 0ea90952 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in __jbd2_log_wait_for_space
2024/03/23 14:13 linux-5.15.y b95c01af2113 0ea90952 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in __jbd2_log_wait_for_space
2024/03/19 11:40 linux-5.15.y b95c01af2113 baa80228 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/01 06:22 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/01 01:02 linux-5.15.y b925f60c6ee7 3ba885bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/30 17:47 linux-5.15.y b925f60c6ee7 3ce4924c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/30 08:42 linux-5.15.y b925f60c6ee7 f10afd69 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/23 18:30 linux-5.15.y c52b9710c83d 21339d7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/13 17:55 linux-5.15.y fa3df276cd36 c8349e48 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/11 02:03 linux-5.15.y cdfd0a7f0139 33b9e058 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/09 21:55 linux-5.15.y 9465fef4ae35 171ec371 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/06 20:49 linux-5.15.y 9465fef4ae35 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/03 12:38 linux-5.15.y 9465fef4ae35 7925100d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/03 10:44 linux-5.15.y 9465fef4ae35 7925100d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/04/02 09:45 linux-5.15.y 9465fef4ae35 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/29 07:21 linux-5.15.y 9465fef4ae35 c52bcb23 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/26 22:31 linux-5.15.y b95c01af2113 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/24 21:30 linux-5.15.y b95c01af2113 0ea90952 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/22 14:15 linux-5.15.y b95c01af2113 4b6cdce6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/03/21 04:37 linux-5.15.y b95c01af2113 5b7d42ae .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2023/03/07 17:43 linux-5.15.y d9b4a0c83a2d d7ea8bc4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __jbd2_log_wait_for_space
* Struck through repros no longer work on HEAD.