syzbot


possible deadlock in __jbd2_log_wait_for_space

Status: upstream: reported C repro on 2023/03/07 18:21
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+c5b10e098e75430412f1@syzkaller.appspotmail.com
First crash: 583d, last: 101d
Fix commit to backport (bisect log) :
tree: upstream
commit 62913ae96de747091c4dacd06d158e7729c1a76d
Author: Theodore Ts'o <tytso@mit.edu>
Date: Wed Mar 8 04:15:49 2023 +0000

  ext4, jbd2: add an optimized bmap for the journal inode

[report pending]
  
Bug presence (2)
Date Name Commit Repro Result
2023/08/15 linux-6.1.y (ToT) 1321ab403b38 C [report] INFO: rcu detected stall in corrupted
2023/08/15 upstream (ToT) 2ccdd1b13c59 C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in __jbd2_log_wait_for_space origin:lts-only C inconclusive 1257 7d11h 583d 0/3 upstream: reported C repro on 2023/03/07 17:44
upstream possible deadlock in __jbd2_log_wait_for_space ext4 C error 34993 546d 794d 22/28 fixed on 2023/06/08 14:41
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2023/09/07 14:16 6h13m fix candidate upstream OK (1) job log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.1.45-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor266/3583 is trying to acquire lock:
ffff88807f7523f8 (&journal->j_checkpoint_mutex){+.+.}-{3:3}, at: __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:88

but task is already holding lock:
ffff88807092ca38 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:756 [inline]
ffff88807092ca38 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xaf/0x3a0 fs/ext4/file.c:279

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&sb->s_type->i_mutex_key#8){++++}-{3:3}:
       lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5669
       down_read+0x43/0x2e0 kernel/locking/rwsem.c:1520
       inode_lock_shared include/linux/fs.h:766 [inline]
       ext4_bmap+0x4b/0x410 fs/ext4/inode.c:3164
       bmap+0xa1/0xd0 fs/inode.c:1840
       jbd2_journal_bmap fs/jbd2/journal.c:977 [inline]
       __jbd2_journal_erase fs/jbd2/journal.c:1789 [inline]
       jbd2_journal_flush+0x5b5/0xc40 fs/jbd2/journal.c:2492
       ext4_ioctl_checkpoint fs/ext4/ioctl.c:1086 [inline]
       __ext4_ioctl fs/ext4/ioctl.c:1594 [inline]
       ext4_ioctl+0x3986/0x5f60 fs/ext4/ioctl.c:1614
       vfs_ioctl fs/ioctl.c:51 [inline]
       __do_sys_ioctl fs/ioctl.c:870 [inline]
       __se_sys_ioctl+0xf1/0x160 fs/ioctl.c:856
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x63/0xcd

-> #0 (&journal->j_checkpoint_mutex){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3098 [inline]
       check_prevs_add kernel/locking/lockdep.c:3217 [inline]
       validate_chain+0x1667/0x58e0 kernel/locking/lockdep.c:3832
       __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5056
       lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5669
       __mutex_lock_common+0x1d4/0x2520 kernel/locking/mutex.c:603
       mutex_lock_io_nested+0x43/0x60 kernel/locking/mutex.c:833
       __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:88
       add_transaction_credits+0x94c/0xc00 fs/jbd2/transaction.c:298
       start_this_handle+0x747/0x1640 fs/jbd2/transaction.c:422
       jbd2__journal_start+0x2d1/0x5c0 fs/jbd2/transaction.c:520
       __ext4_journal_start_sb+0x19b/0x410 fs/ext4/ext4_jbd2.c:105
       __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
       ext4_dirty_inode+0x8b/0x100 fs/ext4/inode.c:6057
       __mark_inode_dirty+0x331/0xf80 fs/fs-writeback.c:2411
       generic_update_time fs/inode.c:1901 [inline]
       inode_update_time fs/inode.c:1914 [inline]
       __file_update_time+0x221/0x240 fs/inode.c:2102
       file_modified_flags+0x3e1/0x480 fs/inode.c:2175
       ext4_write_checks+0x24a/0x2c0 fs/ext4/file.c:264
       ext4_buffered_write_iter+0xbd/0x3a0 fs/ext4/file.c:280
       ext4_file_write_iter+0x1d2/0x18f0
       __kernel_write_iter+0x2ff/0x710 fs/read_write.c:517
       dump_emit_page fs/coredump.c:881 [inline]
       dump_user_range+0x43d/0x8e0 fs/coredump.c:908
       elf_core_dump+0x3cff/0x45b0 fs/binfmt_elf.c:2312
       do_coredump+0x18b7/0x2700 fs/coredump.c:755
       get_signal+0x1454/0x17d0 kernel/signal.c:2848
       arch_do_signal_or_restart+0xb0/0x1a10 arch/x86/kernel/signal.c:871
       exit_to_user_mode_loop+0x6a/0x100 kernel/entry/common.c:168
       exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:204
       irqentry_exit_to_user_mode+0x5/0x30 kernel/entry/common.c:310
       exc_page_fault+0x1c0/0x660 arch/x86/mm/fault.c:1530
       asm_exc_page_fault+0x22/0x30 arch/x86/include/asm/idtentry.h:570

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&sb->s_type->i_mutex_key#8);
                               lock(&journal->j_checkpoint_mutex);
                               lock(&sb->s_type->i_mutex_key#8);
  lock(&journal->j_checkpoint_mutex);

 *** DEADLOCK ***

2 locks held by syz-executor266/3583:
 #0: ffff88807f74e460 (sb_writers#4){.+.+}-{0:0}, at: do_coredump+0x1892/0x2700 fs/coredump.c:754
 #1: ffff88807092ca38 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:756 [inline]
 #1: ffff88807092ca38 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xaf/0x3a0 fs/ext4/file.c:279

stack backtrace:
CPU: 0 PID: 3583 Comm: syz-executor266 Not tainted 6.1.45-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 check_noncircular+0x2fa/0x3b0 kernel/locking/lockdep.c:2178
 check_prev_add kernel/locking/lockdep.c:3098 [inline]
 check_prevs_add kernel/locking/lockdep.c:3217 [inline]
 validate_chain+0x1667/0x58e0 kernel/locking/lockdep.c:3832
 __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5056
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5669
 __mutex_lock_common+0x1d4/0x2520 kernel/locking/mutex.c:603
 mutex_lock_io_nested+0x43/0x60 kernel/locking/mutex.c:833
 __jbd2_log_wait_for_space+0x213/0x760 fs/jbd2/checkpoint.c:88
 add_transaction_credits+0x94c/0xc00 fs/jbd2/transaction.c:298
 start_this_handle+0x747/0x1640 fs/jbd2/transaction.c:422
 jbd2__journal_start+0x2d1/0x5c0 fs/jbd2/transaction.c:520
 __ext4_journal_start_sb+0x19b/0x410 fs/ext4/ext4_jbd2.c:105
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_dirty_inode+0x8b/0x100 fs/ext4/inode.c:6057
 __mark_inode_dirty+0x331/0xf80 fs/fs-writeback.c:2411
 generic_update_time fs/inode.c:1901 [inline]
 inode_update_time fs/inode.c:1914 [inline]
 __file_update_time+0x221/0x240 fs/inode.c:2102
 file_modified_flags+0x3e1/0x480 fs/inode.c:2175
 ext4_write_checks+0x24a/0x2c0 fs/ext4/file.c:264
 ext4_buffered_write_iter+0xbd/0x3a0 fs/ext4/file.c:280
 ext4_file_write_iter+0x1d2/0x18f0
 __kernel_write_iter+0x2ff/0x710 fs/read_write.c:517
 dump_emit_page fs/coredump.c:881 [inline]
 dump_user_range+0x43d/0x8e0 fs/coredump.c:908
 elf_core_dump+0x3cff/0x45b0 fs/binfmt_elf.c:2312
 do_coredump+0x18b7/0x2700 fs/coredump.c:755
 get_signal+0x1454/0x17d0 kernel/signal.c:2848
 arch_do_signal_or_restart+0xb0/0x1a10 arch/x86/kernel/signal.c:871
 exit_to_user_mode_loop+0x6a/0x100 kernel/entry/common.c:168
 exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:204
 irqentry_exit_to_user_mode+0x5/0x30 kernel/entry/common.c:310
 exc_page_fault+0x1c0/0x660 arch/x86/mm/fault.c:1530
 asm_exc_page_fault+0x22/0x30 arch/x86/include/asm/idtentry.h:570
RIP: 0033:0x0
Code: Unable to access opcode bytes at 0xffffffffffffffd6.
RSP: 002b:0000000020000448 EFLAGS: 00010217
RAX: 0000000000000000 RBX: 00007f9b6511d3c8 RCX: 00007f9b65095e79
RDX: 0000000000000000 RSI: 0000000020000440 RDI: 0000000000080400
RBP: 00007f9b6511d3c0 R08: 0000000000000000 R09: 00007f9b650536c0
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f9b650ea198
R13: 00007f9b6511d3cc R14: 0030656c69662f2e R15: 00007fff488d9798
 </TASK>

Crashes (1121):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/08/15 00:09 linux-6.1.y 1321ab403b38 39990d51 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/04/07 15:47 linux-6.1.y 347385861c50 ca620dd8 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/01/02 15:14 linux-6.1.y a507f147e6f0 fb427a07 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/07/02 05:22 linux-6.1.y 99e6a620de00 b294e901 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/07/02 03:37 linux-6.1.y 99e6a620de00 b294e901 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/07/01 18:23 linux-6.1.y 99e6a620de00 b294e901 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/29 06:14 linux-6.1.y 99e6a620de00 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/28 22:46 linux-6.1.y 99e6a620de00 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/22 06:15 linux-6.1.y eb44d83053d6 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/22 04:30 linux-6.1.y eb44d83053d6 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/20 13:41 linux-6.1.y eb44d83053d6 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in __jbd2_log_wait_for_space
2024/06/17 16:07 linux-6.1.y eb44d83053d6 1f11cfd7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/17 04:23 linux-6.1.y eb44d83053d6 f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/13 20:09 linux-6.1.y ae9f2a70d69e a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/12 21:17 linux-6.1.y ae9f2a70d69e f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/12 20:41 linux-6.1.y ae9f2a70d69e f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/12 11:03 linux-6.1.y ae9f2a70d69e f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/11 18:33 linux-6.1.y 88690811da69 4d75f4f7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/10 04:17 linux-6.1.y 88690811da69 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/05 05:44 linux-6.1.y 88690811da69 e1e2c66e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/05 01:30 linux-6.1.y 88690811da69 e1e2c66e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/04 19:44 linux-6.1.y 88690811da69 11f2afa5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/04 05:38 linux-6.1.y 88690811da69 a1feae05 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/04 00:36 linux-6.1.y 88690811da69 a1feae05 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/03 17:49 linux-6.1.y 88690811da69 0aba2352 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/31 19:46 linux-6.1.y 88690811da69 0c378259 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/23 23:18 linux-6.1.y 4078fa637fcd 8f98448e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/23 08:52 linux-6.1.y 4078fa637fcd 4c2072ee .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/18 10:09 linux-6.1.y 4078fa637fcd c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/17 16:53 linux-6.1.y 4078fa637fcd a12e99e7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/05/16 13:58 linux-6.1.y 909ba1f1b414 ef5d53ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
2024/06/29 02:41 linux-6.1.y 99e6a620de00 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/19 06:23 linux-6.1.y eb44d83053d6 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/18 01:06 linux-6.1.y eb44d83053d6 ce6011bc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/12 18:37 linux-6.1.y ae9f2a70d69e f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/12 07:49 linux-6.1.y 88690811da69 4d75f4f7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/08 01:53 linux-6.1.y 88690811da69 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/06/03 19:08 linux-6.1.y 88690811da69 0aba2352 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/29 07:27 linux-6.1.y 88690811da69 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/27 18:38 linux-6.1.y 88690811da69 761766e6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/21 11:54 linux-6.1.y 4078fa637fcd 4c0d3ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/20 09:11 linux-6.1.y 4078fa637fcd c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/20 06:43 linux-6.1.y 4078fa637fcd c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/18 05:23 linux-6.1.y 4078fa637fcd c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/17 23:50 linux-6.1.y 4078fa637fcd a12e99e7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/17 23:30 linux-6.1.y 4078fa637fcd a12e99e7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2024/05/17 06:18 linux-6.1.y 909ba1f1b414 c2e07261 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 possible deadlock in __jbd2_log_wait_for_space
2023/03/07 18:21 linux-6.1.y 42616e0f09fb d7ea8bc4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan possible deadlock in __jbd2_log_wait_for_space
* Struck through repros no longer work on HEAD.