syzbot


possible deadlock in ext4_writepages (2)

Status: upstream: reported on 2024/10/19 20:27
Subsystems: ext4
[Documentation on labels]
Reported-by: syzbot+eb5b4ef634a018917f3c@syzkaller.appspotmail.com
First crash: 50d, last: 2d21h
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [ext4?] possible deadlock in ext4_writepages (2) 0 (1) 2024/10/19 20:27
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in ext4_writepages ext4 1 845d 841d 0/28 auto-obsoleted due to no activity on 2022/12/11 07:00
linux-5.15 possible deadlock in ext4_writepages 2 581d 583d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:09

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.13.0-rc1-syzkaller-00001-ge70140ba0d2b #0 Not tainted
------------------------------------------------------
syz.6.470/10463 is trying to acquire lock:
ffff888034c4cb98 (&sbi->s_writepages_rwsem){++++}-{0:0}, at: ext4_writepages_down_read fs/ext4/ext4.h:1776 [inline]
ffff888034c4cb98 (&sbi->s_writepages_rwsem){++++}-{0:0}, at: ext4_writepages+0x1bf/0x3c0 fs/ext4/inode.c:2823

but task is already holding lock:
ffff88805a328f80 (mapping.invalidate_lock){++++}-{4:4}, at: filemap_invalidate_lock_shared include/linux/fs.h:873 [inline]
ffff88805a328f80 (mapping.invalidate_lock){++++}-{4:4}, at: ext4_page_mkwrite+0x217/0xdf0 fs/ext4/inode.c:6161

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (mapping.invalidate_lock){++++}-{4:4}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       down_read+0xb1/0xa40 kernel/locking/rwsem.c:1524
       filemap_invalidate_lock_shared include/linux/fs.h:873 [inline]
       ext4_page_mkwrite+0x217/0xdf0 fs/ext4/inode.c:6161
       do_page_mkwrite+0x198/0x480 mm/memory.c:3176
       wp_page_shared mm/memory.c:3577 [inline]
       do_wp_page+0x2352/0x5160 mm/memory.c:3727
       handle_pte_fault+0x111e/0x68a0 mm/memory.c:5817
       __handle_mm_fault mm/memory.c:5944 [inline]
       handle_mm_fault+0x1053/0x1ad0 mm/memory.c:6112
       do_user_addr_fault arch/x86/mm/fault.c:1338 [inline]
       handle_page_fault arch/x86/mm/fault.c:1481 [inline]
       exc_page_fault+0x459/0x8b0 arch/x86/mm/fault.c:1539
       asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:623

-> #4 (sb_pagefaults){.+.+}-{0:0}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       percpu_down_read+0x44/0x1b0 include/linux/percpu-rwsem.h:51
       __sb_start_write include/linux/fs.h:1725 [inline]
       sb_start_pagefault include/linux/fs.h:1890 [inline]
       ext4_page_mkwrite+0x1ef/0xdf0 fs/ext4/inode.c:6158
       do_page_mkwrite+0x198/0x480 mm/memory.c:3176
       do_shared_fault mm/memory.c:5398 [inline]
       do_fault mm/memory.c:5460 [inline]
       do_pte_missing mm/memory.c:3979 [inline]
       handle_pte_fault+0x1235/0x68a0 mm/memory.c:5801
       __handle_mm_fault mm/memory.c:5944 [inline]
       handle_mm_fault+0x1053/0x1ad0 mm/memory.c:6112
       do_user_addr_fault arch/x86/mm/fault.c:1389 [inline]
       handle_page_fault arch/x86/mm/fault.c:1481 [inline]
       exc_page_fault+0x2b9/0x8b0 arch/x86/mm/fault.c:1539
       asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:623

-> #3 (&mm->mmap_lock){++++}-{4:4}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       __might_fault+0xc6/0x120 mm/memory.c:6751
       _inline_copy_to_user include/linux/uaccess.h:192 [inline]
       _copy_to_user+0x2c/0xb0 lib/usercopy.c:26
       copy_to_user include/linux/uaccess.h:225 [inline]
       fiemap_fill_next_extent+0x235/0x410 fs/ioctl.c:145
       ocfs2_fiemap_inline fs/ocfs2/extent_map.c:725 [inline]
       ocfs2_fiemap+0xc25/0xf80 fs/ocfs2/extent_map.c:762
       ioctl_fiemap fs/ioctl.c:220 [inline]
       do_vfs_ioctl+0x1c01/0x2e40 fs/ioctl.c:840
       __do_sys_ioctl fs/ioctl.c:904 [inline]
       __se_sys_ioctl+0x80/0x170 fs/ioctl.c:892
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #2 (&ocfs2_quota_ip_alloc_sem_key){++++}-{4:4}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       down_write+0x99/0x220 kernel/locking/rwsem.c:1577
       ocfs2_create_local_dquot+0x1de/0x1d70 fs/ocfs2/quota_local.c:1231
       ocfs2_acquire_dquot+0x833/0xb70 fs/ocfs2/quota_global.c:878
       dqget+0x770/0xeb0 fs/quota/dquot.c:975
       __dquot_initialize+0x2e3/0xec0 fs/quota/dquot.c:1503
       ocfs2_get_init_inode+0x158/0x1c0 fs/ocfs2/namei.c:202
       ocfs2_mknod+0xcfa/0x2b30 fs/ocfs2/namei.c:308
       ocfs2_create+0x1ab/0x470 fs/ocfs2/namei.c:672
       lookup_open fs/namei.c:3649 [inline]
       open_last_lookups fs/namei.c:3748 [inline]
       path_openat+0x1c03/0x3590 fs/namei.c:3984
       do_filp_open+0x27f/0x4e0 fs/namei.c:4014
       do_sys_openat2+0x13e/0x1d0 fs/open.c:1402
       do_sys_open fs/open.c:1417 [inline]
       __do_sys_openat fs/open.c:1433 [inline]
       __se_sys_openat fs/open.c:1428 [inline]
       __x64_sys_openat+0x247/0x2a0 fs/open.c:1428
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #1 (&dquot->dq_lock){+.+.}-{4:4}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       __mutex_lock_common kernel/locking/mutex.c:585 [inline]
       __mutex_lock+0x1ac/0xee0 kernel/locking/mutex.c:735
       wait_on_dquot fs/quota/dquot.c:354 [inline]
       dqget+0x6e6/0xeb0 fs/quota/dquot.c:970
       __dquot_initialize+0x2e3/0xec0 fs/quota/dquot.c:1503
       __ext4_new_inode+0x852/0x4380 fs/ext4/ialloc.c:992
       ext4_ext_migrate+0x6c1/0x13e0 fs/ext4/migrate.c:456
       ext4_ioctl_setflags fs/ext4/ioctl.c:702 [inline]
       ext4_fileattr_set+0xecf/0x16e0 fs/ext4/ioctl.c:1021
       vfs_fileattr_set+0x8ff/0xd50 fs/ioctl.c:696
       ioctl_setflags fs/ioctl.c:728 [inline]
       do_vfs_ioctl+0x2083/0x2e40 fs/ioctl.c:869
       __do_sys_ioctl fs/ioctl.c:904 [inline]
       __se_sys_ioctl+0x80/0x170 fs/ioctl.c:892
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&sbi->s_writepages_rwsem){++++}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3161 [inline]
       check_prevs_add kernel/locking/lockdep.c:3280 [inline]
       validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
       __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5226
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
       percpu_down_read+0x44/0x1b0 include/linux/percpu-rwsem.h:51
       ext4_writepages_down_read fs/ext4/ext4.h:1776 [inline]
       ext4_writepages+0x1bf/0x3c0 fs/ext4/inode.c:2823
       do_writepages+0x35f/0x880 mm/page-writeback.c:2702
       filemap_fdatawrite_wbc mm/filemap.c:397 [inline]
       __filemap_fdatawrite_range mm/filemap.c:430 [inline]
       __filemap_fdatawrite mm/filemap.c:436 [inline]
       filemap_flush+0x1d2/0x260 mm/filemap.c:463
       ext4_convert_inline_data+0x1a5/0x620 fs/ext4/inline.c:2039
       ext4_page_mkwrite+0x21f/0xdf0 fs/ext4/inode.c:6163
       do_page_mkwrite+0x198/0x480 mm/memory.c:3176
       do_shared_fault mm/memory.c:5398 [inline]
       do_fault mm/memory.c:5460 [inline]
       do_pte_missing mm/memory.c:3979 [inline]
       handle_pte_fault+0x1235/0x68a0 mm/memory.c:5801
       __handle_mm_fault mm/memory.c:5944 [inline]
       handle_mm_fault+0x1053/0x1ad0 mm/memory.c:6112
       do_user_addr_fault arch/x86/mm/fault.c:1389 [inline]
       handle_page_fault arch/x86/mm/fault.c:1481 [inline]
       exc_page_fault+0x2b9/0x8b0 arch/x86/mm/fault.c:1539
       asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:623
       rep_movs_alternative+0x4a/0x70 arch/x86/lib/copy_user_64.S:65
       copy_user_generic arch/x86/include/asm/uaccess_64.h:126 [inline]
       raw_copy_to_user arch/x86/include/asm/uaccess_64.h:147 [inline]
       copy_to_user_iter lib/iov_iter.c:25 [inline]
       iterate_ubuf include/linux/iov_iter.h:30 [inline]
       iterate_and_advance2 include/linux/iov_iter.h:300 [inline]
       iterate_and_advance include/linux/iov_iter.h:328 [inline]
       _copy_to_iter+0x29a/0x1e70 lib/iov_iter.c:185
       copy_page_to_iter+0xb1/0x160 lib/iov_iter.c:362
       copy_folio_to_iter include/linux/uio.h:189 [inline]
       filemap_read+0x7e3/0xf50 mm/filemap.c:2697
       new_sync_read fs/read_write.c:484 [inline]
       vfs_read+0x991/0xb70 fs/read_write.c:565
       ksys_read+0x18f/0x2b0 fs/read_write.c:708
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

Chain exists of:
  &sbi->s_writepages_rwsem --> sb_pagefaults --> mapping.invalidate_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  rlock(mapping.invalidate_lock);
                               lock(sb_pagefaults);
                               lock(mapping.invalidate_lock);
  rlock(&sbi->s_writepages_rwsem);

 *** DEADLOCK ***

4 locks held by syz.6.470/10463:
 #0: ffff888032829278 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos+0x254/0x320 fs/file.c:1191
 #1: ffff88806b4fc460 (&mm->mmap_lock){++++}-{4:4}, at: mmap_read_trylock include/linux/mmap_lock.h:163 [inline]
 #1: ffff88806b4fc460 (&mm->mmap_lock){++++}-{4:4}, at: get_mmap_lock_carefully mm/memory.c:6149 [inline]
 #1: ffff88806b4fc460 (&mm->mmap_lock){++++}-{4:4}, at: lock_mm_and_find_vma+0x32/0x2f0 mm/memory.c:6209
 #2: ffff888034c4a518 (sb_pagefaults){.+.+}-{0:0}, at: __sb_start_write include/linux/fs.h:1725 [inline]
 #2: ffff888034c4a518 (sb_pagefaults){.+.+}-{0:0}, at: sb_start_pagefault include/linux/fs.h:1890 [inline]
 #2: ffff888034c4a518 (sb_pagefaults){.+.+}-{0:0}, at: ext4_page_mkwrite+0x1ef/0xdf0 fs/ext4/inode.c:6158
 #3: ffff88805a328f80 (mapping.invalidate_lock){++++}-{4:4}, at: filemap_invalidate_lock_shared include/linux/fs.h:873 [inline]
 #3: ffff88805a328f80 (mapping.invalidate_lock){++++}-{4:4}, at: ext4_page_mkwrite+0x217/0xdf0 fs/ext4/inode.c:6161

stack backtrace:
CPU: 0 UID: 0 PID: 10463 Comm: syz.6.470 Not tainted 6.13.0-rc1-syzkaller-00001-ge70140ba0d2b #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 print_circular_bug+0x13a/0x1b0 kernel/locking/lockdep.c:2074
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2206
 check_prev_add kernel/locking/lockdep.c:3161 [inline]
 check_prevs_add kernel/locking/lockdep.c:3280 [inline]
 validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
 __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5226
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
 percpu_down_read+0x44/0x1b0 include/linux/percpu-rwsem.h:51
 ext4_writepages_down_read fs/ext4/ext4.h:1776 [inline]
 ext4_writepages+0x1bf/0x3c0 fs/ext4/inode.c:2823
 do_writepages+0x35f/0x880 mm/page-writeback.c:2702
 filemap_fdatawrite_wbc mm/filemap.c:397 [inline]
 __filemap_fdatawrite_range mm/filemap.c:430 [inline]
 __filemap_fdatawrite mm/filemap.c:436 [inline]
 filemap_flush+0x1d2/0x260 mm/filemap.c:463
 ext4_convert_inline_data+0x1a5/0x620 fs/ext4/inline.c:2039
 ext4_page_mkwrite+0x21f/0xdf0 fs/ext4/inode.c:6163
 do_page_mkwrite+0x198/0x480 mm/memory.c:3176
 do_shared_fault mm/memory.c:5398 [inline]
 do_fault mm/memory.c:5460 [inline]
 do_pte_missing mm/memory.c:3979 [inline]
 handle_pte_fault+0x1235/0x68a0 mm/memory.c:5801
 __handle_mm_fault mm/memory.c:5944 [inline]
 handle_mm_fault+0x1053/0x1ad0 mm/memory.c:6112
 do_user_addr_fault arch/x86/mm/fault.c:1389 [inline]
 handle_page_fault arch/x86/mm/fault.c:1481 [inline]
 exc_page_fault+0x2b9/0x8b0 arch/x86/mm/fault.c:1539
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:623
RIP: 0010:rep_movs_alternative+0x4a/0x70 arch/x86/lib/copy_user_64.S:71
Code: 75 f1 c3 cc cc cc cc 66 0f 1f 84 00 00 00 00 00 48 8b 06 48 89 07 48 83 c6 08 48 83 c7 08 83 e9 08 74 df 83 f9 08 73 e8 eb c9 <f3> a4 c3 cc cc cc cc 48 89 c8 48 c1 e9 03 83 e0 07 f3 48 a5 89 c1
RSP: 0018:ffffc90000ab7838 EFLAGS: 00050206
RAX: ffffffff84bf3d01 RBX: 0000000000001000 RCX: 0000000000001000
RDX: 0000000000000000 RSI: ffff888056367000 RDI: 0000000020001f00
RBP: ffffc90000ab79b8 R08: ffff888056367fff R09: 1ffff1100ac6cfff
R10: dffffc0000000000 R11: ffffed100ac6d000 R12: ffffc90000ab7d98
R13: 0000000020001f00 R14: ffff888056367000 R15: 1ffff92000156fb3
 copy_user_generic arch/x86/include/asm/uaccess_64.h:126 [inline]
 raw_copy_to_user arch/x86/include/asm/uaccess_64.h:147 [inline]
 copy_to_user_iter lib/iov_iter.c:25 [inline]
 iterate_ubuf include/linux/iov_iter.h:30 [inline]
 iterate_and_advance2 include/linux/iov_iter.h:300 [inline]
 iterate_and_advance include/linux/iov_iter.h:328 [inline]
 _copy_to_iter+0x29a/0x1e70 lib/iov_iter.c:185
 copy_page_to_iter+0xb1/0x160 lib/iov_iter.c:362
 copy_folio_to_iter include/linux/uio.h:189 [inline]
 filemap_read+0x7e3/0xf50 mm/filemap.c:2697
 new_sync_read fs/read_write.c:484 [inline]
 vfs_read+0x991/0xb70 fs/read_write.c:565
 ksys_read+0x18f/0x2b0 fs/read_write.c:708
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f840db7ff19
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f840e9de058 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 00007f840dd45fa0 RCX: 00007f840db7ff19
RDX: 0000000000002020 RSI: 0000000020001f00 RDI: 0000000000000004
RBP: 00007f840dbf3986 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f840dd45fa0 R15: 00007ffe48362908
 </TASK>
EXT4-fs error (device loop6): ext4_mb_generate_buddy:1220: group 0, block bitmap and bg descriptor inconsistent: 25 vs 4128793 free clusters
EXT4-fs (loop6): Delayed block allocation failed for inode 15 at logical offset 0 with max blocks 1 with error 28
EXT4-fs (loop6): This should not happen!! Data will be lost

EXT4-fs (loop6): Total free blocks count 0
EXT4-fs (loop6): Free/Dirty block details
EXT4-fs (loop6): free_blocks=66060288
EXT4-fs (loop6): dirty_blocks=32
EXT4-fs (loop6): Block reservation details
EXT4-fs (loop6): i_reserved_data_blocks=2
----------------
Code disassembly (best guess):
   0:	75 f1                	jne    0xfffffff3
   2:	c3                   	ret
   3:	cc                   	int3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	66 0f 1f 84 00 00 00 	nopw   0x0(%rax,%rax,1)
   e:	00 00
  10:	48 8b 06             	mov    (%rsi),%rax
  13:	48 89 07             	mov    %rax,(%rdi)
  16:	48 83 c6 08          	add    $0x8,%rsi
  1a:	48 83 c7 08          	add    $0x8,%rdi
  1e:	83 e9 08             	sub    $0x8,%ecx
  21:	74 df                	je     0x2
  23:	83 f9 08             	cmp    $0x8,%ecx
  26:	73 e8                	jae    0x10
  28:	eb c9                	jmp    0xfffffff3
* 2a:	f3 a4                	rep movsb %ds:(%rsi),%es:(%rdi) <-- trapping instruction
  2c:	c3                   	ret
  2d:	cc                   	int3
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	48 89 c8             	mov    %rcx,%rax
  34:	48 c1 e9 03          	shr    $0x3,%rcx
  38:	83 e0 07             	and    $0x7,%eax
  3b:	f3 48 a5             	rep movsq %ds:(%rsi),%es:(%rdi)
  3e:	89 c1                	mov    %eax,%ecx

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/02 22:03 upstream e70140ba0d2b b499ea68 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in ext4_writepages
2024/10/15 20:23 upstream 2f87d0916ce0 7eb57b4a .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs possible deadlock in ext4_writepages
* Struck through repros no longer work on HEAD.