syzbot


possible deadlock in fat_write_inode

Status: upstream: reported on 2022/12/25 00:59
Reported-by: syzbot+34f4e24356c17bbf1ccb@syzkaller.appspotmail.com
First crash: 710d, last: 710d

Sample crash report:
IPVS: ftp: loaded support on port[0] = 21
======================================================
IPVS: ftp: loaded support on port[0] = 21
WARNING: possible circular locking dependency detected
4.19.211-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:7/9609 is trying to acquire lock:
00000000d1a8feba (&sbi->s_lock){+.+.}, at: fat_write_inode fs/fat/inode.c:911 [inline]
00000000d1a8feba (&sbi->s_lock){+.+.}, at: fat_write_inode+0xfa/0x180 fs/fat/inode.c:904

but task is already holding lock:
00000000c7ee5ebf ((work_completion)(&(&wb->dwork)->work)){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #6 ((work_completion)(&(&wb->dwork)->work)){+.+.}:
       wb_shutdown+0x172/0x210 mm/backing-dev.c:374
       bdi_unregister+0x169/0x610 mm/backing-dev.c:946
       del_gendisk+0x7f6/0xa80 block/genhd.c:788
       loop_remove drivers/block/loop.c:2066 [inline]
       loop_control_ioctl drivers/block/loop.c:2165 [inline]
       loop_control_ioctl+0x3b1/0x480 drivers/block/loop.c:2131
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:501 [inline]
       do_vfs_ioctl+0xcdb/0x12e0 fs/ioctl.c:688
       ksys_ioctl+0x9b/0xc0 fs/ioctl.c:705
       __do_sys_ioctl fs/ioctl.c:712 [inline]
       __se_sys_ioctl fs/ioctl.c:710 [inline]
       __x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:710
       do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #5 (loop_ctl_mutex){+.+.}:
       lo_open+0x19/0xd0 drivers/block/loop.c:1771
       __blkdev_get+0x372/0x1480 fs/block_dev.c:1494
       blkdev_get+0xb0/0x940 fs/block_dev.c:1627
       blkdev_open+0x202/0x290 fs/block_dev.c:1788
       do_dentry_open+0x4aa/0x1160 fs/open.c:796
       do_last fs/namei.c:3421 [inline]
       path_openat+0x793/0x2df0 fs/namei.c:3537
       do_filp_open+0x18c/0x3f0 fs/namei.c:3567
       do_sys_open+0x3b3/0x520 fs/open.c:1085
       do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #4 (&bdev->bd_mutex){+.+.}:
       blkdev_put+0x30/0x520 fs/block_dev.c:1839
       btrfs_close_bdev fs/btrfs/volumes.c:1033 [inline]
       btrfs_close_one_device fs/btrfs/volumes.c:1057 [inline]
       close_fs_devices.part.0+0x24d/0x8e0 fs/btrfs/volumes.c:1085
       close_fs_devices fs/btrfs/volumes.c:1117 [inline]
       btrfs_close_devices+0x95/0x1f0 fs/btrfs/volumes.c:1103
       close_ctree+0x3c8/0x850 fs/btrfs/disk-io.c:4047
       generic_shutdown_super+0x144/0x370 fs/super.c:456
       kill_anon_super+0x36/0x60 fs/super.c:1032
       btrfs_kill_super+0x49/0x550 fs/btrfs/super.c:2221
       deactivate_locked_super+0x94/0x160 fs/super.c:329
       deactivate_super+0x174/0x1a0 fs/super.c:360
       cleanup_mnt+0x1a8/0x290 fs/namespace.c:1098
       task_work_run+0x148/0x1c0 kernel/task_work.c:113
       tracehook_notify_resume include/linux/tracehook.h:193 [inline]
       exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
       prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
       syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
       do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #3 (&fs_devs->device_list_mutex){+.+.}:
       btrfs_finish_chunk_alloc+0x27b/0xf90 fs/btrfs/volumes.c:4938
       btrfs_create_pending_block_groups+0x242/0x590 fs/btrfs/extent-tree.c:10134
       __btrfs_end_transaction+0x21a/0xb00 fs/btrfs/transaction.c:855
       flush_space+0xa41/0xee0 fs/btrfs/extent-tree.c:4861
       btrfs_async_reclaim_metadata_space+0x466/0x1050 fs/btrfs/extent-tree.c:4977
       process_one_work+0x864/0x1570 kernel/workqueue.c:2153
       worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
       kthread+0x33f/0x460 kernel/kthread.c:259
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #2 (sb_internal#2){.+.+}:
       sb_start_intwrite include/linux/fs.h:1626 [inline]
       start_transaction+0xa37/0xf90 fs/btrfs/transaction.c:528
       btrfs_dirty_inode+0xe3/0x210 fs/btrfs/inode.c:6165
       btrfs_update_time+0x33b/0x3d0 fs/btrfs/inode.c:6207
       update_time fs/inode.c:1675 [inline]
       touch_atime+0x23c/0x2a0 fs/inode.c:1746
       file_accessed include/linux/fs.h:2123 [inline]
       btrfs_file_mmap+0x11b/0x160 fs/btrfs/file.c:2274
       call_mmap include/linux/fs.h:1826 [inline]
       mmap_region+0xc94/0x16b0 mm/mmap.c:1757
       do_mmap+0x8e8/0x1080 mm/mmap.c:1530
       do_mmap_pgoff include/linux/mm.h:2329 [inline]
       vm_mmap_pgoff+0x197/0x200 mm/util.c:357
       ksys_mmap_pgoff+0x298/0x5a0 mm/mmap.c:1580
       do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&mm->mmap_sem){++++}:
       _copy_to_user+0x29/0x100 lib/usercopy.c:25
       copy_to_user include/linux/uaccess.h:155 [inline]
       filldir64+0x26e/0x430 fs/readdir.c:324
       dir_emit_dot include/linux/fs.h:3432 [inline]
       dir_emit_dots include/linux/fs.h:3443 [inline]
       __fat_readdir+0xef1/0x1490 fs/fat/dir.c:575
       iterate_dir+0x473/0x5c0 fs/readdir.c:51
       ksys_getdents64+0x175/0x2b0 fs/readdir.c:357
       __do_sys_getdents64 fs/readdir.c:376 [inline]
       __se_sys_getdents64 fs/readdir.c:373 [inline]
       __x64_sys_getdents64+0x6f/0xb0 fs/readdir.c:373
       do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&sbi->s_lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:937 [inline]
       __mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
       fat_write_inode fs/fat/inode.c:911 [inline]
       fat_write_inode+0xfa/0x180 fs/fat/inode.c:904
       write_inode fs/fs-writeback.c:1244 [inline]
       __writeback_single_inode+0x733/0x11d0 fs/fs-writeback.c:1442
       writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
       wb_writeback+0x28d/0xcc0 fs/fs-writeback.c:1820
       wb_do_writeback fs/fs-writeback.c:1965 [inline]
       wb_workfn+0x29b/0x1250 fs/fs-writeback.c:2006
       process_one_work+0x864/0x1570 kernel/workqueue.c:2153
       worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
       kthread+0x33f/0x460 kernel/kthread.c:259
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

other info that might help us debug this:

Chain exists of:
  &sbi->s_lock --> loop_ctl_mutex --> (work_completion)(&(&wb->dwork)->work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&(&wb->dwork)->work));
                               lock(loop_ctl_mutex);
                               lock((work_completion)(&(&wb->dwork)->work));
  lock(&sbi->s_lock);

 *** DEADLOCK ***

2 locks held by kworker/u4:7/9609:
 #0: 0000000058a0c2fe ((wq_completion)"writeback"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2124
 #1: 00000000c7ee5ebf ((work_completion)(&(&wb->dwork)->work)){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128

stack backtrace:
CPU: 1 PID: 9609 Comm: kworker/u4:7 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: writeback wb_workfn (flush-7:3)
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1222
 check_prev_add kernel/locking/lockdep.c:1866 [inline]
 check_prevs_add kernel/locking/lockdep.c:1979 [inline]
 validate_chain kernel/locking/lockdep.c:2420 [inline]
 __lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3416
 lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3908
 __mutex_lock_common kernel/locking/mutex.c:937 [inline]
 __mutex_lock+0xd7/0x1190 kernel/locking/mutex.c:1078
 fat_write_inode fs/fat/inode.c:911 [inline]
 fat_write_inode+0xfa/0x180 fs/fat/inode.c:904
 write_inode fs/fs-writeback.c:1244 [inline]
 __writeback_single_inode+0x733/0x11d0 fs/fs-writeback.c:1442
 writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
 wb_writeback+0x28d/0xcc0 fs/fs-writeback.c:1820
 wb_do_writeback fs/fs-writeback.c:1965 [inline]
 wb_workfn+0x29b/0x1250 fs/fs-writeback.c:2006
 process_one_work+0x864/0x1570 kernel/workqueue.c:2153
 worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
BTRFS info (device loop3): enabling inode map caching
BTRFS warning (device loop3): excessive commit interval 622039222
BTRFS warning (device <unknown>): duplicate device /dev/loop2 devid 1 generation 8 scanned by syz-executor.2 (15262)
BTRFS warning (device <unknown>): duplicate device /dev/loop2 devid 1 generation 8 scanned by systemd-udevd (15291)
BTRFS info (device loop3): force zlib compression, level 3
BTRFS info (device loop3): using free space tree
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by systemd-udevd (15289)
BTRFS info (device loop3): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15278)
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop3): enabling inode map caching
BTRFS warning (device loop3): excessive commit interval 622039222
BTRFS info (device loop3): force zlib compression, level 3
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15366)
BTRFS info (device loop3): using free space tree
BTRFS info (device loop3): has skinny extents
IPVS: ftp: loaded support on port[0] = 21
kauditd_printk_skb: 61 callbacks suppressed
audit: type=1800 audit(1671929936.963:134): pid=15470 uid=0 auid=4294967295 ses=4294967295 subj==unconfined op=collect_data cause=failed comm="syz-executor.1" name="memory.events" dev="sda1" ino=13943 res=0
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy1 wpan1: encryption failed: -22
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by syz-executor.5 (15451)
BTRFS info (device loop2): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop5 devid 1 generation 8 scanned by systemd-udevd (15493)
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by syz-executor.3 (15474)
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by systemd-udevd (15528)
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): found reiserfs format "3.6" with non-standard journal
REISERFS (device loop5): using ordered data mode
ieee802154 phy0 wpan0: encryption failed: -22
reiserfs: using flush barriers
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): journal params: device loop5, size 512, journal first block 18, max trans len 256, max batch 225, max commit age 30, max trans age 30
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): checking transaction log (loop5)
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): Using r5 hash to sort names
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop5): Created .reiserfs_priv - reserved for xattr storage.
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): enabling inode map caching
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS warning (device loop2): excessive commit interval 622039222
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): force zlib compression, level 3
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): using free space tree
ieee802154 phy0 wpan0: encryption failed: -22
BTRFS info (device loop2): has skinny extents
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
REISERFS (device loop3): found reiserfs format "3.6" with non-standard journal
REISERFS (device loop3): using ordered data mode
reiserfs: using flush barriers
REISERFS (device loop3): journal params: device loop3, size 512, journal first block 18, max trans len 256, max batch 225, max commit age 30, max trans age 30
REISERFS (device loop3): checking transaction log (loop3)
REISERFS (device loop3): Using r5 hash to sort names
BTRFS info (device loop2): enabling inode map caching
REISERFS (device loop3): Created .reiserfs_priv - reserved for xattr storage.
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): enabling inode map caching
BTRFS warning (device loop2): excessive commit interval 622039222
BTRFS info (device loop2): force zlib compression, level 3
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS warning (device <unknown>): duplicate device /dev/loop3 devid 1 generation 8 scanned by syz-executor.3 (15937)

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/12/25 00:59 linux-4.19.y 3f8a27f9e27b 9da18ae8 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 possible deadlock in fat_write_inode
* Struck through repros no longer work on HEAD.