====================================================== WARNING: possible circular locking dependency detected 4.19.50 #22 Not tainted ------------------------------------------------------ kworker/0:2/3215 is trying to acquire lock: 00000000f5398637 (&sb->s_type->i_mutex_key#9){++++}, at: inode_lock include/linux/fs.h:747 [inline] 00000000f5398637 (&sb->s_type->i_mutex_key#9){++++}, at: __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 but task is already holding lock: 000000008b1b6f7c ((work_completion)(&dio->complete_work)){+.+.}, at: process_one_work+0x8b4/0x1750 kernel/workqueue.c:2128 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 ((work_completion)(&dio->complete_work)){+.+.}: process_one_work+0x90f/0x1750 kernel/workqueue.c:2129 worker_thread+0x98/0xe40 kernel/workqueue.c:2296 kthread+0x354/0x420 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415 -> #1 ((wq_completion)"dio/%s"sb->s_id){+.+.}: flush_workqueue+0x126/0x14c0 kernel/workqueue.c:2655 drain_workqueue+0x1b4/0x470 kernel/workqueue.c:2820 destroy_workqueue+0x21/0x6f0 kernel/workqueue.c:4158 __alloc_workqueue_key+0xc69/0xee0 kernel/workqueue.c:4141 sb_init_dio_done_wq+0x39/0x90 fs/direct-io.c:623 dio_set_defer_completion fs/direct-io.c:646 [inline] get_more_blocks fs/direct-io.c:724 [inline] do_direct_IO fs/direct-io.c:1002 [inline] do_blockdev_direct_IO+0x26c1/0x8cb0 fs/direct-io.c:1332 __blockdev_direct_IO+0xa1/0xca fs/direct-io.c:1418 ext4_direct_IO_write fs/ext4/inode.c:3744 [inline] ext4_direct_IO+0x990/0x19e0 fs/ext4/inode.c:3871 generic_file_direct_write+0x20f/0x4b0 mm/filemap.c:3042 __generic_file_write_iter+0x2ee/0x630 mm/filemap.c:3221 ext4_file_write_iter+0xa24/0x1060 fs/ext4/file.c:266 call_write_iter include/linux/fs.h:1820 [inline] aio_write+0x346/0x530 fs/aio.c:1573 __io_submit_one fs/aio.c:1834 [inline] io_submit_one+0x104b/0x2eb0 fs/aio.c:1885 __do_sys_io_submit fs/aio.c:1929 [inline] __se_sys_io_submit fs/aio.c:1900 [inline] __x64_sys_io_submit+0x1aa/0x520 fs/aio.c:1900 do_syscall_64+0xfd/0x620 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #0 (&sb->s_type->i_mutex_key#9){++++}: lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3900 down_write+0x38/0x90 kernel/locking/rwsem.c:70 inode_lock include/linux/fs.h:747 [inline] __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 ext4_sync_file+0x826/0x1450 fs/ext4/fsync.c:120 vfs_fsync_range+0x141/0x230 fs/sync.c:197 generic_write_sync include/linux/fs.h:2744 [inline] dio_complete+0x49a/0x9f0 fs/direct-io.c:329 dio_aio_complete_work+0x20/0x30 fs/direct-io.c:341 process_one_work+0x989/0x1750 kernel/workqueue.c:2153 worker_thread+0x98/0xe40 kernel/workqueue.c:2296 kthread+0x354/0x420 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415 other info that might help us debug this: Chain exists of: &sb->s_type->i_mutex_key#9 --> (wq_completion)"dio/%s"sb->s_id --> (work_completion)(&dio->complete_work) Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((work_completion)(&dio->complete_work)); lock((wq_completion)"dio/%s"sb->s_id); lock((work_completion)(&dio->complete_work)); lock(&sb->s_type->i_mutex_key#9); *** DEADLOCK *** 2 locks held by kworker/0:2/3215: #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 00000000d805dac7 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: process_one_work+0x87e/0x1750 kernel/workqueue.c:2124 #1: 000000008b1b6f7c ((work_completion)(&dio->complete_work)){+.+.}, at: process_one_work+0x8b4/0x1750 kernel/workqueue.c:2128 stack backtrace: CPU: 0 PID: 3215 Comm: kworker/0:2 Not tainted 4.19.50 #22 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: dio/sda1 dio_aio_complete_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1221 check_prev_add kernel/locking/lockdep.c:1861 [inline] check_prevs_add kernel/locking/lockdep.c:1974 [inline] validate_chain kernel/locking/lockdep.c:2415 [inline] __lock_acquire+0x2e6d/0x48f0 kernel/locking/lockdep.c:3411 lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3900 down_write+0x38/0x90 kernel/locking/rwsem.c:70 inode_lock include/linux/fs.h:747 [inline] __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 ext4_sync_file+0x826/0x1450 fs/ext4/fsync.c:120 vfs_fsync_range+0x141/0x230 fs/sync.c:197 generic_write_sync include/linux/fs.h:2744 [inline] dio_complete+0x49a/0x9f0 fs/direct-io.c:329 dio_aio_complete_work+0x20/0x30 fs/direct-io.c:341 process_one_work+0x989/0x1750 kernel/workqueue.c:2153 worker_thread+0x98/0xe40 kernel/workqueue.c:2296 kthread+0x354/0x420 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415 ntfs: (device loop5): parse_options(): NLS character set m%cromanian not found. kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' ntfs: (device loop5): parse_options(): NLS character set m%cromanian not found. kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (00000000701466d6): kobject_uevent_env kobject: 'loop5' (00000000701466d6): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop0' (00000000b01ff277): kobject_uevent_env kobject: 'loop0' (00000000b01ff277): fill_kobj_path: path = '/devices/virtual/block/loop0'