====================================================== WARNING: possible circular locking dependency detected 4.14.149+ #0 Not tainted ------------------------------------------------------ kworker/1:13/15978 is trying to acquire lock: (&sb->s_type->i_mutex_key#9){++++}, at: [<00000000f15e88bb>] inode_lock include/linux/fs.h:724 [inline] (&sb->s_type->i_mutex_key#9){++++}, at: [<00000000f15e88bb>] __generic_file_fsync+0x9e/0x190 fs/libfs.c:981 but task is already holding lock: ((&dio->complete_work)){+.+.}, at: [<0000000039aff442>] process_one_work+0x735/0x1580 kernel/workqueue.c:2109 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 ((&dio->complete_work)){+.+.}: process_one_work+0x789/0x1580 kernel/workqueue.c:2110 worker_thread+0xdd/0xdf0 kernel/workqueue.c:2271 kthread+0x31f/0x430 kernel/kthread.c:232 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:404 -> #1 ("dio/%s"sb->s_id){+.+.}: flush_workqueue+0x118/0x13b0 kernel/workqueue.c:2639 drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2804 destroy_workqueue+0x22/0x5e0 kernel/workqueue.c:4111 __alloc_workqueue_key+0xb48/0xd80 kernel/workqueue.c:4094 sb_init_dio_done_wq+0x34/0x90 fs/direct-io.c:624 dio_set_defer_completion fs/direct-io.c:647 [inline] get_more_blocks fs/direct-io.c:725 [inline] do_direct_IO fs/direct-io.c:1003 [inline] do_blockdev_direct_IO fs/direct-io.c:1336 [inline] __blockdev_direct_IO+0x31cc/0xe24e fs/direct-io.c:1422 ext4_direct_IO_write fs/ext4/inode.c:3730 [inline] ext4_direct_IO+0xa4f/0x2820 fs/ext4/inode.c:3885 generic_file_direct_write+0x1e4/0x430 mm/filemap.c:3035 __generic_file_write_iter+0x209/0x550 mm/filemap.c:3214 ext4_file_write_iter+0x58f/0xdb0 fs/ext4/file.c:268 call_write_iter include/linux/fs.h:1798 [inline] aio_write+0x2ea/0x530 fs/aio.c:1553 io_submit_one fs/aio.c:1641 [inline] do_io_submit+0x8e7/0x13e0 fs/aio.c:1709 do_syscall_64+0x19b/0x520 arch/x86/entry/common.c:292 entry_SYSCALL_64_after_hwframe+0x42/0xb7 -> #0 (&sb->s_type->i_mutex_key#9){++++}: lock_acquire+0x12b/0x360 kernel/locking/lockdep.c:3994 down_write+0x34/0x90 kernel/locking/rwsem.c:54 inode_lock include/linux/fs.h:724 [inline] __generic_file_fsync+0x9e/0x190 fs/libfs.c:981 ext4_sync_file+0x3ac/0x1250 fs/ext4/fsync.c:120 vfs_fsync_range+0x106/0x260 fs/sync.c:196 generic_write_sync include/linux/fs.h:2713 [inline] dio_complete+0x37e/0x860 fs/direct-io.c:330 process_one_work+0x7f1/0x1580 kernel/workqueue.c:2134 worker_thread+0xdd/0xdf0 kernel/workqueue.c:2271 kthread+0x31f/0x430 kernel/kthread.c:232 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:404 other info that might help us debug this: Chain exists of: &sb->s_type->i_mutex_key#9 --> "dio/%s"sb->s_id --> (&dio->complete_work) Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((&dio->complete_work)); lock("dio/%s"sb->s_id); lock((&dio->complete_work)); lock(&sb->s_type->i_mutex_key#9); *** DEADLOCK *** 2 locks held by kworker/1:13/15978: #0: ("dio/%s"sb->s_id){+.+.}, at: [<000000008f63b38d>] process_one_work+0x6ff/0x1580 kernel/workqueue.c:2105 #1: ((&dio->complete_work)){+.+.}, at: [<0000000039aff442>] process_one_work+0x735/0x1580 kernel/workqueue.c:2109 stack backtrace: CPU: 1 PID: 15978 Comm: kworker/1:13 Not tainted 4.14.149+ #0 Workqueue: dio/sda1 dio_aio_complete_work Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0xca/0x134 lib/dump_stack.c:53 print_circular_bug.isra.0.cold+0x2dc/0x425 kernel/locking/lockdep.c:1258 check_prev_add kernel/locking/lockdep.c:1901 [inline] check_prevs_add kernel/locking/lockdep.c:2018 [inline] validate_chain kernel/locking/lockdep.c:2460 [inline] __lock_acquire+0x2f5f/0x4320 kernel/locking/lockdep.c:3487 lock_acquire+0x12b/0x360 kernel/locking/lockdep.c:3994 down_write+0x34/0x90 kernel/locking/rwsem.c:54 inode_lock include/linux/fs.h:724 [inline] __generic_file_fsync+0x9e/0x190 fs/libfs.c:981 ext4_sync_file+0x3ac/0x1250 fs/ext4/fsync.c:120 vfs_fsync_range+0x106/0x260 fs/sync.c:196 generic_write_sync include/linux/fs.h:2713 [inline] dio_complete+0x37e/0x860 fs/direct-io.c:330 process_one_work+0x7f1/0x1580 kernel/workqueue.c:2134 worker_thread+0xdd/0xdf0 kernel/workqueue.c:2271 kthread+0x31f/0x430 kernel/kthread.c:232 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:404 EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs: 7 callbacks suppressed EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: EXT4-fs (sda1): re-mounted. Opts: