====================================================== WARNING: possible circular locking dependency detected 5.0.0+ #6 Not tainted ------------------------------------------------------ kworker/1:3/7902 is trying to acquire lock: 00000000ff25f41d (&sb->s_type->i_mutex_key#9){++++}, at: inode_lock include/linux/fs.h:763 [inline] 00000000ff25f41d (&sb->s_type->i_mutex_key#9){++++}, at: __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 but task is already holding lock: 0000000066ebf615 ((work_completion)(&dio->complete_work)){+.+.}, at: process_one_work+0x8b4/0x1790 kernel/workqueue.c:2148 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 ((work_completion)(&dio->complete_work)){+.+.}: process_one_work+0x90f/0x1790 kernel/workqueue.c:2149 worker_thread+0x98/0xe40 kernel/workqueue.c:2319 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 -> #1 ((wq_completion)"dio/%s"sb->s_id){+.+.}: flush_workqueue+0x126/0x14c0 kernel/workqueue.c:2678 drain_workqueue+0x1b4/0x470 kernel/workqueue.c:2843 destroy_workqueue+0x21/0x6f0 kernel/workqueue.c:4178 sb_init_dio_done_wq+0x77/0x90 fs/direct-io.c:634 dio_set_defer_completion fs/direct-io.c:646 [inline] get_more_blocks fs/direct-io.c:724 [inline] do_direct_IO fs/direct-io.c:1002 [inline] do_blockdev_direct_IO+0x27b7/0x8db0 fs/direct-io.c:1334 __blockdev_direct_IO+0xa1/0xca fs/direct-io.c:1420 ext4_direct_IO_write fs/ext4/inode.c:3775 [inline] ext4_direct_IO+0xe6c/0x1c10 fs/ext4/inode.c:3902 generic_file_direct_write+0x214/0x4b0 mm/filemap.c:3111 __generic_file_write_iter+0x2ee/0x630 mm/filemap.c:3290 ext4_file_write_iter+0x33f/0x1160 fs/ext4/file.c:266 call_write_iter include/linux/fs.h:1869 [inline] aio_write+0x34b/0x540 fs/aio.c:1580 __io_submit_one fs/aio.c:1853 [inline] io_submit_one+0x10ce/0x1cd0 fs/aio.c:1901 __do_sys_io_submit fs/aio.c:1946 [inline] __se_sys_io_submit fs/aio.c:1916 [inline] __x64_sys_io_submit+0x1bd/0x580 fs/aio.c:1916 do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #0 (&sb->s_type->i_mutex_key#9){++++}: lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3841 down_write+0x38/0x90 kernel/locking/rwsem.c:70 inode_lock include/linux/fs.h:763 [inline] __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 ext4_sync_file+0x867/0x14d0 fs/ext4/fsync.c:120 vfs_fsync_range+0x144/0x230 fs/sync.c:197 generic_write_sync include/linux/fs.h:2793 [inline] dio_complete+0x498/0x9f0 fs/direct-io.c:329 dio_aio_complete_work+0x20/0x30 fs/direct-io.c:341 process_one_work+0x98e/0x1790 kernel/workqueue.c:2173 worker_thread+0x98/0xe40 kernel/workqueue.c:2319 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 other info that might help us debug this: Chain exists of: &sb->s_type->i_mutex_key#9 --> (wq_completion)"dio/%s"sb->s_id --> (work_completion)(&dio->complete_work) Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((work_completion)(&dio->complete_work)); lock((wq_completion)"dio/%s"sb->s_id); lock((work_completion)(&dio->complete_work)); lock(&sb->s_type->i_mutex_key#9); *** DEADLOCK *** 2 locks held by kworker/1:3/7902: #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 0000000029935406 ((wq_completion)"dio/%s"sb->s_id){+.+.}, at: process_one_work+0x87e/0x1790 kernel/workqueue.c:2144 #1: 0000000066ebf615 ((work_completion)(&dio->complete_work)){+.+.}, at: process_one_work+0x8b4/0x1790 kernel/workqueue.c:2148 stack backtrace: CPU: 1 PID: 7902 Comm: kworker/1:3 Not tainted 5.0.0+ #6 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: dio/sda1 dio_aio_complete_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1224 check_prev_add kernel/locking/lockdep.c:1866 [inline] check_prevs_add kernel/locking/lockdep.c:1979 [inline] validate_chain kernel/locking/lockdep.c:2350 [inline] __lock_acquire+0x2f00/0x4700 kernel/locking/lockdep.c:3338 lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:3841 down_write+0x38/0x90 kernel/locking/rwsem.c:70 inode_lock include/linux/fs.h:763 [inline] __generic_file_fsync+0xb5/0x200 fs/libfs.c:981 ext4_sync_file+0x867/0x14d0 fs/ext4/fsync.c:120 vfs_fsync_range+0x144/0x230 fs/sync.c:197 generic_write_sync include/linux/fs.h:2793 [inline] dio_complete+0x498/0x9f0 fs/direct-io.c:329 dio_aio_complete_work+0x20/0x30 fs/direct-io.c:341 process_one_work+0x98e/0x1790 kernel/workqueue.c:2173 worker_thread+0x98/0xe40 kernel/workqueue.c:2319 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'rx-0' (0000000080c58f32): kobject_cleanup, parent 00000000f5ea24d4 kobject: 'rx-0' (0000000080c58f32): auto cleanup 'remove' event kobject: 'rx-0' (0000000080c58f32): kobject_uevent_env kobject: 'rx-0' (0000000080c58f32): fill_kobj_path: path = '/devices/virtual/net/eql/queues/rx-0' kobject: 'rx-0' (0000000080c58f32): auto cleanup kobject_del kobject: 'rx-0' (0000000080c58f32): calling ktype release kobject: 'rx-0': free name kobject: 'tx-0' (000000008399d1cd): kobject_cleanup, parent 00000000f5ea24d4 kobject: 'tx-0' (000000008399d1cd): auto cleanup 'remove' event kobject: 'tx-0' (000000008399d1cd): kobject_uevent_env kobject: 'tx-0' (000000008399d1cd): fill_kobj_path: path = '/devices/virtual/net/eql/queues/tx-0' kobject: 'tx-0' (000000008399d1cd): auto cleanup kobject_del kobject: 'tx-0' (000000008399d1cd): calling ktype release kobject: 'tx-0': free name kobject: 'queues' (00000000f5ea24d4): kobject_cleanup, parent (null) kobject: 'queues' (00000000f5ea24d4): calling ktype release kobject: 'queues' (00000000f5ea24d4): kset_release kobject: 'queues': free name kobject: 'eql' (000000001479d3b8): kobject_uevent_env kobject: 'eql' (000000001479d3b8): fill_kobj_path: path = '/devices/virtual/net/eql' kobject: 'eql' (000000001479d3b8): kobject_cleanup, parent (null) kobject: 'eql' (000000001479d3b8): calling ktype release kobject: 'eql': free name kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'eql' (00000000df365b47): kobject_add_internal: parent: 'net', set: 'devices' kobject: 'eql' (00000000df365b47): kobject_uevent_env kobject: 'eql' (00000000df365b47): fill_kobj_path: path = '/devices/virtual/net/eql' kobject: 'queues' (00000000b3833d1b): kobject_add_internal: parent: 'eql', set: '' kobject: 'queues' (00000000b3833d1b): kobject_uevent_env kobject: 'queues' (00000000b3833d1b): kobject_uevent_env: filter function caused the event to drop! kobject: 'rx-0' (000000002f44ff1b): kobject_add_internal: parent: 'queues', set: 'queues' kobject: 'rx-0' (000000002f44ff1b): kobject_uevent_env kobject: 'rx-0' (000000002f44ff1b): fill_kobj_path: path = '/devices/virtual/net/eql/queues/rx-0' kobject: 'tx-0' (00000000b9da4b7a): kobject_add_internal: parent: 'queues', set: 'queues' kobject: 'tx-0' (00000000b9da4b7a): kobject_uevent_env kobject: 'tx-0' (00000000b9da4b7a): fill_kobj_path: path = '/devices/virtual/net/eql/queues/tx-0' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'rx-0' (000000002f44ff1b): kobject_cleanup, parent 00000000b3833d1b kobject: 'rx-0' (000000002f44ff1b): auto cleanup 'remove' event kobject: 'rx-0' (000000002f44ff1b): kobject_uevent_env kobject: 'rx-0' (000000002f44ff1b): fill_kobj_path: path = '/devices/virtual/net/eql/queues/rx-0' kobject: 'rx-0' (000000002f44ff1b): auto cleanup kobject_del kobject: 'rx-0' (000000002f44ff1b): calling ktype release kobject: 'rx-0': free name kobject: 'tx-0' (00000000b9da4b7a): kobject_cleanup, parent 00000000b3833d1b kobject: 'tx-0' (00000000b9da4b7a): auto cleanup 'remove' event kobject: 'tx-0' (00000000b9da4b7a): kobject_uevent_env kobject: 'tx-0' (00000000b9da4b7a): fill_kobj_path: path = '/devices/virtual/net/eql/queues/tx-0' kobject: 'tx-0' (00000000b9da4b7a): auto cleanup kobject_del kobject: 'tx-0' (00000000b9da4b7a): calling ktype release kobject: 'tx-0': free name kobject: 'queues' (00000000b3833d1b): kobject_cleanup, parent (null) kobject: 'queues' (00000000b3833d1b): calling ktype release kobject: 'queues' (00000000b3833d1b): kset_release kobject: 'queues': free name kobject: 'eql' (00000000df365b47): kobject_uevent_env kobject: 'eql' (00000000df365b47): fill_kobj_path: path = '/devices/virtual/net/eql' kobject: 'eql' (00000000df365b47): kobject_cleanup, parent (null) kobject: 'eql' (00000000df365b47): calling ktype release kobject: 'eql': free name kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'eql' (000000004356ed71): kobject_add_internal: parent: 'net', set: 'devices' kobject: 'eql' (000000004356ed71): kobject_uevent_env kobject: 'eql' (000000004356ed71): fill_kobj_path: path = '/devices/virtual/net/eql' kobject: 'queues' (000000005c665abc): kobject_add_internal: parent: 'eql', set: '' kobject: 'queues' (000000005c665abc): kobject_uevent_env kobject: 'queues' (000000005c665abc): kobject_uevent_env: filter function caused the event to drop! kobject: 'rx-0' (00000000aa27f0bc): kobject_add_internal: parent: 'queues', set: 'queues' kobject: 'rx-0' (00000000aa27f0bc): kobject_uevent_env kobject: 'rx-0' (00000000aa27f0bc): fill_kobj_path: path = '/devices/virtual/net/eql/queues/rx-0' kobject: 'tx-0' (00000000ccf7fabf): kobject_add_internal: parent: 'queues', set: 'queues' kobject: 'tx-0' (00000000ccf7fabf): kobject_uevent_env kobject: 'tx-0' (00000000ccf7fabf): fill_kobj_path: path = '/devices/virtual/net/eql/queues/tx-0' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'rx-0' (00000000aa27f0bc): kobject_cleanup, parent 000000005c665abc kobject: 'rx-0' (00000000aa27f0bc): auto cleanup 'remove' event kobject: 'rx-0' (00000000aa27f0bc): kobject_uevent_env kobject: 'rx-0' (00000000aa27f0bc): fill_kobj_path: path = '/devices/virtual/net/eql/queues/rx-0' kobject: 'rx-0' (00000000aa27f0bc): auto cleanup kobject_del kobject: 'rx-0' (00000000aa27f0bc): calling ktype release kobject: 'rx-0': free name kobject: 'tx-0' (00000000ccf7fabf): kobject_cleanup, parent 000000005c665abc kobject: 'tx-0' (00000000ccf7fabf): auto cleanup 'remove' event kobject: 'tx-0' (00000000ccf7fabf): kobject_uevent_env kobject: 'tx-0' (00000000ccf7fabf): fill_kobj_path: path = '/devices/virtual/net/eql/queues/tx-0' kobject: 'tx-0' (00000000ccf7fabf): auto cleanup kobject_del kobject: 'tx-0' (00000000ccf7fabf): calling ktype release kobject: 'tx-0': free name kobject: 'queues' (000000005c665abc): kobject_cleanup, parent (null) kobject: 'queues' (000000005c665abc): calling ktype release kobject: 'queues' (000000005c665abc): kset_release kobject: 'queues': free name kobject: 'eql' (000000004356ed71): kobject_uevent_env kobject: 'eql' (000000004356ed71): fill_kobj_path: path = '/devices/virtual/net/eql' kobject: 'eql' (000000004356ed71): kobject_cleanup, parent (null) kobject: 'eql' (000000004356ed71): calling ktype release kobject: 'eql': free name kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' netlink: 767 bytes leftover after parsing attributes in process `syz-executor.5'. kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' net_ratelimit: 10 callbacks suppressed netlink: zone id is out of range netlink: zone id is out of range netlink: del zone limit has 8 unknown bytes kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' netlink: 767 bytes leftover after parsing attributes in process `syz-executor.5'. netlink: zone id is out of range netlink: zone id is out of range netlink: del zone limit has 8 unknown bytes kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' netlink: 767 bytes leftover after parsing attributes in process `syz-executor.5'. IPVS: Unknown mcast interface: yam0 netlink: zone id is out of range netlink: zone id is out of range netlink: del zone limit has 8 unknown bytes kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' netlink: 767 bytes leftover after parsing attributes in process `syz-executor.5'. IPVS: Unknown mcast interface: yam0 netlink: zone id is out of range IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): kobject_uevent_env IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): kobject_uevent_env IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop2' (00000000da735771): kobject_uevent_env kobject: 'loop2' (00000000da735771): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5' IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): kobject_uevent_env IPVS: Unknown mcast interface: yam0 kobject: 'loop5' (00000000764eeb1b): fill_kobj_path: path = '/devices/virtual/block/loop5'