bisecting fixing commit since 14b58326976de6ef3998eefec1dd7f8b38b97a75 building syzkaller on 5ce135324a84d6de67c913f1b725e3fc5ec39535 testing commit 14b58326976de6ef3998eefec1dd7f8b38b97a75 with gcc (GCC) 8.1.0 kernel signature: 73164b1df0e8a68f05dcd19adf68e74107e3d52c3b5f4711cde342cc775557cf run #0: crashed: possible deadlock in __generic_file_fsync run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK testing current HEAD bae31eef2a167ef160ab2703b6a2f5bbecd98d92 testing commit bae31eef2a167ef160ab2703b6a2f5bbecd98d92 with gcc (GCC) 8.1.0 kernel signature: b5e5cd03c8d88dc0973b4713d6e2d465920536ec74d6a442474f7b5adbe57942 run #0: crashed: possible deadlock in __generic_file_fsync run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK revisions tested: 2, total time: 37m30.20475026s (build: 17m25.543291495s, test: 19m11.322079041s) the crash still happens on HEAD commit msg: Linux 4.14.200 crash: possible deadlock in __generic_file_fsync Bluetooth: hci4 command 0x0419 tx timeout Bluetooth: hci5 command 0x0419 tx timeout Bluetooth: hci0 command 0x0419 tx timeout Bluetooth: hci1 command 0x0419 tx timeout ====================================================== WARNING: possible circular locking dependency detected 4.14.200-syzkaller #0 Not tainted ------------------------------------------------------ kworker/0:0/3 is trying to acquire lock: (&sb->s_type->i_mutex_key#9){+.+.}, at: [] inode_lock include/linux/fs.h:719 [inline] (&sb->s_type->i_mutex_key#9){+.+.}, at: [] __generic_file_fsync+0x8a/0x1a0 fs/libfs.c:987 but task is already holding lock: ((&dio->complete_work)){+.+.}, at: [] process_one_work+0x6b4/0x1620 kernel/workqueue.c:2091 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #2 ((&dio->complete_work)){+.+.}: lock_acquire+0x17e/0x3e0 kernel/locking/lockdep.c:3998 process_one_work+0x702/0x1620 kernel/workqueue.c:2092 worker_thread+0xcc/0xed0 kernel/workqueue.c:2250 kthread+0x338/0x400 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 -> #1 ("dio/%s"sb->s_id){+.+.}: lock_acquire+0x17e/0x3e0 kernel/locking/lockdep.c:3998 flush_workqueue+0xea/0x1340 kernel/workqueue.c:2624 drain_workqueue+0x148/0x3a0 kernel/workqueue.c:2789 destroy_workqueue+0x68/0x600 kernel/workqueue.c:4102 __alloc_workqueue_key+0xa07/0xc80 kernel/workqueue.c:4079 sb_init_dio_done_wq+0x30/0x60 fs/direct-io.c:624 do_blockdev_direct_IO fs/direct-io.c:1286 [inline] __blockdev_direct_IO+0x2591/0xc358 fs/direct-io.c:1422 ext4_direct_IO_write fs/ext4/inode.c:3716 [inline] ext4_direct_IO+0x790/0x1ae0 fs/ext4/inode.c:3857 generic_file_direct_write+0x1ee/0x410 mm/filemap.c:2958 __generic_file_write_iter+0x270/0x580 mm/filemap.c:3137 ext4_file_write_iter+0x279/0xe30 fs/ext4/file.c:270 call_write_iter include/linux/fs.h:1778 [inline] aio_write+0x291/0x4e0 fs/aio.c:1553 io_submit_one fs/aio.c:1647 [inline] do_io_submit+0xc6d/0x1170 fs/aio.c:1709 SYSC_io_submit fs/aio.c:1734 [inline] SyS_io_submit+0xb/0x10 fs/aio.c:1731 do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292 entry_SYSCALL_64_after_hwframe+0x46/0xbb -> #0 (&sb->s_type->i_mutex_key#9){+.+.}: check_prev_add kernel/locking/lockdep.c:1905 [inline] check_prevs_add kernel/locking/lockdep.c:2022 [inline] validate_chain kernel/locking/lockdep.c:2464 [inline] __lock_acquire+0x32e8/0x42d0 kernel/locking/lockdep.c:3491 lock_acquire+0x17e/0x3e0 kernel/locking/lockdep.c:3998 down_write+0x38/0x90 kernel/locking/rwsem.c:54 inode_lock include/linux/fs.h:719 [inline] __generic_file_fsync+0x8a/0x1a0 fs/libfs.c:987 ext4_sync_file+0x938/0x1050 fs/ext4/fsync.c:118 vfs_fsync_range+0xe8/0x290 fs/sync.c:196 generic_write_sync include/linux/fs.h:2682 [inline] dio_complete+0x618/0x9d0 fs/direct-io.c:330 dio_aio_complete_work+0x17/0x20 fs/direct-io.c:342 process_one_work+0x74f/0x1620 kernel/workqueue.c:2116 worker_thread+0xcc/0xed0 kernel/workqueue.c:2250 kthread+0x338/0x400 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 other info that might help us debug this: Chain exists of: &sb->s_type->i_mutex_key#9 --> "dio/%s"sb->s_id --> (&dio->complete_work) Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((&dio->complete_work)); lock("dio/%s"sb->s_id); lock((&dio->complete_work)); lock(&sb->s_type->i_mutex_key#9); *** DEADLOCK *** 2 locks held by kworker/0:0/3: #0: ("dio/%s"sb->s_id){+.+.}, at: [] process_one_work+0x681/0x1620 kernel/workqueue.c:2087 #1: ((&dio->complete_work)){+.+.}, at: [] process_one_work+0x6b4/0x1620 kernel/workqueue.c:2091 stack backtrace: CPU: 0 PID: 3 Comm: kworker/0:0 Not tainted 4.14.200-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: dio/sda1 dio_aio_complete_work Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x14b/0x1f1 lib/dump_stack.c:58 print_circular_bug.isra.17.cold.40+0x2e3/0x41e kernel/locking/lockdep.c:1258 check_prev_add kernel/locking/lockdep.c:1905 [inline] check_prevs_add kernel/locking/lockdep.c:2022 [inline] validate_chain kernel/locking/lockdep.c:2464 [inline] __lock_acquire+0x32e8/0x42d0 kernel/locking/lockdep.c:3491 lock_acquire+0x17e/0x3e0 kernel/locking/lockdep.c:3998 down_write+0x38/0x90 kernel/locking/rwsem.c:54 inode_lock include/linux/fs.h:719 [inline] __generic_file_fsync+0x8a/0x1a0 fs/libfs.c:987 ext4_sync_file+0x938/0x1050 fs/ext4/fsync.c:118 vfs_fsync_range+0xe8/0x290 fs/sync.c:196 generic_write_sync include/linux/fs.h:2682 [inline] dio_complete+0x618/0x9d0 fs/direct-io.c:330 dio_aio_complete_work+0x17/0x20 fs/direct-io.c:342 process_one_work+0x74f/0x1620 kernel/workqueue.c:2116 worker_thread+0xcc/0xed0 kernel/workqueue.c:2250 kthread+0x338/0x400 kernel/kthread.c:232 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404