syzbot


possible deadlock in flush_workqueue

Status: auto-closed as invalid on 2020/03/10 11:29
Reported-by: syzbot+8d9981eb8598591ef6dc@syzkaller.appspotmail.com
First crash: 1599d, last: 1599d
Similar bugs (7)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in flush_workqueue 1 351d 351d 0/3 auto-obsoleted due to no activity on 2023/08/09 18:38
linux-5.15 possible deadlock in flush_workqueue (2) 3 3d13h 14d 0/3 upstream: reported on 2024/03/13 19:04
upstream possible deadlock in flush_workqueue (2) C done done 256 1552d 1986d 15/26 fixed on 2020/01/31 18:49
linux-4.14 possible deadlock in flush_workqueue C done 15 1592d 1685d 1/1 fixed on 2019/12/18 17:48
linux-4.14 possible deadlock in flush_workqueue (2) 3 1555d 1560d 0/1 auto-closed as invalid on 2020/04/22 20:54
upstream possible deadlock in flush_workqueue net C 73762 2002d 2044d 11/26 fixed on 2018/10/11 14:33
linux-4.19 possible deadlock in flush_workqueue 3 1613d 1624d 0/1 auto-closed as invalid on 2020/02/25 05:02

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
4.14.153+ #0 Not tainted
------------------------------------------------------
syz-executor.4/6517 is trying to acquire lock:
 ("dio/%s"sb->s_id){+.+.}, at: [<0000000043d2888d>] flush_workqueue+0xe9/0x13b0 kernel/workqueue.c:2636

but task is already holding lock:
 (&sb->s_type->i_mutex_key#9){++++}, at: [<0000000083e0f094>] inode_trylock include/linux/fs.h:744 [inline]
 (&sb->s_type->i_mutex_key#9){++++}, at: [<0000000083e0f094>] ext4_file_write_iter+0x1b0/0xdb0 fs/ext4/file.c:234

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&sb->s_type->i_mutex_key#9){++++}:
       down_write+0x34/0x90 kernel/locking/rwsem.c:54
       inode_lock include/linux/fs.h:724 [inline]
       __generic_file_fsync+0x9e/0x190 fs/libfs.c:985
       ext4_sync_file+0x3ac/0x1250 fs/ext4/fsync.c:120
       vfs_fsync_range+0x106/0x260 fs/sync.c:196
       generic_write_sync include/linux/fs.h:2713 [inline]
       dio_complete+0x37e/0x860 fs/direct-io.c:330
       process_one_work+0x7f1/0x1580 kernel/workqueue.c:2134
       worker_thread+0xdd/0xdf0 kernel/workqueue.c:2271
       kthread+0x31f/0x430 kernel/kthread.c:232
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:404

-> #1 ((&dio->complete_work)){+.+.}:
       process_one_work+0x789/0x1580 kernel/workqueue.c:2110
       worker_thread+0xdd/0xdf0 kernel/workqueue.c:2271
       kthread+0x31f/0x430 kernel/kthread.c:232
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:404

-> #0 ("dio/%s"sb->s_id){+.+.}:
       lock_acquire+0x12b/0x360 kernel/locking/lockdep.c:3994
       flush_workqueue+0x118/0x13b0 kernel/workqueue.c:2639
       drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2804
       destroy_workqueue+0x22/0x5e0 kernel/workqueue.c:4111
       sb_init_dio_done_wq+0x72/0x90 fs/direct-io.c:635
       do_blockdev_direct_IO fs/direct-io.c:1286 [inline]
       __blockdev_direct_IO+0x2f23/0xe24e fs/direct-io.c:1422
       ext4_direct_IO_write fs/ext4/inode.c:3730 [inline]
       ext4_direct_IO+0xa4f/0x2820 fs/ext4/inode.c:3885
       generic_file_direct_write+0x1e4/0x430 mm/filemap.c:3035
       __generic_file_write_iter+0x209/0x550 mm/filemap.c:3214
       ext4_file_write_iter+0x58f/0xdb0 fs/ext4/file.c:268
       call_write_iter include/linux/fs.h:1798 [inline]
       aio_write+0x2ea/0x530 fs/aio.c:1553
       io_submit_one fs/aio.c:1641 [inline]
       do_io_submit+0x8e7/0x13e0 fs/aio.c:1709
       do_syscall_64+0x19b/0x520 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
  "dio/%s"sb->s_id --> (&dio->complete_work) --> &sb->s_type->i_mutex_key#9

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&sb->s_type->i_mutex_key#9);
                               lock((&dio->complete_work));
                               lock(&sb->s_type->i_mutex_key#9);
  lock("dio/%s"sb->s_id);

 *** DEADLOCK ***

2 locks held by syz-executor.4/6517:
 #0:  (sb_writers#4){.+.+}, at: [<000000009969df6a>] file_start_write include/linux/fs.h:2746 [inline]
 #0:  (sb_writers#4){.+.+}, at: [<000000009969df6a>] aio_write+0x40c/0x530 fs/aio.c:1552
 #1:  (&sb->s_type->i_mutex_key#9){++++}, at: [<0000000083e0f094>] inode_trylock include/linux/fs.h:744 [inline]
 #1:  (&sb->s_type->i_mutex_key#9){++++}, at: [<0000000083e0f094>] ext4_file_write_iter+0x1b0/0xdb0 fs/ext4/file.c:234

stack backtrace:
CPU: 0 PID: 6517 Comm: syz-executor.4 Not tainted 4.14.153+ #0
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0xca/0x134 lib/dump_stack.c:53
 print_circular_bug.isra.0.cold+0x2dc/0x425 kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2f5f/0x4320 kernel/locking/lockdep.c:3487
 lock_acquire+0x12b/0x360 kernel/locking/lockdep.c:3994
 flush_workqueue+0x118/0x13b0 kernel/workqueue.c:2639
 drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2804
 destroy_workqueue+0x22/0x5e0 kernel/workqueue.c:4111
 sb_init_dio_done_wq+0x72/0x90 fs/direct-io.c:635
 do_blockdev_direct_IO fs/direct-io.c:1286 [inline]
 __blockdev_direct_IO+0x2f23/0xe24e fs/direct-io.c:1422
 ext4_direct_IO_write fs/ext4/inode.c:3730 [inline]
 ext4_direct_IO+0xa4f/0x2820 fs/ext4/inode.c:3885
 generic_file_direct_write+0x1e4/0x430 mm/filemap.c:3035
 __generic_file_write_iter+0x209/0x550 mm/filemap.c:3214
 ext4_file_write_iter+0x58f/0xdb0 fs/ext4/file.c:268
 call_write_iter include/linux/fs.h:1798 [inline]
 aio_write+0x2ea/0x530 fs/aio.c:1553
 io_submit_one fs/aio.c:1641 [inline]
 do_io_submit+0x8e7/0x13e0 fs/aio.c:1709
 do_syscall_64+0x19b/0x520 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x45a219
RSP: 002b:00007f74567e7c78 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 000000000045a219
RDX: 0000000020000540 RSI: 0000000000000001 RDI: 00007f74567bf000
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f74567e86d4
R13: 00000000004c18fc R14: 00000000004d5500 R15: 00000000ffffffff

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/11/11 11:28 android-4.14 10e570bfc15a dc438b91 .config console log report ci-android-414-kasan-gce-root
* Struck through repros no longer work on HEAD.