syzbot


possible deadlock in __generic_file_fsync

Status: public: reported syz repro on 2019/04/12 00:01
Reported-by: syzbot+7d66ca77f7bdfbb8e684@syzkaller.appspotmail.com
First crash: 2011d, last: 1600d
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 possible deadlock in __generic_file_fsync ext4 btrfs fat C 577 424d 1772d 0/1 upstream: reported C repro on 2019/06/13 18:12
linux-4.14 possible deadlock in __generic_file_fsync ext4 fat syz error 171 429d 1766d 0/1 upstream: reported syz repro on 2019/06/19 20:20
upstream possible deadlock in __generic_file_fsync ext4 C done unreliable 80 1872d 2012d 0/26 auto-obsoleted due to no activity on 2022/09/04 03:10

Sample crash report:
audit: type=1400 audit(1539755749.782:12): avc:  denied  { read } for  pid=4438 comm="syz-executor0" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
audit: type=1400 audit(1539755749.932:13): avc:  denied  { map } for  pid=4438 comm="syz-executor0" path="/dev/loop-control" dev="devtmpfs" ino=1058 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1

======================================================
WARNING: possible circular locking dependency detected
4.14.76+ #20 Not tainted
------------------------------------------------------
kworker/1:1/68 is trying to acquire lock:
 (&sb->s_type->i_mutex_key#9){+.+.}, at: [<ffffffff921db45e>] inode_lock include/linux/fs.h:713 [inline]
 (&sb->s_type->i_mutex_key#9){+.+.}, at: [<ffffffff921db45e>] __generic_file_fsync+0x9e/0x1a0 fs/libfs.c:981

but task is already holding lock:
 ((&dio->complete_work)){+.+.}, at: [<ffffffff91d2795f>] process_one_work+0x7bf/0x15c0 kernel/workqueue.c:2089

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((&dio->complete_work)){+.+.}:
       process_one_work+0x813/0x15c0 kernel/workqueue.c:2090
       worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
       kthread+0x348/0x420 kernel/kthread.c:232
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402

-> #1 ("dio/%s"sb->s_id){+.+.}:
       flush_workqueue+0xfc/0x1390 kernel/workqueue.c:2616
       drain_workqueue+0x17b/0x3f0 kernel/workqueue.c:2781
       destroy_workqueue+0x23/0x600 kernel/workqueue.c:4088
       sb_init_dio_done_wq+0x5e/0x70 fs/direct-io.c:613
       do_blockdev_direct_IO fs/direct-io.c:1263 [inline]
       __blockdev_direct_IO+0x29e2/0xc4e0 fs/direct-io.c:1399
       ext4_direct_IO_write fs/ext4/inode.c:3710 [inline]
       ext4_direct_IO+0x905/0x27f0 fs/ext4/inode.c:3863
       generic_file_direct_write+0x225/0x430 mm/filemap.c:2940
       __generic_file_write_iter+0x213/0x540 mm/filemap.c:3119
       ext4_file_write_iter+0x4f6/0xe20 fs/ext4/file.c:264
       call_write_iter include/linux/fs.h:1782 [inline]
       aio_write+0x2f2/0x510 fs/aio.c:1551
       io_submit_one fs/aio.c:1639 [inline]
       do_io_submit+0xef4/0x12a0 fs/aio.c:1707
       do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (&sb->s_type->i_mutex_key#9){+.+.}:
       lock_acquire+0x10f/0x380 kernel/locking/lockdep.c:3991
       down_write+0x34/0x90 kernel/locking/rwsem.c:54
       inode_lock include/linux/fs.h:713 [inline]
       __generic_file_fsync+0x9e/0x1a0 fs/libfs.c:981
       ext4_sync_file+0x39e/0x1050 fs/ext4/fsync.c:120
       vfs_fsync_range+0x105/0x260 fs/sync.c:196
       generic_write_sync include/linux/fs.h:2689 [inline]
       dio_complete+0x621/0x800 fs/direct-io.c:308
       process_one_work+0x86e/0x15c0 kernel/workqueue.c:2114
       worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
       kthread+0x348/0x420 kernel/kthread.c:232
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402

other info that might help us debug this:

Chain exists of:
  &sb->s_type->i_mutex_key#9 --> "dio/%s"sb->s_id --> (&dio->complete_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((&dio->complete_work));
                               lock("dio/%s"sb->s_id);
                               lock((&dio->complete_work));
  lock(&sb->s_type->i_mutex_key#9);

 *** DEADLOCK ***

2 locks held by kworker/1:1/68:
 #0:  ("dio/%s"sb->s_id){+.+.}, at: [<ffffffff91d27927>] process_one_work+0x787/0x15c0 kernel/workqueue.c:2085
 #1:  ((&dio->complete_work)){+.+.}, at: [<ffffffff91d2795f>] process_one_work+0x7bf/0x15c0 kernel/workqueue.c:2089

stack backtrace:
CPU: 1 PID: 68 Comm: kworker/1:1 Not tainted 4.14.76+ #20
Workqueue: dio/sda1 dio_aio_complete_work
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0xb9/0x11b lib/dump_stack.c:53
 print_circular_bug.isra.18.cold.43+0x2d3/0x40c kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2ff9/0x4320 kernel/locking/lockdep.c:3487
 lock_acquire+0x10f/0x380 kernel/locking/lockdep.c:3991
 down_write+0x34/0x90 kernel/locking/rwsem.c:54
 inode_lock include/linux/fs.h:713 [inline]
 __generic_file_fsync+0x9e/0x1a0 fs/libfs.c:981
 ext4_sync_file+0x39e/0x1050 fs/ext4/fsync.c:120
 vfs_fsync_range+0x105/0x260 fs/sync.c:196
 generic_write_sync include/linux/fs.h:2689 [inline]
 dio_complete+0x621/0x800 fs/direct-io.c:308
 process_one_work+0x86e/0x15c0 kernel/workqueue.c:2114
 worker_thread+0xdc/0x1000 kernel/workqueue.c:2248
 kthread+0x348/0x420 kernel/kthread.c:232
 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:402
hrtimer: interrupt took 15813 ns

Crashes (262):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/10/17 06:10 android-4.14 6d46bcc5a747 1ba7fd7e .config console log report syz ci-android-414-kasan-gce-root
2019/12/02 18:19 android-4.14 13855a652bd5 f879db37 .config console log report ci-android-414-kasan-gce-root
2019/12/02 17:14 android-4.14 13855a652bd5 f879db37 .config console log report ci-android-414-kasan-gce-root
2019/12/01 21:22 android-4.14 13855a652bd5 a76bf83f .config console log report ci-android-414-kasan-gce-root
2019/11/29 07:41 android-4.14 714ada7cabc7 76357d6f .config console log report ci-android-414-kasan-gce-root
2019/11/24 10:32 android-4.14 437a2a739c5f 598ca6c8 .config console log report ci-android-414-kasan-gce-root
2019/11/23 18:55 android-4.14 437a2a739c5f 598ca6c8 .config console log report ci-android-414-kasan-gce-root
2019/11/22 09:40 android-4.14 7bc77fd33905 8098ea0f .config console log report ci-android-414-kasan-gce-root
2019/11/18 22:17 android-4.14 460dc7c31cef 1daed50a .config console log report ci-android-414-kasan-gce-root
2019/11/17 08:25 android-4.14 460dc7c31cef d5696d51 .config console log report ci-android-414-kasan-gce-root
2019/11/13 12:41 android-4.14 0ac69147fd8c 048f2d49 .config console log report ci-android-414-kasan-gce-root
2019/11/13 03:05 android-4.14 0ac69147fd8c 048f2d49 .config console log report ci-android-414-kasan-gce-root
2019/11/12 16:41 android-4.14 10e570bfc15a 048f2d49 .config console log report ci-android-414-kasan-gce-root
2019/11/12 08:26 android-4.14 10e570bfc15a 048f2d49 .config console log report ci-android-414-kasan-gce-root
2019/11/10 12:12 android-4.14 81144e705f48 dc438b91 .config console log report ci-android-414-kasan-gce-root
2019/11/10 05:21 android-4.14 81144e705f48 dc438b91 .config console log report ci-android-414-kasan-gce-root
2019/11/09 22:56 android-4.14 81144e705f48 dc438b91 .config console log report ci-android-414-kasan-gce-root
2019/11/09 20:41 android-4.14 81144e705f48 dc438b91 .config console log report ci-android-414-kasan-gce-root
2019/11/03 10:59 android-4.14 6409e7e01d11 a41ca8fa .config console log report ci-android-414-kasan-gce-root
2019/11/03 04:48 android-4.14 6409e7e01d11 a41ca8fa .config console log report ci-android-414-kasan-gce-root
2019/11/01 17:21 android-4.14 6409e7e01d11 a41ca8fa .config console log report ci-android-414-kasan-gce-root
2019/10/31 15:02 android-4.14 2bb70f40b08b a41ca8fa .config console log report ci-android-414-kasan-gce-root
2019/10/28 02:43 android-4.14 0b383e2946f5 25bb509e .config console log report ci-android-414-kasan-gce-root
2019/10/27 22:21 android-4.14 0b383e2946f5 25bb509e .config console log report ci-android-414-kasan-gce-root
2019/10/26 00:52 android-4.14 4039d072b0dc c2e837da .config console log report ci-android-414-kasan-gce-root
2019/10/25 13:39 android-4.14 3e5dc4ed2d3f d01bb02a .config console log report ci-android-414-kasan-gce-root
2019/10/23 00:50 android-4.14 7d642373db4c 5681358a .config console log report ci-android-414-kasan-gce-root
2019/10/21 19:13 android-4.14 234de92896af b24d2b8a .config console log report ci-android-414-kasan-gce-root
2019/10/21 11:47 android-4.14 234de92896af 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/21 09:33 android-4.14 234de92896af 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/21 08:24 android-4.14 234de92896af 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/20 17:36 android-4.14 234de92896af 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/20 12:48 android-4.14 234de92896af 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/17 02:20 android-4.14 248a268ad139 8c88c9c1 .config console log report ci-android-414-kasan-gce-root
2019/10/15 21:02 android-4.14 248a268ad139 b5268b89 .config console log report ci-android-414-kasan-gce-root
2019/10/15 13:17 android-4.14 248a268ad139 b5268b89 .config console log report ci-android-414-kasan-gce-root
2019/10/14 02:25 android-4.14 1d75f58e4e19 2f661ec4 .config console log report ci-android-414-kasan-gce-root
2019/10/13 15:26 android-4.14 1d75f58e4e19 2f661ec4 .config console log report ci-android-414-kasan-gce-root
2019/10/13 14:04 android-4.14 1d75f58e4e19 2f661ec4 .config console log report ci-android-414-kasan-gce-root
2019/10/13 05:15 android-4.14 1d75f58e4e19 426631dd .config console log report ci-android-414-kasan-gce-root
2019/10/12 19:39 android-4.14 1d75f58e4e19 426631dd .config console log report ci-android-414-kasan-gce-root
2019/10/11 22:31 android-4.14 5faab626bf1f 426631dd .config console log report ci-android-414-kasan-gce-root
2019/10/10 14:25 android-4.14 3150b5bf7ab8 a4efa8c0 .config console log report ci-android-414-kasan-gce-root
* Struck through repros no longer work on HEAD.