bisecting cause commit starting from 6abab1b81b657ca74b7c443e832d95c87901e75b building syzkaller on 4ebb27982f8984ed57466f87099acc0b250a1b5c testing commit 6abab1b81b657ca74b7c443e832d95c87901e75b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6eec58f6901f407be8c84e72022af0139b6bc5a808ba274eb73d5b6e38ffd4d9 run #0: crashed: possible deadlock in f2fs_write_checkpoint run #1: crashed: possible deadlock in f2fs_write_checkpoint run #2: crashed: possible deadlock in f2fs_write_checkpoint run #3: crashed: possible deadlock in f2fs_write_checkpoint run #4: basic kernel testing failed: BUG: program execution failed: executor NUM: failed to write control pipe: write |NUM: broken pipe run #5: crashed: possible deadlock in f2fs_write_checkpoint run #6: crashed: possible deadlock in f2fs_write_checkpoint run #7: crashed: possible deadlock in f2fs_write_checkpoint run #8: crashed: possible deadlock in f2fs_write_checkpoint run #9: crashed: possible deadlock in f2fs_write_checkpoint run #10: crashed: possible deadlock in f2fs_write_checkpoint run #11: crashed: possible deadlock in f2fs_write_checkpoint run #12: crashed: possible deadlock in f2fs_write_checkpoint run #13: crashed: possible deadlock in f2fs_write_checkpoint run #14: crashed: possible deadlock in f2fs_write_checkpoint run #15: crashed: possible deadlock in f2fs_write_checkpoint run #16: crashed: possible deadlock in f2fs_write_checkpoint run #17: crashed: possible deadlock in f2fs_write_checkpoint run #18: crashed: possible deadlock in f2fs_write_checkpoint run #19: crashed: possible deadlock in f2fs_write_checkpoint testing release v5.16 testing commit df0cc57e057f18e44dac8e6c18aba47ab53202f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6392e7dc90e0a4647080a5029d93ca58348fb8a5852ea8aba24fe1cfc2cb5f6d all runs: OK # git bisect start 6abab1b81b657ca74b7c443e832d95c87901e75b df0cc57e057f18e44dac8e6c18aba47ab53202f9 Bisecting: 7959 revisions left to test after this (roughly 13 steps) [455e73a07f6e288b0061dfcf4fcf54fa9fe06458] Merge tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux testing commit 455e73a07f6e288b0061dfcf4fcf54fa9fe06458 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 688a0317b34949a2fa1a03a23354cdcff72b3cd85e39730081264c9410b248f1 all runs: OK # git bisect good 455e73a07f6e288b0061dfcf4fcf54fa9fe06458 Bisecting: 3979 revisions left to test after this (roughly 12 steps) [82b550fa99f2c73636ca3f84900117c3c3272ef7] Merge tag 'drm-fixes-2022-01-28' of git://anongit.freedesktop.org/drm/drm testing commit 82b550fa99f2c73636ca3f84900117c3c3272ef7 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4e3d44efcf43aeddacf04957b5ad086abd21953f26311b13ac81d54c931aece6 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 82b550fa99f2c73636ca3f84900117c3c3272ef7 Bisecting: 2016 revisions left to test after this (roughly 11 steps) [ab4cdccffff704568beed9d26308479fe34e9008] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git testing commit ab4cdccffff704568beed9d26308479fe34e9008 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cf2a75a4a47e1605230dd22730bf43b5e90b78dcd51862b2bd67c0d93a26a0c3 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad ab4cdccffff704568beed9d26308479fe34e9008 Bisecting: 976 revisions left to test after this (roughly 10 steps) [d505a8841301bf9bdf087133060c45a5d99f9257] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip.git testing commit d505a8841301bf9bdf087133060c45a5d99f9257 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 03ddc957bf6b24cd2ee009bec4929edd1d9948ac402386ef8a60148b9ee2b617 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good d505a8841301bf9bdf087133060c45a5d99f9257 Bisecting: 471 revisions left to test after this (roughly 9 steps) [80998d0eb4f3efbf28ef7497ada5eb164cf74fda] Merge branch 'master' of git://linuxtv.org/media_tree.git testing commit 80998d0eb4f3efbf28ef7497ada5eb164cf74fda compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0d21f12c7fdca1e29df3f2534bd44742108922b58f5378dc610b0532dadb35a9 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad 80998d0eb4f3efbf28ef7497ada5eb164cf74fda Bisecting: 240 revisions left to test after this (roughly 8 steps) [c9be336e7bc2349a4edf2e716018e099a9044565] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git testing commit c9be336e7bc2349a4edf2e716018e099a9044565 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d8b9ad81810b902560b59f7a3478e57fabacdd24b9f9614edfbca64282452e81 all runs: OK # git bisect good c9be336e7bc2349a4edf2e716018e099a9044565 Bisecting: 115 revisions left to test after this (roughly 7 steps) [01609b789dc1dfff14bf26d5ba98f1ec6bab9a3a] Merge branch 'hwmon-next' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git testing commit 01609b789dc1dfff14bf26d5ba98f1ec6bab9a3a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 54f5b7fd5be5de03aa713df68c44706441355f21ec59a1cbe3b573c8e9188c59 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad 01609b789dc1dfff14bf26d5ba98f1ec6bab9a3a Bisecting: 61 revisions left to test after this (roughly 6 steps) [62a7aedd7776a4a08a2052a28c13b59f14bbc480] Merge branch 'iomap-for-next' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux.git testing commit 62a7aedd7776a4a08a2052a28c13b59f14bbc480 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 12cd4c7acedc4a2bc04cc7f8233e7d0e445ec64b45f6303e313faf74e1450630 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad 62a7aedd7776a4a08a2052a28c13b59f14bbc480 Bisecting: 32 revisions left to test after this (roughly 5 steps) [71e8166a11ab140a2f897324a2f027094df24f34] Merge branch 'dev' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git testing commit 71e8166a11ab140a2f897324a2f027094df24f34 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5a61d49a369f1028d14b3c9d7ac715c6e103927676ce55f7a6056a6185aeba06 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad 71e8166a11ab140a2f897324a2f027094df24f34 Bisecting: 16 revisions left to test after this (roughly 4 steps) [b36c9466ce982906a6420e6af0fca73d1c0931b5] fs/ext4: fix comments mentioning i_mutex testing commit b36c9466ce982906a6420e6af0fca73d1c0931b5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 9a8e434a30bd546d1bfb4a93457af10f5582255233fff5222219b4ae560b1c7a all runs: OK # git bisect good b36c9466ce982906a6420e6af0fca73d1c0931b5 Bisecting: 8 revisions left to test after this (roughly 3 steps) [c83580e68d64d1131b20ae3dd2125720cbcf1c59] Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6.git testing commit c83580e68d64d1131b20ae3dd2125720cbcf1c59 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6685aedf53ea2296efe92b2cdff58b77cab20f363cbd16f16678723bd95b1bb9 all runs: OK # git bisect good c83580e68d64d1131b20ae3dd2125720cbcf1c59 Bisecting: 4 revisions left to test after this (roughly 2 steps) [f26f32510fcfa76aedaa7aa9113ad6e574d6d4f2] f2fs: expose discard related parameters in sysfs testing commit f26f32510fcfa76aedaa7aa9113ad6e574d6d4f2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1ad650566abcb7e1d93a308a36541c27990edbfef9f6d6e5c56bf3c5312a7389 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad f26f32510fcfa76aedaa7aa9113ad6e574d6d4f2 Bisecting: 1 revision left to test after this (roughly 1 step) [7d19e3dab0002e527052b0aaf986e8c32e5537bf] f2fs: fix to enable ATGC correctly via gc_idle sysfs interface testing commit 7d19e3dab0002e527052b0aaf986e8c32e5537bf compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8cbb262e7e9a31f3c990811407bdd205b3db7437520638254154c1f9f9e9863c all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad 7d19e3dab0002e527052b0aaf986e8c32e5537bf Bisecting: 0 revisions left to test after this (roughly 0 steps) [e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae] f2fs: move f2fs to use reader-unfair rwsems testing commit e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f0eafe5efbcc6328d119ab1f717a41122fadbdb776781e2422356aae72e0b7d2 all runs: crashed: possible deadlock in f2fs_write_checkpoint # git bisect bad e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae is the first bad commit commit e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae Author: Tim Murray Date: Fri Jan 7 12:48:44 2022 -0800 f2fs: move f2fs to use reader-unfair rwsems f2fs rw_semaphores work better if writers can starve readers, especially for the checkpoint thread, because writers are strictly more important than reader threads. This prevents significant priority inversion between low-priority readers that blocked while trying to acquire the read lock and a second acquisition of the write lock that might be blocking high priority work. Signed-off-by: Tim Murray Signed-off-by: Jaegeuk Kim fs/f2fs/checkpoint.c | 34 ++++++++-------- fs/f2fs/compress.c | 6 +-- fs/f2fs/data.c | 50 +++++++++++------------ fs/f2fs/dir.c | 12 +++--- fs/f2fs/f2fs.h | 110 ++++++++++++++++++++++++++++++++++++++++---------- fs/f2fs/file.c | 112 +++++++++++++++++++++++++-------------------------- fs/f2fs/gc.c | 46 ++++++++++----------- fs/f2fs/inline.c | 4 +- fs/f2fs/namei.c | 34 ++++++++-------- fs/f2fs/node.c | 84 +++++++++++++++++++------------------- fs/f2fs/recovery.c | 4 +- fs/f2fs/segment.c | 44 ++++++++++---------- fs/f2fs/super.c | 56 +++++++++++++------------- fs/f2fs/sysfs.c | 4 +- fs/f2fs/verity.c | 4 +- fs/f2fs/xattr.c | 12 +++--- 16 files changed, 342 insertions(+), 274 deletions(-) parent commit dd81e1c7d5fb126e5fbc5c9e334d7b3ec29a16a0 wasn't tested testing commit dd81e1c7d5fb126e5fbc5c9e334d7b3ec29a16a0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b8589986d57e9a744b28288531dd33534ab01cd57a16d5094f5cf86ad4530005 culprit signature: f0eafe5efbcc6328d119ab1f717a41122fadbdb776781e2422356aae72e0b7d2 parent signature: b8589986d57e9a744b28288531dd33534ab01cd57a16d5094f5cf86ad4530005 revisions tested: 16, total time: 3h33m3.787909707s (build: 2h46m35.328348652s, test: 44m33.742156627s) first bad commit: e4544b63a7ee49e7fbebf35ece0a6acd3b9617ae f2fs: move f2fs to use reader-unfair rwsems recipients (to): ["chao@kernel.org" "jaegeuk@kernel.org" "jaegeuk@kernel.org" "linux-f2fs-devel@lists.sourceforge.net" "timmurray@google.com"] recipients (cc): ["linux-kernel@vger.kernel.org"] crash: possible deadlock in f2fs_write_checkpoint F2FS-fs (loop0): Disable nat_bits due to incorrect cp_ver (13274671252798256465, 0) F2FS-fs (loop0): Try to recover 2th superblock, ret: 0 F2FS-fs (loop0): Mounted with checkpoint version = 15213551 ============================================ WARNING: possible recursive locking detected 5.17.0-rc1-syzkaller #0 Not tainted -------------------------------------------- syz-executor288/4062 is trying to acquire lock: ffff888017bc4390 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_down_write fs/f2fs/f2fs.h:2149 [inline] ffff888017bc4390 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_write_checkpoint+0x466/0x5ae0 fs/f2fs/checkpoint.c:1615 but task is already holding lock: ffff888017bc5338 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_down_write fs/f2fs/f2fs.h:2149 [inline] ffff888017bc5338 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_issue_checkpoint+0x2a0/0x430 fs/f2fs/checkpoint.c:1834 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&sem->internal_rwsem#2); lock(&sem->internal_rwsem#2); *** DEADLOCK *** May be due to missing lock nesting notation 2 locks held by syz-executor288/4062: #0: ffff888076f680e0 (&type->s_umount_key#50){+.+.}-{3:3}, at: deactivate_super+0x5f/0x80 fs/super.c:362 #1: ffff888017bc5338 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_down_write fs/f2fs/f2fs.h:2149 [inline] #1: ffff888017bc5338 (&sem->internal_rwsem#2){++++}-{3:3}, at: f2fs_issue_checkpoint+0x2a0/0x430 fs/f2fs/checkpoint.c:1834 stack backtrace: CPU: 0 PID: 4062 Comm: syz-executor288 Not tainted 5.17.0-rc1-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 print_deadlock_bug kernel/locking/lockdep.c:2956 [inline] check_deadlock kernel/locking/lockdep.c:2999 [inline] validate_chain kernel/locking/lockdep.c:3788 [inline] __lock_acquire.cold+0x12c/0x3a9 kernel/locking/lockdep.c:5027 lock_acquire kernel/locking/lockdep.c:5639 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5604 down_write+0x90/0x150 kernel/locking/rwsem.c:1514 f2fs_down_write fs/f2fs/f2fs.h:2149 [inline] f2fs_write_checkpoint+0x466/0x5ae0 fs/f2fs/checkpoint.c:1615 __write_checkpoint_sync fs/f2fs/checkpoint.c:1745 [inline] f2fs_issue_checkpoint+0x2ad/0x430 fs/f2fs/checkpoint.c:1842 sync_filesystem fs/sync.c:63 [inline] sync_filesystem+0x15c/0x1f0 fs/sync.c:30 generic_shutdown_super+0x64/0x3a0 fs/super.c:445 kill_block_super+0x90/0xd0 fs/super.c:1394 kill_f2fs_super+0x241/0x340 fs/f2fs/super.c:4523 deactivate_locked_super+0x7b/0x130 fs/super.c:332 cleanup_mnt+0x324/0x4d0 fs/namespace.c:1143 task_work_run+0xc0/0x160 kernel/task_work.c:164 exit_task_work include/linux/task_work.h:32 [inline] do_exit+0x9a6/0x2500 kernel/exit.c:806 do_group_exit+0xb2/0x2a0 kernel/exit.c:935 __do_sys_exit_group kernel/exit.c:946 [inline] __se_sys_exit_group kernel/exit.c:944 [inline] __x64_sys_exit_group+0x35/0x40 kernel/exit.c:944 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7fbcb9731929 Code: Unable to access opcode bytes at RIP 0x7fbcb97318ff. RSP: 002b:00007fff0bd71888 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7