ci2 starts bisection 2023-09-29 09:56:29.809242781 +0000 UTC m=+65545.997832302 bisecting cause commit starting from 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d building syzkaller on d265efd8b4153abba6f7b44082305976371beaad ensuring issue is reproducible on original commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b89339242caf7119cb2a9581c0a612492e3645335f879a697ebe6016eeee291d all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] check whether we can drop unnecessary instrumentation disabling configs for [UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 4a4569e3c39c0b1fd85e843e67d2da1d6e361ffa7bacb9e5235a3c3bbd656fb9 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the bug reproduces without the instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed kconfig minimization: base=3876 full=7686 leaves diff=2019 split chunks (needed=false): <2019> split chunk #0 of len 2019 into 5 parts testing without sub-chunk 1/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 9f7862c8f7ea23a1fd53e88dec46ceeaf02aeb377c5f42231921ad59ab927d03 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 51f0283f31ff2364122abe6a4e347b42ef9a95d0fe3297aeb7bf8bbfa73bf76e all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c1a5b5329e216e9858395b3f762d9961855ae8bcf47a5ff38c7a5a02317cbd8a all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6d3b2f3cfa2222726a88ada67b05113f2d43de120807842fe69419da3c2ed9c3 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG], they are not needed testing commit 9ed22ae6be817d7a3f5c15ca22cbc9d3963b481d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: cc14ceb88b2c354d7b2210bf428446bb0802b1e1dc5f5a59a31b2fed642ae3ad all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] the chunk can be dropped disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed picked [v6.5 v6.4 v6.3 v6.1 v5.19 v5.17 v5.15 v5.13 v5.10 v5.7 v5.4 v5.1 v4.19] out of 28 release tags testing release v6.5 testing commit 2dde18cd1d8fac735875f2e4987f11817cc0bc2c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 63840353c566e29a74a1e9f57e1e21f59932f7e873a11c8a53b4b94db7f988c5 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] testing release v6.4 testing commit 6995e2de6891c724bfeb2db33d7b87775f913ad1 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 246659f6ead966df6fa22c7c7b6bf13496145fa21db2c2c44ef9481a56c6e53f all runs: OK false negative chance: 0.000 # git bisect start 2dde18cd1d8fac735875f2e4987f11817cc0bc2c 6995e2de6891c724bfeb2db33d7b87775f913ad1 Bisecting: 7364 revisions left to test after this (roughly 13 steps) [b775d6c5859affe00527cbe74263de05cfe6b9f9] Merge tag 'mips_6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux testing commit b775d6c5859affe00527cbe74263de05cfe6b9f9 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 1263c3363f9d75541a2b398e7a754207f85ba757872554e3cab20c53f286702b all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] # git bisect bad b775d6c5859affe00527cbe74263de05cfe6b9f9 Bisecting: 3061 revisions left to test after this (roughly 12 steps) [3a8a670eeeaa40d87bd38a587438952741980c18] Merge tag 'net-next-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 3a8a670eeeaa40d87bd38a587438952741980c18 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 86aa8de160f157ea9f17c9dc1ff60900125e09bdaa074090ca7c23e3e1e135a2 all runs: OK false negative chance: 0.000 # git bisect good 3a8a670eeeaa40d87bd38a587438952741980c18 Bisecting: 1530 revisions left to test after this (roughly 11 steps) [188d3f80fc6d8451ab5e570becd6a7b2d3033023] drm/amdgpu: vcn_4_0 set instance 0 init sched score to 1 testing commit 188d3f80fc6d8451ab5e570becd6a7b2d3033023 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: af1ed5e0c785f2a2ae648c70dc9d69747dc8999590e705d66af9e2d79a89e646 all runs: OK false negative chance: 0.000 # git bisect good 188d3f80fc6d8451ab5e570becd6a7b2d3033023 Bisecting: 916 revisions left to test after this (roughly 10 steps) [d6048fdc870240e5020343f8af0c825829c232bd] Merge tag 'asoc-v6.5' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus testing commit d6048fdc870240e5020343f8af0c825829c232bd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 0f237ba6bd4bf4635ca9394dc9a13aae48dfd9e85ac4a36f1c72886cb4335e67 all runs: OK false negative chance: 0.000 # git bisect good d6048fdc870240e5020343f8af0c825829c232bd Bisecting: 532 revisions left to test after this (roughly 9 steps) [f8824e151fbfa0ac0a258015d606ea6f4a10251b] Merge tag 'sound-6.5-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit f8824e151fbfa0ac0a258015d606ea6f4a10251b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: de61ee034004e730b3bbd1c8c31ae855f91d5efbd561cbb0e98725276a88ba2b all runs: OK false negative chance: 0.000 # git bisect good f8824e151fbfa0ac0a258015d606ea6f4a10251b Bisecting: 248 revisions left to test after this (roughly 8 steps) [2222dcb0775d36de28992f56455ab3967b30d380] Merge tag 'drm-msm-next-2023-06-18' of https://gitlab.freedesktop.org/drm/msm into drm-next testing commit 2222dcb0775d36de28992f56455ab3967b30d380 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: a2683c4d8fac0b34294e1c3afcb844143b671ae1da16ba3e11e7c3d366a6e70c all runs: OK false negative chance: 0.000 # git bisect good 2222dcb0775d36de28992f56455ab3967b30d380 Bisecting: 127 revisions left to test after this (roughly 7 steps) [0a37714f96d5746268dc09bdd400a215f180ba9b] Merge tag 'dlm-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm testing commit 0a37714f96d5746268dc09bdd400a215f180ba9b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e8b97b84822ced27c5bc43fc3da92c91d51b3a3e3c4019a80187f0e3234bfeeb all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] # git bisect bad 0a37714f96d5746268dc09bdd400a215f180ba9b Bisecting: 65 revisions left to test after this (roughly 6 steps) [2ef6c32a914b85217b44a0a2418e830e520b085e] ext4: avoid updating the superblock on a r/o mount if not needed testing commit 2ef6c32a914b85217b44a0a2418e830e520b085e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 3370ad59e7135615ac903d6f3768227ece1c4292937e5cc63e7ba443445382a6 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] # git bisect bad 2ef6c32a914b85217b44a0a2418e830e520b085e Bisecting: 27 revisions left to test after this (roughly 5 steps) [f52f3d2b9fbab73c776f4d3386393e9bbc83b87d] ext4: Give symbolic names to mballoc criterias testing commit f52f3d2b9fbab73c776f4d3386393e9bbc83b87d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: f078a2a0c0f7b443bc78c30c7402b7008c54f38b35bb133c280e2e4594a7575a all runs: OK false negative chance: 0.000 # git bisect good f52f3d2b9fbab73c776f4d3386393e9bbc83b87d Bisecting: 13 revisions left to test after this (roughly 4 steps) [310ee0902b8d9d0a13a5a13e94688a5863fa29c2] ext4: allow concurrent unaligned dio overwrites testing commit 310ee0902b8d9d0a13a5a13e94688a5863fa29c2 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 04889e9b95019a7b41311c034c27a2cdcc852c6928ada0633c3abcded15924c5 all runs: crashed: WARNING in ext4_iomap_begin representative crash: WARNING in ext4_iomap_begin, types: [WARNING] # git bisect bad 310ee0902b8d9d0a13a5a13e94688a5863fa29c2 Bisecting: 6 revisions left to test after this (roughly 3 steps) [4a2d98447b37bcb68a7f06a1078edcb4f7e6ce7e] ext4: using nofail preallocation in ext4_es_insert_delayed_block() testing commit 4a2d98447b37bcb68a7f06a1078edcb4f7e6ce7e gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ed4377178f35d734e7b62050775fa04a5bb3ba362d42351b7f1faf7f0a60213c all runs: OK false negative chance: 0.000 # git bisect good 4a2d98447b37bcb68a7f06a1078edcb4f7e6ce7e Bisecting: 3 revisions left to test after this (roughly 2 steps) [8782b020ccbef6c4b62f00c86423f4d37ec60932] ext4: make ext4_es_insert_delayed_block() return void testing commit 8782b020ccbef6c4b62f00c86423f4d37ec60932 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 901194674655db3e363ebf01f65eb36193e6f3edadef4074e9f1176a13dba524 all runs: OK false negative chance: 0.000 # git bisect good 8782b020ccbef6c4b62f00c86423f4d37ec60932 Bisecting: 1 revision left to test after this (roughly 1 step) [ab8627e104696b8c1c6953ad5255def5b0821e06] ext4: make ext4_zeroout_es() return void testing commit ab8627e104696b8c1c6953ad5255def5b0821e06 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8f5ce087ca641fbaf75b177059881678f3b4fbe3973645d9d827f0587e66205c all runs: OK false negative chance: 0.000 # git bisect good ab8627e104696b8c1c6953ad5255def5b0821e06 Bisecting: 0 revisions left to test after this (roughly 0 steps) [4c0cfebdf3c34c9cd2c55844f549fa46b1da3164] ext4: clean up mballoc criteria comments testing commit 4c0cfebdf3c34c9cd2c55844f549fa46b1da3164 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: b119c86c6c9b96eea753d05eb9a7daa3ab5b54abfb420f0a263213cc8302f35c all runs: OK false negative chance: 0.000 # git bisect good 4c0cfebdf3c34c9cd2c55844f549fa46b1da3164 310ee0902b8d9d0a13a5a13e94688a5863fa29c2 is the first bad commit commit 310ee0902b8d9d0a13a5a13e94688a5863fa29c2 Author: Brian Foster Date: Tue Mar 14 09:07:59 2023 -0400 ext4: allow concurrent unaligned dio overwrites We've had reports of significant performance regression of sub-block (unaligned) direct writes due to the added exclusivity restrictions in ext4. The purpose of the exclusivity requirement for unaligned direct writes is to avoid data corruption caused by unserialized partial block zeroing in the iomap dio layer across overlapping writes. XFS has similar requirements for the same underlying reasons, yet doesn't suffer the extreme performance regression that ext4 does. The reason for this is that XFS utilizes IOMAP_DIO_OVERWRITE_ONLY mode, which allows for optimistic submission of concurrent unaligned I/O and kicks back writes that require partial block zeroing such that they can be submitted in a safe, exclusive context. Since ext4 already performs most of these checks pre-submission, it can support something similar without necessarily relying on the iomap flag and associated retry mechanism. Update the dio write submission path to allow concurrent submission of unaligned direct writes that are purely overwrite and so will not require block zeroing. To improve readability of the various related checks, move the unaligned I/O handling down into ext4_dio_write_checks(), where the dio draining and force wait logic can immediately follow the locking requirement checks. Finally, the IOMAP_DIO_OVERWRITE_ONLY flag is set to enable a warning check as a precaution should the ext4 overwrite logic ever become inconsistent with the zeroing expectations of iomap dio. The performance improvement of sub-block direct write I/O is shown in the following fio test on a 64xcpu guest vm: Test: fio --name=test --ioengine=libaio --direct=1 --group_reporting --overwrite=1 --thread --size=10G --filename=/mnt/fio --readwrite=write --ramp_time=10s --runtime=60s --numjobs=8 --blocksize=2k --iodepth=256 --allow_file_create=0 v6.2: write: IOPS=4328, BW=8724KiB/s v6.2 (patched): write: IOPS=801k, BW=1565MiB/s Signed-off-by: Brian Foster Reviewed-by: Ritesh Harjani (IBM) Reviewed-by: Jan Kara Link: https://lore.kernel.org/r/20230314130759.642710-1-bfoster@redhat.com Signed-off-by: Theodore Ts'o fs/ext4/file.c | 86 +++++++++++++++++++++++++++++++--------------------------- 1 file changed, 46 insertions(+), 40 deletions(-) accumulated error probability: 0.00 culprit signature: 04889e9b95019a7b41311c034c27a2cdcc852c6928ada0633c3abcded15924c5 parent signature: b119c86c6c9b96eea753d05eb9a7daa3ab5b54abfb420f0a263213cc8302f35c revisions tested: 23, total time: 4h31m34.634175679s (build: 1h15m50.68140335s, test: 3h6m42.033851678s) first bad commit: 310ee0902b8d9d0a13a5a13e94688a5863fa29c2 ext4: allow concurrent unaligned dio overwrites recipients (to): ["bfoster@redhat.com" "jack@suse.cz" "ritesh.list@gmail.com" "tytso@mit.edu"] recipients (cc): [] crash: WARNING in ext4_iomap_begin ------------[ cut here ]------------ WARNING: CPU: 0 PID: 2097 at fs/ext4/inode.c:3304 ext4_iomap_begin+0x2e1/0x2f0 fs/ext4/inode.c:3344 Modules linked in: CPU: 0 PID: 2097 Comm: syz-executor.0 Not tainted 6.4.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/06/2023 RIP: 0010:ext4_iomap_begin+0x2e1/0x2f0 fs/ext4/inode.c:3304 Code: e8 44 59 00 00 45 31 e4 65 48 8b 04 25 28 00 00 00 48 3b 44 24 48 75 1c 44 89 e0 48 83 c4 50 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b 41 bc de ff ff ff eb d4 e8 40 7e 7c 00 f3 0f 1e fa 41 83 e0 RSP: 0018:ffffc90001c0fb68 EFLAGS: 00010202 RAX: 0000000000000016 RBX: ffff888113416110 RCX: 000000000000000b RDX: 00000000fffffffe RSI: 000000000000b114 RDI: ffff888113416110 RBP: 000000000000b114 R08: ffffc90001c0fcb0 R09: ffffc90001c0fd00 R10: ffffc90001c0fcb0 R11: 0000000000000000 R12: 00000000ffffffea R13: ffffc90001c0fe01 R14: ffffc90001c0fcb0 R15: 0000000000000001 FS: 00007ff0270206c0(0000) GS:ffff888237c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff02f819086 CR3: 0000000102fc1000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: iomap_iter+0x178/0x3a0 fs/iomap/iter.c:91 __iomap_dio_rw+0x3be/0x840 fs/iomap/direct-io.c:597 iomap_dio_rw+0xd/0x40 fs/iomap/direct-io.c:688 ext4_dio_write_iter fs/ext4/file.c:603 [inline] ext4_file_write_iter+0x58d/0x960 fs/ext4/file.c:714 call_write_iter include/linux/fs.h:1868 [inline] new_sync_write fs/read_write.c:491 [inline] vfs_write+0x334/0x3f0 fs/read_write.c:584 ksys_pwrite64 fs/read_write.c:699 [inline] __do_sys_pwrite64 fs/read_write.c:709 [inline] __se_sys_pwrite64 fs/read_write.c:706 [inline] __x64_sys_pwrite64+0x6a/0xc0 fs/read_write.c:706 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0x90 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7ff02f7beae9 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ff0270200c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 RAX: ffffffffffffffda RBX: 00007ff02f8de050 RCX: 00007ff02f7beae9 RDX: 0000000000000001 RSI: 0000000020000100 RDI: 0000000000000005 RBP: 00007ff02f80a47a R08: 0000000000000000 R09: 0000000000000000 R10: 000000000000b114 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000006 R14: 00007ff02f8de050 R15: 00007fff2c0d03e8