ci2 starts bisection 2023-06-08 00:00:44.7832714 +0000 UTC m=+23917.020912910 bisecting cause commit starting from a27648c742104a833a01c54becc24429898d85bf building syzkaller on 058b3a5a6a945a55767811552eb7b9f4a20307f8 ensuring issue is reproducible on original commit a27648c742104a833a01c54becc24429898d85bf testing commit a27648c742104a833a01c54becc24429898d85bf gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3b379bc2c5a137eabaf506d72a8c5b15ebb738b1d096b3b3fd8223d35bb8c997 all runs: crashed: WARNING in btrfs_split_ordered_extent testing release v6.3 testing commit 457391b0380335d5e9a5babdec90ac53928b23b4 gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c5ea961418dd6e6fd5845134568a609bc682ca2d9d136da1be9ff8e70ed67187 all runs: OK # git bisect start a27648c742104a833a01c54becc24429898d85bf 457391b0380335d5e9a5babdec90ac53928b23b4 Bisecting: 7275 revisions left to test after this (roughly 13 steps) [6e98b09da931a00bf4e0477d0fa52748bf28fcce] Merge tag 'net-next-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 6e98b09da931a00bf4e0477d0fa52748bf28fcce gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 716a9fdbdf8e328bbf8a80bbdc27a2ea2e35cda2d9c7ba8fa41d0845d27fdd31 all runs: crashed: WARNING in btrfs_split_ordered_extent # git bisect bad 6e98b09da931a00bf4e0477d0fa52748bf28fcce Bisecting: 3926 revisions left to test after this (roughly 12 steps) [088e0c188513b58a0056a488cf5b7df094a8a48a] Merge tag 'platform-drivers-x86-v6.4-1' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86 testing commit 088e0c188513b58a0056a488cf5b7df094a8a48a gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b01c4a279cb9f1eb7b8527a9e7ac67e25761dd2cc29bc0824567842e7505453d all runs: OK # git bisect good 088e0c188513b58a0056a488cf5b7df094a8a48a Bisecting: 1938 revisions left to test after this (roughly 11 steps) [ca288965801572fe41386560d4e6c5cc0e5cc56d] Merge tag 'wireless-next-2023-04-21' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next testing commit ca288965801572fe41386560d4e6c5cc0e5cc56d gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6d3a8044014b895244486b76fcd1b5e535c6db0aae0a81e29b1ae828eb7eaee0 all runs: OK # git bisect good ca288965801572fe41386560d4e6c5cc0e5cc56d Bisecting: 1030 revisions left to test after this (roughly 10 steps) [94fc0792661a96d64a4bb79cf10d0793ecadf76e] Merge tag 'fs_for_v6.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs testing commit 94fc0792661a96d64a4bb79cf10d0793ecadf76e gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5c7b3b3672125eeb0a667a416d3d9a9b66fb032fc463e13a9e2f32fbceba63e3 all runs: OK # git bisect good 94fc0792661a96d64a4bb79cf10d0793ecadf76e Bisecting: 512 revisions left to test after this (roughly 9 steps) [36006b1d5c04692924f011aa949e8788f1c604de] Merge tag 'ata-6.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata testing commit 36006b1d5c04692924f011aa949e8788f1c604de gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1d4f506936203c8be316848eccf0379c5b02b22781eb30b508ab107d09286c3e all runs: crashed: WARNING in btrfs_split_ordered_extent # git bisect bad 36006b1d5c04692924f011aa949e8788f1c604de Bisecting: 291 revisions left to test after this (roughly 8 steps) [5c7ecada25d2086aee607ff7deb69e77faa4aa92] Merge tag 'f2fs-for-6.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs testing commit 5c7ecada25d2086aee607ff7deb69e77faa4aa92 gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8b04b6f182a451f1db4ba8f8add170aebdf1c8dc13a6124af83782c493aec0a4 all runs: crashed: WARNING in btrfs_split_ordered_extent # git bisect bad 5c7ecada25d2086aee607ff7deb69e77faa4aa92 Bisecting: 112 revisions left to test after this (roughly 7 steps) [12be09fe18f2fd9f882ca0acbe14cf121250bcbe] block: async_bio_lock does not need to be bh-safe testing commit 12be09fe18f2fd9f882ca0acbe14cf121250bcbe gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7801de9c46ddb2fa3df45e8eaff7f5cb3be02d43a97ab7d7d840a7acd2356577 all runs: crashed: WARNING in btrfs_split_ordered_extent # git bisect bad 12be09fe18f2fd9f882ca0acbe14cf121250bcbe Bisecting: 56 revisions left to test after this (roughly 6 steps) [b7d463a1d1252c2cd5e9f13c008eb49b8a5f75af] btrfs: store a pointer to the original btrfs_bio in struct compressed_bio testing commit b7d463a1d1252c2cd5e9f13c008eb49b8a5f75af gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4c532f7794914cd3459d70b986283cfaa7c7b83e92b09b2d34436ab6a1b8a27d all runs: OK # git bisect good b7d463a1d1252c2cd5e9f13c008eb49b8a5f75af Bisecting: 28 revisions left to test after this (roughly 5 steps) [b13d57db90b859dadb33ccdb3716c9c3ed0a825d] btrfs: calculate correct amount of space for delayed reference when evicting testing commit b13d57db90b859dadb33ccdb3716c9c3ed0a825d gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b55922ccf803366dafa217c40b9b2afa0e5aebb16559d30cf9450dee094d33d9 all runs: OK # git bisect good b13d57db90b859dadb33ccdb3716c9c3ed0a825d Bisecting: 14 revisions left to test after this (roughly 4 steps) [e44ca71cfe07c5133a35102f2aeb200370614bb2] btrfs: move ordered_extent internal sanity checks into btrfs_split_ordered_extent testing commit e44ca71cfe07c5133a35102f2aeb200370614bb2 gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 660ca3d0864914ab21c2671eac879a7cc0d8e41e0f94073893188399df604942 all runs: OK # git bisect good e44ca71cfe07c5133a35102f2aeb200370614bb2 Bisecting: 7 revisions left to test after this (roughly 3 steps) [b73a6fd1b1efd799c6e3d14a922887f4453fea17] btrfs: split partial dio bios before submit testing commit b73a6fd1b1efd799c6e3d14a922887f4453fea17 gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5e35135335e8b22c3265f24cdfedae6fb7311535037d59bcc8de7602eb1362ae all runs: crashed: WARNING in btrfs_split_ordered_extent # git bisect bad b73a6fd1b1efd799c6e3d14a922887f4453fea17 Bisecting: 3 revisions left to test after this (roughly 2 steps) [f0792b792dbe862847b1d590beed372a01b99af0] btrfs: fold btrfs_clone_ordered_extent into btrfs_split_ordered_extent testing commit f0792b792dbe862847b1d590beed372a01b99af0 gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4abdac25d889902c71fafd0899d69b261c53b2ee2d7da5afa0673fcc887e5dc5 all runs: OK # git bisect good f0792b792dbe862847b1d590beed372a01b99af0 Bisecting: 1 revision left to test after this (roughly 1 step) [7edd339c8a416afed9d58b5d20778d5ee49e079f] btrfs: pass an ordered_extent to btrfs_extract_ordered_extent testing commit 7edd339c8a416afed9d58b5d20778d5ee49e079f gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b96afca44c00f27f0c701cf9746966da4ff6703af657d6b50632eb78e3feeb27 all runs: OK # git bisect good 7edd339c8a416afed9d58b5d20778d5ee49e079f Bisecting: 0 revisions left to test after this (roughly 0 steps) [f0f5329a00ba5e9aec012212e3d9d8c2ff8f5cfa] btrfs: don't split NOCOW extent_maps in btrfs_extract_ordered_extent testing commit f0f5329a00ba5e9aec012212e3d9d8c2ff8f5cfa gcc compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5bf7939f767ad0fb6264f5829f706c819a16cef5e766b29bba58e72d81f49d09 all runs: OK # git bisect good f0f5329a00ba5e9aec012212e3d9d8c2ff8f5cfa b73a6fd1b1efd799c6e3d14a922887f4453fea17 is the first bad commit commit b73a6fd1b1efd799c6e3d14a922887f4453fea17 Author: Boris Burkov Date: Tue Mar 28 14:19:57 2023 +0900 btrfs: split partial dio bios before submit If an application is doing direct io to a btrfs file and experiences a page fault reading from the write buffer, iomap will issue a partial bio, and allow the fs to keep going. However, there was a subtle bug in this code path in the btrfs dio iomap implementation that led to the partial write ending up as a gap in the file's extents and to be read back as zeros. The sequence of events in a partial write, lightly summarized and trimmed down for brevity is as follows: ==== WRITING TASK ==== btrfs_direct_write __iomap_dio_write iomap_iter btrfs_dio_iomap_begin # create full ordered extent iomap_dio_bio_iter bio_iov_iter_get_pages # page fault; partial read submit_bio # partial bio iomap_iter btrfs_dio_iomap_end btrfs_mark_ordered_io_finished # sets BTRFS_ORDERED_IOERR; # submit to finish_ordered_fn wq fault_in_iov_iter_readable # btrfs_direct_write detects partial write __iomap_dio_write iomap_iter btrfs_dio_iomap_begin # create second partial ordered extent iomap_dio_bio_iter bio_iov_iter_get_pages # read all of remainder submit_bio # partial bio with all of remainder iomap_iter btrfs_dio_iomap_end # nothing exciting to do with ordered io ==== DIO ENDIO ==== == FIRST PARTIAL BIO == btrfs_dio_end_io btrfs_mark_ordered_io_finished # bytes_left > 0 # don't submit to finish_ordered_fn wq == SECOND PARTIAL BIO == btrfs_dio_end_io btrfs_mark_ordered_io_finished # bytes_left == 0 # submit to finish_ordered_fn wq ==== BTRFS FINISH ORDERED WQ ==== == FIRST PARTIAL BIO == btrfs_finish_ordered_io # called by dio_iomap_end_io, sees # BTRFS_ORDERED_IOERR, just drops the # ordered_extent ==SECOND PARTIAL BIO== btrfs_finish_ordered_io # called by btrfs_dio_end_io, writes out file # extents, csums, etc... The essence of the problem is that while btrfs_direct_write and iomap properly interact to submit all the correct bios, there is insufficient logic in the btrfs dio functions (btrfs_dio_iomap_begin, btrfs_dio_submit_io, btrfs_dio_end_io, and btrfs_dio_iomap_end) to ensure that every bio is at least a part of a completed ordered_extent. And it is completing an ordered_extent that results in crucial functionality like writing out a file extent for the range. More specifically, btrfs_dio_end_io treats the ordered extent as unfinished but btrfs_dio_iomap_end sets BTRFS_ORDERED_IOERR on it. Thus, the finish io work doesn't result in file extents, csums, etc. In the aftermath, such a file behaves as though it has a hole in it, instead of the purportedly written data. We considered a few options for fixing the bug: 1. treat the partial bio as if we had truncated the file, which would result in properly finishing it. 2. split the ordered extent when submitting a partial bio. 3. cache the ordered extent across calls to __iomap_dio_rw in iter->private, so that we could reuse it and correctly apply several bios to it. I had trouble with 1, and it felt the most like a hack, so I tried 2 and 3. Since 3 has the benefit of also not creating an extra file extent, and avoids an ordered extent lookup during bio submission, it felt like the best option. However, that turned out to re-introduce a deadlock which this code discarding the ordered_extent between faults was meant to fix in the first place. (Link to an explanation of the deadlock below.) Therefore, go with fix 2, which requires a bit more setup work but fixes the corruption without introducing the deadlock, which is fundamentally caused by the ordered extent existing when we attempt to fault in a range that overlaps with it. Put succinctly, what this patch does is: when we submit a dio bio, check if it is partial against the ordered extent stored in dio_data, and if it is, extract the ordered_extent that matches the bio exactly out of the larger ordered_extent. Keep the remaining ordered_extent around in dio_data for cancellation in iomap_end. Thanks to Josef, Christoph, and Filipe with their help figuring out the bug and the fix. Fixes: 51bd9563b678 ("btrfs: fix deadlock due to page faults during direct IO reads and writes") Link: https://bugzilla.redhat.com/show_bug.cgi?id=2169947 Link: https://lore.kernel.org/linux-btrfs/aa1fb69e-b613-47aa-a99e-a0a2c9ed273f@app.fastmail.com/ Link: https://pastebin.com/3SDaH8C6 Link: https://lore.kernel.org/linux-btrfs/20230315195231.GW10580@twin.jikos.cz/T/#t Reviewed-by: Josef Bacik Tested-by: Johannes Thumshirn Signed-off-by: Boris Burkov [ hch: refactored the ordered_extent extraction ] Signed-off-by: Christoph Hellwig Signed-off-by: David Sterba fs/btrfs/inode.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) culprit signature: 5e35135335e8b22c3265f24cdfedae6fb7311535037d59bcc8de7602eb1362ae parent signature: 5bf7939f767ad0fb6264f5829f706c819a16cef5e766b29bba58e72d81f49d09 revisions tested: 16, total time: 8h38m42.310708836s (build: 6h22m45.812467821s, test: 2h13m35.956384355s) first bad commit: b73a6fd1b1efd799c6e3d14a922887f4453fea17 btrfs: split partial dio bios before submit recipients (to): ["boris@bur.io" "dsterba@suse.com" "hch@lst.de" "johannes.thumshirn@wdc.com" "josef@toxicpanda.com"] recipients (cc): [] crash: WARNING in btrfs_split_ordered_extent ------------[ cut here ]------------ WARNING: CPU: 0 PID: 5457 at fs/btrfs/ordered-data.c:1138 btrfs_split_ordered_extent+0x541/0x6b0 Modules linked in: CPU: 0 PID: 5457 Comm: syz-executor.0 Not tainted 6.3.0-rc7-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023 RIP: 0010:btrfs_split_ordered_extent+0x541/0x6b0 fs/btrfs/ordered-data.c:1138 Code: 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 c7 c7 e0 46 2a 8a 48 c7 c6 60 42 2a 8a ba 6c 04 00 00 e8 34 46 62 06 e9 26 fc ff ff <0f> 0b eb 11 0f 0b e9 df fb ff ff 0f 0b eb 06 0f 0b eb 02 0f 0b b8 RSP: 0018:ffffc9000423ee68 EFLAGS: 00010287 RAX: 1ffff110034d8801 RBX: ffff88801a6c4008 RCX: ffffffff833827d6 RDX: 0000000000000000 RSI: ffffffff8a37fde0 RDI: ffffffff8a37fda0 RBP: 0000000000001000 R08: dffffc0000000000 R09: fffffbfff1a7b25e R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000 R13: 0000000000000000 R14: ffff88801a6c4000 R15: ffff8880746d16f8 FS: 00007f3a635fe700(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f270506d3b0 CR3: 000000007b9c4000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: btrfs_extract_ordered_extent+0x111/0xa00 fs/btrfs/inode.c:2621 btrfs_dio_submit_io+0x19e/0x290 fs/btrfs/inode.c:7728 iomap_dio_submit_bio fs/iomap/direct-io.c:75 [inline] iomap_dio_bio_iter+0xb4f/0x10d0 fs/iomap/direct-io.c:354 __iomap_dio_rw+0x1039/0x1ab0 fs/iomap/direct-io.c:595 btrfs_dio_write+0xa6/0xf0 fs/btrfs/inode.c:7761 btrfs_direct_write fs/btrfs/file.c:1529 [inline] btrfs_do_write_iter+0x723/0xeb0 fs/btrfs/file.c:1674 do_iter_write+0x668/0xad0 fs/read_write.c:861 iter_file_splice_write+0x7c4/0xfc0 fs/splice.c:778 do_splice_from fs/splice.c:856 [inline] direct_splice_actor+0xe2/0x1a0 fs/splice.c:1022 splice_direct_to_actor+0x42e/0xa60 fs/splice.c:977 do_splice_direct+0x268/0x3a0 fs/splice.c:1065 do_sendfile+0x4f5/0xc20 fs/read_write.c:1255 __do_sys_sendfile64 fs/read_write.c:1323 [inline] __se_sys_sendfile64+0x143/0x190 fs/read_write.c:1309 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f3a6b68c169 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f3a635fe168 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 RAX: ffffffffffffffda RBX: 00007f3a6b7ac050 RCX: 00007f3a6b68c169 RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000004 RBP: 00007f3a6b6e7ca1 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000008800000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fff1e69518f R14: 00007f3a635fe300 R15: 0000000000022000