ci2 starts bisection 2024-03-11 21:14:56.52534642 +0000 UTC m=+231787.529174043 bisecting fixing commit since 8a67c06094453c7bf84be53a6cdf569985b7355a building syzkaller on af8d2e46418eefb127e9fa9309a63fa60ef7fc66 ensuring issue is reproducible on original commit 8a67c06094453c7bf84be53a6cdf569985b7355a testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: be0b11783c94b6e2518e2106b2e71e98f7536d1cfa69e5ff06ca3214cf83dab5 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] check whether we can drop unnecessary instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: e9fb33256fc1a06701665f70bc57650fb5556969c61e9ffa2ebb17f298350c21 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] the bug reproduces without the instrumentation disabling configs for [KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN], they are not needed kconfig minimization: base=5179 full=6485 leaves diff=250 split chunks (needed=false): <250> split chunk #0 of len 250 into 5 parts testing without sub-chunk 1/5 disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6e12c2f40bf691d0c1406fe7119138422c5f94cc97a689baa605c47ab8523010 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [LOCKDEP ATOMIC_SLEEP HANG LEAK UBSAN KASAN], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 085ba190bda471049e7bc2d7416bec5627bc3a8ce177f5818776f116ff1d46d1 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 381fd0b318923458cf3335cc901278c095218461710990efeac64ce76b0bfa20 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [ATOMIC_SLEEP HANG LEAK UBSAN KASAN LOCKDEP], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: ec149fb58ce9b082e233bf17142cd50fdcce0a2e91519c82a141e7ce6667d89b all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [UBSAN KASAN LOCKDEP ATOMIC_SLEEP HANG LEAK], they are not needed testing commit 8a67c06094453c7bf84be53a6cdf569985b7355a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 failed building 8a67c06094453c7bf84be53a6cdf569985b7355a: net/socket.c:1225: undefined reference to `wext_handle_ioctl' net/socket.c:3420: undefined reference to `compat_wext_handle_ioctl' net/core/net-procfs.c:329: undefined reference to `wext_proc_init' net/core/net-procfs.c:345: undefined reference to `wext_proc_exit' minimized to 50 configs; suspects: [HID_ZEROPLUS USB_NET_CDC_MBIM USB_NET_CDC_SUBSET USB_NET_CDC_SUBSET_ENABLE USB_NET_DM9601 USB_NET_GL620A USB_NET_MCS7830 USB_NET_NET1080 USB_NET_PLUSB USB_NET_RNDIS_HOST USB_NET_SMSC75XX USB_NET_SMSC95XX USB_NET_SR9700 USB_NET_SR9800 USB_NET_ZAURUS USB_OHCI_HCD USB_OHCI_HCD_PCI USB_OHCI_HCD_PLATFORM USB_OTG USB_OTG_FSM USB_PRINTER USB_SERIAL_GENERIC USB_SERIAL_PL2303 USB_STORAGE_ALAUDA USB_STORAGE_CYPRESS_ATACB USB_STORAGE_DATAFAB USB_STORAGE_FREECOM USB_STORAGE_ISD200 USB_STORAGE_JUMPSHOT USB_STORAGE_KARMA USB_STORAGE_ONETOUCH USB_STORAGE_SDDR09 USB_STORAGE_SDDR55 USB_STORAGE_USBAT USB_TRANCEVIBRATOR USB_U_AUDIO USB_U_ETHER USB_U_SERIAL USB_WDM WLAN WLAN_VENDOR_ATH WLAN_VENDOR_ATMEL WLAN_VENDOR_BROADCOM WLAN_VENDOR_INTERSIL WLAN_VENDOR_MARVELL WLAN_VENDOR_MEDIATEK WLAN_VENDOR_MICROCHIP WLAN_VENDOR_PURELIFI WLAN_VENDOR_RALINK WLAN_VENDOR_REALTEK WLAN_VENDOR_RSI WLAN_VENDOR_SILABS WLAN_VENDOR_ZYDAS X86_X32_ABI ZEROPLUS_FF] disabling configs for [HANG LEAK UBSAN KASAN LOCKDEP ATOMIC_SLEEP], they are not needed testing current HEAD 981ffd6df1161bb1348506c1d117a3f95e3cf830 testing commit 981ffd6df1161bb1348506c1d117a3f95e3cf830 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 52f5f059641317295dc997a813487e0c9e706a74971dcfaef59c1c8eee972adc all runs: OK false negative chance: 0.000 # git bisect start 981ffd6df1161bb1348506c1d117a3f95e3cf830 8a67c06094453c7bf84be53a6cdf569985b7355a Bisecting: 4328 revisions left to test after this (roughly 12 steps) [9cba16beca66ad2e66e3a5952f2ccff98375d4fd] clk: qcom: gpucc-sm6350: Fix clock source names determine whether the revision contains the guilty commit checking the merge base b1644a0031cfb3ca2cbd84c92f771f8ebb62302d no existing result, test the revision testing commit b1644a0031cfb3ca2cbd84c92f771f8ebb62302d gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5567dbd80441e21a6ca13b8427640f450b3ba1700d1f09f3b182a1d2fd392213 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] testing commit 9cba16beca66ad2e66e3a5952f2ccff98375d4fd gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2ebfd0ba064b4a1166ec2296bb0b89478a445196166c300a459371cf76b646d4 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good 9cba16beca66ad2e66e3a5952f2ccff98375d4fd Bisecting: 2165 revisions left to test after this (roughly 11 steps) [55d699e2d2ef7ff5a4ca6751f25dd84b29d11b1c] mcb: fix error handling for different scenarios when parsing determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 55d699e2d2ef7ff5a4ca6751f25dd84b29d11b1c gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: bd0a5ca8416dc869ab8826299e7a7e5f94ed3c9f561b2641a3cabacac19d3826 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good 55d699e2d2ef7ff5a4ca6751f25dd84b29d11b1c Bisecting: 1082 revisions left to test after this (roughly 10 steps) [17a8519cb359c3b483fb5c7367efa9a8a508bdea] uio: Fix use-after-free in uio_open determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 17a8519cb359c3b483fb5c7367efa9a8a508bdea gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 86bd9153240333f427e785b917b09c6fcf0e6966c95a93ec0c17cc6c0d42f98e all runs: OK false negative chance: 0.000 # git bisect bad 17a8519cb359c3b483fb5c7367efa9a8a508bdea Bisecting: 541 revisions left to test after this (roughly 9 steps) [9bb392ee53af7d402dbb344092077ff351b78de7] qed: Fix a potential use-after-free in qed_cxt_tables_alloc determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 9bb392ee53af7d402dbb344092077ff351b78de7 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6d55097d404ccbfec1273680d4e2a8ad065af400ecfab384a5b9d068664383f1 all runs: OK false negative chance: 0.000 # git bisect bad 9bb392ee53af7d402dbb344092077ff351b78de7 Bisecting: 270 revisions left to test after this (roughly 8 steps) [12dd4c1bf3bdd1bfdc5fe2b0f78dcbc28bd4d0d6] selftests/net: unix: fix unused variable compiler warning determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 12dd4c1bf3bdd1bfdc5fe2b0f78dcbc28bd4d0d6 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: c0a867838c70301c011dc00564f7a71edbf1be980877a0a085807d42b93d6697 all runs: OK false negative chance: 0.000 # git bisect bad 12dd4c1bf3bdd1bfdc5fe2b0f78dcbc28bd4d0d6 Bisecting: 134 revisions left to test after this (roughly 7 steps) [8e9a64996528cc7441cb779d801ac21b5d269a5f] drm/rockchip: vop: Fix color for RGB888/BGR888 format on VOP full determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 8e9a64996528cc7441cb779d801ac21b5d269a5f gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 5c09eb15d009bbd3001e1359d42b1205effc4921a09e59a65e2ac13248613d3f all runs: OK false negative chance: 0.000 # git bisect bad 8e9a64996528cc7441cb779d801ac21b5d269a5f Bisecting: 67 revisions left to test after this (roughly 6 steps) [41c269083c4d66c9cea1a573f345aa5836bc8960] media: lirc: drop trailing space from scancode transmit determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 41c269083c4d66c9cea1a573f345aa5836bc8960 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 79d898bb16ca7f77c90614dfa654246516c76dbb0d9aa636de7b1cb7f923c4a0 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good 41c269083c4d66c9cea1a573f345aa5836bc8960 Bisecting: 33 revisions left to test after this (roughly 5 steps) [ec4ba3d62f0fdde57cfaaeb7f1df85609b9a86ef] ext4: correct return value of ext4_convert_meta_bg determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit ec4ba3d62f0fdde57cfaaeb7f1df85609b9a86ef gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 2ae91dabc5d7abfb89da300e4c8584893a0028dee3876955d342b541d0c498fa all runs: OK false negative chance: 0.000 # git bisect bad ec4ba3d62f0fdde57cfaaeb7f1df85609b9a86ef Bisecting: 16 revisions left to test after this (roughly 4 steps) [70ff9b65a72885b3a2dfde6709da1f19b85fa696] mptcp: deal with large GSO size determine whether the revision contains the guilty commit revision b1644a0031cfb3ca2cbd84c92f771f8ebb62302d crashed and is reachable testing commit 70ff9b65a72885b3a2dfde6709da1f19b85fa696 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: be20c0e97eedbdcffb7b256c544b4062c9eea228eae69eb02e8a1f4df5c0825d all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good 70ff9b65a72885b3a2dfde6709da1f19b85fa696 Bisecting: 8 revisions left to test after this (roughly 3 steps) [0f3e5f93fe77bc16e632686b7571d296f91a76be] media: qcom: camss: Fix VFE-17x vfe_disable_output() determine whether the revision contains the guilty commit revision 41c269083c4d66c9cea1a573f345aa5836bc8960 crashed and is reachable testing commit 0f3e5f93fe77bc16e632686b7571d296f91a76be gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 6c108326590816f83053987f7e0fa1e9d444405952ea27cbb76ece8f5c066afa all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good 0f3e5f93fe77bc16e632686b7571d296f91a76be Bisecting: 4 revisions left to test after this (roughly 2 steps) [eb2f435be2c46eea47f60baca419b69f01abf6ad] media: qcom: camss: Fix csid-gen2 for test pattern generator determine whether the revision contains the guilty commit revision 9cba16beca66ad2e66e3a5952f2ccff98375d4fd crashed and is reachable testing commit eb2f435be2c46eea47f60baca419b69f01abf6ad gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 8f69e3f2ffc1f6bc90ffab3c7eb6a0e7a1fdfc1f631f6ed7bad9c9b60e47ca80 all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good eb2f435be2c46eea47f60baca419b69f01abf6ad Bisecting: 2 revisions left to test after this (roughly 1 step) [af075d06b34f79476bcd4e2b07c8632d206dad78] ext4: apply umask if ACL support is disabled determine whether the revision contains the guilty commit revision 9cba16beca66ad2e66e3a5952f2ccff98375d4fd crashed and is reachable testing commit af075d06b34f79476bcd4e2b07c8632d206dad78 gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 061ca2fe598929fafdedc528058e7fef06a0af8c0fedb767cfe76c02db637d0a all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good af075d06b34f79476bcd4e2b07c8632d206dad78 Bisecting: 0 revisions left to test after this (roughly 1 step) [32b9fb9a67ec70bbe3afe931b0ea44203150a49a] ext4: mark buffer new if it is unwritten to avoid stale data exposure determine whether the revision contains the guilty commit revision 9cba16beca66ad2e66e3a5952f2ccff98375d4fd crashed and is reachable testing commit 32b9fb9a67ec70bbe3afe931b0ea44203150a49a gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 19cfbc0e691a74ce0627f03d736e48c31b68634da617040fbfa1604620ca4b36 all runs: OK false negative chance: 0.000 # git bisect bad 32b9fb9a67ec70bbe3afe931b0ea44203150a49a Bisecting: 0 revisions left to test after this (roughly 0 steps) [f0cc1368fafd2542f09d18a75aa32288bc49d11b] ext4: correct offset of gdb backup in non meta_bg group to update_backups determine whether the revision contains the guilty commit revision 9cba16beca66ad2e66e3a5952f2ccff98375d4fd crashed and is reachable testing commit f0cc1368fafd2542f09d18a75aa32288bc49d11b gcc compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 kernel signature: 798795fa58c577c0c6b9854dd306070d741461097add33088f9ad30b29717a9d all runs: crashed: kernel BUG in submit_bh_wbc representative crash: kernel BUG in submit_bh_wbc, types: [BUG] # git bisect good f0cc1368fafd2542f09d18a75aa32288bc49d11b 32b9fb9a67ec70bbe3afe931b0ea44203150a49a is the first bad commit commit 32b9fb9a67ec70bbe3afe931b0ea44203150a49a Author: Ojaswin Mujoo Date: Mon Sep 18 16:15:50 2023 +0530 ext4: mark buffer new if it is unwritten to avoid stale data exposure commit 2cd8bdb5efc1e0d5b11a4b7ba6b922fd2736a87f upstream. ** Short Version ** In ext4 with dioread_nolock, we could have a scenario where the bh returned by get_blocks (ext4_get_block_unwritten()) in __block_write_begin_int() has UNWRITTEN and MAPPED flag set. Since such a bh does not have NEW flag set we never zero out the range of bh that is not under write, causing whatever stale data is present in the folio at that time to be written out to disk. To fix this mark the buffer as new, in case it is unwritten, in ext4_get_block_unwritten(). ** Long Version ** The issue mentioned above was resulting in two different bugs: 1. On block size < page size case in ext4, generic/269 was reliably failing with dioread_nolock. The state of the write was as follows: * The write was extending i_size. * The last block of the file was fallocated and had an unwritten extent * We were near ENOSPC and hence we were switching to non-delayed alloc allocation. In this case, the back trace that triggers the bug is as follows: ext4_da_write_begin() /* switch to nodelalloc due to low space */ ext4_write_begin() ext4_should_dioread_nolock() // true since mount flags still have delalloc __block_write_begin(..., ext4_get_block_unwritten) __block_write_begin_int() for(each buffer head in page) { /* first iteration, this is bh1 which contains i_size */ if (!buffer_mapped) get_block() /* returns bh with only UNWRITTEN and MAPPED */ /* second iteration, bh2 */ if (!buffer_mapped) get_block() /* we fail here, could be ENOSPC */ } if (err) /* * this would zero out all new buffers and mark them uptodate. * Since bh1 was never marked new, we skip it here which causes * the bug later. */ folio_zero_new_buffers(); /* ext4_wrte_begin() error handling */ ext4_truncate_failed_write() ext4_truncate() ext4_block_truncate_page() __ext4_block_zero_page_range() if(!buffer_uptodate()) ext4_read_bh_lock() ext4_read_bh() -> ... ext4_submit_bh_wbc() BUG_ON(buffer_unwritten(bh)); /* !!! */ 2. The second issue is stale data exposure with page size >= blocksize with dioread_nolock. The conditions needed for it to happen are same as the previous issue ie dioread_nolock around ENOSPC condition. The issue is also similar where in __block_write_begin_int() when we call ext4_get_block_unwritten() on the buffer_head and the underlying extent is unwritten, we get an unwritten and mapped buffer head. Since it is not new, we never zero out the partial range which is not under write, thus writing stale data to disk. This can be easily observed with the following reproducer: fallocate -l 4k testfile xfs_io -c "pwrite 2k 2k" testfile # hexdump output will have stale data in from byte 0 to 2k in testfile hexdump -C testfile NOTE: To trigger this, we need dioread_nolock enabled and write happening via ext4_write_begin(), which is usually used when we have -o nodealloc. Since dioread_nolock is disabled with nodelalloc, the only alternate way to call ext4_write_begin() is to ensure that delayed alloc switches to nodelalloc ie ext4_da_write_begin() calls ext4_write_begin(). This will usually happen when ext4 is almost full like the way generic/269 was triggering it in Issue 1 above. This might make the issue harder to hit. Hence, for reliable replication, I used the below patch to temporarily allow dioread_nolock with nodelalloc and then mount the disk with -o nodealloc,dioread_nolock. With this you can hit the stale data issue 100% of times: @@ -508,8 +508,8 @@ static inline int ext4_should_dioread_nolock(struct inode *inode) if (ext4_should_journal_data(inode)) return 0; /* temporary fix to prevent generic/422 test failures */ - if (!test_opt(inode->i_sb, DELALLOC)) - return 0; + // if (!test_opt(inode->i_sb, DELALLOC)) + // return 0; return 1; } After applying this patch to mark buffer as NEW, both the above issues are fixed. Signed-off-by: Ojaswin Mujoo Cc: stable@kernel.org Reviewed-by: Jan Kara Reviewed-by: "Ritesh Harjani (IBM)" Link: https://lore.kernel.org/r/d0ed09d70a9733fbb5349c5c7b125caac186ecdf.1695033645.git.ojaswin@linux.ibm.com Signed-off-by: Theodore Ts'o Signed-off-by: Greg Kroah-Hartman fs/ext4/inode.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) accumulated error probability: 0.00 culprit signature: 19cfbc0e691a74ce0627f03d736e48c31b68634da617040fbfa1604620ca4b36 parent signature: 798795fa58c577c0c6b9854dd306070d741461097add33088f9ad30b29717a9d revisions tested: 22, total time: 3h12m46.023463791s (build: 1h12m12.730508296s, test: 1h52m44.668436028s) first good commit: 32b9fb9a67ec70bbe3afe931b0ea44203150a49a ext4: mark buffer new if it is unwritten to avoid stale data exposure recipients (to): ["gregkh@linuxfoundation.org" "jack@suse.cz" "ojaswin@linux.ibm.com" "ritesh.list@gmail.com" "tytso@mit.edu"] recipients (cc): []