ci starts bisection 2025-11-17 15:09:56.194298548 +0000 UTC m=+16678.440514226 bisecting fixing commit since 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 building syzkaller on 7117feecc9626dc60b06fb3e91c0f7632d99d30b ensuring issue is reproducible on original commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 0f541fb185752f7901d2d100ee309c69d2f38bc772fbf498f88fedd7214dd892 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] check whether we can drop unnecessary instrumentation disabling configs for [kasan locking atomic_sleep memleak ubsan bug_or_warning], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 2408e57b76fb517af9474857e044a74a7354b73e333afbed77e5dcb95701f274 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the bug reproduces without the instrumentation disabling configs for [memleak ubsan bug_or_warning kasan locking atomic_sleep], they are not needed kconfig minimization: base=4116 full=8480 leaves diff=2203 split chunks (needed=false): <2203> split chunk #0 of len 2203 into 5 parts testing without sub-chunk 1/5 disabling configs for [atomic_sleep memleak ubsan bug_or_warning kasan locking], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: bb1c41ac1ae68478854e72ec65376659ca99090ac2e07171957393e9e11eddb4 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the chunk can be dropped testing without sub-chunk 2/5 disabling configs for [locking atomic_sleep memleak ubsan bug_or_warning kasan], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: e599a6e85c798c215b5d939ab0c5ab8bae5dee5ed2c24c149b466d5ca41540a0 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the chunk can be dropped testing without sub-chunk 3/5 disabling configs for [memleak ubsan bug_or_warning kasan locking atomic_sleep], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 770d797f5bf437619f4f09b123550aa127252942dc09022ed82165b817514268 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the chunk can be dropped testing without sub-chunk 4/5 disabling configs for [memleak ubsan bug_or_warning kasan locking atomic_sleep], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: a671514822c44b7d821d67752b8bb60c9835892c39e54d19bf782e54e6bb2d80 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the chunk can be dropped testing without sub-chunk 5/5 disabling configs for [memleak ubsan bug_or_warning kasan locking atomic_sleep], they are not needed testing commit 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 4dd7ea721096dcd119790dba75957fd3d207130525a8b4c2eae171373583abfe all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] the chunk can be dropped disabling configs for [kasan locking atomic_sleep memleak ubsan bug_or_warning], they are not needed testing current HEAD 6a23ae0a96a600d1d12557add110e0bb6e32730c testing commit 6a23ae0a96a600d1d12557add110e0bb6e32730c gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: f113c1fd03e14c31f7f592bc76f55881a1af800df53f5946514b63ce891ed3fd all runs: OK false negative chance: 0.000 # git bisect start 6a23ae0a96a600d1d12557add110e0bb6e32730c 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 Bisecting: 14551 revisions left to test after this (roughly 14 steps) [b8d8265a0db8b3e8a6b40e8a0b25da1c00599577] Merge tag 'asoc-v6.18' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-next determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit b8d8265a0db8b3e8a6b40e8a0b25da1c00599577 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: e0eca6da407a21448948cf773d77cf841f88bf49d6b30f5c9629c05b40afbfe5 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good b8d8265a0db8b3e8a6b40e8a0b25da1c00599577 Bisecting: 7640 revisions left to test after this (roughly 13 steps) [f79e772258df311c2cb21594ca0996318e720d28] Merge tag 'media/v6.18-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit f79e772258df311c2cb21594ca0996318e720d28 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: c68f1f27c4ca04eedf47b95d0b845ce48a2365415820e29a849551148966e00a all runs: OK false negative chance: 0.000 # git bisect bad f79e772258df311c2cb21594ca0996318e720d28 Bisecting: 3515 revisions left to test after this (roughly 12 steps) [a8253f807760e9c80eada9e5354e1240ccf325f9] Merge tag 'soc-newsoc-6.18' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit a8253f807760e9c80eada9e5354e1240ccf325f9 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 3f17cabd0882bf2c6a1d52fb2e01d3720f0246c2d1c1c07c16328f3f18d0725b all runs: OK false negative chance: 0.000 # git bisect bad a8253f807760e9c80eada9e5354e1240ccf325f9 Bisecting: 1701 revisions left to test after this (roughly 11 steps) [2cb8eeaf00efc037988910de17ffe592b23941a6] Merge tag 'x86_cache_for_v6.18_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 2cb8eeaf00efc037988910de17ffe592b23941a6 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 511e7299113d4a19b021596f0bd93e9b0a5cb204c638df110d4ccc923a5b1683 all runs: OK false negative chance: 0.000 # git bisect bad 2cb8eeaf00efc037988910de17ffe592b23941a6 Bisecting: 777 revisions left to test after this (roughly 10 steps) [a9401710a5f5681abd2a6f21f9e76bc9f2e81891] Merge tag 'v6.18-rc-part1-smb3-common' of git://git.samba.org/ksmbd determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit a9401710a5f5681abd2a6f21f9e76bc9f2e81891 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 3eb0add32b9e651d7743060e587102e6529dbe33740aaec9d7bc9aab614aebc5 all runs: OK false negative chance: 0.000 # git bisect bad a9401710a5f5681abd2a6f21f9e76bc9f2e81891 Bisecting: 458 revisions left to test after this (roughly 9 steps) [51a24b7deaae5c3561965f5b4b27bb9d686add1c] Merge tag 'trace-tools-v6.17-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 51a24b7deaae5c3561965f5b4b27bb9d686add1c gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: f03431fc7fe2b34c16c9749d799ed5c2ea027141a9bb6b76ddc7292544486dc3 all runs: OK false negative chance: 0.000 # git bisect bad 51a24b7deaae5c3561965f5b4b27bb9d686add1c Bisecting: 229 revisions left to test after this (roughly 8 steps) [39879e3a41061e2fc8313d55bcdbed6f458ae75d] Merge tag 'loongarch-fixes-6.17-2' of git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 39879e3a41061e2fc8313d55bcdbed6f458ae75d gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 35aca151a2bc8fb848bcc12173a4c894e3608fa2e9dd60f7f336257b709e446a all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good 39879e3a41061e2fc8313d55bcdbed6f458ae75d Bisecting: 114 revisions left to test after this (roughly 7 steps) [bf40f4b87761e2ec16efc8e49b9ca0d81f4115d8] Merge tag 'probes-fixes-v6.17-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit bf40f4b87761e2ec16efc8e49b9ca0d81f4115d8 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 78335757bb79d263b931798739230aebc8c5abf86bd8743906996bad2868ff16 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good bf40f4b87761e2ec16efc8e49b9ca0d81f4115d8 Bisecting: 55 revisions left to test after this (roughly 6 steps) [4ff71af020ae59ae2d83b174646fc2ad9fcd4dc4] Merge tag 'net-6.17-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 4ff71af020ae59ae2d83b174646fc2ad9fcd4dc4 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: f420f4a4647efadd8dd4d0c09404ec7c9b190c05eeb0f764701e29c54ba49f1e all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good 4ff71af020ae59ae2d83b174646fc2ad9fcd4dc4 Bisecting: 25 revisions left to test after this (roughly 5 steps) [bb97142197df73fbbb0e6f8629dc1f89ef6960f7] Merge tag 'platform-drivers-x86-v6.17-5' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86 determine whether the revision contains the guilty commit revision bf40f4b87761e2ec16efc8e49b9ca0d81f4115d8 crashed and is reachable testing commit bb97142197df73fbbb0e6f8629dc1f89ef6960f7 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: cd4bcf471909fcc553bc1a291df2a13527a17075444e7e52220da1f51b6917b8 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good bb97142197df73fbbb0e6f8629dc1f89ef6960f7 Bisecting: 11 revisions left to test after this (roughly 4 steps) [2cea0ed9796381b142f46bd8de97bb6b54b1df61] Merge tag 'locking-urgent-2025-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 2cea0ed9796381b142f46bd8de97bb6b54b1df61 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 52f98ab3effddd2b9b0bdf5a52a9b27a1f22c414855aec010ebc074a0f0a6e08 all runs: OK false negative chance: 0.000 # git bisect bad 2cea0ed9796381b142f46bd8de97bb6b54b1df61 Bisecting: 7 revisions left to test after this (roughly 3 steps) [d8743676b12addb982f5d501e9f8def042ef9bdb] Merge tag 'vfs-6.17-rc8.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit d8743676b12addb982f5d501e9f8def042ef9bdb gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: c04e076921f85e7dc92a72fea3e9b3c19255f28ae7958872c8a1b30425418a17 run #0: ignore: lost connection to test machine run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK false negative chance: 0.000 # git bisect bad d8743676b12addb982f5d501e9f8def042ef9bdb Bisecting: 2 revisions left to test after this (roughly 2 steps) [4d428dca252c858bfac691c31fa95d26cd008706] netfs: fix reference leak determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 4d428dca252c858bfac691c31fa95d26cd008706 gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 966ed6920078d6dfcfe0846e96aba340e58791ae17a5666df11a7c1d211afe4e all runs: OK false negative chance: 0.000 # git bisect bad 4d428dca252c858bfac691c31fa95d26cd008706 Bisecting: 0 revisions left to test after this (roughly 1 step) [9158c6bb245113d4966df9b2ba602197a379412e] afs: Fix potential null pointer dereference in afs_put_server determine whether the revision contains the guilty commit revision 4871b7cb27f480f6ecce804f81d4b9ee27281dd2 crashed and is reachable testing commit 9158c6bb245113d4966df9b2ba602197a379412e gcc compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 kernel signature: 129d09b916d3b9654dced9957697189716e99d884b23a221bcaef23857eb5031 all runs: crashed: INFO: task hung in v9fs_evict_inode representative crash: INFO: task hung in v9fs_evict_inode, types: [HANG] # git bisect good 9158c6bb245113d4966df9b2ba602197a379412e 4d428dca252c858bfac691c31fa95d26cd008706 is the first bad commit commit 4d428dca252c858bfac691c31fa95d26cd008706 Author: Max Kellermann Date: Thu Sep 25 14:08:20 2025 +0100 netfs: fix reference leak Commit 20d72b00ca81 ("netfs: Fix the request's work item to not require a ref") modified netfs_alloc_request() to initialize the reference counter to 2 instead of 1. The rationale was that the requet's "work" would release the second reference after completion (via netfs_{read,write}_collection_worker()). That works most of the time if all goes well. However, it leaks this additional reference if the request is released before the I/O operation has been submitted: the error code path only decrements the reference counter once and the work item will never be queued because there will never be a completion. This has caused outages of our whole server cluster today because tasks were blocked in netfs_wait_for_outstanding_io(), leading to deadlocks in Ceph (another bug that I will address soon in another patch). This was caused by a netfs_pgpriv2_begin_copy_to_cache() call which failed in fscache_begin_write_operation(). The leaked netfs_io_request was never completed, leaving `netfs_inode.io_count` with a positive value forever. All of this is super-fragile code. Finding out which code paths will lead to an eventual completion and which do not is hard to see: - Some functions like netfs_create_write_req() allocate a request, but will never submit any I/O. - netfs_unbuffered_read_iter_locked() calls netfs_unbuffered_read() and then netfs_put_request(); however, netfs_unbuffered_read() can also fail early before submitting the I/O request, therefore another netfs_put_request() call must be added there. A rule of thumb is that functions that return a `netfs_io_request` do not submit I/O, and all of their callers must be checked. For my taste, the whole netfs code needs an overhaul to make reference counting easier to understand and less fragile & obscure. But to fix this bug here and now and produce a patch that is adequate for a stable backport, I tried a minimal approach that quickly frees the request object upon early failure. I decided against adding a second netfs_put_request() each time because that would cause code duplication which obscures the code further. Instead, I added the function netfs_put_failed_request() which frees such a failed request synchronously under the assumption that the reference count is exactly 2 (as initially set by netfs_alloc_request() and never touched), verified by a WARN_ON_ONCE(). It then deinitializes the request object (without going through the "cleanup_work" indirection) and frees the allocation (with RCU protection to protect against concurrent access by netfs_requests_seq_start()). All code paths that fail early have been changed to call netfs_put_failed_request() instead of netfs_put_request(). Additionally, I have added a netfs_put_request() call to netfs_unbuffered_read() as explained above because the netfs_put_failed_request() approach does not work there. Fixes: 20d72b00ca81 ("netfs: Fix the request's work item to not require a ref") Signed-off-by: Max Kellermann Signed-off-by: David Howells cc: Paulo Alcantara cc: netfs@lists.linux.dev cc: linux-fsdevel@vger.kernel.org cc: stable@vger.kernel.org Signed-off-by: Christian Brauner fs/netfs/buffered_read.c | 10 +++++----- fs/netfs/direct_read.c | 7 ++++++- fs/netfs/direct_write.c | 6 +++++- fs/netfs/internal.h | 1 + fs/netfs/objects.c | 30 +++++++++++++++++++++++++++--- fs/netfs/read_pgpriv2.c | 2 +- fs/netfs/read_single.c | 2 +- fs/netfs/write_issue.c | 3 +-- 8 files changed, 47 insertions(+), 14 deletions(-) accumulated error probability: 0.00 culprit signature: 966ed6920078d6dfcfe0846e96aba340e58791ae17a5666df11a7c1d211afe4e parent signature: 129d09b916d3b9654dced9957697189716e99d884b23a221bcaef23857eb5031 revisions tested: 22, total time: 11h3m14.601983382s (build: 6h44m34.909861389s, test: 3h19m19.193266039s) first good commit: 4d428dca252c858bfac691c31fa95d26cd008706 netfs: fix reference leak recipients (to): ["brauner@kernel.org" "dhowells@redhat.com" "max.kellermann@ionos.com"] recipients (cc): []