bisecting fixing commit since 63849c8f410717eb2e6662f3953ff674727303e7 building syzkaller on 2e9971bbbfb4df6ba0118353163a7703f3dbd6ec testing commit 63849c8f410717eb2e6662f3953ff674727303e7 with gcc (GCC) 8.1.0 kernel signature: bcd6cb0f18458f164e8338b13764e6e39b76b37491a51d3100897ad3c98ec1a0 run #0: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #1: crashed: WARNING in percpu_ref_exit run #2: crashed: WARNING in percpu_ref_exit run #3: crashed: WARNING in percpu_ref_exit run #4: crashed: WARNING in percpu_ref_exit run #5: crashed: WARNING in percpu_ref_exit run #6: crashed: WARNING in percpu_ref_exit run #7: crashed: WARNING in percpu_ref_exit run #8: crashed: WARNING in percpu_ref_exit run #9: crashed: WARNING in percpu_ref_exit testing current HEAD eccc876724927ff3b9ff91f36f7b6b159e948f0c testing commit eccc876724927ff3b9ff91f36f7b6b159e948f0c with gcc (GCC) 8.1.0 kernel signature: dc4ffa14775a78d40f9d0a88d6df1c07d96009701afbe3709138438a39ef6fa1 all runs: OK # git bisect start eccc876724927ff3b9ff91f36f7b6b159e948f0c 63849c8f410717eb2e6662f3953ff674727303e7 Bisecting: 32723 revisions left to test after this (roughly 15 steps) [e9919e11e219eaa5e8041b7b1a196839143e9125] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input testing commit e9919e11e219eaa5e8041b7b1a196839143e9125 with gcc (GCC) 8.1.0 kernel signature: 062f5d334fc8c8046983a8e57d3ed6dd529e816dae78d39803c893c2b6f6e21c all runs: OK # git bisect bad e9919e11e219eaa5e8041b7b1a196839143e9125 Bisecting: 16362 revisions left to test after this (roughly 14 steps) [829f3b9401fe7cc3c1f3642bb2520751a42a87df] Merge tag 'pstore-v5.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux testing commit 829f3b9401fe7cc3c1f3642bb2520751a42a87df with gcc (GCC) 8.1.0 kernel signature: aa1736bc6bbfa358c4a2781d4a36bb150bc0a1706415850a455080677a26893c all runs: OK # git bisect bad 829f3b9401fe7cc3c1f3642bb2520751a42a87df Bisecting: 8001 revisions left to test after this (roughly 13 steps) [f365ab31efacb70bed1e821f7435626e0b2528a6] Merge tag 'drm-next-2020-04-01' of git://anongit.freedesktop.org/drm/drm testing commit f365ab31efacb70bed1e821f7435626e0b2528a6 with gcc (GCC) 8.1.0 kernel signature: bbe20531ce116c02e9b341830be1073d912cea942bec35de8724a828773c64bf all runs: OK # git bisect bad f365ab31efacb70bed1e821f7435626e0b2528a6 Bisecting: 4208 revisions left to test after this (roughly 12 steps) [1f944f976d7ef8a29d1ad296253d3a9387c58e62] Merge tag 'tty-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit 1f944f976d7ef8a29d1ad296253d3a9387c58e62 with gcc (GCC) 8.1.0 kernel signature: 9e8f1aa9c370eaf14e3e8557bbae771fd752d1d1e4fa81b938bcbd2f9da2c3ce all runs: OK # git bisect bad 1f944f976d7ef8a29d1ad296253d3a9387c58e62 Bisecting: 2183 revisions left to test after this (roughly 11 steps) [59838093be51ee9447f6ad05483d697b6fa0368d] Merge tag 'driver-core-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core testing commit 59838093be51ee9447f6ad05483d697b6fa0368d with gcc (GCC) 8.1.0 kernel signature: d062fb2fd7c9f38a682e21bd6b718b84565ae807b555911a7c55c15b27a350fa all runs: OK # git bisect bad 59838093be51ee9447f6ad05483d697b6fa0368d Bisecting: 983 revisions left to test after this (roughly 10 steps) [0f751396346f5cfb6d02abe1985af53717b23c3d] Merge tag 'tpmdd-next-20200316' of git://git.infradead.org/users/jjs/linux-tpmdd testing commit 0f751396346f5cfb6d02abe1985af53717b23c3d with gcc (GCC) 8.1.0 kernel signature: e21f5fdde805503915caa7e6441f1dccb610bfbe18c64919c88777eaa953599f all runs: OK # git bisect bad 0f751396346f5cfb6d02abe1985af53717b23c3d Bisecting: 490 revisions left to test after this (roughly 9 steps) [1f81c5efc020314b2db30d77efe228b7e117750d] tools/power turbostat: Fix missing SYS_LPI counter on some Chromebooks testing commit 1f81c5efc020314b2db30d77efe228b7e117750d with gcc (GCC) 8.1.0 kernel signature: 774e52a88a932c57174c810c0dda0e5061ae8b6141c1ad62b86506402bb05ae1 all runs: OK # git bisect bad 1f81c5efc020314b2db30d77efe228b7e117750d Bisecting: 268 revisions left to test after this (roughly 8 steps) [807f030b44ccbb26a346df6f6438628315d9ad98] Merge branch 'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs testing commit 807f030b44ccbb26a346df6f6438628315d9ad98 with gcc (GCC) 8.1.0 kernel signature: 2de2679278dcec2cb50955a7c491ce70186d88c9e1289a37451a9bdc256db1b6 all runs: OK # git bisect bad 807f030b44ccbb26a346df6f6438628315d9ad98 Bisecting: 108 revisions left to test after this (roughly 7 steps) [378fee2e6b12f31ab3749e0aa4ed0a63be23e822] Merge tag 'char-misc-5.6-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc testing commit 378fee2e6b12f31ab3749e0aa4ed0a63be23e822 with gcc (GCC) 8.1.0 kernel signature: 416197e64dbae96d353ffa3ba1ed2dea4a69ca04f3675adc765c28a9769073f9 all runs: OK # git bisect bad 378fee2e6b12f31ab3749e0aa4ed0a63be23e822 Bisecting: 54 revisions left to test after this (roughly 6 steps) [5dfcc13902bfb6d252b84e234bfc4cdba76c1069] Merge tag 'block-5.6-2020-03-07' of git://git.kernel.dk/linux-block testing commit 5dfcc13902bfb6d252b84e234bfc4cdba76c1069 with gcc (GCC) 8.1.0 kernel signature: e1311ce753f8e0811f517c421fd933126ea29af5a1a72e34b27504e9762f16c2 run #0: crashed: WARNING in percpu_ref_exit run #1: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #2: crashed: WARNING in percpu_ref_exit run #3: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #4: crashed: WARNING in percpu_ref_exit run #5: crashed: WARNING in percpu_ref_exit run #6: crashed: WARNING in percpu_ref_exit run #7: crashed: WARNING in percpu_ref_exit run #8: crashed: WARNING in percpu_ref_exit run #9: crashed: WARNING in percpu_ref_exit # git bisect good 5dfcc13902bfb6d252b84e234bfc4cdba76c1069 Bisecting: 18 revisions left to test after this (roughly 5 steps) [fd3f6cc9806c2f10b886f3ad78c9e192fb1bffd9] Merge tag 'usb-5.6-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb testing commit fd3f6cc9806c2f10b886f3ad78c9e192fb1bffd9 with gcc (GCC) 8.1.0 kernel signature: cbf1cc9cd2598a770e4f257197a615c4484748ba3e296baa5e4df02b95391b9f all runs: OK # git bisect bad fd3f6cc9806c2f10b886f3ad78c9e192fb1bffd9 Bisecting: 17 revisions left to test after this (roughly 4 steps) [61a09258f2e5b48ad0605131cae9a33ce4d01a9d] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma testing commit 61a09258f2e5b48ad0605131cae9a33ce4d01a9d with gcc (GCC) 8.1.0 kernel signature: 70e05fe7720ff5843fedf277fe9f1683e4efb8f2da5f7ffb0c7898d53e5988c0 all runs: OK # git bisect bad 61a09258f2e5b48ad0605131cae9a33ce4d01a9d Bisecting: 8 revisions left to test after this (roughly 3 steps) [a4e63bce1414df7ab6eb82ca9feb8494ce13e554] RDMA/odp: Ensure the mm is still alive before creating an implicit child testing commit a4e63bce1414df7ab6eb82ca9feb8494ce13e554 with gcc (GCC) 8.1.0 kernel signature: 8fa8656971afe71579d5e675b52aef7dd1e82bd25f1e819a32db76abac5b4492 run #0: crashed: WARNING in percpu_ref_exit run #1: crashed: WARNING in percpu_ref_exit run #2: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #3: crashed: WARNING in percpu_ref_exit run #4: crashed: WARNING in percpu_ref_exit run #5: crashed: WARNING in percpu_ref_exit run #6: crashed: WARNING in percpu_ref_exit run #7: crashed: WARNING in percpu_ref_exit run #8: crashed: WARNING in percpu_ref_exit run #9: crashed: WARNING in percpu_ref_exit # git bisect good a4e63bce1414df7ab6eb82ca9feb8494ce13e554 Bisecting: 4 revisions left to test after this (roughly 2 steps) [f0e20b8943509d81200cef5e30af2adfddba0f5c] io_uring: fix lockup with timeouts testing commit f0e20b8943509d81200cef5e30af2adfddba0f5c with gcc (GCC) 8.1.0 kernel signature: 569d4251d5dc56a59e0d1713576899948ec59698cb2e8ef5fddded0c67631f00 all runs: OK # git bisect bad f0e20b8943509d81200cef5e30af2adfddba0f5c Bisecting: 1 revision left to test after this (roughly 1 step) [80ad894382bf1d73eb688c29714fa10c0afcf2e7] io-wq: remove io_wq_flush and IO_WQ_WORK_INTERNAL testing commit 80ad894382bf1d73eb688c29714fa10c0afcf2e7 with gcc (GCC) 8.1.0 kernel signature: 9b2f8a015dbec2b32568a11ce991a59f22c1305ec593300def5a19a42b2c938c run #0: crashed: WARNING in percpu_ref_exit run #1: crashed: WARNING in percpu_ref_exit run #2: crashed: WARNING in percpu_ref_exit run #3: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #4: crashed: KASAN: use-after-free Read in percpu_ref_switch_to_atomic_rcu run #5: crashed: WARNING in percpu_ref_exit run #6: crashed: WARNING in percpu_ref_exit run #7: crashed: WARNING in percpu_ref_exit run #8: crashed: WARNING in percpu_ref_exit run #9: crashed: WARNING in percpu_ref_exit # git bisect good 80ad894382bf1d73eb688c29714fa10c0afcf2e7 Bisecting: 0 revisions left to test after this (roughly 0 steps) [c1e2148f8ecb26863b899d402a823dab8e26efd1] io_uring: free fixed_file_data after RCU grace period testing commit c1e2148f8ecb26863b899d402a823dab8e26efd1 with gcc (GCC) 8.1.0 kernel signature: 8f379cc1345c3e78f6c4b8380abda0b9f442c84ce26112583fed9b6c88c9c003 all runs: OK # git bisect bad c1e2148f8ecb26863b899d402a823dab8e26efd1 c1e2148f8ecb26863b899d402a823dab8e26efd1 is the first bad commit commit c1e2148f8ecb26863b899d402a823dab8e26efd1 Author: Jens Axboe Date: Wed Mar 4 07:25:50 2020 -0700 io_uring: free fixed_file_data after RCU grace period The percpu refcount protects this structure, and we can have an atomic switch in progress when exiting. This makes it unsafe to just free the struct normally, and can trigger the following KASAN warning: BUG: KASAN: use-after-free in percpu_ref_switch_to_atomic_rcu+0xfa/0x1b0 Read of size 1 at addr ffff888181a19a30 by task swapper/0/0 CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.6.0-rc4+ #5747 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014 Call Trace: dump_stack+0x76/0xa0 print_address_description.constprop.0+0x3b/0x60 ? percpu_ref_switch_to_atomic_rcu+0xfa/0x1b0 ? percpu_ref_switch_to_atomic_rcu+0xfa/0x1b0 __kasan_report.cold+0x1a/0x3d ? percpu_ref_switch_to_atomic_rcu+0xfa/0x1b0 percpu_ref_switch_to_atomic_rcu+0xfa/0x1b0 rcu_core+0x370/0x830 ? percpu_ref_exit+0x50/0x50 ? rcu_note_context_switch+0x7b0/0x7b0 ? run_rebalance_domains+0x11d/0x140 __do_softirq+0x10a/0x3e9 irq_exit+0xd5/0xe0 smp_apic_timer_interrupt+0x86/0x200 apic_timer_interrupt+0xf/0x20 RIP: 0010:default_idle+0x26/0x1f0 Fix this by punting the final exit and free of the struct to RCU, then we know that it's safe to do so. Jann suggested the approach of using a double rcu callback to achieve this. It's important that we do a nested call_rcu() callback, as otherwise the free could be ordered before the atomic switch, even if the latter was already queued. Reported-by: syzbot+e017e49c39ab484ac87a@syzkaller.appspotmail.com Suggested-by: Jann Horn Reviewed-by: Paul E. McKenney Signed-off-by: Jens Axboe fs/io_uring.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) culprit signature: 8f379cc1345c3e78f6c4b8380abda0b9f442c84ce26112583fed9b6c88c9c003 parent signature: 9b2f8a015dbec2b32568a11ce991a59f22c1305ec593300def5a19a42b2c938c revisions tested: 18, total time: 4h51m19.136393906s (build: 1h52m14.950650881s, test: 2h57m10.75328876s) first good commit: c1e2148f8ecb26863b899d402a823dab8e26efd1 io_uring: free fixed_file_data after RCU grace period recipients (to): ["axboe@kernel.dk" "axboe@kernel.dk" "io-uring@vger.kernel.org" "paulmck@kernel.org"] recipients (cc): ["linux-fsdevel@vger.kernel.org" "linux-kernel@vger.kernel.org" "viro@zeniv.linux.org.uk"]