bisecting cause commit starting from f66ed1ebbfde37631fba289f7c399eaa70632abf building syzkaller on 5457883a514281287bbd81364c4e26e25828563d testing commit f66ed1ebbfde37631fba289f7c399eaa70632abf with gcc (GCC) 8.1.0 kernel signature: c2ea45160c5b7dc3639d97e3e48a4423ebe3f379377de1dc4bbd92560de75a14 run #0: crashed: KASAN: use-after-free Read in rpc_net_ns run #1: crashed: KASAN: use-after-free Read in rpc_net_ns run #2: crashed: KASAN: use-after-free Read in rpc_net_ns run #3: crashed: KASAN: use-after-free Read in rpc_net_ns run #4: crashed: KASAN: use-after-free Read in rpc_net_ns run #5: crashed: KASAN: use-after-free Read in rpc_net_ns run #6: crashed: KASAN: use-after-free Read in rpc_net_ns run #7: crashed: KASAN: use-after-free Read in rpc_net_ns run #8: crashed: KASAN: use-after-free Read in rpc_net_ns run #9: OK testing release v5.6 testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0 kernel signature: f6ef16e518a0ffc3e32592cb5160581a7f390f80cdb3744b9533ad4a05e238ee all runs: OK # git bisect start f66ed1ebbfde37631fba289f7c399eaa70632abf 7111951b8d4973bda27ff663f2cf18b663d15b48 Bisecting: 6397 revisions left to test after this (roughly 13 steps) [f365ab31efacb70bed1e821f7435626e0b2528a6] Merge tag 'drm-next-2020-04-01' of git://anongit.freedesktop.org/drm/drm testing commit f365ab31efacb70bed1e821f7435626e0b2528a6 with gcc (GCC) 8.1.0 kernel signature: c530c9ba7f0d35e48edae85d9bb7103c22af5a90b513f0d2cc4638311b3efd61 all runs: OK # git bisect good f365ab31efacb70bed1e821f7435626e0b2528a6 Bisecting: 3197 revisions left to test after this (roughly 12 steps) [5364abc57993b3bf60c41923cb98a8f1a594e749] Merge tag 'arc-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc testing commit 5364abc57993b3bf60c41923cb98a8f1a594e749 with gcc (GCC) 8.1.0 kernel signature: 3f39d10f2c969057a5e74364e9b7faea324245cc3826bd5b82a5970da027c67a all runs: OK # git bisect good 5364abc57993b3bf60c41923cb98a8f1a594e749 Bisecting: 1597 revisions left to test after this (roughly 11 steps) [413a103cf6e507f6304ec42b89ed45428942c43f] Merge tag 'tag-chrome-platform-for-v5.7' of git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux testing commit 413a103cf6e507f6304ec42b89ed45428942c43f with gcc (GCC) 8.1.0 kernel signature: 00b7134fdf035216558d19b0400c671385c6b960498fb77b3e1fc607b5256ec1 all runs: OK # git bisect good 413a103cf6e507f6304ec42b89ed45428942c43f Bisecting: 798 revisions left to test after this (roughly 10 steps) [ceb1adbacb4971cd47533d667f91ed06a38d7d4a] Merge tag 'mtd/fixes-for-5.7-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux testing commit ceb1adbacb4971cd47533d667f91ed06a38d7d4a with gcc (GCC) 8.1.0 kernel signature: f3596c3792d420101c7dd623e3cafb892f6495daec61ed31f516507528295d8c all runs: OK # git bisect good ceb1adbacb4971cd47533d667f91ed06a38d7d4a Bisecting: 404 revisions left to test after this (roughly 9 steps) [5ef58e29078261ef5195c7fee74768546b863182] Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit 5ef58e29078261ef5195c7fee74768546b863182 with gcc (GCC) 8.1.0 kernel signature: 72f68955a12945e5de520d20b6bb458b7a99257f4157c5cf924991025399e2a2 all runs: OK # git bisect good 5ef58e29078261ef5195c7fee74768546b863182 Bisecting: 205 revisions left to test after this (roughly 8 steps) [c5f33785719596047b0bb9df98d39fab9d1c51da] Merge tag 'tty-5.7-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit c5f33785719596047b0bb9df98d39fab9d1c51da with gcc (GCC) 8.1.0 kernel signature: 66e490ca476b0e35fb1084f5e35a0d1c63fe623287ad68e698b6d4baab2f10fb all runs: OK # git bisect good c5f33785719596047b0bb9df98d39fab9d1c51da Bisecting: 103 revisions left to test after this (roughly 7 steps) [39e16d93424b61e0b5bd182e308a56d5f0e489d6] Merge tag 'selinux-pr-20200430' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux testing commit 39e16d93424b61e0b5bd182e308a56d5f0e489d6 with gcc (GCC) 8.1.0 kernel signature: af229ef31bd098e4f4f41da105aa2759bb5a58e0606dd42658a04902ed627543 all runs: OK # git bisect good 39e16d93424b61e0b5bd182e308a56d5f0e489d6 Bisecting: 52 revisions left to test after this (roughly 6 steps) [c536419022fe3f363ca0c84848dacb3ef3b1f031] Merge tag 'sound-5.7-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit c536419022fe3f363ca0c84848dacb3ef3b1f031 with gcc (GCC) 8.1.0 kernel signature: c3a6b4606c5b2cfdc188afe7bd41accbd18f9677d514e7525469c547a9464e10 all runs: OK # git bisect good c536419022fe3f363ca0c84848dacb3ef3b1f031 Bisecting: 27 revisions left to test after this (roughly 5 steps) [690e2aba7beb1ef06352803bea41a68a3c695015] Merge tag 'vfio-v5.7-rc4' of git://github.com/awilliam/linux-vfio testing commit 690e2aba7beb1ef06352803bea41a68a3c695015 with gcc (GCC) 8.1.0 kernel signature: 8aec308571889a6e7b5d4e4459b41abc460d28d95fa59f0e86948924e2aa8345 all runs: OK # git bisect good 690e2aba7beb1ef06352803bea41a68a3c695015 Bisecting: 14 revisions left to test after this (roughly 4 steps) [ed6889db63d24600e523ac28fbece33201906611] Merge tag 'dmaengine-fix-5.7-rc4' of git://git.infradead.org/users/vkoul/slave-dma testing commit ed6889db63d24600e523ac28fbece33201906611 with gcc (GCC) 8.1.0 kernel signature: 785dfd074b4abeacd0159452f23449f40b9feebcaa97d6efc77cbc03cdf2a84a all runs: OK # git bisect good ed6889db63d24600e523ac28fbece33201906611 Bisecting: 7 revisions left to test after this (roughly 3 steps) [dff58530c4ca8ce7ee5a74db431c6e35362cf682] NFSv4.1: fix handling of backchannel binding in BIND_CONN_TO_SESSION testing commit dff58530c4ca8ce7ee5a74db431c6e35362cf682 with gcc (GCC) 8.1.0 kernel signature: ee586d71e59e3ae45de73c3803b601237bd96cfa71e973973284f1f0f9e8b116 all runs: crashed: KASAN: use-after-free Read in rpc_net_ns # git bisect bad dff58530c4ca8ce7ee5a74db431c6e35362cf682 Bisecting: 3 revisions left to test after this (roughly 2 steps) [7648f939cb919b9d15c21fff8cd9eba908d595dc] nfs: Fix potential posix_acl refcnt leak in nfs3_set_acl testing commit 7648f939cb919b9d15c21fff8cd9eba908d595dc with gcc (GCC) 8.1.0 kernel signature: 3ec93dda0b2178444c740e9fec680301b57512e178c8f96398a95a2424a73ed4 all runs: OK # git bisect good 7648f939cb919b9d15c21fff8cd9eba908d595dc Bisecting: 1 revision left to test after this (roughly 1 step) [6e47666ef93dc9c4011407df8d2de1dd1ed39f25] NFSv4: Remove unreachable error condition due to rpc_run_task() testing commit 6e47666ef93dc9c4011407df8d2de1dd1ed39f25 with gcc (GCC) 8.1.0 kernel signature: d6678f21a9783573646414be8a29e253a206a6e1830a7c5dbe6528ac521a58dc all runs: OK # git bisect good 6e47666ef93dc9c4011407df8d2de1dd1ed39f25 Bisecting: 0 revisions left to test after this (roughly 0 steps) [7c4310ff56422ea43418305d22bbc5fe19150ec4] SUNRPC: defer slow parts of rpc_free_client() to a workqueue. testing commit 7c4310ff56422ea43418305d22bbc5fe19150ec4 with gcc (GCC) 8.1.0 kernel signature: 513041026f65f2cc8ffbfaa30235d1be1f51a120d61d66c768a5b9a71bee1062 all runs: crashed: KASAN: use-after-free Read in rpc_net_ns # git bisect bad 7c4310ff56422ea43418305d22bbc5fe19150ec4 7c4310ff56422ea43418305d22bbc5fe19150ec4 is the first bad commit commit 7c4310ff56422ea43418305d22bbc5fe19150ec4 Author: NeilBrown Date: Fri Apr 3 14:33:41 2020 +1100 SUNRPC: defer slow parts of rpc_free_client() to a workqueue. The rpciod workqueue is on the write-out path for freeing dirty memory, so it is important that it never block waiting for memory to be allocated - this can lead to a deadlock. rpc_execute() - which is often called by an rpciod work item - calls rcp_task_release_client() which can lead to rpc_free_client(). rpc_free_client() makes two calls which could potentially block wating for memory allocation. rpc_clnt_debugfs_unregister() calls into debugfs and will block while any of the debugfs files are being accessed. In particular it can block while any of the 'open' methods are being called and all of these use malloc for one thing or another. So this can deadlock if the memory allocation waits for NFS to complete some writes via rpciod. rpc_clnt_remove_pipedir() can take the inode_lock() and while it isn't obvious that memory allocations can happen while the lock it held, it is safer to assume they might and to not let rpciod call rpc_clnt_remove_pipedir(). So this patch moves these two calls (together with the final kfree() and rpciod_down()) into a work-item to be run from the system work-queue. rpciod can continue its important work, and the final stages of the free can happen whenever they happen. I have seen this deadlock on a 4.12 based kernel where debugfs used synchronize_srcu() when removing objects. synchronize_srcu() requires a workqueue and there were no free workther threads and none could be allocated. While debugsfs no longer uses SRCU, I believe the deadlock is still possible. Signed-off-by: NeilBrown Signed-off-by: Trond Myklebust include/linux/sunrpc/clnt.h | 8 +++++++- net/sunrpc/clnt.c | 21 +++++++++++++++++---- 2 files changed, 24 insertions(+), 5 deletions(-) culprit signature: 513041026f65f2cc8ffbfaa30235d1be1f51a120d61d66c768a5b9a71bee1062 parent signature: d6678f21a9783573646414be8a29e253a206a6e1830a7c5dbe6528ac521a58dc revisions tested: 16, total time: 4h9m25.270147216s (build: 1h34m38.196546725s, test: 2h33m29.945105541s) first bad commit: 7c4310ff56422ea43418305d22bbc5fe19150ec4 SUNRPC: defer slow parts of rpc_free_client() to a workqueue. cc: ["anna.schumaker@netapp.com" "bfields@fieldses.org" "chuck.lever@oracle.com" "davem@davemloft.net" "kuba@kernel.org" "linux-kernel@vger.kernel.org" "linux-nfs@vger.kernel.org" "neilb@suse.de" "netdev@vger.kernel.org" "trond.myklebust@hammerspace.com"] crash: KASAN: use-after-free Read in rpc_net_ns ================================================================== BUG: KASAN: use-after-free in rpc_net_ns+0x174/0x180 net/sunrpc/clnt.c:1506 Read of size 8 at addr ffff8880a86708d8 by task kworker/0:38/2822 CPU: 0 PID: 2822 Comm: kworker/0:38 Not tainted 5.7.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events rpc_free_client_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 print_address_description.constprop.8.cold.10+0x9/0x317 mm/kasan/report.c:382 __kasan_report.cold.11+0x35/0x4d mm/kasan/report.c:511 kasan_report+0x32/0x50 mm/kasan/common.c:625 rpc_net_ns+0x174/0x180 net/sunrpc/clnt.c:1506 rpc_clnt_remove_pipedir net/sunrpc/clnt.c:111 [inline] rpc_free_client_work+0x11/0x40 net/sunrpc/clnt.c:892 process_one_work+0x908/0x15d0 kernel/workqueue.c:2268 worker_thread+0x82/0xb50 kernel/workqueue.c:2414 kthread+0x340/0x410 kernel/kthread.c:268 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Allocated by task 24611: save_stack+0x19/0x40 mm/kasan/common.c:49 set_track mm/kasan/common.c:57 [inline] __kasan_kmalloc.constprop.17+0xc1/0xd0 mm/kasan/common.c:495 __do_kmalloc mm/slab.c:3656 [inline] __kmalloc+0x164/0x7b0 mm/slab.c:3665 kmalloc include/linux/slab.h:560 [inline] kzalloc include/linux/slab.h:669 [inline] xprt_alloc+0x26/0x720 net/sunrpc/xprt.c:1658 xs_setup_xprt.part.12+0x45/0x2b0 net/sunrpc/xprtsock.c:2734 xs_setup_xprt net/sunrpc/xprtsock.c:2844 [inline] xs_setup_udp+0x72/0x800 net/sunrpc/xprtsock.c:2849 xprt_create_transport+0x11b/0x41f net/sunrpc/xprt.c:1905 rpc_create+0x225/0x540 net/sunrpc/clnt.c:581 nfs_create_rpc_client+0x42d/0x640 fs/nfs/client.c:535 nfs_init_client+0x4c/0xd0 fs/nfs/client.c:652 nfs_init_server+0x2d9/0xd50 fs/nfs/client.c:691 nfs_create_server+0x12f/0x5e0 fs/nfs/client.c:978 nfs_try_get_tree+0x154/0x810 fs/nfs/super.c:922 vfs_get_tree+0x7e/0x2c0 fs/super.c:1547 do_new_mount fs/namespace.c:2816 [inline] do_mount+0x10c5/0x17b0 fs/namespace.c:3141 __do_sys_mount fs/namespace.c:3350 [inline] __se_sys_mount fs/namespace.c:3327 [inline] __x64_sys_mount+0x15d/0x1b0 fs/namespace.c:3327 do_syscall_64+0xc6/0x620 arch/x86/entry/common.c:295 entry_SYSCALL_64_after_hwframe+0x49/0xb3 Freed by task 3964: save_stack+0x19/0x40 mm/kasan/common.c:49 set_track mm/kasan/common.c:57 [inline] kasan_set_free_info mm/kasan/common.c:317 [inline] __kasan_slab_free+0xf7/0x140 mm/kasan/common.c:456 __cache_free mm/slab.c:3426 [inline] kmem_cache_free_bulk+0x7f/0x280 mm/slab.c:3721 kfree_bulk include/linux/slab.h:412 [inline] kfree_rcu_work+0x17d/0x450 kernel/rcu/tree.c:2859 process_one_work+0x908/0x15d0 kernel/workqueue.c:2268 worker_thread+0x82/0xb50 kernel/workqueue.c:2414 kthread+0x340/0x410 kernel/kthread.c:268 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 The buggy address belongs to the object at ffff8880a8670000 which belongs to the cache kmalloc-4k of size 4096 The buggy address is located 2264 bytes inside of 4096-byte region [ffff8880a8670000, ffff8880a8671000) The buggy address belongs to the page: page:ffffea0002a19c00 refcount:1 mapcount:0 mapping:000000003b9d48a9 index:0x0 head:ffffea0002a19c00 order:1 compound_mapcount:0 flags: 0xfffe0000010200(slab|head) raw: 00fffe0000010200 ffffea0001e08008 ffffea0001d95108 ffff8880aa402000 raw: 0000000000000000 ffff8880a8670000 0000000100000001 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffff8880a8670780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8880a8670800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb >ffff8880a8670880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff8880a8670900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8880a8670980: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ==================================================================