ci starts bisection 2023-06-24 18:58:57.867191432 +0000 UTC m=+389803.528677866 bisecting cause commit starting from 8d2be868b42c08290509c60515865f4de24ea704 building syzkaller on 09ffe269727719aad37ea8145eb57fefb0097165 ensuring issue is reproducible on original commit 8d2be868b42c08290509c60515865f4de24ea704 testing commit 8d2be868b42c08290509c60515865f4de24ea704 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 04cb3128851f1e1d5bb07fa6ec8910dbd44ecb8833fc04863dd2371c28b65a88 all runs: crashed: KASAN: slab-use-after-free Read in __vhost_vq_attach_worker testing release v6.3 testing commit 457391b0380335d5e9a5babdec90ac53928b23b4 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 064149e58d044069b7ee6379de017e3c74d2cd5f50b3c78af2e9425135dd4618 all runs: OK too many neither good nor bad results, skipping this commit # git bisect start 8d2be868b42c08290509c60515865f4de24ea704 457391b0380335d5e9a5babdec90ac53928b23b4 Bisecting: 13953 revisions left to test after this (roughly 14 steps) [cc3c44c9fda264c6d401be04e95449a57c1231c6] Merge tag 'drm-fixes-2023-05-12' of git://anongit.freedesktop.org/drm/drm testing commit cc3c44c9fda264c6d401be04e95449a57c1231c6 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 3c0a128f86170163b8c84e6c1d0cccc3b5a8e3e7f67334a22937076bd77c54e6 all runs: OK too many neither good nor bad results, skipping this commit # git bisect good cc3c44c9fda264c6d401be04e95449a57c1231c6 Bisecting: 6384 revisions left to test after this (roughly 13 steps) [b79972202e5ed4ffa22f1bc6e0e185cdff505413] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git testing commit b79972202e5ed4ffa22f1bc6e0e185cdff505413 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1c0d6469144f4e61472c9b4f72daf3e6cb8ec971ba996436162be0491c9a6e0a all runs: OK too many neither good nor bad results, skipping this commit # git bisect good b79972202e5ed4ffa22f1bc6e0e185cdff505413 Bisecting: 3017 revisions left to test after this (roughly 12 steps) [cca41cc0b5485a0ec20707316c1a00082c01a2af] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git testing commit cca41cc0b5485a0ec20707316c1a00082c01a2af gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 22ee33916042c6deb2a565533fe36b8fa77b3464cb719edc88e14cab9bb0882d all runs: OK too many neither good nor bad results, skipping this commit # git bisect good cca41cc0b5485a0ec20707316c1a00082c01a2af Bisecting: 1539 revisions left to test after this (roughly 11 steps) [2671712f643bfaa8692f6fdee6d0ec5437327d2d] Merge branch 'driver-core-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git testing commit 2671712f643bfaa8692f6fdee6d0ec5437327d2d gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ce28e1a99f4cb8fceac71a52512bfb2ba005105cd73196c71eb937b26b49a48b all runs: OK too many neither good nor bad results, skipping this commit # git bisect good 2671712f643bfaa8692f6fdee6d0ec5437327d2d Bisecting: 761 revisions left to test after this (roughly 10 steps) [211b14bd2caa92e440859928549e2d3ce1042e82] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine.git testing commit 211b14bd2caa92e440859928549e2d3ce1042e82 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 911f267582694d5d55c74f0a3d002758bbf1cf82512a805b8cf16d5e04aebaf3 all runs: OK too many neither good nor bad results, skipping this commit # git bisect good 211b14bd2caa92e440859928549e2d3ce1042e82 Bisecting: 377 revisions left to test after this (roughly 9 steps) [45bd06286f78bbfbb1ff3d8d4a0d6e8bdb929c76] Merge branch 'gpio/for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux.git testing commit 45bd06286f78bbfbb1ff3d8d4a0d6e8bdb929c76 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6401cbc5649e59ee5616e4ab5d3d05362c07dd48f2deca29a35d7647fe077a80 all runs: crashed: KASAN: slab-use-after-free Read in __vhost_vq_attach_worker # git bisect bad 45bd06286f78bbfbb1ff3d8d4a0d6e8bdb929c76 Bisecting: 191 revisions left to test after this (roughly 8 steps) [5fc136f3f48ce648d65e28b966a6b3493135af3d] Merge patch series "qla2xxx klocwork fixes" testing commit 5fc136f3f48ce648d65e28b966a6b3493135af3d gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 87154a9dc8543bf08641919e2a12ff77c2ffa1c66a0e9fb78dd1af626e7d6f10 all runs: OK too many neither good nor bad results, skipping this commit # git bisect good 5fc136f3f48ce648d65e28b966a6b3493135af3d Bisecting: 68 revisions left to test after this (roughly 7 steps) [1a77932e7c920c1731306959be59e88472960e3f] Merge branch 'linux-next' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git testing commit 1a77932e7c920c1731306959be59e88472960e3f gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c26e43aa3a2feb75c4c76511fa29d37b21f2b7e7d625be797c5463a3afbb9a44 all runs: crashed: KASAN: slab-use-after-free Read in __vhost_vq_attach_worker # git bisect bad 1a77932e7c920c1731306959be59e88472960e3f Bisecting: 61 revisions left to test after this (roughly 6 steps) [5999ac55b19337c451ba5fe9a19b49d18d5a8fea] virtio_ring: introduce virtqueue_add_sg() testing commit 5999ac55b19337c451ba5fe9a19b49d18d5a8fea gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: dc8e9dd70e1f7d5fd3d6a531e403164d89e07e0357839182de48835e8e35db58 all runs: crashed: KASAN: slab-use-after-free Read in __vhost_vq_attach_worker # git bisect bad 5999ac55b19337c451ba5fe9a19b49d18d5a8fea Bisecting: 30 revisions left to test after this (roughly 5 steps) [23eebafdd79840990a8eada0db434d6e88633b1e] vDPA/ifcvf: implement new accessors for vq_state testing commit 23eebafdd79840990a8eada0db434d6e88633b1e gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d7ba9f5c623865430bed84be9b4aac3f3e46dd1c66ba012b1047ee5db4a2cbad all runs: OK too many neither good nor bad results, skipping this commit # git bisect good 23eebafdd79840990a8eada0db434d6e88633b1e Bisecting: 15 revisions left to test after this (roughly 4 steps) [a9e96e6a75ae4ea18807e4709781d96ca4e505d8] vhost: remove vhost_work_queue testing commit a9e96e6a75ae4ea18807e4709781d96ca4e505d8 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 494326161afb8da05a749d0d8e4e0447f0c7019024f664a02f1b1ba622d0c786 all runs: OK too many neither good nor bad results, skipping this commit # git bisect good a9e96e6a75ae4ea18807e4709781d96ca4e505d8 Bisecting: 7 revisions left to test after this (roughly 3 steps) [42adbb4cddbe62c85479452f6d34b767cf87b13f] virtio_ring: put mapping error check in vring_map_one_sg testing commit 42adbb4cddbe62c85479452f6d34b767cf87b13f gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4926f73e9805fee06b6b389196898aa6fb2baf1664b310df71ef330e61abe38b all runs: crashed: KASAN: slab-use-after-free Read in __vhost_vq_attach_worker # git bisect bad 42adbb4cddbe62c85479452f6d34b767cf87b13f Bisecting: 3 revisions left to test after this (roughly 2 steps) [8d56ae16b9cfff0fefb1e7b0b2b84f3d41329477] vhost_scsi: add support for worker ioctls testing commit 8d56ae16b9cfff0fefb1e7b0b2b84f3d41329477 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fbd953fc83aa50d129f8e5f70a8f877a73f624e03ee6f0318fb12bb9092b3052 all runs: crashed: KASAN: slab-use-after-free Read in vhost_dev_set_owner # git bisect bad 8d56ae16b9cfff0fefb1e7b0b2b84f3d41329477 Bisecting: 1 revision left to test after this (roughly 1 step) [fe2b89437bb346c1f2022dea1947b1e3c7ed1ddd] vhost: replace single worker pointer with xarray testing commit fe2b89437bb346c1f2022dea1947b1e3c7ed1ddd gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e2a6d0fb64e61a8d364eea52d60c6bcbb5d87c04ae7079d50c2d57cda815b3e2 all runs: OK too many neither good nor bad results, skipping this commit # git bisect good fe2b89437bb346c1f2022dea1947b1e3c7ed1ddd Bisecting: 0 revisions left to test after this (roughly 0 steps) [21a18f4a51896fde11002165f0e7340f4131d6a0] vhost: allow userspace to create workers testing commit 21a18f4a51896fde11002165f0e7340f4131d6a0 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 42013857c0ff3a0c0f9cd6b343665f541869b5198d35079ced623d114e695cf3 all runs: crashed: KASAN: slab-use-after-free Read in vhost_dev_set_owner # git bisect bad 21a18f4a51896fde11002165f0e7340f4131d6a0 21a18f4a51896fde11002165f0e7340f4131d6a0 is the first bad commit commit 21a18f4a51896fde11002165f0e7340f4131d6a0 Author: Mike Christie Date: Mon Jun 12 20:32:46 2023 -0500 vhost: allow userspace to create workers For vhost-scsi with 3 vqs or more and a workload that tries to use them in parallel like: fio --filename=/dev/sdb --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=128 --numjobs=3 the single vhost worker thread will become a bottlneck and we are stuck at around 500K IOPs no matter how many jobs, virtqueues, and CPUs are used. To better utilize virtqueues and available CPUs, this patch allows userspace to create workers and bind them to vqs. You can have N workers per dev and also share N workers with M vqs on that dev. This patch adds the interface related code and the next patch will hook vhost-scsi into it. The patches do not try to hook net and vsock into the interface because: 1. multiple workers don't seem to help vsock. The problem is that with only 2 virtqueues we never fully use the existing worker when doing bidirectional tests. This seems to match vhost-scsi where we don't see the worker as a bottleneck until 3 virtqueues are used. 2. net already has a way to use multiple workers. Signed-off-by: Mike Christie Message-Id: <20230613013248.12196-16-michael.christie@oracle.com> Signed-off-by: Michael S. Tsirkin drivers/vhost/vhost.c | 141 ++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 3 + include/uapi/linux/vhost.h | 33 +++++++++ include/uapi/linux/vhost_types.h | 16 +++++ 4 files changed, 192 insertions(+), 1 deletion(-) culprit signature: 42013857c0ff3a0c0f9cd6b343665f541869b5198d35079ced623d114e695cf3 parent signature: e2a6d0fb64e61a8d364eea52d60c6bcbb5d87c04ae7079d50c2d57cda815b3e2 revisions tested: 17, total time: 5h54m10.981940055s (build: 2h51m9.285969357s, test: 2h56m49.094398958s) first bad commit: 21a18f4a51896fde11002165f0e7340f4131d6a0 vhost: allow userspace to create workers recipients (to): ["michael.christie@oracle.com" "mst@redhat.com"] recipients (cc): [] crash: KASAN: slab-use-after-free Read in vhost_dev_set_owner ================================================================== BUG: KASAN: slab-use-after-free in __vhost_vq_attach_worker drivers/vhost/vhost.c:631 [inline] BUG: KASAN: slab-use-after-free in vhost_dev_set_owner+0x96c/0xa70 drivers/vhost/vhost.c:823 Read of size 4 at addr ffff888028657d5c by task syz-executor.3/938 CPU: 1 PID: 938 Comm: syz-executor.3 Not tainted 6.4.0-rc7-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x64/0xb0 lib/dump_stack.c:106 print_address_description.constprop.0+0x2c/0x3c0 mm/kasan/report.c:351 print_report mm/kasan/report.c:462 [inline] kasan_report+0x11c/0x130 mm/kasan/report.c:572 __vhost_vq_attach_worker drivers/vhost/vhost.c:631 [inline] vhost_dev_set_owner+0x96c/0xa70 drivers/vhost/vhost.c:823 vhost_net_set_owner drivers/vhost/net.c:1687 [inline] vhost_net_ioctl+0x885/0x12c0 drivers/vhost/net.c:1737 vfs_ioctl fs/ioctl.c:51 [inline] __do_sys_ioctl fs/ioctl.c:870 [inline] __se_sys_ioctl fs/ioctl.c:856 [inline] __x64_sys_ioctl+0x123/0x190 fs/ioctl.c:856 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f957ce8c389 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f957db20168 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00007f957cfabf80 RCX: 00007f957ce8c389 RDX: 0000000000000000 RSI: 000040000000af01 RDI: 0000000000000003 RBP: 00007f957ced7493 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fffd42f82ff R14: 00007f957db20300 R15: 0000000000022000 Allocated by task 938: kasan_save_stack+0x22/0x40 mm/kasan/common.c:45 kasan_set_track+0x25/0x30 mm/kasan/common.c:52 ____kasan_kmalloc mm/kasan/common.c:374 [inline] ____kasan_kmalloc mm/kasan/common.c:333 [inline] __kasan_kmalloc+0xa2/0xb0 mm/kasan/common.c:383 kmalloc include/linux/slab.h:559 [inline] kzalloc include/linux/slab.h:680 [inline] vhost_worker_create+0x8b/0x2b0 drivers/vhost/vhost.c:596 vhost_dev_set_owner+0x34a/0xa70 drivers/vhost/vhost.c:811 vhost_net_set_owner drivers/vhost/net.c:1687 [inline] vhost_net_ioctl+0x885/0x12c0 drivers/vhost/net.c:1737 vfs_ioctl fs/ioctl.c:51 [inline] __do_sys_ioctl fs/ioctl.c:870 [inline] __se_sys_ioctl fs/ioctl.c:856 [inline] __x64_sys_ioctl+0x123/0x190 fs/ioctl.c:856 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd Freed by task 954: kasan_save_stack+0x22/0x40 mm/kasan/common.c:45 kasan_set_track+0x25/0x30 mm/kasan/common.c:52 kasan_save_free_info+0x2e/0x40 mm/kasan/generic.c:521 ____kasan_slab_free mm/kasan/common.c:236 [inline] ____kasan_slab_free+0x160/0x1c0 mm/kasan/common.c:200 kasan_slab_free include/linux/kasan.h:162 [inline] slab_free_hook mm/slub.c:1781 [inline] slab_free_freelist_hook+0x8b/0x1c0 mm/slub.c:1807 slab_free mm/slub.c:3786 [inline] __kmem_cache_free+0xaf/0x2d0 mm/slub.c:3799 vhost_worker_destroy drivers/vhost/vhost.c:569 [inline] vhost_workers_free drivers/vhost/vhost.c:584 [inline] vhost_dev_cleanup+0x5a2/0x7b0 drivers/vhost/vhost.c:922 vhost_dev_reset_owner+0x1d/0x150 drivers/vhost/vhost.c:854 vhost_net_reset_owner drivers/vhost/net.c:1621 [inline] vhost_net_ioctl+0x91b/0x12c0 drivers/vhost/net.c:1735 vfs_ioctl fs/ioctl.c:51 [inline] __do_sys_ioctl fs/ioctl.c:870 [inline] __se_sys_ioctl fs/ioctl.c:856 [inline] __x64_sys_ioctl+0x123/0x190 fs/ioctl.c:856 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd The buggy address belongs to the object at ffff888028657d40 which belongs to the cache kmalloc-cg-32 of size 32 The buggy address is located 28 bytes inside of freed 32-byte region [ffff888028657d40, ffff888028657d60) The buggy address belongs to the physical page: page:ffffea0000a195c0 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff888028657d00 pfn:0x28657 memcg:ffff888020c47001 flags: 0xfff00000000200(slab|node=0|zone=1|lastcpupid=0x7ff) page_type: 0xffffffff() raw: 00fff00000000200 ffff88801144d8c0 ffffea0000ab0a40 dead000000000004 raw: ffff888028657d00 000000008040003e 00000001ffffffff ffff888020c47001 page dumped because: kasan: bad access detected page_owner tracks the page as allocated page last allocated via order 0, migratetype Unmovable, gfp_mask 0x112cc0(GFP_USER|__GFP_NOWARN|__GFP_NORETRY), pid 12845, tgid 12837 (syz-executor.2), ts 118702808671, free_ts 118701766031 set_page_owner include/linux/page_owner.h:31 [inline] post_alloc_hook+0x2db/0x350 mm/page_alloc.c:1731 prep_new_page mm/page_alloc.c:1738 [inline] get_page_from_freelist+0xf41/0x2c00 mm/page_alloc.c:3502 __alloc_pages+0x1cb/0x4a0 mm/page_alloc.c:4768 alloc_slab_page mm/slub.c:1851 [inline] allocate_slab+0x25f/0x390 mm/slub.c:1998 new_slab mm/slub.c:2051 [inline] ___slab_alloc+0xa91/0x1400 mm/slub.c:3192 __slab_alloc.constprop.0+0x56/0xa0 mm/slub.c:3291 __slab_alloc_node mm/slub.c:3344 [inline] slab_alloc_node mm/slub.c:3441 [inline] __kmem_cache_alloc_node+0x136/0x320 mm/slub.c:3490 kmalloc_trace+0x26/0xe0 mm/slab_common.c:1057 kmalloc include/linux/slab.h:559 [inline] kzalloc include/linux/slab.h:680 [inline] vhost_worker_create+0x8b/0x2b0 drivers/vhost/vhost.c:596 vhost_dev_set_owner+0x34a/0xa70 drivers/vhost/vhost.c:811 vhost_net_set_owner drivers/vhost/net.c:1687 [inline] vhost_net_ioctl+0x885/0x12c0 drivers/vhost/net.c:1737 vfs_ioctl fs/ioctl.c:51 [inline] __do_sys_ioctl fs/ioctl.c:870 [inline] __se_sys_ioctl fs/ioctl.c:856 [inline] __x64_sys_ioctl+0x123/0x190 fs/ioctl.c:856 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd page last free stack trace: reset_page_owner include/linux/page_owner.h:24 [inline] free_pages_prepare mm/page_alloc.c:1302 [inline] free_unref_page_prepare+0x62e/0xcb0 mm/page_alloc.c:2564 free_unref_page+0x33/0x370 mm/page_alloc.c:2659 mm_free_pgd kernel/fork.c:806 [inline] __mmdrop+0xc3/0x3f0 kernel/fork.c:924 exit_mm kernel/exit.c:567 [inline] do_exit+0x85d/0x24d0 kernel/exit.c:861 do_group_exit+0xb4/0x250 kernel/exit.c:1024 get_signal+0x1d36/0x1fa0 kernel/signal.c:2876 arch_do_signal_or_restart+0x79/0x5c0 arch/x86/kernel/signal.c:306 exit_to_user_mode_loop kernel/entry/common.c:168 [inline] exit_to_user_mode_prepare+0x11f/0x240 kernel/entry/common.c:204 __syscall_exit_to_user_mode_work kernel/entry/common.c:286 [inline] syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:297 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86 entry_SYSCALL_64_after_hwframe+0x63/0xcd Memory state around the buggy address: ffff888028657c00: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc ffff888028657c80: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc >ffff888028657d00: fb fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc ^ ffff888028657d80: 00 00 00 00 fc fc fc fc fa fb fb fb fc fc fc fc ffff888028657e00: 00 00 00 00 fc fc fc fc fa fb fb fb fc fc fc fc ==================================================================