bisecting cause commit starting from ef78e5ec9214376c5cb989f5da70b02d0c117b66 building syzkaller on 4b6d14f266bcae8f1856f987c2194f36eadef83b testing commit ef78e5ec9214376c5cb989f5da70b02d0c117b66 with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.19 testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.18 testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.17 testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.16 testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.15 testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0 all runs: crashed: INFO: task hung in vhost_net_stop_vq testing release v4.14 testing commit bebc6082da0a9f5d47a1ea2edc099bf671058bd4 with gcc (GCC) 8.1.0 all runs: crashed: possible deadlock in vhost_chr_write_iter testing release v4.13 testing commit 569dbb88e80deb68974ef6fdd6a13edb9d686261 with gcc (GCC) 8.1.0 all runs: crashed: possible deadlock in vhost_chr_write_iter testing release v4.12 testing commit 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c with gcc (GCC) 8.1.0 all runs: crashed: possible deadlock in vhost_chr_write_iter testing release v4.11 testing commit a351e9b9fc24e982ec2f0e76379a49826036da12 with gcc (GCC) 7.3.0 all runs: crashed: possible deadlock in vhost_chr_write_iter testing release v4.10 testing commit c470abd4fde40ea6a0846a2beab642a578c0b8cd with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_chr_write_iter testing release v4.9 testing commit 69973b830859bc6529a7a0468ba0d80ee5117826 with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg testing release v4.8 testing commit c8d2bc9bc39ebea8437fd974fdbc21847bb897a3 with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg testing release v4.7 testing commit 523d939ef98fd712632d93a5a2b588e477a7565e with gcc (GCC) 5.5.0 all runs: OK # git bisect start v4.8 v4.7 Bisecting: 7344 revisions left to test after this (roughly 13 steps) [e61c10e468a42512f5fad74c00b62af5cc19f65f] sh: add device tree source for J2 FPGA on Mimas v2 board testing commit e61c10e468a42512f5fad74c00b62af5cc19f65f with gcc (GCC) 5.5.0 all runs: OK # git bisect good e61c10e468a42512f5fad74c00b62af5cc19f65f Bisecting: 3672 revisions left to test after this (roughly 12 steps) [b6e8d4aa1110306378af0f3472a6b85a1f039a16] rapidio: add RapidIO channelized messaging driver testing commit b6e8d4aa1110306378af0f3472a6b85a1f039a16 with gcc (GCC) 5.5.0 all runs: OK # git bisect good b6e8d4aa1110306378af0f3472a6b85a1f039a16 Bisecting: 1836 revisions left to test after this (roughly 11 steps) [694d0d0bb2030d2e36df73e2d23d5770511dbc8d] Linux 4.8-rc2 testing commit 694d0d0bb2030d2e36df73e2d23d5770511dbc8d with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad 694d0d0bb2030d2e36df73e2d23d5770511dbc8d Bisecting: 895 revisions left to test after this (roughly 10 steps) [0cda611386b2fcbf8bb32e9a5d82bfed4856fc36] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma testing commit 0cda611386b2fcbf8bb32e9a5d82bfed4856fc36 with gcc (GCC) 5.5.0 all runs: OK # git bisect good 0cda611386b2fcbf8bb32e9a5d82bfed4856fc36 Bisecting: 433 revisions left to test after this (roughly 9 steps) [4305f42401b29e2e024bd064618faf25aef5cb69] Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus testing commit 4305f42401b29e2e024bd064618faf25aef5cb69 with gcc (GCC) 5.5.0 all runs: OK # git bisect good 4305f42401b29e2e024bd064618faf25aef5cb69 Bisecting: 218 revisions left to test after this (roughly 8 steps) [d52c0569bab4edc888832df44dc7ac28517134f6] x86/apic/x2apic, smp/hotplug: Don't use before alloc in x2apic_cluster_probe() testing commit d52c0569bab4edc888832df44dc7ac28517134f6 with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad d52c0569bab4edc888832df44dc7ac28517134f6 Bisecting: 107 revisions left to test after this (roughly 7 steps) [0cbbc422d56668528f6efd1234fe908010284082] Merge tag 'xfs-rmap-for-linus-4.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs testing commit 0cbbc422d56668528f6efd1234fe908010284082 with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad 0cbbc422d56668528f6efd1234fe908010284082 Bisecting: 52 revisions left to test after this (roughly 6 steps) [835c92d43b29eb354abdbd5475308a474d7efdfa] Merge branch 'work.const-qstr' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs testing commit 835c92d43b29eb354abdbd5475308a474d7efdfa with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad 835c92d43b29eb354abdbd5475308a474d7efdfa Bisecting: 34 revisions left to test after this (roughly 5 steps) [b226acab2f6aaa45c2af27279b63f622b23a44bd] VSOCK: Use kvfree() testing commit b226acab2f6aaa45c2af27279b63f622b23a44bd with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad b226acab2f6aaa45c2af27279b63f622b23a44bd Bisecting: 9 revisions left to test after this (roughly 3 steps) [06a8fc78367d070720af960dcecec917d3ae5f3b] VSOCK: Introduce virtio_vsock_common.ko testing commit 06a8fc78367d070720af960dcecec917d3ae5f3b with gcc (GCC) 5.5.0 all runs: OK # git bisect good 06a8fc78367d070720af960dcecec917d3ae5f3b Bisecting: 4 revisions left to test after this (roughly 2 steps) [a9709d6874d55130663567577a9b05c35138cc6b] vhost: convert pre sorted vhost memory array to interval tree testing commit a9709d6874d55130663567577a9b05c35138cc6b with gcc (GCC) 5.5.0 all runs: OK # git bisect good a9709d6874d55130663567577a9b05c35138cc6b Bisecting: 2 revisions left to test after this (roughly 1 step) [6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c] vhost: new device IOTLB API testing commit 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in vhost_process_iotlb_msg # git bisect bad 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c Bisecting: 0 revisions left to test after this (roughly 0 steps) [b2fbd8b0737803f527bc7671a14bd6736d78b5d7] vhost: drop vringh dependency testing commit b2fbd8b0737803f527bc7671a14bd6736d78b5d7 with gcc (GCC) 5.5.0 all runs: OK # git bisect good b2fbd8b0737803f527bc7671a14bd6736d78b5d7 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c is the first bad commit commit 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c Author: Jason Wang Date: Thu Jun 23 02:04:32 2016 -0400 vhost: new device IOTLB API This patch tries to implement an device IOTLB for vhost. This could be used with userspace(qemu) implementation of DMA remapping to emulate an IOMMU for the guest. The idea is simple, cache the translation in a software device IOTLB (which is implemented as an interval tree) in vhost and use vhost_net file descriptor for reporting IOTLB miss and IOTLB update/invalidation. When vhost meets an IOTLB miss, the fault address, size and access can be read from the file. After userspace finishes the translation, it writes the translated address to the vhost_net file to update the device IOTLB. When device IOTLB is enabled by setting VIRTIO_F_IOMMU_PLATFORM all vq addresses set by ioctl are treated as iova instead of virtual address and the accessing can only be done through IOTLB instead of direct userspace memory access. Before each round or vq processing, all vq metadata is prefetched in device IOTLB to make sure no translation fault happens during vq processing. In most cases, virtqueues are contiguous even in virtual address space. The IOTLB translation for virtqueue itself may make it a little slower. We might add fast path cache on top of this patch. Signed-off-by: Jason Wang [mst: use virtio feature bit: VHOST_F_DEVICE_IOTLB -> VIRTIO_F_IOMMU_PLATFORM ] [mst: fix build warnings ] Signed-off-by: Michael S. Tsirkin [ weiyj.lk: missing unlock on error ] Signed-off-by: Wei Yongjun drivers/vhost/net.c | 59 ++++- drivers/vhost/vhost.c | 636 ++++++++++++++++++++++++++++++++++++++++++--- drivers/vhost/vhost.h | 32 ++- include/uapi/linux/vhost.h | 28 ++ 4 files changed, 705 insertions(+), 50 deletions(-) revisions tested: 27, total time: 4h32m53.324711886s (build: 1h45m10.482901797s, test: 2h42m14.374645875s) first bad commit: 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c vhost: new device IOTLB API cc: ["jasowang@redhat.com" "kvm@vger.kernel.org" "linux-kernel@vger.kernel.org" "mst@redhat.com" "netdev@vger.kernel.org" "virtualization@lists.linux-foundation.org" "weiyj.lk@gmail.com"] crash: possible deadlock in vhost_process_iotlb_msg IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready 8021q: adding VLAN 0 to HW filter on device team0 IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready 8021q: adding VLAN 0 to HW filter on device team0 ============================================= [ INFO: possible recursive locking detected ] 4.7.0+ #1 Not tainted --------------------------------------------- syz-executor1/6823 is trying to acquire lock: (&vq->mutex){+.+...}, at: [] vhost_dev_lock_vqs drivers/vhost/vhost.c:844 [inline] (&vq->mutex){+.+...}, at: [] vhost_process_iotlb_msg+0xe0/0x9e0 drivers/vhost/vhost.c:930 but task is already holding lock: (&vq->mutex){+.+...}, at: [] vhost_dev_lock_vqs drivers/vhost/vhost.c:844 [inline] (&vq->mutex){+.+...}, at: [] vhost_process_iotlb_msg+0xe0/0x9e0 drivers/vhost/vhost.c:930 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&vq->mutex); lock(&vq->mutex); *** DEADLOCK *** May be due to missing lock nesting notation 1 lock held by syz-executor1/6823: #0: (&vq->mutex){+.+...}, at: [] vhost_dev_lock_vqs drivers/vhost/vhost.c:844 [inline] #0: (&vq->mutex){+.+...}, at: [] vhost_process_iotlb_msg+0xe0/0x9e0 drivers/vhost/vhost.c:930 stack backtrace: CPU: 0 PID: 6823 Comm: syz-executor1 Not tainted 4.7.0+ #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 1ffffffff0d55e4a ffff880064f2f830 ffffffff829c1956 ffffffff87e092b0 ffff880065b68830 ffffffff87e092b0 0000000000a90548 ffff880064f2f9e0 ffffffff81442660 ffffffff858265d0 ffff880065b68000 ffff880064f2f8a8 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xe6/0x120 lib/dump_stack.c:51 [] print_deadlock_bug kernel/locking/lockdep.c:1723 [inline] [] check_deadlock kernel/locking/lockdep.c:1767 [inline] [] validate_chain kernel/locking/lockdep.c:2245 [inline] [] __lock_acquire+0x710/0x3cf0 kernel/locking/lockdep.c:3330 [] lock_acquire+0x196/0x480 kernel/locking/lockdep.c:3741 [] __mutex_lock_common kernel/locking/mutex.c:521 [inline] [] mutex_lock_nested+0xa8/0xb00 kernel/locking/mutex.c:621 [] vhost_dev_lock_vqs drivers/vhost/vhost.c:844 [inline] [] vhost_process_iotlb_msg+0xe0/0x9e0 drivers/vhost/vhost.c:930 [] vhost_chr_write_iter+0xe2/0x110 drivers/vhost/vhost.c:977 [] vhost_net_chr_write_iter+0x50/0x80 drivers/vhost/net.c:1202 [] new_sync_write fs/read_write.c:499 [inline] [] __vfs_write+0x303/0x740 fs/read_write.c:512 [] vfs_write+0x147/0x4a0 fs/read_write.c:560 [] SYSC_write fs/read_write.c:607 [inline] [] SyS_write+0xcb/0x1a0 fs/read_write.c:599 [] entry_SYSCALL_64_fastpath+0x23/0xc1 kobject: 'loop2' (ffff8800688cf3a0): kobject_uevent_env kobject: 'loop2' (ffff8800688cf3a0): fill_kobj_path: path = '/devices/virtual/block/loop2' kobject: 'loop3' (ffff880069fab3e0): kobject_uevent_env kobject: 'loop3' (ffff880069fab3e0): fill_kobj_path: path = '/devices/virtual/block/loop3' kobject: 'loop4' (ffff88006a0bd420): kobject_uevent_env kobject: 'loop4' (ffff88006a0bd420): fill_kobj_path: path = '/devices/virtual/block/loop4' kobject: 'loop0' (ffff880069325320): kobject_uevent_env kobject: 'loop0' (ffff880069325320): fill_kobj_path: path = '/devices/virtual/block/loop0' kobject: 'loop5' (ffff880069b53460): kobject_uevent_env kobject: 'loop5' (ffff880069b53460): fill_kobj_path: path = '/devices/virtual/block/loop5'