bisecting cause commit starting from 6d21a41b7b1f46d5d5c3ddc26b55c5c4a6a826b9 building syzkaller on 7bb222f7bcce6f16c2e110f4c3270e009aaf55e7 testing commit 6d21a41b7b1f46d5d5c3ddc26b55c5c4a6a826b9 with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: BUG: Bad rss-counter state run #2: crashed: BUG: Bad rss-counter state run #3: crashed: BUG: Bad rss-counter state run #4: crashed: WARNING in __mmdrop run #5: crashed: BUG: Bad rss-counter state run #6: crashed: BUG: Bad rss-counter state run #7: crashed: BUG: Bad rss-counter state run #8: crashed: WARNING in __mmdrop run #9: crashed: BUG: Bad rss-counter state testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 all runs: OK # git bisect start 6d21a41b7b1f46d5d5c3ddc26b55c5c4a6a826b9 v5.2 Bisecting: 7135 revisions left to test after this (roughly 13 steps) [ef8f3d48afd6a17a0dae8c277c2f539c2f19fd16] Merge branch 'akpm' (patches from Andrew) testing commit ef8f3d48afd6a17a0dae8c277c2f539c2f19fd16 with gcc (GCC) 8.1.0 all runs: OK # git bisect good ef8f3d48afd6a17a0dae8c277c2f539c2f19fd16 Bisecting: 3567 revisions left to test after this (roughly 12 steps) [c633324e311243586675e732249339685e5d6faa] mm/cma.c: fail if fixed declaration can't be honored testing commit c633324e311243586675e732249339685e5d6faa with gcc (GCC) 8.1.0 all runs: OK # git bisect good c633324e311243586675e732249339685e5d6faa Bisecting: 1689 revisions left to test after this (roughly 11 steps) [e2b9da9d078c014cc59e01b4109b746e70e57b5e] Merge remote-tracking branch 'arm-soc/for-next' testing commit e2b9da9d078c014cc59e01b4109b746e70e57b5e with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: BUG: Bad rss-counter state run #2: crashed: BUG: Bad rss-counter state run #3: crashed: BUG: Bad rss-counter state run #4: crashed: BUG: Bad rss-counter state run #5: crashed: WARNING in __mmdrop run #6: crashed: WARNING in __mmdrop run #7: crashed: WARNING in __mmdrop run #8: crashed: WARNING in __mmdrop run #9: crashed: WARNING in __mmdrop # git bisect bad e2b9da9d078c014cc59e01b4109b746e70e57b5e Bisecting: 948 revisions left to test after this (roughly 10 steps) [879d1d31cdf06b980cfecbd1927628dbf03b0e55] ARM: Document merges testing commit 879d1d31cdf06b980cfecbd1927628dbf03b0e55 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 879d1d31cdf06b980cfecbd1927628dbf03b0e55 Bisecting: 484 revisions left to test after this (roughly 9 steps) [edafb6fe42cfa98f4abf8c63acc5f4db011ed7b9] Merge tag 'rtc-5.3' of git://git.kernel.org/pub/scm/linux/kernel/git/abelloni/linux testing commit edafb6fe42cfa98f4abf8c63acc5f4db011ed7b9 with gcc (GCC) 8.1.0 all runs: OK # git bisect good edafb6fe42cfa98f4abf8c63acc5f4db011ed7b9 Bisecting: 242 revisions left to test after this (roughly 8 steps) [fdcec00405fae0befdd7bbcbe738b7325e5746fb] Merge tag 'rproc-v5.3' of git://github.com/andersson/remoteproc testing commit fdcec00405fae0befdd7bbcbe738b7325e5746fb with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: WARNING in __mmdrop run #2: crashed: WARNING in __mmdrop run #3: crashed: WARNING in __mmdrop run #4: crashed: BUG: Bad rss-counter state run #5: crashed: WARNING in __mmdrop run #6: crashed: WARNING in __mmdrop run #7: crashed: BUG: Bad rss-counter state run #8: crashed: BUG: Bad rss-counter state run #9: crashed: BUG: Bad rss-counter state # git bisect bad fdcec00405fae0befdd7bbcbe738b7325e5746fb Bisecting: 104 revisions left to test after this (roughly 7 steps) [e02cb1f5934053acfa72b7def8bef9b20e7ab818] Merge branches 'clk-ti', 'clk-samsung', 'clk-imx' and 'clk-allwinner' into clk-next testing commit e02cb1f5934053acfa72b7def8bef9b20e7ab818 with gcc (GCC) 8.1.0 all runs: OK # git bisect good e02cb1f5934053acfa72b7def8bef9b20e7ab818 Bisecting: 60 revisions left to test after this (roughly 6 steps) [47c9e0cef01494aa512e924b100160206295f45e] Merge branches 'clk-rpi-cpufreq', 'clk-tegra', 'clk-simplify-provider.h', 'clk-sprd' and 'clk-at91' into clk-next testing commit 47c9e0cef01494aa512e924b100160206295f45e with gcc (GCC) 8.1.0 all runs: OK # git bisect good 47c9e0cef01494aa512e924b100160206295f45e Bisecting: 35 revisions left to test after this (roughly 5 steps) [37d4607ebbbf5d8b74cbcb9434a5ce6897a51864] Merge tag 'vfio-v5.3-rc1' of git://github.com/awilliam/linux-vfio testing commit 37d4607ebbbf5d8b74cbcb9434a5ce6897a51864 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 37d4607ebbbf5d8b74cbcb9434a5ce6897a51864 Bisecting: 18 revisions left to test after this (roughly 4 steps) [3a1d5384b7decbff6519daa9c65a35665e227323] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost testing commit 3a1d5384b7decbff6519daa9c65a35665e227323 with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: BUG: Bad rss-counter state run #2: crashed: BUG: Bad rss-counter state run #3: crashed: BUG: Bad rss-counter state run #4: crashed: BUG: Bad rss-counter state run #5: crashed: WARNING in __mmdrop run #6: crashed: WARNING in __mmdrop run #7: crashed: WARNING in __mmdrop run #8: crashed: WARNING in __mmdrop run #9: crashed: WARNING in __mmdrop # git bisect bad 3a1d5384b7decbff6519daa9c65a35665e227323 Bisecting: 8 revisions left to test after this (roughly 3 steps) [8447d84e35f2c3b2e76f80319d0b547a9521bafa] dt-bindings: virtio-mmio: Add IOMMU description testing commit 8447d84e35f2c3b2e76f80319d0b547a9521bafa with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: BUG: Bad rss-counter state run #2: crashed: BUG: Bad rss-counter state run #3: crashed: BUG: Bad rss-counter state run #4: crashed: BUG: Bad rss-counter state run #5: crashed: BUG: Bad rss-counter state run #6: crashed: BUG: Bad rss-counter state run #7: crashed: WARNING in __mmdrop run #8: crashed: BUG: Bad rss-counter state run #9: crashed: WARNING in __mmdrop # git bisect bad 8447d84e35f2c3b2e76f80319d0b547a9521bafa Bisecting: 3 revisions left to test after this (roughly 2 steps) [4942e8254d93b31ea14cf63a9dfd10a8112caafa] vhost: introduce helpers to get the size of metadata area testing commit 4942e8254d93b31ea14cf63a9dfd10a8112caafa with gcc (GCC) 8.1.0 all runs: OK # git bisect good 4942e8254d93b31ea14cf63a9dfd10a8112caafa Bisecting: 1 revision left to test after this (roughly 1 step) [7f466032dc9e5a61217f22ea34b2df932786bbfc] vhost: access vq metadata through kernel virtual address testing commit 7f466032dc9e5a61217f22ea34b2df932786bbfc with gcc (GCC) 8.1.0 run #0: crashed: BUG: Bad rss-counter state run #1: crashed: WARNING in __mmdrop run #2: crashed: WARNING in __mmdrop run #3: crashed: WARNING in __mmdrop run #4: crashed: BUG: Bad rss-counter state run #5: crashed: BUG: Bad rss-counter state run #6: crashed: BUG: Bad rss-counter state run #7: crashed: WARNING in __mmdrop run #8: crashed: BUG: Bad rss-counter state run #9: crashed: BUG: Bad rss-counter state # git bisect bad 7f466032dc9e5a61217f22ea34b2df932786bbfc Bisecting: 0 revisions left to test after this (roughly 0 steps) [feebcaeac79ad86fb289ef55fa92f4a97ab8314e] vhost: factor out setting vring addr and num testing commit feebcaeac79ad86fb289ef55fa92f4a97ab8314e with gcc (GCC) 8.1.0 all runs: OK # git bisect good feebcaeac79ad86fb289ef55fa92f4a97ab8314e 7f466032dc9e5a61217f22ea34b2df932786bbfc is the first bad commit commit 7f466032dc9e5a61217f22ea34b2df932786bbfc Author: Jason Wang Date: Fri May 24 04:12:18 2019 -0400 vhost: access vq metadata through kernel virtual address It was noticed that the copy_to/from_user() friends that was used to access virtqueue metdata tends to be very expensive for dataplane implementation like vhost since it involves lots of software checks, speculation barriers, hardware feature toggling (e.g SMAP). The extra cost will be more obvious when transferring small packets since the time spent on metadata accessing become more significant. This patch tries to eliminate those overheads by accessing them through direct mapping of those pages. Invalidation callbacks is implemented for co-operation with general VM management (swap, KSM, THP or NUMA balancing). We will try to get the direct mapping of vq metadata before each round of packet processing if it doesn't exist. If we fail, we will simplely fallback to copy_to/from_user() friends. This invalidation and direct mapping access are synchronized through spinlock and RCU. All matedata accessing through direct map is protected by RCU, and the setup or invalidation are done under spinlock. This method might does not work for high mem page which requires temporary mapping so we just fallback to normal copy_to/from_user() and may not for arch that has virtual tagged cache since extra cache flushing is needed to eliminate the alias. This will result complex logic and bad performance. For those archs, this patch simply go for copy_to/from_user() friends. This is done by ruling out kernel mapping codes through ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE. Note that this is only done when device IOTLB is not enabled. We could use similar method to optimize IOTLB in the future. Tests shows at most about 23% improvement on TX PPS when using virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell: SMAP on | SMAP off Before: 5.2Mpps | 7.1Mpps After: 6.4Mpps | 8.2Mpps Cc: Andrea Arcangeli Cc: James Bottomley Cc: Christoph Hellwig Cc: David Miller Cc: Jerome Glisse Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-parisc@vger.kernel.org Signed-off-by: Jason Wang Signed-off-by: Michael S. Tsirkin :040000 040000 c368bf7940686a2134ad239d743eb0f5846c15cf 26c5227d261b8ee18909d17eb2dd9631fe1c5b2b M drivers revisions tested: 16, total time: 3h47m13.001701913s (build: 1h33m5.265457212s, test: 2h8m40.552183076s) first bad commit: 7f466032dc9e5a61217f22ea34b2df932786bbfc vhost: access vq metadata through kernel virtual address cc: ["aarcange@redhat.com" "davem@davemloft.net" "hch@infradead.org" "james.bottomley@hansenpartnership.com" "jasowang@redhat.com" "jglisse@redhat.com" "linux-arm-kernel@lists.infradead.org" "linux-mm@kvack.org" "linux-parisc@vger.kernel.org" "mst@redhat.com"] crash: BUG: Bad rss-counter state BUG: Bad rss-counter state mm:000000001b47cd5c idx:0 val:241 BUG: Bad rss-counter state mm:000000001b47cd5c idx:1 val:544 BUG: non-zero pgtables_bytes on freeing mm: 73728