bisecting cause commit starting from 57902dc0670c2926b0b5cc93a1e2ddf09d53e706 building syzkaller on 1eedba3600f43b5623dfc2148a0fb2f0dbe7fdce testing commit 57902dc0670c2926b0b5cc93a1e2ddf09d53e706 with gcc (GCC) 8.1.0 run #0: crashed: BUG: soft lockup in jump_label_update_timeout run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 all runs: OK # git bisect start 57902dc0670c2926b0b5cc93a1e2ddf09d53e706 v4.20 Bisecting: 6751 revisions left to test after this (roughly 13 steps) [0f4991e8fd48987ae476a92cdee6bfec4aff31b8] kernel/fork.c: mark 'stack_vm_area' with __maybe_unused testing commit 0f4991e8fd48987ae476a92cdee6bfec4aff31b8 with gcc (GCC) 8.1.0 run #0: crashed: BUG: soft lockup in kvm_vm_release run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 0f4991e8fd48987ae476a92cdee6bfec4aff31b8 Bisecting: 3287 revisions left to test after this (roughly 12 steps) [42b00f122cfbfed79fc29b0b3610f3abbb1e3864] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit 42b00f122cfbfed79fc29b0b3610f3abbb1e3864 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 42b00f122cfbfed79fc29b0b3610f3abbb1e3864 Bisecting: 1643 revisions left to test after this (roughly 11 steps) [dac9597606432574f3c64b68decaec3c56ca750f] Merge branch 'mt2712' testing commit dac9597606432574f3c64b68decaec3c56ca750f with gcc (GCC) 8.1.0 all runs: OK # git bisect good dac9597606432574f3c64b68decaec3c56ca750f Bisecting: 912 revisions left to test after this (roughly 10 steps) [7f9f852c75e7d776b078813586c76a2bc7dca993] Merge tag 'modules-for-v4.21' of git://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux testing commit 7f9f852c75e7d776b078813586c76a2bc7dca993 with gcc (GCC) 8.1.0 run #0: crashed: INFO: rcu detected stall in jump_label_update_timeout run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 7f9f852c75e7d776b078813586c76a2bc7dca993 Bisecting: 373 revisions left to test after this (roughly 9 steps) [116b081c285d89dc6ece72eeecc6aa3979e8b54e] Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 116b081c285d89dc6ece72eeecc6aa3979e8b54e with gcc (GCC) 8.1.0 run #0: crashed: BUG: soft lockup in kvm_vcpu_ioctl run #1: crashed: BUG: soft lockup in kvm_vm_ioctl run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 116b081c285d89dc6ece72eeecc6aa3979e8b54e Bisecting: 176 revisions left to test after this (roughly 8 steps) [1eefdec18eded41833401cfd64749643ff72e7da] Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 1eefdec18eded41833401cfd64749643ff72e7da with gcc (GCC) 8.1.0 run #0: crashed: BUG: soft lockup in kvm_vm_ioctl run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 1eefdec18eded41833401cfd64749643ff72e7da Bisecting: 83 revisions left to test after this (roughly 7 steps) [eaaf055f27a0eaaed0cdb0d3aa8d7fb892829ccb] Merge branches 'bug.2018.11.12a', 'consolidate.2018.12.01a', 'doc.2018.11.12a', 'fixes.2018.11.12a', 'initrd.2018.11.08b', 'sil.2018.11.12a' and 'srcu.2018.11.27a' into HEAD testing commit eaaf055f27a0eaaed0cdb0d3aa8d7fb892829ccb with gcc (GCC) 8.1.0 all runs: OK # git bisect good eaaf055f27a0eaaed0cdb0d3aa8d7fb892829ccb Bisecting: 45 revisions left to test after this (roughly 5 steps) [eed9688f8513189295887e5a27ec7f576754b60e] Merge branch 'ras-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit eed9688f8513189295887e5a27ec7f576754b60e with gcc (GCC) 8.1.0 all runs: OK # git bisect good eed9688f8513189295887e5a27ec7f576754b60e Bisecting: 26 revisions left to test after this (roughly 5 steps) [792bf4d871dea8b69be2aaabdd320d7c6ed15985] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 792bf4d871dea8b69be2aaabdd320d7c6ed15985 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 792bf4d871dea8b69be2aaabdd320d7c6ed15985 Bisecting: 13 revisions left to test after this (roughly 4 steps) [786fa29e9cb6810e21ab0d9c41a81d81d54d1d1b] locking/lockdep: Make concurrent lockdep_reset_lock() calls safe testing commit 786fa29e9cb6810e21ab0d9c41a81d81d54d1d1b with gcc (GCC) 8.1.0 all runs: OK # git bisect good 786fa29e9cb6810e21ab0d9c41a81d81d54d1d1b Bisecting: 6 revisions left to test after this (roughly 3 steps) [3db5e0ba8b8f4aee631d7ee04b7a11c56cfdc213] efi/libstub: Disable some warnings for x86{,_64} testing commit 3db5e0ba8b8f4aee631d7ee04b7a11c56cfdc213 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 3db5e0ba8b8f4aee631d7ee04b7a11c56cfdc213 Bisecting: 3 revisions left to test after this (roughly 2 steps) [1debf0958fa27b7c469dbf22754929ec59a7c0e7] x86/efi: Don't unmap EFI boot services code/data regions for EFI_OLD_MEMMAP and EFI_MIXED_MODE testing commit 1debf0958fa27b7c469dbf22754929ec59a7c0e7 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 1debf0958fa27b7c469dbf22754929ec59a7c0e7 Bisecting: 1 revision left to test after this (roughly 1 step) [80eb865768703c0f85a0603762742ae1dedf21f0] sched/fair: Clean up comment in nohz_idle_balance() testing commit 80eb865768703c0f85a0603762742ae1dedf21f0 with gcc (GCC) 8.1.0 run #0: crashed: BUG: soft lockup in kvm_vm_ioctl run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 80eb865768703c0f85a0603762742ae1dedf21f0 Bisecting: 0 revisions left to test after this (roughly 0 steps) [fe27b0de8dfcdf8482558ce5d25e697fe74d851e] locking/lockdep: Stop using RCU primitives to access 'all_lock_classes' testing commit fe27b0de8dfcdf8482558ce5d25e697fe74d851e with gcc (GCC) 8.1.0 all runs: OK # git bisect good fe27b0de8dfcdf8482558ce5d25e697fe74d851e 80eb865768703c0f85a0603762742ae1dedf21f0 is the first bad commit commit 80eb865768703c0f85a0603762742ae1dedf21f0 Author: Andrea Parri Date: Tue Nov 27 12:01:10 2018 +0100 sched/fair: Clean up comment in nohz_idle_balance() Concerning the comment associated to the atomic_fetch_andnot() in nohz_idle_balance(), Vincent explains [1]: "[...] the comment is useless and can be removed [...] it was referring to a line code above the comment that was present in a previous iteration of the patchset. This line disappeared in final version but the comment has stayed." So remove the comment. Vincent also points out that the full ordering associated to the atomic_fetch_andnot() primitive could be relaxed, but this patch insists on the current more conservative/fully ordered solution: "Performance" isn't a concern, stay away from "correctness"/subtle relaxed (re)ordering if possible..., just make sure not to confuse the next reader with misleading/out-of-date comments. [1] http://lkml.kernel.org/r/CAKfTPtBjA-oCBRkO6__npQwL3+HLjzk7riCcPU1R7YdO-EpuZg@mail.gmail.com Suggested-by: Vincent Guittot Signed-off-by: Andrea Parri Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Link: https://lkml.kernel.org/r/20181127110110.5533-1-andrea.parri@amarulasolutions.com Signed-off-by: Ingo Molnar :040000 040000 56b67bc0d1fe64d99448b18d6409e60238ecb7e7 e36f1c54942fc14badc07d9a053b399a4c79199f M kernel revisions tested: 16, total time: 4h33m48.448124818s (build: 1h32m39.790096032s, test: 2h55m49.713684178s) first bad commit: 80eb865768703c0f85a0603762742ae1dedf21f0 sched/fair: Clean up comment in nohz_idle_balance() cc: ["andrea.parri@amarulasolutions.com" "mingo@kernel.org" "peterz@infradead.org" "tglx@linutronix.de" "torvalds@linux-foundation.org"] crash: BUG: soft lockup in kvm_vm_ioctl kvm: vcpu 0: requested 34784 ns lapic timer period limited to 200000 ns kvm: vcpu 0: requested 34784 ns lapic timer period limited to 200000 ns kvm: vcpu 0: requested 34784 ns lapic timer period limited to 200000 ns kvm: vcpu 0: requested 34784 ns lapic timer period limited to 200000 ns kvm: vcpu 0: requested 34784 ns lapic timer period limited to 200000 ns watchdog: BUG: soft lockup - CPU#0 stuck for 123s! [syz-executor.4:8634] Modules linked in: irq event stamp: 54608 hardirqs last enabled at (54607): [] trace_hardirqs_on_thunk+0x1a/0x1c hardirqs last disabled at (54608): [] trace_hardirqs_off_thunk+0x1a/0x1c softirqs last enabled at (23390): [] __do_softirq+0x67f/0x993 kernel/softirq.c:319 softirqs last disabled at (23347): [] invoke_softirq kernel/softirq.c:373 [inline] softirqs last disabled at (23347): [] irq_exit+0x17f/0x1c0 kernel/softirq.c:413 CPU: 0 PID: 8634 Comm: syz-executor.4 Not tainted 4.20.0-rc6+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__read_once_size include/linux/compiler.h:182 [inline] RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline] RIP: 0010:smp_call_function_single+0x169/0x360 kernel/smp.c:302 Code: 00 45 85 ff 0f 84 85 00 00 00 44 89 e7 4c 89 f1 4c 89 ea 48 8d 74 24 40 e8 04 fb ff ff 41 89 c4 8b 44 24 58 a8 01 74 0b f3 90 <8b> 54 24 58 83 e2 01 75 f5 bf 01 00 00 00 e8 f4 2d ea ff 65 8b 15 RSP: 0018:ffff888092477440 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000003 RBX: 1ffff1101248ee8c RCX: 0000000000000830 RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000830 RBP: ffff888092477508 R08: ffffed1015d65bc9 R09: ffffed1015d65bc8 R10: ffffed1015d65bc8 R11: ffff8880aeb2de47 R12: 0000000000000000 R13: ffffffff8122d9e0 R14: 0000000000000000 R15: 0000000000000001 FS: 00007f19260e9700(0000) GS:ffff8880aea00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000625208 CR3: 00000000a4e62000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: smp_call_function_many+0x512/0x790 kernel/smp.c:434 smp_call_function+0x36/0x60 kernel/smp.c:492 on_each_cpu+0x2a/0x180 kernel/smp.c:604 text_poke_bp+0xe0/0x195 arch/x86/kernel/alternative.c:808 __jump_label_transform+0x25d/0x350 arch/x86/kernel/jump_label.c:93 arch_jump_label_transform+0x2a/0x40 arch/x86/kernel/jump_label.c:101 __jump_label_update+0xf2/0x180 kernel/jump_label.c:391 jump_label_update+0x15c/0x330 kernel/jump_label.c:754 static_key_slow_inc_cpuslocked+0x150/0x1f0 kernel/jump_label.c:131 static_key_slow_inc+0x15/0x20 kernel/jump_label.c:146 kvm_create_lapic+0x153/0x1d0 arch/x86/kvm/lapic.c:2284 kvm_arch_vcpu_init+0x5b6/0x7b0 arch/x86/kvm/x86.c:8923 kvm_vcpu_init+0x269/0x350 arch/x86/kvm/../../../virt/kvm/kvm_main.c:315 vmx_create_vcpu+0xf7/0x1a30 arch/x86/kvm/vmx.c:11530 kvm_arch_vcpu_create+0x52/0xd0 arch/x86/kvm/x86.c:8615 kvm_vm_ioctl_create_vcpu arch/x86/kvm/../../../virt/kvm/kvm_main.c:2482 [inline] kvm_vm_ioctl+0x45c/0x1680 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2985 vfs_ioctl fs/ioctl.c:46 [inline] file_ioctl fs/ioctl.c:509 [inline] do_vfs_ioctl+0x199/0x10d0 fs/ioctl.c:696 ksys_ioctl+0x62/0x90 fs/ioctl.c:713 __do_sys_ioctl fs/ioctl.c:720 [inline] __se_sys_ioctl fs/ioctl.c:718 [inline] __x64_sys_ioctl+0x6e/0xb0 fs/ioctl.c:718 do_syscall_64+0xd6/0x4e0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457f89 Code: ad b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f19260e8c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457f89 RDX: 0000000000000000 RSI: 000000000000ae41 RDI: 0000000000000004 RBP: 000000000073bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f19260e96d4 R13: 00000000004c09e5 R14: 00000000004d2758 R15: 00000000ffffffff Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 8639 Comm: syz-executor.2 Not tainted 4.20.0-rc6+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:rep_nop arch/x86/include/asm/processor.h:660 [inline] RIP: 0010:cpu_relax arch/x86/include/asm/processor.h:665 [inline] RIP: 0010:mutex_spin_on_owner+0x233/0x320 kernel/locking/mutex.c:552 Code: c3 be 08 00 00 00 4c 89 ef e8 59 d0 45 00 4c 89 e8 48 c1 e8 03 42 80 3c 30 00 0f 85 e8 00 00 00 49 8b 45 00 a8 01 75 93 f3 90 3a fe ff ff 0f 0b e8 f1 48 07 00 84 c0 0f 85 13 fe ff ff 48 c7 RSP: 0018:ffff888078e27628 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000 RBP: ffff888078e27660 R08: fffffbfff103ef31 R09: fffffbfff103ef30 R10: fffffbfff103ef30 R11: ffffffff881f7987 R12: fffffbfff1023628 R13: ffffffff881f7980 R14: dffffc0000000000 R15: ffff8880a41fa2c0 FS: 00007f73db1b1700(0000) GS:ffff8880aeb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000004cddd0 CR3: 000000007dba1000 CR4: 00000000001426e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: mutex_optimistic_spin kernel/locking/mutex.c:646 [inline] __mutex_lock_common kernel/locking/mutex.c:928 [inline] __mutex_lock+0xc64/0x11f0 kernel/locking/mutex.c:1072 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1087 jump_label_lock kernel/jump_label.c:28 [inline] static_key_slow_inc_cpuslocked+0xc6/0x1f0 kernel/jump_label.c:128 static_key_slow_inc+0x15/0x20 kernel/jump_label.c:146 kvm_create_lapic+0x153/0x1d0 arch/x86/kvm/lapic.c:2284 kvm_arch_vcpu_init+0x5b6/0x7b0 arch/x86/kvm/x86.c:8923 kvm_vcpu_init+0x269/0x350 arch/x86/kvm/../../../virt/kvm/kvm_main.c:315 vmx_create_vcpu+0xf7/0x1a30 arch/x86/kvm/vmx.c:11530 kvm_arch_vcpu_create+0x52/0xd0 arch/x86/kvm/x86.c:8615 kvm_vm_ioctl_create_vcpu arch/x86/kvm/../../../virt/kvm/kvm_main.c:2482 [inline] kvm_vm_ioctl+0x45c/0x1680 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2985 vfs_ioctl fs/ioctl.c:46 [inline] file_ioctl fs/ioctl.c:509 [inline] do_vfs_ioctl+0x199/0x10d0 fs/ioctl.c:696 ksys_ioctl+0x62/0x90 fs/ioctl.c:713 __do_sys_ioctl fs/ioctl.c:720 [inline] __se_sys_ioctl fs/ioctl.c:718 [inline] __x64_sys_ioctl+0x6e/0xb0 fs/ioctl.c:718 do_syscall_64+0xd6/0x4e0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x457f89 Code: ad b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f73db1b0c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457f89 RDX: 0000000000000000 RSI: 000000000000ae41 RDI: 0000000000000004 RBP: 000000000073bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f73db1b16d4 R13: 00000000004c09e5 R14: 00000000004d2758 R15: 00000000ffffffff