bisecting cause commit starting from 7c832d2f9b959e3181370c8b0dacaf9efe13fc05 building syzkaller on 0c5d9412d774262384cbdbe9d672b077364ed776 testing commit 7c832d2f9b959e3181370c8b0dacaf9efe13fc05 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2c741ddeb7fc67524c922aaa5c48c13160e428aae5fafa5e45db74e614c0a53b run #0: crashed: possible deadlock in wake_up_all_idle_cpus run #1: crashed: possible deadlock in wake_up_all_idle_cpus run #2: crashed: possible deadlock in wake_up_all_idle_cpus run #3: crashed: possible deadlock in wake_up_all_idle_cpus run #4: crashed: possible deadlock in wake_up_all_idle_cpus run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: crashed: possible deadlock in perf_event_ctx_lock_nested run #7: crashed: possible deadlock in wake_up_all_idle_cpus run #8: crashed: possible deadlock in wake_up_all_idle_cpus run #9: crashed: possible deadlock in wake_up_all_idle_cpus run #10: crashed: possible deadlock in wake_up_all_idle_cpus run #11: crashed: possible deadlock in wake_up_all_idle_cpus run #12: crashed: possible deadlock in wake_up_all_idle_cpus run #13: crashed: possible deadlock in wake_up_all_idle_cpus run #14: crashed: possible deadlock in wake_up_all_idle_cpus run #15: crashed: possible deadlock in wake_up_all_idle_cpus run #16: crashed: possible deadlock in wake_up_all_idle_cpus run #17: crashed: possible deadlock in wake_up_all_idle_cpus run #18: crashed: possible deadlock in wake_up_all_idle_cpus run #19: crashed: possible deadlock in wake_up_all_idle_cpus testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: de5d0644639cc5c360c5e4129b2c3253ccb03fa29846cb1eb6936ee1da8e68ac all runs: OK # git bisect start 7c832d2f9b959e3181370c8b0dacaf9efe13fc05 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 10492 revisions left to test after this (roughly 13 steps) [14e2bc4e8c40a876c1ab5597320d523c12a97f39] Merge tag 'nfsd-5.15-1' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux testing commit 14e2bc4e8c40a876c1ab5597320d523c12a97f39 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b5ac7330f00980c4a84c1dced6473dd4592419cabd72572af26ec7f821dbbbbb all runs: OK # git bisect good 14e2bc4e8c40a876c1ab5597320d523c12a97f39 Bisecting: 5703 revisions left to test after this (roughly 12 steps) [fc25206d8fae5eeb06a24fa3cb3f31848ec2c146] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git testing commit fc25206d8fae5eeb06a24fa3cb3f31848ec2c146 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 26e4ec5ffb2a705ead90642fac71623b9723ec5aac47cfb06556b8762bdc641e all runs: OK # git bisect good fc25206d8fae5eeb06a24fa3cb3f31848ec2c146 Bisecting: 2914 revisions left to test after this (roughly 12 steps) [029e7360a7206f14b0c0cbee2e96fb070846d8a1] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input.git testing commit 029e7360a7206f14b0c0cbee2e96fb070846d8a1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bc622aba2f14dbf5aa1d9ecfec2610598f4e2ffd3160702b03661ebb3f63dbff all runs: OK # git bisect good 029e7360a7206f14b0c0cbee2e96fb070846d8a1 Bisecting: 1517 revisions left to test after this (roughly 11 steps) [d8a625620ad72c878ac6cfc3a64039bbd76db079] Merge branch 'next' of git://github.com/awilliam/linux-vfio.git testing commit d8a625620ad72c878ac6cfc3a64039bbd76db079 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7ae6824bd7b02f70640b16bd4b375271bb5cd934aa10af608ade641bad8e173e run #0: crashed: possible deadlock in wake_up_all_idle_cpus run #1: crashed: possible deadlock in wake_up_all_idle_cpus run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in wake_up_all_idle_cpus run #4: crashed: possible deadlock in wake_up_all_idle_cpus run #5: crashed: possible deadlock in wake_up_all_idle_cpus run #6: crashed: possible deadlock in wake_up_all_idle_cpus run #7: crashed: possible deadlock in wake_up_all_idle_cpus run #8: crashed: possible deadlock in wake_up_all_idle_cpus run #9: crashed: possible deadlock in wake_up_all_idle_cpus # git bisect bad d8a625620ad72c878ac6cfc3a64039bbd76db079 Bisecting: 704 revisions left to test after this (roughly 10 steps) [5c6d304fdad0e311dcea28c335888a676d798150] Merge branch 'edac-for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git testing commit 5c6d304fdad0e311dcea28c335888a676d798150 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: db285b9cdf81c013cc51d955d9b4143ebde7e2bda442ec05d2ae77b50db1927e all runs: crashed: possible deadlock in wake_up_all_idle_cpus # git bisect bad 5c6d304fdad0e311dcea28c335888a676d798150 Bisecting: 348 revisions left to test after this (roughly 9 steps) [7780d7d7f0eb1053b604c59b1f225dffc4217823] Merge branch 'next-testing' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security.git testing commit 7780d7d7f0eb1053b604c59b1f225dffc4217823 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f2e9efe08ccd5bf45da022ad6f4821b1168875de4d0b985452204547d1d858e8 run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: boot failed: general protection fault in hctx_lock run #9: boot failed: general protection fault in hctx_lock # git bisect good 7780d7d7f0eb1053b604c59b1f225dffc4217823 Bisecting: 184 revisions left to test after this (roughly 8 steps) [a5dd661e53635877debbf48045913266b429950a] Merge remote-tracking branch 'tip/sched/core' into tip-master testing commit a5dd661e53635877debbf48045913266b429950a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7666cc16720fef3281e5910d8fbb501ce9d171c993effebc77d2c6e6069b82ea all runs: crashed: possible deadlock in wake_up_all_idle_cpus # git bisect bad a5dd661e53635877debbf48045913266b429950a Bisecting: 86 revisions left to test after this (roughly 6 steps) [e3e941e633873a6203f0f71afe7c9a4a813fcb83] Merge remote-tracking branch 'tip/objtool/core' into tip-master testing commit e3e941e633873a6203f0f71afe7c9a4a813fcb83 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7873d09d131934028624cc2cc85e80229545fa7dafc48ee5358ae2834131cf8c all runs: OK # git bisect good e3e941e633873a6203f0f71afe7c9a4a813fcb83 Bisecting: 43 revisions left to test after this (roughly 6 steps) [00c034108a76e8282809b7f25fa6ff147a9c6893] sched/numa: Remove the redundant member numa_group::fault_cpus testing commit 00c034108a76e8282809b7f25fa6ff147a9c6893 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1ce65b4ed5c09eb4c8b4351fee178fdc5ed67493ab4059d25c01df5dfe625b08 all runs: crashed: possible deadlock in wake_up_all_idle_cpus # git bisect bad 00c034108a76e8282809b7f25fa6ff147a9c6893 Bisecting: 21 revisions left to test after this (roughly 5 steps) [c597bfddc9e9e8a63817252b67c3ca0e544ace26] sched: Provide Kconfig support for default dynamic preempt mode testing commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e48575d1186391f4517b7af92cd490c1793c75c880dcdc54ad2d7c830a07d176 all runs: OK # git bisect good c597bfddc9e9e8a63817252b67c3ca0e544ace26 Bisecting: 10 revisions left to test after this (roughly 4 steps) [539fbb5be0da56ffa1434b4f56521a0522bd1d61] sched: Disable TTWU_QUEUE on RT testing commit 539fbb5be0da56ffa1434b4f56521a0522bd1d61 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6a9969bf9b7e629c5d2316bc0552573fee1cbac105091101d9f0ed24a57f659c all runs: OK # git bisect good 539fbb5be0da56ffa1434b4f56521a0522bd1d61 Bisecting: 5 revisions left to test after this (roughly 3 steps) [f6ac18fafcf6cc5e41c26766d12ad335ed81012e] sched: Improve try_invoke_on_locked_down_task() testing commit f6ac18fafcf6cc5e41c26766d12ad335ed81012e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d5b2108f35a7820b4800efdb315f4d105b324a6921450a21acd4fc9d271f3a0c all runs: OK # git bisect good f6ac18fafcf6cc5e41c26766d12ad335ed81012e Bisecting: 2 revisions left to test after this (roughly 2 steps) [8850cb663b5cda04d33f9cfbc38889d73d3c8e24] sched: Simplify wake_up_*idle*() testing commit 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5e4481519e2817a47aa6c4641dfe8a97b71b2893be2696563484281ccdc016ac run #0: crashed: possible deadlock in wake_up_all_idle_cpus run #1: crashed: possible deadlock in wake_up_all_idle_cpus run #2: crashed: possible deadlock in wake_up_all_idle_cpus run #3: crashed: possible deadlock in wake_up_all_idle_cpus run #4: crashed: possible deadlock in wake_up_all_idle_cpus run #5: crashed: possible deadlock in wake_up_all_idle_cpus run #6: crashed: possible deadlock in perf_event_ctx_lock_nested run #7: crashed: possible deadlock in wake_up_all_idle_cpus run #8: crashed: possible deadlock in wake_up_all_idle_cpus run #9: crashed: possible deadlock in wake_up_all_idle_cpus # git bisect bad 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 Bisecting: 0 revisions left to test after this (roughly 1 step) [00619f7c650e4e46c650cb2e2fd5f438b32dc64b] sched,livepatch: Use task_call_func() testing commit 00619f7c650e4e46c650cb2e2fd5f438b32dc64b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a8a150390fa5ab20a62915d9af4865c4fd36a9bbdc8865ed7112b5e3e6e22291 all runs: OK # git bisect good 00619f7c650e4e46c650cb2e2fd5f438b32dc64b 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 is the first bad commit commit 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 Author: Peter Zijlstra Date: Tue Sep 21 22:16:02 2021 +0200 sched: Simplify wake_up_*idle*() Simplify and make wake_up_if_idle() more robust, also don't iterate the whole machine with preempt_disable() in it's caller: wake_up_all_idle_cpus(). This prepares for another wake_up_if_idle() user that needs a full do_idle() cycle. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Vasily Gorbik Tested-by: Vasily Gorbik # on s390 Link: https://lkml.kernel.org/r/20210929152428.769328779@infradead.org kernel/sched/core.c | 14 +++++--------- kernel/smp.c | 6 +++--- 2 files changed, 8 insertions(+), 12 deletions(-) culprit signature: 5e4481519e2817a47aa6c4641dfe8a97b71b2893be2696563484281ccdc016ac parent signature: a8a150390fa5ab20a62915d9af4865c4fd36a9bbdc8865ed7112b5e3e6e22291 revisions tested: 16, total time: 3h39m50.717112592s (build: 1h46m35.00420189s, test: 1h51m29.659644697s) first bad commit: 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 sched: Simplify wake_up_*idle*() recipients (to): ["gor@linux.ibm.com" "juri.lelli@redhat.com" "linux-kernel@vger.kernel.org" "mingo@redhat.com" "peterz@infradead.org" "peterz@infradead.org" "vincent.guittot@linaro.org"] recipients (cc): ["bigeasy@linutronix.de" "bristot@redhat.com" "bsegall@google.com" "dietmar.eggemann@arm.com" "jgross@suse.com" "mgorman@suse.de" "namit@vmware.com" "rostedt@goodmis.org"] crash: possible deadlock in wake_up_all_idle_cpus ====================================================== WARNING: possible circular locking dependency detected 5.15.0-rc4-syzkaller #0 Not tainted ------------------------------------------------------ syz-executor.0/9041 is trying to acquire lock: ffffffff8aa25110 (cpu_hotplug_lock){++++}-{0:0}, at: wake_up_all_idle_cpus+0xc/0x50 kernel/smp.c:1173 but task is already holding lock: ffff88807b6c1628 (&mm->mmap_lock#2){++++}-{3:3}, at: mmap_write_lock_killable include/linux/mmap_lock.h:87 [inline] ffff88807b6c1628 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0x12e/0x210 mm/util.c:517 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #3 (&mm->mmap_lock#2){++++}-{3:3}: __might_fault mm/memory.c:5262 [inline] __might_fault+0xe8/0x160 mm/memory.c:5247 _copy_from_user+0x20/0xf0 lib/usercopy.c:13 copy_from_user include/linux/uaccess.h:192 [inline] memdup_user+0x49/0x90 mm/util.c:177 strndup_user+0x42/0x90 mm/util.c:232 perf_event_set_filter kernel/events/core.c:10512 [inline] _perf_ioctl+0x171/0x1850 kernel/events/core.c:5659 perf_ioctl+0x61/0x90 kernel/events/core.c:5730 vfs_ioctl fs/ioctl.c:51 [inline] __do_sys_ioctl fs/ioctl.c:874 [inline] __se_sys_ioctl fs/ioctl.c:860 [inline] __x64_sys_ioctl+0x11f/0x190 fs/ioctl.c:860 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae -> #2 (&cpuctx_mutex){+.+.}-{3:3}: __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x131/0x12f0 kernel/locking/mutex.c:729 perf_event_init_cpu+0x146/0x350 kernel/events/core.c:13295 perf_event_init+0x37d/0x3d3 kernel/events/core.c:13342 start_kernel+0x23f/0x35c init/main.c:1055 secondary_startup_64_no_verify+0xb0/0xbb -> #1 (pmus_lock){+.+.}-{3:3}: __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x131/0x12f0 kernel/locking/mutex.c:729 perf_event_init_cpu+0xae/0x350 kernel/events/core.c:13289 cpuhp_invoke_callback+0x2f0/0x830 kernel/cpu.c:190 cpuhp_invoke_callback_range kernel/cpu.c:665 [inline] cpuhp_up_callbacks kernel/cpu.c:693 [inline] _cpu_up+0x2f8/0x5f0 kernel/cpu.c:1368 cpu_up kernel/cpu.c:1404 [inline] cpu_up+0x95/0x100 kernel/cpu.c:1376 bringup_nonboot_cpus+0xab/0xd0 kernel/cpu.c:1470 smp_init+0x23/0x106 kernel/smp.c:1092 kernel_init_freeable+0x3a0/0x605 init/main.c:1606 kernel_init+0x14/0x120 init/main.c:1505 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295 -> #0 (cpu_hotplug_lock){++++}-{0:0}: check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x2985/0x5410 kernel/locking/lockdep.c:5015 lock_acquire kernel/locking/lockdep.c:5625 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5590 percpu_down_read include/linux/percpu-rwsem.h:51 [inline] cpus_read_lock+0x39/0xc0 kernel/cpu.c:308 wake_up_all_idle_cpus+0xc/0x50 kernel/smp.c:1173 cpu_latency_qos_apply kernel/power/qos.c:249 [inline] cpu_latency_qos_remove_request.part.0+0x81/0x210 kernel/power/qos.c:328 snd_pcm_hw_params+0x115e/0x1780 sound/core/pcm_native.c:784 snd_pcm_oss_change_params_locked+0x132f/0x3050 sound/core/oss/pcm_oss.c:947 snd_pcm_oss_change_params sound/core/oss/pcm_oss.c:1091 [inline] snd_pcm_oss_mmap+0x391/0x4a0 sound/core/oss/pcm_oss.c:2910 call_mmap include/linux/fs.h:2168 [inline] mmap_region+0xb0f/0x1480 mm/mmap.c:1787 do_mmap+0x5ca/0xd80 mm/mmap.c:1575 vm_mmap_pgoff+0x163/0x210 mm/util.c:519 ksys_mmap_pgoff+0x3b5/0x5f0 mm/mmap.c:1624 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae other info that might help us debug this: Chain exists of: cpu_hotplug_lock --> &cpuctx_mutex --> &mm->mmap_lock#2 Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&mm->mmap_lock#2); lock(&cpuctx_mutex); lock(&mm->mmap_lock#2); lock(cpu_hotplug_lock); *** DEADLOCK *** 2 locks held by syz-executor.0/9041: #0: ffff88807b6c1628 (&mm->mmap_lock#2){++++}-{3:3}, at: mmap_write_lock_killable include/linux/mmap_lock.h:87 [inline] #0: ffff88807b6c1628 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0x12e/0x210 mm/util.c:517 #1: ffff88807b8cd440 (&runtime->oss.params_lock){+.+.}-{3:3}, at: snd_pcm_oss_change_params sound/core/oss/pcm_oss.c:1086 [inline] #1: ffff88807b8cd440 (&runtime->oss.params_lock){+.+.}-{3:3}, at: snd_pcm_oss_mmap+0x385/0x4a0 sound/core/oss/pcm_oss.c:2910 stack backtrace: CPU: 1 PID: 9041 Comm: syz-executor.0 Not tainted 5.15.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2131 check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x2985/0x5410 kernel/locking/lockdep.c:5015 lock_acquire kernel/locking/lockdep.c:5625 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5590 percpu_down_read include/linux/percpu-rwsem.h:51 [inline] cpus_read_lock+0x39/0xc0 kernel/cpu.c:308 wake_up_all_idle_cpus+0xc/0x50 kernel/smp.c:1173 cpu_latency_qos_apply kernel/power/qos.c:249 [inline] cpu_latency_qos_remove_request.part.0+0x81/0x210 kernel/power/qos.c:328 snd_pcm_hw_params+0x115e/0x1780 sound/core/pcm_native.c:784 snd_pcm_oss_change_params_locked+0x132f/0x3050 sound/core/oss/pcm_oss.c:947 snd_pcm_oss_change_params sound/core/oss/pcm_oss.c:1091 [inline] snd_pcm_oss_mmap+0x391/0x4a0 sound/core/oss/pcm_oss.c:2910 call_mmap include/linux/fs.h:2168 [inline] mmap_region+0xb0f/0x1480 mm/mmap.c:1787 do_mmap+0x5ca/0xd80 mm/mmap.c:1575 vm_mmap_pgoff+0x163/0x210 mm/util.c:519 ksys_mmap_pgoff+0x3b5/0x5f0 mm/mmap.c:1624 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f9730a8ba39 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f9730201188 EFLAGS: 00000246 ORIG_RAX: 0000000000000009 RAX: ffffffffffffffda RBX: 00007f9730b8ef60 RCX: 00007f9730a8ba39 RDX: 0000000001800003 RSI: 0000000000800000 RDI: 0000000020000000 RBP: 00007f9730ae5c5f R08: 0000000000000004 R09: 0000000000000000 R10: 0000000000000012 R11: 0000000000000246 R12: 0000000000000000 R13: 00007ffef0657faf R14: 00007f9730201300 R15: 0000000000022000