bisecting cause commit starting from 60e8840126bdcb60bccef74c3f962742183c681f building syzkaller on 24dc29dba51a8ae7ae86ebc04521a0b2223c531f testing commit 60e8840126bdcb60bccef74c3f962742183c681f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 74de4c3db541117be375df734ddc8ac3ba0842579089ad4a2b8776db22869d54 run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: crashed: possible deadlock in perf_event_ctx_lock_nested run #7: crashed: possible deadlock in perf_event_ctx_lock_nested run #8: crashed: possible deadlock in perf_event_ctx_lock_nested run #9: crashed: possible deadlock in perf_event_ctx_lock_nested run #10: crashed: possible deadlock in perf_event_ctx_lock_nested run #11: crashed: possible deadlock in perf_event_ctx_lock_nested run #12: crashed: possible deadlock in perf_event_ctx_lock_nested run #13: OK run #14: crashed: possible deadlock in perf_event_ctx_lock_nested run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: de5d0644639cc5c360c5e4129b2c3253ccb03fa29846cb1eb6936ee1da8e68ac all runs: OK # git bisect start 60e8840126bdcb60bccef74c3f962742183c681f 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 10849 revisions left to test after this (roughly 13 steps) [5ffc06ebeaab65d8e08df6953caf8155df2c45f8] Merge tag 'char-misc-5.15-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc testing commit 5ffc06ebeaab65d8e08df6953caf8155df2c45f8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 40a68d9b277d490530fe64fda58e520acf0284421ad04dd1fe242ff502ef44aa all runs: OK # git bisect good 5ffc06ebeaab65d8e08df6953caf8155df2c45f8 Bisecting: 5118 revisions left to test after this (roughly 12 steps) [8afab2a2e332025c1477d6dcb0695b76875842ae] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git testing commit 8afab2a2e332025c1477d6dcb0695b76875842ae compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 393aa9609c2d1f915aae33a28eedc01c811eefb4a8fd22fefc0e87769220ca42 all runs: OK # git bisect good 8afab2a2e332025c1477d6dcb0695b76875842ae Bisecting: 2509 revisions left to test after this (roughly 11 steps) [b9361f3cb7e016eb15033d8c9db94a3f336a495b] Merge branch 'auto-latest' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git testing commit b9361f3cb7e016eb15033d8c9db94a3f336a495b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d8b64937913598788dd5bd2e0c9a3efd47ec80e34195f7f5221b545710362e46 run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: OK run #7: OK run #8: crashed: possible deadlock in perf_event_ctx_lock_nested run #9: OK # git bisect bad b9361f3cb7e016eb15033d8c9db94a3f336a495b Bisecting: 1284 revisions left to test after this (roughly 10 steps) [8a7df10375989c6173cc5391abc493003eee3934] Merge branch 'for-linux-next' of git://anongit.freedesktop.org/drm/drm-misc testing commit 8a7df10375989c6173cc5391abc493003eee3934 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 641f45610ca446ef0697a130891ef9f28202bc9573163357ed804cf1895e5851 all runs: OK # git bisect good 8a7df10375989c6173cc5391abc493003eee3934 Bisecting: 525 revisions left to test after this (roughly 9 steps) [8e99ea1ea613353946d4d8a78c2ceacfbbf7a5d9] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git testing commit 8e99ea1ea613353946d4d8a78c2ceacfbbf7a5d9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: af7f906c5ed55a0d4651ca97ff0f8877c8022ea0667672c11dc2ad4c29655ed1 run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: boot failed: general protection fault in hctx_lock # git bisect good 8e99ea1ea613353946d4d8a78c2ceacfbbf7a5d9 Bisecting: 258 revisions left to test after this (roughly 8 steps) [b1ca1108c45154f3f0a87fb780ae6ffce6c34d67] Merge branch 'mailbox-for-next' of git://git.linaro.org/landing-teams/working/fujitsu/integration.git testing commit b1ca1108c45154f3f0a87fb780ae6ffce6c34d67 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 53b5859e05f6989f5ab285a76ccf81c73c0f8bc7ccf261d46d728bb1b3488595 run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: boot failed: general protection fault in hctx_lock run #9: boot failed: general protection fault in hctx_lock # git bisect good b1ca1108c45154f3f0a87fb780ae6ffce6c34d67 Bisecting: 132 revisions left to test after this (roughly 7 steps) [8008765e74ce7a60c279861e80516fc946f91555] Merge branch 'perf/urgent' testing commit 8008765e74ce7a60c279861e80516fc946f91555 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 33007c4d0e40ecdc0ce0980a1c59b0637654ddfa7ac9744f3e824e5ad2d0e7eb run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 8008765e74ce7a60c279861e80516fc946f91555 Bisecting: 65 revisions left to test after this (roughly 6 steps) [09089db79859cbccccd8df95b034f36f7027efa6] irq_work: Also rcuwait for !IRQ_WORK_HARD_IRQ on PREEMPT_RT testing commit 09089db79859cbccccd8df95b034f36f7027efa6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fb6ad569a41b86648418ef9c89f52c452f4405dde59f262d1c19dea335d33d99 all runs: crashed: possible deadlock in perf_event_ctx_lock_nested # git bisect bad 09089db79859cbccccd8df95b034f36f7027efa6 Bisecting: 29 revisions left to test after this (roughly 5 steps) [8d491de6edc27138806cae6e8eca455beb325b62] sched: Move mmdrop to RCU on RT testing commit 8d491de6edc27138806cae6e8eca455beb325b62 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 424ef8e79a8b370d300d9b4675727e0b5645f9f11831944f9f199e1a2b479099 all runs: OK # git bisect good 8d491de6edc27138806cae6e8eca455beb325b62 Bisecting: 14 revisions left to test after this (roughly 4 steps) [f9ec6fea201429b5a3f76319e943989f1a1e25ef] sched/topology: Remove unused numa_distance in cpu_attach_domain() testing commit f9ec6fea201429b5a3f76319e943989f1a1e25ef compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5ecefff8c133567b6bb03bc2d3c8924ff77b3b94d16979a054bf7e2517c0c55f run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: crashed: possible deadlock in perf_event_ctx_lock_nested run #7: crashed: possible deadlock in perf_event_ctx_lock_nested run #8: OK run #9: OK # git bisect bad f9ec6fea201429b5a3f76319e943989f1a1e25ef Bisecting: 7 revisions left to test after this (roughly 3 steps) [f6ac18fafcf6cc5e41c26766d12ad335ed81012e] sched: Improve try_invoke_on_locked_down_task() testing commit f6ac18fafcf6cc5e41c26766d12ad335ed81012e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d5b2108f35a7820b4800efdb315f4d105b324a6921450a21acd4fc9d271f3a0c all runs: OK # git bisect good f6ac18fafcf6cc5e41c26766d12ad335ed81012e Bisecting: 3 revisions left to test after this (roughly 2 steps) [5de62ea84abd732ded7c5569426fd71c0420f83e] sched,livepatch: Use wake_up_if_idle() testing commit 5de62ea84abd732ded7c5569426fd71c0420f83e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5e4481519e2817a47aa6c4641dfe8a97b71b2893be2696563484281ccdc016ac run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 5de62ea84abd732ded7c5569426fd71c0420f83e Bisecting: 1 revision left to test after this (roughly 1 step) [00619f7c650e4e46c650cb2e2fd5f438b32dc64b] sched,livepatch: Use task_call_func() testing commit 00619f7c650e4e46c650cb2e2fd5f438b32dc64b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a8a150390fa5ab20a62915d9af4865c4fd36a9bbdc8865ed7112b5e3e6e22291 all runs: OK # git bisect good 00619f7c650e4e46c650cb2e2fd5f438b32dc64b Bisecting: 0 revisions left to test after this (roughly 0 steps) [8850cb663b5cda04d33f9cfbc38889d73d3c8e24] sched: Simplify wake_up_*idle*() testing commit 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5e4481519e2817a47aa6c4641dfe8a97b71b2893be2696563484281ccdc016ac run #0: crashed: possible deadlock in perf_event_ctx_lock_nested run #1: crashed: possible deadlock in perf_event_ctx_lock_nested run #2: crashed: possible deadlock in perf_event_ctx_lock_nested run #3: crashed: possible deadlock in perf_event_ctx_lock_nested run #4: crashed: possible deadlock in perf_event_ctx_lock_nested run #5: crashed: possible deadlock in perf_event_ctx_lock_nested run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 is the first bad commit commit 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 Author: Peter Zijlstra Date: Tue Sep 21 22:16:02 2021 +0200 sched: Simplify wake_up_*idle*() Simplify and make wake_up_if_idle() more robust, also don't iterate the whole machine with preempt_disable() in it's caller: wake_up_all_idle_cpus(). This prepares for another wake_up_if_idle() user that needs a full do_idle() cycle. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Vasily Gorbik Tested-by: Vasily Gorbik # on s390 Link: https://lkml.kernel.org/r/20210929152428.769328779@infradead.org kernel/sched/core.c | 14 +++++--------- kernel/smp.c | 6 +++--- 2 files changed, 8 insertions(+), 12 deletions(-) culprit signature: 5e4481519e2817a47aa6c4641dfe8a97b71b2893be2696563484281ccdc016ac parent signature: a8a150390fa5ab20a62915d9af4865c4fd36a9bbdc8865ed7112b5e3e6e22291 revisions tested: 16, total time: 4h18m27.055523901s (build: 1h52m56.695486043s, test: 2h23m35.982993788s) first bad commit: 8850cb663b5cda04d33f9cfbc38889d73d3c8e24 sched: Simplify wake_up_*idle*() recipients (to): ["gor@linux.ibm.com" "juri.lelli@redhat.com" "linux-kernel@vger.kernel.org" "mingo@redhat.com" "peterz@infradead.org" "peterz@infradead.org" "vincent.guittot@linaro.org"] recipients (cc): ["bigeasy@linutronix.de" "bristot@redhat.com" "bsegall@google.com" "dietmar.eggemann@arm.com" "jgross@suse.com" "mgorman@suse.de" "namit@vmware.com" "rostedt@goodmis.org"] crash: possible deadlock in perf_event_ctx_lock_nested ====================================================== WARNING: possible circular locking dependency detected 5.15.0-rc4-syzkaller #0 Not tainted ------------------------------------------------------ syz-executor.2/20377 is trying to acquire lock: ffff88807a1e6a28 (&mm->mmap_lock#2){++++}-{3:3}, at: __might_fault+0x8a/0x160 mm/memory.c:5261 but task is already holding lock: ffff8880b9e379b8 (&cpuctx_mutex){+.+.}-{3:3}, at: perf_event_ctx_lock_nested+0x18f/0x320 kernel/events/core.c:1357 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #3 (&cpuctx_mutex){+.+.}-{3:3}: __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x131/0x12f0 kernel/locking/mutex.c:729 perf_event_init_cpu+0x146/0x350 kernel/events/core.c:13295 perf_event_init+0x37d/0x3d3 kernel/events/core.c:13342 start_kernel+0x23f/0x35c init/main.c:1055 secondary_startup_64_no_verify+0xb0/0xbb -> #2 (pmus_lock){+.+.}-{3:3}: __mutex_lock_common kernel/locking/mutex.c:596 [inline] __mutex_lock+0x131/0x12f0 kernel/locking/mutex.c:729 perf_event_init_cpu+0xae/0x350 kernel/events/core.c:13289 cpuhp_invoke_callback+0x2f0/0x830 kernel/cpu.c:190 cpuhp_invoke_callback_range kernel/cpu.c:665 [inline] cpuhp_up_callbacks kernel/cpu.c:693 [inline] _cpu_up+0x2f8/0x5f0 kernel/cpu.c:1368 cpu_up kernel/cpu.c:1404 [inline] cpu_up+0x95/0x100 kernel/cpu.c:1376 bringup_nonboot_cpus+0xab/0xd0 kernel/cpu.c:1470 smp_init+0x23/0x106 kernel/smp.c:1092 kernel_init_freeable+0x3a0/0x605 init/main.c:1606 kernel_init+0x14/0x120 init/main.c:1505 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295 -> #1 (cpu_hotplug_lock){++++}-{0:0}: percpu_down_read include/linux/percpu-rwsem.h:51 [inline] cpus_read_lock+0x39/0xc0 kernel/cpu.c:308 wake_up_all_idle_cpus+0xc/0x50 kernel/smp.c:1173 cpu_latency_qos_apply kernel/power/qos.c:249 [inline] cpu_latency_qos_remove_request.part.0+0x81/0x210 kernel/power/qos.c:328 snd_pcm_hw_params+0x115e/0x1780 sound/core/pcm_native.c:784 snd_pcm_oss_change_params_locked+0x132f/0x3050 sound/core/oss/pcm_oss.c:947 snd_pcm_oss_change_params sound/core/oss/pcm_oss.c:1091 [inline] snd_pcm_oss_mmap+0x391/0x4a0 sound/core/oss/pcm_oss.c:2910 call_mmap include/linux/fs.h:2168 [inline] mmap_region+0xb0f/0x1480 mm/mmap.c:1787 do_mmap+0x5ca/0xd80 mm/mmap.c:1575 vm_mmap_pgoff+0x163/0x210 mm/util.c:519 ksys_mmap_pgoff+0x3b5/0x5f0 mm/mmap.c:1624 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae -> #0 (&mm->mmap_lock#2){++++}-{3:3}: check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x2985/0x5410 kernel/locking/lockdep.c:5015 lock_acquire kernel/locking/lockdep.c:5625 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5590 __might_fault mm/memory.c:5262 [inline] __might_fault+0xe8/0x160 mm/memory.c:5247 _copy_to_user+0x20/0xc0 lib/usercopy.c:28 copy_to_user include/linux/uaccess.h:200 [inline] perf_read_group kernel/events/core.c:5329 [inline] __perf_read kernel/events/core.c:5396 [inline] perf_read+0x5ef/0x7e0 kernel/events/core.c:5415 do_loop_readv_writev fs/read_write.c:750 [inline] do_loop_readv_writev fs/read_write.c:737 [inline] do_iter_read+0x3f0/0x6a0 fs/read_write.c:792 vfs_readv+0xc3/0x130 fs/read_write.c:910 do_readv+0x10b/0x280 fs/read_write.c:947 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae other info that might help us debug this: Chain exists of: &mm->mmap_lock#2 --> pmus_lock --> &cpuctx_mutex Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&cpuctx_mutex); lock(pmus_lock); lock(&cpuctx_mutex); lock(&mm->mmap_lock#2); *** DEADLOCK *** 1 lock held by syz-executor.2/20377: #0: ffff8880b9e379b8 (&cpuctx_mutex){+.+.}-{3:3}, at: perf_event_ctx_lock_nested+0x18f/0x320 kernel/events/core.c:1357 stack backtrace: CPU: 0 PID: 20377 Comm: syz-executor.2 Not tainted 5.15.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2131 check_prev_add kernel/locking/lockdep.c:3051 [inline] check_prevs_add kernel/locking/lockdep.c:3174 [inline] validate_chain kernel/locking/lockdep.c:3789 [inline] __lock_acquire+0x2985/0x5410 kernel/locking/lockdep.c:5015 lock_acquire kernel/locking/lockdep.c:5625 [inline] lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5590 __might_fault mm/memory.c:5262 [inline] __might_fault+0xe8/0x160 mm/memory.c:5247 _copy_to_user+0x20/0xc0 lib/usercopy.c:28 copy_to_user include/linux/uaccess.h:200 [inline] perf_read_group kernel/events/core.c:5329 [inline] __perf_read kernel/events/core.c:5396 [inline] perf_read+0x5ef/0x7e0 kernel/events/core.c:5415 do_loop_readv_writev fs/read_write.c:750 [inline] do_loop_readv_writev fs/read_write.c:737 [inline] do_iter_read+0x3f0/0x6a0 fs/read_write.c:792 vfs_readv+0xc3/0x130 fs/read_write.c:910 do_readv+0x10b/0x280 fs/read_write.c:947 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f5ad76f4a39 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f5ad6e49188 EFLAGS: 00000246 ORIG_RAX: 0000000000000013 RAX: ffffffffffffffda RBX: 00007f5ad77f8020 RCX: 00007f5ad76f4a39 RDX: 0000000000000001 RSI: 00000000200002c0 RDI: 0000000000000007 RBP: 00007f5ad774ec5f R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fff1c0c392f R14: 00007f5ad6e49300 R15: 0000000000022000