bisecting fixing commit since 9fa690a2a016e1b55356835f047b952e67d3d73a building syzkaller on 5ed23f9aa677d71bc60f61df8e05046151868541 testing commit 9fa690a2a016e1b55356835f047b952e67d3d73a with gcc (GCC) 8.1.0 kernel signature: 7d0a4675ef3f9a6b80febb70907ba5d75cac7ac4de7f2e1728bf8a43ffb9206d all runs: crashed: possible deadlock in perf_event_release_kernel testing current HEAD 01364dad1d4577e27a57729d41053f661bb8a5b9 testing commit 01364dad1d4577e27a57729d41053f661bb8a5b9 with gcc (GCC) 8.1.0 kernel signature: c8818a6d679f7c8fbaba37f217d5a469edaa0baab6de95936029ee97d4cec595 all runs: crashed: possible deadlock in perf_event_release_kernel revisions tested: 2, total time: 23m53.320587923s (build: 15m55.6887703s, test: 7m23.135999267s) the crash still happens on HEAD commit msg: Linux 4.14.174 crash: possible deadlock in perf_event_release_kernel NOHZ: local_softirq_pending 08 ====================================================== WARNING: possible circular locking dependency detected 4.14.174-syzkaller #0 Not tainted ------------------------------------------------------ syz-executor.0/9419 is trying to acquire lock: (event_mutex){+.+.}, at: [] perf_trace_destroy+0x1c/0x100 kernel/trace/trace_event_perf.c:234 but task is already holding lock: (&event->child_mutex){+.+.}, at: [] perf_event_release_kernel+0x1e3/0x7b0 kernel/events/core.c:4397 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #5 (&event->child_mutex){+.+.}: lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 perf_event_for_each_child+0x7f/0x140 kernel/events/core.c:4682 _perf_ioctl kernel/events/core.c:4869 [inline] perf_ioctl+0x4be/0xdd0 kernel/events/core.c:4881 vfs_ioctl fs/ioctl.c:46 [inline] file_ioctl fs/ioctl.c:500 [inline] do_vfs_ioctl+0x180/0xfb0 fs/ioctl.c:684 SYSC_ioctl fs/ioctl.c:701 [inline] SyS_ioctl+0x74/0x80 fs/ioctl.c:692 do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292 entry_SYSCALL_64_after_hwframe+0x42/0xb7 -> #4 (&cpuctx_mutex){+.+.}: lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 perf_event_init_cpu+0xb6/0x160 kernel/events/core.c:11234 perf_event_init+0x2cd/0x301 kernel/events/core.c:11281 start_kernel+0x365/0x5de init/main.c:620 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:399 x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:380 secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240 -> #3 (pmus_lock){+.+.}: lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 perf_event_init_cpu+0x2a/0x160 kernel/events/core.c:11228 cpuhp_invoke_callback+0x191/0x1610 kernel/cpu.c:184 cpuhp_up_callbacks kernel/cpu.c:572 [inline] _cpu_up+0x21e/0x540 kernel/cpu.c:1140 do_cpu_up+0x80/0x130 kernel/cpu.c:1175 cpu_up+0xe/0x10 kernel/cpu.c:1183 smp_init+0x69/0x10c kernel/smp.c:578 kernel_init_freeable+0x2d2/0x4ae init/main.c:1066 kernel_init+0xc/0x105 init/main.c:998 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404 -> #2 (cpu_hotplug_lock.rw_sem){++++}: lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline] percpu_down_read include/linux/percpu-rwsem.h:59 [inline] cpus_read_lock+0x38/0xa0 kernel/cpu.c:295 static_key_slow_inc+0xd/0x20 kernel/jump_label.c:123 tracepoint_add_func kernel/tracepoint.c:223 [inline] tracepoint_probe_register_prio+0x4de/0x6e0 kernel/tracepoint.c:283 tracepoint_probe_register+0xe/0x10 kernel/tracepoint.c:304 trace_event_reg+0x14d/0x340 kernel/trace/trace_events.c:305 perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline] perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline] perf_trace_init+0x3ce/0x9d0 kernel/trace/trace_event_perf.c:221 perf_tp_event_init+0x68/0xd0 kernel/events/core.c:8117 perf_try_init_event+0x138/0x1c0 kernel/events/core.c:9353 perf_init_event kernel/events/core.c:9391 [inline] perf_event_alloc+0xe09/0x2220 kernel/events/core.c:9651 SYSC_perf_event_open+0x447/0x21b0 kernel/events/core.c:10108 SyS_perf_event_open+0x9/0x10 kernel/events/core.c:9994 do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292 entry_SYSCALL_64_after_hwframe+0x42/0xb7 -> #1 (tracepoints_mutex){+.+.}: lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 tracepoint_probe_register_prio+0x30/0x6e0 kernel/tracepoint.c:279 tracepoint_probe_register+0xe/0x10 kernel/tracepoint.c:304 trace_event_reg+0x14d/0x340 kernel/trace/trace_events.c:305 perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline] perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline] perf_trace_init+0x3ce/0x9d0 kernel/trace/trace_event_perf.c:221 perf_tp_event_init+0x68/0xd0 kernel/events/core.c:8117 perf_try_init_event+0x138/0x1c0 kernel/events/core.c:9353 perf_init_event kernel/events/core.c:9391 [inline] perf_event_alloc+0xe09/0x2220 kernel/events/core.c:9651 SYSC_perf_event_open+0x447/0x21b0 kernel/events/core.c:10108 SyS_perf_event_open+0x9/0x10 kernel/events/core.c:9994 do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292 entry_SYSCALL_64_after_hwframe+0x42/0xb7 -> #0 (event_mutex){+.+.}: check_prev_add kernel/locking/lockdep.c:1901 [inline] check_prevs_add kernel/locking/lockdep.c:2018 [inline] validate_chain kernel/locking/lockdep.c:2460 [inline] __lock_acquire+0x2e94/0x4500 kernel/locking/lockdep.c:3487 lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 perf_trace_destroy+0x1c/0x100 kernel/trace/trace_event_perf.c:234 tp_perf_event_destroy+0x9/0x10 kernel/events/core.c:8101 _free_event+0x2e9/0xd50 kernel/events/core.c:4238 free_event+0x27/0x30 kernel/events/core.c:4265 perf_event_release_kernel+0x311/0x7b0 kernel/events/core.c:4409 perf_release+0x32/0x50 kernel/events/core.c:4435 __fput+0x232/0x750 fs/file_table.c:210 ____fput+0x9/0x10 fs/file_table.c:244 task_work_run+0xe5/0x170 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x16a/0x1b0 arch/x86/entry/common.c:164 prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline] syscall_return_slowpath arch/x86/entry/common.c:270 [inline] do_syscall_64+0x416/0x5b0 arch/x86/entry/common.c:297 entry_SYSCALL_64_after_hwframe+0x42/0xb7 other info that might help us debug this: Chain exists of: event_mutex --> &cpuctx_mutex --> &event->child_mutex Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&event->child_mutex); lock(&cpuctx_mutex); lock(&event->child_mutex); lock(event_mutex); *** DEADLOCK *** 2 locks held by syz-executor.0/9419: #0: (&ctx->mutex){+.+.}, at: [] perf_event_release_kernel+0x1d9/0x7b0 kernel/events/core.c:4396 #1: (&event->child_mutex){+.+.}, at: [] perf_event_release_kernel+0x1e3/0x7b0 kernel/events/core.c:4397 stack backtrace: CPU: 0 PID: 9419 Comm: syz-executor.0 Not tainted 4.14.174-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0xf7/0x13b lib/dump_stack.c:58 print_circular_bug.isra.40.cold.67+0x1bd/0x27d kernel/locking/lockdep.c:1258 check_prev_add kernel/locking/lockdep.c:1901 [inline] check_prevs_add kernel/locking/lockdep.c:2018 [inline] validate_chain kernel/locking/lockdep.c:2460 [inline] __lock_acquire+0x2e94/0x4500 kernel/locking/lockdep.c:3487 lock_acquire+0x173/0x400 kernel/locking/lockdep.c:3994 __mutex_lock_common kernel/locking/mutex.c:756 [inline] __mutex_lock+0xef/0x14c0 kernel/locking/mutex.c:893 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908 perf_trace_destroy+0x1c/0x100 kernel/trace/trace_event_perf.c:234 tp_perf_event_destroy+0x9/0x10 kernel/events/core.c:8101 _free_event+0x2e9/0xd50 kernel/events/core.c:4238 free_event+0x27/0x30 kernel/events/core.c:4265 perf_event_release_kernel+0x311/0x7b0 kernel/events/core.c:4409 perf_release+0x32/0x50 kernel/events/core.c:4435 __fput+0x232/0x750 fs/file_table.c:210 ____fput+0x9/0x10 fs/file_table.c:244 task_work_run+0xe5/0x170 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x16a/0x1b0 arch/x86/entry/common.c:164 prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline] syscall_return_slowpath arch/x86/entry/common.c:270 [inline] do_syscall_64+0x416/0x5b0 arch/x86/entry/common.c:297 entry_SYSCALL_64_after_hwframe+0x42/0xb7 RIP: 0033:0x414ee1 RSP: 002b:00007ffcb1c7cfc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 000000000000000a RCX: 0000000000414ee1 RDX: 0000000000000000 RSI: 0000000000000081 RDI: 0000000000000009 RBP: 0000000000000000 R08: 0000000000763bc0 R09: ffffffffffffffff R10: 00007ffcb1c7d090 R11: 0000000000000293 R12: 000000000075c070 R13: 0000000000000003 R14: 0000000000763bc8 R15: 000000000075c07c