syzbot


possible deadlock in perf_event_for_each_child

Status: upstream: reported C repro on 2019/04/14 09:58
Reported-by: syzbot+c82316340fea6502aaeb@syzkaller.appspotmail.com
First crash: 2070d, last: 654d
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in perf_event_for_each_child perf 135 2507d 2516d 0/28 auto-closed as invalid on 2019/02/22 10:22
android-414 possible deadlock in perf_event_for_each_child 9 1945d 2072d 0/1 auto-closed as invalid on 2019/12/15 20:17
Fix bisection attempts (7)
Created Duration User Patch Repo Result
2022/01/28 01:01 26m bisect fix linux-4.14.y OK (0) job log log
2021/11/16 23:38 26m bisect fix linux-4.14.y OK (0) job log log
2021/10/17 22:56 24m bisect fix linux-4.14.y OK (0) job log log
2021/09/17 22:34 21m bisect fix linux-4.14.y OK (0) job log log
2021/08/18 22:12 22m bisect fix linux-4.14.y OK (0) job log log
2021/07/19 21:49 23m bisect fix linux-4.14.y OK (0) job log log
2021/06/14 14:43 22m bisect fix linux-4.14.y OK (0) job log log

Sample crash report:
IPVS: ftp: loaded support on port[0] = 21
======================================================
WARNING: possible circular locking dependency detected
4.14.216-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor858/8018 is trying to acquire lock:
 (&event->child_mutex){+.+.}, at: [<ffffffff8162e492>] perf_event_for_each_child+0x82/0x140 kernel/events/core.c:4688

but task is already holding lock:
 (&cpuctx_mutex){+.+.}, at: [<ffffffff81632efd>] perf_event_ctx_lock_nested+0x14d/0x2c0 kernel/events/core.c:1241

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (&cpuctx_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       perf_event_init_cpu+0xb7/0x170 kernel/events/core.c:11250
       perf_event_init+0x2cc/0x308 kernel/events/core.c:11297
       start_kernel+0x46a/0x770 init/main.c:620
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #4 (pmus_lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       perf_event_init_cpu+0x2c/0x170 kernel/events/core.c:11244
       cpuhp_invoke_callback+0x1e6/0x1a80 kernel/cpu.c:184
       cpuhp_up_callbacks kernel/cpu.c:572 [inline]
       _cpu_up+0x219/0x500 kernel/cpu.c:1144
       do_cpu_up+0x9a/0x160 kernel/cpu.c:1179
       smp_init+0x197/0x1ac kernel/smp.c:578
       kernel_init_freeable+0x3f4/0x614 init/main.c:1068
       kernel_init+0xd/0x166 init/main.c:1000
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #3 (cpu_hotplug_lock.rw_sem){++++}:
       percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
       percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
       cpus_read_lock+0x39/0xc0 kernel/cpu.c:295
       static_key_slow_inc+0xe/0x20 kernel/jump_label.c:123
       tracepoint_add_func+0x517/0x750 kernel/tracepoint.c:223
       tracepoint_probe_register_prio kernel/tracepoint.c:283 [inline]
       tracepoint_probe_register+0x8c/0xc0 kernel/tracepoint.c:304
       trace_event_reg+0x272/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x424/0xa30 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x79/0xf0 kernel/events/core.c:8138
       perf_try_init_event+0x15b/0x1f0 kernel/events/core.c:9369
       perf_init_event kernel/events/core.c:9407 [inline]
       perf_event_alloc.part.0+0xe2d/0x2640 kernel/events/core.c:9667
       perf_event_alloc kernel/events/core.c:10020 [inline]
       SYSC_perf_event_open kernel/events/core.c:10124 [inline]
       SyS_perf_event_open+0x67f/0x24b0 kernel/events/core.c:10010
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

-> #2 (tracepoints_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       tracepoint_probe_register_prio kernel/tracepoint.c:279 [inline]
       tracepoint_probe_register+0x68/0xc0 kernel/tracepoint.c:304
       trace_event_reg+0x272/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x424/0xa30 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x79/0xf0 kernel/events/core.c:8138
       perf_try_init_event+0x15b/0x1f0 kernel/events/core.c:9369
       perf_init_event kernel/events/core.c:9407 [inline]
       perf_event_alloc.part.0+0xe2d/0x2640 kernel/events/core.c:9667
       perf_event_alloc kernel/events/core.c:10020 [inline]
       SYSC_perf_event_open kernel/events/core.c:10124 [inline]
       SyS_perf_event_open+0x67f/0x24b0 kernel/events/core.c:10010
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

-> #1 (event_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       perf_trace_destroy+0x23/0xf0 kernel/trace/trace_event_perf.c:234
       _free_event+0x321/0xe20 kernel/events/core.c:4244
       free_event+0x32/0x40 kernel/events/core.c:4271
       perf_event_release_kernel+0x368/0x8a0 kernel/events/core.c:4415
       perf_release+0x33/0x40 kernel/events/core.c:4441
       __fput+0x25f/0x7a0 fs/file_table.c:210
       task_work_run+0x11f/0x190 kernel/task_work.c:113
       exit_task_work include/linux/task_work.h:22 [inline]
       do_exit+0xa44/0x2850 kernel/exit.c:868
       do_group_exit+0x100/0x2e0 kernel/exit.c:965
       get_signal+0x38d/0x1ca0 kernel/signal.c:2423
       do_signal+0x7c/0x1550 arch/x86/kernel/signal.c:814
       exit_to_usermode_loop+0x160/0x200 arch/x86/entry/common.c:160
       prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
       syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
       do_syscall_64+0x4a3/0x640 arch/x86/entry/common.c:297
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

-> #0 (&event->child_mutex){+.+.}:
       lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       perf_event_for_each_child+0x82/0x140 kernel/events/core.c:4688
       perf_event_for_each kernel/events/core.c:4705 [inline]
       _perf_ioctl+0x3db/0x1a40 kernel/events/core.c:4873
       perf_ioctl+0x55/0x80 kernel/events/core.c:4887
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x46/0xbb

other info that might help us debug this:

Chain exists of:
  &event->child_mutex --> pmus_lock --> &cpuctx_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&cpuctx_mutex);
                               lock(pmus_lock);
                               lock(&cpuctx_mutex);
  lock(&event->child_mutex);

 *** DEADLOCK ***

1 lock held by syz-executor858/8018:
 #0:  (&cpuctx_mutex){+.+.}, at: [<ffffffff81632efd>] perf_event_ctx_lock_nested+0x14d/0x2c0 kernel/events/core.c:1241

stack backtrace:
CPU: 0 PID: 8018 Comm: syz-executor858 Not tainted 4.14.216-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x1b2/0x281 lib/dump_stack.c:58
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1905 [inline]
 check_prevs_add kernel/locking/lockdep.c:2022 [inline]
 validate_chain kernel/locking/lockdep.c:2464 [inline]
 __lock_acquire+0x2e0e/0x3f20 kernel/locking/lockdep.c:3491
 lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
 __mutex_lock_common kernel/locking/mutex.c:756 [inline]
 __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
 perf_event_for_each_child+0x82/0x140 kernel/events/core.c:4688
 perf_event_for_each kernel/events/core.c:4705 [inline]
 _perf_ioctl+0x3db/0x1a40 kernel/events/core.c:4873
 perf_ioctl+0x55/0x80 kernel/events/core.c:4887
 vfs_ioctl fs/ioctl.c:46 [inline]
 file_ioctl fs/ioctl.c:500 [inline]
 do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
 SYSC_ioctl fs/ioctl.c:701 [inline]
 SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
 do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x46/0xbb
RIP: 0033:0x44a8c9
RSP: 002b:00007f24b291cdb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00000000006dbc28 RCX: 000000000044a8c9
RDX: 0000000000000003 RSI: 0000000000002403 RDI: 0000000000000004
RBP: 00000000006dbc20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dbc2c
R13: 00007ffdf2e14cbf R14: 00007f24b291d9c0 R15: 0000000000000000

Crashes (1482):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/01/19 00:20 linux-4.14.y 2762b48e9611 63631df1 .config console log report syz C ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/28 15:15 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/27 17:10 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/14 09:59 linux-4.14.y a8ad60f2af58 93ae7e0a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/13 23:43 linux-4.14.y a8ad60f2af58 957959cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/10 21:17 linux-4.14.y a8ad60f2af58 e29a17f5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/02/08 02:35 linux-4.14.y a8ad60f2af58 15c3d445 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/01/24 15:24 linux-4.14.y 3949d1610004 9dfcf09c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/01/20 17:36 linux-4.14.y 97205fccccdc dd15ff29 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/01/14 16:53 linux-4.14.y c4215ee4771b a63719e7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2023/01/10 11:26 linux-4.14.y c4215ee4771b 48bc529a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/21 12:10 linux-4.14.y c4215ee4771b d3e76707 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/20 18:37 linux-4.14.y c4215ee4771b d3e76707 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/20 05:38 linux-4.14.y c4215ee4771b c52b2efb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/16 00:24 linux-4.14.y c4215ee4771b 6f9c033e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/13 21:05 linux-4.14.y 65afe34ac33d f6511626 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/13 08:39 linux-4.14.y 65afe34ac33d 67be1ae7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/09 09:49 linux-4.14.y 65afe34ac33d 1034e5fa .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/02 04:34 linux-4.14.y 179ef7fe8677 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/12/02 01:23 linux-4.14.y 179ef7fe8677 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/30 21:26 linux-4.14.y 179ef7fe8677 4c2a66e8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/30 15:32 linux-4.14.y 179ef7fe8677 4c2a66e8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/27 04:05 linux-4.14.y 179ef7fe8677 f4470a7b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/25 18:50 linux-4.14.y e911713e40ca 74a66371 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/23 01:10 linux-4.14.y e911713e40ca 9da37ae8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/11 02:04 linux-4.14.y e911713e40ca 3ead01ad .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/05 11:42 linux-4.14.y a901bb6c7db7 6d752409 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/04 22:00 linux-4.14.y a901bb6c7db7 6d752409 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/11/03 00:38 linux-4.14.y a85772d7ba90 08977f5d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/10/20 15:10 linux-4.14.y 9d5c0b3a8e1a b31320fc .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/10/12 20:21 linux-4.14.y 9d5c0b3a8e1a 16a9c9e0 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/10/10 14:20 linux-4.14.y 9d5c0b3a8e1a 5bcf0c31 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/10/04 15:39 linux-4.14.y 9d5c0b3a8e1a eab8f949 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/10/01 08:15 linux-4.14.y 9d5c0b3a8e1a feb56351 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/25 20:24 linux-4.14.y 4edbf74132a4 0042f2b4 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/22 02:33 linux-4.14.y 4edbf74132a4 60af5050 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/21 07:38 linux-4.14.y 4edbf74132a4 c4b8ccfd .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/21 01:28 linux-4.14.y 4edbf74132a4 c4b8ccfd .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/14 02:15 linux-4.14.y 65640c873dcf b884348d .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/09 12:40 linux-4.14.y 65640c873dcf 90058bdc .config console log report info ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/09 02:10 linux-4.14.y 65640c873dcf f3027468 .config console log report info ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/08 13:23 linux-4.14.y 65640c873dcf f3027468 .config console log report info ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/09/05 12:22 linux-4.14.y 65640c873dcf 922294ab .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2022/08/29 16:49 linux-4.14.y e548869f356f d7593c58 .config console log report info [disk image] [vmlinux] ci2-linux-4-14 possible deadlock in perf_event_for_each_child
2021/01/17 10:56 linux-4.14.y f79dc86058bc 813be542 .config console log report info ci2-linux-4-14
2019/04/14 09:28 linux-4.14.y 1ec8f1f0bffe 505ab413 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.