possible deadlock in perf_read

Status: upstream: reported C repro on 2019/07/25 19:47
Reported-by: syzbot+93e350984a1eedc29917@syzkaller.appspotmail.com
First crash: 2587d, last: 1602d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Last patch testing requests (4)
Created Duration User Patch Repo Result
2023/02/26 15:32 42m retest repro linux-4.14.y report log
2023/02/26 14:32 12m retest repro linux-4.14.y report log
2022/11/07 20:30 10m retest repro linux-4.14.y report log
2022/11/07 19:30 11m retest repro linux-4.14.y report log
▶ ▼ Fix bisection attempts (15)
Created Duration User Patch Repo Result
2022/03/14 01:31 0m bisect fix linux-4.14.y error job log
2022/02/11 16:16 29m bisect fix linux-4.14.y OK (0) job log log
2022/01/12 15:53 23m bisect fix linux-4.14.y OK (0) job log log
2021/12/13 15:27 25m bisect fix linux-4.14.y OK (0) job log log
2021/11/13 14:59 21m bisect fix linux-4.14.y OK (0) job log log
2021/10/14 14:28 31m bisect fix linux-4.14.y OK (0) job log log
2021/09/14 13:54 24m bisect fix linux-4.14.y OK (0) job log log
2021/08/15 13:31 22m bisect fix linux-4.14.y OK (0) job log log
2021/07/16 13:06 24m bisect fix linux-4.14.y OK (0) job log log
2021/06/16 12:15 21m bisect fix linux-4.14.y OK (0) job log log
2021/03/12 11:03 21m bisect fix linux-4.14.y OK (0) job log log
2020/08/17 15:43 27m bisect fix linux-4.14.y OK (0) job log log
2020/04/27 07:49 23m bisect fix linux-4.14.y OK (0) job log log
2020/03/28 07:26 22m bisect fix linux-4.14.y OK (0) job log log
2020/02/03 22:41 24m bisect fix linux-4.14.y OK (0) job log log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
4.14.150 #0 Not tainted
------------------------------------------------------
syz-executor071/7190 is trying to acquire lock:
 (&event->child_mutex){+.+.}, at: [<ffffffff816bfab0>] perf_read_group kernel/events/core.c:4536 [inline]
 (&event->child_mutex){+.+.}, at: [<ffffffff816bfab0>] __perf_read kernel/events/core.c:4618 [inline]
 (&event->child_mutex){+.+.}, at: [<ffffffff816bfab0>] perf_read+0x2c0/0x820 kernel/events/core.c:4633

but task is already holding lock:
 (&cpuctx_mutex){+.+.}, at: [<ffffffff816bd1a0>] perf_event_ctx_lock_nested+0x150/0x2c0 kernel/events/core.c:1235

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (&cpuctx_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0xc2/0x170 kernel/events/core.c:11223
       perf_event_init+0x2d8/0x31a kernel/events/core.c:11270
       start_kernel+0x3b6/0x6fd init/main.c:621
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:381
       x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:362
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #4 (pmus_lock){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0x2f/0x170 kernel/events/core.c:11217
       cpuhp_invoke_callback+0x1ea/0x1ab0 kernel/cpu.c:184
       cpuhp_up_callbacks kernel/cpu.c:572 [inline]
       _cpu_up+0x228/0x530 kernel/cpu.c:1134
       do_cpu_up+0x121/0x150 kernel/cpu.c:1169
       cpu_up+0x1b/0x20 kernel/cpu.c:1177
       smp_init+0x157/0x170 kernel/smp.c:578
       kernel_init_freeable+0x30b/0x532 init/main.c:1067
       kernel_init+0x12/0x162 init/main.c:999
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #3 (cpu_hotplug_lock.rw_sem){++++}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
       percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
       cpus_read_lock+0x3d/0xc0 kernel/cpu.c:295
       static_key_slow_inc+0x13/0x30 kernel/jump_label.c:123
       tracepoint_add_func kernel/tracepoint.c:223 [inline]
       tracepoint_probe_register_prio+0x4d6/0x6d0 kernel/tracepoint.c:283
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9342
       perf_init_event kernel/events/core.c:9380 [inline]
       perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
       perf_event_alloc kernel/events/core.c:9993 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #2 (tracepoints_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       tracepoint_probe_register_prio+0x36/0x6d0 kernel/tracepoint.c:279
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9342
       perf_init_event kernel/events/core.c:9380 [inline]
       perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
       perf_event_alloc kernel/events/core.c:9993 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (event_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
       tp_perf_event_destroy+0x16/0x20 kernel/events/core.c:8090
       _free_event+0x330/0xe70 kernel/events/core.c:4238
       free_event+0x38/0x50 kernel/events/core.c:4265
       perf_event_release_kernel+0x364/0x880 kernel/events/core.c:4409
       perf_release+0x37/0x50 kernel/events/core.c:4435
       __fput+0x275/0x7a0 fs/file_table.c:210
       ____fput+0x16/0x20 fs/file_table.c:244
       task_work_run+0x114/0x190 kernel/task_work.c:113
       tracehook_notify_resume include/linux/tracehook.h:191 [inline]
       exit_to_usermode_loop+0x1da/0x220 arch/x86/entry/common.c:164
       prepare_exit_to_usermode arch/x86/entry/common.c:199 [inline]
       syscall_return_slowpath arch/x86/entry/common.c:270 [inline]
       do_syscall_64+0x4bc/0x640 arch/x86/entry/common.c:297
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (&event->child_mutex){+.+.}:
       check_prev_add kernel/locking/lockdep.c:1901 [inline]
       check_prevs_add kernel/locking/lockdep.c:2018 [inline]
       validate_chain kernel/locking/lockdep.c:2460 [inline]
       __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_read_group kernel/events/core.c:4536 [inline]
       __perf_read kernel/events/core.c:4618 [inline]
       perf_read+0x2c0/0x820 kernel/events/core.c:4633
       do_loop_readv_writev fs/read_write.c:695 [inline]
       do_loop_readv_writev fs/read_write.c:682 [inline]
       do_iter_read+0x3e2/0x5b0 fs/read_write.c:919
       vfs_readv+0xd3/0x130 fs/read_write.c:981
       do_readv+0x10a/0x2d0 fs/read_write.c:1014
       SYSC_readv fs/read_write.c:1101 [inline]
       SyS_readv+0x28/0x30 fs/read_write.c:1098
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
  &event->child_mutex --> pmus_lock --> &cpuctx_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&cpuctx_mutex);
                               lock(pmus_lock);
                               lock(&cpuctx_mutex);
  lock(&event->child_mutex);

 *** DEADLOCK ***

1 lock held by syz-executor071/7190:
 #0:  (&cpuctx_mutex){+.+.}, at: [<ffffffff816bd1a0>] perf_event_ctx_lock_nested+0x150/0x2c0 kernel/events/core.c:1235

stack backtrace:
CPU: 0 PID: 7190 Comm: syz-executor071 Not tainted 4.14.150 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x138/0x197 lib/dump_stack.c:53
 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
 __mutex_lock_common kernel/locking/mutex.c:756 [inline]
 __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 perf_read_group kernel/events/core.c:4536 [inline]
 __perf_read kernel/events/core.c:4618 [inline]
 perf_read+0x2c0/0x820 kernel/events/core.c:4633
 do_loop_readv_writev fs/read_write.c:695 [inline]
 do_loop_readv_writev fs/read_write.c:682 [inline]
 do_iter_read+0x3e2/0x5b0 fs/read_write.c:919
 vfs_readv+0xd3/0x130 fs/read_write.c:981
 do_readv+0x10a/0x2d0 fs/read_write.c:1014
 SYSC_readv fs/read_write.c:1101 [inline]
 SyS_readv+0x28/0x30 fs/read_write.c:1098
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x442489
RSP: 002b:00007fff64e6bc58 EFLAGS: 00000246 ORIG_RAX: 0000000000000013
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000442489
RDX: 0000000000000001 RSI: 0000000020000200 RDI: 0000000000000004
RBP: 0000000000000000 R08: 0000000000000045 R09: 0000000000000045
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000099
R13: 00000000004032b0 R14: 0000000000000000 R15: 0000000000000000

Crashes (62):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/10/29 06:51 linux-4.14.y b98aebd29824 5ea87a66 .config console log report syz C ci2-linux-4-14
2019/10/17 16:17 linux-4.14.y e132c8d7b58d 8c88c9c1 .config console log report syz C ci2-linux-4-14
2022/04/05 18:17 linux-4.14.y 74766a973637 0127c10f .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2022/04/05 16:55 linux-4.14.y 74766a973637 0127c10f .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/05/17 12:10 linux-4.14.y 7d7d1c0ab3eb a2eb125d .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/05/17 09:34 linux-4.14.y 7d7d1c0ab3eb a2eb125d .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/05/13 14:51 linux-4.14.y 7d7d1c0ab3eb ecb594cb .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/05/09 02:29 linux-4.14.y 7d7d1c0ab3eb bc5434be .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/04/23 01:44 linux-4.14.y cf256fbcbe34 590921a5 .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/04/10 13:55 linux-4.14.y 958e517f4e16 6a81331a .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/04/04 17:35 linux-4.14.y bd634aa64163 6a81331a .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/04/03 10:43 linux-4.14.y bd634aa64163 6a81331a .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/02/10 10:49 linux-4.14.y 2c8a3fceddf0 9c8b8541 .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/02/03 10:47 linux-4.14.y 2c8a3fceddf0 624dad51 .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/02/02 22:57 linux-4.14.y 2c8a3fceddf0 624dad51 .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/01/31 18:15 linux-4.14.y 2c8a3fceddf0 fc9fd31e .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/01/22 21:57 linux-4.14.y 2762b48e9611 52e37319 .config console log report info ci2-linux-4-14 possible deadlock in perf_read
2021/01/17 10:41 linux-4.14.y f79dc86058bc 813be542 .config console log report info ci2-linux-4-14
2021/01/02 09:41 linux-4.14.y 1752938529c6 79264ae3 .config console log report info ci2-linux-4-14
2020/12/23 20:36 linux-4.14.y 3f2ecb86cb90 c2c1d1dd .config console log report info ci2-linux-4-14
2020/11/24 19:06 linux-4.14.y 87335852c5d9 e34b696c .config console log report info ci2-linux-4-14
2020/11/24 05:23 linux-4.14.y 0df445b0f0da 1ab681a4 .config console log report info ci2-linux-4-14
2020/11/11 01:13 linux-4.14.y 27ce4f2a6817 cca87986 .config console log report info ci2-linux-4-14
2020/10/18 12:16 linux-4.14.y cbfa1702aaf6 fea47c01 .config console log report info ci2-linux-4-14
2020/09/28 22:30 linux-4.14.y cbfa1702aaf6 6bfdbe89 .config console log report info ci2-linux-4-14
2020/09/27 20:27 linux-4.14.y cbfa1702aaf6 5dd8aee8 .config console log report info ci2-linux-4-14
2020/09/10 05:41 linux-4.14.y 458a534cac0c ac7ca78e .config console log report ci2-linux-4-14
2020/09/08 23:42 linux-4.14.y 2f166cdcf8a9 abf9ba4f .config console log report ci2-linux-4-14
2020/07/18 15:43 linux-4.14.y b850307b279c 9c812472 .config console log report ci2-linux-4-14
2020/07/17 00:22 linux-4.14.y b850307b279c 54b3c45e .config console log report ci2-linux-4-14
2020/07/16 10:58 linux-4.14.y b850307b279c b090c643 .config console log report ci2-linux-4-14
2020/07/08 20:27 linux-4.14.y b850307b279c 9f9845eb .config console log report ci2-linux-4-14
2020/07/05 15:39 linux-4.14.y b850307b279c 22f87567 .config console log report ci2-linux-4-14
2020/07/04 13:58 linux-4.14.y b850307b279c 4f739670 .config console log report ci2-linux-4-14
2020/06/29 15:29 linux-4.14.y b850307b279c ce9ef6f2 .config console log report ci2-linux-4-14
2020/06/26 01:51 linux-4.14.y b850307b279c aea82c00 .config console log report ci2-linux-4-14
2020/06/25 23:51 linux-4.14.y b850307b279c aea82c00 .config console log report ci2-linux-4-14
2020/06/24 20:33 linux-4.14.y b850307b279c 9d60b18e .config console log report ci2-linux-4-14
2020/06/24 13:41 linux-4.14.y b850307b279c 41694dbf .config console log report ci2-linux-4-14
2020/06/22 01:49 linux-4.14.y b850307b279c 4f2acff9 .config console log report ci2-linux-4-14
2020/06/22 01:49 linux-4.14.y b850307b279c 4f2acff9 .config console log report ci2-linux-4-14
2020/06/15 06:17 linux-4.14.y b850307b279c 2a22c77a .config console log report ci2-linux-4-14
2020/06/02 19:50 linux-4.14.y 4f68020fef1c 52fd7b7d .config console log report ci2-linux-4-14
2020/05/31 06:11 linux-4.14.y 4f68020fef1c 6f3e1c7c .config console log report ci2-linux-4-14
2020/05/30 02:15 linux-4.14.y 4f68020fef1c bed08304 .config console log report ci2-linux-4-14
2020/05/28 23:02 linux-4.14.y 4f68020fef1c c7192a2f .config console log report ci2-linux-4-14
2020/05/26 21:41 linux-4.14.y a41ba30d9df2 9072c126 .config console log report ci2-linux-4-14
2020/05/13 00:13 linux-4.14.y ab9dfda23248 a44eb8f7 .config console log report ci2-linux-4-14
2020/05/07 07:24 linux-4.14.y d71f695ce745 4618eb2d .config console log report ci2-linux-4-14
2020/05/04 17:46 linux-4.14.y 773e2b1cd56a 58ae5e18 .config console log report ci2-linux-4-14
2020/02/27 07:26 linux-4.14.y 98db2bf27b9e 251aabb7 .config console log report ci2-linux-4-14
2020/01/04 22:41 linux-4.14.y 84f5ad468100 68256974 .config console log report ci2-linux-4-14
2019/12/28 13:17 linux-4.14.y e1f7d50ae3a3 af6b8ef8 .config console log report ci2-linux-4-14
2019/12/14 17:45 linux-4.14.y a844dc4c5442 eef6e580 .config console log report ci2-linux-4-14
2019/11/11 14:07 linux-4.14.y c9fda4f22428 dc438b91 .config console log report ci2-linux-4-14
2019/10/17 14:57 linux-4.14.y e132c8d7b58d 8c88c9c1 .config console log report ci2-linux-4-14
2019/10/12 06:46 linux-4.14.y e132c8d7b58d 426631dd .config console log report ci2-linux-4-14
2019/10/10 18:33 linux-4.14.y 42327896f194 d52eff28 .config console log report ci2-linux-4-14
2019/10/03 04:18 linux-4.14.y f6e27dbb1afa 2e29b534 .config console log report ci2-linux-4-14
2019/09/21 19:48 linux-4.14.y f6e27dbb1afa d96e88f3 .config console log report ci2-linux-4-14
2019/09/21 19:47 linux-4.14.y f6e27dbb1afa d96e88f3 .config console log report ci2-linux-4-14
2019/07/25 18:46 linux-4.14.y ff33472c282e 0d7a1249 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.