syzbot


INFO: rcu detected stall in sys_perf_event_open (2)

Status: upstream: reported syz repro on 2024/09/23 05:55
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+5284a86a0b0a31ab266a@syzkaller.appspotmail.com
First crash: 138d, last: 28d
Fix commit to backport (bisect log) :
tree: upstream
commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5
Author: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Date: Fri Apr 7 23:05:26 2023 +0000

  softirq: Add trace points for tasklet entry/exit

  
Bug presence (2)
Date Name Commit Repro Result
2024/11/22 linux-6.1.y (ToT) b67dc5c9ade9 C [report] INFO: rcu detected stall in smp_call_function
2024/11/22 upstream (ToT) 28eb75e178d3 C Didn't crash
Similar bugs (10)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in sys_perf_event_open (2) origin:lts-only C error 5 17d 396d 0/3 upstream: reported C repro on 2024/01/09 18:19
upstream INFO: rcu detected stall in sys_perf_event_open (3) perf 1 1859d 1859d 0/28 closed as invalid on 2020/01/08 05:23
upstream INFO: rcu detected stall in sys_perf_event_open (4) perf 1 1858d 1858d 0/28 closed as invalid on 2020/01/09 08:13
upstream INFO: rcu detected stall in sys_perf_event_open (2) perf 8 1893d 1895d 0/28 closed as invalid on 2019/12/04 14:14
linux-6.1 INFO: rcu detected stall in sys_perf_event_open 1 243d 243d 0/3 auto-obsoleted due to no activity on 2024/09/19 01:45
linux-5.15 INFO: rcu detected stall in sys_perf_event_open 1 661d 661d 0/3 auto-obsoleted due to no activity on 2023/08/17 04:33
upstream INFO: rcu detected stall in sys_perf_event_open perf 1 2338d 2336d 0/28 auto-closed as invalid on 2019/03/14 17:19
android-5-15 BUG: soft lockup in sys_perf_event_open 1 278d 278d 0/2 auto-obsoleted due to no activity on 2024/08/04 07:07
upstream BUG: soft lockup in sys_perf_event_open perf 1 254d 250d 0/28 auto-obsoleted due to no activity on 2024/08/29 01:52
upstream BUG: soft lockup in sys_perf_event_open (2) kvm fs 6 21d 129d 0/28 moderation: reported on 2024/10/03 01:56
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2024/12/03 12:42 7h31m fix candidate upstream OK (1) job log

Sample crash report:
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-.... } 2641 jiffies s: 27773 root: 0x1/.
rcu: blocking rcu_node structures (internal RCU debug):
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 17658 Comm: syz.1.6096 Not tainted 6.1.118-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:__lock_acquire+0xc8d/0x1f80 kernel/locking/lockdep.c:5019
Code: 00 00 e8 56 5d 00 00 31 f6 85 c0 0f 84 8f 08 00 00 48 c7 c0 60 81 33 97 48 c1 e8 03 48 bf 00 00 00 00 00 fc ff df 0f b6 04 38 <84> c0 0f 85 ea 0b 00 00 83 3d d4 dc c8 15 00 75 39 48 63 9c 24 80
RSP: 0018:ffffc900000070c0 EFLAGS: 00000802
RAX: 0000000000000004 RBX: ffffffff90bbb2d8 RCX: ffffffff816b024a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: dffffc0000000000
RBP: ffff88801f672918 R08: dffffc0000000000 R09: fffffbfff224624d
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff88801f672898
R13: ffff88801f671dc0 R14: ffff88801f672938 R15: 1ffff11003ece527
FS:  00007f67eb3546c0(0000) GS:ffff8880b8e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b3225ffff CR3: 000000002d2d1000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
 rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 rcu_read_lock include/linux/rcupdate.h:791 [inline]
 __perf_output_begin kernel/events/ring_buffer.c:163 [inline]
 perf_output_begin_forward+0xcf/0xaf0 kernel/events/ring_buffer.c:271
 __perf_event_output kernel/events/core.c:7708 [inline]
 perf_event_output_forward+0x1e5/0x330 kernel/events/core.c:7726
 __perf_event_overflow+0x45e/0x640 kernel/events/core.c:9472
 perf_tp_event+0x35a/0xc80 kernel/events/core.c:10005
 perf_trace_run_bpf_submit+0xf7/0x1d0 kernel/events/core.c:9978
 perf_trace_preemptirq_template+0x2d1/0x3d0 include/trace/events/preemptirq.h:14
 trace_irq_enable_rcuidle+0xd5/0x140 include/trace/events/preemptirq.h:40
 trace_hardirqs_on+0x24/0x40 kernel/trace/trace_preemptirq.c:44
 asm_sysvec_irq_work+0x16/0x20 arch/x86/include/asm/idtentry.h:728
RIP: 0010:rcu_read_unlock_special+0x84/0x540 kernel/rcu/tree_plugin.h:682
Code: f1 f1 f1 00 f2 f2 f2 49 89 04 17 66 41 c7 44 17 09 f3 f3 41 c6 44 17 0b f3 65 44 8b 25 7d 28 8e 7e 41 f7 c4 00 00 f0 00 74 45 <48> c7 44 24 20 0e 36 e0 45 4a c7 04 3a 00 00 00 00 66 42 c7 44 3a
RSP: 0018:ffffc90000007ac0 EFLAGS: 00000206
RAX: ad59148710775900 RBX: 1ffff92000000f60 RCX: ffffffff816b024a
RDX: dffffc0000000000 RSI: ffffffff8b0c01c0 RDI: ffffffff8b5d7800
RBP: ffffc90000007ba0 R08: dffffc0000000000 R09: fffffbfff2246255
R10: 0000000000000000 R11: dffffc0000000001 R12: ffffffff8d32fe00
R13: ffff88801f672218 R14: ffffc90000007b00 R15: 1ffff92000000f5c
 __rcu_read_unlock+0x92/0x100 kernel/rcu/tree_plugin.h:426
 rcu_read_unlock include/linux/rcupdate.h:823 [inline]
 ieee80211_iterate_active_interfaces_atomic+0x153/0x170 net/mac80211/util.c:826
 mac80211_hwsim_beacon+0xd0/0x1e0 drivers/net/wireless/mac80211_hwsim.c:2142
 __run_hrtimer kernel/time/hrtimer.c:1689 [inline]
 __hrtimer_run_queues+0x5e5/0xe50 kernel/time/hrtimer.c:1753
 hrtimer_run_softirq+0x196/0x2c0 kernel/time/hrtimer.c:1770
 handle_softirqs+0x2ee/0xa40 kernel/softirq.c:571
 __do_softirq kernel/softirq.c:605 [inline]
 invoke_softirq kernel/softirq.c:445 [inline]
 __irq_exit_rcu+0x157/0x240 kernel/softirq.c:654
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:666
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1118 [inline]
 sysvec_apic_timer_interrupt+0xa0/0xc0 arch/x86/kernel/apic/apic.c:1118
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:691
RIP: 0010:lock_acquire+0x26f/0x5a0 kernel/locking/lockdep.c:5666
Code: 2b 00 74 08 4c 89 f7 e8 ef 7b 77 00 f6 44 24 61 02 0f 85 84 01 00 00 41 f7 c7 00 02 00 00 74 01 fb 48 c7 44 24 40 0e 36 e0 45 <4b> c7 44 25 00 00 00 00 00 43 c7 44 25 09 00 00 00 00 43 c7 44 25
RSP: 0018:ffffc90005be7a60 EFLAGS: 00000206
RAX: 0000000000000001 RBX: 1ffff92000b7cf58 RCX: ffffffff816b4e82
RDX: dffffc0000000000 RSI: ffffffff8b0c1340 RDI: ffffffff8b5d7800
RBP: ffffc90005be7ba8 R08: dffffc0000000000 R09: fffffbfff2246255
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff92000b7cf54
R13: dffffc0000000000 R14: ffffc90005be7ac0 R15: 0000000000000246
 __mutex_lock_common kernel/locking/mutex.c:603 [inline]
 __mutex_lock+0x132/0xd80 kernel/locking/mutex.c:747
 __do_sys_perf_event_open kernel/events/core.c:12749 [inline]
 __se_sys_perf_event_open+0x43ff/0x4960 kernel/events/core.c:12331
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:81
 entry_SYSCALL_64_after_hwframe+0x68/0xd2
RIP: 0033:0x7f67ea57e819
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f67eb354038 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 00007f67ea735fa0 RCX: 00007f67ea57e819
RDX: ffffffffffffffff RSI: 0000000000000000 RDI: 0000000020000480
RBP: 00007f67ea5f175e R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f67ea735fa0 R15: 00007ffce8622a08
 </TASK>

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/11/22 03:43 linux-6.1.y b67dc5c9ade9 4b25d554 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf INFO: rcu detected stall in sys_perf_event_open
2025/01/12 02:40 linux-6.1.y c63962be84ef 6dbc6a9b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf INFO: rcu detected stall in sys_perf_event_open
2024/09/23 05:54 linux-6.1.y e526b12bf916 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf INFO: rcu detected stall in sys_perf_event_open
* Struck through repros no longer work on HEAD.