syzbot


INFO: rcu detected stall in process_one_work (8)

Status: auto-obsoleted due to no activity on 2024/03/18 23:07
Subsystems: kernel
[Documentation on labels]
First crash: 180d, last: 130d
Similar bugs (7)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in process_one_work 2 14d 51d 0/3 upstream: reported on 2024/03/08 03:59
upstream INFO: rcu detected stall in process_one_work (2) kernel 1 1477d 1477d 0/26 auto-closed as invalid on 2020/07/10 22:43
upstream INFO: rcu detected stall in process_one_work (4) kernel 1 952d 952d 0/26 auto-closed as invalid on 2021/12/18 04:26
upstream INFO: rcu detected stall in process_one_work kernel 8 1717d 2159d 0/26 auto-closed as invalid on 2019/11/14 14:39
upstream INFO: rcu detected stall in process_one_work (5) kernel 1 597d 597d 0/26 auto-obsoleted due to no activity on 2022/12/07 18:00
linux-6.1 INFO: rcu detected stall in process_one_work origin:upstream C 1 7d20h 109d 0/3 upstream: reported C repro on 2024/01/09 18:19
upstream INFO: rcu detected stall in process_one_work (3) kernel C done done 1 1299d 1341d 15/26 fixed on 2020/11/16 12:12

Sample crash report:
ip6_tunnel: ip6tnl1 xmit: Local address not yet configured!
ip6_tunnel: ip6tnl3 xmit: Local address not yet configured!
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	1-...!: (1 GPs behind) idle=1a2c/1/0x4000000000000000 softirq=52862/52863 fqs=56
rcu: 	(detected by 0, t=10502 jiffies, g=94493, q=257 ncpus=2)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 11438 Comm: kworker/1:1 Not tainted 6.7.0-rc6-syzkaller-00022-g55cb5f43689d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
Workqueue: events_freezable_power_ sync_hw_clock
RIP: 0010:__sanitizer_cov_trace_pc+0x59/0x60 kernel/kcov.c:225
Code: 82 d8 15 00 00 83 f8 02 75 20 48 8b 8a e0 15 00 00 8b 92 dc 15 00 00 48 8b 01 48 83 c0 01 48 39 d0 73 07 48 89 01 48 89 34 c1 <c3> 66 0f 1f 44 00 00 f3 0f 1e fa 41 57 41 56 49 89 d6 41 55 41 54
RSP: 0018:ffffc900003f8e38 EFLAGS: 00000046
RAX: 0000000080010001 RBX: ffff8880b992ba40 RCX: ffffffff81751a5f
RDX: ffff88802537c0c0 RSI: ffffffff817515a9 RDI: 0000000000000001
RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000002 R12: ffff8880b992b940
R13: 18611ccef77f3968 R14: ffff8880480ba340 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd49961e000 CR3: 000000007c38b000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1657 [inline]
 __hrtimer_run_queues+0x4d9/0xc20 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x31b/0x800 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1065 [inline]
 __sysvec_apic_timer_interrupt+0x105/0x400 arch/x86/kernel/apic/apic.c:1082
 sysvec_apic_timer_interrupt+0x90/0xb0 arch/x86/kernel/apic/apic.c:1076
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:lock_acquire+0x1ef/0x520 kernel/locking/lockdep.c:5722
Code: c1 05 4d 95 9b 7e 83 f8 01 0f 85 b4 02 00 00 9c 58 f6 c4 02 0f 85 9f 02 00 00 48 85 ed 74 01 fb 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 8b 84 24
RSP: 0018:ffffc9000abf7c00 EFLAGS: 00000206
RAX: dffffc0000000000 RBX: 1ffff9200157ef82 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffffff8acca240 RDI: ffffffff8b2ed3e0
RBP: 0000000000000200 R08: 0000000000000001 R09: fffffbfff21b93c2
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: ffff88801307b538 R15: 0000000000000000
 process_one_work+0x7cd/0x15d0 kernel/workqueue.c:2602
 process_scheduled_works kernel/workqueue.c:2700 [inline]
 worker_thread+0x8b9/0x1290 kernel/workqueue.c:2781
 kthread+0x2c6/0x3a0 kernel/kthread.c:388
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:242
 </TASK>
rcu: rcu_preempt kthread starved for 10220 jiffies! g94493 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28240 pid:16    tgid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5376 [inline]
 __schedule+0xedb/0x5af0 kernel/sched/core.c:6688
 __schedule_loop kernel/sched/core.c:6763 [inline]
 schedule+0xe9/0x270 kernel/sched/core.c:6778
 schedule_timeout+0x137/0x290 kernel/time/timer.c:2167
 rcu_gp_fqs_loop+0x1ec/0xb10 kernel/rcu/tree.c:1631
 rcu_gp_kthread+0x24b/0x380 kernel/rcu/tree.c:1830
 kthread+0x2c6/0x3a0 kernel/kthread.c:388
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:242
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 0 PID: 5159 Comm: kworker/0:5 Not tainted 6.7.0-rc6-syzkaller-00022-g55cb5f43689d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
Workqueue: events bpf_prog_free_deferred
RIP: 0010:csd_lock_wait kernel/smp.c:311 [inline]
RIP: 0010:smp_call_function_many_cond+0x4e4/0x1550 kernel/smp.c:855
Code: 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 fe b6 0b 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 24 0e 00 00 8b 43 08 31
RSP: 0018:ffffc9000365f870 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b99414e0 RCX: ffffffff817aaaa8
RDX: ffff888079e22040 RSI: ffffffff817aaa82 RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000003 R12: ffffed101732829d
R13: 0000000000000001 R14: ffff8880b99414e8 R15: ffff8880b983d900
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000564881431dd8 CR3: 000000007c38b000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x40/0x90 kernel/smp.c:1023
 __purge_vmap_area_lazy+0x203/0x2170 mm/vmalloc.c:1749
 _vm_unmap_aliases+0x693/0x9b0 mm/vmalloc.c:2306
 vm_reset_perms mm/vmalloc.c:2753 [inline]
 vfree+0x533/0x7a0 mm/vmalloc.c:2832
 bpf_prog_unlock_free include/linux/filter.h:924 [inline]
 bpf_jit_free include/linux/filter.h:1210 [inline]
 bpf_prog_free_deferred+0x534/0x720 kernel/bpf/core.c:2744
 process_one_work+0x886/0x15d0 kernel/workqueue.c:2627
 process_scheduled_works kernel/workqueue.c:2700 [inline]
 worker_thread+0x8b9/0x1290 kernel/workqueue.c:2781
 kthread+0x2c6/0x3a0 kernel/kthread.c:388
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:242
 </TASK>

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/12/19 23:04 upstream 55cb5f43689d 3ad490ea .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in process_one_work
2023/11/06 07:45 linux-next e27090b1413f 500bfdc4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: rcu detected stall in process_one_work
2023/10/31 04:08 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing c70793fb7632 b5729d82 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in process_one_work
* Struck through repros no longer work on HEAD.