syzbot


INFO: rcu detected stall in call_usermodehelper_exec_work (2)

Status: closed as invalid on 2022/02/08 10:00
Subsystems: kernel
[Documentation on labels]
First crash: 861d, last: 851d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in call_usermodehelper_exec_work cgroups mm 1 1541d 1541d 0/26 closed as invalid on 2020/01/09 08:13

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 GPs behind) idle=21d/1/0x4000000000000000 softirq=98348/98349 fqs=0 
	(detected by 1, t=10502 jiffies, g=171413, q=24)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 12675 Comm: kworker/u4:7 Not tainted 5.16.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound call_usermodehelper_exec_work
RIP: 0010:rcu_lockdep_current_cpu_online+0x7/0x150 kernel/rcu/tree.c:1166
Code: 4c 24 30 4c 8b 54 24 28 48 8b 54 24 20 48 8b 44 24 18 e9 41 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 65 8b 15 69 51 9f 7e <81> e2 00 00 f0 00 b8 01 00 00 00 75 0a 8b 15 1a 0e 2e 0c 85 d2 75
RSP: 0018:ffffc90000007cb8 EFLAGS: 00000002
RAX: 0000000000000001 RBX: 1ffff92000000f9c RCX: ffffffff815cb4b8
RDX: 0000000000010003 RSI: 0000000000010004 RDI: ffffffff8b5628a0
RBP: 0000000000000001 R08: 0000000000000000 R09: ffffffff8d912bd7
R10: fffffbfff1b2257a R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: ffff8880459dc300 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32625000 CR3: 000000004072a000 CR4: 0000000000350ef0
DR0: 0700000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 rcu_read_lock_held_common kernel/rcu/update.c:112 [inline]
 rcu_read_lock_held_common kernel/rcu/update.c:102 [inline]
 rcu_read_lock_sched_held+0x25/0x70 kernel/rcu/update.c:123
 trace_lock_acquire include/trace/events/lock.h:13 [inline]
 lock_acquire+0x442/0x510 kernel/locking/lockdep.c:5608
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:349 [inline]
 advance_sched+0x53/0x9a0 net/sched/sch_taprio.c:714
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x609/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:___slab_alloc+0x791/0xfe0 mm/slub.c:3088
Code: 48 89 df e8 51 9c ff ff 41 81 e5 00 02 00 00 0f 85 c0 04 00 00 9c 58 f6 c4 02 0f 85 73 05 00 00 4d 85 ed 74 01 fb 48 8b 45 c8 <65> 48 2b 04 25 28 00 00 00 0f 85 b1 06 00 00 48 8d 65 d0 4c 89 f0
RSP: 0018:ffffc90002a5f7c0 EFLAGS: 00000206
RAX: ee31e0a37e098900 RBX: ffff8880b9c40600 RCX: 1ffffffff1ff15e6
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90002a5f8a0 R08: 0000000000000001 R09: ffffffff8ff72a47
R10: 0000000000000001 R11: 000000000008808a R12: 0000000000000000
R13: 0000000000000200 R14: ffff888030f44a00 R15: 0000000000040600
 __slab_alloc.constprop.0+0x4d/0xa0 mm/slub.c:3109
 slab_alloc_node mm/slub.c:3200 [inline]
 slab_alloc mm/slub.c:3242 [inline]
 kmem_cache_alloc+0x35c/0x3a0 mm/slub.c:3247
 copy_sighand kernel/fork.c:1593 [inline]
 copy_process+0x2354/0x75a0 kernel/fork.c:2185
 kernel_clone+0xe7/0xab0 kernel/fork.c:2582
 kernel_thread+0xb5/0xf0 kernel/fork.c:2634
 call_usermodehelper_exec_work kernel/umh.c:174 [inline]
 call_usermodehelper_exec_work+0xcc/0x180 kernel/umh.c:160
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: rcu_preempt kthread starved for 10502 jiffies! g171413 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:27528 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4972 [inline]
 __schedule+0xa9a/0x4940 kernel/sched/core.c:6253
 schedule+0xd2/0x260 kernel/sched/core.c:6326
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2128
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 12768 Comm: syz-executor.1 Not tainted 5.16.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_check_gp_kthread_starvation.cold+0x1fb/0x200 kernel/rcu/tree_stall.h:481
 print_other_cpu_stall kernel/rcu/tree_stall.h:586 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq+0x2125/0x2200 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1421
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x452/0xc20 kernel/smp.c:969
Code: 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 80 75 0b 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 33 06 00 00 8b 43 08 31
RSP: 0018:ffffc90009eefc08 EFLAGS: 00000246
RAX: 0000000000040000 RBX: ffff8880b9c41d40 RCX: ffffc900049b1000
RDX: 0000000000040000 RSI: ffffffff816c23e0 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff816c2406 R11: 0000000000000000 R12: ffffed10173883a9
R13: 0000000000000000 R14: ffff8880b9c41d48 R15: 0000000000000001
 clock_was_set+0x599/0x790 kernel/time/hrtimer.c:974
 do_settimeofday64 kernel/time/timekeeping.c:1327 [inline]
 do_settimeofday64+0x3e2/0x5c0 kernel/time/timekeeping.c:1293
 do_sys_settimeofday64 kernel/time/time.c:195 [inline]
 do_sys_settimeofday64+0x1de/0x260 kernel/time/time.c:169
 __do_sys_clock_settime kernel/time/posix-timers.c:1079 [inline]
 __se_sys_clock_settime kernel/time/posix-timers.c:1067 [inline]
 __x64_sys_clock_settime+0x1a1/0x280 kernel/time/posix-timers.c:1067
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7fcb52fe6ae9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fcb5055c188 EFLAGS: 00000246 ORIG_RAX: 00000000000000e3
RAX: ffffffffffffffda RBX: 00007fcb530f9f60 RCX: 00007fcb52fe6ae9
RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000000
RBP: 00007fcb53040f6d R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffeedb4263f R14: 00007fcb5055c300 R15: 0000000000022000
 </TASK>
vkms_vblank_simulate: vblank timer overrun
vkms_vblank_simulate: vblank timer overrun
----------------
Code disassembly (best guess):
   0:	4c 24 30             	rex.WR and $0x30,%al
   3:	4c 8b 54 24 28       	mov    0x28(%rsp),%r10
   8:	48 8b 54 24 20       	mov    0x20(%rsp),%rdx
   d:	48 8b 44 24 18       	mov    0x18(%rsp),%rax
  12:	e9 41 fe ff ff       	jmpq   0xfffffe58
  17:	66 66 2e 0f 1f 84 00 	data16 nopw %cs:0x0(%rax,%rax,1)
  1e:	00 00 00 00
  22:	90                   	nop
  23:	65 8b 15 69 51 9f 7e 	mov    %gs:0x7e9f5169(%rip),%edx        # 0x7e9f5193
* 2a:	81 e2 00 00 f0 00    	and    $0xf00000,%edx <-- trapping instruction
  30:	b8 01 00 00 00       	mov    $0x1,%eax
  35:	75 0a                	jne    0x41
  37:	8b 15 1a 0e 2e 0c    	mov    0xc2e0e1a(%rip),%edx        # 0xc2e0e57
  3d:	85 d2                	test   %edx,%edx
  3f:	75                   	.byte 0x75

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/11/28 10:29 upstream 3498e7f2bb41 63eeac02 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in call_usermodehelper_exec_work
2021/11/18 17:48 bpf-next dd7f091fd22b 31a30fc0 .config console log report info ci-upstream-bpf-next-kasan-gce INFO: rcu detected stall in call_usermodehelper_exec_work
* Struck through repros no longer work on HEAD.