syzbot


INFO: rcu detected stall in flush_memcg_stats_work

Status: auto-closed as invalid on 2021/11/14 09:36
Reported-by: syzbot+@syzkaller.appspotmail.com
First crash: 440d, last: 440d

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 GPs behind) idle=c15/1/0x4000000000000000 softirq=95129/95130 fqs=15 
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P4632/1:b..l
	(detected by 1, t=10502 jiffies, g=174261, q=248)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 148 Comm: kworker/u4:2 Not tainted 5.15.0-rc1-next-20210915-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound flush_memcg_stats_work
RIP: 0010:rcu_lockdep_current_cpu_online kernel/rcu/tree.c:1169 [inline]
RIP: 0010:rcu_lockdep_current_cpu_online+0x28/0x150 kernel/rcu/tree.c:1160
Code: 00 90 65 8b 15 a9 11 a0 7e 81 e2 00 00 f0 00 b8 01 00 00 00 75 0a 8b 15 62 68 0c 0c 85 d2 75 01 c3 55 53 65 ff 05 88 11 a0 7e <e8> 63 fc d0 07 48 c7 c3 c0 28 03 00 83 f8 07 89 c5 0f 87 f5 00 00
RSP: 0018:ffffc90000007cd0 EFLAGS: 00000086
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff8161d084
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff8880b9c32a08
RBP: 1ffff92000000fa1 R08: 0000000000000000 R09: ffff8880b9c32a0b
R10: ffffed1017386541 R11: 0000000000000000 R12: 0000000000000001
R13: 173ea2e1ae7d45c3 R14: ffff888018518aa0 R15: 173ea2e1ae7d45c3
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004e4bc5 CR3: 000000003ba7c000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 rcu_read_lock_held_common kernel/rcu/update.c:112 [inline]
 rcu_read_lock_held_common kernel/rcu/update.c:102 [inline]
 rcu_read_lock_sched_held+0x25/0x70 kernel/rcu/update.c:123
 trace_lock_release include/trace/events/lock.h:58 [inline]
 lock_release+0x522/0x720 kernel/locking/lockdep.c:5636
 __raw_spin_unlock include/linux/spinlock_api_smp.h:150 [inline]
 _raw_spin_unlock+0x12/0x40 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:408 [inline]
 advance_sched+0x435/0x9a0 net/sched/sch_taprio.c:759
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x609/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 aa e4 25 f8 48 89 ef e8 f2 5a 26 f8 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 <e8> f3 11 1a f8 65 8b 05 6c 6a cc 76 85 c0 74 0a 5b 5d c3 e8 80 bb
RSP: 0018:ffffc900012afcf8 EFLAGS: 00000206
RAX: 0000000000000006 RBX: 0000000000000200 RCX: 1ffffffff1fa2a26
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000001
RBP: ffffffff8b9c97a0 R08: 0000000000000001 R09: ffffffff8fcfb987
R10: 0000000000000001 R11: 0000000000000000 R12: ffffc900012afdb0
R13: ffffffff8bad71f0 R14: ffff8881411a8300 R15: ffff888010c69800
 mem_cgroup_flush_stats mm/memcontrol.c:5366 [inline]
 mem_cgroup_flush_stats mm/memcontrol.c:5361 [inline]
 flush_memcg_stats_work+0x39/0x50 mm/memcontrol.c:5378
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2297
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2444
 kthread+0x3e5/0x4d0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
task:syz-executor.5  state:R  running task     stack:29032 pid: 4632 ppid:  6949 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:4955 [inline]
 __schedule+0x940/0x26f0 kernel/sched/core.c:6236
 preempt_schedule_irq+0x4e/0x90 kernel/sched/core.c:6636
 irqentry_exit+0x31/0x80 kernel/entry/common.c:427
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:lock_acquire+0x1ef/0x510 kernel/locking/lockdep.c:5593
Code: 87 a6 7e 83 f8 01 0f 85 b4 02 00 00 9c 58 f6 c4 02 0f 85 9f 02 00 00 48 83 7c 24 08 00 74 01 fb 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 8b 84 24
RSP: 0018:ffffc9000cf4f978 EFLAGS: 00000206
RAX: dffffc0000000000 RBX: 1ffff920019e9f31 RCX: b060b6157ab86be6
RDX: 1ffff11011cb985f RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8fcfb947
R10: fffffbfff1f9f728 R11: 0000000000000000 R12: 0000000000000002
R13: 0000000000000000 R14: ffffffff8b97ffa0 R15: 0000000000000000
 rcu_lock_acquire include/linux/rcupdate.h:268 [inline]
 rcu_read_lock include/linux/rcupdate.h:688 [inline]
 ext4_get_group_desc+0x146/0x500 fs/ext4/balloc.c:284
 recently_deleted fs/ext4/ialloc.c:685 [inline]
 find_inode_bit+0x1c4/0x580 fs/ext4/ialloc.c:731
 __ext4_new_inode+0xc64/0x5ba0 fs/ext4/ialloc.c:1066
 ext4_symlink+0x485/0xd40 fs/ext4/namei.c:3293
 vfs_symlink fs/namei.c:4250 [inline]
 vfs_symlink+0x479/0x730 fs/namei.c:4235
 do_symlinkat+0x261/0x2e0 fs/namei.c:4279
 __do_sys_symlink fs/namei.c:4301 [inline]
 __se_sys_symlink fs/namei.c:4299 [inline]
 __x64_sys_symlink+0x75/0x90 fs/namei.c:4299
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x465f07
RSP: 002b:00007fff05b944b8 EFLAGS: 00000206 ORIG_RAX: 0000000000000058
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000465f07
RDX: 00007fff05b945a7 RSI: 00000000004bfd65 RDI: 00007fff05b94590
RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fff05b94350
R10: 00007fff05b94207 R11: 0000000000000206 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000001 R15: 00007fff05b94590
rcu: rcu_preempt kthread starved for 10472 jiffies! g174261 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28696 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:4955 [inline]
 __schedule+0x940/0x26f0 kernel/sched/core.c:6236
 schedule+0xd3/0x270 kernel/sched/core.c:6315
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2128
 kthread+0x3e5/0x4d0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 4638 Comm: syz-executor.0 Not tainted 5.15.0-rc1-next-20210915-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1ae/0x220 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_check_gp_kthread_starvation.cold+0x1fb/0x200 kernel/rcu/tree_stall.h:481
 print_other_cpu_stall kernel/rcu/tree_stall.h:586 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq+0x2125/0x2200 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1421
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x447/0xc90 kernel/smp.c:969
Code: 0b 00 85 ed 74 4d 48 b8 00 00 00 00 00 fc ff df 4d 89 f4 4c 89 f5 49 c1 ec 03 83 e5 07 49 01 c4 83 c5 03 e8 8b 44 0b 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 95 06 00 00 8b 43 08 31
RSP: 0018:ffffc9000d20f708 EFLAGS: 00000246
RAX: 0000000000040000 RBX: ffff8880b9c39ce0 RCX: ffffc90002e81000
RDX: 0000000000040000 RSI: ffffffff816ad155 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff816ad17b R11: 0000000000000000 R12: ffffed101738739d
R13: 0000000000000000 R14: ffff8880b9c39ce8 R15: 0000000000000001
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1135
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:929 [inline]
 text_poke_bp_batch+0x1b3/0x560 arch/x86/kernel/alternative.c:1114
 text_poke_bp+0xad/0x110 arch/x86/kernel/alternative.c:1314
 arch_static_call_transform+0x48/0x90 arch/x86/kernel/static_call.c:101
 __static_call_update+0xcf/0x620 kernel/static_call.c:136
 tracepoint_update_call kernel/tracepoint.c:317 [inline]
 tracepoint_add_func+0xa18/0xe50 kernel/tracepoint.c:358
 tracepoint_probe_register_prio kernel/tracepoint.c:511 [inline]
 tracepoint_probe_register+0xa1/0xf0 kernel/tracepoint.c:531
 trace_event_reg+0x28f/0x350 kernel/trace/trace_events.c:513
 perf_trace_event_reg kernel/trace/trace_event_perf.c:129 [inline]
 perf_trace_event_init+0x549/0xa20 kernel/trace/trace_event_perf.c:204
 perf_trace_init+0x1a0/0x2f0 kernel/trace/trace_event_perf.c:228
 perf_tp_event_init+0xa2/0x120 kernel/events/core.c:9735
 perf_try_init_event+0x12a/0x560 kernel/events/core.c:11193
 perf_init_event kernel/events/core.c:11257 [inline]
 perf_event_alloc.part.0+0xf16/0x3b10 kernel/events/core.c:11548
 perf_event_alloc kernel/events/core.c:11971 [inline]
 __do_sys_perf_event_open+0x4ae/0x3130 kernel/events/core.c:12069
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4665f9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f83d81df188 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 00000000004665f9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000200
RBP: 00000000004bfcc4 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 000000000056bf80
R13: 00007ffc2769eccf R14: 00007f83d81df300 R15: 0000000000022000
----------------
Code disassembly (best guess):
   0:	00 90 65 8b 15 a9    	add    %dl,-0x56ea749b(%rax)
   6:	11 a0 7e 81 e2 00    	adc    %esp,0xe2817e(%rax)
   c:	00 f0                	add    %dh,%al
   e:	00 b8 01 00 00 00    	add    %bh,0x1(%rax)
  14:	75 0a                	jne    0x20
  16:	8b 15 62 68 0c 0c    	mov    0xc0c6862(%rip),%edx        # 0xc0c687e
  1c:	85 d2                	test   %edx,%edx
  1e:	75 01                	jne    0x21
  20:	c3                   	retq
  21:	55                   	push   %rbp
  22:	53                   	push   %rbx
  23:	65 ff 05 88 11 a0 7e 	incl   %gs:0x7ea01188(%rip)        # 0x7ea011b2
* 2a:	e8 63 fc d0 07       	callq  0x7d0fc92 <-- trapping instruction
  2f:	48 c7 c3 c0 28 03 00 	mov    $0x328c0,%rbx
  36:	83 f8 07             	cmp    $0x7,%eax
  39:	89 c5                	mov    %eax,%ebp
  3b:	0f                   	.byte 0xf
  3c:	87 f5                	xchg   %esi,%ebp

Crashes (1):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-linux-next-kasan-gce-root 2021/09/15 09:33 linux-next 70ced02f322f 07e953c1 .config log report info INFO: rcu detected stall in flush_memcg_stats_work
* Struck through repros no longer work on HEAD.