INFO: rcu detected stall in iterate_cleanup

Kernel	Title	Rank 🛈	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in iterate_cleanup_work (2) netfilter	1	C	done	inconclusive	4	1097d	1853d	0/29	upstream: reported C repro on 2020/07/07 07:23

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: (detected by 1, t=10502 jiffies, g=6893, q=113) rcu: All QSes seen, last rcu_preempt kthread activity 10503 (4294955133-4294944630), jiffies_till_next_fqs=1, root ->qsmask 0x0 kworker/1:8 R running task 28712 2703 2 0x80004008 Workqueue: events iterate_cleanup_work Call Trace: <IRQ> sched_show_task+0x40f/0x560 kernel/sched/core.c:5954 print_other_cpu_stall kernel/rcu/tree_stall.h:410 [inline] check_cpu_stall kernel/rcu/tree_stall.h:538 [inline] rcu_pending kernel/rcu/tree.c:2827 [inline] rcu_sched_clock_irq+0x1861/0x1ad0 kernel/rcu/tree.c:2271 update_process_times+0x12d/0x180 kernel/time/timer.c:1726 tick_sched_handle kernel/time/tick-sched.c:167 [inline] tick_sched_timer+0x263/0x420 kernel/time/tick-sched.c:1310 __run_hrtimer kernel/time/hrtimer.c:1517 [inline] __hrtimer_run_queues+0x403/0x840 kernel/time/hrtimer.c:1579 hrtimer_interrupt+0x38c/0xda0 kernel/time/hrtimer.c:1641 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1110 [inline] smp_apic_timer_interrupt+0x109/0x280 arch/x86/kernel/apic/apic.c:1135 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829 </IRQ> RIP: 0010:__memcg_kmem_uncharge+0x24/0x1a0 mm/memcontrol.c:2909 Code: 1f 80 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 50 89 f3 49 89 fc 48 b8 00 00 00 00 00 fc ff df 4c 8d 77 38 4d 89 f5 <49> c1 ed 03 41 80 7c 05 00 00 74 08 4c 89 f7 e8 48 07 fb ff 4d 8b RSP: 0018:ffffc900078a78a8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 RAX: dffffc0000000000 RBX: 0000000000000000 RCX: ffff88809fcf2000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffea000255ddc0 RBP: ffffc900078a78d8 R08: 000000000003a728 R09: ffffed1012c55749 R10: ffffed1012c55749 R11: 0000000000000000 R12: ffffea000255ddc0 R13: ffffea000255ddf8 R14: ffffea000255ddf8 R15: ffff8880962aba38 memcg_kmem_uncharge include/linux/memcontrol.h:1402 [inline] free_thread_stack+0x12e/0x590 kernel/fork.c:284 release_task_stack kernel/fork.c:440 [inline] put_task_stack+0xa3/0x130 kernel/fork.c:451 finish_task_switch+0x3f1/0x550 kernel/sched/core.c:3256 context_switch kernel/sched/core.c:3388 [inline] __schedule+0x9a8/0xcc0 kernel/sched/core.c:4081 preempt_schedule_irq+0xc1/0x140 kernel/sched/core.c:4338 retint_kernel+0x1b/0x2b RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:752 [inline] RIP: 0010:lock_is_held_type+0x25c/0x2b0 kernel/locking/lockdep.c:4524 Code: 68 95 2a 89 48 c1 e8 03 42 80 3c 30 00 74 0c 48 c7 c7 68 95 2a 89 e8 33 50 57 00 48 83 3d c3 be ce 07 00 74 56 4c 89 e7 57 9d <0f> 1f 44 00 00 89 d8 48 83 c4 28 5b 41 5c 41 5d 41 5e 41 5f 5d c3 RSP: 0018:ffffc900078a7b30 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 RAX: 1ffffffff12552ad RBX: 0000000000000000 RCX: ffffffff815c1e17 RDX: dffffc0000000000 RSI: ffff88809fcf28e0 RDI: 0000000000000286 RBP: ffffc900078a7b80 R08: ffff88809fcf2858 R09: fffffbfff140cf3c R10: fffffbfff140cf3c R11: 0000000000000000 R12: 0000000000000286 R13: ffff88809fcf2894 R14: dffffc0000000000 R15: 1ffff11013f9e512 lock_is_held include/linux/lockdep.h:361 [inline] ___might_sleep+0x84/0x5e0 kernel/sched/core.c:6765 get_next_corpse net/netfilter/nf_conntrack_core.c:2013 [inline] nf_ct_iterate_cleanup+0x30d/0x390 net/netfilter/nf_conntrack_core.c:2035 nf_ct_iterate_cleanup_net+0xf9/0x150 net/netfilter/nf_conntrack_core.c:2120 iterate_cleanup_work+0x4f/0x100 net/netfilter/nf_nat_masquerade.c:216 process_one_work+0x7ef/0x10d0 kernel/workqueue.c:2264 worker_thread+0xc01/0x1630 kernel/workqueue.c:2410 kthread+0x332/0x350 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 rcu: rcu_preempt kthread starved for 10537 jiffies! g6893 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 rcu: RCU grace-period kthread stack dump: rcu_preempt R running task 28984 10 2 0x80004000 Call Trace: context_switch kernel/sched/core.c:3385 [inline] __schedule+0x9a0/0xcc0 kernel/sched/core.c:4081 schedule+0x181/0x210 kernel/sched/core.c:4155 schedule_timeout+0x14f/0x240 kernel/time/timer.c:1895 rcu_gp_fqs_loop kernel/rcu/tree.c:1661 [inline] rcu_gp_kthread+0xed8/0x1770 kernel/rcu/tree.c:1821 kthread+0x332/0x350 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352