INFO: rcu detected stall in addrconf_dad

INFO: rcu detected stall in addrconf_dad_work
Status: fixed on 2019/12/06 10:33
Reported-by: syzbot+360efe4e8b5dbf168f54@syzkaller.appspotmail.com
Fix commit: cc243e24 sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero
First crash: 261d, last: 254d

Fix bisection: fixed by (bisect log):

commit cc243e2427cef2a5dd7367cb0e0b846503350ffe
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Sun Sep 8 20:40:51 2019 +0000

sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero

similar bugs (6):
Kernel	Title	Repro	Bisected	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in addrconf_dad_work (4)			8	137d	137d	0/17	closed as invalid on 2020/01/09 08:13
upstream	INFO: rcu detected stall in addrconf_dad_work (3)			6	137d	137d	0/17	closed as invalid on 2020/01/08 05:23
upstream	INFO: rcu detected stall in addrconf_dad_work (2)			15	172d	174d	0/17	closed as invalid on 2019/12/04 14:14
upstream	INFO: rcu detected stall in addrconf_dad_work	C	cause	126	252d	257d	14/17	fixed on 2019/10/09 10:54
linux-4.19	INFO: rcu detected stall in addrconf_dad_work (2)	C	fix	1	156d	156d	1/1	fixed on 2020/01/19 15:05
linux-4.19	INFO: rcu detected stall in addrconf_dad_work	C	fix	19	248d	260d	1/1	fixed on 2019/12/07 19:18

Sample crash report:

IPv6: ADDRCONF(NETDEV_UP): hsr0: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
IPv6: ADDRCONF(NETDEV_UP): vxcan1: link is not ready
8021q: adding VLAN 0 to HW filter on device batadv0
INFO: rcu_preempt self-detected stall on CPU
	1-...: (10499 ticks this GP) idle=052/140000000000001/0 softirq=9528/9528 fqs=32 
	 (t=10500 jiffies g=982 c=981 q=17)
rcu_preempt kthread starved for 10435 jiffies! g982 c981 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
rcu_preempt     I29776     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 1
CPU: 1 PID: 6777 Comm: kworker/1:3 Not tainted 4.14.143 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x138/0x197 lib/dump_stack.c:53
 nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
 rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
 print_cpu_stall kernel/rcu/tree.c:1542 [inline]
 check_cpu_stall kernel/rcu/tree.c:1610 [inline]
 __rcu_pending kernel/rcu/tree.c:3390 [inline]
 rcu_pending kernel/rcu/tree.c:3452 [inline]
 rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
 update_process_times+0x31/0x70 kernel/time/timer.c:1588
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
 tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
 __run_hrtimer kernel/time/hrtimer.c:1220 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:60
RSP: 0018:ffff8880a8a6f548 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff10
RAX: ffff888081ae3d38 RBX: ffff888081ae3d38 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff888081ae3dd0 RDI: ffff888081ae3d40
RBP: ffff8880a8a6f5a0 R08: 0000000000000000 R09: ffff888089be2fb8
R10: ffff888089be2f98 R11: ffff888089be2600 R12: dffffc0000000000
R13: ffff888081ae3b40 R14: ffff888081ae3dd0 R15: ffff888081ae3dc0
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip6_finish_output2+0x9ab/0x21b0 net/ipv6/ip6_output.c:120
 ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:462 [inline]
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
 ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
 addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
INFO: rcu_sched detected stalls on CPUs/tasks:
	1-...: (10501 ticks this GP) idle=052/140000000000000/0 softirq=9528/9528 fqs=33 
	(detected by 0, t=10543 jiffies, g=688, c=687, q=0)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 6777 Comm: kworker/1:3 Not tainted 4.14.143 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
task: ffff888089be2600 task.stack: ffff8880a8a68000
RIP: 0010:hhf_dequeue+0x67/0xa60 net/sched/sch_hhf.c:426
RSP: 0018:ffff8880a8a6f550 EFLAGS: 00000246
RAX: ffff888081ae3dc0 RBX: ffff888081ae3d38 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff888081ae3dd0 RDI: ffff888081ae3d40
RBP: ffff8880a8a6f5a0 R08: 0000000000000000 R09: ffff888089be2fb8
R10: ffff888089be2f98 R11: ffff888089be2600 R12: dffffc0000000000
R13: ffff888081ae3b40 R14: ffff888081ae3dc0 R15: ffff888081ae3dc0
FS:  0000000000000000(0000) GS:ffff8880aef00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004c7368 CR3: 000000000766a000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip6_finish_output2+0x9ab/0x21b0 net/ipv6/ip6_output.c:120
 ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:462 [inline]
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
 ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
 addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: 02 00 00 48 89 45 d0 48 c1 e8 03 48 89 45 c0 e8 80 51 6d fc 48 8b 45 c8 80 38 00 0f 85 53 07 00 00 49 8b 85 80 02 00 00 4d 89 fe <49> 39 c7 0f 84 3e 04 00 00 e8 5b 51 6d fc 4c 89 f0 48 c1 e8 03 
rcu_sched kthread starved for 10478 jiffies! g688 c687 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
rcu_sched       I29824     9      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

Crashes (18):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	Maintainers
ci2-linux-4-14	2019/09/13 22:32	linux-4.14.y	e2cd24b6	32d59357	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/13 10:18	linux-4.14.y	e2cd24b6	40fa42bc	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/10 16:18	linux-4.14.y	e2cd24b6	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/10 06:30	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/10 01:21	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/09 20:57	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/09 17:21	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/09 14:10	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/09 05:04	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/09 03:09	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, dsahern@gmail.com, edumazet@google.com, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, lorenzo.bianconi@redhat.com, netdev@vger.kernel.org, vasilykh@arista.com
ci2-linux-4-14	2019/09/08 22:28	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/08 16:42	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/08 02:41	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/08 00:04	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/07 19:29	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/07 18:00	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/07 07:05	linux-4.14.y	414510bc	a60cb4cd	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci2-linux-4-14	2019/09/07 00:24	linux-4.14.y	414510bc	acb5b744	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com