INFO: rcu detected stall in netlink

INFO: rcu detected stall in netlink_sendmsg (3)
Status: fixed on 2020/02/18 14:31
Reported-by: syzbot+@syzkaller.appspotmail.com
Fix commit: d9e15a27 pkt_sched: fq: do not accept silly TCA_FQ_QUANTUM
First crash: 348d, last: 295d

Cause bisection: the issue happens on the oldest tested release (bisect log)
Crash: INFO: rcu detected stall in corrupted (log)
Repro: C syz .config

similar bugs (4):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in netlink_sendmsg (2)	C	done		11	444d	450d	14/17	fixed on 2019/10/09 10:54
linux-4.19	INFO: rcu detected stall in netlink_sendmsg	C		done	3	443d	448d	1/1	fixed on 2019/12/10 03:52
upstream	INFO: rcu detected stall in netlink_sendmsg	syz			83	654d	709d	12/17	fixed on 2019/03/06 07:43
upstream	INFO: rcu detected stall in netlink_sendmsg (4)	C	done		11	1d12h	246d	0/17	upstream: reported C repro on 2020/03/31 17:43

Sample crash report:

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (1 GPs behind) idle=be2/1/0x4000000000000002 softirq=12989/12991 fqs=0 
	(t=10500 jiffies g=9037 q=154)
rcu: rcu_preempt kthread starved for 10500 jiffies! g9037 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: RCU grace-period kthread stack dump:
rcu_preempt     I29272    10      2 0x80004000
Call Trace:
 context_switch kernel/sched/core.c:3385 [inline]
 __schedule+0x934/0x1f90 kernel/sched/core.c:4081
 schedule+0xdc/0x2b0 kernel/sched/core.c:4155
 schedule_timeout+0x486/0xc50 kernel/time/timer.c:1895
 rcu_gp_fqs_loop kernel/rcu/tree.c:1661 [inline]
 rcu_gp_kthread+0x9b2/0x18d0 kernel/rcu/tree.c:1821
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Sending NMI from CPU 1 to CPUs 0:
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.340 msecs
NMI backtrace for cpu 0
CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.5.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:check_kcov_mode kernel/kcov.c:153 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x1e/0x50 kernel/kcov.c:187
Code: 00 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 65 48 8b 04 25 c0 1e 02 00 65 8b 15 84 f1 8c 7e 81 e2 00 01 1f 00 48 8b 75 08 <75> 2b 8b 90 80 13 00 00 83 fa 02 75 20 48 8b 88 88 13 00 00 8b 80
RSP: 0018:ffffc90000cfe958 EFLAGS: 00000206
RAX: ffff8880a99ee240 RBX: 0000000000000000 RCX: ffffffff8644d3d3
RDX: 0000000000000100 RSI: ffffffff8644d647 RDI: ffff8880903bde08
RBP: ffffc90000cfe958 R08: ffff8880a99ee240 R09: fffffbfff16599c4
R10: fffffbfff16599c3 R11: ffffffff8b2cce1f R12: ffff88809658f2d8
R13: ffff8880903bddc0 R14: dffffc0000000000 R15: ffff88809658f000
FS:  0000000000000000(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dbf70 CR3: 000000000986d000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 fq_flow_add_tail net/sched/sch_fq.c:152 [inline]
 fq_dequeue+0x777/0x16a0 net/sched/sch_fq.c:518
 dequeue_skb net/sched/sch_generic.c:263 [inline]
 qdisc_restart net/sched/sch_generic.c:366 [inline]
 __qdisc_run+0x1a5/0x1770 net/sched/sch_generic.c:384
 __dev_xmit_skb net/core/dev.c:3677 [inline]
 __dev_queue_xmit+0x163f/0x35c0 net/core/dev.c:3982
 dev_queue_xmit+0x18/0x20 net/core/dev.c:4046
 br_dev_queue_push_xmit+0x3f3/0x5e0 net/bridge/br_forward.c:52
 br_nf_dev_queue_xmit+0x34e/0x14b0 net/bridge/br_netfilter_hooks.c:800
 NF_HOOK include/linux/netfilter.h:307 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 br_nf_post_routing+0x14e9/0x1d10 net/bridge/br_netfilter_hooks.c:848
 nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
 nf_hook include/linux/netfilter.h:262 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 br_forward_finish+0x215/0x3f0 net/bridge/br_forward.c:65
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1019
 br_nf_forward_finish+0x667/0xa80 net/bridge/br_netfilter_hooks.c:564
 NF_HOOK include/linux/netfilter.h:307 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 br_nf_forward_ip net/bridge/br_netfilter_hooks.c:634 [inline]
 br_nf_forward_ip+0xc65/0x21d0 net/bridge/br_netfilter_hooks.c:575
 nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
 nf_hook include/linux/netfilter.h:262 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 __br_forward+0x393/0xaf0 net/bridge/br_forward.c:109
 deliver_clone+0x61/0xc0 net/bridge/br_forward.c:125
 br_flood+0x325/0x3d0 net/bridge/br_forward.c:232
 br_handle_frame_finish+0xb46/0x1670 net/bridge/br_input.c:162
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1019
 br_nf_pre_routing_finish_ipv6+0x6fa/0xdb0 net/bridge/br_netfilter_ipv6.c:206
 NF_HOOK include/linux/netfilter.h:307 [inline]
 br_nf_pre_routing_ipv6+0x456/0x830 net/bridge/br_netfilter_ipv6.c:236
 br_nf_pre_routing+0x1896/0x22b3 net/bridge/br_netfilter_hooks.c:505
 nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
 nf_hook_bridge_pre net/bridge/br_input.c:224 [inline]
 br_handle_frame+0x806/0x1340 net/bridge/br_input.c:349
 __netif_receive_skb_core+0xfbc/0x30b0 net/core/dev.c:5051
 __netif_receive_skb_one_core+0xa8/0x1a0 net/core/dev.c:5148
 __netif_receive_skb+0x2c/0x1d0 net/core/dev.c:5264
 process_backlog+0x206/0x750 net/core/dev.c:6095
 napi_poll net/core/dev.c:6532 [inline]
 net_rx_action+0x508/0x1120 net/core/dev.c:6600
 __do_softirq+0x262/0x98c kernel/softirq.c:292
 run_ksoftirqd kernel/softirq.c:603 [inline]
 run_ksoftirqd+0x8e/0x110 kernel/softirq.c:595
 smpboot_thread_fn+0x6a3/0xa40 kernel/smpboot.c:165
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
NMI backtrace for cpu 1
CPU: 1 PID: 9744 Comm: syz-executor183 Not tainted 5.5.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree_stall.h:254
 print_cpu_stall kernel/rcu/tree_stall.h:455 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:529 [inline]
 rcu_pending kernel/rcu/tree.c:2827 [inline]
 rcu_sched_clock_irq.cold+0x509/0xc02 kernel/rcu/tree.c:2271
 update_process_times+0x2d/0x70 kernel/time/timer.c:1726
 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:167
 tick_sched_timer+0x53/0x140 kernel/time/tick-sched.c:1310
 __run_hrtimer kernel/time/hrtimer.c:1517 [inline]
 __hrtimer_run_queues+0x364/0xe40 kernel/time/hrtimer.c:1579
 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1641
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1110 [inline]
 smp_apic_timer_interrupt+0x160/0x610 arch/x86/kernel/apic/apic.c:1135
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
 </IRQ>
RIP: 0010:native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:61
Code: a8 98 ea f9 eb 8a cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 94 2a 60 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d 84 2a 60 00 fb f4 <c3> cc 55 48 89 e5 41 57 41 56 41 55 41 54 53 e8 7e 6d 9a f9 e8 e9
RSP: 0018:ffffc90002df71d0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff132669e RBX: ffff88809658f0e8 RCX: 0000000000000002
RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffff88808ce80d54
RBP: ffffc90002df71f0 R08: 1ffffffff16599c3 R09: fffffbfff16599c4
R10: fffffbfff16599c3 R11: ffffffff8b2cce1f R12: 0000000000000003
R13: 0000000000000286 R14: 0000000000000000 R15: 0000000000000001
 pv_wait arch/x86/include/asm/paravirt.h:648 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x9ba/0xc40 kernel/locking/qspinlock.c:507
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:638 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:81 [inline]
 do_raw_spin_lock+0x21d/0x2f0 kernel/locking/spinlock_debug.c:113
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:136 [inline]
 _raw_spin_lock_bh+0x3b/0x50 kernel/locking/spinlock.c:175
 spin_lock_bh include/linux/spinlock.h:343 [inline]
 sch_tree_lock include/net/sch_generic.h:553 [inline]
 fq_change+0x17f/0xe60 net/sched/sch_fq.c:769
 qdisc_change net/sched/sch_api.c:1321 [inline]
 tc_modify_qdisc+0xfd5/0x1cd0 net/sched/sch_api.c:1623
 rtnetlink_rcv_msg+0x45e/0xaf0 net/core/rtnetlink.c:5424
 netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
 rtnetlink_rcv+0x1d/0x30 net/core/rtnetlink.c:5442
 netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
 netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
 netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
 sock_sendmsg_nosec net/socket.c:639 [inline]
 sock_sendmsg+0xd7/0x130 net/socket.c:659
 ____sys_sendmsg+0x753/0x880 net/socket.c:2330
 ___sys_sendmsg+0x100/0x170 net/socket.c:2384
 __sys_sendmsg+0x105/0x1d0 net/socket.c:2417
 __do_sys_sendmsg net/socket.c:2426 [inline]
 __se_sys_sendmsg net/socket.c:2424 [inline]
 __x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4428b9
Code: e8 ac 07 03 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 1b 0a fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fff0f88a1f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004428b9
RDX: 0000000000000000 RSI: 0000000020000240 RDI: 0000000000000003
RBP: 00000000000465c9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000004037f0 R14: 0000000000000000 R15: 0000000000000000

Crashes (6):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	Maintainers
ci-upstream-kasan-gce-root	2019/12/21 09:15	upstream	6398b9fc	bc586918	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci-upstream-net-this-kasan-gce	2019/12/20 02:23	net	6fa9a115	36650b4b	.config	log	report	syz	C	davem@davemloft.net, jhs@mojatatu.com, jiri@resnulli.us, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, xiyou.wangcong@gmail.com
ci-upstream-kasan-gce-selinux-root	2020/01/26 16:14	upstream	2821e26f	dd56146d	.config	log	report			a@unstable.cc, b.a.t.m.a.n@lists.open-mesh.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, mareklindner@neomailbox.ch, netdev@vger.kernel.org, sven@narfation.org, sw@simonwunderlich.de
ci-upstream-kasan-gce	2020/01/22 10:27	upstream	d96d875e	8eda0b95	.config	log	report			a@unstable.cc, b.a.t.m.a.n@lists.open-mesh.org, davem@davemloft.net, linux-kernel@vger.kernel.org, mareklindner@neomailbox.ch, netdev@vger.kernel.org, sven@narfation.org, sw@simonwunderlich.de
ci-upstream-net-kasan-gce	2020/02/11 09:48	net-next	fdfa3a67	084454ae	.config	log	report			a@unstable.cc, b.a.t.m.a.n@lists.open-mesh.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, mareklindner@neomailbox.ch, netdev@vger.kernel.org, sven@narfation.org, sw@simonwunderlich.de
ci-upstream-net-kasan-gce	2020/02/03 10:31	net-next	9f68e365	93e5e335	.config	log	report			a@unstable.cc, b.a.t.m.a.n@lists.open-mesh.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, mareklindner@neomailbox.ch, netdev@vger.kernel.org, sven@narfation.org, sw@simonwunderlich.de