syzbot


INFO: rcu detected stall in sch_direct_xmit

Status: auto-closed as invalid on 2019/02/22 10:09
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+536ae84aa971d463d902@syzkaller.appspotmail.com
First crash: 2216d, last: 2214d
Discussions (1)
Title Replies (including bot) Last reply
INFO: rcu detected stall in sch_direct_xmit 0 (1) 2018/04/01 08:16

Sample crash report:
INFO: rcu_sched self-detected stall on CPU
	0-...!: (1 GPs behind) idle=426/1/4611686018427387906 softirq=97346/97347 fqs=238 
	 (t=125000 jiffies g=44021 c=44020 q=910)
rcu_sched kthread starved for 124031 jiffies! g44021 c44020 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
rcu_sched       I24560     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2862 [inline]
 __schedule+0x8fb/0x1ec0 kernel/sched/core.c:3440
 schedule+0xf5/0x430 kernel/sched/core.c:3499
 schedule_timeout+0x118/0x230 kernel/time/timer.c:1801
 rcu_gp_kthread+0x9dd/0x18e0 kernel/rcu/tree.c:2230
 kthread+0x33c/0x400 kernel/kthread.c:238
 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:406
NMI backtrace for cpu 0
CPU: 0 PID: 4242 Comm: syz-executor4 Not tainted 4.16.0-rc6+ #288
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x24d lib/dump_stack.c:53
 nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103
 nmi_trigger_cpumask_backtrace+0x123/0x180 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:156 [inline]
 rcu_dump_cpu_stacks+0x186/0x1de kernel/rcu/tree.c:1375
 print_cpu_stall kernel/rcu/tree.c:1524 [inline]
 check_cpu_stall.isra.61+0xbb8/0x15b0 kernel/rcu/tree.c:1592
 __rcu_pending kernel/rcu/tree.c:3361 [inline]
 rcu_pending kernel/rcu/tree.c:3423 [inline]
 rcu_check_callbacks+0x238/0xd20 kernel/rcu/tree.c:2763
 update_process_times+0x30/0x60 kernel/time/timer.c:1636
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:162
 tick_sched_timer+0x42/0x120 kernel/time/tick-sched.c:1194
 __run_hrtimer kernel/time/hrtimer.c:1349 [inline]
 __hrtimer_run_queues+0x39c/0xec0 kernel/time/hrtimer.c:1411
 hrtimer_interrupt+0x2a5/0x6f0 kernel/time/hrtimer.c:1469
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
 smp_apic_timer_interrupt+0x14a/0x700 arch/x86/kernel/apic/apic.c:1050
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:857
 </IRQ>
RIP: 0010:rep_nop arch/x86/include/asm/processor.h:657 [inline]
RIP: 0010:cpu_relax arch/x86/include/asm/processor.h:662 [inline]
RIP: 0010:virt_spin_lock arch/x86/include/asm/qspinlock.h:69 [inline]
RIP: 0010:native_queued_spin_lock_slowpath+0x1c2/0xfa0 kernel/locking/qspinlock.c:305
RSP: 0018:ffff8801ae384390 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff12
RAX: 0000000000000001 RBX: ffff8801aee96600 RCX: dffffc0000000000
RDX: 0000000000000004 RSI: ffff8801ae384460 RDI: ffff8801aee96600
RBP: ffff8801ae384708 R08: ffffffff85841401 R09: 0000000000000000
R10: ffff8801ae384628 R11: ffff8801aee96618 R12: ffffed0035c7088c
R13: ffff8801ae3846e0 R14: 1ffff10035c70878 R15: 0000000000000003
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:669 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:30 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:90 [inline]
 do_raw_spin_lock+0x185/0x220 kernel/locking/spinlock_debug.c:113
 __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_lock+0x32/0x40 kernel/locking/spinlock.c:144
 spin_lock include/linux/spinlock.h:310 [inline]
 __netif_tx_lock include/linux/netdevice.h:3582 [inline]
 sch_direct_xmit+0x361/0x1140 net/sched/sch_generic.c:325
 qdisc_restart net/sched/sch_generic.c:393 [inline]
 __qdisc_run+0x57d/0x18e0 net/sched/sch_generic.c:401
 __dev_xmit_skb net/core/dev.c:3249 [inline]
 __dev_queue_xmit+0xb8b/0x2fc0 net/core/dev.c:3557
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3622
 br_dev_queue_push_xmit+0x196/0x5a0 net/bridge/br_forward.c:55
 NF_HOOK include/linux/netfilter.h:288 [inline]
 br_forward_finish+0xc8/0x530 net/bridge/br_forward.c:67
 NF_HOOK include/linux/netfilter.h:288 [inline]
 __br_forward+0x533/0xc80 net/bridge/br_forward.c:112
 br_flood+0x665/0x770 net/bridge/br_forward.c:225
 br_dev_xmit+0xa68/0x1550 net/bridge/br_device.c:87
 __netdev_start_xmit include/linux/netdevice.h:4087 [inline]
 netdev_start_xmit include/linux/netdevice.h:4096 [inline]
 xmit_one net/core/dev.c:3058 [inline]
 dev_hard_start_xmit+0x24e/0xac0 net/core/dev.c:3074
 __dev_queue_xmit+0x26bf/0x2fc0 net/core/dev.c:3589
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3622
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x91a/0x1550 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd60 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:277 [inline]
 ip_mc_output+0x271/0x1350 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:444 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 iptunnel_xmit+0x556/0x810 net/ipv4/ip_tunnel_core.c:91
 ip_tunnel_xmit+0x177b/0x3550 net/ipv4/ip_tunnel.c:768
 __gre_xmit+0x546/0x8b0 net/ipv4/ip_gre.c:449
 erspan_xmit+0x779/0x22a0 net/ipv4/ip_gre.c:731
 __netdev_start_xmit include/linux/netdevice.h:4087 [inline]
 netdev_start_xmit include/linux/netdevice.h:4096 [inline]
 xmit_one net/core/dev.c:3058 [inline]
 dev_hard_start_xmit+0x24e/0xac0 net/core/dev.c:3074
 sch_direct_xmit+0x40d/0x1140 net/sched/sch_generic.c:327
 qdisc_restart net/sched/sch_generic.c:393 [inline]
 __qdisc_run+0x57d/0x18e0 net/sched/sch_generic.c:401
 __dev_xmit_skb net/core/dev.c:3249 [inline]
 __dev_queue_xmit+0xb8b/0x2fc0 net/core/dev.c:3557
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3622
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x91a/0x1550 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd60 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:277 [inline]
 ip_output+0x1d2/0x860 net/ipv4/ip_output.c:405
 dst_output include/net/dst.h:444 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1414
 ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1434
 raw_sendmsg+0x1d4d/0x26b0 net/ipv4/raw.c:684
 inet_sendmsg+0x11f/0x5e0 net/ipv4/af_inet.c:763
 sock_sendmsg_nosec net/socket.c:629 [inline]
 sock_sendmsg+0xca/0x110 net/socket.c:639
 SYSC_sendto+0x361/0x5c0 net/socket.c:1748
 SyS_sendto+0x40/0x50 net/socket.c:1716
 do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x454e79
RSP: 002b:00007f6054f74c68 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00007f6054f756d4 RCX: 0000000000454e79
RDX: 0000000000000000 RSI: 0000000020000040 RDI: 0000000000000013
RBP: 000000000072bea0 R08: 0000000020000080 R09: 0000000000000010
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000004f2 R14: 00000000006f9750 R15: 0000000000000000

Crashes (112):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/03/31 17:46 net-next-old c0b6edef0bf0 0174c6c8 .config console log report ci-upstream-net-kasan-gce
2018/03/31 14:21 net-next-old c0b6edef0bf0 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 11:43 net-next-old c0b6edef0bf0 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 03:21 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 03:15 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 03:01 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 03:00 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 02:45 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 01:24 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 01:06 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 00:53 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 00:53 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/31 00:42 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/30 23:49 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/30 19:25 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/30 19:20 net-next-old 6f14f49ce5eb 8fbce0e4 .config console log report ci-upstream-net-kasan-gce
2018/03/30 16:36 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 16:30 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 16:02 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 14:53 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 14:07 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 13:51 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 13:51 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 12:41 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 12:35 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 12:02 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:56 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:46 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:29 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:24 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:08 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 11:01 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 10:44 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 10:23 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 09:58 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 09:47 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 09:00 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 08:49 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 08:32 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 07:39 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 07:28 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 07:16 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
2018/03/30 05:32 net-next-old 18845557fd6f d47f0ed6 .config console log report ci-upstream-net-kasan-gce
* Struck through repros no longer work on HEAD.