INFO: rcu detected stall in do_softirq

Status: closed as dup on 2018/02/14 15:09
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+6a74dabfc3393d3e527f@syzkaller.appspotmail.com
First crash: 3162d, last: 3149d
▶ ▼ Duplicate of
Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported
INFO: rcu detected stall in __xfrm_decode_session net 1 2 3066d 3120d
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
Re: INFO: rcu detected stall in do_softirq 2 (2) 2018/02/14 15:09

Sample crash report:
INFO: rcu_sched self-detected stall on CPU
INFO: rcu_sched detected stalls on CPUs/tasks:
	1-....: (125000 ticks this GP) idle=976/140000000000001/0 softirq=39052/39052 fqs=31240 
	(detected by 0, t=125002 jiffies, g=22660, c=22659, q=728)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 19071 Comm: syz-executor4 Not tainted 4.15.0-rc7+ #187
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:inb arch/x86/include/asm/io.h:348 [inline]
RIP: 0010:io_serial_in+0x6b/0x90 drivers/tty/serial/8250/8250_port.c:434
RSP: 0018:ffff8801db305620 EFLAGS: 00000002
RAX: dffffc0000000020 RBX: 00000000000003fd RCX: 0000000000000000
RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8886ef18
RBP: ffff8801db305630 R08: 1ffff1003b660a1b R09: 000000000000000c
R10: ffff8801db3055a0 R11: 0000000000000060 R12: ffffffff8886eee0
R13: 0000000000000020 R14: fffffbfff110de21 R15: fffffbfff110dde5
FS:  00007f55b1761700(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020141fe4 CR3: 00000001d562e004 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 serial_in drivers/tty/serial/8250/8250.h:111 [inline]
 wait_for_xmitr+0x93/0x1e0 drivers/tty/serial/8250/8250_port.c:2033
 serial8250_console_putchar+0x1f/0x60 drivers/tty/serial/8250/8250_port.c:3170
 uart_console_write+0xac/0xe0 drivers/tty/serial/serial_core.c:1858
 serial8250_console_write+0x64f/0xa40 drivers/tty/serial/8250/8250_port.c:3236
 univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:590
 call_console_drivers kernel/printk/printk.c:1574 [inline]
 console_unlock+0x788/0xd80 kernel/printk/printk.c:2233
 vprintk_emit+0x4ad/0x590 kernel/printk/printk.c:1757
 vprintk_default+0x28/0x30 kernel/printk/printk.c:1796
 vprintk_func+0x57/0xc0 kernel/printk/printk_safe.c:379
 printk+0xaa/0xca kernel/printk/printk.c:1829
 print_cpu_stall_info_begin kernel/rcu/tree_plugin.h:1657 [inline]
 print_cpu_stall kernel/rcu/tree.c:1594 [inline]
 check_cpu_stall.isra.61+0x845/0x15b0 kernel/rcu/tree.c:1676
 __rcu_pending kernel/rcu/tree.c:3440 [inline]
 rcu_pending kernel/rcu/tree.c:3502 [inline]
 rcu_check_callbacks+0x256/0xd00 kernel/rcu/tree.c:2842
 update_process_times+0x30/0x60 kernel/time/timer.c:1628
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:162
 tick_sched_timer+0x42/0x120 kernel/time/tick-sched.c:1194
 __run_hrtimer kernel/time/hrtimer.c:1211 [inline]
 __hrtimer_run_queues+0x358/0xe20 kernel/time/hrtimer.c:1275
 hrtimer_interrupt+0x1c2/0x5e0 kernel/time/hrtimer.c:1309
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
 smp_apic_timer_interrupt+0x14a/0x700 arch/x86/kernel/apic/apic.c:1050
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:920
RIP: 0010:_decode_session6+0x58c/0x1380 net/ipv6/xfrm6_policy.c:148
RSP: 0018:ffff8801db3066d8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
RAX: 0000000000000005 RBX: ffff8801d0f81a40 RCX: ffff8801d0f81b04
RDX: 1ffff1003a1f0362 RSI: ffff8801a9840840 RDI: ffff8801d0f81ac4
RBP: ffff8801db306760 R08: 0000000000000001 R09: ffff8801db306868
R10: 000000000000000b R11: ffffed003b660d17 R12: 0000000000000018
R13: ffff8801db306868 R14: 0000000000005360 R15: dffffc0000000000
 __xfrm_decode_session+0x68/0x110 net/xfrm/xfrm_policy.c:2358
 xfrm_decode_session_reverse include/net/xfrm.h:1213 [inline]
 icmpv6_route_lookup+0x356/0x660 net/ipv6/icmp.c:372
 icmp6_send+0x17cd/0x28c0 net/ipv6/icmp.c:551
 icmpv6_send+0x142/0x280 net/ipv6/ip6_icmp.c:43
 ip6_input_finish+0x11ef/0x17a0 net/ipv6/ip6_input.c:305
 NF_HOOK include/linux/netfilter.h:288 [inline]
 ip6_input+0xdb/0x560 net/ipv6/ip6_input.c:327
 dst_input include/net/dst.h:449 [inline]
 ip6_rcv_finish+0x1a9/0x7a0 net/ipv6/ip6_input.c:71
 NF_HOOK include/linux/netfilter.h:288 [inline]
 ipv6_rcv+0xf37/0x1fa0 net/ipv6/ip6_input.c:208
 __netif_receive_skb_core+0x1a41/0x3460 net/core/dev.c:4538
 __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4603
 process_backlog+0x203/0x740 net/core/dev.c:5283
 napi_poll net/core/dev.c:5681 [inline]
 net_rx_action+0x792/0x1910 net/core/dev.c:5747
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1133
 </IRQ>
 do_softirq.part.21+0x14d/0x190 kernel/softirq.c:329
 do_softirq kernel/softirq.c:177 [inline]
 __local_bh_enable_ip+0x1ee/0x230 kernel/softirq.c:182
 local_bh_enable include/linux/bottom_half.h:32 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:727 [inline]
 ip6_finish_output2+0xba0/0x23a0 net/ipv6/ip6_output.c:121
 ip6_finish_output+0x698/0xaf0 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:277 [inline]
 ip6_output+0x1eb/0x840 net/ipv6/ip6_output.c:171
Lost 114 message(s)!
	1-....: (125000 ticks this GP) idle=976/140000000000001/0 softirq=39052/39052 fqs=31499 
	 (t=126040 jiffies g=22660 c=22659 q=728)
NMI backtrace for cpu 1
CPU: 1 PID: 19071 Comm: syz-executor4 Not tainted 4.15.0-rc7+ #187
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103
 nmi_trigger_cpumask_backtrace+0x122/0x180 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:156 [inline]
 rcu_dump_cpu_stacks+0x186/0x1d9 kernel/rcu/tree.c:1459
 print_cpu_stall kernel/rcu/tree.c:1608 [inline]
 check_cpu_stall.isra.61+0xbb8/0x15b0 kernel/rcu/tree.c:1676
 __rcu_pending kernel/rcu/tree.c:3440 [inline]
 rcu_pending kernel/rcu/tree.c:3502 [inline]
 rcu_check_callbacks+0x256/0xd00 kernel/rcu/tree.c:2842
 update_process_times+0x30/0x60 kernel/time/timer.c:1628
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:162
 tick_sched_timer+0x42/0x120 kernel/time/tick-sched.c:1194
 __run_hrtimer kernel/time/hrtimer.c:1211 [inline]
 __hrtimer_run_queues+0x358/0xe20 kernel/time/hrtimer.c:1275
 hrtimer_interrupt+0x1c2/0x5e0 kernel/time/hrtimer.c:1309
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
 smp_apic_timer_interrupt+0x14a/0x700 arch/x86/kernel/apic/apic.c:1050
 apic_timer_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:920
RIP: 0010:_decode_session6+0x58c/0x1380 net/ipv6/xfrm6_policy.c:148
RSP: 0018:ffff8801db3066d8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
RAX: 0000000000000005 RBX: ffff8801d0f81a40 RCX: ffff8801d0f81b04
RDX: 1ffff1003a1f0362 RSI: ffff8801a9840840 RDI: ffff8801d0f81ac4
RBP: ffff8801db306760 R08: 0000000000000001 R09: ffff8801db306868
R10: 000000000000000b R11: ffffed003b660d17 R12: 0000000000000018
R13: ffff8801db306868 R14: 0000000000005360 R15: dffffc0000000000
 __xfrm_decode_session+0x68/0x110 net/xfrm/xfrm_policy.c:2358
 xfrm_decode_session_reverse include/net/xfrm.h:1213 [inline]
 icmpv6_route_lookup+0x356/0x660 net/ipv6/icmp.c:372
 icmp6_send+0x17cd/0x28c0 net/ipv6/icmp.c:551
 icmpv6_send+0x142/0x280 net/ipv6/ip6_icmp.c:43
 ip6_input_finish+0x11ef/0x17a0 net/ipv6/ip6_input.c:305
 NF_HOOK include/linux/netfilter.h:288 [inline]
 ip6_input+0xdb/0x560 net/ipv6/ip6_input.c:327
 dst_input include/net/dst.h:449 [inline]
 ip6_rcv_finish+0x1a9/0x7a0 net/ipv6/ip6_input.c:71
 NF_HOOK include/linux/netfilter.h:288 [inline]
 ipv6_rcv+0xf37/0x1fa0 net/ipv6/ip6_input.c:208
 __netif_receive_skb_core+0x1a41/0x3460 net/core/dev.c:4538
 __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4603
 process_backlog+0x203/0x740 net/core/dev.c:5283
 napi_poll net/core/dev.c:5681 [inline]
 net_rx_action+0x792/0x1910 net/core/dev.c:5747
 __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
 do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1133
 </IRQ>
 do_softirq.part.21+0x14d/0x190 kernel/softirq.c:329
 do_softirq kernel/softirq.c:177 [inline]
 __local_bh_enable_ip+0x1ee/0x230 kernel/softirq.c:182
 local_bh_enable include/linux/bottom_half.h:32 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:727 [inline]
 ip6_finish_output2+0xba0/0x23a0 net/ipv6/ip6_output.c:121
 ip6_finish_output+0x698/0xaf0 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:277 [inline]
 ip6_output+0x1eb/0x840 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:443 [inline]
 ip6_local_out+0x95/0x160 net/ipv6/output_core.c:176
 ip6_send_skb+0xa1/0x330 net/ipv6/ip6_output.c:1675
 ip6_push_pending_frames+0xb3/0xe0 net/ipv6/ip6_output.c:1695
 rawv6_push_pending_frames net/ipv6/raw.c:616 [inline]
 rawv6_sendmsg+0x2eed/0x3e70 net/ipv6/raw.c:935
 inet_sendmsg+0x11f/0x5e0 net/ipv4/af_inet.c:764
 sock_sendmsg_nosec net/socket.c:630 [inline]
 sock_sendmsg+0xca/0x110 net/socket.c:640
 SYSC_sendto+0x361/0x5c0 net/socket.c:1721
 SyS_sendto+0x40/0x50 net/socket.c:1689
 entry_SYSCALL_64_fastpath+0x23/0x9a
RIP: 0033:0x452cf9
RSP: 002b:00007f55b1760c58 EFLAGS: 00000212 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 000000000071bea0 RCX: 0000000000452cf9
RDX: 0000000000000020 RSI: 0000000020f93fb2 RDI: 0000000000000013
RBP: 0000000000000489 R08: 0000000020141fe4 R09: 000000000000001c
R10: 0000000000000000 R11: 0000000000000212 R12: 00000000006f4d78
R13: 00000000ffffffff R14: 00007f55b17616d4 R15: 0000000000000000
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.252 msecs

Crashes (11):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/01/14 02:47 net-next-old 6bd39bc3da0f c9e7aeae .config console log report ci-upstream-net-kasan-gce
2018/01/10 08:28 net-next-old 61ad64080e03 1f60c828 .config console log report ci-upstream-net-kasan-gce
2018/01/10 07:11 net-next-old 61ad64080e03 1f60c828 .config console log report ci-upstream-net-kasan-gce
2018/01/10 01:48 net-next-old 61ad64080e03 1f60c828 .config console log report ci-upstream-net-kasan-gce
2018/01/08 19:51 net-next-old f66faae2f80a 93b4c6f1 .config console log report ci-upstream-net-kasan-gce
2018/01/07 20:39 net-next-old d0adb51edb73 19c05fff .config console log report ci-upstream-net-kasan-gce
2018/01/07 10:24 net-next-old d0adb51edb73 19c05fff .config console log report ci-upstream-net-kasan-gce
2018/01/05 00:56 net-next-old 8a4816cad00b 00193447 .config console log report ci-upstream-net-kasan-gce
2018/01/04 07:56 net-next-old 4b24dd802280 00193447 .config console log report ci-upstream-net-kasan-gce
2018/01/01 14:48 net-next-old 6bb8824732f6 00193447 .config console log report ci-upstream-net-kasan-gce
2018/01/06 10:46 mmots 69eed2290e1d 00193447 .config console log report ci-upstream-mmots-kasan-gce
* Struck through repros no longer work on HEAD.