syzbot


INFO: rcu detected stall in tcp_write_timer

Status: upstream: reported on 2024/08/22 21:30
Reported-by: syzbot+06b635a512eb322a9bb9@syzkaller.appspotmail.com
First crash: 24d, last: 2d14h
Similar bugs (15)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in tcp_write_timer (2) bpf 2 1322d 1377d 0/28 auto-closed as invalid on 2021/05/03 11:59
upstream INFO: rcu detected stall in tcp_write_timer net 3 1871d 1866d 0/28 auto-closed as invalid on 2019/10/25 14:11
upstream INFO: rcu detected stall in tcp_write_timer (3) net 1 1189d 1189d 0/28 auto-closed as invalid on 2021/09/13 13:17
linux-6.1 INFO: rcu detected stall in tcp_write_timer 6 19d 87d 0/3 upstream: reported on 2024/06/21 02:50
upstream BUG: soft lockup in tcp_write_timer (4) kasan mm 4 72d 97d 26/28 fixed on 2024/07/09 19:14
linux-4.14 INFO: rcu detected stall in tcp_write_timer 4 1473d 1716d 0/1 auto-closed as invalid on 2021/01/02 05:45
linux-4.19 INFO: rcu detected stall in tcp_write_timer 2 1602d 1656d 0/1 auto-closed as invalid on 2020/08/26 06:46
linux-4.19 BUG: soft lockup in tcp_write_timer (3) 2 625d 638d 0/1 upstream: reported on 2022/12/17 21:41
linux-4.19 BUG: soft lockup in tcp_write_timer (2) 2 1035d 1098d 0/1 auto-closed as invalid on 2022/03/16 10:56
linux-4.19 BUG: soft lockup in tcp_write_timer 1 1264d 1264d 0/1 auto-closed as invalid on 2021/07/30 14:52
upstream BUG: soft lockup in tcp_write_timer (2) kvm 1 874d 874d 0/28 auto-closed as invalid on 2022/06/24 22:31
upstream BUG: soft lockup in tcp_write_timer (3) net 6 251d 359d 0/28 closed as invalid on 2024/03/18 17:07
android-5-15 BUG: soft lockup in tcp_write_timer 11 35d 148d 0/2 premoderation: reported on 2024/04/21 15:01
linux-4.14 BUG: soft lockup in tcp_write_timer 2 1592d 1671d 0/1 auto-closed as invalid on 2020/09/05 12:42
upstream BUG: soft lockup in tcp_write_timer net 11 1871d 1879d 0/28 auto-closed as invalid on 2019/10/25 14:11

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P26905/1:b..l
	(detected by 1, t=10502 jiffies, g=110881, q=470)
task:syz.4.8605      state:R  running task     stack:24824 pid:26905 ppid: 24780 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5027 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
 preempt_schedule_common+0x83/0xd0 kernel/sched/core.c:6549
 preempt_schedule+0xd9/0xe0 kernel/sched/core.c:6574
 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:34
 htab_unlock_bucket kernel/bpf/hashtab.c:204 [inline]
 __htab_percpu_map_update_elem+0x5e6/0x690 kernel/bpf/hashtab.c:1253
 bpf_percpu_hash_update+0x110/0x1c0 kernel/bpf/hashtab.c:2240
 bpf_map_update_value+0x268/0x6c0 kernel/bpf/syscall.c:196
 generic_map_update_batch+0x54d/0x8b0 kernel/bpf/syscall.c:1421
 bpf_map_do_batch+0x4d0/0x620
 __sys_bpf+0x55c/0x670
 __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
 __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f34c9d79e79
RSP: 002b:00007f34c81f6038 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f34c9f15f80 RCX: 00007f34c9d79e79
RDX: 0000000000000038 RSI: 00000000200005c0 RDI: 000000000000001a
RBP: 00007f34c9de7916 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f34c9f15f80 R15: 00007fffe4a38e28
 </TASK>
rcu: rcu_preempt kthread starved for 10524 jiffies! g110881 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:26400 pid:   15 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5027 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
 schedule+0x11b/0x1f0 kernel/sched/core.c:6456
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1914
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 26907 Comm: syz.1.8606 Not tainted 5.15.165-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
 rcu_check_gp_kthread_starvation+0x1d2/0x240 kernel/rcu/tree_stall.h:487
 print_other_cpu_stall+0x137a/0x14d0 kernel/rcu/tree_stall.h:592
 check_cpu_stall kernel/rcu/tree_stall.h:745 [inline]
 rcu_pending kernel/rcu/tree.c:3932 [inline]
 rcu_sched_clock_irq+0xa38/0x1150 kernel/rcu/tree.c:2619
 update_process_times+0x196/0x200 kernel/time/timer.c:1818
 tick_sched_handle kernel/time/tick-sched.c:254 [inline]
 tick_sched_timer+0x386/0x550 kernel/time/tick-sched.c:1473
 __run_hrtimer kernel/time/hrtimer.c:1686 [inline]
 __hrtimer_run_queues+0x55b/0xcf0 kernel/time/hrtimer.c:1750
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1812
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
 __sysvec_apic_timer_interrupt+0x139/0x470 arch/x86/kernel/apic/apic.c:1102
 sysvec_apic_timer_interrupt+0x3e/0xb0 arch/x86/kernel/apic/apic.c:1096
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:free_unref_page+0x213/0x2d0 mm/page_alloc.c:3418
Code: 2b 00 74 08 4c 89 ff e8 0b c5 0a 00 f6 44 24 41 02 0f 85 83 00 00 00 41 f7 c4 00 02 00 00 74 01 fb 48 c7 44 24 20 0e 36 e0 45 <4b> c7 44 35 00 00 00 00 00 66 43 c7 44 35 09 00 00 43 c6 44 35 0b
RSP: 0018:ffffc90000dd0820 EFLAGS: 00000206
RAX: 3e1b0edeeb59e300 RBX: 1ffff920001ba10c RCX: ffffffff81631a88
RDX: dffffc0000000000 RSI: ffffffff8a8b2a20 RDI: ffffffff8ad8f7c0
RBP: ffffc90000dd0900 R08: dffffc0000000000 R09: fffffbfff1f8e230
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000246
R13: dffffc0000000000 R14: 1ffff920001ba108 R15: ffffc90000dd0860
 put_page include/linux/mm.h:1247 [inline]
 __skb_frag_unref include/linux/skbuff.h:3236 [inline]
 skb_release_data+0x411/0x8a0 net/core/skbuff.c:672
 skb_release_all net/core/skbuff.c:742 [inline]
 __kfree_skb+0x4c/0x60 net/core/skbuff.c:756
 tcp_write_queue_purge+0x132/0x3e0 net/ipv4/tcp.c:2963
 tcp_done_with_error+0x3d/0xc0 net/ipv4/tcp_input.c:4358
 tcp_write_err net/ipv4/tcp_timer.c:70 [inline]
 tcp_write_timeout net/ipv4/tcp_timer.c:273 [inline]
 tcp_retransmit_timer+0x112b/0x2680 net/ipv4/tcp_timer.c:543
 tcp_write_timer_handler+0x1f7/0x970 net/ipv4/tcp_timer.c:655
 tcp_write_timer+0x12e/0x280 net/ipv4/tcp_timer.c:675
 call_timer_fn+0x16d/0x560 kernel/time/timer.c:1451
 expire_timers kernel/time/timer.c:1496 [inline]
 __run_timers+0x67c/0x890 kernel/time/timer.c:1767
 run_timer_softirq+0x63/0xf0 kernel/time/timer.c:1780
 handle_softirqs+0x3a7/0x930 kernel/softirq.c:558
 __do_softirq kernel/softirq.c:592 [inline]
 invoke_softirq kernel/softirq.c:432 [inline]
 __irq_exit_rcu+0x157/0x240 kernel/softirq.c:641
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:653
 sysvec_apic_timer_interrupt+0x91/0xb0 arch/x86/kernel/apic/apic.c:1096
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:lock_release+0x62d/0x9a0 kernel/locking/lockdep.c:5647
Code: 3c 3b 00 74 08 4c 89 f7 e8 50 51 67 00 f6 84 24 91 00 00 00 02 75 6f 41 f7 c5 00 02 00 00 74 01 fb 48 c7 44 24 60 0e 36 e0 45 <4b> c7 04 27 00 00 00 00 4b c7 44 27 08 00 00 00 00 65 48 8b 04 25
RSP: 0018:ffffc90003126b00 EFLAGS: 00000206
RAX: 0000000000000001 RBX: 1ffff92000624d72 RCX: ffffc90003126b03
RDX: 0000000000000002 RSI: ffffffff8a8b3cc0 RDI: ffffffff8ad8f7c0
RBP: ffffc90003126c30 R08: dffffc0000000000 R09: fffffbfff1bd2a56
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff92000624d6c
R13: 0000000000000246 R14: ffffc90003126b90 R15: dffffc0000000000
 rcu_read_unlock include/linux/rcupdate.h:772 [inline]
 BPF_PROG_RUN_ARRAY include/linux/bpf.h:1346 [inline]
 trace_call_bpf+0x5a6/0x660 kernel/trace/bpf_trace.c:127
 perf_trace_run_bpf_submit+0x7b/0x1d0 kernel/events/core.c:9981
 perf_trace_lock_acquire+0x3bf/0x4a0 include/trace/events/lock.h:13
 trace_lock_acquire include/trace/events/lock.h:13 [inline]
 lock_acquire+0x4c6/0x4f0 kernel/locking/lockdep.c:5594
 rcu_lock_acquire+0x2a/0x30 include/linux/rcupdate.h:312
 rcu_read_lock include/linux/rcupdate.h:739 [inline]
 nf_hook+0x9f/0x430 include/linux/netfilter.h:226
 __ip_local_out+0x3b2/0x4f0 net/ipv4/ip_output.c:115
 ip_local_out net/ipv4/ip_output.c:124 [inline]
 __ip_queue_xmit+0x11d9/0x1ce0 net/ipv4/ip_output.c:532
 __tcp_transmit_skb+0x1f39/0x3810 net/ipv4/tcp_output.c:1402
 tcp_transmit_skb net/ipv4/tcp_output.c:1420 [inline]
 tcp_write_xmit+0x19e5/0x65f0 net/ipv4/tcp_output.c:2705
 __tcp_push_pending_frames+0x90/0x250 net/ipv4/tcp_output.c:2890
 tcp_sendmsg_locked+0x20c6/0x3a90 net/ipv4/tcp.c:1408
 tcp_sendmsg+0x2c/0x40 net/ipv4/tcp.c:1457
 sock_sendmsg_nosec net/socket.c:704 [inline]
 __sock_sendmsg net/socket.c:716 [inline]
 ____sys_sendmsg+0x59e/0x8f0 net/socket.c:2431
 ___sys_sendmsg+0x252/0x2e0 net/socket.c:2485
 __sys_sendmsg net/socket.c:2514 [inline]
 __do_sys_sendmsg net/socket.c:2523 [inline]
 __se_sys_sendmsg+0x19a/0x260 net/socket.c:2521
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7f725ba81e79
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f7259efe038 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007f725bc1df80 RCX: 00007f725ba81e79
RDX: 00000000000052cc RSI: 0000000020000040 RDI: 0000000000000006
RBP: 00007f725baef916 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f725bc1df80 R15: 00007ffca8baa808
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/08/22 21:29 linux-5.15.y fa93fa65db6e ca02180f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf INFO: rcu detected stall in tcp_write_timer
2024/09/14 04:39 linux-5.15.y 3a5928702e71 b58f933c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf BUG: soft lockup in tcp_write_timer
* Struck through repros no longer work on HEAD.