syzbot


BUG: soft lockup in tcp_write_timer (4)

Status: fixed on 2024/07/09 19:14
Subsystems: kasan mm
[Documentation on labels]
Fix commit: 22f008128625 USB: class: cdc-wdm: Fix CPU lockup caused by excessive log messages
First crash: 88d, last: 63d
Similar bugs (15)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in tcp_write_timer (3) 2 616d 630d 0/1 upstream: reported on 2022/12/17 21:41
linux-4.14 BUG: soft lockup in tcp_write_timer 2 1583d 1662d 0/1 auto-closed as invalid on 2020/09/05 12:42
linux-4.19 BUG: soft lockup in tcp_write_timer (2) 2 1026d 1090d 0/1 auto-closed as invalid on 2022/03/16 10:56
linux-4.19 BUG: soft lockup in tcp_write_timer 1 1255d 1255d 0/1 auto-closed as invalid on 2021/07/30 14:52
upstream BUG: soft lockup in tcp_write_timer net 11 1862d 1870d 0/27 auto-closed as invalid on 2019/10/25 14:11
upstream BUG: soft lockup in tcp_write_timer (2) kvm 1 866d 866d 0/27 auto-closed as invalid on 2022/06/24 22:31
upstream BUG: soft lockup in tcp_write_timer (3) net 6 243d 350d 0/27 closed as invalid on 2024/03/18 17:07
android-5-15 BUG: soft lockup in tcp_write_timer 11 26d 139d 0/2 premoderation: reported on 2024/04/21 15:01
upstream INFO: rcu detected stall in tcp_write_timer (2) bpf 2 1313d 1368d 0/27 auto-closed as invalid on 2021/05/03 11:59
linux-5.15 INFO: rcu detected stall in tcp_write_timer 1 16d 16d 0/3 upstream: reported on 2024/08/22 21:30
upstream INFO: rcu detected stall in tcp_write_timer (3) net 1 1180d 1180d 0/27 auto-closed as invalid on 2021/09/13 13:17
linux-6.1 INFO: rcu detected stall in tcp_write_timer 6 10d 78d 0/3 upstream: reported on 2024/06/21 02:50
upstream INFO: rcu detected stall in tcp_write_timer net 3 1862d 1857d 0/27 auto-closed as invalid on 2019/10/25 14:11
linux-4.14 INFO: rcu detected stall in tcp_write_timer 4 1464d 1707d 0/1 auto-closed as invalid on 2021/01/02 05:45
linux-4.19 INFO: rcu detected stall in tcp_write_timer 2 1593d 1647d 0/1 auto-closed as invalid on 2020/08/26 06:46

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 143s! [syz-executor.4:9045]
CPU#0 Utilization every 22s during lockup:
	#1:   0% system,	  7% softirq,	 94% hardirq,	  0% idle
	#2:   0% system,	  7% softirq,	 94% hardirq,	  0% idle
	#3:   1% system,	  7% softirq,	 94% hardirq,	  0% idle
	#4:   0% system,	  7% softirq,	 94% hardirq,	  0% idle
	#5:   0% system,	  7% softirq,	 94% hardirq,	  0% idle
CPU#0 Detect HardIRQ Time exceeds 50%. Most frequent HardIRQs:
	#1: 9         	irq#35
Modules linked in:
irq event stamp: 14768927
hardirqs last  enabled at (14768926): [<ffffffff8b869773>] irqentry_exit+0x63/0x90 kernel/entry/common.c:357
hardirqs last disabled at (14768927): [<ffffffff8b86733e>] sysvec_apic_timer_interrupt+0xe/0xc0 arch/x86/kernel/apic/apic.c:1043
softirqs last  enabled at (10502): [<ffffffff815a0b84>] __do_softirq kernel/softirq.c:588 [inline]
softirqs last  enabled at (10502): [<ffffffff815a0b84>] invoke_softirq kernel/softirq.c:428 [inline]
softirqs last  enabled at (10502): [<ffffffff815a0b84>] __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
softirqs last disabled at (10505): [<ffffffff815a0b84>] __do_softirq kernel/softirq.c:588 [inline]
softirqs last disabled at (10505): [<ffffffff815a0b84>] invoke_softirq kernel/softirq.c:428 [inline]
softirqs last disabled at (10505): [<ffffffff815a0b84>] __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
CPU: 0 PID: 9045 Comm: syz-executor.4 Not tainted 6.10.0-rc2-syzkaller-00242-g36534d3c5453 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
RIP: 0010:unwind_next_frame+0x1645/0x2a00 arch/x86/kernel/unwind_orc.c:596
Code: ea 01 00 00 49 39 c4 0f 86 e1 01 00 00 48 05 a8 00 00 00 4c 39 f0 0f 86 27 06 00 00 4c 39 e0 0f 87 1e 06 00 00 e8 db 20 55 00 <4c> 8b 74 24 48 48 8b 44 24 10 48 8d b8 80 00 00 00 e8 35 21 00 00
RSP: 0018:ffffc90000007408 EFLAGS: 00000246
RAX: ffffffff814101a0 RBX: 0000000000000001 RCX: ffff888058a3da00
RDX: 0000000000000103 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900093d0000 R08: ffffffff81410170 R09: ffffffff8140ff56
R10: 0000000000000003 R11: ffff888058a3da00 R12: 1ffff92000000e9e
R13: dffffc0000000000 R14: 1ffff92000000e9d R15: ffffc900093d8000
FS:  00007f24585ff6c0(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32821000 CR3: 0000000051ed2000 CR4: 00000000003506f0
DR0: 0000000020000300 DR1: 0000000020000300 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff1 DR7: 0000000000000600
Call Trace:
 <IRQ>
 arch_stack_walk+0x151/0x1b0 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x118/0x1d0 kernel/stacktrace.c:122
 save_stack+0xfb/0x1f0 mm/page_owner.c:156
 __reset_page_owner+0x75/0x3f0 mm/page_owner.c:297
 reset_page_owner include/linux/page_owner.h:25 [inline]
 free_pages_prepare mm/page_alloc.c:1088 [inline]
 free_unref_page+0xd22/0xea0 mm/page_alloc.c:2565
 __folio_put+0x3b9/0x620 mm/swap.c:129
 skb_page_unref include/linux/skbuff_ref.h:44 [inline]
 __skb_frag_unref include/linux/skbuff_ref.h:57 [inline]
 skb_release_data+0x467/0x880 net/core/skbuff.c:1102
 skb_release_all net/core/skbuff.c:1173 [inline]
 __kfree_skb+0x55/0x70 net/core/skbuff.c:1187
 tcp_write_queue_purge+0x136/0x2f0 net/ipv4/tcp.c:3000
 tcp_write_err net/ipv4/tcp_timer.c:80 [inline]
 tcp_write_timeout net/ipv4/tcp_timer.c:297 [inline]
 tcp_retransmit_timer+0x1762/0x3370 net/ipv4/tcp_timer.c:583
 tcp_write_timer+0x132/0x290 net/ipv4/tcp_timer.c:716
 call_timer_fn+0x18e/0x650 kernel/time/timer.c:1792
 expire_timers kernel/time/timer.c:1843 [inline]
 __run_timers kernel/time/timer.c:2417 [inline]
 __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2428
 run_timer_base kernel/time/timer.c:2437 [inline]
 run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2447
 handle_softirqs+0x2c4/0x970 kernel/softirq.c:554
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:649
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:finish_task_switch+0x1ea/0x870 kernel/sched/core.c:5282
Code: c9 50 e8 99 c9 0b 00 48 83 c4 08 4c 89 f7 e8 cd 38 00 00 0f 1f 44 00 00 4c 89 f7 e8 60 c7 2a 0a e8 5b f8 36 00 fb 48 8b 5d c0 <48> 8d bb f8 15 00 00 48 89 f8 48 c1 e8 03 49 be 00 00 00 00 00 fc
RSP: 0018:ffffc900093d7828 EFLAGS: 00000282
RAX: ff51e34f93feca00 RBX: ffff888058a3da00 RCX: ffffffff8172d78a
RDX: dffffc0000000000 RSI: ffffffff8bcabb80 RDI: ffffffff8c1ff380
RBP: ffffc900093d7870 R08: ffffffff92fab5b7 R09: 1ffffffff25f56b6
R10: dffffc0000000000 R11: fffffbfff25f56b7 R12: 1ffff11017287ea7
R13: dffffc0000000000 R14: ffff8880b943e7c0 R15: ffff8880b943f538
 context_switch kernel/sched/core.c:5411 [inline]
 __schedule+0x17f0/0x4a20 kernel/sched/core.c:6745
 preempt_schedule_irq+0xfb/0x1c0 kernel/sched/core.c:7067
 irqentry_exit+0x5e/0x90 kernel/entry/common.c:354
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:lock_acquire+0x264/0x550 kernel/locking/lockdep.c:5758
Code: 2b 00 74 08 4c 89 f7 e8 2a 82 89 00 f6 44 24 61 02 0f 85 85 01 00 00 41 f7 c7 00 02 00 00 74 01 fb 48 c7 44 24 40 0e 36 e0 45 <4b> c7 44 25 00 00 00 00 00 43 c7 44 25 09 00 00 00 00 43 c7 44 25
RSP: 0018:ffffc900093d7c00 EFLAGS: 00000206
RAX: 0000000000000001 RBX: 1ffff9200127af8c RCX: 0000000000000001
RDX: dffffc0000000000 RSI: ffffffff8bcacd00 RDI: ffffffff8c1ff380
RBP: ffffc900093d7d58 R08: ffffffff92fab5b7 R09: 1ffffffff25f56b6
R10: dffffc0000000000 R11: fffffbfff25f56b7 R12: 1ffff9200127af88
R13: dffffc0000000000 R14: ffffc900093d7c60 R15: 0000000000000246
 __might_fault+0xc6/0x120 mm/memory.c:6233
 clear_rseq_cs kernel/rseq.c:257 [inline]
 rseq_ip_fixup kernel/rseq.c:291 [inline]
 __rseq_handle_notify_resume+0x63a/0x14e0 kernel/rseq.c:329
 rseq_handle_notify_resume include/linux/rseq.h:38 [inline]
 resume_user_mode_work include/linux/resume_user_mode.h:62 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:114 [inline]
 exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
 __syscall_exit_to_user_mode_work kernel/entry/common.c:207 [inline]
 syscall_exit_to_user_mode+0x114/0x370 kernel/entry/common.c:218
 do_syscall_64+0x100/0x230 arch/x86/entry/common.c:89
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f2458a7cea9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f24585ff0c8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: 0000000000000003 RBX: 00007f2458bb3f80 RCX: 00007f2458a7cea9
RDX: ffefffffffffffff RSI: 0000000000000000 RDI: 0000000020000700
RBP: 00007f2458aebff4 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f2458bb3f80 R15: 00007ffc46925018
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 5322 Comm: kworker/u8:15 Not tainted 6.10.0-rc2-syzkaller-00242-g36534d3c5453 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:311 [inline]
RIP: 0010:smp_call_function_many_cond+0x1860/0x29d0 kernel/smp.c:855
Code: 45 8b 65 00 44 89 e6 83 e6 01 31 ff e8 99 09 0c 00 41 83 e4 01 49 bc 00 00 00 00 00 fc ff df 75 07 e8 44 05 0c 00 eb 38 f3 90 <42> 0f b6 04 23 84 c0 75 11 41 f7 45 00 01 00 00 00 74 1e e8 28 05
RSP: 0018:ffffc90009217700 EFLAGS: 00000293
RAX: ffffffff818a2158 RBX: 1ffff11017288c11 RCX: ffff8880207e9e00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900092178e0 R08: ffffffff818a2127 R09: 1ffffffff25f56b0
R10: dffffc0000000000 R11: fffffbfff25f56b1 R12: dffffc0000000000
R13: ffff8880b9446088 R14: ffff8880b953f900 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880b9500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000560e4c4c7180 CR3: 000000000e132000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1023
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2069 [inline]
 text_poke_bp_batch+0x352/0xb30 arch/x86/kernel/alternative.c:2279
 text_poke_flush arch/x86/kernel/alternative.c:2470 [inline]
 text_poke_finish+0x30/0x50 arch/x86/kernel/alternative.c:2477
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x136/0x260 kernel/jump_label.c:205
 static_key_enable+0x1a/0x20 kernel/jump_label.c:218
 toggle_allocation_gate+0xb5/0x250 mm/kfence/core.c:826
 process_one_work kernel/workqueue.c:3231 [inline]
 process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
 worker_thread+0x86d/0xd70 kernel/workqueue.c:3393
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/06/11 11:30 bpf 36534d3c5453 c7d7d815 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce BUG: soft lockup in tcp_write_timer
2024/06/13 17:02 bpf-next 041c1dc988fd 2aa5052f .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-next-kasan-gce BUG: soft lockup in tcp_write_timer
2024/06/17 21:42 upstream 2ccbdf43d5e7 1f11cfd7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in tcp_write_timer
2024/07/06 02:53 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 179264157bba 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in tcp_write_timer
* Struck through repros no longer work on HEAD.