syzbot

 sign-in | mailing list | source | docs
🐞 Open [657]
🐞 Fixed [64]
🐞 Invalid [739]
⬇ Missing Backports [78]
≡ Crashes
Kernel Health Bugs/Month Bug Lifetimes Fuzzing
💬 Send us feedback

INFO: rcu detected stall in hub_event

Status: upstream: reported on 2024/08/13 15:49
Reported-by: syzbot+88898457bbccad1cb490@syzkaller.appspotmail.com
First crash: 167d, last: 81d
▶ ▼ Similar bugs (5)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in hub_event usb C done 1216 2d00h 1894d 19/28 upstream: reported C repro on 2019/11/21 14:45
linux-6.1 INFO: rcu detected stall in hub_event origin:upstream C error 5 41d 534d 0/3 upstream: reported C repro on 2023/08/12 16:34
android-5-15 BUG: soft lockup in hub_event 12 215d 286d 0/2 auto-obsoleted due to no activity on 2024/09/23 11:46
android-54 BUG: soft lockup in hub_event C 1 505d 505d 0/2 auto-obsoleted due to no activity on 2024/02/11 21:05
android-6-1 BUG: soft lockup in hub_event 1 593d 593d 0/2 auto-obsoleted due to no activity on 2023/09/12 05:32

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 ticks this GP) idle=dc9/1/0x4000000000000000 softirq=14747/14747 fqs=0 
	(detected by 1, t=10502 jiffies, g=18009, q=213)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5191 Comm: kworker/0:13 Not tainted 5.15.170-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: usb_hub_wq hub_event
RIP: 0010:rb_add_cached include/linux/rbtree.h:183 [inline]
RIP: 0010:timerqueue_add+0x1df/0x280 lib/timerqueue.c:40
Code: 00 00 00 fc ff df 41 80 7c 05 00 00 74 08 4c 89 ff e8 85 19 9c fd 4c 8b 7c 24 10 4e 89 7c 35 00 45 84 e4 75 40 e8 21 4e 52 fd <31> db 4c 8b 64 24 08 eb 69 e8 13 4e 52 fd ba 18 00 00 00 4c 89 ff
RSP: 0018:ffffc90000007cc0 EFLAGS: 00000006
RAX: ffffffff842e259f RBX: ffff8880767a8b40 RCX: ffff8880261e1dc0
RDX: 0000000000010000 RSI: 0000000000000000 RDI: ffff8880767a8b58
RBP: ffff88802cf78340 R08: dffffc0000000000 R09: ffff8880767a8b48
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: 1ffff110059ef069 R14: 0000000000000008 R15: ffff8880767a8b40
FS:  0000000000000000(0000) GS:ffff8880b9000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b3061bff8 CR3: 00000000666f9000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 enqueue_hrtimer+0x1a0/0x390 kernel/time/hrtimer.c:1092
 __run_hrtimer kernel/time/hrtimer.c:1705 [inline]
 __hrtimer_run_queues+0x6b6/0xcf0 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1097 [inline]
 __sysvec_apic_timer_interrupt+0x13b/0x4b0 arch/x86/kernel/apic/apic.c:1114
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1108 [inline]
 sysvec_apic_timer_interrupt+0x9b/0xc0 arch/x86/kernel/apic/apic.c:1108
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:676
RIP: 0010:console_unlock+0xe53/0x12b0 kernel/printk/printk.c:2749
Code: 00 48 83 7c 24 18 00 75 19 e8 59 06 1a 00 eb 18 e8 52 06 1a 00 e8 6d b1 cd 08 48 83 7c 24 18 00 74 e7 e8 40 06 1a 00 fb 31 ff <44> 89 f6 e8 95 09 1a 00 31 db 45 85 f6 0f 95 c0 89 c1 0a 4c 24 17
RSP: 0018:ffffc900040e7180 EFLAGS: 00000246
RAX: ffffffff81666d80 RBX: 0000000000000000 RCX: 0000000000100000
RDX: ffffc900103d6000 RSI: 00000000000224f4 RDI: 0000000000000000
RBP: ffffc900040e7410 R08: ffffffff81666d4b R09: fffffbfff20ec621
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffffffff1a4c62e
R13: ffffffff8d2631a8 R14: 0000000000000000 R15: dffffc0000000000
 vprintk_emit+0xbf/0x150 kernel/printk/printk.c:2274
 dev_vprintk_emit+0x2aa/0x330 drivers/base/core.c:4621
 dev_printk_emit+0xd9/0x120 drivers/base/core.c:4632
 _dev_info+0x11e/0x170 drivers/base/core.c:4690
 announce_device drivers/usb/core/hub.c:2311 [inline]
 usb_new_device+0x81f/0x18e0 drivers/usb/core/hub.c:2574
 hub_port_connect drivers/usb/core/hub.c:5455 [inline]
 hub_port_connect_change drivers/usb/core/hub.c:5595 [inline]
 port_event drivers/usb/core/hub.c:5741 [inline]
 hub_event+0x2cdf/0x54c0 drivers/usb/core/hub.c:5823
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
rcu: rcu_preempt kthread timer wakeup didn't happen for 10501 jiffies! g18009 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: 	Possible timer handling issue on cpu=0 timer-softirq=6538
rcu: rcu_preempt kthread starved for 10502 jiffies! g18009 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:26144 pid:   15 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5027 [inline]
 __schedule+0x12c4/0x45b0 kernel/sched/core.c:6373
 schedule+0x11b/0x1f0 kernel/sched/core.c:6456
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1914
 rcu_gp_fqs_loop+0x2bf/0x1080 kernel/rcu/tree.c:1972
 rcu_gp_kthread+0xa4/0x360 kernel/rcu/tree.c:2145
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5191 Comm: kworker/0:13 Not tainted 5.15.170-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: usb_hub_wq hub_event
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x4/0x60 kernel/kcov.c:205
Code: 00 00 00 0f 1f 00 53 48 89 fb e8 17 00 00 00 48 8b 3d e8 f4 68 0c 48 89 de 5b e9 b7 f3 48 00 cc cc cc cc cc cc cc 48 8b 04 24 <65> 48 8b 0d f4 fb 81 7e 65 8b 15 f5 fb 81 7e 81 e2 00 01 ff 00 74
RSP: 0018:ffffc90000007d20 EFLAGS: 00000096
RAX: ffffffff816fa3fa RBX: 0000000000000000 RCX: 0000000000010003
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000001
RBP: 0000000000000000 R08: dffffc0000000000 R09: fffffbfff2d4989e
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff8880b902a300 R14: ffff8880767a8b40 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b9000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b3061bff8 CR3: 00000000666f9000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 arch_static_branch arch/x86/include/asm/jump_label.h:27 [inline]
 static_key_false include/linux/jump_label.h:212 [inline]
 trace_hrtimer_start include/trace/events/timer.h:199 [inline]
 debug_activate kernel/time/hrtimer.c:476 [inline]
 enqueue_hrtimer+0x3a/0x390 kernel/time/hrtimer.c:1084
 __run_hrtimer kernel/time/hrtimer.c:1705 [inline]
 __hrtimer_run_queues+0x6b6/0xcf0 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1097 [inline]
 __sysvec_apic_timer_interrupt+0x13b/0x4b0 arch/x86/kernel/apic/apic.c:1114
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1108 [inline]
 sysvec_apic_timer_interrupt+0x9b/0xc0 arch/x86/kernel/apic/apic.c:1108
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:676
RIP: 0010:console_unlock+0xe53/0x12b0 kernel/printk/printk.c:2749
Code: 00 48 83 7c 24 18 00 75 19 e8 59 06 1a 00 eb 18 e8 52 06 1a 00 e8 6d b1 cd 08 48 83 7c 24 18 00 74 e7 e8 40 06 1a 00 fb 31 ff <44> 89 f6 e8 95 09 1a 00 31 db 45 85 f6 0f 95 c0 89 c1 0a 4c 24 17
RSP: 0018:ffffc900040e7180 EFLAGS: 00000246
RAX: ffffffff81666d80 RBX: 0000000000000000 RCX: 0000000000100000
RDX: ffffc900103d6000 RSI: 00000000000224f4 RDI: 0000000000000000
RBP: ffffc900040e7410 R08: ffffffff81666d4b R09: fffffbfff20ec621
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffffffff1a4c62e
R13: ffffffff8d2631a8 R14: 0000000000000000 R15: dffffc0000000000
 vprintk_emit+0xbf/0x150 kernel/printk/printk.c:2274
 dev_vprintk_emit+0x2aa/0x330 drivers/base/core.c:4621
 dev_printk_emit+0xd9/0x120 drivers/base/core.c:4632
 _dev_info+0x11e/0x170 drivers/base/core.c:4690
 announce_device drivers/usb/core/hub.c:2311 [inline]
 usb_new_device+0x81f/0x18e0 drivers/usb/core/hub.c:2574
 hub_port_connect drivers/usb/core/hub.c:5455 [inline]
 hub_port_connect_change drivers/usb/core/hub.c:5595 [inline]
 port_event drivers/usb/core/hub.c:5741 [inline]
 hub_event+0x2cdf/0x54c0 drivers/usb/core/hub.c:5823
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
watchdog: BUG: soft lockup - CPU#1 stuck for 246s! [syz.3.876:6445]
Modules linked in:
irq event stamp: 50998
hardirqs last  enabled at (50997): [<ffffffff8a400d06>] asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:676
hardirqs last disabled at (50998): [<ffffffff8a34123a>] sysvec_apic_timer_interrupt+0xa/0xc0 arch/x86/kernel/apic/apic.c:1108
softirqs last  enabled at (660): [<ffffffff814d9907>] __do_softirq kernel/softirq.c:592 [inline]
softirqs last  enabled at (660): [<ffffffff814d9907>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last  enabled at (660): [<ffffffff814d9907>] __irq_exit_rcu+0x157/0x240 kernel/softirq.c:641
softirqs last disabled at (421): [<ffffffff814d9907>] __do_softirq kernel/softirq.c:592 [inline]
softirqs last disabled at (421): [<ffffffff814d9907>] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last disabled at (421): [<ffffffff814d9907>] __irq_exit_rcu+0x157/0x240 kernel/softirq.c:641
CPU: 1 PID: 6445 Comm: syz.3.876 Not tainted 5.15.170-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0xa93/0xd90 kernel/smp.c:969
Code: 04 03 84 c0 0f 85 84 00 00 00 45 8b 7d 00 44 89 fe 83 e6 01 31 ff e8 5c d5 0b 00 41 83 e7 01 75 07 e8 f1 d1 0b 00 eb 41 f3 90 <48> b8 00 00 00 00 00 fc ff df 0f b6 04 03 84 c0 75 11 41 f7 45 00
RSP: 0018:ffffc900031a7220 EFLAGS: 00000246
RAX: ffffffff8174a1f4 RBX: 1ffff11017208541 RCX: 0000000000040000
RDX: ffffc90004109000 RSI: 000000000003ffff RDI: 0000000000040000
RBP: ffffc900031a7360 R08: ffffffff8174a1c4 R09: fffffbfff20ec622
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: ffff8880b9042a08 R14: ffff8880b913b380 R15: 0000000000000001
FS:  00007fed1a4896c0(0000) GS:ffff8880b9100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000110c301c91 CR3: 0000000058501000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x3b/0x80 kernel/smp.c:1135
 __purge_vmap_area_lazy+0x294/0x1740 mm/vmalloc.c:1683
 _vm_unmap_aliases+0x453/0x4e0 mm/vmalloc.c:2107
 change_page_attr_set_clr+0x308/0x1050 arch/x86/mm/pat/set_memory.c:1740
 change_page_attr_clear arch/x86/mm/pat/set_memory.c:1797 [inline]
 set_memory_ro+0xa1/0xe0 arch/x86/mm/pat/set_memory.c:1943
 bpf_jit_binary_lock_ro include/linux/filter.h:892 [inline]
 bpf_int_jit_compile+0xbf36/0xc6c0 arch/x86/net/bpf_jit_comp.c:2422
 bpf_prog_select_runtime+0x701/0x9f0 kernel/bpf/core.c:1930
 bpf_prog_load+0x131c/0x1b60 kernel/bpf/syscall.c:2357
 __sys_bpf+0x343/0x670 kernel/bpf/syscall.c:4651
 __do_sys_bpf kernel/bpf/syscall.c:4755 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:4753 [inline]
 __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:4753
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7fed1c011719
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fed1a489038 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007fed1c1c8f80 RCX: 00007fed1c011719
RDX: 0000000000000090 RSI: 00000000200000c0 RDI: 0000000000000005
RBP: 00007fed1c08439e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fed1c1c8f80 R15: 00007ffffc9f7d78
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5191 Comm: kworker/0:13 Not tainted 5.15.170-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: usb_hub_wq hub_event
RIP: 0010:advance_sched+0x200/0x940 net/sched/sch_taprio.c:733
Code: 8b 7c 24 10 e8 f1 94 48 f9 48 8b 44 24 10 48 c7 00 00 00 00 00 48 8b 44 24 20 48 89 44 24 08 31 c0 48 89 44 24 20 48 8b 04 24 <48> 85 c0 4c 89 7c 24 60 0f 84 73 06 00 00 48 89 6c 24 40 4c 89 6c
RSP: 0018:ffffc90000007cc0 EFLAGS: 00000006
RAX: ffff88805cbcaa80 RBX: ffff88802cf78330 RCX: ffff8880261e1dc0
RDX: 0000000000010000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffff88802cf782e8 R08: ffffffff8881a988 R09: 0000000000000003
R10: ffffffffffffffff R11: dffffc0000000001 R12: ffff88802cf78000
R13: ffff88802cf78340 R14: dffffc0000000000 R15: ffff88802cf78328
FS:  0000000000000000(0000) GS:ffff8880b9000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b3061bff8 CR3: 00000000666f9000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1688 [inline]
 __hrtimer_run_queues+0x598/0xcf0 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1097 [inline]
 __sysvec_apic_timer_interrupt+0x13b/0x4b0 arch/x86/kernel/apic/apic.c:1114
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1108 [inline]
 sysvec_apic_timer_interrupt+0x9b/0xc0 arch/x86/kernel/apic/apic.c:1108
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:676
RIP: 0010:console_unlock+0xe53/0x12b0 kernel/printk/printk.c:2749
Code: 00 48 83 7c 24 18 00 75 19 e8 59 06 1a 00 eb 18 e8 52 06 1a 00 e8 6d b1 cd 08 48 83 7c 24 18 00 74 e7 e8 40 06 1a 00 fb 31 ff <44> 89 f6 e8 95 09 1a 00 31 db 45 85 f6 0f 95 c0 89 c1 0a 4c 24 17
RSP: 0018:ffffc900040e7180 EFLAGS: 00000246
RAX: ffffffff81666d80 RBX: 0000000000000000 RCX: 0000000000100000
RDX: ffffc900103d6000 RSI: 00000000000224f4 RDI: 0000000000000000
RBP: ffffc900040e7410 R08: ffffffff81666d4b R09: fffffbfff20ec621
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffffffff1a4c62e
R13: ffffffff8d2631a8 R14: 0000000000000000 R15: dffffc0000000000
 vprintk_emit+0xbf/0x150 kernel/printk/printk.c:2274
 dev_vprintk_emit+0x2aa/0x330 drivers/base/core.c:4621
 dev_printk_emit+0xd9/0x120 drivers/base/core.c:4632
 _dev_info+0x11e/0x170 drivers/base/core.c:4690
 announce_device drivers/usb/core/hub.c:2311 [inline]
 usb_new_device+0x81f/0x18e0 drivers/usb/core/hub.c:2574
 hub_port_connect drivers/usb/core/hub.c:5455 [inline]
 hub_port_connect_change drivers/usb/core/hub.c:5595 [inline]
 port_event drivers/usb/core/hub.c:5741 [inline]
 hub_event+0x2cdf/0x54c0 drivers/usb/core/hub.c:5823
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/11/07 14:47 linux-5.15.y 72244eab0dad 867e44df .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in hub_event
2024/08/13 15:49 linux-5.15.y 7e89efd3ae1c f21a18ca .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: rcu detected stall in hub_event
* Struck through repros no longer work on HEAD.