syzbot


BUG: soft lockup in __hrtimer_run_queues (3)

Status: moderation: reported on 2025/01/03 03:13
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+683c8619374c4c5afe1c@syzkaller.appspotmail.com
First crash: 6d04h, last: 6d04h
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream BUG: soft lockup in __hrtimer_run_queues (2) kernel 1 189d 185d 0/28 auto-obsoleted due to no activity on 2024/09/27 22:42
upstream BUG: soft lockup in __hrtimer_run_queues kernel 2 1450d 1495d 0/28 auto-closed as invalid on 2021/04/15 12:33
upstream INFO: rcu detected stall in __hrtimer_run_queues kernel C inconclusive done 22 868d 1414d 0/28 auto-obsoleted due to no activity on 2022/12/19 03:45

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 144s! [syz.5.1327:11745]
Modules linked in:
irq event stamp: 13238035
hardirqs last  enabled at (13238034): [<ffffffff8bc8aa13>] irqentry_exit+0x63/0x90 kernel/entry/common.c:357
hardirqs last disabled at (13238035): [<ffffffff8bc8862e>] sysvec_apic_timer_interrupt+0xe/0xc0 arch/x86/kernel/apic/apic.c:1049
softirqs last  enabled at (13228252): [<ffffffff8161d3a7>] __do_softirq kernel/softirq.c:595 [inline]
softirqs last  enabled at (13228252): [<ffffffff8161d3a7>] invoke_softirq kernel/softirq.c:435 [inline]
softirqs last  enabled at (13228252): [<ffffffff8161d3a7>] __irq_exit_rcu+0xf7/0x220 kernel/softirq.c:662
softirqs last disabled at (13228255): [<ffffffff8161d3a7>] __do_softirq kernel/softirq.c:595 [inline]
softirqs last disabled at (13228255): [<ffffffff8161d3a7>] invoke_softirq kernel/softirq.c:435 [inline]
softirqs last disabled at (13228255): [<ffffffff8161d3a7>] __irq_exit_rcu+0xf7/0x220 kernel/softirq.c:662
CPU: 0 UID: 0 PID: 11745 Comm: syz.5.1327 Not tainted 6.13.0-rc3-syzkaller-ge84a3bf7f4aa #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xd8/0x140 kernel/locking/spinlock.c:194
Code: 9c 8f 44 24 20 42 80 3c 23 00 74 08 4c 89 f7 e8 5e 79 3a f6 f6 44 24 21 02 75 52 41 f7 c7 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> d3 26 a2 f5 65 8b 05 74 7a 38 74 85 c0 74 43 48 c7 04 24 0e 36
RSP: 0018:ffffc90000007be0 EFLAGS: 00000206
RAX: cc1ce4fbe94d2b00 RBX: 1ffff92000000f80 RCX: ffffffff817b275a
RDX: dffffc0000000000 RSI: ffffffff8c0a9760 RDI: 0000000000000001
RBP: ffffc90000007c70 R08: ffffffff942a48b7 R09: 1ffffffff2854916
R10: dffffc0000000000 R11: fffffbfff2854917 R12: dffffc0000000000
R13: 1ffff92000000f7c R14: ffffc90000007c00 R15: 0000000000000246
FS:  00007fbb770f46c0(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2b50177bac CR3: 000000005a9a6000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000082
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1735 [inline]
 __hrtimer_run_queues+0x477/0xd30 kernel/time/hrtimer.c:1803
 hrtimer_run_softirq+0x19a/0x2c0 kernel/time/hrtimer.c:1820
 handle_softirqs+0x2d4/0x9b0 kernel/softirq.c:561
 __do_softirq kernel/softirq.c:595 [inline]
 invoke_softirq kernel/softirq.c:435 [inline]
 __irq_exit_rcu+0xf7/0x220 kernel/softirq.c:662
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:678
 instr_sysvec_irq_work arch/x86/kernel/irq_work.c:17 [inline]
 sysvec_irq_work+0xa3/0xc0 arch/x86/kernel/irq_work.c:17
 </IRQ>
 <TASK>
 asm_sysvec_irq_work+0x1a/0x20 arch/x86/include/asm/idtentry.h:738
RIP: 0010:preempt_schedule_irq+0xf6/0x1c0 kernel/sched/core.c:7078
Code: 89 f5 49 c1 ed 03 eb 0d 48 f7 03 08 00 00 00 0f 84 8b 00 00 00 bf 01 00 00 00 e8 25 c4 a3 f5 e8 60 71 dd f5 fb bf 01 00 00 00 <e8> 85 ab ff ff 43 80 7c 3d 00 00 74 08 4c 89 f7 e8 05 19 3c f6 48
RSP: 0018:ffffc900048ef980 EFLAGS: 00000282
RAX: cc1ce4fbe94d2b00 RBX: 1ffff9200091df38 RCX: ffffffff817b275a
RDX: dffffc0000000000 RSI: ffffffff8c0a9760 RDI: 0000000000000001
RBP: ffffc900048efa40 R08: ffffffff942a48b7 R09: 1ffffffff2854916
R10: dffffc0000000000 R11: fffffbfff2854917 R12: 1ffff9200091df30
R13: 1ffff9200091df34 R14: ffffc900048ef9a0 R15: dffffc0000000000
 irqentry_exit+0x5e/0x90 kernel/entry/common.c:354
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:put_cpu_partial+0x185/0x250
Code: 44 24 10 00 02 00 00 75 3f f7 c3 00 02 00 00 75 44 4d 85 e4 74 0b 4c 89 ff 4c 89 e6 e8 b4 e3 ff ff 65 48 8b 04 25 28 00 00 00 <48> 3b 44 24 18 0f 85 b4 00 00 00 48 83 c4 20 5b 41 5c 41 5d 41 5e
RSP: 0018:ffffc900048efb08 EFLAGS: 00000292
RAX: cc1ce4fbe94d2b00 RBX: 0000000000000282 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: 0000000000000000 R08: ffffffff942a48b7 R09: 1ffffffff2854916
R10: dffffc0000000000 R11: fffffbfff2854917 R12: ffffea00009c4200
R13: ffffea0001f89e00 R14: ffff888027b80000 R15: ffff88801ac42140
 __slab_free+0x290/0x380 mm/slub.c:4483
 qlink_free mm/kasan/quarantine.c:163 [inline]
 qlist_free_all+0x9a/0x140 mm/kasan/quarantine.c:179
 kasan_quarantine_reduce+0x14f/0x170 mm/kasan/quarantine.c:286
 __kasan_slab_alloc+0x23/0x80 mm/kasan/common.c:329
 kasan_slab_alloc include/linux/kasan.h:250 [inline]
 slab_post_alloc_hook mm/slub.c:4119 [inline]
 slab_alloc_node mm/slub.c:4168 [inline]
 kmem_cache_alloc_noprof+0x1d9/0x380 mm/slub.c:4175
 alloc_empty_file+0x9e/0x1d0 fs/file_table.c:228
 alloc_file fs/file_table.c:345 [inline]
 alloc_file_pseudo+0x1da/0x290 fs/file_table.c:376
 sock_alloc_file+0xb8/0x280 net/socket.c:468
 sock_map_fd net/socket.c:493 [inline]
 __sys_socket+0x1dd/0x3c0 net/socket.c:1709
 __do_sys_socket net/socket.c:1714 [inline]
 __se_sys_socket net/socket.c:1712 [inline]
 __x64_sys_socket+0x7a/0x90 net/socket.c:1712
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fbb76385d29
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fbb770f4038 EFLAGS: 00000246 ORIG_RAX: 0000000000000029
RAX: ffffffffffffffda RBX: 00007fbb76575fa0 RCX: 00007fbb76385d29
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000010
RBP: 00007fbb76401b08 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fbb76575fa0 R15: 00007ffdb8e6d288
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 35 Comm: kworker/u8:2 Not tainted 6.13.0-rc3-syzkaller-ge84a3bf7f4aa #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:340 [inline]
RIP: 0010:smp_call_function_many_cond+0x19f3/0x2c60 kernel/smp.c:884
Code: 45 8b 65 00 44 89 e6 83 e6 01 31 ff e8 56 e9 0b 00 41 83 e4 01 49 bc 00 00 00 00 00 fc ff df 75 07 e8 01 e5 0b 00 eb 38 f3 90 <42> 0f b6 04 23 84 c0 75 11 41 f7 45 00 01 00 00 00 74 1e e8 e5 e4
RSP: 0018:ffffc90000ab76e0 EFLAGS: 00000293
RAX: ffffffff81938f0b RBX: 1ffff110170c8c89 RCX: ffff88801dea1e00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90000ab78e0 R08: ffffffff81938eda R09: 1ffffffff2854910
R10: dffffc0000000000 R11: fffffbfff2854911 R12: dffffc0000000000
R13: ffff8880b8646448 R14: ffff8880b873fac0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880b8700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000558cf4ad3a28 CR3: 000000000e736000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1051
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2114 [inline]
 text_poke_bp_batch+0x352/0xb30 arch/x86/kernel/alternative.c:2324
 text_poke_flush arch/x86/kernel/alternative.c:2515 [inline]
 text_poke_finish+0x30/0x50 arch/x86/kernel/alternative.c:2522
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x136/0x260 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 toggle_allocation_gate+0xbc/0x260 mm/kfence/core.c:849
 process_one_work kernel/workqueue.c:3229 [inline]
 process_scheduled_works+0xa66/0x1840 kernel/workqueue.c:3310
 worker_thread+0x870/0xd30 kernel/workqueue.c:3391
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/30 03:08 bpf e84a3bf7f4aa d3ccff63 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce BUG: soft lockup in __hrtimer_run_queues
* Struck through repros no longer work on HEAD.