syzbot


BUG: soft lockup in perf_sched_delayed

Status: premoderation: reported on 2024/06/16 15:51
Reported-by: syzbot+4ddc1ade9d3ea91ba115@syzkaller.appspotmail.com
First crash: 36d, last: 36d

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 246s! [kworker/0:1:20]
Modules linked in:
CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G        W         5.15.149-syzkaller-00165-g85445b5a2107 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
Workqueue: events perf_sched_delayed
RIP: 0010:__sanitizer_cov_trace_pc+0x1/0x60 kernel/kcov.c:190
Code: 00 00 0f 0b 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb e8 13 00 00 00 48 8b 3d 74 1e 96 05 48 89 de e8 84 76 41 00 5b 5d c3 cc 55 <48> 89 e5 48 8b 45 08 65 48 8b 0d 10 36 92 7e 65 8b 15 11 36 92 7e
RSP: 0018:ffffc90000147970 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 1ffff1103ee2784d RCX: ffff8881003362c0
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90000147a98 R08: ffffffff8165929a R09: ffffed103ee071d3
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000001
R13: ffff8881f713c268 R14: ffff8881f7038e80 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8881f7000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2ef32000 CR3: 000000000680f000 CR4: 00000000003506b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x40/0x80 kernel/smp.c:1135
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1201 [inline]
 text_poke_bp_batch+0x1c4/0x5d0 arch/x86/kernel/alternative.c:1392
 text_poke_flush arch/x86/kernel/alternative.c:1560 [inline]
 text_poke_finish+0x1a/0x30 arch/x86/kernel/alternative.c:1567
 arch_jump_label_transform_apply+0x15/0x30 arch/x86/kernel/jump_label.c:146
 __jump_label_update+0x36a/0x380 kernel/jump_label.c:459
 jump_label_update+0x3af/0x450 kernel/jump_label.c:830
 static_key_disable_cpuslocked+0xcd/0x1b0 kernel/jump_label.c:207
 static_key_disable+0x1a/0x30 kernel/jump_label.c:215
 perf_sched_delayed+0x64/0x80 kernel/events/core.c:4970
 process_one_work+0x6bb/0xc10 kernel/workqueue.c:2325
 worker_thread+0xad5/0x12a0 kernel/workqueue.c:2472
 kthread+0x421/0x510 kernel/kthread.c:337
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 1971 Comm: syz-executor.1 Tainted: G        W         5.15.149-syzkaller-00165-g85445b5a2107 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
RIP: 0010:native_halt arch/x86/include/asm/irqflags.h:57 [inline]
RIP: 0010:halt arch/x86/include/asm/irqflags.h:98 [inline]
RIP: 0010:kvm_wait+0x117/0x180 arch/x86/kernel/kvm.c:912
Code: 48 c1 e8 03 42 0f b6 04 20 84 c0 44 8b 74 24 1c 75 53 41 0f b6 45 00 44 38 f0 0f 85 63 ff ff ff 66 90 0f 00 2d 7a 02 f3 03 f4 <e9> 54 ff ff ff fa 4c 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 44 8b
RSP: 0000:ffffc900001d07a0 EFLAGS: 00000046
RAX: 0000000000000003 RBX: 1ffff9200003a0f8 RCX: ffffffff8154fb7f
RDX: dffffc0000000000 RSI: 0000000000000003 RDI: ffff8881f7137540
RBP: ffffc900001d0850 R08: dffffc0000000000 R09: ffffed103ee26ea9
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff8881f7137540 R14: 0000000000000003 R15: 1ffff9200003a0fc
FS:  00007ff97ac696c0(0000) GS:ffff8881f7100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2f222000 CR3: 000000012ecf7000 CR4: 00000000003506a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 pv_wait arch/x86/include/asm/paravirt.h:597 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:470 [inline]
 __pv_queued_spin_lock_slowpath+0x6bc/0xc40 kernel/locking/qspinlock.c:508
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:585 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:85 [inline]
 do_raw_spin_lock include/linux/spinlock.h:187 [inline]
 __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_lock+0x139/0x1b0 kernel/locking/spinlock.c:154
 __queue_work+0x58c/0xcd0
 __queue_delayed_work+0x182/0x1f0 kernel/workqueue.c:1686
 mod_delayed_work_on+0xee/0x190 kernel/workqueue.c:1760
 kblockd_mod_delayed_work_on+0x2a/0x40 block/blk-core.c:1658
 blk_mq_kick_requeue_list block/blk-mq.c:840 [inline]
 blk_mq_add_to_requeue_list+0x29c/0x300 block/blk-mq.c:835
 blk_flush_queue_rq block/blk-flush.c:136 [inline]
 blk_kick_flush block/blk-flush.c:348 [inline]
 blk_flush_complete_seq+0xaf9/0xd00 block/blk-flush.c:211
 mq_flush_data_end_io+0x34d/0x480 block/blk-flush.c:369
 __blk_mq_end_request+0x3ac/0x400 block/blk-mq.c:569
 scsi_end_request+0x3c4/0x7c0 drivers/scsi/scsi_lib.c:578
 scsi_io_completion+0x1aa/0x460 drivers/scsi/scsi_lib.c:940
 scsi_finish_command+0x307/0x420 drivers/scsi/scsi.c:207
 scsi_complete+0x155/0x4d0 drivers/scsi/scsi_lib.c:1433
 blk_complete_reqs block/blk-mq.c:590 [inline]
 blk_done_softirq+0xfd/0x140 block/blk-mq.c:595
 __do_softirq+0x26d/0x5bf kernel/softirq.c:565
 do_softirq+0xf6/0x150 kernel/softirq.c:452
 </IRQ>
 <TASK>
 __local_bh_enable_ip+0x75/0x80 kernel/softirq.c:379
 __raw_spin_unlock_bh include/linux/spinlock_api_smp.h:176 [inline]
 _raw_spin_unlock_bh+0x51/0x60 kernel/locking/spinlock.c:210
 sock_hash_delete_elem+0x2a2/0x2f0 net/core/sock_map.c:945
 bpf_prog_2c29ac5cdc6b1842+0x3a/0x7fc
 bpf_dispatcher_nop_func include/linux/bpf.h:785 [inline]
 __bpf_prog_run include/linux/filter.h:625 [inline]
 bpf_prog_run include/linux/filter.h:632 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1880 [inline]
 bpf_trace_run1+0xbf/0x1c0 kernel/trace/bpf_trace.c:1916
 __bpf_trace_workqueue_activate_work+0x1d/0x30 include/trace/events/workqueue.h:59
 __traceiter_workqueue_activate_work+0x68/0xb0 include/trace/events/workqueue.h:59
 trace_workqueue_activate_work include/trace/events/workqueue.h:59 [inline]
 __queue_work+0xc18/0xcd0 kernel/workqueue.c:1528
 __queue_delayed_work+0x182/0x1f0 kernel/workqueue.c:1686
 mod_delayed_work_on+0xee/0x190 kernel/workqueue.c:1760
 kblockd_mod_delayed_work_on+0x2a/0x40 block/blk-core.c:1658
 __blk_mq_delay_run_hw_queue+0x4c3/0x580 block/blk-mq.c:1599
 blk_mq_run_hw_queue+0x2a0/0x3c0 block/blk-mq.c:1644
 blk_mq_sched_insert_requests+0x1b7/0x330 block/blk-mq-sched.c:519
 blk_mq_flush_plug_list+0x5d8/0x7c0 block/blk-mq.c:1968
 blk_flush_plug_list+0x44b/0x490 block/blk-core.c:1756
 blk_schedule_flush_plug include/linux/blkdev.h:1250 [inline]
 io_schedule_prepare kernel/sched/core.c:8609 [inline]
 io_schedule_timeout+0x91/0x130 kernel/sched/core.c:8628
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common_io+0x187/0x2e0 kernel/sched/completion.c:123
 wait_for_completion_io_timeout+0x9/0x10 kernel/sched/completion.c:191
 submit_bio_wait+0x15b/0x210 block/bio.c:1237
 blkdev_issue_discard+0x139/0x1d0 block/blk-lib.c:143
 sb_issue_discard include/linux/blkdev.h:1326 [inline]
 ext4_issue_discard+0x24b/0x480 fs/ext4/mballoc.c:3666
 ext4_trim_extent fs/ext4/mballoc.c:6431 [inline]
 ext4_try_to_trim_range+0x6dc/0x1180 fs/ext4/mballoc.c:6486
 ext4_trim_all_free fs/ext4/mballoc.c:6548 [inline]
 ext4_trim_fs+0xd9b/0x16a0 fs/ext4/mballoc.c:6639
 __ext4_ioctl fs/ext4/ioctl.c:1122 [inline]
 ext4_ioctl+0x21ad/0x5830 fs/ext4/ioctl.c:1276
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:874 [inline]
 __se_sys_ioctl+0x114/0x190 fs/ioctl.c:860
 __x64_sys_ioctl+0x7b/0x90 fs/ioctl.c:860
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x61/0xcb
RIP: 0033:0x7ff97b8eeea9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ff97ac690c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007ff97ba25f80 RCX: 00007ff97b8eeea9
RDX: 0000000020000040 RSI: 00000000c0185879 RDI: 0000000000000009
RBP: 00007ff97b95dff4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007ff97ba25f80 R15: 00007ffff24c5968
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/06/16 15:50 android13-5.15-lts 85445b5a2107 f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in perf_sched_delayed
* Struck through repros no longer work on HEAD.