syzbot


INFO: rcu detected stall in blk_mq_run_hw_queues

Status: auto-obsoleted due to no activity on 2025/01/24 01:10
Subsystems: block net virt
[Documentation on labels]
First crash: 197d, last: 95d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 INFO: rcu detected stall in blk_mq_run_hw_queues 1 357d 357d 0/3 auto-obsoleted due to no activity on 2024/05/17 10:13

Sample crash report:
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: {
 1-...D
 } 2654 jiffies s: 33833 root: 0x2/.
rcu: blocking rcu_node structures (internal RCU debug):
Sending NMI from CPU 0 to CPUs 1:
 stack:25424 pid:13317 tgid:13317 ppid:8153   flags:0x00000002
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 342 Comm: kworker/1:1H Not tainted 6.12.0-rc4-syzkaller-00052-gc6d9e43954bf #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: kblockd blk_mq_requeue_work
RIP: 0010:__sanitizer_cov_trace_pc+0x5a/0x70 kernel/kcov.c:223
Code: 35 8b 82 54 15 00 00 85 c0 74 2b 8b 82 30 15 00 00 83 f8 02 75 20 48 8b 8a 38 15 00 00 8b 92 34 15 00 00 48 8b 01 48 83 c0 01 <48> 39 d0 73 07 48 89 01 48 89 34 c1 c3 cc cc cc cc 0f 1f 44 00 00
RSP: 0018:ffffc900001b7ba8 EFLAGS: 00000016
RAX: 0000000000040000 RBX: ffffffff8728c781 RCX: ffffc90001202000
RDX: 0000000000040000 RSI: ffffffff86e67a60 RDI: ffffc900001b7c18
RBP: ffffffff8728c782 R08: 0000000000000005 R09: 0000000000000009
R10: 0000000000000005 R11: 00000000004e2dd8 R12: ffffc900001b7ce0
R13: ffffc900001b7c18 R14: 0000000000000035 R15: 0000000000000005
FS:  0000000000000000(0000) GS:ffff8881f5900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4ab8c11000 CR3: 0000000116670000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 skip_atoi+0x10/0xe0 lib/vsprintf.c:169
 format_decode+0x95c/0xba0 lib/vsprintf.c:2571
 vsnprintf+0x13d/0x1880 lib/vsprintf.c:2755
 sprintf+0xcd/0x110 lib/vsprintf.c:3007
 print_time kernel/printk/printk.c:1362 [inline]
 info_print_prefix+0x25c/0x350 kernel/printk/printk.c:1388
 record_print_text+0x141/0x400 kernel/printk/printk.c:1437
 printk_get_next_message+0x2a6/0x670 kernel/printk/printk.c:2978
 console_emit_next_record kernel/printk/printk.c:3046 [inline]
 console_flush_all+0x6ec/0xc60 kernel/printk/printk.c:3180
 __console_flush_and_unlock kernel/printk/printk.c:3239 [inline]
 console_unlock+0xd9/0x210 kernel/printk/printk.c:3279
 vprintk_emit+0x424/0x6f0 kernel/printk/printk.c:2407
 vprintk+0x7f/0xa0 kernel/printk/printk_safe.c:68
 _printk+0xc8/0x100 kernel/printk/printk.c:2432
 sched_show_task kernel/sched/core.c:7597 [inline]
 sched_show_task+0x3c1/0x5f0 kernel/sched/core.c:7579
 show_state_filter+0xee/0x320 kernel/sched/core.c:7649
 k_spec drivers/tty/vt/keyboard.c:667 [inline]
 k_spec+0xed/0x150 drivers/tty/vt/keyboard.c:656
 kbd_keycode drivers/tty/vt/keyboard.c:1522 [inline]
 kbd_event+0xcbd/0x17a0 drivers/tty/vt/keyboard.c:1541
 input_handler_events_default+0x116/0x1b0 drivers/input/input.c:2549
 input_pass_values+0x777/0x8e0 drivers/input/input.c:126
 input_event_dispose drivers/input/input.c:352 [inline]
 input_handle_event+0xb30/0x14d0 drivers/input/input.c:369
 input_event drivers/input/input.c:398 [inline]
 input_event+0x83/0xa0 drivers/input/input.c:390
 hidinput_hid_event+0xa12/0x2410 drivers/hid/hid-input.c:1719
 hid_process_event+0x4b7/0x5e0 drivers/hid/hid-core.c:1540
 hid_input_array_field+0x330/0x710 drivers/hid/hid-core.c:1644
 hid_process_report drivers/hid/hid-core.c:1694 [inline]
 hid_report_raw_event+0xa02/0x11c0 drivers/hid/hid-core.c:2040
 __hid_input_report.constprop.0+0x341/0x440 drivers/hid/hid-core.c:2110
 hid_irq_in+0x35e/0x870 drivers/hid/usbhid/hid-core.c:285
 __usb_hcd_giveback_urb+0x389/0x6e0 drivers/usb/core/hcd.c:1650
 usb_hcd_giveback_urb+0x396/0x450 drivers/usb/core/hcd.c:1734
 dummy_timer+0x17f0/0x3930 drivers/usb/gadget/udc/dummy_hcd.c:1993
 __run_hrtimer kernel/time/hrtimer.c:1691 [inline]
 __hrtimer_run_queues+0x20a/0xae0 kernel/time/hrtimer.c:1755
 hrtimer_run_softirq+0x17d/0x350 kernel/time/hrtimer.c:1772
 handle_softirqs+0x206/0x8d0 kernel/softirq.c:554
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu kernel/softirq.c:637 [inline]
 irq_exit_rcu+0xac/0x110 kernel/softirq.c:649
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0x90/0xb0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x31/0x80 kernel/locking/spinlock.c:194
Code: f5 53 48 8b 74 24 10 48 89 fb 48 83 c7 18 e8 f6 82 42 fa 48 89 df e8 0e 00 43 fa f7 c5 00 02 00 00 75 23 9c 58 f6 c4 02 75 37 <bf> 01 00 00 00 e8 b5 2f 37 fa 65 8b 05 56 ed 12 79 85 c0 74 16 5b
RSP: 0018:ffffc9000186f618 EFLAGS: 00000246
RAX: 0000000000000002 RBX: ffffffff935af548 RCX: 1ffffffff1f5bf87
RDX: 0000000000000000 RSI: ffffffff8727f220 RDI: ffffffff8746ebc0
RBP: 0000000000000a06 R08: 0000000000000001 R09: fffffbfff1f561c2
R10: ffffffff8fab0e17 R11: 0000000000000000 R12: 1ffff9200030deca
R13: ffff8881033b11c0 R14: 0000000000000000 R15: ffffffff935af548
 __debug_object_init+0x34a/0x480 lib/debugobjects.c:645
 scsi_initialize_rq+0xa1/0x160 drivers/scsi/scsi_lib.c:1212
 scsi_init_command+0x217/0x290 drivers/scsi/scsi_lib.c:1248
 scsi_prepare_cmd drivers/scsi/scsi_lib.c:1635 [inline]
 scsi_queue_rq+0x1887/0x3360 drivers/scsi/scsi_lib.c:1825
 blk_mq_dispatch_rq_list+0x440/0x1e20 block/blk-mq.c:2030
 __blk_mq_sched_dispatch_requests+0x219/0x1620 block/blk-mq-sched.c:301
 blk_mq_sched_dispatch_requests+0xd4/0x150 block/blk-mq-sched.c:331
 blk_mq_run_hw_queue+0x645/0x9a0 block/blk-mq.c:2243
 blk_mq_run_hw_queues+0x113/0x290 block/blk-mq.c:2292
 blk_mq_requeue_work+0x649/0x8d0 block/blk-mq.c:1477
 process_one_work+0x9c5/0x1ba0 kernel/workqueue.c:3229
 process_scheduled_works kernel/workqueue.c:3310 [inline]
 worker_thread+0x6c8/0xf00 kernel/workqueue.c:3391
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5328 [inline]
 __schedule+0x105f/0x34b0 kernel/sched/core.c:6690
 do_task_dead+0xd6/0x110 kernel/sched/core.c:6706
 do_exit+0x1de7/0x2ce0 kernel/exit.c:990
 do_group_exit+0xd3/0x2a0 kernel/exit.c:1088
 __do_sys_exit_group kernel/exit.c:1099 [inline]
 __se_sys_exit_group kernel/exit.c:1097 [inline]
 __x64_sys_exit_group+0x3e/0x50 kernel/exit.c:1097
 x64_sys_call+0x14a9/0x16a0 arch/x86/include/generated/asm/syscalls_64.h:232
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f824b140a90
RSP: 002b:00007ffd93531138 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007f824b231860 RCX: 00007f824b140a90
RDX: 00000000000000e7 RSI: 000000000000003c RDI: 0000000000000001
RBP: 00007f824b231860 R08: 0000000000000001 R09: 2e91b3eb1fb8892d
R10: 00007ffd93530ff0 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 00007f824b235658 R15: 0000000000000001
 </TASK>
task:kworker/u8:5    state:I stack:30928 pid:13326 tgid:13326 ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5328 [inline]
 __schedule+0x105f/0x34b0 kernel/sched/core.c:6690
 __schedule_loop kernel/sched/core.c:6767 [inline]
 schedule+0xe7/0x350 kernel/sched/core.c:6782
 worker_thread+0x2de/0xf00 kernel/workqueue.c:3406
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Showing all locks held in the system:
7 locks held by kworker/1:1H/342:
 #0: ffff888100eef548 ((wq_completion)kblockd){+.+.}-{0:0}, at: process_one_work+0x129b/0x1ba0 kernel/workqueue.c:3204
 #1: ffffc9000186fd80 ((work_completion)(&(&q->requeue_work)->work)){+.+.}-{0:0}, at: process_one_work+0x921/0x1ba0 kernel/workqueue.c:3205
 #2: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #2: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #2: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: blk_mq_run_hw_queue+0x619/0x9a0 block/blk-mq.c:2243
 #3: ffff888116459230 (&dev->event_lock){..-.}-{2:2}, at: input_event drivers/input/input.c:397 [inline]
 #3: ffff888116459230 (&dev->event_lock){..-.}-{2:2}, at: input_event+0x70/0xa0 drivers/input/input.c:390
 #4: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #4: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #4: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: input_pass_values+0x8b/0x8e0 drivers/input/input.c:118
 #5: ffffffff89387a58 (kbd_event_lock){..-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
 #5: ffffffff89387a58 (kbd_event_lock){..-.}-{2:2}, at: kbd_event+0x8a/0x17a0 drivers/tty/vt/keyboard.c:1535
 #6: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #6: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #6: ffffffff88ebb140 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x7f/0x390 kernel/locking/lockdep.c:6720
2 locks held by getty/2920:
 #0: ffff8881124190a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x80 drivers/tty/tty_ldisc.c:243
 #1: ffffc900000432f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xfba/0x1480 drivers/tty/n_tty.c:2211
6 locks held by kworker/0:6/5487:
 #0: ffff888105ef6d48 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_one_work+0x129b/0x1ba0 kernel/workqueue.c:3204
 #1: ffffc9000f127d80 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_one_work+0x921/0x1ba0 kernel/workqueue.c:3205
 #2: ffff88810af0f190 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #2: ffff88810af0f190 (&dev->mutex){....}-{3:3}, at: hub_event+0x1be/0x4f40 drivers/usb/core/hub.c:5849
 #3: ffff888137284190 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #3: ffff888137284190 (&dev->mutex){....}-{3:3}, at: usb_disconnect+0x10a/0x920 drivers/usb/core/hub.c:2295
 #4: ffff8881196fe160 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #4: ffff8881196fe160 (&dev->mutex){....}-{3:3}, at: __device_driver_lock drivers/base/dd.c:1095 [inline]
 #4: ffff8881196fe160 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0xa4/0x610 drivers/base/dd.c:1293
 #5: ffff8881169b5a20 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #5: ffff8881169b5a20 (&dev->mutex){....}-{3:3}, at: __device_driver_lock drivers/base/dd.c:1095 [inline]
 #5: ffff8881169b5a20 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0xa4/0x610 drivers/base/dd.c:1293
3 locks held by kworker/1:0/9255:
1 lock held by syz.2.2161/12936:
 #0: ffffffff88ec6a38 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock+0x282/0x3b0 kernel/rcu/tree_exp.h:297
1 lock held by syz.0.2162/12939:
 #0: ffffffff88ec6a38 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock+0x1a4/0x3b0 kernel/rcu/tree_exp.h:329

=============================================

keytouch 0003:0926:3333.0130: can't resubmit intr, dummy_hcd.3-1/input0, status -19
keytouch 0003:0926:3333.0130: usb_submit_urb(ctrl) failed: -19

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/10/26 01:02 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing c6d9e43954bf 045e728d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in blk_mq_run_hw_queues
2024/08/01 04:48 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing d483f034f032 1e9c4cf3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in blk_mq_run_hw_queues
2024/07/16 02:30 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing b72749301112 e8709b21 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in blk_mq_run_hw_queues
* Struck through repros no longer work on HEAD.