syzbot


INFO: task hung in io_put_sq_data

Status: auto-obsoleted due to no activity on 2024/08/18 03:55
Subsystems: io-uring
[Documentation on labels]
First crash: 253d, last: 253d

Sample crash report:
INFO: task kworker/u8:3:51 blocked for more than 143 seconds.
      Not tainted 6.9.0-syzkaller-09429-g61307b7be41a #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u8:3    state:D stack:20688 pid:51    tgid:51    ppid:2      flags:0x00004000
Workqueue: iou_exit io_ring_exit_work
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5408 [inline]
 __schedule+0x17e8/0x4a50 kernel/sched/core.c:6745
 __schedule_loop kernel/sched/core.c:6822 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6837
 schedule_timeout+0xb0/0x310 kernel/time/timer.c:2557
 do_wait_for_common kernel/sched/completion.c:95 [inline]
 __wait_for_common kernel/sched/completion.c:116 [inline]
 wait_for_common kernel/sched/completion.c:127 [inline]
 wait_for_completion+0x355/0x620 kernel/sched/completion.c:148
 io_put_sq_data+0x7e/0xe0 io_uring/sqpoll.c:74
 io_sq_thread_finish+0x20e/0x250 io_uring/sqpoll.c:99
 io_ring_ctx_free+0x28/0xea0 io_uring/io_uring.c:2637
 io_ring_exit_work+0x80f/0x8a0 io_uring/io_uring.c:2885
 process_one_work kernel/workqueue.c:3231 [inline]
 process_scheduled_works+0xa2e/0x1830 kernel/workqueue.c:3312
 worker_thread+0x86d/0xd70 kernel/workqueue.c:3393
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Showing all locks held in the system:
2 locks held by kworker/u8:0/11:
1 lock held by khungtaskd/30:
 #0: ffffffff8e333d20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
 #0: ffffffff8e333d20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
 #0: ffffffff8e333d20 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x2a0 kernel/locking/lockdep.c:6614
2 locks held by kworker/u8:3/51:
 #0: ffff888017fd5948 ((wq_completion)iou_exit){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3206 [inline]
 #0: ffff888017fd5948 ((wq_completion)iou_exit){+.+.}-{0:0}, at: process_scheduled_works+0x90a/0x1830 kernel/workqueue.c:3312
 #1: ffffc90000bb7d00 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3207 [inline]
 #1: ffffc90000bb7d00 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_scheduled_works+0x945/0x1830 kernel/workqueue.c:3312
5 locks held by kworker/u8:7/1105:
 #0: ffff888015ecd948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3206 [inline]
 #0: ffff888015ecd948 ((wq_completion)netns){+.+.}-{0:0}, at: process_scheduled_works+0x90a/0x1830 kernel/workqueue.c:3312
 #1: ffffc9000479fd00 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3207 [inline]
 #1: ffffc9000479fd00 (net_cleanup_work){+.+.}-{0:0}, at: process_scheduled_works+0x945/0x1830 kernel/workqueue.c:3312
 #2: ffffffff8f5d3c90 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0x16a/0xcc0 net/core/net_namespace.c:594
 #3: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0xe9/0xa90 net/core/dev.c:11745
 #4: ffffffff8e3390f8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:291 [inline]
 #4: ffffffff8e3390f8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x381/0x830 kernel/rcu/tree_exp.h:939
2 locks held by getty/4845:
 #0: ffff88802af140a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
 #1: ffffc9000312b2f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6b5/0x1e10 drivers/tty/n_tty.c:2201
3 locks held by kworker/1:5/5176:
 #0: ffff888015078948 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3206 [inline]
 #0: ffff888015078948 ((wq_completion)events){+.+.}-{0:0}, at: process_scheduled_works+0x90a/0x1830 kernel/workqueue.c:3312
 #1: ffffc900053bfd00 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3207 [inline]
 #1: ffffc900053bfd00 ((linkwatch_work).work){+.+.}-{0:0}, at: process_scheduled_works+0x945/0x1830 kernel/workqueue.c:3312
 #2: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: linkwatch_event+0xe/0x60 net/core/link_watch.c:276
3 locks held by vhost-6314/6328:
2 locks held by syz-executor.4/8961:
 #0: ffff88802306a0e0 (&type->s_umount_key#33){++++}-{3:3}, at: __super_lock fs/super.c:56 [inline]
 #0: ffff88802306a0e0 (&type->s_umount_key#33){++++}-{3:3}, at: __super_lock_excl fs/super.c:71 [inline]
 #0: ffff88802306a0e0 (&type->s_umount_key#33){++++}-{3:3}, at: deactivate_super+0xb5/0xf0 fs/super.c:504
 #1: ffffffff8e3ec7e8 (lock#3){+.+.}-{3:3}, at: __lru_add_drain_all+0x66/0x570 mm/swap.c:857
1 lock held by syz-executor.0/9599:
 #0: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x842/0x10d0 net/core/rtnetlink.c:6592
1 lock held by syz-executor.1/9698:
 #0: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x842/0x10d0 net/core/rtnetlink.c:6592
1 lock held by syz-executor.2/9708:
 #0: ffffffff8f5e04c8 (rtnl_mutex){+.+.}-{3:3}, at: ip_mroute_setsockopt+0x15b/0x1190 net/ipv4/ipmr.c:1369
2 locks held by syz-executor.3/9718:
 #0: ffff888056cc60e0 (&type->s_umount_key#63/1){+.+.}-{3:3}, at: alloc_super+0x221/0x9d0 fs/super.c:343
 #1: ffffffff8e3ec7e8 (lock#3){+.+.}-{3:3}, at: __lru_add_drain_all+0x66/0x570 mm/swap.c:857

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 30 Comm: khungtaskd Not tainted 6.9.0-syzkaller-09429-g61307b7be41a #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
 nmi_cpu_backtrace+0x49c/0x4d0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x198/0x320 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:160 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:223 [inline]
 watchdog+0xfde/0x1020 kernel/hung_task.c:379
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 9718 Comm: syz-executor.3 Not tainted 6.9.0-syzkaller-09429-g61307b7be41a #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
RIP: 0010:kasan_check_range+0xb/0x290 mm/kasan/generic.c:188
Code: 90 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 55 41 57 41 56 41 54 <53> b0 01 48 85 f6 0f 84 a0 01 00 00 4c 8d 04 37 49 39 f8 0f 82 56
RSP: 0018:ffffc90000a187d8 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81727604
RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffff8faccc68
RBP: ffffc90000a18960 R08: 0000000000000001 R09: 0000000000000000
R10: dffffc0000000000 R11: fffff52000143134 R12: 1ffff92000143108
R13: dffffc0000000000 R14: 0000000000000000 R15: ffff88807cb18000
FS:  00007f439168f6c0(0000) GS:ffff8880b9500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b30626000 CR3: 000000004c642000 CR4: 0000000000350ef0
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 instrument_atomic_read include/linux/instrumented.h:68 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 cpumask_test_cpu include/linux/cpumask.h:545 [inline]
 cpu_online include/linux/cpumask.h:1160 [inline]
 trace_lock_acquire include/trace/events/lock.h:24 [inline]
 lock_acquire+0xd4/0x550 kernel/locking/lockdep.c:5725
 do_write_seqcount_begin_nested include/linux/seqlock.h:469 [inline]
 do_write_seqcount_begin include/linux/seqlock.h:495 [inline]
 psi_account_irqtime+0x1b2/0x5d0 kernel/sched/psi.c:1016
 update_rq_clock_task kernel/sched/core.c:726 [inline]
 update_rq_clock+0x22b/0x8a0 kernel/sched/core.c:770
 ttwu_queue kernel/sched/core.c:4056 [inline]
 try_to_wake_up+0x872/0x1470 kernel/sched/core.c:4378
 call_timer_fn+0x190/0x650 kernel/time/timer.c:1792
 expire_timers kernel/time/timer.c:1843 [inline]
 __run_timers kernel/time/timer.c:2417 [inline]
 __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2428
 run_timer_base kernel/time/timer.c:2437 [inline]
 run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2447
 handle_softirqs+0x2d8/0x990 kernel/softirq.c:554
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:649
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:console_trylock_spinning kernel/printk/printk.c:2010 [inline]
RIP: 0010:vprintk_emit+0x576/0x770 kernel/printk/printk.c:2347
Code: 02 20 00 4c 21 e3 0f 85 3a 01 00 00 e8 b3 fd 1f 00 4d 89 ec 4d 85 ff 75 07 e8 a6 fd 1f 00 eb 06 e8 9f fd 1f 00 fb 44 8b 3c 24 <48> c7 c7 60 f8 20 8e 31 f6 ba 01 00 00 00 31 c9 41 b8 01 00 00 00
RSP: 0018:ffffc900099bef60 EFLAGS: 00000283
RAX: ffffffff81763291 RBX: 0000000000000000 RCX: 0000000000040000
RDX: ffffc9000af77000 RSI: 000000000001a5aa RDI: 000000000001a5ab
RBP: ffffc900099bf050 R08: ffffffff8176326f R09: 1ffffffff25f42cf
R10: dffffc0000000000 R11: fffffbfff25f42d0 R12: dffffc0000000000
R13: dffffc0000000000 R14: ffffffff817630ef R15: 0000000000000050
 _printk+0xd5/0x120 kernel/printk/printk.c:2373
 __nla_validate_parse+0x2702/0x2f70 lib/nlattr.c:647
 __nla_parse+0x40/0x60 lib/nlattr.c:732
 nla_parse_deprecated include/net/netlink.h:708 [inline]
 nfnetlink_rcv_batch net/netfilter/nfnetlink.c:515 [inline]
 nfnetlink_rcv_skb_batch net/netfilter/nfnetlink.c:644 [inline]
 nfnetlink_rcv+0x13c9/0x2a80 net/netfilter/nfnetlink.c:662
 netlink_unicast_kernel net/netlink/af_netlink.c:1335 [inline]
 netlink_unicast+0x7ec/0x980 net/netlink/af_netlink.c:1361
 netlink_sendmsg+0x8e1/0xcb0 net/netlink/af_netlink.c:1905
 sock_sendmsg_nosec net/socket.c:730 [inline]
 __sock_sendmsg+0x223/0x270 net/socket.c:745
 ____sys_sendmsg+0x525/0x7d0 net/socket.c:2585
 ___sys_sendmsg net/socket.c:2639 [inline]
 __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2668
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f439087cee9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f439168f0c8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007f43909abf80 RCX: 00007f439087cee9
RDX: 0000000000000000 RSI: 0000000020000040 RDI: 0000000000000008
RBP: 00007f43908c949e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f43909abf80 R15: 00007fff85cbefe8
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/05/20 03:46 upstream 61307b7be41a c0f1611a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in io_put_sq_data
* Struck through repros no longer work on HEAD.