syzbot


INFO: task hung in io_req_task_cancel

Status: auto-obsoleted due to no activity on 2023/05/26 13:07
Subsystems: io-uring
[Documentation on labels]
First crash: 703d, last: 703d

Sample crash report:
INFO: task syz-executor.5:12983 blocked for more than 143 seconds.
      Not tainted 6.2.0-syzkaller-10827-g489fa31ea873 #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor.5  state:D stack:28024 pid:12983 ppid:12940  flags:0x00004006
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5304 [inline]
 __schedule+0xcce/0x5b20 kernel/sched/core.c:6622
 schedule+0xde/0x1a0 kernel/sched/core.c:6698
 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:6757
 __mutex_lock_common kernel/locking/mutex.c:679 [inline]
 __mutex_lock+0xa3b/0x1350 kernel/locking/mutex.c:747
 io_tw_lock io_uring/io_uring.h:305 [inline]
 io_req_task_cancel+0xb1/0x110 io_uring/io_uring.c:1412
 handle_tw_list io_uring/io_uring.c:1184 [inline]
 tctx_task_work+0x2d7/0xb30 io_uring/io_uring.c:1246
 task_work_run+0x16f/0x270 kernel/task_work.c:179
 get_signal+0x1c7/0x25b0 kernel/signal.c:2635
 arch_do_signal_or_restart+0x79/0x5c0 arch/x86/kernel/signal.c:306
 exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
 exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:203
 __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
 syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:296
 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f9f8028c0f9
RSP: 002b:00007f9f7edbc168 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: 0000000000002412 RBX: 00007f9f803ac120 RCX: 00007f9f8028c0f9
RDX: 0000000000000000 RSI: 0000000000002412 RDI: 0000000000000003
RBP: 00007f9f802e7ae9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f9f804cfb1f R14: 00007f9f7edbc300 R15: 0000000000022000
 </TASK>

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/12:
 #0: ffffffff8c796830 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:510
1 lock held by rcu_tasks_trace/13:
 #0: ffffffff8c796530 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}
, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:510
1 lock held by khungtaskd/28:
 #0: ffffffff8c797380 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x340 kernel/locking/lockdep.c:6495
2 locks held by getty/4761:
 #0: ffff88802bdda098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x26/0x80 drivers/tty/tty_ldisc.c:244
 #1: ffffc900015802f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xef4/0x13e0 drivers/tty/n_tty.c:2177
5 locks held by kworker/0:6/5172:
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
 #0: ffff888017c39d38 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_one_work+0x86d/0x1820 kernel/workqueue.c:2361
 #1: ffffc90004d2fda8 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_one_work+0x8a1/0x1820 kernel/workqueue.c:2365
 #2: ffff888147821190 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:832 [inline]
 #2: ffff888147821190 (&dev->mutex){....}-{3:3}, at: hub_event+0x1c0/0x4e40 drivers/usb/core/hub.c:5739
 #3: ffff88807d729190 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:832 [inline]
 #3: ffff88807d729190 (&dev->mutex){....}-{3:3}, at: __device_attach+0x76/0x4b0 drivers/base/dd.c:973
 #4: ffff88802be69118 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:832 [inline]
 #4: ffff88802be69118 (&dev->mutex){....}-{3:3}, at: __device_attach+0x76/0x4b0 drivers/base/dd.c:973
2 locks held by kworker/0:10/5932:
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x86d/0x1820 kernel/workqueue.c:2361
 #1: ffffc9001514fda8 ((work_completion)(&rew->rew_work)){+.+.}-{0:0}, at: process_one_work+0x8a1/0x1820 kernel/workqueue.c:2365
1 lock held by syz-executor.2/12887:
1 lock held by syz-executor.1/12882:
1 lock held by syz-executor.5/12888:
1 lock held by syz-executor.5/12896:
 #0: ffff88807a5840a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_tw_lock io_uring/io_uring.h:305 [inline]
 #0: ffff88807a5840a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_req_task_cancel+0xb1/0x110 io_uring/io_uring.c:1412
1 lock held by syz-executor.2/12969:
1 lock held by syz-executor.2/12971:
 #0: ffff88804b79c0a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_tw_lock io_uring/io_uring.h:305 [inline]
 #0: ffff88804b79c0a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_req_task_cancel+0xb1/0x110 io_uring/io_uring.c:1412
1 lock held by syz-executor.1/12976:
1 lock held by syz-executor.1/12980:
 #0: ffff88802cbf20a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_tw_lock io_uring/io_uring.h:305 [inline]
 #0: ffff88802cbf20a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_req_task_cancel+0xb1/0x110 io_uring/io_uring.c:1412
1 lock held by syz-executor.5/12979:
1 lock held by syz-executor.5/12983:
 #0: ffff888029bd40a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_tw_lock io_uring/io_uring.h:305 [inline]
 #0: ffff888029bd40a8 (&ctx->uring_lock){+.+.}-{3:3}, at: io_req_task_cancel+0xb1/0x110 io_uring/io_uring.c:1412
3 locks held by kworker/u4:3/13799:
2 locks held by syz-executor.2/14237:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: ip_tunnel_delete_nets+0x94/0x3a0 net/ipv4/ip_tunnel.c:1121
2 locks held by syz-executor.2/14252:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: xfrmi_exit_batch_net+0x81/0x3e0 net/xfrm/xfrm_interface_core.c:966
2 locks held by syz-executor.2/14256:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: wg_netns_pre_exit+0x1b/0x210 drivers/net/wireguard/device.c:420
2 locks held by syz-executor.5/14245:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: vti6_exit_batch_net+0x86/0x730 net/ipv6/ip6_vti.c:1183
3 locks held by syz-executor.5/14246:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: ip_tunnel_delete_nets+0x94/0x3a0 net/ipv4/ip_tunnel.c:1121
 #2: ffffffff8c7a2678 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:293 [inline]
 #2: ffffffff8c7a2678 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x64a/0x770 kernel/rcu/tree_exp.h:989
2 locks held by syz-executor.5/14254:
 #0: ffffffff8e0f0610 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4cb/0x8e0 net/core/net_namespace.c:486
 #1: ffffffff8e104248 (rtnl_mutex){+.+.}-{3:3}, at: ip_tunnel_delete_nets+0x94/0x3a0 net/ipv4/ip_tunnel.c:1121
1 lock held by syz-executor.3/14262:
1 lock held by syz-executor.0/14268:

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 28 Comm: khungtaskd Not tainted 6.2.0-syzkaller-10827-g489fa31ea873 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/16/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xd9/0x150 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x316/0x3e0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x33f/0x460 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:222 [inline]
 watchdog+0xe94/0x11e0 kernel/hung_task.c:379
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 12882 Comm: syz-executor.1 Not tainted 6.2.0-syzkaller-10827-g489fa31ea873 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/16/2023
RIP: 0010:lockdep_hardirqs_on_prepare+0xc/0x410 kernel/locking/lockdep.c:4276
Code: e8 99 94 70 00 e9 6d ff ff ff 48 89 34 24 e8 4b 94 70 00 48 8b 34 24 eb a7 0f 1f 44 00 00 f3 0f 1e fa 48 c7 c0 70 81 78 8e 55 <48> ba 00 00 00 00 00 fc ff df 48 89 c1 53 83 e0 07 48 c1 e9 03 83
RSP: 0018:ffffc9000d1376f0 EFLAGS: 00000082
RAX: ffffffff8e788170 RBX: ffff8880235d3a80 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
RBP: ffff8880465fb780 R08: 0000000000000000 R09: ffffffff8e784c57
R10: fffffbfff1cf098a R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffc9000d1378c8 R15: 0000000080810f00
FS:  00007f1cc1b51700(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000201e3030 CR3: 00000000289cd000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 trace_hardirqs_on+0x36/0x40 kernel/trace/trace_preemptirq.c:61
 __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:159 [inline]
 _raw_spin_unlock_irq+0x23/0x50 kernel/locking/spinlock.c:202
 spin_unlock_irq include/linux/spinlock.h:400 [inline]
 copy_process+0x322/0x76c0 kernel/fork.c:2094
 create_io_thread+0xaf/0xf0 kernel/fork.c:2632
 create_io_worker+0x260/0x630 io_uring/io-wq.c:820
 io_wqe_create_worker io_uring/io-wq.c:310 [inline]
 io_wqe_enqueue+0x6c2/0xc20 io_uring/io-wq.c:936
 io_queue_iowq+0x282/0x5d0 io_uring/io_uring.c:475
 io_req_task_submit+0x190/0x290 io_uring/io_uring.c:1423
 handle_tw_list io_uring/io_uring.c:1184 [inline]
 tctx_task_work+0x2d7/0xb30 io_uring/io_uring.c:1246
 task_work_run+0x16f/0x270 kernel/task_work.c:179
 get_signal+0x1c7/0x25b0 kernel/signal.c:2635
 arch_do_signal_or_restart+0x79/0x5c0 arch/x86/kernel/signal.c:306
 exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
 exit_to_user_mode_prepare+0x15f/0x250 kernel/entry/common.c:203
 __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
 syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:296
 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f1cc0e8c0f9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f1cc1b51168 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: 00000000000040b3 RBX: 00007f1cc0fabf80 RCX: 00007f1cc0e8c0f9
RDX: 0000000000000000 RSI: 00000000000040b3 RDI: 0000000000000003
RBP: 00007f1cc0ee7ae9 R08: 0000000020000000 R09: 0000000000000008
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f1cc10cfb1f R14: 00007f1cc1b51300 R15: 0000000000022000
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/02/25 12:58 upstream 489fa31ea873 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in io_req_task_cancel
* Struck through repros no longer work on HEAD.