syzbot


INFO: task hung in io_ring_exit_work

Status: upstream: reported syz repro on 2023/04/10 16:53
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+b2439fdfc6b698781fc9@syzkaller.appspotmail.com
First crash: 407d, last: 310d
Fix commit to backport (bisect log) :
tree: upstream
commit 3954cf4338becb1f140bb6fa4f5e9a42f2529b86
Author: Christoph Hellwig <hch@lst.de>
Date: Mon Aug 22 06:14:24 2022 +0000

  devres: remove devm_ioremap_np

  
Bug presence (2)
Date Name Commit Repro Result
2023/07/16 linux-5.15.y (ToT) d54cfc420586 C [report] INFO: task hung in io_ring_exit_work
2023/07/16 upstream (ToT) 831fe284d827 C Didn't crash
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in io_ring_exit_work io-uring C unreliable unreliable 820 322d 1142d 0/26 auto-obsoleted due to no activity on 2023/09/12 07:44
linux-6.1 INFO: task hung in io_ring_exit_work 8 324d 414d 0/3 auto-obsoleted due to no activity on 2023/10/10 09:34
Fix bisection attempts (1)
Created Duration User Patch Repo Result
2023/09/03 22:49 5h23m fix candidate upstream job log (1)

Sample crash report:
INFO: task kworker/u4:7:4004 blocked for more than 143 seconds.
      Not tainted 5.15.120-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:7    state:D stack:25472 pid: 4004 ppid:     2 flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5026 [inline]
 __schedule+0x12c4/0x4590 kernel/sched/core.c:6372
 schedule+0x11b/0x1f0 kernel/sched/core.c:6455
 schedule_timeout+0xac/0x300 kernel/time/timer.c:1860
 do_wait_for_common+0x2d9/0x480 kernel/sched/completion.c:85
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x48/0x60 kernel/sched/completion.c:138
 io_ring_exit_work+0x6a9/0x1330 io_uring/io_uring.c:9725
 process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2307
 worker_thread+0xaca/0x1280 kernel/workqueue.c:2454
 kthread+0x3f6/0x4f0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>

Showing all locks held in the system:
1 lock held by khungtaskd/27:
 #0: ffffffff8c91e6a0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30
2 locks held by kworker/u4:2/154:
 #0: ffff888011c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2280
 #1: ffffc90001a8fd20 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2282
2 locks held by getty/3265:
 #0: ffff88814b25b098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x21/0x70 drivers/tty/tty_ldisc.c:252
 #1: ffffc90002bb32e8 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6af/0x1da0 drivers/tty/n_tty.c:2147
2 locks held by kworker/u4:7/4004:
 #0: ffff888011c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2280
 #1: ffffc90004937d20 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2282
2 locks held by kworker/u4:11/4358:
 #0: ffff888011c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x78a/0x10c0 kernel/workqueue.c:2280
 #1: ffffc90005fb7d20 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x7d0/0x10c0 kernel/workqueue.c:2282
1 lock held by syz-executor.5/13217:

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 27 Comm: khungtaskd Not tainted 5.15.120-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/03/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 nmi_cpu_backtrace+0x46a/0x4a0 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x181/0x2a0 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:210 [inline]
 watchdog+0xe72/0xeb0 kernel/hung_task.c:295
 kthread+0x3f6/0x4f0 kernel/kthread.c:319
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 2961 Comm: udevd Not tainted 5.15.120-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/03/2023
RIP: 0010:hlock_class kernel/locking/lockdep.c:197 [inline]
RIP: 0010:__lock_acquire+0xfb7/0x1ff0 kernel/locking/lockdep.c:5007
Code: 0e 00 00 8b 1b 81 e3 ff 1f 00 00 89 d8 c1 e8 06 48 8d 3c c5 00 21 bd 8f be 08 00 00 00 e8 c1 ec 66 00 48 0f a3 1d a9 9d 5a 0e <0f> 83 be 01 00 00 48 8d 04 5b 48 c1 e0 06 48 8d 98 00 80 8c 8f 48
RSP: 0018:ffffc90002b9f600 EFLAGS: 00000057
RAX: 0000000000000001 RBX: 0000000000000150 RCX: ffffffff8162834f
RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffff8fbd2128
RBP: b12163e75ced83e9 R08: dffffc0000000000 R09: fffffbfff1f7a426
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000001
R13: ffff88807df24668 R14: ffff88807df23b80 R15: ffff88807df246b8
FS:  00007f4e878d6c80(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555555ab9938 CR3: 000000001d8ac000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5622
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
 kasan_quarantine_reduce+0x64/0x180 mm/kasan/quarantine.c:257
 __kasan_slab_alloc+0x2f/0xc0 mm/kasan/common.c:444
 kasan_slab_alloc include/linux/kasan.h:254 [inline]
 slab_post_alloc_hook+0x53/0x380 mm/slab.h:519
 slab_alloc_node mm/slub.c:3220 [inline]
 slab_alloc mm/slub.c:3228 [inline]
 kmem_cache_alloc+0xf3/0x280 mm/slub.c:3233
 getname_flags+0xb8/0x4e0 fs/namei.c:138
 user_path_at_empty+0x2a/0x180 fs/namei.c:2852
 user_path_at include/linux/namei.h:57 [inline]
 vfs_statx+0x106/0x3b0 fs/stat.c:221
 vfs_fstatat fs/stat.c:243 [inline]
 __do_sys_newfstatat fs/stat.c:411 [inline]
 __se_sys_newfstatat fs/stat.c:405 [inline]
 __x64_sys_newfstatat+0x12c/0x1b0 fs/stat.c:405
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x61/0xcb
RIP: 0033:0x7f4e87a015f4
Code: 64 c7 00 09 00 00 00 83 c8 ff c3 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 00 00 00 00 41 89 ca b8 06 01 00 00 0f 05 <45> 31 c0 3d 00 f0 ff ff 76 10 48 8b 15 03 a8 0d 00 f7 d8 41 83 c8
RSP: 002b:00007ffeb37a0c28 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
RAX: ffffffffffffffda RBX: 000055f2ef2936e0 RCX: 00007f4e87a015f4
RDX: 00007ffeb37a0c38 RSI: 000055f2ef283897 RDI: 00000000ffffff9c
RBP: 000055f2f071ec28 R08: 0005f9c4183b50c0 R09: 0000000000000010
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 0005f9c4183b50c0 R15: 000055f2ef287dca
 </TASK>

Crashes (11):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/07/16 06:42 linux-5.15.y d54cfc420586 35d9ecc5 .config console log report syz [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/07/02 07:37 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/07/02 07:06 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/07/02 06:47 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/06/27 20:50 linux-5.15.y f67653019430 4cd5bb25 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/04/10 16:53 linux-5.15.y d86dfc4d95cd 71147e29 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan INFO: task hung in io_ring_exit_work
2023/07/02 08:27 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 INFO: task hung in io_ring_exit_work
2023/07/02 08:09 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 INFO: task hung in io_ring_exit_work
2023/07/02 06:42 linux-5.15.y 4af60700a60c bfc47836 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 INFO: task hung in io_ring_exit_work
2023/04/25 10:44 linux-5.15.y 3299fb36854f 65320f8e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 INFO: task hung in io_ring_exit_work
2023/04/16 15:58 linux-5.15.y 4fdad925aa1a ec410564 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 INFO: task hung in io_ring_exit_work
* Struck through repros no longer work on HEAD.