INFO: task hung in io_ring_exit

INFO: task hung in io_ring_exit_work
Status: upstream: reported C repro on 2021/04/05 09:16
Reported-by: syzbot+93f72b3885406bb09e0d@syzkaller.appspotmail.com
First crash: 141d, last: 3h33m

Cause bisection: introduced by (bisect log) [no-op commit]:
commit 7e923e6a3ceb877497dd9ee70d71fa33b94f332b
Author: Peter Zijlstra <peterz@infradead.org>
Date: Wed Dec 9 15:06:06 2020 +0000

locking/selftests: Add local_lock inversion tests

Crash: WARNING: ODEBUG bug in netdev_run_todo (log)
Repro: C syz .config

Patch testing requests:
Created	Duration	User	Repo	Result
2021/07/19 13:55	17m	mail@anirudhrb.com	upstream	OK
2021/06/09 07:49	18m	mail@anirudhrb.com	linux-next	error
2021/05/13 21:15	13m	asml.silence@gmail.com	https://github.com/isilence/linux.git syz_test7	report log
2021/05/07 19:53	15m	asml.silence@gmail.com	git://git.kernel.dk/linux-block io_uring-5.13	OK
2021/04/08 14:02	17m	asml.silence@gmail.com	https://github.com/isilence/linux.git syz_test3	report log
2021/04/08 13:39	17m	asml.silence@gmail.com	https://github.com/isilence/linux.git syz_test2	report log
2021/04/08 00:42	17m	asml.silence@gmail.com	https://github.com/isilence/linux.git syz_test2	report log
2021/04/07 19:55	17m	asml.silence@gmail.com	https://github.com/isilence/linux.git syz_test	report log
2021/04/05 19:15	17m	asml.silence@gmail.com	https://github.com/isilence/linux.git iowq_cancel_unbounded	report log

Sample crash report:

INFO: task kworker/u4:6:3091 blocked for more than 143 seconds.
      Not tainted 5.12.0-rc6-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:6    state:D stack:24792 pid: 3091 ppid:     2 flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
 context_switch kernel/sched/core.c:4322 [inline]
 __schedule+0x911/0x21b0 kernel/sched/core.c:5073
 schedule+0xcf/0x270 kernel/sched/core.c:5152
 schedule_timeout+0x1db/0x250 kernel/time/timer.c:1868
 do_wait_for_common kernel/sched/completion.c:85 [inline]
 __wait_for_common kernel/sched/completion.c:106 [inline]
 wait_for_common kernel/sched/completion.c:117 [inline]
 wait_for_completion+0x168/0x270 kernel/sched/completion.c:138
 io_ring_exit_work+0x4e8/0x12d0 fs/io_uring.c:8596
 process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
 kthread+0x3b1/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Showing all locks held in the system:
2 locks held by kworker/u4:5/235:
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc900019bfda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
1 lock held by khungtaskd/1630:
 #0: ffffffff8bf74320 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x53/0x260 kernel/locking/lockdep.c:6327
2 locks held by kworker/u4:6/3091:
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc90001cbfda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
1 lock held by in:imklog/8101:
 #0: ffff88801523b270 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:961
2 locks held by kworker/u4:1/11499:
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc9000d957da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
2 locks held by syz-executor633/15066:
 #0: ffff8880b9d35198 (&rq->lock){-.-.}-{2:2}, at: rq_lock kernel/sched/sched.h:1321 [inline]
 #0: ffff8880b9d35198 (&rq->lock){-.-.}-{2:2}, at: __schedule+0x21c/0x21b0 kernel/sched/core.c:4990
 #1: ffff8880b9d1f948 (&per_cpu_ptr(group->pcpu, cpu)->seq){-.-.}-{0:0}, at: psi_task_switch+0x305/0x440 kernel/sched/psi.c:833
1 lock held by syz-executor633/15068:
 #0: ffffffff8bf7cee8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:290 [inline]
 #0: ffffffff8bf7cee8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x4fa/0x620 kernel/rcu/tree_exp.h:836

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 1630 Comm: khungtaskd Not tainted 5.12.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x44/0xd7 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1b3/0x230 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:209 [inline]
 watchdog+0xd48/0xfb0 kernel/hung_task.c:294
 kthread+0x3b1/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 8393 Comm: syz-executor633 Not tainted 5.12.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:wait_consider_task+0x99/0x3b40 kernel/exit.c:1288
Code: 28 00 00 00 48 89 84 24 f0 00 00 00 31 c0 e8 ae 8e 2f 00 49 8d 8f ec 04 00 00 48 89 c8 48 89 4c 24 10 48 c1 e8 03 0f b6 14 28 <48> 89 c8 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 4b 1d 00 00 45
RSP: 0018:ffffc9000146fb20 EFLAGS: 00000a07
RAX: 1ffff11015477b35 RBX: 0000000000000000 RCX: ffff8880aa3bd9ac
RDX: 0000000000000000 RSI: ffffffff814462a2 RDI: ffffc9000146fd20
RBP: dffffc0000000000 R08: 0000000000000000 R09: ffffffff8bc0a083
R10: ffffffff8144a0e2 R11: 0000000000000001 R12: ffffc9000146fd20
R13: ffff888020ff9c40 R14: 0000000000000000 R15: ffff8880aa3bd4c0
FS:  0000000001688300(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f42f9b4c000 CR3: 0000000025681000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 do_wait_thread kernel/exit.c:1397 [inline]
 do_wait+0x376/0xa00 kernel/exit.c:1468
 kernel_wait4+0x14c/0x260 kernel/exit.c:1630
 __do_sys_wait4+0x13f/0x150 kernel/exit.c:1658
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x444d06
Code: 0f 1f 40 00 31 c9 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 49 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 11 b8 3d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5a c3 90 48 83 ec 28 89 54 24 14 48 89 74 24
RSP: 002b:00007ffe758f7cb8 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
RAX: ffffffffffffffda RBX: 00000000000d35a7 RCX: 0000000000444d06
RDX: 0000000040000001 RSI: 00007ffe758f7ce4 RDI: 00000000ffffffff
RBP: 00000000000019f2 R08: 0000000000000000 R09: 00007ffe75923090
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe758f7ce4
R13: 00007ffe758f7d40 R14: 000000000000028f R15: 00007ffe758f7d20

Crashes (161):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce	2021/04/05 09:16	upstream	e49d033bddf5	6a81331a	.config	log	report	syz	C		INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/08/05 15:00	upstream	251a1524293d	7f7bb950	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/08/05 13:32	upstream	251a1524293d	7f7bb950	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/08/05 03:25	upstream	251a1524293d	b97d64c9	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-selinux-root	2021/08/03 03:16	upstream	c500bee1c5b2	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/08/02 16:49	upstream	c500bee1c5b2	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-smack-root	2021/08/01 13:34	upstream	f3438b4c4e69	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-smack-root	2021/08/01 11:26	upstream	f3438b4c4e69	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/30 23:17	upstream	764a5bc89b12	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/28 17:42	upstream	7d549995d4e0	17d6ab15	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/28 16:16	upstream	7d549995d4e0	17d6ab15	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/28 12:26	upstream	7d549995d4e0	17d6ab15	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/28 00:53	upstream	7d549995d4e0	17d6ab15	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/27 19:07	upstream	ff1176468d36	fd511809	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/27 10:01	upstream	ff1176468d36	fd511809	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-smack-root	2021/07/27 06:29	upstream	ff1176468d36	fd511809	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-smack-root	2021/07/26 16:04	upstream	ff1176468d36	fd511809	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/26 00:35	upstream	d8079fac1681	4d1b57d4	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/22 20:40	upstream	3d5895cd3517	302e51de	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/22 08:27	upstream	7b6ae471e541	29c3f20f	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/22 04:34	upstream	7b6ae471e541	29c3f20f	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/21 03:16	upstream	8cae8cd89f05	1b201b48	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-selinux-root	2021/07/20 05:14	upstream	2734d6c1b1a0	bc48c9ab	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-smack-root	2021/07/18 23:19	upstream	1d67c8d993ba	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-selinux-root	2021/07/18 22:17	upstream	1d67c8d993ba	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/18 12:33	upstream	ccbb22b9ab86	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/18 00:40	upstream	d980cc0620ae	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/17 23:21	upstream	d980cc0620ae	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/17 21:04	upstream	d980cc0620ae	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/17 14:39	upstream	d980cc0620ae	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/16 09:50	upstream	dd9c7df94c1b	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/16 03:52	upstream	dd9c7df94c1b	f115ae98	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/15 08:11	upstream	8096acd7442e	b9a2f64e	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/15 01:40	upstream	8096acd7442e	94e0b707	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/14 14:42	upstream	40226a3d96ef	484502bd	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/14 02:54	upstream	40226a3d96ef	fa0594c3	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-root	2021/07/13 14:28	upstream	7fef2edf7cc7	f415556d	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/13 10:01	upstream	7fef2edf7cc7	f415556d	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/12 13:59	upstream	e73f0f0ee754	a4869c92	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/12 08:47	upstream	e73f0f0ee754	a4869c92	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/11 05:46	upstream	3dbdb38e2869	8f5a7b8c	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/10 05:47	upstream	3dbdb38e2869	8f5a7b8c	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-selinux-root	2021/07/09 21:14	upstream	3dbdb38e2869	281e815f	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/09 06:53	upstream	3dbdb38e2869	1b20171a	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/07/08 10:38	upstream	3dbdb38e2869	95793bce	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce	2021/03/17 14:43	upstream	1df27313f50a	fdb2bb2c	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-kasan-gce-386	2021/07/25 16:14	upstream	7ffca2bb9d8b	4d1b57d4	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-linux-next-kasan-gce-root	2021/08/02 13:28	linux-next	8d4b477da1a8	6c236867	.config	log	report			info	INFO: task hung in io_ring_exit_work
ci-upstream-linux-next-kasan-gce-root	2021/07/23 14:37	linux-next	90d856e71443	bc5f1d88	.config	log	report			info	INFO: task hung in io_ring_exit_work