possible deadlock in io_sq_thread

possible deadlock in io_sq_thread_finish
Status: upstream: reported C repro on 2021/03/07 09:49
Reported-by: syzbot+ac39856cb1b332dbbdda@syzkaller.appspotmail.com
Fix commit: f4e61f0c9add x86/kvm: Fix broken irq restoration in kvm_wait
Patched on: [ci-qemu-upstream ci-qemu-upstream-386 ci-qemu2-arm64 ci-qemu2-arm64-compat ci-qemu2-arm64-mte ci-qemu2-riscv64 ci-upstream-bpf-kasan-gce ci-upstream-bpf-next-kasan-gce ci-upstream-gce-leak ci-upstream-kasan-gce ci-upstream-kasan-gce-386 ci-upstream-kasan-gce-root ci-upstream-kasan-gce-selinux-root ci-upstream-kasan-gce-smack-root ci-upstream-kmsan-gce ci-upstream-kmsan-gce-386 ci-upstream-linux-next-kasan-gce-root ci-upstream-net-kasan-gce ci-upstream-net-this-kasan-gce ci2-upstream-kcsan-gce ci2-upstream-usb], missing on: [ci-qemu2-arm32]
First crash: 139d, last: 136d

Cause bisection: introduced by (bisect log) [merge commit]:
commit a4dec04c7ff4307973ba502ce7b27330d9fe04b7
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Wed Feb 24 17:54:24 2021 +0000

Merge tag 'dma-mapping-5.12' of git://git.infradead.org/users/hch/dma-mapping

Crash: WARNING in kvm_wait (log)
Repro: C syz .config

Fix bisection: fixed by (bisect log) :
commit f4e61f0c9add3b00bd5f2df3c814d688849b8707
Author: Wanpeng Li <wanpengli@tencent.com>
Date: Mon Mar 15 06:55:28 2021 +0000

x86/kvm: Fix broken irq restoration in kvm_wait

Patch testing requests:
Created	Duration	User	Repo	Result
2021/03/10 14:37	18m	axboe@kernel.dk	git://git.kernel.dk/linux-block io_uring-5.12	OK
2021/03/09 23:45	16m	axboe@kernel.dk	git://git.kernel.dk/linux-block io_uring-5.12	report log
2021/03/09 23:34	10m	axboe@kernel.dk	git://git.kernel.dk/linux-block io_uring-5.12	report log
2021/03/09 14:57	10m	axboe@kernel.dk	git://git.kernel.dk/linux-block io_uring-5.12	report log

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.12.0-rc2-syzkaller #0 Not tainted
--------------------------------------------
kworker/u4:7/8696 is trying to acquire lock:
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_stop fs/io_uring.c:7099 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_put_sq_data fs/io_uring.c:7115 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_finish+0x408/0x650 fs/io_uring.c:7139

but task is already holding lock:
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park fs/io_uring.c:7088 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park+0x63/0xc0 fs/io_uring.c:7082

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&sqd->lock);
  lock(&sqd->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by kworker/u4:7/8696:
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc9000253fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
 #2: ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park fs/io_uring.c:7088 [inline]
 #2: ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park+0x63/0xc0 fs/io_uring.c:7082

stack backtrace:
CPU: 0 PID: 8696 Comm: kworker/u4:7 Not tainted 5.12.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound io_ring_exit_work
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 print_deadlock_bug kernel/locking/lockdep.c:2829 [inline]
 check_deadlock kernel/locking/lockdep.c:2872 [inline]
 validate_chain kernel/locking/lockdep.c:3661 [inline]
 __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900
 lock_acquire kernel/locking/lockdep.c:5510 [inline]
 lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475
 __mutex_lock_common kernel/locking/mutex.c:946 [inline]
 __mutex_lock+0x139/0x1120 kernel/locking/mutex.c:1093
 io_sq_thread_stop fs/io_uring.c:7099 [inline]
 io_put_sq_data fs/io_uring.c:7115 [inline]
 io_sq_thread_finish+0x408/0x650 fs/io_uring.c:7139
 io_ring_ctx_free fs/io_uring.c:8408 [inline]
 io_ring_exit_work+0x82/0x9a0 fs/io_uring.c:8539
 process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
 kthread+0x3b1/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Crashes (4):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info	Title
ci-upstream-kasan-gce-root	2021/03/09 14:03	upstream	144c79ef3353	09fbf400	.config	log	report	syz	C		possible deadlock in io_sq_thread_finish
ci-upstream-kasan-gce-root	2021/03/09 15:58	upstream	144c79ef3353	09fbf400	.config	log	report			info	possible deadlock in io_sq_thread_finish
ci-upstream-kasan-gce-root	2021/03/09 13:24	upstream	144c79ef3353	09fbf400	.config	log	report			info	possible deadlock in io_sq_thread_finish
ci-upstream-kasan-gce-root	2021/03/07 09:19	upstream	a38fd8748464	e4b4d570	.config	log	report			info	possible deadlock in io_sq_thread_finish