syzbot


possible deadlock in io_sq_thread_finish

Status: fixed on 2021/11/10 00:50
Subsystems: fs
[Documentation on labels]
Reported-by: syzbot+ac39856cb1b332dbbdda@syzkaller.appspotmail.com
Fix commit: f4e61f0c9add x86/kvm: Fix broken irq restoration in kvm_wait
First crash: 1147d, last: 1144d
Cause bisection: introduced by (bisect log) [merge commit]:
commit a4dec04c7ff4307973ba502ce7b27330d9fe04b7
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Wed Feb 24 17:54:24 2021 +0000

  Merge tag 'dma-mapping-5.12' of git://git.infradead.org/users/hch/dma-mapping

Crash: WARNING in kvm_wait (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit f4e61f0c9add3b00bd5f2df3c814d688849b8707
Author: Wanpeng Li <wanpengli@tencent.com>
Date: Mon Mar 15 06:55:28 2021 +0000

  x86/kvm: Fix broken irq restoration in kvm_wait

  
Discussions (2)
Title Replies (including bot) Last reply
[syzbot] possible deadlock in io_sq_thread_finish 6 (13) 2021/04/08 19:43
Re: [syzbot] possible deadlock in io_sq_thread_finish 2 (2) 2021/03/10 14:26
Last patch testing requests (4)
Created Duration User Patch Repo Result
2021/03/10 14:37 18m axboe@kernel.dk git://git.kernel.dk/linux-block io_uring-5.12 OK
2021/03/09 23:45 16m axboe@kernel.dk git://git.kernel.dk/linux-block io_uring-5.12 report log
2021/03/09 23:34 10m axboe@kernel.dk git://git.kernel.dk/linux-block io_uring-5.12 report log
2021/03/09 14:57 10m axboe@kernel.dk git://git.kernel.dk/linux-block io_uring-5.12 report log

Sample crash report:
============================================
WARNING: possible recursive locking detected
5.12.0-rc2-syzkaller #0 Not tainted
--------------------------------------------
kworker/u4:7/8696 is trying to acquire lock:
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_stop fs/io_uring.c:7099 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_put_sq_data fs/io_uring.c:7115 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_finish+0x408/0x650 fs/io_uring.c:7139

but task is already holding lock:
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park fs/io_uring.c:7088 [inline]
ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park+0x63/0xc0 fs/io_uring.c:7082

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&sqd->lock);
  lock(&sqd->lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by kworker/u4:7/8696:
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff888010469138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x871/0x1600 kernel/workqueue.c:2246
 #1: ffffc9000253fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x8a5/0x1600 kernel/workqueue.c:2250
 #2: ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park fs/io_uring.c:7088 [inline]
 #2: ffff888015395870 (&sqd->lock){+.+.}-{3:3}, at: io_sq_thread_park+0x63/0xc0 fs/io_uring.c:7082

stack backtrace:
CPU: 0 PID: 8696 Comm: kworker/u4:7 Not tainted 5.12.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound io_ring_exit_work
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 print_deadlock_bug kernel/locking/lockdep.c:2829 [inline]
 check_deadlock kernel/locking/lockdep.c:2872 [inline]
 validate_chain kernel/locking/lockdep.c:3661 [inline]
 __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900
 lock_acquire kernel/locking/lockdep.c:5510 [inline]
 lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5475
 __mutex_lock_common kernel/locking/mutex.c:946 [inline]
 __mutex_lock+0x139/0x1120 kernel/locking/mutex.c:1093
 io_sq_thread_stop fs/io_uring.c:7099 [inline]
 io_put_sq_data fs/io_uring.c:7115 [inline]
 io_sq_thread_finish+0x408/0x650 fs/io_uring.c:7139
 io_ring_ctx_free fs/io_uring.c:8408 [inline]
 io_ring_exit_work+0x82/0x9a0 fs/io_uring.c:8539
 process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
 kthread+0x3b1/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/03/09 14:03 upstream 144c79ef3353 09fbf400 .config console log report syz C ci-upstream-kasan-gce-root possible deadlock in io_sq_thread_finish
2021/03/09 15:58 upstream 144c79ef3353 09fbf400 .config console log report info ci-upstream-kasan-gce-root possible deadlock in io_sq_thread_finish
2021/03/09 13:24 upstream 144c79ef3353 09fbf400 .config console log report info ci-upstream-kasan-gce-root possible deadlock in io_sq_thread_finish
2021/03/07 09:19 upstream a38fd8748464 e4b4d570 .config console log report info ci-upstream-kasan-gce-root possible deadlock in io_sq_thread_finish
* Struck through repros no longer work on HEAD.