syzbot

 sign-in | mailing list | source | docs
🐞 Open [988] ≡ Subsystems 🐞 Fixed [5236] 🐞 Invalid [12507] ⬇ Missing Backports [83] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

possible deadlock in io_submit_one (3)

Status: fixed on 2020/05/10 10:42
Subsystems: fs
[Documentation on labels]
Reported-by: syzbot+343f75cdeea091340956@syzkaller.appspotmail.com
Fix commit: 63f818f46af9 proc: Use a dedicated lock in struct pid
First crash: 1483d, last: 1476d
Cause bisection: introduced by (bisect log) :
commit 7bc3e6e55acf065500a24621f3b313e7e5998acf
Author: Eric W. Biederman <ebiederm@xmission.com>
Date: Thu Feb 20 00:22:26 2020 +0000

  proc: Use a list of inodes to flush from proc

Crash: possible deadlock in io_submit_one (log)
Repro: syz .config
  
▶ ▼ Discussions (3)
Title Replies (including bot) Last reply
[GIT PULL] proc fix for 5.7-rc1 2 (2) 2020/04/10 20:40
[PATCH] proc: Use a dedicated lock in struct pid 2 (2) 2020/04/09 08:35
possible deadlock in io_submit_one (3) 0 (3) 2020/04/07 09:31
▶ ▼ Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 possible deadlock in io_submit_one syz done 406 1664d 1842d 1/1 fixed on 2019/12/17 14:31
upstream possible deadlock in io_submit_one fs C done 1779 1736d 1906d 12/26 fixed on 2019/07/29 13:39
upstream possible deadlock in io_submit_one (2) fuse 353 1674d 1712d 13/26 fixed on 2019/10/15 23:40

Sample crash report:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.6.0-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor.0/8516 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:353 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: aio_poll fs/aio.c:1767 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: io_submit_one+0x10f5/0x1a80 fs/aio.c:1878

and this task is already holding:
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:378 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: aio_poll fs/aio.c:1765 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: io_submit_one+0x10cb/0x1a80 fs/aio.c:1878
which would create a new lock dependency:
 (&ctx->ctx_lock){..-.}-{2:2} -> (&pid->wait_pidfd){+.+.}-{2:2}

but this new dependency connects a SOFTIRQ-irq-safe lock:
 (&ctx->ctx_lock){..-.}-{2:2}

... which became SOFTIRQ-irq-safe at:
  lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
  __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
  _raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
  spin_lock_irq include/linux/spinlock.h:378 [inline]
  free_ioctx_users+0x30/0x1c0 fs/aio.c:618
  percpu_ref_put_many include/linux/percpu-refcount.h:309 [inline]
  percpu_ref_put+0x18d/0x1a0 include/linux/percpu-refcount.h:325
  rcu_do_batch kernel/rcu/tree.c:2206 [inline]
  rcu_core+0x816/0x1120 kernel/rcu/tree.c:2433
  __do_softirq+0x268/0x80c kernel/softirq.c:292
  run_ksoftirqd+0x60/0xf0 kernel/softirq.c:604
  smpboot_thread_fn+0x572/0x970 kernel/smpboot.c:165
  kthread+0x353/0x380 kernel/kthread.c:268
  ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

to a SOFTIRQ-irq-unsafe lock:
 (&pid->wait_pidfd){+.+.}-{2:2}

... which became SOFTIRQ-irq-unsafe at:
...
  lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
  _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
  spin_lock include/linux/spinlock.h:353 [inline]
  proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
  proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
  proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
  proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
  __lookup_slow+0x240/0x370 fs/namei.c:1530
  lookup_slow fs/namei.c:1547 [inline]
  walk_component+0x442/0x680 fs/namei.c:1846
  link_path_walk+0x66d/0xba0 fs/namei.c:2165
  path_openat+0x21d/0x38b0 fs/namei.c:3342
  do_filp_open+0x191/0x3a0 fs/namei.c:3373
  do_sys_openat2+0x463/0x770 fs/open.c:1148
  do_sys_open fs/open.c:1164 [inline]
  ksys_open include/linux/syscalls.h:1386 [inline]
  __do_sys_open fs/open.c:1170 [inline]
  __se_sys_open fs/open.c:1168 [inline]
  __x64_sys_open+0x1af/0x1e0 fs/open.c:1168
  do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
  entry_SYSCALL_64_after_hwframe+0x49/0xb3

other info that might help us debug this:

 Possible interrupt unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&pid->wait_pidfd);
                               local_irq_disable();
                               lock(&ctx->ctx_lock);
                               lock(&pid->wait_pidfd);
  <Interrupt>
    lock(&ctx->ctx_lock);

 *** DEADLOCK ***

1 lock held by syz-executor.0/8516:
 #0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:378 [inline]
 #0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: aio_poll fs/aio.c:1765 [inline]
 #0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
 #0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: io_submit_one+0x10cb/0x1a80 fs/aio.c:1878

the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&ctx->ctx_lock){..-.}-{2:2} {
   IN-SOFTIRQ-W at:
                    lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
                    __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
                    _raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
                    spin_lock_irq include/linux/spinlock.h:378 [inline]
                    free_ioctx_users+0x30/0x1c0 fs/aio.c:618
                    percpu_ref_put_many include/linux/percpu-refcount.h:309 [inline]
                    percpu_ref_put+0x18d/0x1a0 include/linux/percpu-refcount.h:325
                    rcu_do_batch kernel/rcu/tree.c:2206 [inline]
                    rcu_core+0x816/0x1120 kernel/rcu/tree.c:2433
                    __do_softirq+0x268/0x80c kernel/softirq.c:292
                    run_ksoftirqd+0x60/0xf0 kernel/softirq.c:604
                    smpboot_thread_fn+0x572/0x970 kernel/smpboot.c:165
                    kthread+0x353/0x380 kernel/kthread.c:268
                    ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
   INITIAL USE at:
                   lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
                   __raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
                   _raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
                   spin_lock_irq include/linux/spinlock.h:378 [inline]
                   aio_poll fs/aio.c:1765 [inline]
                   __io_submit_one fs/aio.c:1841 [inline]
                   io_submit_one+0x10cb/0x1a80 fs/aio.c:1878
                   __do_sys_io_submit fs/aio.c:1937 [inline]
                   __se_sys_io_submit+0x117/0x220 fs/aio.c:1907
                   do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
                   entry_SYSCALL_64_after_hwframe+0x49/0xb3
 }
 ... key      at: [<ffffffff8b5abc40>] ioctx_alloc.__key+0x0/0x10
 ... acquired at:
   lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
   spin_lock include/linux/spinlock.h:353 [inline]
   aio_poll fs/aio.c:1767 [inline]
   __io_submit_one fs/aio.c:1841 [inline]
   io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
   __do_sys_io_submit fs/aio.c:1937 [inline]
   __se_sys_io_submit+0x117/0x220 fs/aio.c:1907
   do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
   entry_SYSCALL_64_after_hwframe+0x49/0xb3


the dependencies between the lock to be acquired
 and SOFTIRQ-irq-unsafe lock:
-> (&pid->wait_pidfd){+.+.}-{2:2} {
   HARDIRQ-ON-W at:
                    lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
                    spin_lock include/linux/spinlock.h:353 [inline]
                    proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
                    proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
                    proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
                    proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
                    __lookup_slow+0x240/0x370 fs/namei.c:1530
                    lookup_slow fs/namei.c:1547 [inline]
                    walk_component+0x442/0x680 fs/namei.c:1846
                    link_path_walk+0x66d/0xba0 fs/namei.c:2165
                    path_openat+0x21d/0x38b0 fs/namei.c:3342
                    do_filp_open+0x191/0x3a0 fs/namei.c:3373
                    do_sys_openat2+0x463/0x770 fs/open.c:1148
                    do_sys_open fs/open.c:1164 [inline]
                    ksys_open include/linux/syscalls.h:1386 [inline]
                    __do_sys_open fs/open.c:1170 [inline]
                    __se_sys_open fs/open.c:1168 [inline]
                    __x64_sys_open+0x1af/0x1e0 fs/open.c:1168
                    do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
                    entry_SYSCALL_64_after_hwframe+0x49/0xb3
   SOFTIRQ-ON-W at:
                    lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
                    spin_lock include/linux/spinlock.h:353 [inline]
                    proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
                    proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
                    proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
                    proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
                    __lookup_slow+0x240/0x370 fs/namei.c:1530
                    lookup_slow fs/namei.c:1547 [inline]
                    walk_component+0x442/0x680 fs/namei.c:1846
                    link_path_walk+0x66d/0xba0 fs/namei.c:2165
                    path_openat+0x21d/0x38b0 fs/namei.c:3342
                    do_filp_open+0x191/0x3a0 fs/namei.c:3373
                    do_sys_openat2+0x463/0x770 fs/open.c:1148
                    do_sys_open fs/open.c:1164 [inline]
                    ksys_open include/linux/syscalls.h:1386 [inline]
                    __do_sys_open fs/open.c:1170 [inline]
                    __se_sys_open fs/open.c:1168 [inline]
                    __x64_sys_open+0x1af/0x1e0 fs/open.c:1168
                    do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
                    entry_SYSCALL_64_after_hwframe+0x49/0xb3
   INITIAL USE at:
                   lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
                   __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                   _raw_spin_lock_irqsave+0x9e/0xc0 kernel/locking/spinlock.c:159
                   __wake_up_common_lock kernel/sched/wait.c:122 [inline]
                   __wake_up+0xb8/0x150 kernel/sched/wait.c:142
                   do_notify_pidfd kernel/signal.c:1900 [inline]
                   do_notify_parent+0x167/0xce0 kernel/signal.c:1927
                   exit_notify kernel/exit.c:660 [inline]
                   do_exit+0x12c5/0x1f80 kernel/exit.c:816
                   call_usermodehelper_exec_async+0x47c/0x480 kernel/umh.c:125
                   ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
 }
 ... key      at: [<ffffffff8aafb790>] alloc_pid.__key+0x0/0x10
 ... acquired at:
   lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
   spin_lock include/linux/spinlock.h:353 [inline]
   aio_poll fs/aio.c:1767 [inline]
   __io_submit_one fs/aio.c:1841 [inline]
   io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
   __do_sys_io_submit fs/aio.c:1937 [inline]
   __se_sys_io_submit+0x117/0x220 fs/aio.c:1907
   do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
   entry_SYSCALL_64_after_hwframe+0x49/0xb3


stack backtrace:
CPU: 1 PID: 8516 Comm: syz-executor.0 Not tainted 5.6.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1e9/0x30e lib/dump_stack.c:118
 print_bad_irq_dependency kernel/locking/lockdep.c:2132 [inline]
 check_irq_usage kernel/locking/lockdep.c:2330 [inline]
 check_prev_add kernel/locking/lockdep.c:2519 [inline]
 check_prevs_add kernel/locking/lockdep.c:2620 [inline]
 validate_chain+0x8479/0x8920 kernel/locking/lockdep.c:3237
 __lock_acquire+0x116c/0x2b90 kernel/locking/lockdep.c:4344
 lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:353 [inline]
 aio_poll fs/aio.c:1767 [inline]
 __io_submit_one fs/aio.c:1841 [inline]
 io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
 __do_sys_io_submit fs/aio.c:1937 [inline]
 __se_sys_io_submit+0x117/0x220 fs/aio.c:1907
 do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
 entry_SYSCALL_64_after_hwframe+0x49/0xb3
RIP: 0033:0x45c889
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f0f7d9dcc78 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
RAX: ffffffffffffffda RBX: 00007f0f7d9dd6d4 RCX: 000000000045c889
RDX: 0000000020000040 RSI: 0000000020000103 RDI: 00007f0f7d9de000
RBP: 000000000076bfa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000001f9 R14: 00000000004c426b R15: 000000000076bfac

Crashes (45):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/04/08 20:52 upstream f5e94d10e4c4 db9bcd4b .config console log report syz ci-upstream-kasan-gce-smack-root
2020/04/10 11:15 upstream 87ebc45d2d32 a8c6a3f8 .config console log report syz ci-upstream-kasan-gce-386
2020/04/07 01:35 upstream 7e63420847ae 99a96044 .config console log report syz ci-upstream-kasan-gce-386
2020/04/09 20:06 upstream 5d30bcacd91a a8c6a3f8 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/09 18:50 upstream 5d30bcacd91a a8c6a3f8 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/09 06:51 upstream ae46d2aa6a7f a8c6a3f8 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/09 05:55 upstream ae46d2aa6a7f a8c6a3f8 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/08 23:11 upstream ae46d2aa6a7f db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/08 20:34 upstream f5e94d10e4c4 db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/08 11:11 upstream f5e94d10e4c4 db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/08 10:58 upstream f5e94d10e4c4 db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/07 22:13 upstream 763dede1b248 db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/07 21:13 upstream 763dede1b248 db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/07 19:58 upstream 7e63420847ae db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/07 18:53 upstream 7e63420847ae db9bcd4b .config console log report ci-upstream-kasan-gce-smack-root
2020/04/06 20:45 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/06 20:04 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/03 13:11 upstream bef7b2a7be28 5ed396e6 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/03 11:41 upstream bef7b2a7be28 5ed396e6 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/02 22:49 upstream 7be97138e727 a34e2c33 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/02 22:15 upstream 7be97138e727 a34e2c33 .config console log report ci-upstream-kasan-gce-smack-root
2020/04/10 13:31 upstream c0cc271173b2 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/10 12:21 upstream c0cc271173b2 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/10 10:56 upstream 87ebc45d2d32 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/10 10:23 upstream 87ebc45d2d32 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/09 21:18 upstream 87ebc45d2d32 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/09 20:04 upstream 87ebc45d2d32 a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/09 07:10 upstream 5d30bcacd91a a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/09 05:59 upstream 5d30bcacd91a a8c6a3f8 .config console log report ci-upstream-kasan-gce-386
2020/04/09 00:11 upstream ae46d2aa6a7f db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/08 23:36 upstream ae46d2aa6a7f db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/08 11:42 upstream f5e94d10e4c4 db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/08 10:07 upstream f5e94d10e4c4 db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/07 22:21 upstream 763dede1b248 db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/07 19:08 upstream 7e63420847ae db9bcd4b .config console log report ci-upstream-kasan-gce-386
2020/04/07 09:21 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-386
2020/04/06 21:50 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-386
2020/04/06 21:24 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-386
2020/04/06 20:42 upstream 7e63420847ae 99a96044 .config console log report ci-upstream-kasan-gce-386
2020/04/03 12:20 upstream bef7b2a7be28 5ed396e6 .config console log report ci-upstream-kasan-gce-386
2020/04/03 09:47 upstream 7be97138e727 a34e2c33 .config console log report ci-upstream-kasan-gce-386
2020/04/02 22:46 upstream 7be97138e727 a34e2c33 .config console log report ci-upstream-kasan-gce-386
2020/04/02 21:47 upstream 7be97138e727 a34e2c33 .config console log report ci-upstream-kasan-gce-386
* Struck through repros no longer work on HEAD.