syzbot
possible deadlock in io_submit_one (3)
Status:
fixed on 2020/05/10 10:42
Subsystems:
fs
[Documentation on labels]
Reported-by: syzbot+343f75cdeea091340956@syzkaller.appspotmail.com
Fix commit:
63f818f46af9
proc: Use a dedicated lock in struct pid
First crash: 2303d, last: 2296d
Sample crash report:
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.6.0-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor.0/8516 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:353 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: aio_poll fs/aio.c:1767 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
ffff8880939b8148 (&pid->wait_pidfd){+.+.}-{2:2}, at: io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
and this task is already holding:
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:378 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: aio_poll fs/aio.c:1765 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: io_submit_one+0x10cb/0x1a80 fs/aio.c:1878
which would create a new lock dependency:
(&ctx->ctx_lock){..-.}-{2:2} -> (&pid->wait_pidfd){+.+.}-{2:2}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&ctx->ctx_lock){..-.}-{2:2}
... which became SOFTIRQ-irq-safe at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
_raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
spin_lock_irq include/linux/spinlock.h:378 [inline]
free_ioctx_users+0x30/0x1c0 fs/aio.c:618
percpu_ref_put_many include/linux/percpu-refcount.h:309 [inline]
percpu_ref_put+0x18d/0x1a0 include/linux/percpu-refcount.h:325
rcu_do_batch kernel/rcu/tree.c:2206 [inline]
rcu_core+0x816/0x1120 kernel/rcu/tree.c:2433
__do_softirq+0x268/0x80c kernel/softirq.c:292
run_ksoftirqd+0x60/0xf0 kernel/softirq.c:604
smpboot_thread_fn+0x572/0x970 kernel/smpboot.c:165
kthread+0x353/0x380 kernel/kthread.c:268
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
to a SOFTIRQ-irq-unsafe lock:
(&pid->wait_pidfd){+.+.}-{2:2}
... which became SOFTIRQ-irq-unsafe at:
...
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
__lookup_slow+0x240/0x370 fs/namei.c:1530
lookup_slow fs/namei.c:1547 [inline]
walk_component+0x442/0x680 fs/namei.c:1846
link_path_walk+0x66d/0xba0 fs/namei.c:2165
path_openat+0x21d/0x38b0 fs/namei.c:3342
do_filp_open+0x191/0x3a0 fs/namei.c:3373
do_sys_openat2+0x463/0x770 fs/open.c:1148
do_sys_open fs/open.c:1164 [inline]
ksys_open include/linux/syscalls.h:1386 [inline]
__do_sys_open fs/open.c:1170 [inline]
__se_sys_open fs/open.c:1168 [inline]
__x64_sys_open+0x1af/0x1e0 fs/open.c:1168
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&pid->wait_pidfd);
local_irq_disable();
lock(&ctx->ctx_lock);
lock(&pid->wait_pidfd);
<Interrupt>
lock(&ctx->ctx_lock);
*** DEADLOCK ***
1 lock held by syz-executor.0/8516:
#0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:378 [inline]
#0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: aio_poll fs/aio.c:1765 [inline]
#0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: __io_submit_one fs/aio.c:1841 [inline]
#0: ffff888090b38658 (&ctx->ctx_lock){..-.}-{2:2}, at: io_submit_one+0x10cb/0x1a80 fs/aio.c:1878
the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&ctx->ctx_lock){..-.}-{2:2} {
IN-SOFTIRQ-W at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
_raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
spin_lock_irq include/linux/spinlock.h:378 [inline]
free_ioctx_users+0x30/0x1c0 fs/aio.c:618
percpu_ref_put_many include/linux/percpu-refcount.h:309 [inline]
percpu_ref_put+0x18d/0x1a0 include/linux/percpu-refcount.h:325
rcu_do_batch kernel/rcu/tree.c:2206 [inline]
rcu_core+0x816/0x1120 kernel/rcu/tree.c:2433
__do_softirq+0x268/0x80c kernel/softirq.c:292
run_ksoftirqd+0x60/0xf0 kernel/softirq.c:604
smpboot_thread_fn+0x572/0x970 kernel/smpboot.c:165
kthread+0x353/0x380 kernel/kthread.c:268
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
INITIAL USE at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock_irq include/linux/spinlock_api_smp.h:128 [inline]
_raw_spin_lock_irq+0x67/0x80 kernel/locking/spinlock.c:167
spin_lock_irq include/linux/spinlock.h:378 [inline]
aio_poll fs/aio.c:1765 [inline]
__io_submit_one fs/aio.c:1841 [inline]
io_submit_one+0x10cb/0x1a80 fs/aio.c:1878
__do_sys_io_submit fs/aio.c:1937 [inline]
__se_sys_io_submit+0x117/0x220 fs/aio.c:1907
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
}
... key at: [<ffffffff8b5abc40>] ioctx_alloc.__key+0x0/0x10
... acquired at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
aio_poll fs/aio.c:1767 [inline]
__io_submit_one fs/aio.c:1841 [inline]
io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
__do_sys_io_submit fs/aio.c:1937 [inline]
__se_sys_io_submit+0x117/0x220 fs/aio.c:1907
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
the dependencies between the lock to be acquired
and SOFTIRQ-irq-unsafe lock:
-> (&pid->wait_pidfd){+.+.}-{2:2} {
HARDIRQ-ON-W at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
__lookup_slow+0x240/0x370 fs/namei.c:1530
lookup_slow fs/namei.c:1547 [inline]
walk_component+0x442/0x680 fs/namei.c:1846
link_path_walk+0x66d/0xba0 fs/namei.c:2165
path_openat+0x21d/0x38b0 fs/namei.c:3342
do_filp_open+0x191/0x3a0 fs/namei.c:3373
do_sys_openat2+0x463/0x770 fs/open.c:1148
do_sys_open fs/open.c:1164 [inline]
ksys_open include/linux/syscalls.h:1386 [inline]
__do_sys_open fs/open.c:1170 [inline]
__se_sys_open fs/open.c:1168 [inline]
__x64_sys_open+0x1af/0x1e0 fs/open.c:1168
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
SOFTIRQ-ON-W at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
proc_pid_make_inode+0x187/0x2d0 fs/proc/base.c:1880
proc_pid_instantiate+0x4b/0x1a0 fs/proc/base.c:3285
proc_pid_lookup+0x218/0x2f0 fs/proc/base.c:3320
proc_root_lookup+0x1b/0x50 fs/proc/root.c:243
__lookup_slow+0x240/0x370 fs/namei.c:1530
lookup_slow fs/namei.c:1547 [inline]
walk_component+0x442/0x680 fs/namei.c:1846
link_path_walk+0x66d/0xba0 fs/namei.c:2165
path_openat+0x21d/0x38b0 fs/namei.c:3342
do_filp_open+0x191/0x3a0 fs/namei.c:3373
do_sys_openat2+0x463/0x770 fs/open.c:1148
do_sys_open fs/open.c:1164 [inline]
ksys_open include/linux/syscalls.h:1386 [inline]
__do_sys_open fs/open.c:1170 [inline]
__se_sys_open fs/open.c:1168 [inline]
__x64_sys_open+0x1af/0x1e0 fs/open.c:1168
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
INITIAL USE at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x9e/0xc0 kernel/locking/spinlock.c:159
__wake_up_common_lock kernel/sched/wait.c:122 [inline]
__wake_up+0xb8/0x150 kernel/sched/wait.c:142
do_notify_pidfd kernel/signal.c:1900 [inline]
do_notify_parent+0x167/0xce0 kernel/signal.c:1927
exit_notify kernel/exit.c:660 [inline]
do_exit+0x12c5/0x1f80 kernel/exit.c:816
call_usermodehelper_exec_async+0x47c/0x480 kernel/umh.c:125
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
}
... key at: [<ffffffff8aafb790>] alloc_pid.__key+0x0/0x10
... acquired at:
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
aio_poll fs/aio.c:1767 [inline]
__io_submit_one fs/aio.c:1841 [inline]
io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
__do_sys_io_submit fs/aio.c:1937 [inline]
__se_sys_io_submit+0x117/0x220 fs/aio.c:1907
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
stack backtrace:
CPU: 1 PID: 8516 Comm: syz-executor.0 Not tainted 5.6.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1e9/0x30e lib/dump_stack.c:118
print_bad_irq_dependency kernel/locking/lockdep.c:2132 [inline]
check_irq_usage kernel/locking/lockdep.c:2330 [inline]
check_prev_add kernel/locking/lockdep.c:2519 [inline]
check_prevs_add kernel/locking/lockdep.c:2620 [inline]
validate_chain+0x8479/0x8920 kernel/locking/lockdep.c:3237
__lock_acquire+0x116c/0x2b90 kernel/locking/lockdep.c:4344
lock_acquire+0x169/0x480 kernel/locking/lockdep.c:4923
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:353 [inline]
aio_poll fs/aio.c:1767 [inline]
__io_submit_one fs/aio.c:1841 [inline]
io_submit_one+0x10f5/0x1a80 fs/aio.c:1878
__do_sys_io_submit fs/aio.c:1937 [inline]
__se_sys_io_submit+0x117/0x220 fs/aio.c:1907
do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:295
entry_SYSCALL_64_after_hwframe+0x49/0xb3
RIP: 0033:0x45c889
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f0f7d9dcc78 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
RAX: ffffffffffffffda RBX: 00007f0f7d9dd6d4 RCX: 000000000045c889
RDX: 0000000020000040 RSI: 0000000020000103 RDI: 00007f0f7d9de000
RBP: 000000000076bfa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000001f9 R14: 00000000004c426b R15: 000000000076bfac
Struck through repros no longer work on HEAD.