syzbot

 sign-in | mailing list | source | docs
🐞 Open [988] ≡ Subsystems 🐞 Fixed [5242] 🐞 Invalid [12509] ⬇ Missing Backports [83] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

possible deadlock in do_io_accounting (3)

Status: fixed on 2020/08/18 22:40
Subsystems: fs
[Documentation on labels]
Reported-by: syzbot+87a1b40b8fcdc9d40bd0@syzkaller.appspotmail.com
Fix commit: 76518d379885 proc: io_accounting: Use new infrastructure to fix deadlocks in execve
First crash: 1584d, last: 1492d
Cause bisection: the cause commit could be any of (bisect log):
  9211bfbff80a netfilter: add missing IS_ENABLED(CONFIG_BRIDGE_NETFILTER) checks to header-file.
  47e640af2e49 netfilter: add missing IS_ENABLED(CONFIG_NF_TABLES) check to header-file.
  a1b2f04ea527 netfilter: add missing includes to a number of header-files.
  0abc8bf4f284 netfilter: add missing IS_ENABLED(CONFIG_NF_CONNTRACK) checks to some header-files.
  bd96b4c75675 netfilter: inline four headers files into another one.
  43dd16efc7f2 netfilter: nf_tables: store data in offload context registers
  78458e3e08cd netfilter: add missing IS_ENABLED(CONFIG_NETFILTER) checks to some header-files.
  20a9379d9a03 netfilter: remove "#ifdef __KERNEL__" guards from some headers.
  bd8699e9e292 netfilter: nft_bitwise: add offload support
  2a475c409fe8 kbuild: remove all netfilter headers from header-test blacklist.
  7e59b3fea2a2 netfilter: remove unnecessary spaces
  1b90af292e71 ipvs: Improve robustness to the ipvs sysctl
  5785cf15fd74 netfilter: nf_tables: add missing prototypes.
  0a30ba509fde netfilter: nf_nat_proto: make tables static
  e84fb4b3666d netfilter: conntrack: use shared sysctl constants
  105333435b4f netfilter: connlabels: prefer static lock initialiser
  8c0bb7873815 netfilter: synproxy: rename mss synproxy_options field
  c162610c7db2 Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next
   
Fix bisection: fixed by (bisect log) :
commit 76518d3798855242817e8a8ed76b2d72f4415624
Author: Bernd Edlinger <bernd.edlinger@hotmail.de>
Date: Fri Mar 20 20:27:41 2020 +0000

  proc: io_accounting: Use new infrastructure to fix deadlocks in execve

  
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
possible deadlock in do_io_accounting (3) 0 (4) 2020/04/25 14:32
▶ ▼ Similar bugs (8)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-49 possible deadlock in do_io_accounting C 4521 1605d 1841d 0/3 public: reported C repro on 2019/04/12 00:00
android-44 possible deadlock in do_io_accounting C 28 1607d 1842d 0/2 public: reported C repro on 2019/04/11 08:44
linux-4.19 possible deadlock in do_io_accounting C done 4 1669d 1781d 1/1 fixed on 2019/12/10 20:49
linux-4.19 possible deadlock in do_io_accounting (2) syz error 26 1322d 1585d 0/1 upstream: reported syz repro on 2019/12/24 07:12
upstream possible deadlock in do_io_accounting fs syz 1003 1856d 2334d 0/26 closed as dup on 2017/12/12 21:27
upstream possible deadlock in do_io_accounting (2) fs 1 1830d 1827d 0/26 auto-closed as invalid on 2019/10/20 09:03
android-414 possible deadlock in do_io_accounting syz 70 1858d 1841d 0/1 public: reported syz repro on 2019/04/12 00:01
linux-4.14 possible deadlock in do_io_accounting C error 56 819d 1768d 0/1 upstream: reported C repro on 2019/06/24 03:04

Sample crash report:
overlayfs: failed to resolve './file0': -2
======================================================
WARNING: possible circular locking dependency detected
5.5.0-rc3-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor763/9723 is trying to acquire lock:
ffff8880a95dfed0 (&sig->cred_guard_mutex){+.+.}, at: do_io_accounting+0x1f4/0x820 fs/proc/base.c:2773

but task is already holding lock:
ffff8880a24999a0 (&p->lock){+.+.}, at: seq_read+0x71/0x1170 fs/seq_file.c:161

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&p->lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:956 [inline]
       __mutex_lock+0x156/0x13c0 kernel/locking/mutex.c:1103
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1118
       seq_read+0x71/0x1170 fs/seq_file.c:161
       do_loop_readv_writev fs/read_write.c:714 [inline]
       do_loop_readv_writev fs/read_write.c:701 [inline]
       do_iter_read+0x4a4/0x660 fs/read_write.c:935
       vfs_readv+0xf0/0x160 fs/read_write.c:997
       kernel_readv fs/splice.c:365 [inline]
       default_file_splice_read+0x4fb/0xa20 fs/splice.c:422
       do_splice_to+0x127/0x180 fs/splice.c:892
       splice_direct_to_actor+0x320/0xa30 fs/splice.c:971
       do_splice_direct+0x1da/0x2a0 fs/splice.c:1080
       do_sendfile+0x597/0xd00 fs/read_write.c:1464
       __do_sys_sendfile64 fs/read_write.c:1525 [inline]
       __se_sys_sendfile64 fs/read_write.c:1511 [inline]
       __x64_sys_sendfile64+0x1dd/0x220 fs/read_write.c:1511
       do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (sb_writers#3){.+.+}:
       percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
       __sb_start_write+0x241/0x460 fs/super.c:1674
       sb_start_write include/linux/fs.h:1650 [inline]
       mnt_want_write+0x3f/0xc0 fs/namespace.c:354
       ovl_want_write+0x76/0xa0 fs/overlayfs/util.c:21
       ovl_create_object+0xb3/0x2c0 fs/overlayfs/dir.c:596
       ovl_create+0x28/0x30 fs/overlayfs/dir.c:627
       lookup_open+0x12d5/0x1a90 fs/namei.c:3241
       do_last fs/namei.c:3331 [inline]
       path_openat+0x14a2/0x4500 fs/namei.c:3537
       do_filp_open+0x1a1/0x280 fs/namei.c:3567
       do_sys_open+0x3fe/0x5d0 fs/open.c:1097
       __do_sys_open fs/open.c:1115 [inline]
       __se_sys_open fs/open.c:1110 [inline]
       __x64_sys_open+0x7e/0xc0 fs/open.c:1110
       do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&ovl_i_mutex_dir_key[depth]){++++}:
       down_read+0x95/0x430 kernel/locking/rwsem.c:1495
       inode_lock_shared include/linux/fs.h:801 [inline]
       do_last fs/namei.c:3330 [inline]
       path_openat+0x1e37/0x4500 fs/namei.c:3537
       do_filp_open+0x1a1/0x280 fs/namei.c:3567
       do_open_execat+0x137/0x690 fs/exec.c:856
       __do_execve_file.isra.0+0x1702/0x22b0 fs/exec.c:1761
       do_execveat_common fs/exec.c:1867 [inline]
       do_execve fs/exec.c:1884 [inline]
       __do_sys_execve fs/exec.c:1960 [inline]
       __se_sys_execve fs/exec.c:1955 [inline]
       __x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
       do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&sig->cred_guard_mutex){+.+.}:
       check_prev_add kernel/locking/lockdep.c:2476 [inline]
       check_prevs_add kernel/locking/lockdep.c:2581 [inline]
       validate_chain kernel/locking/lockdep.c:2971 [inline]
       __lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
       lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4485
       __mutex_lock_common kernel/locking/mutex.c:956 [inline]
       __mutex_lock+0x156/0x13c0 kernel/locking/mutex.c:1103
       mutex_lock_killable_nested+0x16/0x20 kernel/locking/mutex.c:1133
       do_io_accounting+0x1f4/0x820 fs/proc/base.c:2773
       proc_tgid_io_accounting+0x23/0x30 fs/proc/base.c:2822
       proc_single_show+0xfd/0x1c0 fs/proc/base.c:756
       seq_read+0x4ca/0x1170 fs/seq_file.c:229
       do_loop_readv_writev fs/read_write.c:714 [inline]
       do_loop_readv_writev fs/read_write.c:701 [inline]
       do_iter_read+0x4a4/0x660 fs/read_write.c:935
       vfs_readv+0xf0/0x160 fs/read_write.c:997
       kernel_readv fs/splice.c:365 [inline]
       default_file_splice_read+0x4fb/0xa20 fs/splice.c:422
       do_splice_to+0x127/0x180 fs/splice.c:892
       splice_direct_to_actor+0x320/0xa30 fs/splice.c:971
       do_splice_direct+0x1da/0x2a0 fs/splice.c:1080
       do_sendfile+0x597/0xd00 fs/read_write.c:1464
       __do_sys_sendfile64 fs/read_write.c:1525 [inline]
       __se_sys_sendfile64 fs/read_write.c:1511 [inline]
       __x64_sys_sendfile64+0x1dd/0x220 fs/read_write.c:1511
       do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
  &sig->cred_guard_mutex --> sb_writers#3 --> &p->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&p->lock);
                               lock(sb_writers#3);
                               lock(&p->lock);
  lock(&sig->cred_guard_mutex);

 *** DEADLOCK ***

2 locks held by syz-executor763/9723:
 #0: ffff888098910428 (sb_writers#9){.+.+}, at: file_start_write include/linux/fs.h:2885 [inline]
 #0: ffff888098910428 (sb_writers#9){.+.+}, at: do_sendfile+0x9b9/0xd00 fs/read_write.c:1463
 #1: ffff8880a24999a0 (&p->lock){+.+.}, at: seq_read+0x71/0x1170 fs/seq_file.c:161

stack backtrace:
CPU: 0 PID: 9723 Comm: syz-executor763 Not tainted 5.5.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 print_circular_bug.isra.0.cold+0x163/0x172 kernel/locking/lockdep.c:1685
 check_noncircular+0x32e/0x3e0 kernel/locking/lockdep.c:1809
 check_prev_add kernel/locking/lockdep.c:2476 [inline]
 check_prevs_add kernel/locking/lockdep.c:2581 [inline]
 validate_chain kernel/locking/lockdep.c:2971 [inline]
 __lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
 lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4485
 __mutex_lock_common kernel/locking/mutex.c:956 [inline]
 __mutex_lock+0x156/0x13c0 kernel/locking/mutex.c:1103
 mutex_lock_killable_nested+0x16/0x20 kernel/locking/mutex.c:1133
 do_io_accounting+0x1f4/0x820 fs/proc/base.c:2773
 proc_tgid_io_accounting+0x23/0x30 fs/proc/base.c:2822
 proc_single_show+0xfd/0x1c0 fs/proc/base.c:756
 seq_read+0x4ca/0x1170 fs/seq_file.c:229
 do_loop_readv_writev fs/read_write.c:714 [inline]
 do_loop_readv_writev fs/read_write.c:701 [inline]
 do_iter_read+0x4a4/0x660 fs/read_write.c:935
 vfs_readv+0xf0/0x160 fs/read_write.c:997
 kernel_readv fs/splice.c:365 [inline]
 default_file_splice_read+0x4fb/0xa20 fs/splice.c:422
 do_splice_to+0x127/0x180 fs/splice.c:892
 splice_direct_to_actor+0x320/0xa30 fs/splice.c:971
 do_splice_direct+0x1da/0x2a0 fs/splice.c:1080
 do_sendfile+0x597/0xd00 fs/read_write.c:1464
 __do_sys_sendfile64 fs/read_write.c:1525 [inline]
 __se_sys_sendfile64 fs/read_write.c:1511 [inline]
 __x64_sys_sendfile64+0x1dd/0x220 fs/read_write.c:1511
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4496d9
Code: e8 9c e6 ff ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 3b 05 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f91dddd5db8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 00000000006e5a18 RCX: 00000000004496d9
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000005
RBP: 00000000006e5a10 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000209 R11: 0000000000000246 R12: 00000000006e5a1c
R13: 00007ffc82ca42cf R14: 00007f91dddd69c0 R15: 20c49ba5e353f7cf

Crashes (39):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/12/27 11:54 upstream 46cf053efec6 be5c2c81 .config console log report syz C ci-upstream-kasan-gce-root
2019/12/27 03:03 upstream 46cf053efec6 be5c2c81 .config console log report syz ci-upstream-kasan-gce-selinux-root
2020/03/26 04:11 upstream 1b649e0bcae7 e8e6c7d2 .config console log report ci-upstream-kasan-gce-root
2020/03/21 07:09 upstream 5ad0ec0b8652 2c31c529 .config console log report ci-upstream-kasan-gce-root
2020/03/18 21:07 upstream 5076190daded 0a96a13c .config console log report ci-upstream-kasan-gce-root
2020/03/17 06:50 upstream fb33c6510d55 749688d2 .config console log report ci-upstream-kasan-gce-root
2020/03/16 14:47 upstream fb33c6510d55 749688d2 .config console log report ci-upstream-kasan-gce-root
2020/03/04 12:42 upstream 63623fd44972 c88c7b75 .config console log report ci-upstream-kasan-gce-selinux-root
2020/02/26 17:23 upstream f8788d86ab28 59b57593 .config console log report ci-upstream-kasan-gce-root
2020/02/16 23:21 upstream 11a48a5a18c6 1f448cd6 .config console log report ci-upstream-kasan-gce-root
2020/02/14 08:04 upstream b19e8c684703 5d7b90f1 .config console log report ci-upstream-kasan-gce-root
2020/02/13 16:23 upstream 0bf999f9c5e7 c5ed587f .config console log report ci-upstream-kasan-gce-root
2020/02/10 15:14 upstream bb6d3fb354c5 18847f55 .config console log report ci-upstream-kasan-gce-root
2020/02/09 01:51 upstream d4f309ca4118 06150bf1 .config console log report ci-upstream-kasan-gce-root
2020/02/07 22:55 upstream 41dcd67e8868 06150bf1 .config console log report ci-upstream-kasan-gce-root
2020/02/07 03:40 upstream 90568ecf5615 06150bf1 .config console log report ci-upstream-kasan-gce-selinux-root
2020/02/06 15:33 upstream 4c46bef2e96a c91cbc9d .config console log report ci-upstream-kasan-gce-root
2020/02/04 02:51 upstream 754beeec1d90 93e5e335 .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/28 21:36 upstream c677124e631d c8e81ce4 .config console log report ci-upstream-kasan-gce-root
2020/01/26 21:31 upstream a45ea48e2bcd dd56146d .config console log report ci-upstream-kasan-gce-root
2020/01/26 01:14 upstream 2821e26f3a0a f4e7270e .config console log report ci-upstream-kasan-gce-root
2020/01/22 17:26 upstream d96d875ef5dd 8eda0b95 .config console log report ci-upstream-kasan-gce-root
2020/01/20 08:38 upstream def9d2780727 0342f8c7 .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/15 09:50 upstream 95e20af9fb9c fa12bd3c .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/15 09:25 upstream 95e20af9fb9c fa12bd3c .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/14 20:25 upstream e033e7d4a808 fa12bd3c .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/14 01:35 upstream b3a987b0264d 32881205 .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/14 01:02 upstream b3a987b0264d 32881205 .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/13 12:44 upstream b3a987b0264d 99565c1a .config console log report ci-upstream-kasan-gce-root
2020/01/12 11:51 upstream 6327edceb62b 31290a45 .config console log report ci-upstream-kasan-gce-root
2020/01/09 05:24 upstream b07f636fca1c ddc3e859 .config console log report ci-upstream-kasan-gce-root
2019/12/26 11:24 upstream 46cf053efec6 be5c2c81 .config console log report ci-upstream-kasan-gce-selinux-root
2019/12/25 04:28 upstream 46cf053efec6 be5c2c81 .config console log report ci-upstream-kasan-gce-root
2020/03/21 11:07 linux-next 770fbb32d34e 4288d95e .config console log report ci-upstream-linux-next-kasan-gce-root
2020/03/05 01:32 linux-next c99b17ac0399 c88c7b75 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/02/17 07:59 linux-next c25a951c50dc 1f448cd6 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/02/10 00:56 linux-next 6dff1565d69c 35f5e45e .config console log report ci-upstream-linux-next-kasan-gce-root
2020/01/12 11:53 linux-next 6c09d7dbb7d3 31290a45 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/01/12 11:28 linux-next 6c09d7dbb7d3 31290a45 .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.