bisecting cause commit starting from bdc5461b23ca293a2a83c851423aae0dd32a48c0 building syzkaller on 59b57593586656c1d5be820aeed0e751087e6ac6 testing commit bdc5461b23ca293a2a83c851423aae0dd32a48c0 with gcc (GCC) 8.1.0 kernel signature: 1bdcc0238be81632ee4b13a6eb94182495e8ced767c3ae5f81e828f0a74f0ac8 all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: 79eb95f49fd148534972d6a5d91447a8d9ef1fe55bc8c6da2af35caa17aa251a all runs: OK # git bisect start bdc5461b23ca293a2a83c851423aae0dd32a48c0 d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 Bisecting: 8054 revisions left to test after this (roughly 13 steps) [ddaefe8947b48b638f726cf89730ecc1000ebcc3] Merge tag 'modules-for-v5.6' of git://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux testing commit ddaefe8947b48b638f726cf89730ecc1000ebcc3 with gcc (GCC) 8.1.0 kernel signature: ff9695042d55b13d30ffd8c360c69a50a7c5307541f9e01174e558ab1a7ac135 run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: boot failed: can't ssh into the instance # git bisect good ddaefe8947b48b638f726cf89730ecc1000ebcc3 Bisecting: 4024 revisions left to test after this (roughly 12 steps) [88f8bbfa94ce18eff7b322256ec4b5f885dea969] Merge tag 'drm-fixes-2020-02-21' of git://anongit.freedesktop.org/drm/drm testing commit 88f8bbfa94ce18eff7b322256ec4b5f885dea969 with gcc (GCC) 8.1.0 kernel signature: a6d8ff368607abe033481e8cb2d129dc3c4c282187afcb1b115d18c79a9b35f2 all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 88f8bbfa94ce18eff7b322256ec4b5f885dea969 Bisecting: 2013 revisions left to test after this (roughly 11 steps) [11777ee8b04acab07c96959e9c6ac6a1603d0958] Merge branch 'i2c/for-5.6' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux testing commit 11777ee8b04acab07c96959e9c6ac6a1603d0958 with gcc (GCC) 8.1.0 kernel signature: b1991d3bc2b487fc88dfff1180e717d9d8052a2f875de407e43c97f1dce50d7c all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 11777ee8b04acab07c96959e9c6ac6a1603d0958 Bisecting: 1009 revisions left to test after this (roughly 10 steps) [9717c1cea16e3eae81ca226f4c3670bb799b61ad] Merge tag 'drm-next-2020-02-04' of git://anongit.freedesktop.org/drm/drm testing commit 9717c1cea16e3eae81ca226f4c3670bb799b61ad with gcc (GCC) 8.1.0 kernel signature: 6a1764ddba2674c694f1feef2fc108225c42185abee395fabb6328c8a35130ef all runs: OK # git bisect good 9717c1cea16e3eae81ca226f4c3670bb799b61ad Bisecting: 513 revisions left to test after this (roughly 9 steps) [0384066381ed5572cf1f57f8d01eaccd3f6d4785] Merge tag 'libata-5.6-2020-02-05' of git://git.kernel.dk/linux-block testing commit 0384066381ed5572cf1f57f8d01eaccd3f6d4785 with gcc (GCC) 8.1.0 kernel signature: a89af0a292ed56f7632ecb66def7f260d3847264c5bb31e7f93139cd76022c84 all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 0384066381ed5572cf1f57f8d01eaccd3f6d4785 Bisecting: 239 revisions left to test after this (roughly 8 steps) [71c3a888cbcaf453aecf8d2f8fb003271d28073f] Merge tag 'powerpc-5.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux testing commit 71c3a888cbcaf453aecf8d2f8fb003271d28073f with gcc (GCC) 8.1.0 kernel signature: 2e14bbd6680b94d9686270c039fda3193bed9d7988d231d23075b7ce6a5f6193 all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 71c3a888cbcaf453aecf8d2f8fb003271d28073f Bisecting: 130 revisions left to test after this (roughly 7 steps) [34b5a946a9543ce38d8ad1aacc4362533a813db7] powerpc: configs: Cleanup old Kconfig options testing commit 34b5a946a9543ce38d8ad1aacc4362533a813db7 with gcc (GCC) 8.1.0 kernel signature: 96b140ee1f56204d7a3ec14af2aa4fac3e35508974b742e5a6c37d6b29c86166 all runs: OK # git bisect good 34b5a946a9543ce38d8ad1aacc4362533a813db7 Bisecting: 65 revisions left to test after this (roughly 6 steps) [190535f7cf50f2d6d6e603715201c58cd6ec696b] include/linux/cpumask.h: don't calculate length of the input string testing commit 190535f7cf50f2d6d6e603715201c58cd6ec696b with gcc (GCC) 8.1.0 kernel signature: 485d678a226b378d892199411197b1a66c71c218e9e7e6bd57a419cd32244646 all runs: OK # git bisect good 190535f7cf50f2d6d6e603715201c58cd6ec696b Bisecting: 30 revisions left to test after this (roughly 5 steps) [a45ad71e8995eed2b95c6ef0f4c442da0c4f6677] Merge tag 'rproc-v5.6' of git://git.kernel.org/pub/scm/linux/kernel/git/andersson/remoteproc testing commit a45ad71e8995eed2b95c6ef0f4c442da0c4f6677 with gcc (GCC) 8.1.0 kernel signature: 69117f9d01eedcdf6cde498cb358fbe03a40b7278cf02ea37c3d76960db39cc2 all runs: OK # git bisect good a45ad71e8995eed2b95c6ef0f4c442da0c4f6677 Bisecting: 15 revisions left to test after this (roughly 4 steps) [7f879e1a94ac99586abf0659c03f35c1e48279c4] Merge tag 'ovl-update-5.6' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs testing commit 7f879e1a94ac99586abf0659c03f35c1e48279c4 with gcc (GCC) 8.1.0 kernel signature: 9fb31911724f334553b541481f43448067b3775ba4e94015c7ae72de153094be all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 7f879e1a94ac99586abf0659c03f35c1e48279c4 Bisecting: 7 revisions left to test after this (roughly 3 steps) [07f1e59637a8e5a8bddba5da7567d46635da510f] ovl: generalize the lower_fs[] array testing commit 07f1e59637a8e5a8bddba5da7567d46635da510f with gcc (GCC) 8.1.0 kernel signature: cdec9096fd4fa0292cae56c8d9d1eff645793e85eeb5c98c130135d8c0a91703 all runs: OK # git bisect good 07f1e59637a8e5a8bddba5da7567d46635da510f Bisecting: 3 revisions left to test after this (roughly 2 steps) [5dcdc43e24a1810d3c3f4959af3d0c8e0026d863] vfs: add vfs_iocb_iter_[read|write] helper functions testing commit 5dcdc43e24a1810d3c3f4959af3d0c8e0026d863 with gcc (GCC) 8.1.0 kernel signature: 710c2f3e4f0969d1e6a329580296328428ae460fd128edea3ef694b2ef6201b1 all runs: OK # git bisect good 5dcdc43e24a1810d3c3f4959af3d0c8e0026d863 Bisecting: 1 revision left to test after this (roughly 1 step) [1a980b8cbf0059a5308eea61522f232fd03002e2] ovl: add splice file read write helper testing commit 1a980b8cbf0059a5308eea61522f232fd03002e2 with gcc (GCC) 8.1.0 kernel signature: 3460c6dba6f21090bfefa662d4569fe62bbdff23f7606e5fc91cf93e05195e83 all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 1a980b8cbf0059a5308eea61522f232fd03002e2 Bisecting: 0 revisions left to test after this (roughly 0 steps) [2406a307ac7ddfd7effeeaff6947149ec6a95b4e] ovl: implement async IO routines testing commit 2406a307ac7ddfd7effeeaff6947149ec6a95b4e with gcc (GCC) 8.1.0 kernel signature: 91c9441c3a28d62803858a19e6f9e3424314f09b44aa980f520e6baa657e223b all runs: crashed: WARNING: lock held when returning to user space in ovl_write_iter # git bisect bad 2406a307ac7ddfd7effeeaff6947149ec6a95b4e 2406a307ac7ddfd7effeeaff6947149ec6a95b4e is the first bad commit commit 2406a307ac7ddfd7effeeaff6947149ec6a95b4e Author: Jiufei Xue Date: Wed Nov 20 17:45:26 2019 +0800 ovl: implement async IO routines A performance regression was observed since linux v4.19 with aio test using fio with iodepth 128 on overlayfs. The queue depth of the device was always 1 which is unexpected. After investigation, it was found that commit 16914e6fc7e1 ("ovl: add ovl_read_iter()") and commit 2a92e07edc5e ("ovl: add ovl_write_iter()") resulted in vfs_iter_{read,write} being called on underlying filesystem, which always results in syncronous IO. Implement async IO for stacked reading and writing. This resolves the performance regresion. This is implemented by allocating a new kiocb for submitting the AIO request on the underlying filesystem. When the request is completed, the new kiocb is freed and the completion callback is called on the original iocb. Signed-off-by: Jiufei Xue Signed-off-by: Miklos Szeredi fs/overlayfs/file.c | 109 ++++++++++++++++++++++++++++++++++++++++++----- fs/overlayfs/overlayfs.h | 2 + fs/overlayfs/super.c | 14 ++++-- 3 files changed, 110 insertions(+), 15 deletions(-) culprit signature: 91c9441c3a28d62803858a19e6f9e3424314f09b44aa980f520e6baa657e223b parent signature: 710c2f3e4f0969d1e6a329580296328428ae460fd128edea3ef694b2ef6201b1 revisions tested: 16, total time: 3h37m25.905279038s (build: 1h43m59.840556542s, test: 1h52m13.218041511s) first bad commit: 2406a307ac7ddfd7effeeaff6947149ec6a95b4e ovl: implement async IO routines cc: ["jiufei.xue@linux.alibaba.com" "linux-kernel@vger.kernel.org" "linux-unionfs@vger.kernel.org" "miklos@szeredi.hu" "mszeredi@redhat.com"] crash: WARNING: lock held when returning to user space in ovl_write_iter ================================================ WARNING: lock held when returning to user space! 5.5.0-rc7-syzkaller #0 Not tainted ------------------------------------------------ syz-executor.3/8445 is leaving the kernel with locks still held! 1 lock held by syz-executor.3/8445: #0: ffff888099162428 (sb_writers#3){.+.+}, at: file_start_write include/linux/fs.h:2897 [inline] #0: ffff888099162428 (sb_writers#3){.+.+}, at: ovl_write_iter+0x859/0x10d0 fs/overlayfs/file.c:345 overlayfs: filesystem on './file0' not supported as upperdir