bisecting fixing commit since f5d582777bcb1c7ff19a5a2343f66ea01de401c6 building syzkaller on 7795ae03c0d2358a40130693e40e0fcab5232ed2 testing commit f5d582777bcb1c7ff19a5a2343f66ea01de401c6 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: crashed: kernel panic: corrupted stack end in wb_workfn run #5: crashed: kernel panic: corrupted stack end in wb_workfn run #6: OK run #7: OK run #8: OK run #9: OK testing current HEAD 3039fadf2bfdc104dc963820c305778c7c1a6229 testing commit 3039fadf2bfdc104dc963820c305778c7c1a6229 with gcc (GCC) 8.1.0 all runs: OK # git bisect start 3039fadf2bfdc104dc963820c305778c7c1a6229 f5d582777bcb1c7ff19a5a2343f66ea01de401c6 Bisecting: 29231 revisions left to test after this (roughly 15 steps) [36e4523aaf4a35de963e190064b53839fa131653] drm/atomic: Wire file_priv through for property changes testing commit 36e4523aaf4a35de963e190064b53839fa131653 with gcc (GCC) 8.1.0 all runs: OK # git bisect bad 36e4523aaf4a35de963e190064b53839fa131653 Bisecting: 14607 revisions left to test after this (roughly 14 steps) [a655fe9f194842693258f43b5382855db1c2f654] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net testing commit a655fe9f194842693258f43b5382855db1c2f654 with gcc (GCC) 8.1.0 all runs: OK # git bisect bad a655fe9f194842693258f43b5382855db1c2f654 Bisecting: 7288 revisions left to test after this (roughly 13 steps) [af7ddd8a627c62a835524b3f5b471edbbbcce025] Merge tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping testing commit af7ddd8a627c62a835524b3f5b471edbbbcce025 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: crashed: kernel panic: corrupted stack end in wb_workfn run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good af7ddd8a627c62a835524b3f5b471edbbbcce025 Bisecting: 3682 revisions left to test after this (roughly 12 steps) [78e8696c234ab637c4dd516cabeac344d84ec10b] Merge tag 'dmaengine-4.21-rc1' of git://git.infradead.org/users/vkoul/slave-dma testing commit 78e8696c234ab637c4dd516cabeac344d84ec10b with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: crashed: kernel panic: corrupted stack end in wb_workfn run #5: crashed: kernel panic: corrupted stack end in wb_workfn run #6: crashed: kernel panic: corrupted stack end in wb_workfn run #7: OK run #8: OK run #9: OK # git bisect good 78e8696c234ab637c4dd516cabeac344d84ec10b Bisecting: 1841 revisions left to test after this (roughly 11 steps) [8f45fa2724a67f660a3b089950cb51eb9fef63d1] Merge tag 'linux-kselftest-5.0-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest testing commit 8f45fa2724a67f660a3b089950cb51eb9fef63d1 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 8f45fa2724a67f660a3b089950cb51eb9fef63d1 Bisecting: 920 revisions left to test after this (roughly 10 steps) [e215278548f9ea412e2be794efa1c84c6817c657] net: hns3: fix netif_napi_del() not do problem when unloading testing commit e215278548f9ea412e2be794efa1c84c6817c657 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in worker_thread run #1: crashed: BUG: unable to handle kernel NULL pointer dereference in kmem_cache_free run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in sys_open run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good e215278548f9ea412e2be794efa1c84c6817c657 Bisecting: 460 revisions left to test after this (roughly 9 steps) [4ca124f4d96d7c976f2753c874d095c0de83d280] dt-bindings: net: mvneta: add phys property testing commit 4ca124f4d96d7c976f2753c874d095c0de83d280 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: OK run #5: crashed: kernel panic: corrupted stack end in wb_workfn run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 4ca124f4d96d7c976f2753c874d095c0de83d280 Bisecting: 232 revisions left to test after this (roughly 8 steps) [74e96711e3379fc66630f2a1d184947f80cf2c48] Merge tag 'platform-drivers-x86-v5.0-2' of git://git.infradead.org/linux-platform-drivers-x86 testing commit 74e96711e3379fc66630f2a1d184947f80cf2c48 with gcc (GCC) 8.1.0 all runs: OK # git bisect bad 74e96711e3379fc66630f2a1d184947f80cf2c48 Bisecting: 117 revisions left to test after this (roughly 7 steps) [74c953ca5f6b4d5f1daa1ef34f4317e15c1a2987] efi/arm64: Fix debugfs crash by adding a terminator for ptdump marker testing commit 74c953ca5f6b4d5f1daa1ef34f4317e15c1a2987 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 74c953ca5f6b4d5f1daa1ef34f4317e15c1a2987 Bisecting: 60 revisions left to test after this (roughly 6 steps) [89401be6580eab6419bab1f553144131e7a335dc] Merge branch 'efi-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 89401be6580eab6419bab1f553144131e7a335dc with gcc (GCC) 8.1.0 all runs: OK # git bisect bad 89401be6580eab6419bab1f553144131e7a335dc Bisecting: 25 revisions left to test after this (roughly 5 steps) [3cde55ee7921609331178c84cca485491c97df2a] Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit 3cde55ee7921609331178c84cca485491c97df2a with gcc (GCC) 8.1.0 all runs: OK # git bisect bad 3cde55ee7921609331178c84cca485491c97df2a Bisecting: 15 revisions left to test after this (roughly 4 steps) [db7ddeab3ce5d64c9696e70d61f45ea9909cd196] lib/test_kmod.c: potential double free in error handling testing commit db7ddeab3ce5d64c9696e70d61f45ea9909cd196 with gcc (GCC) 8.1.0 all runs: OK # git bisect bad db7ddeab3ce5d64c9696e70d61f45ea9909cd196 Bisecting: 7 revisions left to test after this (roughly 3 steps) [80409c65e2c6cd1540045ee01fc55e50d95e0983] mm: migrate: make buffer_migrate_page_norefs() actually succeed testing commit 80409c65e2c6cd1540045ee01fc55e50d95e0983 with gcc (GCC) 8.1.0 all runs: OK # git bisect bad 80409c65e2c6cd1540045ee01fc55e50d95e0983 Bisecting: 3 revisions left to test after this (roughly 2 steps) [36c0f7f0f89984bb21e6d0f92d776faf7be73096] arch: unexport asm/shmparam.h for all architectures testing commit 36c0f7f0f89984bb21e6d0f92d776faf7be73096 with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: crashed: kernel panic: corrupted stack end in wb_workfn run #5: crashed: kernel panic: corrupted stack end in wb_workfn run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 36c0f7f0f89984bb21e6d0f92d776faf7be73096 Bisecting: 1 revision left to test after this (roughly 1 step) [a8e911d13540487942d53137c156bd7707f66e5d] x86_64: increase stack size for KASAN_EXTRA testing commit a8e911d13540487942d53137c156bd7707f66e5d with gcc (GCC) 8.1.0 all runs: OK # git bisect bad a8e911d13540487942d53137c156bd7707f66e5d Bisecting: 0 revisions left to test after this (roughly 0 steps) [1ac25013fb9e4ed595cd608a406191e93520881e] mm/hugetlb.c: teach follow_hugetlb_page() to handle FOLL_NOWAIT testing commit 1ac25013fb9e4ed595cd608a406191e93520881e with gcc (GCC) 8.1.0 run #0: crashed: kernel panic: corrupted stack end in wb_workfn run #1: crashed: kernel panic: corrupted stack end in wb_workfn run #2: crashed: kernel panic: corrupted stack end in wb_workfn run #3: crashed: kernel panic: corrupted stack end in wb_workfn run #4: OK run #5: crashed: kernel panic: corrupted stack end in wb_workfn run #6: crashed: kernel panic: corrupted stack end in wb_workfn run #7: crashed: kernel panic: corrupted stack end in wb_workfn run #8: OK run #9: OK # git bisect good 1ac25013fb9e4ed595cd608a406191e93520881e a8e911d13540487942d53137c156bd7707f66e5d is the first bad commit commit a8e911d13540487942d53137c156bd7707f66e5d Author: Qian Cai Date: Fri Feb 1 14:20:20 2019 -0800 x86_64: increase stack size for KASAN_EXTRA If the kernel is configured with KASAN_EXTRA, the stack size is increasted significantly because this option sets "-fstack-reuse" to "none" in GCC [1]. As a result, it triggers stack overrun quite often with 32k stack size compiled using GCC 8. For example, this reproducer https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/madvise/madvise06.c triggers a "corrupted stack end detected inside scheduler" very reliably with CONFIG_SCHED_STACK_END_CHECK enabled. There are just too many functions that could have a large stack with KASAN_EXTRA due to large local variables that have been called over and over again without being able to reuse the stacks. Some noticiable ones are size 7648 shrink_page_list 3584 xfs_rmap_convert 3312 migrate_page_move_mapping 3312 dev_ethtool 3200 migrate_misplaced_transhuge_page 3168 copy_process There are other 49 functions are over 2k in size while compiling kernel with "-Wframe-larger-than=" even with a related minimal config on this machine. Hence, it is too much work to change Makefiles for each object to compile without "-fsanitize-address-use-after-scope" individually. [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715#c23 Although there is a patch in GCC 9 to help the situation, GCC 9 probably won't be released in a few months and then it probably take another 6-month to 1-year for all major distros to include it as a default. Hence, the stack usage with KASAN_EXTRA can be revisited again in 2020 when GCC 9 is everywhere. Until then, this patch will help users avoid stack overrun. This has already been fixed for arm64 for the same reason via 6e8830674ea ("arm64: kasan: Increase stack size for KASAN_EXTRA"). Link: http://lkml.kernel.org/r/20190109215209.2903-1-cai@lca.pw Signed-off-by: Qian Cai Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Dmitry Vyukov Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds :040000 040000 b87804e8b90f11861a9569d7a1b36dce1cc3783f 4f0dfc40aa80c6230e91a55ca015ebd5ec149f59 M arch revisions tested: 18, total time: 4h47m58.972745339s (build: 1h43m9.865799036s, test: 2h58m41.326753428s) first good commit: a8e911d13540487942d53137c156bd7707f66e5d x86_64: increase stack size for KASAN_EXTRA cc: ["akpm@linux-foundation.org" "aryabinin@virtuozzo.com" "bp@alien8.de" "cai@lca.pw" "dvyukov@google.com" "glider@google.com" "hpa@zytor.com" "mingo@redhat.com" "tglx@linutronix.de" "torvalds@linux-foundation.org"]