bisecting cause commit starting from 6174f05255e65622ff3340257879a4c0f858b0df building syzkaller on a0092f9dfdd33924abe5cf5565e4ec4748217c7b testing commit 6174f05255e65622ff3340257879a4c0f858b0df with gcc (GCC) 8.1.0 kernel signature: ef326bb9272caddacae59bcaa4ec547cfaa3cd6d5ebdff7850a75f22fd0c6749 run #0: crashed: BUG: unable to handle kernel paging request in ip6_finish_output2 run #1: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #2: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #3: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #4: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #5: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #6: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #7: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #8: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked run #9: crashed: BUG: KFENCE: use-after-free in kernfs_path_from_node_locked testing release v5.9 testing commit bbf5c979011a099af5dc76498918ed7df445635b with gcc (GCC) 8.1.0 kernel signature: ae997c81c8480492441e712d3c127e3da959156d4e937c464609e101fa84e28c all runs: OK # git bisect start 6174f05255e65622ff3340257879a4c0f858b0df bbf5c979011a099af5dc76498918ed7df445635b Bisecting: 12846 revisions left to test after this (roughly 14 steps) [ceae608a54898fff2aa0aba358fe81af027ef8c9] Merge tag 'pwm/for-5.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/thierry.reding/linux-pwm testing commit ceae608a54898fff2aa0aba358fe81af027ef8c9 with gcc (GCC) 8.1.0 kernel signature: 9f4d3a5afb1f2fdc7b7125ae2f9cf8aad1d13d4610dc947aa17143e4137c6b6e all runs: OK # git bisect good ceae608a54898fff2aa0aba358fe81af027ef8c9 Bisecting: 6413 revisions left to test after this (roughly 13 steps) [078278853344b396cc380e599a1a86011726d6fb] Merge remote-tracking branch 'v4l-dvb/master' testing commit 078278853344b396cc380e599a1a86011726d6fb with gcc (GCC) 8.1.0 kernel signature: 9bf6e8032c06bee6561cacb537a93109e71b9524e675e6e1f6c87cdee99c19aa all runs: OK # git bisect good 078278853344b396cc380e599a1a86011726d6fb Bisecting: 3228 revisions left to test after this (roughly 12 steps) [98630f6adbb4f700dab6a7219bdb894b24ef8b19] Merge remote-tracking branch 'drm-misc/for-linux-next' testing commit 98630f6adbb4f700dab6a7219bdb894b24ef8b19 with gcc (GCC) 8.1.0 kernel signature: f56fa814910d60ad74d5f9d412f6164c36c2e3bcbc94efe2fa8ce5169c08a367 all runs: OK # git bisect good 98630f6adbb4f700dab6a7219bdb894b24ef8b19 Bisecting: 1619 revisions left to test after this (roughly 11 steps) [299e32071516df4981c5f3a722ed53b25114618b] Merge remote-tracking branch 'usb-serial/usb-next' testing commit 299e32071516df4981c5f3a722ed53b25114618b with gcc (GCC) 8.1.0 kernel signature: a4768abd47f6de97e94c350c759289b6d64cc8ae86af331529fb6b8e252c4abf all runs: OK # git bisect good 299e32071516df4981c5f3a722ed53b25114618b Bisecting: 826 revisions left to test after this (roughly 10 steps) [74ded69fb6ec1f492381605287994a8dd3e28a8a] Merge remote-tracking branch 'gpio-brgl/gpio/for-next' testing commit 74ded69fb6ec1f492381605287994a8dd3e28a8a with gcc (GCC) 8.1.0 kernel signature: 08a7d6de735c974aa07a4b0a56e6e7d75452f3be80636f1ef3efc30540272278 all runs: OK # git bisect good 74ded69fb6ec1f492381605287994a8dd3e28a8a Bisecting: 424 revisions left to test after this (roughly 9 steps) [f8945513d17e61ff3f0eb31e957c5dd9417d6032] Revert "ASoC/SoundWire: rt715-sdca: First version of rt715 sdw sdca codec driver" testing commit f8945513d17e61ff3f0eb31e957c5dd9417d6032 with gcc (GCC) 8.1.0 kernel signature: 426db4531f6b95bff2813e7d8395e6508bb6bac9a2a4fcfd1e15636309f250bc all runs: OK # git bisect good f8945513d17e61ff3f0eb31e957c5dd9417d6032 Bisecting: 212 revisions left to test after this (roughly 8 steps) [659316014bab138170ee07e585e7b4e5314562fb] mm, page_alloc: do not rely on the order of page_poison and init_on_alloc/free parameters testing commit 659316014bab138170ee07e585e7b4e5314562fb with gcc (GCC) 8.1.0 kernel signature: 4961b9da66f7988cb8eb09eca36807d59c74308636c84f43d0a72d34413bec71 run #0: crashed: WARNING in ip6t_do_table run #1: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_flush_dev run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK reproducer seems to be flaky # git bisect bad 659316014bab138170ee07e585e7b4e5314562fb Bisecting: 105 revisions left to test after this (roughly 7 steps) [574be7a4e4e2c146f055631e6bc49efe22cb27bf] arm64: mremap speedup - enable HAVE_MOVE_PUD testing commit 574be7a4e4e2c146f055631e6bc49efe22cb27bf with gcc (GCC) 8.1.0 kernel signature: 5ca642788f68aa827f76ee65277966736193db7587ccc8744b1b1f56d52f3597 all runs: OK # git bisect good 574be7a4e4e2c146f055631e6bc49efe22cb27bf Bisecting: 52 revisions left to test after this (roughly 6 steps) [acaaa15ef00fc5bb980ef9dad8a494a637f1a77d] mm, page_alloc: disable pcplists during memory offline testing commit acaaa15ef00fc5bb980ef9dad8a494a637f1a77d with gcc (GCC) 8.1.0 kernel signature: 99599eac94e6a99565f76f35fa2fd33332ebea55e50b58f39f92aac5a1ddee19 run #0: crashed: BUG: unable to handle kernel NULL pointer dereference in ip6_finish_output2 run #1: crashed: BUG: unable to handle kernel paging request in ip6_finish_output2 run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #3: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad acaaa15ef00fc5bb980ef9dad8a494a637f1a77d Bisecting: 26 revisions left to test after this (roughly 5 steps) [8f690aa2fa6ffa963d5fe48d0919c5820478d293] mm/vmalloc: rework the drain logic testing commit 8f690aa2fa6ffa963d5fe48d0919c5820478d293 with gcc (GCC) 8.1.0 kernel signature: 3249eeb31b5b08e089fbe6c3a91708292b98d7e5537c9af7493b806ee8a8e71d run #0: crashed: BUG: unable to handle kernel paging request in neigh_remove_one run #1: crashed: BUG: unable to handle kernel paging request in neigh_remove_one run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in ip6_finish_output2 run #3: crashed: BUG: unable to handle kernel paging request in neigh_remove_one run #4: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #5: crashed: WARNING: locking bug in neigh_periodic_work run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 8f690aa2fa6ffa963d5fe48d0919c5820478d293 Bisecting: 12 revisions left to test after this (roughly 4 steps) [2c73e83113bbabd563a982094e262feaaa466cc3] mm,hwpoison: drain pcplists before bailing out for non-buddy zero-refcount page testing commit 2c73e83113bbabd563a982094e262feaaa466cc3 with gcc (GCC) 8.1.0 kernel signature: c2266e157a14aef8619c8a9f2f2510fd5fe54a0913ff0adaf111e9d239faaa9f run #0: crashed: BUG: unable to handle kernel paging request in neigh_remove_one run #1: crashed: WARNING: locking bug in neigh_periodic_work run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 2c73e83113bbabd563a982094e262feaaa466cc3 Bisecting: 6 revisions left to test after this (roughly 3 steps) [0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0] mm: mmap_lock: add tracepoints around lock acquisition testing commit 0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0 with gcc (GCC) 8.1.0 kernel signature: ef2305fe4c26062ee22a26dbd187e733d378e0d0d430538e1b4a79834df592f4 run #0: crashed: BUG: unable to handle kernel NULL pointer dereference in ip6_finish_output2 run #1: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0 Bisecting: 2 revisions left to test after this (roughly 2 steps) [5055c067d757ec300180c815868ef711ee44744c] mm/mmap.c: fix the adjusted length error testing commit 5055c067d757ec300180c815868ef711ee44744c with gcc (GCC) 8.1.0 kernel signature: a95b5548090b52862aa50b8e0ef0f7563de4ea96d36af03481d6cd7d2f750697 all runs: OK # git bisect good 5055c067d757ec300180c815868ef711ee44744c Bisecting: 0 revisions left to test after this (roughly 1 step) [ea2f2a6ea9d7965d7a000b5b3f782071df36a58c] mm/page_vma_mapped.c: add colon to fix kernel-doc markups error for check_pte testing commit ea2f2a6ea9d7965d7a000b5b3f782071df36a58c with gcc (GCC) 8.1.0 kernel signature: 3c3ee3cc844639a17cfc444706fb7898744242179c085926b8290f64a898655c all runs: OK # git bisect good ea2f2a6ea9d7965d7a000b5b3f782071df36a58c 0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0 is the first bad commit commit 0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0 Author: Axel Rasmussen Date: Tue Nov 24 16:37:42 2020 +1100 mm: mmap_lock: add tracepoints around lock acquisition The goal of these tracepoints is to be able to debug lock contention issues. This lock is acquired on most (all?) mmap / munmap / page fault operations, so a multi-threaded process which does a lot of these can experience significant contention. We trace just before we start acquisition, when the acquisition returns (whether it succeeded or not), and when the lock is released (or downgraded). The events are broken out by lock type (read / write). The events are also broken out by memcg path. For container-based workloads, users often think of several processes in a memcg as a single logical "task", so collecting statistics at this level is useful. The end goal is to get latency information. This isn't directly included in the trace events. Instead, users are expected to compute the time between "start locking" and "acquire returned", using e.g. synthetic events or BPF. The benefit we get from this is simpler code. Because we use tracepoint_enabled() to decide whether or not to trace, this patch has effectively no overhead unless tracepoints are enabled at runtime. If tracepoints are enabled, there is a performance impact, but how much depends on exactly what e.g. the BPF program does. [rostedt@goodmis.org: in-depth examples of tracepoint_enabled() usage, and per-cpu-per-context buffer design] Link: https://lkml.kernel.org/r/20201105211739.568279-2-axelrasmussen@google.com Signed-off-by: Axel Rasmussen Acked-by: Vlastimil Babka Cc: Steven Rostedt Cc: Ingo Molnar Cc: Michel Lespinasse Cc: Daniel Jordan Cc: Jann Horn Cc: Chinwen Chang Cc: Davidlohr Bueso Cc: David Rientjes Cc: Laurent Dufour Cc: Yafang Shao Signed-off-by: Andrew Morton Signed-off-by: Stephen Rothwell include/linux/mmap_lock.h | 94 ++++++++++++++++++-- include/trace/events/mmap_lock.h | 107 ++++++++++++++++++++++ mm/Makefile | 2 +- mm/mmap_lock.c | 187 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 384 insertions(+), 6 deletions(-) create mode 100644 include/trace/events/mmap_lock.h create mode 100644 mm/mmap_lock.c culprit signature: ef2305fe4c26062ee22a26dbd187e733d378e0d0d430538e1b4a79834df592f4 parent signature: 3c3ee3cc844639a17cfc444706fb7898744242179c085926b8290f64a898655c Reproducer flagged being flaky revisions tested: 16, total time: 3h59m9.696383193s (build: 1h11m53.60063578s, test: 2h45m28.65785626s) first bad commit: 0f818c4bc1f3dc0d6d0ea916e0ab30cf5e75f4c0 mm: mmap_lock: add tracepoints around lock acquisition recipients (to): ["akpm@linux-foundation.org" "axelrasmussen@google.com" "sfr@canb.auug.org.au" "vbabka@suse.cz"] recipients (cc): [] crash: BUG: unable to handle kernel NULL pointer dereference in neigh_periodic_work BUG: kernel NULL pointer dereference, address: 000000000000006f #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 11f60f067 P4D 11f60f067 PUD 11f3ab067 PMD 0 Oops: 0000 [#1] PREEMPT SMP CPU: 1 PID: 17 Comm: kworker/1:0 Not tainted 5.10.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events_power_efficient neigh_periodic_work RIP: 0010:__lock_acquire+0x5d0/0x1ee0 kernel/locking/lockdep.c:4703 Code: 2f 89 85 10 09 00 00 0f 87 04 01 00 00 3b 05 3b f8 28 05 41 bf 01 00 00 00 0f 86 35 01 00 00 89 05 29 f8 28 05 e9 2a 01 00 00 <48> 81 3f 20 44 2c 86 41 ba 00 00 00 00 45 0f 45 d0 83 fb 01 0f 87 RSP: 0018:ffffc90000ce7cd0 EFLAGS: 00010002 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000006f RBP: ffff888100c3cbc0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000006f FS: 0000000000000000(0000) GS:ffff88813bd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000006f CR3: 000000011f61c000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: lock_acquire+0xd0/0x3d0 kernel/locking/lockdep.c:5437 __raw_write_lock include/linux/rwlock_api_smp.h:210 [inline] _raw_write_lock+0x2a/0x40 kernel/locking/spinlock.c:295 neigh_periodic_work+0xbe/0x2f0 net/core/neighbour.c:923 process_one_work+0x273/0x600 kernel/workqueue.c:2272 worker_thread+0x38/0x380 kernel/workqueue.c:2418 kthread+0x144/0x170 kernel/kthread.c:292 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296 Modules linked in: CR2: 000000000000006f ---[ end trace 4bb107b254e3e3f2 ]--- RIP: 0010:__lock_acquire+0x5d0/0x1ee0 kernel/locking/lockdep.c:4703 Code: 2f 89 85 10 09 00 00 0f 87 04 01 00 00 3b 05 3b f8 28 05 41 bf 01 00 00 00 0f 86 35 01 00 00 89 05 29 f8 28 05 e9 2a 01 00 00 <48> 81 3f 20 44 2c 86 41 ba 00 00 00 00 45 0f 45 d0 83 fb 01 0f 87 RSP: 0018:ffffc90000ce7cd0 EFLAGS: 00010002 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000006f RBP: ffff888100c3cbc0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000006f FS: 0000000000000000(0000) GS:ffff88813bd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000006f CR3: 000000011f61c000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400