bisecting cause commit starting from 91265a6da44d13296bdb5b3fbafa19d06be462f2 building syzkaller on e2d91b1d0dd8c8b4760986ec8114469246022bb8 testing commit 91265a6da44d13296bdb5b3fbafa19d06be462f2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: adc62ae07dfc728220de5680b4d0b1a23b2d0c46f465eec6fcdf90aa408aa27f run #0: crashed: WARNING in kernfs_remove_by_name_ns run #1: crashed: WARNING in kernfs_remove_by_name_ns run #2: crashed: WARNING in kernfs_remove_by_name_ns run #3: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #4: crashed: WARNING in kernfs_remove_by_name_ns run #5: crashed: WARNING in kernfs_remove_by_name_ns run #6: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #7: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #8: crashed: WARNING in kernfs_remove_by_name_ns run #9: crashed: WARNING in kernfs_remove_by_name_ns run #10: crashed: WARNING in kernfs_remove_by_name_ns run #11: crashed: WARNING in kernfs_remove_by_name_ns run #12: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #13: crashed: WARNING in kernfs_remove_by_name_ns run #14: crashed: WARNING in kernfs_remove_by_name_ns run #15: crashed: WARNING in kernfs_remove_by_name_ns run #16: crashed: WARNING in kernfs_remove_by_name_ns run #17: crashed: WARNING in kernfs_remove_by_name_ns run #18: crashed: WARNING in kernfs_remove_by_name_ns run #19: crashed: WARNING: kobject bug in erofs_unregister_sysfs testing release v5.16 testing commit df0cc57e057f18e44dac8e6c18aba47ab53202f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1fd9ab6ec52dc5d6eec160897ae3403fe7938723519a9996a423afd6588ef1fc failed: failed to create VM pool: failed to create GCE image: create image operation failed: &{Code:PERMISSIONS_ERROR Location: Message:Required 'read' permission for 'disks/ci-upstream-linux-next-kasan-gce-root-bisect-job-bisect-job-image.tar.gz' ForceSendFields:[] NullFields:[]}. testing release v5.15 testing commit 8bb7eca972ad531c9b149c0a51ab43a417385813 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d129517d67eba16b8ed6fc5963dd9e6a424ece301dd97a6d5aff57914402ade6 all runs: OK # git bisect start 91265a6da44d13296bdb5b3fbafa19d06be462f2 8bb7eca972ad531c9b149c0a51ab43a417385813 Bisecting: 19007 revisions left to test after this (roughly 14 steps) [8efd0d9c316af470377894a6a0f9ff63ce18c177] Merge tag '5.17-net-next' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 8efd0d9c316af470377894a6a0f9ff63ce18c177 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4cdfdf29abb1d8a20d52dbe32f441828c06586f758bdb8339b5da3d910ee9828 all runs: OK # git bisect good 8efd0d9c316af470377894a6a0f9ff63ce18c177 Bisecting: 9536 revisions left to test after this (roughly 13 steps) [1c9566edd537c5f15672fea1cb3f9f5fd7d5211f] Revert "soc: mediatek: mmsys: add mmsys reset control for MT8186" testing commit 1c9566edd537c5f15672fea1cb3f9f5fd7d5211f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ba1abbb2ee07c6614421fca0e3921ea82cb810880a2a2882a4a31b6025ed6429 all runs: OK # git bisect good 1c9566edd537c5f15672fea1cb3f9f5fd7d5211f Bisecting: 4291 revisions left to test after this (roughly 12 steps) [c9a988a08dff2e7a61bcf518cd8307b1659031ef] Merge branch 'drm-next' of git://git.freedesktop.org/git/drm/drm.git testing commit c9a988a08dff2e7a61bcf518cd8307b1659031ef compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 76da76e4dd8edb2cc0ceeadeabb4cd1c361486bceebfc0184ac801d59b7d9eb6 all runs: OK # git bisect good c9a988a08dff2e7a61bcf518cd8307b1659031ef Bisecting: 2128 revisions left to test after this (roughly 11 steps) [718ed2df8b3a87c344b3ddd7a2c86490632018f8] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy.git testing commit 718ed2df8b3a87c344b3ddd7a2c86490632018f8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e8bc36c3639567834e39adcfcea488b80cf75f2d63077171fd29ba89c0ba3859 all runs: OK # git bisect good 718ed2df8b3a87c344b3ddd7a2c86490632018f8 Bisecting: 1004 revisions left to test after this (roughly 10 steps) [f1644aca5f4cce50055241aaff27807aa31ea389] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl.git testing commit f1644aca5f4cce50055241aaff27807aa31ea389 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a4404d24d69021fa158cd546bbd734aabc5108371cb5716526d94877f9a455f6 all runs: OK # git bisect good f1644aca5f4cce50055241aaff27807aa31ea389 Bisecting: 475 revisions left to test after this (roughly 9 steps) [26dd056b2b1105da6f569198d75f4590abcac41b] Merge branch 'for-next' of git://git.infradead.org/users/willy/pagecache.git testing commit 26dd056b2b1105da6f569198d75f4590abcac41b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0013127fb59f39ee6eca7634a8b3efac92a19ef489da2ebb1f2fc8990e794608 all runs: OK # git bisect good 26dd056b2b1105da6f569198d75f4590abcac41b Bisecting: 237 revisions left to test after this (roughly 8 steps) [2c7e0c7a2386db3d0c910ae1df09ade268509941] mm/damon/dbgfs-test: fix is_target_id() change testing commit 2c7e0c7a2386db3d0c910ae1df09ade268509941 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 307d3e618909ae3d3b06341b8fff8b37cad3d2acb8dabec981122cdb0e4faf43 run #0: crashed: WARNING in kernfs_remove_by_name_ns run #1: crashed: WARNING in kernfs_remove_by_name_ns run #2: crashed: WARNING in kernfs_remove_by_name_ns run #3: crashed: WARNING in kernfs_remove_by_name_ns run #4: crashed: WARNING in kernfs_remove_by_name_ns run #5: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #6: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #7: crashed: WARNING in kernfs_remove_by_name_ns run #8: crashed: WARNING in kernfs_remove_by_name_ns run #9: crashed: WARNING: kobject bug in erofs_unregister_sysfs # git bisect bad 2c7e0c7a2386db3d0c910ae1df09ade268509941 Bisecting: 118 revisions left to test after this (roughly 7 steps) [dc56ea366cae75495a02fa92f7746650490ae760] mm: rename zap_skip_check_mapping() to should_zap_page() testing commit dc56ea366cae75495a02fa92f7746650490ae760 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6a9ff0d3cc65f87881b5da34a754179d3151181369a76d8f0010902ed23d70b6 run #0: basic kernel testing failed: timed out run #1: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #2: crashed: WARNING in kernfs_remove_by_name_ns run #3: crashed: WARNING in kernfs_remove_by_name_ns run #4: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #5: crashed: WARNING in kernfs_remove_by_name_ns run #6: crashed: WARNING in kernfs_remove_by_name_ns run #7: crashed: WARNING in kernfs_remove_by_name_ns run #8: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #9: crashed: WARNING in kernfs_remove_by_name_ns # git bisect bad dc56ea366cae75495a02fa92f7746650490ae760 Bisecting: 59 revisions left to test after this (roughly 6 steps) [205b62214269f3b49e717aaafe812c1e1eb819ae] mm: unexport page_init_poison testing commit 205b62214269f3b49e717aaafe812c1e1eb819ae compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 84285089a771d34aef4cf1ab046f5747d78b7d955904ae74afe83440adf613b7 all runs: OK # git bisect good 205b62214269f3b49e717aaafe812c1e1eb819ae Bisecting: 29 revisions left to test after this (roughly 5 steps) [8e74703ba24499f5a263a6a35e8741f80e92b2e8] mm/memcg: disable migration instead of preemption in drain_all_stock(). testing commit 8e74703ba24499f5a263a6a35e8741f80e92b2e8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 4ae0f5ff325280e5db176ea049bd11cf46cc8bf54c94568dcaef007083244a34 all runs: OK # git bisect good 8e74703ba24499f5a263a6a35e8741f80e92b2e8 Bisecting: 14 revisions left to test after this (roughly 4 steps) [dc604f89941a3fccef7b3cbcc69982e431b9fb6a] mm: list_lru: rename list_lru_per_memcg to list_lru_memcg testing commit dc604f89941a3fccef7b3cbcc69982e431b9fb6a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 625b129bcf242d89dc09cac439f3be57c13c69715a29d4a61af7eec9201a9245 run #0: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #1: crashed: WARNING in kernfs_remove_by_name_ns run #2: crashed: WARNING in kernfs_remove_by_name_ns run #3: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #4: crashed: WARNING in kernfs_remove_by_name_ns run #5: crashed: WARNING in kernfs_remove_by_name_ns run #6: crashed: WARNING in kernfs_remove_by_name_ns run #7: crashed: WARNING in kernfs_remove_by_name_ns run #8: crashed: WARNING in kernfs_remove_by_name_ns run #9: crashed: WARNING in kernfs_remove_by_name_ns # git bisect bad dc604f89941a3fccef7b3cbcc69982e431b9fb6a Bisecting: 7 revisions left to test after this (roughly 3 steps) [d624d1eb66c90f6ded10cde6e6aa8fc614bb6856] xarray: use kmem_cache_alloc_lru to allocate xa_node testing commit d624d1eb66c90f6ded10cde6e6aa8fc614bb6856 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f266cb7b7743e3d29111811ac31b826e3043b401723a47cb423f6dd7e1a7f3fd all runs: OK # git bisect good d624d1eb66c90f6ded10cde6e6aa8fc614bb6856 Bisecting: 3 revisions left to test after this (roughly 2 steps) [235b972dbd81510e68196dbb4c907da106864fdd] mm: list_lru: replace linear array with xarray testing commit 235b972dbd81510e68196dbb4c907da106864fdd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2fb6d84e2daa20bda692cb9f5af3e45718e6bf2475ce042fdcbbc92e783fcb55 run #0: crashed: WARNING in kernfs_remove_by_name_ns run #1: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #2: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #3: crashed: WARNING in kernfs_remove_by_name_ns run #4: crashed: WARNING in kernfs_remove_by_name_ns run #5: crashed: WARNING in kernfs_remove_by_name_ns run #6: crashed: WARNING in kernfs_remove_by_name_ns run #7: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #8: crashed: WARNING in kernfs_remove_by_name_ns run #9: crashed: WARNING in kernfs_remove_by_name_ns # git bisect bad 235b972dbd81510e68196dbb4c907da106864fdd Bisecting: 1 revision left to test after this (roughly 1 step) [2768c206f2c3e95c0e5cf2e7f846103fda7cd429] mm: list_lru: allocate list_lru_one only when needed testing commit 2768c206f2c3e95c0e5cf2e7f846103fda7cd429 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 129bd9a2ecb46a347bd6d9e1f4639a49e805e359ee42af58845842008de865c4 run #0: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #1: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #2: crashed: WARNING in kernfs_remove_by_name_ns run #3: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #4: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #5: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #6: crashed: WARNING in kernfs_remove_by_name_ns run #7: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #8: crashed: WARNING: kobject bug in erofs_unregister_sysfs run #9: crashed: WARNING: kobject bug in erofs_unregister_sysfs # git bisect bad 2768c206f2c3e95c0e5cf2e7f846103fda7cd429 Bisecting: 0 revisions left to test after this (roughly 0 steps) [787db852711d7a3ffe90baf424aa4669c98da13d] mm: memcontrol: move memcg_online_kmem() to mem_cgroup_css_online() testing commit 787db852711d7a3ffe90baf424aa4669c98da13d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: af58f834aa433426b9046b490dbfe60c1abd935e67f02bf28305152443ac5f27 run #0: basic kernel testing failed: BUG: program execution failed: executor NUM: exit status NUM run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 787db852711d7a3ffe90baf424aa4669c98da13d 2768c206f2c3e95c0e5cf2e7f846103fda7cd429 is the first bad commit commit 2768c206f2c3e95c0e5cf2e7f846103fda7cd429 Author: Muchun Song Date: Thu Mar 3 12:15:36 2022 +1100 mm: list_lru: allocate list_lru_one only when needed In our server, we found a suspected memory leak problem. The kmalloc-32 consumes more than 6GB of memory. Other kmem_caches consume less than 2GB memory. After our in-depth analysis, the memory consumption of kmalloc-32 slab cache is the cause of list_lru_one allocation. crash> p memcg_nr_cache_ids memcg_nr_cache_ids = $2 = 24574 memcg_nr_cache_ids is very large and memory consumption of each list_lru can be calculated with the following formula. num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) There are 4 numa nodes in our system, so each list_lru consumes ~3MB. crash> list super_blocks | wc -l 952 Every mount will register 2 list lrus, one is for inode, another is for dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 MB (~5.6GB). But the number of memory cgroup is less than 500. So I guess more than 12286 containers have been deployed on this machine (I do not know why there are so many containers, it may be a user's bug or the user really want to do that). And memcg_nr_cache_ids has not been reduced to a suitable value. This can waste a lot of memory. Now the infrastructure for dynamic list_lru_one allocation is ready, so remove statically allocated memory code to save memory. Link: https://lkml.kernel.org/r/20220228122126.37293-11-songmuchun@bytedance.com Signed-off-by: Muchun Song Cc: Alex Shi Cc: Anna Schumaker Cc: Chao Yu Cc: Dave Chinner Cc: Fam Zheng Cc: Jaegeuk Kim Cc: Johannes Weiner Cc: Kari Argillander Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Qi Zheng Cc: Roman Gushchin Cc: Shakeel Butt Cc: Theodore Ts'o Cc: Trond Myklebust Cc: Vladimir Davydov Cc: Vlastimil Babka Cc: Wei Yang Cc: Xiongchun Duan Cc: Yang Shi Signed-off-by: Andrew Morton Signed-off-by: Stephen Rothwell include/linux/list_lru.h | 7 +-- mm/list_lru.c | 121 ++++++++++++++++++++++++++--------------------- mm/memcontrol.c | 6 ++- 3 files changed, 77 insertions(+), 57 deletions(-) culprit signature: 129bd9a2ecb46a347bd6d9e1f4639a49e805e359ee42af58845842008de865c4 parent signature: af58f834aa433426b9046b490dbfe60c1abd935e67f02bf28305152443ac5f27 revisions tested: 18, total time: 3h39m17.960775974s (build: 2h24m18.132819798s, test: 1h13m5.932212293s) first bad commit: 2768c206f2c3e95c0e5cf2e7f846103fda7cd429 mm: list_lru: allocate list_lru_one only when needed recipients (to): ["akpm@linux-foundation.org" "sfr@canb.auug.org.au" "songmuchun@bytedance.com"] recipients (cc): [] crash: WARNING: kobject bug in erofs_unregister_sysfs RDX: 0000000020000000 RSI: 0000000020000100 RDI: 00007ffdc37fdc10 RBP: 00007ffdc37fdc10 R08: 00007ffdc37fdc50 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000286 R12: 00000000200000b0 R13: 0000000000000004 R14: 0000000000000005 R15: 0000000000000002 ------------[ cut here ]------------ kobject: '(null)' (ffff88801c085990): is not initialized, yet kobject_put() is being called. WARNING: CPU: 0 PID: 4059 at lib/kobject.c:750 kobject_put+0x17d/0x410 lib/kobject.c:750 Modules linked in: CPU: 0 PID: 4059 Comm: syz-executor202 Not tainted 5.17.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:kobject_put+0x17d/0x410 lib/kobject.c:750 Code: ff e9 1a ff ff ff 48 89 e8 48 c1 e8 03 80 3c 18 00 0f 85 10 02 00 00 48 8b 75 00 48 89 ea 48 c7 c7 40 89 45 89 e8 ef 6b 8d 04 <0f> 0b e9 c7 fe ff ff c3 4d 89 f9 49 c7 c0 60 91 45 89 48 89 e9 4c RSP: 0018:ffffc900023bfc48 EFLAGS: 00010286 RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffffffff89426fe0 RDI: fffff52000477f7b RBP: ffff88801c085990 R08: 0000000000000001 R09: ffff8880b9e34187 R10: ffffed10173c6830 R11: 0000000000000001 R12: ffff88801c0859cc R13: ffff88807c1fe678 R14: ffff88807c1fe000 R15: ffff88807c1fe068 FS: 0000555555b1c300(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fcedd930b04 CR3: 000000001d4ec000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: erofs_unregister_sysfs+0x41/0x60 fs/erofs/sysfs.c:225 erofs_put_super+0x32/0xb0 fs/erofs/super.c:771 generic_shutdown_super+0x12e/0x3a0 fs/super.c:462 kill_block_super+0x90/0xd0 fs/super.c:1394 erofs_kill_sb+0x46/0x160 fs/erofs/super.c:752 deactivate_locked_super+0x7b/0x130 fs/super.c:332 get_tree_bdev+0x4a5/0x680 fs/super.c:1294 vfs_get_tree+0x7f/0x2c0 fs/super.c:1497 do_new_mount fs/namespace.c:3026 [inline] path_mount+0x7e8/0x1a40 fs/namespace.c:3356 do_mount fs/namespace.c:3369 [inline] __do_sys_mount fs/namespace.c:3577 [inline] __se_sys_mount fs/namespace.c:3554 [inline] __x64_sys_mount+0x1f5/0x260 fs/namespace.c:3554 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7fcedd8d742a Code: 48 c7 c2 c0 ff ff ff f7 d8 64 89 02 b8 ff ff ff ff eb d2 e8 a8 00 00 00 0f 1f 84 00 00 00 00 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 c0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffdc37fdbf8 EFLAGS: 00000286 ORIG_RAX: 00000000000000a5 RAX: ffffffffffffffda RBX: 00007ffdc37fdc50 RCX: 00007fcedd8d742a RDX: 0000000020000000 RSI: 0000000020000100 RDI: 00007ffdc37fdc10 RBP: 00007ffdc37fdc10 R08: 00007ffdc37fdc50 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000286 R12: 00000000200000b0 R13: 0000000000000004 R14: 0000000000000005 R15: 0000000000000002