bisecting cause commit starting from 7d68e38288421ebd6d62f695f91f6c8bde8a323a building syzkaller on d4f4eca56fbea6f58a4d5adfd19cb5e0dc32fe46 testing commit 7d68e38288421ebd6d62f695f91f6c8bde8a323a with gcc (GCC) 8.1.0 kernel signature: 2b934fca9067d396721bdbda2052f011738a08970a056183cfeb34e142e23048 all runs: crashed: WARNING in pskb_expand_head testing release v5.10 testing commit 2c85ebc57b3e1817b6ce1a6b703928e113a90442 with gcc (GCC) 8.1.0 kernel signature: 395a614079c9030036488f72c030c4708fae86b9aba956ca15d15003efd3a593 all runs: OK # git bisect start 7d68e38288421ebd6d62f695f91f6c8bde8a323a 2c85ebc57b3e1817b6ce1a6b703928e113a90442 Bisecting: 7193 revisions left to test after this (roughly 13 steps) [9d0d886799e49e0f6d51e70c823416919544fdb7] Merge branch 'i2c/for-5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux testing commit 9d0d886799e49e0f6d51e70c823416919544fdb7 with gcc (GCC) 8.1.0 kernel signature: 59463ecc1fc0e052ec3459cf9d969923756f1868a75327f17ec5b2c9f5730850 all runs: OK # git bisect good 9d0d886799e49e0f6d51e70c823416919544fdb7 Bisecting: 3598 revisions left to test after this (roughly 12 steps) [48c1c40ab40cb087b992e7b77518c3a2926743cc] Merge tag 'arm-soc-drivers-5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 48c1c40ab40cb087b992e7b77518c3a2926743cc with gcc (GCC) 8.1.0 kernel signature: f2a3efc3d1ab19c8c12c6d17d85775866cb47c9fb8cc672b3ae9b894ef92a5b8 all runs: OK # git bisect good 48c1c40ab40cb087b992e7b77518c3a2926743cc Bisecting: 1860 revisions left to test after this (roughly 11 steps) [8552d28e140110fc935b39a6bfaf33c8ce3a1ad5] Merge tag 'm68knommu-for-v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu testing commit 8552d28e140110fc935b39a6bfaf33c8ce3a1ad5 with gcc (GCC) 8.1.0 kernel signature: 846894fc88f3f440724aa32d05cd8b66874f64c6831032f2a698e8a2aa1963fd all runs: OK # git bisect good 8552d28e140110fc935b39a6bfaf33c8ce3a1ad5 Bisecting: 930 revisions left to test after this (roughly 10 steps) [4beb17e553b49c3dd74505c9f361e756aaae653e] net: qrtr: fix null-ptr-deref in qrtr_ns_remove testing commit 4beb17e553b49c3dd74505c9f361e756aaae653e with gcc (GCC) 8.1.0 kernel signature: 0a588e2e3dc790f0ffa1162fae8de1671ba542f21094ac02b505f7ed8591df7c all runs: OK # git bisect good 4beb17e553b49c3dd74505c9f361e756aaae653e Bisecting: 448 revisions left to test after this (roughly 9 steps) [65f0d2414b7079556fbbcc070b3d1c9f9587606d] Merge tag 'sound-5.11-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 65f0d2414b7079556fbbcc070b3d1c9f9587606d with gcc (GCC) 8.1.0 kernel signature: 846dc8337e24958604d4a4abc7e23f03f15b0582455af193a10490be76780cf0 all runs: OK # git bisect good 65f0d2414b7079556fbbcc070b3d1c9f9587606d Bisecting: 224 revisions left to test after this (roughly 8 steps) [3c51fa5d2afe7a4909b53af5019635326389dd29] net: phy: ar803x: disable extended next page bit testing commit 3c51fa5d2afe7a4909b53af5019635326389dd29 with gcc (GCC) 8.1.0 kernel signature: c06451de13dd5568fd840211826e34a6e3cd50c84160f2a8e5826fdb67c39e92 all runs: OK # git bisect good 3c51fa5d2afe7a4909b53af5019635326389dd29 Bisecting: 88 revisions left to test after this (roughly 7 steps) [1d9f03c0a15fa01aa14fb295cbc1236403fceb0b] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit 1d9f03c0a15fa01aa14fb295cbc1236403fceb0b with gcc (GCC) 8.1.0 kernel signature: 98c46a96ccd336aeedbc72b38c43a6654c33eb75dae0cf3452558e3c02ca79a6 all runs: crashed: WARNING in pskb_expand_head # git bisect bad 1d9f03c0a15fa01aa14fb295cbc1236403fceb0b Bisecting: 68 revisions left to test after this (roughly 6 steps) [13a9499e833387fcc7a53915bbe5cddf3c336b59] mptcp: fix locking in mptcp_disconnect() testing commit 13a9499e833387fcc7a53915bbe5cddf3c336b59 with gcc (GCC) 8.1.0 kernel signature: 6fbee355c5a69bacc3e8745cea8e1728b5626000784ab808889bf5242e1508f7 all runs: crashed: WARNING in pskb_expand_head # git bisect bad 13a9499e833387fcc7a53915bbe5cddf3c336b59 Bisecting: 33 revisions left to test after this (roughly 5 steps) [20bc80b6f582ad1151c52ca09ab66b472768c9c8] mptcp: more strict state checking for acks testing commit 20bc80b6f582ad1151c52ca09ab66b472768c9c8 with gcc (GCC) 8.1.0 kernel signature: ed5cc65b83faa0ec118e509332c9bd0a660608258392818579a32a850f13115d all runs: OK # git bisect good 20bc80b6f582ad1151c52ca09ab66b472768c9c8 Bisecting: 16 revisions left to test after this (roughly 4 steps) [8ad2a970d2010add3963e7219eb50367ab3fa4eb] cxgb4/chtls: Fix tid stuck due to wrong update of qid testing commit 8ad2a970d2010add3963e7219eb50367ab3fa4eb with gcc (GCC) 8.1.0 kernel signature: 895e38bfdc603f0c17b8a0a8295ffdd847e1f1a51af8c09f0940233ae8508ccd all runs: OK # git bisect good 8ad2a970d2010add3963e7219eb50367ab3fa4eb Bisecting: 8 revisions left to test after this (roughly 3 steps) [93089de91e85743942a5f804850d4f0846e5402b] MAINTAINERS: altx: move Jay Cliburn to CREDITS testing commit 93089de91e85743942a5f804850d4f0846e5402b with gcc (GCC) 8.1.0 kernel signature: 45f8427033fadc70c89a85e2dd4bb49f3b6d01ef61a488087f49581a55da3fb5 all runs: crashed: WARNING in pskb_expand_head # git bisect bad 93089de91e85743942a5f804850d4f0846e5402b Bisecting: 3 revisions left to test after this (roughly 2 steps) [b76889ff51bfee318bea15891420e5aefd2833a0] net: stmmac: fix taprio schedule configuration testing commit b76889ff51bfee318bea15891420e5aefd2833a0 with gcc (GCC) 8.1.0 kernel signature: 25bc39a714341efe6fb435ca10da34c01abba00d1c3bc591d5d3912122422faf run #0: boot failed: can't ssh into the instance run #1: boot failed: can't ssh into the instance run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good b76889ff51bfee318bea15891420e5aefd2833a0 Bisecting: 1 revision left to test after this (roughly 1 step) [7da17624e7948d5d9660b910f8079d26d26ce453] nt: usb: USB_RTL8153_ECM should not default to y testing commit 7da17624e7948d5d9660b910f8079d26d26ce453 with gcc (GCC) 8.1.0 kernel signature: 25bc39a714341efe6fb435ca10da34c01abba00d1c3bc591d5d3912122422faf all runs: OK # git bisect good 7da17624e7948d5d9660b910f8079d26d26ce453 Bisecting: 0 revisions left to test after this (roughly 0 steps) [3226b158e67cfaa677fd180152bfb28989cb2fac] net: avoid 32 x truesize under-estimation for tiny skbs testing commit 3226b158e67cfaa677fd180152bfb28989cb2fac with gcc (GCC) 8.1.0 kernel signature: 45f8427033fadc70c89a85e2dd4bb49f3b6d01ef61a488087f49581a55da3fb5 all runs: crashed: WARNING in pskb_expand_head # git bisect bad 3226b158e67cfaa677fd180152bfb28989cb2fac 3226b158e67cfaa677fd180152bfb28989cb2fac is the first bad commit commit 3226b158e67cfaa677fd180152bfb28989cb2fac Author: Eric Dumazet Date: Wed Jan 13 08:18:19 2021 -0800 net: avoid 32 x truesize under-estimation for tiny skbs Both virtio net and napi_get_frags() allocate skbs with a very small skb->head While using page fragments instead of a kmalloc backed skb->head might give a small performance improvement in some cases, there is a huge risk of under estimating memory usage. For both GOOD_COPY_LEN and GRO_MAX_HEAD, we can fit at least 32 allocations per page (order-3 page in x86), or even 64 on PowerPC We have been tracking OOM issues on GKE hosts hitting tcp_mem limits but consuming far more memory for TCP buffers than instructed in tcp_mem[2] Even if we force napi_alloc_skb() to only use order-0 pages, the issue would still be there on arches with PAGE_SIZE >= 32768 This patch makes sure that small skb head are kmalloc backed, so that other objects in the slab page can be reused instead of being held as long as skbs are sitting in socket queues. Note that we might in the future use the sk_buff napi cache, instead of going through a more expensive __alloc_skb() Another idea would be to use separate page sizes depending on the allocated length (to never have more than 4 frags per page) I would like to thank Greg Thelen for his precious help on this matter, analysing crash dumps is always a time consuming task. Fixes: fd11a83dd363 ("net: Pull out core bits of __netdev_alloc_skb and add __napi_alloc_skb") Signed-off-by: Eric Dumazet Cc: Paolo Abeni Cc: Greg Thelen Reviewed-by: Alexander Duyck Acked-by: Michael S. Tsirkin Link: https://lore.kernel.org/r/20210113161819.1155526-1-eric.dumazet@gmail.com Signed-off-by: Jakub Kicinski net/core/skbuff.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) culprit signature: 45f8427033fadc70c89a85e2dd4bb49f3b6d01ef61a488087f49581a55da3fb5 parent signature: 25bc39a714341efe6fb435ca10da34c01abba00d1c3bc591d5d3912122422faf revisions tested: 16, total time: 3h36m36.432932869s (build: 1h18m29.406697141s, test: 2h16m33.162363046s) first bad commit: 3226b158e67cfaa677fd180152bfb28989cb2fac net: avoid 32 x truesize under-estimation for tiny skbs recipients (to): ["alexanderduyck@fb.com" "edumazet@google.com" "kuba@kernel.org" "mst@redhat.com"] recipients (cc): [] crash: WARNING in pskb_expand_head RBP: 00007f0b3bf62ca0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001 R13: 00007ffdba4407ff R14: 00007f0b3bf639c0 R15: 000000000119bf8c ------------[ cut here ]------------ WARNING: CPU: 1 PID: 10173 at mm/page_alloc.c:4976 __alloc_pages_nodemask+0x235/0x430 mm/page_alloc.c:4976 Modules linked in: CPU: 1 PID: 10173 Comm: syz-executor.2 Not tainted 5.11.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__alloc_pages_nodemask+0x235/0x430 mm/page_alloc.c:4976 Code: 65 8b 05 4e 7d c2 7e a9 00 ff ff 00 75 09 4d 85 e4 0f 84 c1 00 00 00 41 be 41 00 00 00 e9 bb fe ff ff 81 e7 00 20 00 00 75 02 <0f> 0b 45 31 ed eb 9f 65 8b 05 bd 22 c2 7e 83 f8 07 0f 87 75 01 00 RSP: 0018:ffffc90002813c00 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000012 RDI: 0000000000000000 RBP: 0000000000000012 R08: 00000000000a2a20 R09: 00000000ffffffff R10: 0000000000000001 R11: 0000000000000003 R12: 0000000020010300 R13: 0000000020010300 R14: 00000000ffffffff R15: ffffffff82c86ecc FS: 00007f0b3bf63700(0000) GS:ffff88813bd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1d56c47000 CR3: 000000011eb64000 CR4: 00000000001506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __alloc_pages include/linux/gfp.h:511 [inline] __alloc_pages_node include/linux/gfp.h:524 [inline] alloc_pages_node include/linux/gfp.h:538 [inline] kmalloc_large_node+0x4c/0xf0 mm/slub.c:3984 __kmalloc_node_track_caller+0x1e6/0x260 mm/slub.c:4481 __kmalloc_reserve.isra.57+0x6d/0x80 net/core/skbuff.c:150 pskb_expand_head+0x6c/0x290 net/core/skbuff.c:1634 __skb_grow include/linux/skbuff.h:2738 [inline] tun_napi_alloc_frags drivers/net/tun.c:1377 [inline] tun_get_user+0xb3e/0x15f0 drivers/net/tun.c:1734 tun_chr_write_iter+0x4d/0x80 drivers/net/tun.c:1932 call_write_iter include/linux/fs.h:1901 [inline] new_sync_write+0x117/0x1a0 fs/read_write.c:518 vfs_write+0x25d/0x390 fs/read_write.c:605 ksys_write+0x5a/0xd0 fs/read_write.c:658 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x45e219 Code: 0d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 db b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f0b3bf62c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000000000045e219 RDX: 000000002001016f RSI: 0000000020000380 RDI: 0000000000000003 RBP: 00007f0b3bf62ca0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001 R13: 00007ffdba4407ff R14: 00007f0b3bf639c0 R15: 000000000119bf8c