bisecting fixing commit since 59126901f200f5fc907153468b03c64e0081b6e6 building syzkaller on abf9ba4fc75d9b29af15625d44dcfc1360fad3b7 testing commit 59126901f200f5fc907153468b03c64e0081b6e6 with gcc (GCC) 8.1.0 kernel signature: 6c9af558ef95aeb096f34d676fc459e11a17d370058cc32590b019644e896267 run #0: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: crashed: BUG: unable to handle kernel paging request in dump_schedule run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #3: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #4: crashed: BUG: unable to handle kernel paging request in dump_schedule run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK reproducer seems to be flaky testing current HEAD 83d09ad4b950651a95d37697f1493c00d888d0db testing commit 83d09ad4b950651a95d37697f1493c00d888d0db with gcc (GCC) 8.1.0 kernel signature: 9d22d58b73665927053ec5e41bb01e68e5395934bd861e79987b428284d86423 all runs: OK # git bisect start 83d09ad4b950651a95d37697f1493c00d888d0db 59126901f200f5fc907153468b03c64e0081b6e6 Bisecting: 16652 revisions left to test after this (roughly 14 steps) [0dfbe4c646bf06a85c3d70572a8b8aa6ebffe3d5] perf vendor events: Fix DRAM_BW_Use 0 issue for CLX/SKX testing commit 0dfbe4c646bf06a85c3d70572a8b8aa6ebffe3d5 with gcc (GCC) 8.1.0 kernel signature: 722e51d2556067760efbd89d1230040ec9276ceb3ead95d20006e3e6779310b7 run #0: crashed: WARNING: refcount bug in ip6_dst_destroy run #1: crashed: stack segment fault in dump_schedule run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 0dfbe4c646bf06a85c3d70572a8b8aa6ebffe3d5 Bisecting: 8252 revisions left to test after this (roughly 13 steps) [0cee54c890a40051928991072e5d1cd279611dfd] Merge tag 'usb-5.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb testing commit 0cee54c890a40051928991072e5d1cd279611dfd with gcc (GCC) 8.1.0 kernel signature: 9d58655fe413c57e97f67d14f93abfec040bc8e8c9071d3d3ab9d018e079346a all runs: OK # git bisect bad 0cee54c890a40051928991072e5d1cd279611dfd Bisecting: 4198 revisions left to test after this (roughly 12 steps) [a6b5e026e6238cbdd51e3c9b77cc3c79a7c24a9a] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit a6b5e026e6238cbdd51e3c9b77cc3c79a7c24a9a with gcc (GCC) 8.1.0 kernel signature: 6f271bbd4d0fdfa7fac56e8f8545557fb90342d5406edc5f5148125d3744c1ea all runs: OK # git bisect bad a6b5e026e6238cbdd51e3c9b77cc3c79a7c24a9a Bisecting: 2100 revisions left to test after this (roughly 11 steps) [74ea610602e607338d7301e0061d6b33b6b2ad0a] net/tls: add CHACHA20-POLY1305 configuration testing commit 74ea610602e607338d7301e0061d6b33b6b2ad0a with gcc (GCC) 8.1.0 kernel signature: 04e0c88eee3608bd462c6dff375ead83e48e26c4818cea6f7aebaaa896fc1494 run #0: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #1: crashed: stack segment fault in dump_schedule run #2: crashed: BUG: unable to handle kernel paging request in dump_schedule run #3: crashed: BUG: unable to handle kernel paging request in dump_schedule run #4: crashed: BUG: unable to handle kernel paging request in dump_schedule run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 74ea610602e607338d7301e0061d6b33b6b2ad0a Bisecting: 1057 revisions left to test after this (roughly 10 steps) [7f376f1917d7461e05b648983e8d2aea9d0712b2] Merge tag 'mtd/fixes-for-5.10-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux testing commit 7f376f1917d7461e05b648983e8d2aea9d0712b2 with gcc (GCC) 8.1.0 kernel signature: e607054a1e4a1dfa18b58b1ead4524a1330762daad237da9a07365d85257e066 run #0: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #2: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 7f376f1917d7461e05b648983e8d2aea9d0712b2 Bisecting: 521 revisions left to test after this (roughly 9 steps) [d9054a1ff585ba01029584ab730efc794603d68f] lwt: Disable BH too in run_lwt_bpf() testing commit d9054a1ff585ba01029584ab730efc794603d68f with gcc (GCC) 8.1.0 kernel signature: 3bc0219ce29419bee62de249cce61f24b511aad6df70e9e7bc1d3d2f2b6e9a07 run #0: crashed: BUG: unable to handle kernel paging request in dump_schedule run #1: crashed: BUG: unable to handle kernel paging request in dump_schedule run #2: crashed: BUG: unable to handle kernel paging request in dump_schedule run #3: crashed: stack segment fault in dump_schedule run #4: crashed: BUG: unable to handle kernel paging request in dump_schedule run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good d9054a1ff585ba01029584ab730efc794603d68f Bisecting: 263 revisions left to test after this (roughly 8 steps) [12c0ab6658dea4709189c3730d2431c52808428e] Merge branch 'akpm' (patches from Andrew) testing commit 12c0ab6658dea4709189c3730d2431c52808428e with gcc (GCC) 8.1.0 kernel signature: 5dc980086cb3e76cea3328713234dadc4a3727627e4d88910e0f841b71aedb46 run #0: crashed: BUG: unable to handle kernel paging request in dump_schedule run #1: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #3: crashed: BUG: unable to handle kernel paging request in dump_schedule run #4: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #5: crashed: BUG: unable to handle kernel paging request in dump_schedule run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 12c0ab6658dea4709189c3730d2431c52808428e Bisecting: 165 revisions left to test after this (roughly 7 steps) [9fca90cf28920c6d0723d7efd1eae0b0fb90309c] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma testing commit 9fca90cf28920c6d0723d7efd1eae0b0fb90309c with gcc (GCC) 8.1.0 kernel signature: 22bfbcd7af5e9c78d5684133dd17a195d734ef98ff16957e0a61d5a48122f1d6 run #0: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #1: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #2: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #3: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #4: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}. run #5: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #6: crashed: BUG: unable to handle kernel paging request in dump_schedule run #7: crashed: BUG: unable to handle kernel paging request in dump_schedule run #8: crashed: BUG: unable to handle kernel paging request in dump_schedule run #9: OK # git bisect good 9fca90cf28920c6d0723d7efd1eae0b0fb90309c Bisecting: 82 revisions left to test after this (roughly 6 steps) [d9838b1d39283c1200c13f9076474c7624b8ec34] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf testing commit d9838b1d39283c1200c13f9076474c7624b8ec34 with gcc (GCC) 8.1.0 kernel signature: b5fc102325b7f67c40b982d9537cadada6b5864fc375df85116df7bcf3b90d0c all runs: OK # git bisect bad d9838b1d39283c1200c13f9076474c7624b8ec34 Bisecting: 41 revisions left to test after this (roughly 5 steps) [c02bd115b1d25931159f89c7d9bf47a30f5d4b41] Revert "geneve: pull IP header before ECN decapsulation" testing commit c02bd115b1d25931159f89c7d9bf47a30f5d4b41 with gcc (GCC) 8.1.0 kernel signature: 870ddaaa2bc806f3e80cf392f5d8dbc16a5f7c6e37e47b9152f225a359df9714 run #0: crashed: BUG: unable to handle kernel paging request in dump_schedule run #1: crashed: stack segment fault in dump_schedule run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #3: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good c02bd115b1d25931159f89c7d9bf47a30f5d4b41 Bisecting: 18 revisions left to test after this (roughly 4 steps) [b7e4ba9a91dffd298d940b4d3f173121ff829a32] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf testing commit b7e4ba9a91dffd298d940b4d3f173121ff829a32 with gcc (GCC) 8.1.0 kernel signature: d0a84344718b938302fe86b580287ae44ea765b530700c8cdcdce05671a69bfd all runs: OK # git bisect bad b7e4ba9a91dffd298d940b4d3f173121ff829a32 Bisecting: 13 revisions left to test after this (roughly 4 steps) [1beb7830d3b285b28f7cde3644d59d2590a47e51] ice: avoid premature Rx buffer reuse testing commit 1beb7830d3b285b28f7cde3644d59d2590a47e51 with gcc (GCC) 8.1.0 kernel signature: 8f09c9f7509c2ec3ec5a8c6252634514272d7c97cc54811a40f55a7e53f9d842 run #0: crashed: stack segment fault in dump_schedule run #1: crashed: stack segment fault in dump_schedule run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #3: crashed: BUG: unable to handle kernel paging request in dump_schedule run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK # git bisect good 1beb7830d3b285b28f7cde3644d59d2590a47e51 Bisecting: 6 revisions left to test after this (roughly 3 steps) [ba603d9d7b1215c72513d7c7aa02b6775fd4891b] net/mlx4_en: Handle TX error CQE testing commit ba603d9d7b1215c72513d7c7aa02b6775fd4891b with gcc (GCC) 8.1.0 kernel signature: 9f34233e2c6a210087afbd3ffaa426bffd2fb403173a719b39a422945fccd06b run #0: crashed: WARNING: refcount bug in copy_semundo run #1: crashed: BUG: unable to handle kernel paging request in dump_schedule run #2: crashed: BUG: unable to handle kernel paging request in dump_schedule run #3: crashed: BUG: unable to handle kernel NULL pointer dereference in dump_schedule run #4: OK run #5: OK run #6: OK run #7: OK run #8: crashed: BUG: unable to handle kernel paging request in dump_schedule run #9: OK # git bisect good ba603d9d7b1215c72513d7c7aa02b6775fd4891b Bisecting: 3 revisions left to test after this (roughly 2 steps) [42f1c27120906a54e73101a7d6a12f58813f6a9f] netfilter: nftables: comment indirect serialization of commit_mutex with rtnl_mutex testing commit 42f1c27120906a54e73101a7d6a12f58813f6a9f with gcc (GCC) 8.1.0 kernel signature: 2f52dd373e4477faa6d8ea3b837dcb16a619781ae1ac0855fa124c20fa0fd1df all runs: OK # git bisect bad 42f1c27120906a54e73101a7d6a12f58813f6a9f Bisecting: 0 revisions left to test after this (roughly 1 step) [917d80d376ffbaa9725fde9e3c0282f63643f278] netfilter: nft_dynset: fix timeouts later than 23 days testing commit 917d80d376ffbaa9725fde9e3c0282f63643f278 with gcc (GCC) 8.1.0 kernel signature: 89bf3e382f169735478881ef3f54acfef8487ff5c95679ca359542d1ba28a25a all runs: OK # git bisect bad 917d80d376ffbaa9725fde9e3c0282f63643f278 Bisecting: 0 revisions left to test after this (roughly 0 steps) [cc00bcaa589914096edef7fb87ca5cee4a166b5c] netfilter: x_tables: Switch synchronization to RCU testing commit cc00bcaa589914096edef7fb87ca5cee4a166b5c with gcc (GCC) 8.1.0 kernel signature: 8ce53839c506fffc6d01b0631b6dd393dacf566eed03b3295ca0ae7a7cce94fd all runs: OK # git bisect bad cc00bcaa589914096edef7fb87ca5cee4a166b5c cc00bcaa589914096edef7fb87ca5cee4a166b5c is the first bad commit commit cc00bcaa589914096edef7fb87ca5cee4a166b5c Author: Subash Abhinov Kasiviswanathan Date: Wed Nov 25 11:27:22 2020 -0700 netfilter: x_tables: Switch synchronization to RCU When running concurrent iptables rules replacement with data, the per CPU sequence count is checked after the assignment of the new information. The sequence count is used to synchronize with the packet path without the use of any explicit locking. If there are any packets in the packet path using the table information, the sequence count is incremented to an odd value and is incremented to an even after the packet process completion. The new table value assignment is followed by a write memory barrier so every CPU should see the latest value. If the packet path has started with the old table information, the sequence counter will be odd and the iptables replacement will wait till the sequence count is even prior to freeing the old table info. However, this assumes that the new table information assignment and the memory barrier is actually executed prior to the counter check in the replacement thread. If CPU decides to execute the assignment later as there is no user of the table information prior to the sequence check, the packet path in another CPU may use the old table information. The replacement thread would then free the table information under it leading to a use after free in the packet processing context- Unable to handle kernel NULL pointer dereference at virtual address 000000000000008e pc : ip6t_do_table+0x5d0/0x89c lr : ip6t_do_table+0x5b8/0x89c ip6t_do_table+0x5d0/0x89c ip6table_filter_hook+0x24/0x30 nf_hook_slow+0x84/0x120 ip6_input+0x74/0xe0 ip6_rcv_finish+0x7c/0x128 ipv6_rcv+0xac/0xe4 __netif_receive_skb+0x84/0x17c process_backlog+0x15c/0x1b8 napi_poll+0x88/0x284 net_rx_action+0xbc/0x23c __do_softirq+0x20c/0x48c This could be fixed by forcing instruction order after the new table information assignment or by switching to RCU for the synchronization. Fixes: 80055dab5de0 ("netfilter: x_tables: make xt_replace_table wait until old rules are not used anymore") Reported-by: Sean Tranchetti Reported-by: kernel test robot Suggested-by: Florian Westphal Signed-off-by: Subash Abhinov Kasiviswanathan Signed-off-by: Pablo Neira Ayuso include/linux/netfilter/x_tables.h | 5 +++- net/ipv4/netfilter/arp_tables.c | 14 +++++------ net/ipv4/netfilter/ip_tables.c | 14 +++++------ net/ipv6/netfilter/ip6_tables.c | 14 +++++------ net/netfilter/x_tables.c | 49 ++++++++++++-------------------------- 5 files changed, 40 insertions(+), 56 deletions(-) parent commit 819f56bad110cb27a8be3232467986e2baebe069 wasn't tested testing commit 819f56bad110cb27a8be3232467986e2baebe069 with gcc (GCC) 8.1.0 kernel signature: ee57559e142aae0401ed62f1403ec169df45048762bf05829aa5e7213c177850 culprit signature: 8ce53839c506fffc6d01b0631b6dd393dacf566eed03b3295ca0ae7a7cce94fd parent signature: ee57559e142aae0401ed62f1403ec169df45048762bf05829aa5e7213c177850 Reproducer flagged being flaky revisions tested: 18, total time: 4h58m59.387102553s (build: 1h36m35.432799984s, test: 3h20m27.830647094s) first good commit: cc00bcaa589914096edef7fb87ca5cee4a166b5c netfilter: x_tables: Switch synchronization to RCU recipients (to): ["coreteam@netfilter.org" "davem@davemloft.net" "fw@strlen.de" "kadlec@netfilter.org" "kuba@kernel.org" "netdev@vger.kernel.org" "netfilter-devel@vger.kernel.org" "pablo@netfilter.org" "pablo@netfilter.org" "subashab@codeaurora.org" "yoshfuji@linux-ipv6.org"] recipients (cc): ["linux-kernel@vger.kernel.org"]