bisecting fixing commit since 7cc2a8ea104820dd9e702202621e8fd4d9f6c8cf building syzkaller on 510951950dc0ee69cfdaf746061d3dbe31b49fd8 testing commit 7cc2a8ea104820dd9e702202621e8fd4d9f6c8cf with gcc (GCC) 8.1.0 kernel signature: dd5bcac7cc69ce310218a7047caa021b7c1d910e1bc1c63dcba8123abee9efe9 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: boot failed: can't ssh into the instance run #4: boot failed: can't ssh into the instance run #5: boot failed: can't ssh into the instance run #6: OK run #7: OK run #8: OK run #9: OK reproducer seems to be flaky testing current HEAD 467f8165a2b0e6accf3d0dd9c8089b1dbde29f7f testing commit 467f8165a2b0e6accf3d0dd9c8089b1dbde29f7f with gcc (GCC) 8.1.0 kernel signature: c7f36506f2c24d1a1f2b312865dad8dda49848daf3132c8af5a2ee21664e522b all runs: OK # git bisect start 467f8165a2b0e6accf3d0dd9c8089b1dbde29f7f 7cc2a8ea104820dd9e702202621e8fd4d9f6c8cf Bisecting: 24243 revisions left to test after this (roughly 15 steps) [726eb70e0d34dc4bc4dada71f52bba8ed638431e] Merge tag 'char-misc-5.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc testing commit 726eb70e0d34dc4bc4dada71f52bba8ed638431e with gcc (GCC) 8.1.0 kernel signature: cf2d82cb91fdc31f209e6516b7aaf15d93a6239e36f2caabd00b78c1b8dbb573 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor338393436" "root@10.128.0.241:./syz-executor338393436"]: exit status 1 ssh: connect to host 10.128.0.241 port 22: Connection timed out lost connection run #8: OK run #9: OK # git bisect good 726eb70e0d34dc4bc4dada71f52bba8ed638431e Bisecting: 12121 revisions left to test after this (roughly 14 steps) [d57801c45f53e3da999e2a0beb932717fe335c41] uio: uio_fsl_elbc_gpcm: use device-managed allocators testing commit d57801c45f53e3da999e2a0beb932717fe335c41 with gcc (GCC) 8.1.0 kernel signature: ec013be1f2c63e149989c23650318f3e82339433d8a08b379da5e73cac5df662 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: crashed: INFO: rcu detected stall in tipc_release run #8: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor935769680" "root@10.128.1.28:./syz-executor935769680"]: exit status 1 ssh: connect to host 10.128.1.28 port 22: Connection timed out lost connection run #9: basic kernel testing failed: failed to copy test binary to VM: timedout ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor239326069" "root@10.128.0.189:./syz-executor239326069"] Warning: Permanently added '10.128.0.189' (ECDSA) to the list of known hosts. # git bisect good d57801c45f53e3da999e2a0beb932717fe335c41 Bisecting: 5872 revisions left to test after this (roughly 13 steps) [c367caf1a38b6f0a1aababafd88b00fefa625f9e] Merge tag 'sound-5.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit c367caf1a38b6f0a1aababafd88b00fefa625f9e with gcc (GCC) 8.1.0 kernel signature: b8eebcf043230c3c3b9ec4427c2d9504c72a37e2b230cf0ba3ecfc4af8540485 all runs: OK # git bisect bad c367caf1a38b6f0a1aababafd88b00fefa625f9e Bisecting: 3118 revisions left to test after this (roughly 12 steps) [1ac0884d5474fea8dc6ceabbd0e870d1bf4b7b42] Merge tag 'core-entry-2020-12-14' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 1ac0884d5474fea8dc6ceabbd0e870d1bf4b7b42 with gcc (GCC) 8.1.0 kernel signature: ba8679d3be7aca62a4bdf6cbaa7e71447ea6362c180f3a4ef9f2e329069acbf2 all runs: OK # git bisect bad 1ac0884d5474fea8dc6ceabbd0e870d1bf4b7b42 Bisecting: 1355 revisions left to test after this (roughly 11 steps) [b10733527bfd864605c33ab2e9a886eec317ec39] Merge tag 'amd-drm-next-5.11-2020-12-09' of git://people.freedesktop.org/~agd5f/linux into drm-next testing commit b10733527bfd864605c33ab2e9a886eec317ec39 with gcc (GCC) 8.1.0 kernel signature: 1ad71da3e3d49c3cb80075f5d6d43b0a57760e6e9119e5339a26be8fab47d503 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: crashed: INFO: rcu detected stall in corrupted run #8: OK run #9: OK # git bisect good b10733527bfd864605c33ab2e9a886eec317ec39 Bisecting: 599 revisions left to test after this (roughly 9 steps) [fab0fca1da5cdc48be051715cd9787df04fdce3a] Merge tag 'media/v5.11-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media testing commit fab0fca1da5cdc48be051715cd9787df04fdce3a with gcc (GCC) 8.1.0 kernel signature: c5aec59d7fa39b491938521df8b8fb6a20f550b0f487a483bf5d9a59e52aa567 all runs: OK # git bisect bad fab0fca1da5cdc48be051715cd9787df04fdce3a Bisecting: 377 revisions left to test after this (roughly 9 steps) [63288c829b1a5991d8f8c15cab596108ed206ba6] media: pixfmt-compressed.rst: fix 'bullet' formatting testing commit 63288c829b1a5991d8f8c15cab596108ed206ba6 with gcc (GCC) 8.1.0 kernel signature: de5fa15379711840bbbaaf70c9b9f3cd375489c2439f294ff7a394c5665c9f78 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor506946200" "root@10.128.0.196:./syz-executor506946200"]: exit status 1 ssh: connect to host 10.128.0.196 port 22: Connection timed out lost connection run #7: OK run #8: OK run #9: OK # git bisect good 63288c829b1a5991d8f8c15cab596108ed206ba6 Bisecting: 187 revisions left to test after this (roughly 8 steps) [94801e5c6d461045726e1563ba2369ef7ce21dbf] Merge tag 'pinctrl-v5.10-3' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl testing commit 94801e5c6d461045726e1563ba2369ef7ce21dbf with gcc (GCC) 8.1.0 kernel signature: 0a592743a59c0240ea356b98d4d631718b670f3c180e2deb9e84bbc0f246a122 all runs: OK # git bisect bad 94801e5c6d461045726e1563ba2369ef7ce21dbf Bisecting: 99 revisions left to test after this (roughly 7 steps) [d9838b1d39283c1200c13f9076474c7624b8ec34] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf testing commit d9838b1d39283c1200c13f9076474c7624b8ec34 with gcc (GCC) 8.1.0 kernel signature: c25435f612649caf0f921a509edad1728cd249cfb3a8553bdb70ddc264031992 all runs: OK # git bisect bad d9838b1d39283c1200c13f9076474c7624b8ec34 Bisecting: 44 revisions left to test after this (roughly 6 steps) [3eca859008a75a4ad363db65b0fe83be1a3d5ad1] igb: use xdp_do_flush testing commit 3eca859008a75a4ad363db65b0fe83be1a3d5ad1 with gcc (GCC) 8.1.0 kernel signature: f0cf502c60f4861c8753f9fce68122c4b0616f843e644cd148102ed9d73fd923 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor118782895" "root@10.128.0.16:./syz-executor118782895"]: exit status 1 ssh: connect to host 10.128.0.16 port 22: Connection timed out lost connection run #8: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor199081401" "root@10.128.15.202:./syz-executor199081401"]: exit status 1 ssh: connect to host 10.128.15.202 port 22: Connection timed out lost connection run #9: OK # git bisect good 3eca859008a75a4ad363db65b0fe83be1a3d5ad1 Bisecting: 22 revisions left to test after this (roughly 5 steps) [3615bdf6d9b19db12b1589861609b4f1c6a8d303] selftests/bpf: Fix "dubious pointer arithmetic" test testing commit 3615bdf6d9b19db12b1589861609b4f1c6a8d303 with gcc (GCC) 8.1.0 kernel signature: c46b9500aadae2ce79754e0481077bd5d011d9029cbe5181a91432e5603fd040 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor773072979" "root@10.128.15.194:./syz-executor773072979"]: exit status 1 ssh: connect to host 10.128.15.194 port 22: Connection timed out lost connection run #8: OK run #9: OK # git bisect good 3615bdf6d9b19db12b1589861609b4f1c6a8d303 Bisecting: 13 revisions left to test after this (roughly 4 steps) [9a25a30ee54c61a186f3f00d9797f1cc43886167] Merge branch 'mlx4_en-fixes' testing commit 9a25a30ee54c61a186f3f00d9797f1cc43886167 with gcc (GCC) 8.1.0 kernel signature: 1440b2ac007041901dac59fb436d9af1545434cdacaaf632cc310090c55458c1 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in tipc_release run #7: crashed: INFO: rcu detected stall in tipc_release run #8: crashed: INFO: rcu detected stall in tipc_release run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor623123255" "root@10.128.15.201:./syz-executor623123255"]: exit status 1 ssh: connect to host 10.128.15.201 port 22: Connection timed out lost connection # git bisect good 9a25a30ee54c61a186f3f00d9797f1cc43886167 Bisecting: 8 revisions left to test after this (roughly 3 steps) [88287773ff6f53c47e1902ae3ae19084ef5c69aa] Merge branch '1GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/net-queue testing commit 88287773ff6f53c47e1902ae3ae19084ef5c69aa with gcc (GCC) 8.1.0 kernel signature: 1440b2ac007041901dac59fb436d9af1545434cdacaaf632cc310090c55458c1 run #0: crashed: INFO: rcu detected stall in tipc_release run #1: crashed: INFO: rcu detected stall in tipc_release run #2: crashed: INFO: rcu detected stall in tipc_release run #3: crashed: INFO: rcu detected stall in tipc_release run #4: crashed: INFO: rcu detected stall in tipc_release run #5: crashed: INFO: rcu detected stall in tipc_release run #6: crashed: INFO: rcu detected stall in tipc_release run #7: crashed: INFO: rcu detected stall in tipc_release run #8: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor676065921" "root@10.128.0.197:./syz-executor676065921"]: exit status 1 ssh: connect to host 10.128.0.197 port 22: Connection timed out lost connection run #9: OK # git bisect good 88287773ff6f53c47e1902ae3ae19084ef5c69aa Bisecting: 4 revisions left to test after this (roughly 2 steps) [2d94b20b95b009eec1a267dcf026b01af627c0cd] netfilter: nft_ct: Remove confirmation check for NFT_CT_ID testing commit 2d94b20b95b009eec1a267dcf026b01af627c0cd with gcc (GCC) 8.1.0 kernel signature: 43cbd282dec600c60180e299ca21e1924a2615aef9cdd42a7340f91c80e82526 all runs: OK # git bisect bad 2d94b20b95b009eec1a267dcf026b01af627c0cd Bisecting: 1 revision left to test after this (roughly 1 step) [917d80d376ffbaa9725fde9e3c0282f63643f278] netfilter: nft_dynset: fix timeouts later than 23 days testing commit 917d80d376ffbaa9725fde9e3c0282f63643f278 with gcc (GCC) 8.1.0 kernel signature: 8014114795cbe6040bca84e8900820d7d12cb1728e401a169a56c834b0740ecb all runs: OK # git bisect bad 917d80d376ffbaa9725fde9e3c0282f63643f278 Bisecting: 0 revisions left to test after this (roughly 0 steps) [cc00bcaa589914096edef7fb87ca5cee4a166b5c] netfilter: x_tables: Switch synchronization to RCU testing commit cc00bcaa589914096edef7fb87ca5cee4a166b5c with gcc (GCC) 8.1.0 kernel signature: 2dd5c2a60f6379fe77fb398b28e3cfa53cc55c2c0bc1889cc2b11759bd3d4d96 all runs: OK # git bisect bad cc00bcaa589914096edef7fb87ca5cee4a166b5c cc00bcaa589914096edef7fb87ca5cee4a166b5c is the first bad commit commit cc00bcaa589914096edef7fb87ca5cee4a166b5c Author: Subash Abhinov Kasiviswanathan Date: Wed Nov 25 11:27:22 2020 -0700 netfilter: x_tables: Switch synchronization to RCU When running concurrent iptables rules replacement with data, the per CPU sequence count is checked after the assignment of the new information. The sequence count is used to synchronize with the packet path without the use of any explicit locking. If there are any packets in the packet path using the table information, the sequence count is incremented to an odd value and is incremented to an even after the packet process completion. The new table value assignment is followed by a write memory barrier so every CPU should see the latest value. If the packet path has started with the old table information, the sequence counter will be odd and the iptables replacement will wait till the sequence count is even prior to freeing the old table info. However, this assumes that the new table information assignment and the memory barrier is actually executed prior to the counter check in the replacement thread. If CPU decides to execute the assignment later as there is no user of the table information prior to the sequence check, the packet path in another CPU may use the old table information. The replacement thread would then free the table information under it leading to a use after free in the packet processing context- Unable to handle kernel NULL pointer dereference at virtual address 000000000000008e pc : ip6t_do_table+0x5d0/0x89c lr : ip6t_do_table+0x5b8/0x89c ip6t_do_table+0x5d0/0x89c ip6table_filter_hook+0x24/0x30 nf_hook_slow+0x84/0x120 ip6_input+0x74/0xe0 ip6_rcv_finish+0x7c/0x128 ipv6_rcv+0xac/0xe4 __netif_receive_skb+0x84/0x17c process_backlog+0x15c/0x1b8 napi_poll+0x88/0x284 net_rx_action+0xbc/0x23c __do_softirq+0x20c/0x48c This could be fixed by forcing instruction order after the new table information assignment or by switching to RCU for the synchronization. Fixes: 80055dab5de0 ("netfilter: x_tables: make xt_replace_table wait until old rules are not used anymore") Reported-by: Sean Tranchetti Reported-by: kernel test robot Suggested-by: Florian Westphal Signed-off-by: Subash Abhinov Kasiviswanathan Signed-off-by: Pablo Neira Ayuso include/linux/netfilter/x_tables.h | 5 +++- net/ipv4/netfilter/arp_tables.c | 14 +++++------ net/ipv4/netfilter/ip_tables.c | 14 +++++------ net/ipv6/netfilter/ip6_tables.c | 14 +++++------ net/netfilter/x_tables.c | 49 ++++++++++++-------------------------- 5 files changed, 40 insertions(+), 56 deletions(-) parent commit 819f56bad110cb27a8be3232467986e2baebe069 wasn't tested testing commit 819f56bad110cb27a8be3232467986e2baebe069 with gcc (GCC) 8.1.0 kernel signature: 791f35806bf706912fe304affafdf0caa53cfdaadeca76050981468091d6d271 culprit signature: 2dd5c2a60f6379fe77fb398b28e3cfa53cc55c2c0bc1889cc2b11759bd3d4d96 parent signature: 791f35806bf706912fe304affafdf0caa53cfdaadeca76050981468091d6d271 Reproducer flagged being flaky revisions tested: 18, total time: 4h40m2.648328726s (build: 1h32m38.196720735s, test: 3h5m17.544584492s) first good commit: cc00bcaa589914096edef7fb87ca5cee4a166b5c netfilter: x_tables: Switch synchronization to RCU recipients (to): ["coreteam@netfilter.org" "davem@davemloft.net" "fw@strlen.de" "kadlec@netfilter.org" "kuba@kernel.org" "kuznet@ms2.inr.ac.ru" "netdev@vger.kernel.org" "netfilter-devel@vger.kernel.org" "pablo@netfilter.org" "pablo@netfilter.org" "subashab@codeaurora.org" "yoshfuji@linux-ipv6.org"] recipients (cc): ["linux-kernel@vger.kernel.org"]