bisecting cause commit starting from 4703d9119972bf586d2cca76ec6438f819ffa30e building syzkaller on 2e95ab335759ed7e1c246c2057c84d813a2c29e1 testing commit 4703d9119972bf586d2cca76ec6438f819ffa30e with gcc (GCC) 8.1.0 kernel signature: 1e1f9f4c5529e2251d8e0f0a0a0fb7ad1ffb6d262765eaad21ae63f225ef2471 all runs: crashed: INFO: rcu detected stall in ip_set_udel testing release v5.4 testing commit 219d54332a09e8d8741c1e1982f5eae56099de85 with gcc (GCC) 8.1.0 kernel signature: c5f8ace8c238a33c4bb6492547fa284f6301e86f7505420a2d2b0e424602957a all runs: crashed: INFO: rcu detected stall in ip_set_udel testing release v5.3 testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0 kernel signature: e4dc014671210342e7b062b91cdb05f440cb353703c05a927c3806d548d328aa run #0: crashed: INFO: task hung in nfnetlink_rcv_msg run #1: crashed: INFO: task hung in nfnetlink_rcv_msg run #2: crashed: INFO: task hung in rcu_barrier run #3: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor737667219" "root@10.128.15.203:./syz-executor737667219"]: exit status 1 ssh: connect to host 10.128.15.203 port 22: Connection timed out lost connection run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor500802168" "root@10.128.0.149:./syz-executor500802168"]: exit status 1 Connection timed out during banner exchange lost connection run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor626909290" "root@10.128.0.157:./syz-executor626909290"]: exit status 1 ssh: connect to host 10.128.0.157 port 22: Connection timed out lost connection run #6: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor154139841" "root@10.128.15.202:./syz-executor154139841"]: exit status 1 ssh: connect to host 10.128.15.202 port 22: Connection timed out lost connection run #7: OK run #8: OK run #9: OK testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 kernel signature: 12b42859db9e1e706d7d9b3d1d16bc05851df9440bd6a9f76039be3b8005a57f all runs: crashed: INFO: rcu detected stall in ip_set_udel testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 kernel signature: fa682a6faf24cbed00df22c6536d60e80357da6ba45e6dfca52215074533d903 all runs: crashed: INFO: rcu detected stall in ip_set_udel testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 kernel signature: cfa812c07492a020db857256812a9f5e14d80695f9294991b3b593b89f579c04 all runs: crashed: INFO: rcu detected stall in ip_set_udel testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 kernel signature: dd752d3f54e39edc8a8ad2c6343e78a1960430aff54db13732139008389e32d2 all runs: OK # git bisect start 1c163f4c7b3f621efff9b28a47abb36f7378d783 8fe28cb58bcb235034b64cbbb7550a8a43fd88be Bisecting: 7011 revisions left to test after this (roughly 13 steps) [af7ddd8a627c62a835524b3f5b471edbbbcce025] Merge tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping testing commit af7ddd8a627c62a835524b3f5b471edbbbcce025 with gcc (GCC) 8.1.0 kernel signature: 6eed4021fdd8d73acc6603697ead3b8b84c9dcd5edfe930ae8b8e6ec6b966bb6 all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad af7ddd8a627c62a835524b3f5b471edbbbcce025 Bisecting: 3448 revisions left to test after this (roughly 12 steps) [792bf4d871dea8b69be2aaabdd320d7c6ed15985] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 792bf4d871dea8b69be2aaabdd320d7c6ed15985 with gcc (GCC) 8.1.0 kernel signature: c64eaed03b8ce57adec1bf5964ab6e631577cc8e07f7af34cd2729dcdecf0b1d all runs: OK # git bisect good 792bf4d871dea8b69be2aaabdd320d7c6ed15985 Bisecting: 1729 revisions left to test after this (roughly 11 steps) [aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c] linux/netlink.h: drop unnecessary extern prefix testing commit aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c with gcc (GCC) 8.1.0 kernel signature: a38e900c85eef6bba7abc4ef038a16e488fe1da1aeb7f52cf2c7e9ea442b88ca all runs: OK # git bisect good aa9d6e0f33aea8a1879e7e53fe0e436943f9ce0c Bisecting: 835 revisions left to test after this (roughly 10 steps) [e0c38a4d1f196a4b17d2eba36afff8f656a4f1de] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next testing commit e0c38a4d1f196a4b17d2eba36afff8f656a4f1de with gcc (GCC) 8.1.0 kernel signature: 4c55bf5c8cf12a1ac42077003d1339823a85531992e6247db2a5be16a8ccbe29 all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad e0c38a4d1f196a4b17d2eba36afff8f656a4f1de Bisecting: 384 revisions left to test after this (roughly 9 steps) [8d6973327ee84c2f40dd9efd8928d4a1186c96e2] Merge tag 'powerpc-4.21-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux testing commit 8d6973327ee84c2f40dd9efd8928d4a1186c96e2 with gcc (GCC) 8.1.0 kernel signature: 6b9c8fe8e80617da12b517ff33b9c2d310cd54ca8c96ce4cc0b59089c711f86c all runs: OK # git bisect good 8d6973327ee84c2f40dd9efd8928d4a1186c96e2 Bisecting: 222 revisions left to test after this (roughly 8 steps) [e770454fabde2e0f8fb3e5039a2b6df8f128bc9b] Merge branch 'expand-txtimestamp-selftest' testing commit e770454fabde2e0f8fb3e5039a2b6df8f128bc9b with gcc (GCC) 8.1.0 kernel signature: 480e9f201fecca4fafe29243933075b644f0774b941db2982b1b4f4e70cd7825 all runs: OK # git bisect good e770454fabde2e0f8fb3e5039a2b6df8f128bc9b Bisecting: 106 revisions left to test after this (roughly 7 steps) [c3e533692527046fb55020e7fac8c4272644ba45] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next testing commit c3e533692527046fb55020e7fac8c4272644ba45 with gcc (GCC) 8.1.0 kernel signature: 76cfabda24b2d80e5c2d1ea84c17f5f1377385b23d46e15fb11f08a0abc6cb4e all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad c3e533692527046fb55020e7fac8c4272644ba45 Bisecting: 57 revisions left to test after this (roughly 6 steps) [9df95e8ec568f98d89fe2c72342714296ac6ce27] bpf: sparc64: Enable sparc64 jit to provide bpf_line_info testing commit 9df95e8ec568f98d89fe2c72342714296ac6ce27 with gcc (GCC) 8.1.0 kernel signature: 834acb516a9f2717ed1fcf6e3ef2016f79268282a22c2afeb847eb01bc8d0ecf all runs: OK # git bisect good 9df95e8ec568f98d89fe2c72342714296ac6ce27 Bisecting: 28 revisions left to test after this (roughly 5 steps) [06aa151ad1fc74a49b45336672515774a678d78d] netfilter: ipt_CLUSTERIP: check MAC address when duplicate config is set testing commit 06aa151ad1fc74a49b45336672515774a678d78d with gcc (GCC) 8.1.0 kernel signature: 370153c86a6a6e7b5c5a093b0783a6a1d4a289e0cfb6c2e877c7a77de7a03bc3 all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad 06aa151ad1fc74a49b45336672515774a678d78d Bisecting: 14 revisions left to test after this (roughly 4 steps) [a504b703bb1da526a01593da0e4be2af9d9f5fa8] netfilter: nat: limit port clash resolution attempts testing commit a504b703bb1da526a01593da0e4be2af9d9f5fa8 with gcc (GCC) 8.1.0 kernel signature: 8ffb98c110420e33e68a4ef1f8ea7b4a8acc235498a247dc4fa12cc35e9fbf0c all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad a504b703bb1da526a01593da0e4be2af9d9f5fa8 Bisecting: 6 revisions left to test after this (roughly 3 steps) [b9660987692230b381b64c1f1e912febe142c390] netfilter: nf_flow_table: simplify nf_flow_offload_gc_step() testing commit b9660987692230b381b64c1f1e912febe142c390 with gcc (GCC) 8.1.0 kernel signature: 4b12b2277b07aef9a744c3053dd9f3007eefa140b90e1d90fb24fe90ef0e1586 all runs: crashed: INFO: rcu detected stall in ip_set_udel # git bisect bad b9660987692230b381b64c1f1e912febe142c390 Bisecting: 3 revisions left to test after this (roughly 2 steps) [23c42a403a9cfdbad6004a556c927be7dd61a8ee] netfilter: ipset: Introduction of new commands and protocol version 7 testing commit 23c42a403a9cfdbad6004a556c927be7dd61a8ee with gcc (GCC) 8.1.0 kernel signature: 2ec72b92d8ad187461931edbfd8bacbf3ef2a33c5b2cd2449980524b11051b62 run #0: crashed: INFO: task hung in nfnetlink_rcv_msg run #1: crashed: INFO: task hung in nfnetlink_rcv_msg run #2: crashed: INFO: task hung in corrupted run #3: crashed: INFO: task hung in corrupted run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor304225288" "root@10.128.0.121:./syz-executor304225288"]: exit status 1 ssh: connect to host 10.128.0.121 port 22: Connection timed out lost connection run #5: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor290846663" "root@10.128.0.251:./syz-executor290846663"]: exit status 1 ssh: connect to host 10.128.0.251 port 22: Connection timed out lost connection run #6: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor804933009" "root@10.128.10.28:./syz-executor804933009"]: exit status 1 ssh: connect to host 10.128.10.28 port 22: Connection timed out lost connection run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor861215723" "root@10.128.10.56:./syz-executor861215723"]: exit status 1 ssh: connect to host 10.128.10.56 port 22: Connection timed out lost connection run #8: OK run #9: OK # git bisect bad 23c42a403a9cfdbad6004a556c927be7dd61a8ee Bisecting: 0 revisions left to test after this (roughly 1 step) [29edbc3ebdb0faa934114f14bf12fc0b784d4f1b] netfilter: ipset: Make invalid MAC address checks consistent testing commit 29edbc3ebdb0faa934114f14bf12fc0b784d4f1b with gcc (GCC) 8.1.0 kernel signature: 9d45c42c021898160745ed774cfe332a38d4766da1e29e205f7d64d83da3e1ea all runs: OK # git bisect good 29edbc3ebdb0faa934114f14bf12fc0b784d4f1b 23c42a403a9cfdbad6004a556c927be7dd61a8ee is the first bad commit commit 23c42a403a9cfdbad6004a556c927be7dd61a8ee Author: Jozsef Kadlecsik Date: Sat Oct 27 15:07:40 2018 +0200 netfilter: ipset: Introduction of new commands and protocol version 7 Two new commands (IPSET_CMD_GET_BYNAME, IPSET_CMD_GET_BYINDEX) are introduced. The new commands makes possible to eliminate the getsockopt operation (in iptables set/SET match/target) and thus use only netlink communication between userspace and kernel for ipset. With the new protocol version, userspace can exactly know which functionality is supported by the running kernel. Both the kernel and userspace is fully backward compatible. Signed-off-by: Jozsef Kadlecsik include/linux/netfilter/ipset/ip_set.h | 2 +- include/uapi/linux/netfilter/ipset/ip_set.h | 19 ++-- net/netfilter/ipset/ip_set_core.c | 164 +++++++++++++++++++++++++--- 3 files changed, 160 insertions(+), 25 deletions(-) culprit signature: 2ec72b92d8ad187461931edbfd8bacbf3ef2a33c5b2cd2449980524b11051b62 parent signature: 9d45c42c021898160745ed774cfe332a38d4766da1e29e205f7d64d83da3e1ea revisions tested: 20, total time: 4h30m25.164738244s (build: 1h59m4.427356857s, test: 2h29m36.609282352s) first bad commit: 23c42a403a9cfdbad6004a556c927be7dd61a8ee netfilter: ipset: Introduction of new commands and protocol version 7 cc: ["coreteam@netfilter.org" "davem@davemloft.net" "fw@strlen.de" "kadlec@blackhole.kfki.hu" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org" "netfilter-devel@vger.kernel.org" "pablo@netfilter.org"] crash: INFO: task hung in corrupted IPv6: ADDRCONF(NETDEV_UP): vlan0: link is not ready IPv6: ADDRCONF(NETDEV_UP): vlan1: link is not ready NOHZ: local_softirq_pending 08 INFO: task syz-executor.4:6672 blocked for more than 140 seconds. IPv6: ADDRCONF(NETDEV_CHANGE): macvlan0: link becomes ready Not tainted 4.19.0-rc6-syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. IPv6: ADDRCONF(NETDEV_CHANGE): macvlan1: link becomes ready syz-executor.4 D24048 6672 1 0x00000004 IPv6: ADDRCONF(NETDEV_CHANGE): veth1_vlan: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): veth0_vlan: link becomes ready Call Trace: context_switch kernel/sched/core.c:2825 [inline] __schedule+0x78c/0x1c00 kernel/sched/core.c:3473 IPv6: ADDRCONF(NETDEV_CHANGE): vlan0: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): vlan1: link becomes ready INFO: lockdep is turned off. NMI backtrace for cpu 1 CPU: 1 PID: 1076 Comm: khungtaskd Not tainted 4.19.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x109/0x15a lib/dump_stack.c:113 nmi_cpu_backtrace.cold.3+0x3e/0x76 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0xf5/0x119 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_all_cpu_backtrace include/linux/nmi.h:144 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:204 [inline] watchdog+0x57c/0x9d0 kernel/hung_task.c:265 kthread+0x324/0x3e0 kernel/kthread.c:246 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:413 Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 6809 Comm: syz-executor.5 Not tainted 4.19.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:hash_ip4_del+0x65/0xab0 net/netfilter/ipset/ip_set_hash_gen.h:849 Code: 8d bc 24 88 00 00 00 49 8b 9c 24 b0 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 05 09 00 00 <49> 8b 84 24 88 00 00 00 48 89 45 d0 e8 ca 1d af fb 85 c0 74 0d 80 RSP: 0018:ffff88009858efd8 EFLAGS: 00000246 RAX: dffffc0000000000 RBX: ffff880096f93300 RCX: ffff88009858f160 RDX: 1ffff100138a2a91 RSI: ffff88009858f0e0 RDI: ffff88009c515488 RBP: ffff88009858f060 R08: 0000000000000001 R09: 00000000ac1414bb R10: 0000000000000008 R11: dffffc0000000000 R12: ffff88009c515400 R13: 0000000000000001 R14: ffff88009858f160 R15: 0000000080930c49 FS: 00007f6cdcfa0700(0000) GS:ffff8800aea00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000020000300 CR3: 00000000941fa000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: hash_ip4_uadt+0x47c/0x700 net/netfilter/ipset/ip_set_hash_ip.c:155 call_ad+0x178/0x530 net/netfilter/ipset/ip_set_core.c:1515 ip_set_udel+0x411/0xbe0 net/netfilter/ipset/ip_set_core.c:1651 nfnetlink_rcv_msg+0x942/0xc10 net/netfilter/nfnetlink.c:228 netlink_rcv_skb+0x142/0x390 net/netlink/af_netlink.c:2476 nfnetlink_rcv+0x163/0x3b0 net/netfilter/nfnetlink.c:560 netlink_unicast_kernel net/netlink/af_netlink.c:1310 [inline] netlink_unicast+0x443/0x650 net/netlink/af_netlink.c:1336 netlink_sendmsg+0x765/0xc40 net/netlink/af_netlink.c:1917 sock_sendmsg_nosec net/socket.c:621 [inline] sock_sendmsg+0xb5/0xf0 net/socket.c:631 ___sys_sendmsg+0x647/0x950 net/socket.c:2116 __sys_sendmsg+0xd9/0x180 net/socket.c:2154 __do_sys_sendmsg net/socket.c:2163 [inline] __se_sys_sendmsg net/socket.c:2161 [inline] __x64_sys_sendmsg+0x73/0xb0 net/socket.c:2161 do_syscall_64+0xd0/0x4d0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45b349 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f6cdcf9fc78 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007f6cdcfa06d4 RCX: 000000000045b349 RDX: 0000000000000040 RSI: 0000000020000080 RDI: 0000000000000004 RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 00000000000008ea R14: 00000000004ca43c R15: 000000000075bf2c