bisecting cause commit starting from 045df37e743c7448931131988e99e8fe0cc92a54 building syzkaller on 0a8d1a965a65f2d6cc93d1180443f5ad658919b1 testing commit 045df37e743c7448931131988e99e8fe0cc92a54 with gcc (GCC) 8.1.0 all runs: crashed: WARNING: suspicious RCU usage in fib_dump_info_fnhe testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 all runs: OK # git bisect start 045df37e743c7448931131988e99e8fe0cc92a54 v5.1 Bisecting: 7859 revisions left to test after this (roughly 13 steps) [a2d635decbfa9c1e4ae15cb05b68b2559f7f827c] Merge tag 'drm-next-2019-05-09' of git://anongit.freedesktop.org/drm/drm testing commit a2d635decbfa9c1e4ae15cb05b68b2559f7f827c with gcc (GCC) 8.1.0 all runs: OK # git bisect good a2d635decbfa9c1e4ae15cb05b68b2559f7f827c Bisecting: 3927 revisions left to test after this (roughly 12 steps) [ab02888e39212af2d1dddc565cd67192548b9fd8] Merge tag 'armsoc-defconfig' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit ab02888e39212af2d1dddc565cd67192548b9fd8 with gcc (GCC) 8.1.0 all runs: OK # git bisect good ab02888e39212af2d1dddc565cd67192548b9fd8 Bisecting: 1963 revisions left to test after this (roughly 11 steps) [2190cd1974abbafeeb17ce654e679c5d646842e1] net: axienet: add X86 and ARM as supported platforms testing commit 2190cd1974abbafeeb17ce654e679c5d646842e1 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 2190cd1974abbafeeb17ce654e679c5d646842e1 Bisecting: 981 revisions left to test after this (roughly 10 steps) [f85d208658468b1a298f31daddb05a7684969cd4] treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 451 testing commit f85d208658468b1a298f31daddb05a7684969cd4 with gcc (GCC) 8.1.0 all runs: OK # git bisect good f85d208658468b1a298f31daddb05a7684969cd4 Bisecting: 452 revisions left to test after this (roughly 9 steps) [13091aa30535b719e269f20a7bc34002bf5afae5] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net testing commit 13091aa30535b719e269f20a7bc34002bf5afae5 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 13091aa30535b719e269f20a7bc34002bf5afae5 Bisecting: 232 revisions left to test after this (roughly 8 steps) [c036f7dabc34ff14fb8a4a04cf3d53afb435715a] Merge tag 'nfs-for-5.2-3' of git://git.linux-nfs.org/projects/anna/linux-nfs testing commit c036f7dabc34ff14fb8a4a04cf3d53afb435715a with gcc (GCC) 8.1.0 all runs: OK # git bisect good c036f7dabc34ff14fb8a4a04cf3d53afb435715a Bisecting: 96 revisions left to test after this (roughly 7 steps) [dca73a65a68329ee386d3ff473152bac66eaab39] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit dca73a65a68329ee386d3ff473152bac66eaab39 with gcc (GCC) 8.1.0 all runs: OK # git bisect good dca73a65a68329ee386d3ff473152bac66eaab39 Bisecting: 48 revisions left to test after this (roughly 6 steps) [de467c116ca2ffc141d2abf8aef18b360fb19b21] fjes: no need to check return value of debugfs_create functions testing commit de467c116ca2ffc141d2abf8aef18b360fb19b21 with gcc (GCC) 8.1.0 all runs: OK # git bisect good de467c116ca2ffc141d2abf8aef18b360fb19b21 Bisecting: 24 revisions left to test after this (roughly 5 steps) [8bc81c570831f5b739a1c8dfe547b828ef398dfb] tipc: remove the unnecessary msg->req check from tipc_nl_compat_bearer_set testing commit 8bc81c570831f5b739a1c8dfe547b828ef398dfb with gcc (GCC) 8.1.0 all runs: OK # git bisect good 8bc81c570831f5b739a1c8dfe547b828ef398dfb Bisecting: 12 revisions left to test after this (roughly 4 steps) [3401bfb1638efdd0b721d03c51c48171a0cde8c6] ipv6/route: Don't match on fc_nh_id if not set in ip6_route_del() testing commit 3401bfb1638efdd0b721d03c51c48171a0cde8c6 with gcc (GCC) 8.1.0 all runs: crashed: WARNING: suspicious RCU usage in fib_dump_info_fnhe # git bisect bad 3401bfb1638efdd0b721d03c51c48171a0cde8c6 Bisecting: 5 revisions left to test after this (roughly 3 steps) [97236cda3ae5a133f0c3b7295ddc746d1ea2c9c2] net: macb: use GRO testing commit 97236cda3ae5a133f0c3b7295ddc746d1ea2c9c2 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 97236cda3ae5a133f0c3b7295ddc746d1ea2c9c2 Bisecting: 2 revisions left to test after this (roughly 2 steps) [d948974ccc6613b30636014f76700de3aad7e9b7] ipv4/route: Allow NULL flowinfo in rt_fill_info() testing commit d948974ccc6613b30636014f76700de3aad7e9b7 with gcc (GCC) 8.1.0 all runs: OK # git bisect good d948974ccc6613b30636014f76700de3aad7e9b7 Bisecting: 0 revisions left to test after this (roughly 1 step) [ef11209d421976fe0990b43dbf91e2d6918813d0] Revert "net/ipv6: Bail early if user only wants cloned entries" testing commit ef11209d421976fe0990b43dbf91e2d6918813d0 with gcc (GCC) 8.1.0 all runs: crashed: WARNING: suspicious RCU usage in fib_dump_info_fnhe # git bisect bad ef11209d421976fe0990b43dbf91e2d6918813d0 Bisecting: 0 revisions left to test after this (roughly 0 steps) [ee28906fd7a1437ca77a60a99b6b9c6d676220f8] ipv4: Dump route exceptions if requested testing commit ee28906fd7a1437ca77a60a99b6b9c6d676220f8 with gcc (GCC) 8.1.0 all runs: crashed: WARNING: suspicious RCU usage in fib_dump_info_fnhe # git bisect bad ee28906fd7a1437ca77a60a99b6b9c6d676220f8 ee28906fd7a1437ca77a60a99b6b9c6d676220f8 is the first bad commit commit ee28906fd7a1437ca77a60a99b6b9c6d676220f8 Author: Stefano Brivio Date: Fri Jun 21 17:45:23 2019 +0200 ipv4: Dump route exceptions if requested Since commit 4895c771c7f0 ("ipv4: Add FIB nexthop exceptions."), cached exception routes are stored as a separate entity, so they are not dumped on a FIB dump, even if the RTM_F_CLONED flag is passed. This implies that the command 'ip route list cache' doesn't return any result anymore. If the RTM_F_CLONED is passed, and strict checking requested, retrieve nexthop exception routes and dump them. If no strict checking is requested, filtering can't be performed consistently: dump everything in that case. With this, we need to add an argument to the netlink callback in order to track how many entries were already dumped for the last leaf included in a partial netlink dump. A single additional argument is sufficient, even if we traverse logically nested structures (nexthop objects, hash table buckets, bucket chains): it doesn't matter if we stop in the middle of any of those, because they are always traversed the same way. As an example, s_i values in [], s_fa values in (): node (fa) #1 [1] nexthop #1 bucket #1 -> #0 in chain (1) bucket #2 -> #0 in chain (2) -> #1 in chain (3) -> #2 in chain (4) bucket #3 -> #0 in chain (5) -> #1 in chain (6) nexthop #2 bucket #1 -> #0 in chain (7) -> #1 in chain (8) bucket #2 -> #0 in chain (9) -- node (fa) #2 [2] nexthop #1 bucket #1 -> #0 in chain (1) -> #1 in chain (2) bucket #2 -> #0 in chain (3) it doesn't matter if we stop at (3), (4), (7) for "node #1", or at (2) for "node #2": walking flattens all that. It would even be possible to drop the distinction between the in-tree (s_i) and in-node (s_fa) counter, but a further improvement might advise against this. This is only as accurate as the existing tracking mechanism for leaves: if a partial dump is restarted after exceptions are removed or expired, we might skip some non-dumped entries. To improve this, we could attach a 'sernum' attribute (similar to the one used for IPv6) to nexthop entities, and bump this counter whenever exceptions change: having a distinction between the two counters would make this more convenient. Listing of exception routes (modified routes pre-3.5) was tested against these versions of kernel and iproute2: iproute2 kernel 4.14.0 4.15.0 4.19.0 5.0.0 5.1.0 3.5-rc4 + + + + + 4.4 4.9 4.14 4.15 4.19 5.0 5.1 fixed + + + + + v7: - Move loop over nexthop objects to route.c, and pass struct fib_info and table ID to it, not a struct fib_alias (suggested by David Ahern) - While at it, note that the NULL check on fa->fa_info is redundant, and the check on RTNH_F_DEAD is also not consistent with what's done with regular route listing: just keep it for nhc_flags - Rename entry point function for dumping exceptions to fib_dump_info_fnhe(), and rearrange arguments for consistency with fib_dump_info() - Rename fnhe_dump_buckets() to fnhe_dump_bucket() and make it handle one bucket at a time - Expand commit message to describe why we can have a single "skip" counter for all exceptions stored in bucket chains in nexthop objects (suggested by David Ahern) v6: - Rebased onto net-next - Loop over nexthop paths too. Move loop over fnhe buckets to route.c, avoids need to export rt_fill_info() and to touch exceptions from fib_trie.c. Pass NULL as flow to rt_fill_info(), it now allows that (suggested by David Ahern) Fixes: 4895c771c7f0 ("ipv4: Add FIB nexthop exceptions.") Signed-off-by: Stefano Brivio Reviewed-by: David Ahern Signed-off-by: David S. Miller :040000 040000 5035f508bb29245d9c8cedfe775f6188e8c59265 594dc3249898d220e04cb5b808512ee9e50f3c63 M include :040000 040000 717d2fb895155de5f6856085e710e27930ce651f 2983f28d064c0ed69dabd62637bb45a5e0050a45 M net revisions tested: 16, total time: 3h59m52.964339863s (build: 1h30m59.118951451s, test: 2h23m25.718712449s) first bad commit: ee28906fd7a1437ca77a60a99b6b9c6d676220f8 ipv4: Dump route exceptions if requested cc: ["davem@davemloft.net" "dsahern@gmail.com" "kuznet@ms2.inr.ac.ru" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org" "sbrivio@redhat.com" "yoshfuji@linux-ipv6.org"] crash: WARNING: suspicious RCU usage in fib_dump_info_fnhe ============================= WARNING: suspicious RCU usage 5.2.0-rc5+ #1 Not tainted ----------------------------- net/ipv4/route.c:2875 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 1 lock held by syz-executor.1/7594: #0: 0000000025de6ea8 (rtnl_mutex){+.+.}, at: netlink_dump+0xcc/0x10c0 net/netlink/af_netlink.c:2199 stack backtrace: CPU: 1 PID: 7594 Comm: syz-executor.1 Not tainted 5.2.0-rc5+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 lockdep_rcu_suspicious+0x14a/0x153 kernel/locking/lockdep.c:5250 fib_dump_info_fnhe+0x70a/0xbc0 net/ipv4/route.c:2875 fn_trie_dump_leaf net/ipv4/fib_trie.c:2141 [inline] fib_table_dump+0x683/0xca0 net/ipv4/fib_trie.c:2175 inet_dump_fib+0x343/0x9c0 net/ipv4/fib_frontend.c:1004 rtnl_dump_all+0x1ca/0x3a0 net/core/rtnetlink.c:3445 netlink_dump+0x49e/0x10c0 net/netlink/af_netlink.c:2244 __netlink_dump_start+0x52b/0x810 net/netlink/af_netlink.c:2352 netlink_dump_start include/linux/netlink.h:226 [inline] rtnetlink_rcv_msg+0x57a/0x8f0 net/core/rtnetlink.c:5182 netlink_rcv_skb+0x13c/0x380 net/netlink/af_netlink.c:2477 rtnetlink_rcv+0x10/0x20 net/core/rtnetlink.c:5237 netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline] netlink_unicast+0x43f/0x630 net/netlink/af_netlink.c:1328 netlink_sendmsg+0x765/0xc50 net/netlink/af_netlink.c:1917 sock_sendmsg_nosec net/socket.c:646 [inline] sock_sendmsg+0xb5/0xf0 net/socket.c:665 sock_write_iter+0x1e9/0x3d0 net/socket.c:994 call_write_iter include/linux/fs.h:1872 [inline] new_sync_write+0x3fd/0x7e0 fs/read_write.c:483 __vfs_write+0x94/0x110 fs/read_write.c:496 vfs_write+0x150/0x4e0 fs/read_write.c:558 ksys_write+0x105/0x220 fs/read_write.c:611 __do_sys_write fs/read_write.c:623 [inline] __se_sys_write fs/read_write.c:620 [inline] __x64_sys_write+0x6e/0xb0 fs/read_write.c:620 do_syscall_64+0xd0/0x530 arch/x86/entry/common.c:301 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x459519 Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007ffa05f45c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000459519 RDX: 000000000000001c RSI: 0000000020000000 RDI: 0000000000000003 RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffa05f466d4 R13: 00000000004c58be R14: 00000000004df8c0 R15: 00000000ffffffff