bisecting cause commit starting from d8b9faec54ae4bc2fff68bcd0befa93ace8256ce building syzkaller on d895b3be2cffeab3004bc0087a9f3e2b2b790892 testing commit d8b9faec54ae4bc2fff68bcd0befa93ace8256ce with gcc (GCC) 8.1.0 kernel signature: 14b5af02dfb33ef16f8affde2ca49a1f3d63331c5e54db717698db903295e194 all runs: crashed: possible deadlock in __dev_queue_xmit testing release v5.7 testing commit 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 with gcc (GCC) 8.1.0 kernel signature: 9ba73a1b54ec1e6f13f979cc5ea06ffb99d89a5ea6872c9c5e20ca5417f4d360 all runs: OK # git bisect start d8b9faec54ae4bc2fff68bcd0befa93ace8256ce 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 Bisecting: 8699 revisions left to test after this (roughly 13 steps) [886d7de631da71e30909980fdbf318f7caade262] Merge branch 'akpm' (patches from Andrew) testing commit 886d7de631da71e30909980fdbf318f7caade262 with gcc (GCC) 8.1.0 kernel signature: 2d2bd6ab7387557e06544af103b72deb0e92ebcc83edfa795eb9ff6d51af9aea all runs: crashed: possible deadlock in __dev_queue_xmit # git bisect bad 886d7de631da71e30909980fdbf318f7caade262 Bisecting: 4325 revisions left to test after this (roughly 12 steps) [039aeb9deb9291f3b19c375a8bc6fa7f768996cc] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit 039aeb9deb9291f3b19c375a8bc6fa7f768996cc with gcc (GCC) 8.1.0 kernel signature: 92e69b7c09fe4b443d3f7853e9437c58fb003d83dac22705572645df2ccbef9d all runs: OK # git bisect good 039aeb9deb9291f3b19c375a8bc6fa7f768996cc Bisecting: 2162 revisions left to test after this (roughly 11 steps) [9785b92b4443f2862495c9aa0ee8caed6f43523d] mlxsw: spectrum: Add packet traps for BFD packets testing commit 9785b92b4443f2862495c9aa0ee8caed6f43523d with gcc (GCC) 8.1.0 kernel signature: e45cba6aa383868e4df08faa5232b9fb8cef6cf326fc6227ef8784e39e8ef42e all runs: crashed: possible deadlock in __dev_queue_xmit # git bisect bad 9785b92b4443f2862495c9aa0ee8caed6f43523d Bisecting: 1049 revisions left to test after this (roughly 10 steps) [5d9e4722c74e8868d5fe2f8749de80928eb4a1d1] Merge tag 'wireless-drivers-next-2020-05-07' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next testing commit 5d9e4722c74e8868d5fe2f8749de80928eb4a1d1 with gcc (GCC) 8.1.0 kernel signature: 19dba88957a92c69e711564df7a7086e8f98e7703178daf677e79d88d96bbb01 all runs: crashed: possible deadlock in __dev_queue_xmit # git bisect bad 5d9e4722c74e8868d5fe2f8749de80928eb4a1d1 Bisecting: 556 revisions left to test after this (roughly 9 steps) [da4063bdfcfa70ec57a6c25f772ac6378b1584ad] netlink: allow NLA_MSECS to have range validation testing commit da4063bdfcfa70ec57a6c25f772ac6378b1584ad with gcc (GCC) 8.1.0 kernel signature: 180f5059ef63126c699022bf825775fda230abd6ee81cf840631c8e2068428ac all runs: OK # git bisect good da4063bdfcfa70ec57a6c25f772ac6378b1584ad Bisecting: 278 revisions left to test after this (roughly 8 steps) [58618ef85546726cf27c38ddc1b022c703b7a6ad] net: nxp: Fix use correct return type for ndo_start_xmit() testing commit 58618ef85546726cf27c38ddc1b022c703b7a6ad with gcc (GCC) 8.1.0 kernel signature: 13ae6735cdee7771182349dbcb92ed65c24e2868857906af83cc0395e35b6259 all runs: crashed: possible deadlock in __dev_queue_xmit # git bisect bad 58618ef85546726cf27c38ddc1b022c703b7a6ad Bisecting: 163 revisions left to test after this (roughly 7 steps) [5b95dea31636ce93660930d16172fe75589b2e70] Merge branch 'net-smc-extent-buffer-mapping-and-port-handling' testing commit 5b95dea31636ce93660930d16172fe75589b2e70 with gcc (GCC) 8.1.0 kernel signature: 88c5b5fd0b53f33bfe6d79b09e021eb583268bc7d2ef74b2c4ee36a69d64fe55 all runs: OK # git bisect good 5b95dea31636ce93660930d16172fe75589b2e70 Bisecting: 81 revisions left to test after this (roughly 6 steps) [80f8443fcdaa27871a233d08e9142612e6ade77c] net: ena: avoid unnecessary admin command when RSS function set fails testing commit 80f8443fcdaa27871a233d08e9142612e6ade77c with gcc (GCC) 8.1.0 kernel signature: 99c9bd6b9659bbc050e30a45202220ac7ac082809d84903b7f60e86c0e596307 all runs: OK # git bisect good 80f8443fcdaa27871a233d08e9142612e6ade77c Bisecting: 40 revisions left to test after this (roughly 5 steps) [f0ec4f1d32ad49a23b93156949208dd9348e3590] net/smc: save state of last sent CDC message testing commit f0ec4f1d32ad49a23b93156949208dd9348e3590 with gcc (GCC) 8.1.0 kernel signature: 6e6229f2a24efc3ac8be1a0b9a2fbacc0d992875949012fdb458602ffdf1557d all runs: OK # git bisect good f0ec4f1d32ad49a23b93156949208dd9348e3590 Bisecting: 19 revisions left to test after this (roughly 4 steps) [09be4c47abe384fe8bd6a6f25012013acacc7729] Merge branch 'net-ipa-I-O-map-SMEM-and-IMEM' testing commit 09be4c47abe384fe8bd6a6f25012013acacc7729 with gcc (GCC) 8.1.0 kernel signature: dcfa1d7c3ece2229602a6ebd715dd9ac024e8694562b3b796b8c88d80e2e30ae all runs: OK # git bisect good 09be4c47abe384fe8bd6a6f25012013acacc7729 Bisecting: 9 revisions left to test after this (roughly 3 steps) [885a26bae0223cac7f939a4a549f2df6c7f89bbd] arm64: dts: ti: k3-am65-mcu: add cpsw cpts node testing commit 885a26bae0223cac7f939a4a549f2df6c7f89bbd with gcc (GCC) 8.1.0 kernel signature: 59d784d9c53ababa498c6077e60392d156c7e82e9f1d586e751d31c307470f0a all runs: OK # git bisect good 885a26bae0223cac7f939a4a549f2df6c7f89bbd Bisecting: 4 revisions left to test after this (roughly 2 steps) [1a33e10e4a95cb109ff1145098175df3113313ef] net: partially revert dynamic lockdep key changes testing commit 1a33e10e4a95cb109ff1145098175df3113313ef with gcc (GCC) 8.1.0 kernel signature: a8440eab1621b007c2bd82776cf2a1ee75df3759cdef52a86726f879f5ef88f1 all runs: crashed: possible deadlock in __dev_queue_xmit # git bisect bad 1a33e10e4a95cb109ff1145098175df3113313ef Bisecting: 2 revisions left to test after this (roughly 1 step) [29390928fe9a7cac7f8b1479f0f285034f16eb6f] arm64: dts: ti: k3-j721e-mcu: add mcu cpsw cpts node testing commit 29390928fe9a7cac7f8b1479f0f285034f16eb6f with gcc (GCC) 8.1.0 kernel signature: f3f3cac7032933ea7840daa592cb772a09b49fabb69306898b380dd60fa39cf2 all runs: OK # git bisect good 29390928fe9a7cac7f8b1479f0f285034f16eb6f Bisecting: 0 revisions left to test after this (roughly 1 step) [ea84c842900872f5b8d6d4754cf0fa37d6672011] Merge branch 'net-ethernet-ti-k3-introduce-common-platform-time-sync-driver-cpts' testing commit ea84c842900872f5b8d6d4754cf0fa37d6672011 with gcc (GCC) 8.1.0 kernel signature: 23076198e5b31be24ffd5a9f0477ce325b8ca0cdf7c230078b3a67e76e964fd9 all runs: OK # git bisect good ea84c842900872f5b8d6d4754cf0fa37d6672011 1a33e10e4a95cb109ff1145098175df3113313ef is the first bad commit commit 1a33e10e4a95cb109ff1145098175df3113313ef Author: Cong Wang Date: Sat May 2 22:22:19 2020 -0700 net: partially revert dynamic lockdep key changes This patch reverts the folowing commits: commit 064ff66e2bef84f1153087612032b5b9eab005bd "bonding: add missing netdev_update_lockdep_key()" commit 53d374979ef147ab51f5d632dfe20b14aebeccd0 "net: avoid updating qdisc_xmit_lock_key in netdev_update_lockdep_key()" commit 1f26c0d3d24125992ab0026b0dab16c08df947c7 "net: fix kernel-doc warning in " commit ab92d68fc22f9afab480153bd82a20f6e2533769 "net: core: add generic lockdep keys" but keeps the addr_list_lock_key because we still lock addr_list_lock nestedly on stack devices, unlikely xmit_lock this is safe because we don't take addr_list_lock on any fast path. Reported-and-tested-by: syzbot+aaa6fa4949cc5d9b7b25@syzkaller.appspotmail.com Cc: Dmitry Vyukov Cc: Taehee Yoo Signed-off-by: Cong Wang Acked-by: Taehee Yoo Signed-off-by: David S. Miller drivers/net/bonding/bond_main.c | 1 + drivers/net/ethernet/netronome/nfp/nfp_net_repr.c | 16 ++++ drivers/net/hamradio/bpqether.c | 20 +++++ drivers/net/hyperv/netvsc_drv.c | 2 + drivers/net/ipvlan/ipvlan_main.c | 2 + drivers/net/macsec.c | 2 + drivers/net/macvlan.c | 2 + drivers/net/ppp/ppp_generic.c | 2 + drivers/net/team/team.c | 1 + drivers/net/vrf.c | 1 + drivers/net/wireless/intersil/hostap/hostap_hw.c | 22 ++++++ include/linux/netdevice.h | 27 +++++-- net/8021q/vlan_dev.c | 21 ++++++ net/batman-adv/soft-interface.c | 30 ++++++++ net/bluetooth/6lowpan.c | 8 ++ net/core/dev.c | 90 ++++++++++++++++++----- net/dsa/slave.c | 12 +++ net/ieee802154/6lowpan/core.c | 8 ++ net/l2tp/l2tp_eth.c | 1 + net/netrom/af_netrom.c | 21 ++++++ net/rose/af_rose.c | 21 ++++++ net/sched/sch_generic.c | 17 +++-- 22 files changed, 294 insertions(+), 33 deletions(-) culprit signature: a8440eab1621b007c2bd82776cf2a1ee75df3759cdef52a86726f879f5ef88f1 parent signature: 23076198e5b31be24ffd5a9f0477ce325b8ca0cdf7c230078b3a67e76e964fd9 revisions tested: 16, total time: 3h39m12.129805328s (build: 1h34m0.812023749s, test: 2h3m11.269253877s) first bad commit: 1a33e10e4a95cb109ff1145098175df3113313ef net: partially revert dynamic lockdep key changes recipients (to): ["ap420073@gmail.com" "davem@davemloft.net" "syzbot+aaa6fa4949cc5d9b7b25@syzkaller.appspotmail.com" "xiyou.wangcong@gmail.com"] recipients (cc): [] crash: possible deadlock in __dev_queue_xmit ============================================ WARNING: possible recursive locking detected 5.7.0-rc2-syzkaller #0 Not tainted -------------------------------------------- syz-executor.0/8519 is trying to acquire lock: ffff8880909d0498 (_xmit_ETHER){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:353 [inline] ffff8880909d0498 (_xmit_ETHER){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4068 [inline] ffff8880909d0498 (_xmit_ETHER){+.-.}-{2:2}, at: __dev_queue_xmit+0x20d2/0x2990 net/core/dev.c:4116 but task is already holding lock: ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:353 [inline] ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4068 [inline] ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: sch_direct_xmit+0x1f3/0xff0 net/sched/sch_generic.c:311 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(_xmit_ETHER); lock(_xmit_ETHER); *** DEADLOCK *** May be due to missing lock nesting notation 11 locks held by syz-executor.0/8519: #0: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: l3mdev_l3_out include/net/l3mdev.h:179 [inline] #0: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: l3mdev_ip6_out include/net/l3mdev.h:200 [inline] #0: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: rawv6_send_hdrinc net/ipv6/raw.c:677 [inline] #0: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: rawv6_sendmsg+0x179b/0x3100 net/ipv6/raw.c:944 #1: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline] #1: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x1c1/0x2340 net/ipv6/ip6_output.c:103 #2: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x205/0x2990 net/core/dev.c:4055 #3: ffff8880a40d1258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:363 [inline] #3: ffff8880a40d1258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:159 [inline] #3: ffff8880a40d1258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:128 [inline] #3: ffff8880a40d1258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3736 [inline] #3: ffff8880a40d1258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1fcc/0x2990 net/core/dev.c:4089 #4: ffff8880a40d1148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_hh_output include/net/neighbour.h:499 [inline] #4: ffff8880a40d1148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_output include/net/neighbour.h:508 [inline] #4: ffff8880a40d1148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: ip6_finish_output2+0xc90/0x2340 net/ipv6/ip6_output.c:117 #5: ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:353 [inline] #5: ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4068 [inline] #5: ffff88809e079498 (_xmit_ETHER){+.-.}-{2:2}, at: sch_direct_xmit+0x1f3/0xff0 net/sched/sch_generic.c:311 #6: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: icmpv6_send+0x0/0x150 net/ipv6/ip6_icmp.c:31 #7: ffff8880928570e0 (k-slock-AF_INET6){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:363 [inline] #7: ffff8880928570e0 (k-slock-AF_INET6){+...}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:117 [inline] #7: ffff8880928570e0 (k-slock-AF_INET6){+...}-{2:2}, at: icmp6_send+0xa91/0x1e80 net/ipv6/icmp.c:538 #8: ffffffff88db7800 (rcu_read_lock){....}-{1:2}, at: icmp6_send+0xfa5/0x1e80 net/ipv6/icmp.c:598 #9: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline] #9: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: ip6_finish_output2+0x1c1/0x2340 net/ipv6/ip6_output.c:103 #10: ffffffff88db77a0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x205/0x2990 net/core/dev.c:4055 stack backtrace: CPU: 1 PID: 8519 Comm: syz-executor.0 Not tainted 5.7.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 print_deadlock_bug kernel/locking/lockdep.c:2410 [inline] check_deadlock kernel/locking/lockdep.c:2451 [inline] validate_chain kernel/locking/lockdep.c:3221 [inline] __lock_acquire.cold.67+0x148/0x3da kernel/locking/lockdep.c:4355 lock_acquire+0x1e3/0x970 kernel/locking/lockdep.c:4934 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151 spin_lock include/linux/spinlock.h:353 [inline] __netif_tx_lock include/linux/netdevice.h:4068 [inline] __dev_queue_xmit+0x20d2/0x2990 net/core/dev.c:4116 neigh_output include/net/neighbour.h:510 [inline] ip6_finish_output2+0xcc4/0x2340 net/ipv6/ip6_output.c:117 NF_HOOK_COND include/linux/netfilter.h:296 [inline] ip6_output+0x200/0x700 net/ipv6/ip6_output.c:176 ip6_send_skb+0x8d/0x2e0 net/ipv6/ip6_output.c:1865 icmp6_send+0x1677/0x1e80 net/ipv6/icmp.c:617 icmpv6_send+0x89/0x150 net/ipv6/ip6_icmp.c:43 ip6_link_failure+0x1d/0x3a0 net/ipv6/route.c:2640 dst_link_failure include/net/dst.h:418 [inline] ip_tunnel_xmit+0x1273/0x27e1 net/ipv4/ip_tunnel.c:820 erspan_xmit+0x815/0x2f00 net/ipv4/ip_gre.c:683 __netdev_start_xmit include/linux/netdevice.h:4587 [inline] netdev_start_xmit include/linux/netdevice.h:4601 [inline] xmit_one net/core/dev.c:3545 [inline] dev_hard_start_xmit+0x167/0x790 net/core/dev.c:3561 sch_direct_xmit+0x27a/0xff0 net/sched/sch_generic.c:313 qdisc_restart net/sched/sch_generic.c:376 [inline] __qdisc_run+0x3f0/0x1750 net/sched/sch_generic.c:384 qdisc_run include/net/pkt_sched.h:134 [inline] __dev_xmit_skb net/core/dev.c:3736 [inline] __dev_queue_xmit+0x19ba/0x2990 net/core/dev.c:4089 neigh_hh_output include/net/neighbour.h:499 [inline] neigh_output include/net/neighbour.h:508 [inline] ip6_finish_output2+0xc90/0x2340 net/ipv6/ip6_output.c:117 NF_HOOK_COND include/linux/netfilter.h:296 [inline] ip6_output+0x200/0x700 net/ipv6/ip6_output.c:176 dst_output include/net/dst.h:435 [inline] NF_HOOK include/linux/netfilter.h:307 [inline] rawv6_send_hdrinc net/ipv6/raw.c:687 [inline] rawv6_sendmsg+0x1960/0x3100 net/ipv6/raw.c:944 sock_sendmsg_nosec net/socket.c:652 [inline] sock_sendmsg+0xac/0xe0 net/socket.c:672 sock_write_iter+0x218/0x380 net/socket.c:1004 call_write_iter include/linux/fs.h:1907 [inline] new_sync_write+0x531/0x6c0 fs/read_write.c:484 vfs_write+0x18b/0x490 fs/read_write.c:559 ksys_write+0x180/0x1c0 fs/read_write.c:612 do_syscall_64+0xc6/0x620 arch/x86/entry/common.c:295 entry_SYSCALL_64_after_hwframe+0x49/0xb3 RIP: 0033:0x45cc79 Code: 2d b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb b5 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f6046831c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 00000000000358c0 RCX: 000000000045cc79 RDX: 0000000000000028 RSI: 0000000020000140 RDI: 0000000000000005 RBP: 000000000078bfe0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000078bfac R13: 00007ffd74c70c6f R14: 00007f60468329c0 R15: 000000000078bfac