possible deadlock in sch_direct

possible deadlock in sch_direct_xmit (2)
Status: upstream: reported on 2020/04/29 00:59
Reported-by: syzbot+e18ac85757292b7baf96@syzkaller.appspotmail.com
First crash: 513d, last: 4d11h

similar bugs (5):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
android-44	possible deadlock in sch_direct_xmit	C			240	656d	893d	0/2	public: reported C repro on 2019/04/11 08:44
upstream	possible deadlock in sch_direct_xmit	C	done	done	1548	663d	1342d	17/22	fixed on 2020/04/17 19:57
linux-4.14	possible deadlock in sch_direct_xmit				1	840d	840d	0/1	auto-closed as invalid on 2019/10/25 08:40
linux-4.14	possible deadlock in sch_direct_xmit (2)				1	673d	673d	0/1	auto-closed as invalid on 2020/03/15 19:58
linux-4.19	possible deadlock in sch_direct_xmit				1	841d	841d	0/1	auto-closed as invalid on 2019/10/25 08:50

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.14.0-rc7-syzkaller #0 Not tainted
--------------------------------------------
syz-executor.0/3182 is trying to acquire lock:
ffff888038277398 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff888038277398 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
ffff888038277398 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1a1/0x5b0 net/sched/sch_generic.c:340

but task is already holding lock:
ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1a1/0x5b0 net/sched/sch_generic.c:340

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

7 locks held by syz-executor.0/3182:
 #0: ffffffff8c717f20 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:267
 #1: ffffffff8c717f20 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:267
 #2: ffff888012bc6258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #2: ffff888012bc6258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:173 [inline]
 #2: ffff888012bc6258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3873 [inline]
 #2: ffff888012bc6258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x1340/0x34b0 net/core/dev.c:4253
 #3: ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #3: ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
 #3: ffff8880169ebc98 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1a1/0x5b0 net/sched/sch_generic.c:340
 #4: ffffffff8c717f20 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:267
 #5: ffffffff8c717f20 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:267
 #6: ffff888019744258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #6: ffff888019744258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:173 [inline]
 #6: ffff888019744258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3873 [inline]
 #6: ffff888019744258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x1340/0x34b0 net/core/dev.c:4253

stack backtrace:
CPU: 0 PID: 3182 Comm: syz-executor.0 Not tainted 5.14.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1ae/0x29f lib/dump_stack.c:105
 __lock_acquire+0x2615/0x6100 kernel/locking/lockdep.c:4853
 lock_acquire+0x182/0x4a0 kernel/locking/lockdep.c:5625
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:354 [inline]
 __netif_tx_lock include/linux/netdevice.h:4379 [inline]
 sch_direct_xmit+0x1a1/0x5b0 net/sched/sch_generic.c:340
 __dev_xmit_skb net/core/dev.c:3886 [inline]
 __dev_queue_xmit+0x1a2e/0x34b0 net/core/dev.c:4253
 neigh_output include/net/neighbour.h:510 [inline]
 ip_finish_output2+0x79a/0x19f0 net/ipv4/ip_output.c:230
 iptunnel_xmit+0x46b/0x810 net/ipv4/ip_tunnel_core.c:82
 ip_tunnel_xmit+0x1bbd/0x25f0 net/ipv4/ip_tunnel.c:810
 __gre_xmit net/ipv4/ip_gre.c:471 [inline]
 erspan_xmit+0x11f3/0x29a0 net/ipv4/ip_gre.c:709
 __netdev_start_xmit include/linux/netdevice.h:4944 [inline]
 netdev_start_xmit include/linux/netdevice.h:4958 [inline]
 xmit_one net/core/dev.c:3659 [inline]
 dev_hard_start_xmit+0x20b/0x450 net/core/dev.c:3675
 sch_direct_xmit+0x292/0x5b0 net/sched/sch_generic.c:342
 __dev_xmit_skb net/core/dev.c:3886 [inline]
 __dev_queue_xmit+0x1a2e/0x34b0 net/core/dev.c:4253
 neigh_output include/net/neighbour.h:510 [inline]
 ip_finish_output2+0x79a/0x19f0 net/ipv4/ip_output.c:230
 dst_output include/net/dst.h:448 [inline]
 ip_local_out net/ipv4/ip_output.c:126 [inline]
 ip_send_skb+0x113/0x1a0 net/ipv4/ip_output.c:1572
 udp_send_skb+0x97e/0x1240 net/ipv4/udp.c:966
 udp_sendmsg+0x21ee/0x3260 net/ipv4/udp.c:1253
 udpv6_sendmsg+0xa5c/0x3590 net/ipv6/udp.c:1359
 sock_sendmsg_nosec net/socket.c:703 [inline]
 sock_sendmsg net/socket.c:723 [inline]
 ____sys_sendmsg+0x5a2/0x900 net/socket.c:2392
 ___sys_sendmsg net/socket.c:2446 [inline]
 __sys_sendmmsg+0x500/0x790 net/socket.c:2532
 __do_sys_sendmmsg net/socket.c:2561 [inline]
 __se_sys_sendmmsg net/socket.c:2558 [inline]
 __x64_sys_sendmmsg+0x9c/0xb0 net/socket.c:2558
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4665f9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fdf4bfc3188 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 00000000004665f9
RDX: 0400000000000132 RSI: 0000000020004d80 RDI: 0000000000000003
RBP: 00000000004bfcc4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 000000000056bf80
R13: 00007ffedb45ba2f R14: 00007fdf4bfc3300 R15: 0000000000022000

Crashes (54):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Title
ci-upstream-kasan-gce-smack-root	2021/08/27 23:37	upstream	77dd11439b86	d5a29e53	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-kasan-gce-smack-root	2021/07/27 22:05	upstream	ff1176468d36	17d6ab15	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-kasan-gce-386	2021/09/07 13:47	upstream	4b93c544e90e	6ca60148	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/09/15 00:55	net	550ac9c1aaaa	07e953c1	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/09/02 13:35	net	57f780f1c433	15cea0a3	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/08/23 22:23	net	82a44ae113b7	b599f2fc	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/07/31 23:07	net	c7d102232649	6c236867	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/06/28 04:35	net	d6765985a42a	9d2ab5df	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-this-kasan-gce	2021/06/25 15:01	net	be7f62eebaff	ae6bf8dd	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/09/15 21:12	net-next	5706383b30cf	07e953c1	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/09/15 13:04	net-next	339133f6c318	07e953c1	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/09/07 06:44	net-next	27151f177827	6ca60148	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/09/06 23:03	net-next	27151f177827	6ca60148	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/08/10 16:13	net-next	4ef3960ea19c	6972b106	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/08/08 10:53	net-next	f9be84db09d2	6972b106	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/07/14 20:03	net-next	5e437416ff66	94e0b707	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/05/28 08:51	net-next	af9207adb6d9	858ea628	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/03/28 20:40	net-next	a1281601f88e	a8529b82	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/01/23 16:26	net-next	59a49d9617e2	52e37319	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-linux-next-kasan-gce-root	2021/07/28 06:21	linux-next	42d0b5f52c9b	17d6ab15	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-kasan-gce	2020/10/02 17:38	upstream	472e5b056f00	4969d6ca	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/09/27 14:33	net	059432495e20	5dd8aee8	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/08/12 18:37	net	633f5b6bca9b	bc15f7db	.config	log	report
ci-upstream-net-this-kasan-gce	2020/06/30 15:17	net	0574e2000fc3	a2cdad9d	.config	log	report
ci-upstream-net-kasan-gce	2020/12/26 05:45	net-next	3db1a3fa9880	821e0b09	.config	log	report	info
ci-upstream-net-kasan-gce	2020/11/28 16:06	net-next	5c39f26e67c9	76b4dcc7	.config	log	report	info
ci-upstream-bpf-next-kasan-gce	2020/07/07 16:17	bpf-next	cb8e59cc8720	51095195	.config	log	report
ci-upstream-net-kasan-gce	2020/06/29 07:05	net-next	b08866f42a87	a2cdad9d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/11 20:04	bpf-next	cb8e59cc8720	3ab7a05a	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 22:43	bpf-next	cb8e59cc8720	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 04:08	bpf-next	cb8e59cc8720	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 03:58	bpf-next	cb8e59cc8720	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 16:50	bpf-next	cb8e59cc8720	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 15:17	bpf-next	cb8e59cc8720	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/04 04:04	bpf-next	e8224bfe7729	a5ce5de0	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 20:18	bpf-next	9a25c1df24a6	52fd7b7d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/31 22:56	bpf-next	551f08b1d8ea	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 15:21	bpf-next	551f08b1d8ea	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 12:53	bpf-next	551f08b1d8ea	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 03:21	bpf-next	551f08b1d8ea	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 22:03	bpf-next	551f08b1d8ea	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 15:30	bpf-next	9b185fa42176	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 12:17	bpf-next	9b185fa42176	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 11:08	bpf-next	9b185fa42176	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 22:19	bpf-next	c41f9b7371d0	ec153193	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 04:32	bpf-next	c409dc81b682	9072c126	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 07:43	bpf-next	a152b85984a0	11284182	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/23 10:13	bpf-next	a5dfaa2ab940	9682898d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/22 12:37	bpf-next	29ae90d22175	5afa2ddd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/18 05:16	bpf-next	96586dd9268d	37bccd4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 17:15	bpf-next	3f2eaebb5450	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 15:50	bpf-next	3f2eaebb5450	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 23:01	bpf-next	3f2eaebb5450	b8bb8e5f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 00:53	bpf-next	3f2eaebb5450	03d97a1b	.config	log	report