possible deadlock in sch_direct

possible deadlock in sch_direct_xmit (2)
Status: upstream: reported on 2020/04/29 00:59
Reported-by: syzbot+e18ac85757292b7baf96@syzkaller.appspotmail.com
First crash: 316d, last: 42d

similar bugs (5):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
android-44	possible deadlock in sch_direct_xmit	C			240	459d	695d	0/2	public: reported C repro on 2019/04/11 08:44
upstream	possible deadlock in sch_direct_xmit	C	done	done	1548	466d	1145d	17/21	fixed on 2020/04/17 19:57
linux-4.14	possible deadlock in sch_direct_xmit				1	643d	643d	0/1	auto-closed as invalid on 2019/10/25 08:40
linux-4.14	possible deadlock in sch_direct_xmit (2)				1	476d	476d	0/1	auto-closed as invalid on 2020/03/15 19:58
linux-4.19	possible deadlock in sch_direct_xmit				1	644d	644d	0/1	auto-closed as invalid on 2019/10/25 08:50

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.11.0-rc4-syzkaller #0 Not tainted
--------------------------------------------
syz-executor.1/2665 is trying to acquire lock:
ffff88805c384398 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff88805c384398 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4202 [inline]
ffff88805c384398 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311

but task is already holding lock:
ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4202 [inline]
ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

9 locks held by syz-executor.1/2665:
 #0: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #0: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x290/0x21b0 net/ipv4/ip_output.c:216
 #1: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1d7/0x2e00 net/core/dev.c:4086
 #2: ffff88801973f258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #2: ffff88801973f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:162 [inline]
 #2: ffff88801973f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:130 [inline]
 #2: ffff88801973f258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3765 [inline]
 #2: ffff88801973f258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1373/0x2e00 net/core/dev.c:4120
 #3: ffff88801973f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output net/core/neighbour.c:1490 [inline]
 #3: ffff88801973f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output+0x4d8/0x7e0 net/core/neighbour.c:1470
 #4: ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #4: ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4202 [inline]
 #4: ffff8880237c4098 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311
 #5: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #5: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x290/0x21b0 net/ipv4/ip_output.c:216
 #6: ffffffff8b373840 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1d7/0x2e00 net/core/dev.c:4086
 #7: ffff888014c6f258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #7: ffff888014c6f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:162 [inline]
 #7: ffff888014c6f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:130 [inline]
 #7: ffff888014c6f258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3765 [inline]
 #7: ffff888014c6f258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1373/0x2e00 net/core/dev.c:4120
 #8: ffff888014c6f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output net/core/neighbour.c:1490 [inline]
 #8: ffff888014c6f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output+0x4d8/0x7e0 net/core/neighbour.c:1470

stack backtrace:
CPU: 1 PID: 2665 Comm: syz-executor.1 Not tainted 5.11.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x107/0x163 lib/dump_stack.c:120
 print_deadlock_bug kernel/locking/lockdep.c:2761 [inline]
 check_deadlock kernel/locking/lockdep.c:2804 [inline]
 validate_chain kernel/locking/lockdep.c:3595 [inline]
 __lock_acquire.cold+0x114/0x39e kernel/locking/lockdep.c:4832
 lock_acquire kernel/locking/lockdep.c:5437 [inline]
 lock_acquire+0x29d/0x740 kernel/locking/lockdep.c:5402
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:354 [inline]
 __netif_tx_lock include/linux/netdevice.h:4202 [inline]
 sch_direct_xmit+0x256/0xbd0 net/sched/sch_generic.c:311
 qdisc_restart net/sched/sch_generic.c:376 [inline]
 __qdisc_run+0x4ba/0x15f0 net/sched/sch_generic.c:384
 qdisc_run include/net/pkt_sched.h:136 [inline]
 qdisc_run include/net/pkt_sched.h:128 [inline]
 __dev_xmit_skb net/core/dev.c:3765 [inline]
 __dev_queue_xmit+0x14b9/0x2e00 net/core/dev.c:4120
 neigh_resolve_output net/core/neighbour.c:1490 [inline]
 neigh_resolve_output+0x4d8/0x7e0 net/core/neighbour.c:1470
 neigh_output include/net/neighbour.h:510 [inline]
 ip_finish_output2+0x83d/0x21b0 net/ipv4/ip_output.c:230
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_mc_output+0x212/0x700 net/ipv4/ip_output.c:417
 dst_output include/net/dst.h:441 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:126
 iptunnel_xmit+0x5a3/0x9c0 net/ipv4/ip_tunnel_core.c:82
 ip_tunnel_xmit+0x1116/0x2b00 net/ipv4/ip_tunnel.c:806
 erspan_xmit+0x1109/0x2760 net/ipv4/ip_gre.c:702
 __netdev_start_xmit include/linux/netdevice.h:4764 [inline]
 netdev_start_xmit include/linux/netdevice.h:4778 [inline]
 xmit_one net/core/dev.c:3574 [inline]
 dev_hard_start_xmit+0x1eb/0x920 net/core/dev.c:3590
 sch_direct_xmit+0x2e1/0xbd0 net/sched/sch_generic.c:313
 qdisc_restart net/sched/sch_generic.c:376 [inline]
 __qdisc_run+0x4ba/0x15f0 net/sched/sch_generic.c:384
 qdisc_run include/net/pkt_sched.h:136 [inline]
 qdisc_run include/net/pkt_sched.h:128 [inline]
 __dev_xmit_skb net/core/dev.c:3765 [inline]
 __dev_queue_xmit+0x14b9/0x2e00 net/core/dev.c:4120
 neigh_resolve_output net/core/neighbour.c:1490 [inline]
 neigh_resolve_output+0x4d8/0x7e0 net/core/neighbour.c:1470
 neigh_output include/net/neighbour.h:510 [inline]
 ip_finish_output2+0x83d/0x21b0 net/ipv4/ip_output.c:230
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_mc_output+0x212/0x700 net/ipv4/ip_output.c:417
 dst_output include/net/dst.h:441 [inline]
 ip_local_out net/ipv4/ip_output.c:126 [inline]
 ip_send_skb+0xd4/0x260 net/ipv4/ip_output.c:1567
 udp_send_skb+0x6cd/0x11a0 net/ipv4/udp.c:947
 udp_sendmsg+0x1c76/0x2720 net/ipv4/udp.c:1234
 inet_sendmsg+0x99/0xe0 net/ipv4/af_inet.c:817
 sock_sendmsg_nosec net/socket.c:652 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:672
 ____sys_sendmsg+0x331/0x810 net/socket.c:2345
 ___sys_sendmsg+0xf3/0x170 net/socket.c:2399
 __sys_sendmmsg+0x195/0x470 net/socket.c:2489
 __do_sys_sendmmsg net/socket.c:2518 [inline]
 __se_sys_sendmmsg net/socket.c:2515 [inline]
 __x64_sys_sendmmsg+0x99/0x100 net/socket.c:2515
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x45e219
Code: 0d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 db b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fd1f58e5c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000000000045e219
RDX: 000000000800001d RSI: 0000000020007fc0 RDI: 0000000000000003
RBP: 000000000119bfc8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000119bf8c
R13: 00007ffebd887ccf R14: 00007fd1f58e69c0 R15: 000000000119bf8c

Crashes (35):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Title
ci-upstream-net-kasan-gce	2021/01/23 16:26	net-next	59a49d96	52e37319	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-kasan-gce	2020/10/02 17:38	upstream	472e5b05	4969d6ca	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/09/27 14:33	net	05943249	5dd8aee8	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/08/12 18:37	net	633f5b6b	bc15f7db	.config	log	report
ci-upstream-net-this-kasan-gce	2020/06/30 15:17	net	0574e200	a2cdad9d	.config	log	report
ci-upstream-net-kasan-gce	2020/12/26 05:45	net-next	3db1a3fa	821e0b09	.config	log	report	info
ci-upstream-net-kasan-gce	2020/11/28 16:06	net-next	5c39f26e	76b4dcc7	.config	log	report	info
ci-upstream-bpf-next-kasan-gce	2020/07/07 16:17	bpf-next	cb8e59cc	51095195	.config	log	report
ci-upstream-net-kasan-gce	2020/06/29 07:05	net-next	b08866f4	a2cdad9d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/11 20:04	bpf-next	cb8e59cc	3ab7a05a	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 22:43	bpf-next	cb8e59cc	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 04:08	bpf-next	cb8e59cc	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 03:58	bpf-next	cb8e59cc	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 16:50	bpf-next	cb8e59cc	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 15:17	bpf-next	cb8e59cc	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/04 04:04	bpf-next	e8224bfe	a5ce5de0	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 20:18	bpf-next	9a25c1df	52fd7b7d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/31 22:56	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 15:21	bpf-next	551f08b1	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 12:53	bpf-next	551f08b1	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 03:21	bpf-next	551f08b1	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 22:03	bpf-next	551f08b1	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 15:30	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 12:17	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 11:08	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 22:19	bpf-next	c41f9b73	ec153193	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 04:32	bpf-next	c409dc81	9072c126	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 07:43	bpf-next	a152b859	11284182	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/23 10:13	bpf-next	a5dfaa2a	9682898d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/22 12:37	bpf-next	29ae90d2	5afa2ddd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/18 05:16	bpf-next	96586dd9	37bccd4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 17:15	bpf-next	3f2eaebb	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 15:50	bpf-next	3f2eaebb	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 23:01	bpf-next	3f2eaebb	b8bb8e5f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 00:53	bpf-next	3f2eaebb	03d97a1b	.config	log	report