possible deadlock in sch_direct

possible deadlock in sch_direct_xmit (2)
Status: upstream: reported on 2020/04/29 00:59
Reported-by: syzbot+e18ac85757292b7baf96@syzkaller.appspotmail.com
First crash: 421d, last: 22d

similar bugs (5):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
android-44	possible deadlock in sch_direct_xmit	C			240	563d	800d	0/2	public: reported C repro on 2019/04/11 08:44
upstream	possible deadlock in sch_direct_xmit	C	done	done	1548	571d	1250d	17/22	fixed on 2020/04/17 19:57
linux-4.14	possible deadlock in sch_direct_xmit				1	748d	747d	0/1	auto-closed as invalid on 2019/10/25 08:40
linux-4.14	possible deadlock in sch_direct_xmit (2)				1	581d	581d	0/1	auto-closed as invalid on 2020/03/15 19:58
linux-4.19	possible deadlock in sch_direct_xmit				1	749d	749d	0/1	auto-closed as invalid on 2019/10/25 08:50

Sample crash report:

============================================
WARNING: possible recursive locking detected
5.13.0-rc3-syzkaller #0 Not tainted
--------------------------------------------
systemd-udevd/4860 is trying to acquire lock:
ffff888020737398 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff888020737398 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
ffff888020737398 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x30f/0xba0 net/sched/sch_generic.c:334

but task is already holding lock:
ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x30f/0xba0 net/sched/sch_generic.c:334

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

11 locks held by systemd-udevd/4860:
 #0: ffffffff8c144de8 (kernfs_mutex){+.+.}-{3:3}, at: kernfs_getlink fs/kernfs/symlink.c:119 [inline]
 #0: ffffffff8c144de8 (kernfs_mutex){+.+.}-{3:3}, at: kernfs_iop_get_link fs/kernfs/symlink.c:138 [inline]
 #0: ffffffff8c144de8 (kernfs_mutex){+.+.}-{3:3}, at: kernfs_iop_get_link+0xf1/0x6e0 fs/kernfs/symlink.c:126
 #1: ffffc90000dc0d70 ((&in_dev->mr_ifc_timer)){+.-.}-{0:0}, at: lockdep_copy_map include/linux/lockdep.h:35 [inline]
 #1: ffffc90000dc0d70 ((&in_dev->mr_ifc_timer)){+.-.}-{0:0}, at: call_timer_fn+0xd5/0x6b0 kernel/time/timer.c:1421
 #2: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #2: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x290/0x21f0 net/ipv4/ip_output.c:216
 #3: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1da/0x3130 net/core/dev.c:4180
 #4: ffff88803115f258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #4: ffff88803115f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:163 [inline]
 #4: ffff88803115f258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:130 [inline]
 #4: ffff88803115f258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3857 [inline]
 #4: ffff88803115f258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1311/0x3130 net/core/dev.c:4214
 #5: ffff88803115f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output net/core/neighbour.c:1495 [inline]
 #5: ffff88803115f148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_resolve_output+0x50e/0x820 net/core/neighbour.c:1475
 #6: ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #6: ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4379 [inline]
 #6: ffff88802eaf5898 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x30f/0xba0 net/sched/sch_generic.c:334
 #7: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #7: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: ip_finish_output2+0x290/0x21f0 net/ipv4/ip_output.c:216
 #8: ffffffff8bf792c0 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1da/0x3130 net/core/dev.c:4180
 #9: ffff888067866258 (&sch->seqlock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:364 [inline]
 #9: ffff888067866258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:163 [inline]
 #9: ffff888067866258 (&sch->seqlock){+...}-{2:2}, at: qdisc_run include/net/pkt_sched.h:130 [inline]
 #9: ffff888067866258 (&sch->seqlock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3857 [inline]
 #9: ffff888067866258 (&sch->seqlock){+...}-{2:2}, at: __dev_queue_xmit+0x1311/0x3130 net/core/dev.c:4214
 #10: ffff888067866148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_hh_output include/net/neighbour.h:499 [inline]
 #10: ffff888067866148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: neigh_output include/net/neighbour.h:508 [inline]
 #10: ffff888067866148 (dev->qdisc_running_key ?: &qdisc_running_key){+...}-{0:0}, at: ip_finish_output2+0xeec/0x21f0 net/ipv4/ip_output.c:230

stack backtrace:
CPU: 1 PID: 4860 Comm: systemd-udevd Not tainted 5.13.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x141/0x1d7 lib/dump_stack.c:120
 print_deadlock_bug kernel/locking/lockdep.c:2831 [inline]
 check_deadlock kernel/locking/lockdep.c:2874 [inline]
 validate_chain kernel/locking/lockdep.c:3663 [inline]
 __lock_acquire.cold+0x22f/0x3b4 kernel/locking/lockdep.c:4902
 lock_acquire kernel/locking/lockdep.c:5512 [inline]
 lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5477
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:354 [inline]
 __netif_tx_lock include/linux/netdevice.h:4379 [inline]
 sch_direct_xmit+0x30f/0xba0 net/sched/sch_generic.c:334
 qdisc_restart net/sched/sch_generic.c:401 [inline]
 __qdisc_run+0x4bc/0x1680 net/sched/sch_generic.c:409
 qdisc_run include/net/pkt_sched.h:131 [inline]
 qdisc_run include/net/pkt_sched.h:128 [inline]
 __dev_xmit_skb net/core/dev.c:3857 [inline]
 __dev_queue_xmit+0x13ca/0x3130 net/core/dev.c:4214
 neigh_hh_output include/net/neighbour.h:499 [inline]
 neigh_output include/net/neighbour.h:508 [inline]
 ip_finish_output2+0xeec/0x21f0 net/ipv4/ip_output.c:230
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_output+0x196/0x310 net/ipv4/ip_output.c:432
 dst_output include/net/dst.h:448 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:126
 iptunnel_xmit+0x5a3/0x9c0 net/ipv4/ip_tunnel_core.c:82
 ip_tunnel_xmit+0x1116/0x2b10 net/ipv4/ip_tunnel.c:804
 erspan_xmit+0x12c7/0x2970 net/ipv4/ip_gre.c:702
 __netdev_start_xmit include/linux/netdevice.h:4944 [inline]
 netdev_start_xmit include/linux/netdevice.h:4958 [inline]
 xmit_one net/core/dev.c:3654 [inline]
 dev_hard_start_xmit+0x1eb/0x920 net/core/dev.c:3670
 sch_direct_xmit+0x19f/0xba0 net/sched/sch_generic.c:336
 qdisc_restart net/sched/sch_generic.c:401 [inline]
 __qdisc_run+0x4bc/0x1680 net/sched/sch_generic.c:409
 qdisc_run include/net/pkt_sched.h:131 [inline]
 qdisc_run include/net/pkt_sched.h:128 [inline]
 __dev_xmit_skb net/core/dev.c:3857 [inline]
 __dev_queue_xmit+0x13ca/0x3130 net/core/dev.c:4214
 neigh_resolve_output net/core/neighbour.c:1495 [inline]
 neigh_resolve_output+0x50e/0x820 net/core/neighbour.c:1475
 neigh_output include/net/neighbour.h:510 [inline]
 ip_finish_output2+0x83d/0x21f0 net/ipv4/ip_output.c:230
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x396/0x640 net/ipv4/ip_output.c:290
 ip_finish_output+0x35/0x200 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_output+0x196/0x310 net/ipv4/ip_output.c:432
 dst_output include/net/dst.h:448 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:126
 igmpv3_send_cr net/ipv4/igmp.c:719 [inline]
 igmp_ifc_timer_expire+0x7a7/0xf30 net/ipv4/igmp.c:807
 call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1431
 expire_timers kernel/time/timer.c:1476 [inline]
 __run_timers.part.0+0x67c/0xa50 kernel/time/timer.c:1745
 __run_timers kernel/time/timer.c:1726 [inline]
 run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1758
 __do_softirq+0x29b/0x9f6 kernel/softirq.c:559
 invoke_softirq kernel/softirq.c:433 [inline]
 __irq_exit_rcu+0x136/0x200 kernel/softirq.c:637
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:647
RIP: 0010:memcpy_erms+0x8/0x10 arch/x86/lib/memcpy_64.S:56
Code: cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 a4 <c3> 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35
RSP: 0018:ffffc9000154fd28 EFLAGS: 00000202
RAX: ffff88802d6fe012 RBX: 0000000000000006 RCX: 0000000000000000
RDX: 0000000000000005 RSI: ffffffff89c131c5 RDI: ffff88802d6fe017
RBP: ffff8880122ec000 R08: 0000000000000001 R09: ffff88802d6fe016
R10: ffffed1005adfc02 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff888018e2e3a0 R14: ffff88802d6fe00c R15: ffff888018e2e3d8
 memcpy include/linux/fortify-string.h:191 [inline]
 kernfs_get_target_path fs/kernfs/symlink.c:102 [inline]
 kernfs_getlink fs/kernfs/symlink.c:120 [inline]
 kernfs_iop_get_link fs/kernfs/symlink.c:138 [inline]
 kernfs_iop_get_link+0x40d/0x6e0 fs/kernfs/symlink.c:126
 vfs_readlink+0x1d7/0x390 fs/namei.c:4795
 do_readlinkat+0x27e/0x2f0 fs/stat.c:437
 __do_sys_readlinkat fs/stat.c:452 [inline]
 __se_sys_readlinkat fs/stat.c:449 [inline]
 __x64_sys_readlinkat+0x93/0xf0 fs/stat.c:449
 do_syscall_64+0x3a/0xb0 arch/x86/entry/common.c:47
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7fb735d0c0ba
Code: 48 8b 0d e1 bd 2b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 0b 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ae bd 2b 00 f7 d8 64 89 01 48
RSP: 002b:00007fff9e839388 EFLAGS: 00000206 ORIG_RAX: 000000000000010b
RAX: ffffffffffffffda RBX: 000055f5dee99400 RCX: 00007fb735d0c0ba
RDX: 000055f5dee99400 RSI: 00007fff9e839410 RDI: 00000000ffffff9c
RBP: 0000000000000064 R08: 00007fb735fc8bc8 R09: 0000000000000070
R10: 0000000000000063 R11: 0000000000000206 R12: 00007fff9e839410
R13: 00000000ffffff9c R14: 00007fff9e8393e0 R15: 0000000000000063

Crashes (37):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Title
ci-upstream-net-kasan-gce	2021/05/28 08:51	net-next	af9207ad	858ea628	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/03/28 20:40	net-next	a1281601	a8529b82	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-net-kasan-gce	2021/01/23 16:26	net-next	59a49d96	52e37319	.config	log	report	info	possible deadlock in sch_direct_xmit
ci-upstream-kasan-gce	2020/10/02 17:38	upstream	472e5b05	4969d6ca	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/09/27 14:33	net	05943249	5dd8aee8	.config	log	report	info
ci-upstream-net-this-kasan-gce	2020/08/12 18:37	net	633f5b6b	bc15f7db	.config	log	report
ci-upstream-net-this-kasan-gce	2020/06/30 15:17	net	0574e200	a2cdad9d	.config	log	report
ci-upstream-net-kasan-gce	2020/12/26 05:45	net-next	3db1a3fa	821e0b09	.config	log	report	info
ci-upstream-net-kasan-gce	2020/11/28 16:06	net-next	5c39f26e	76b4dcc7	.config	log	report	info
ci-upstream-bpf-next-kasan-gce	2020/07/07 16:17	bpf-next	cb8e59cc	51095195	.config	log	report
ci-upstream-net-kasan-gce	2020/06/29 07:05	net-next	b08866f4	a2cdad9d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/11 20:04	bpf-next	cb8e59cc	3ab7a05a	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 22:43	bpf-next	cb8e59cc	e6b89e4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 04:08	bpf-next	cb8e59cc	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/06 03:58	bpf-next	cb8e59cc	f243c88f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 16:50	bpf-next	cb8e59cc	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/05 15:17	bpf-next	cb8e59cc	2420d1bc	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/04 04:04	bpf-next	e8224bfe	a5ce5de0	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/06/02 20:18	bpf-next	9a25c1df	52fd7b7d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/31 22:56	bpf-next	551f08b1	a0331e89	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 15:21	bpf-next	551f08b1	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 12:53	bpf-next	551f08b1	954bd312	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/30 03:21	bpf-next	551f08b1	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 22:03	bpf-next	551f08b1	3905eaae	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 15:30	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 12:17	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/29 11:08	bpf-next	9b185fa4	d19ed305	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 22:19	bpf-next	c41f9b73	ec153193	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/27 04:32	bpf-next	c409dc81	9072c126	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/25 07:43	bpf-next	a152b859	11284182	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/23 10:13	bpf-next	a5dfaa2a	9682898d	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/22 12:37	bpf-next	29ae90d2	5afa2ddd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/05/18 05:16	bpf-next	96586dd9	37bccd4e	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 17:15	bpf-next	3f2eaebb	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/26 15:50	bpf-next	3f2eaebb	99b258dd	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 23:01	bpf-next	3f2eaebb	b8bb8e5f	.config	log	report
ci-upstream-bpf-next-kasan-gce	2020/04/25 00:53	bpf-next	3f2eaebb	03d97a1b	.config	log	report