possible deadlock in sch_direct

possible deadlock in sch_direct_xmit
Status: fixed on 2020/04/17 19:57
Reported-by: syzbot+29cc278357da941e304e@syzkaller.appspotmail.com
Fix commit: 323ebb61e32b net: use listified RX for handling GRO_NORMAL skbs
First crash: 1587d, last: 905d

Cause bisection: introduced by (bisect log) :
commit c84bed440e4e11a973e8c0254d0dfaccfca41fb0
Author: Xin Long <lucien.xin@gmail.com>
Date: Sun Oct 1 14:00:56 2017 +0000

ip_gre: erspan device should keep dst

Crash: possible deadlock in sch_direct_xmit (log)
Repro: C syz .config

Fix bisection: fixed by (bisect log) :
commit 323ebb61e32b478e2432c5a3cbf9e2ca678a9609
Author: Edward Cree <ecree@solarflare.com>
Date: Tue Aug 6 13:53:55 2019 +0000

net: use listified RX for handling GRO_NORMAL skbs

similar bugs (6):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
android-44	possible deadlock in sch_direct_xmit	C			240	897d	1134d	0/2	public: reported C repro on 2019/04/11 08:44
upstream	possible deadlock in sch_direct_xmit (2)	C	error	unreliable	90	7d23h	750d	0/22	upstream: reported C repro on 2020/04/29 00:59
linux-4.19	possible deadlock in sch_direct_xmit (2)	C		error	7	130d	235d	0/1	upstream: reported C repro on 2021/09/26 01:30
linux-4.14	possible deadlock in sch_direct_xmit				1	1081d	1081d	0/1	auto-closed as invalid on 2019/10/25 08:40
linux-4.14	possible deadlock in sch_direct_xmit (2)				1	915d	915d	0/1	auto-closed as invalid on 2020/03/15 19:58
linux-4.19	possible deadlock in sch_direct_xmit				1	1083d	1083d	0/1	auto-closed as invalid on 2019/10/25 08:50

Sample crash report:

============================================
WARNING: possible recursive locking detected
4.15.0-rc7+ #260 Not tainted
--------------------------------------------
syzkaller218108/3657 is trying to acquire lock:
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185

but task is already holding lock:
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

10 locks held by syzkaller218108/3657:
 #0:  (&tfile->napi_mutex){+.+.}, at: [<00000000129cc53a>] tun_get_user+0xe5a/0x3710 drivers/net/tun.c:1636
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] arch_static_branch arch/x86/include/asm/jump_label.h:36 [inline]
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] static_key_false include/linux/jump_label.h:142 [inline]
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] netif_receive_skb_internal+0xa2/0x670 net/core/dev.c:4585
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] spin_trylock include/linux/spinlock.h:320 [inline]
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] icmp_xmit_lock net/ipv4/icmp.c:219 [inline]
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] icmp_send+0x75e/0x19d0 net/ipv4/icmp.c:668
 #3:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #3:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] ip_finish_output2+0x2b6/0x1500 net/ipv4/ip_output.c:213
 #4:  (rcu_read_lock_bh){....}, at: [<000000007d2deb0a>] __dev_queue_xmit+0x294/0x2920 net/core/dev.c:3434
 #5:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<00000000d2573a7c>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185
 #7:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #7:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] ip_finish_output2+0x2b6/0x1500 net/ipv4/ip_output.c:213
 #8:  (rcu_read_lock_bh){....}, at: [<000000007d2deb0a>] __dev_queue_xmit+0x294/0x2920 net/core/dev.c:3434
 #9:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<00000000d2573a7c>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3533

stack backtrace:
CPU: 0 PID: 3657 Comm: syzkaller218108 Not tainted 4.15.0-rc7+ #260
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 print_deadlock_bug kernel/locking/lockdep.c:1756 [inline]
 check_deadlock kernel/locking/lockdep.c:1800 [inline]
 validate_chain kernel/locking/lockdep.c:2396 [inline]
 __lock_acquire+0xe8f/0x3e00 kernel/locking/lockdep.c:3426
 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
 spin_lock include/linux/spinlock.h:310 [inline]
 __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185
 __dev_xmit_skb net/core/dev.c:3201 [inline]
 __dev_queue_xmit+0x1ce2/0x2920 net/core/dev.c:3468
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x8d2/0x1500 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd10 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x277/0x1360 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:460 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 iptunnel_xmit+0x556/0x810 net/ipv4/ip_tunnel_core.c:91
 ip_tunnel_xmit+0x1780/0x3650 net/ipv4/ip_tunnel.c:786
 __gre_xmit+0x546/0x8b0 net/ipv4/ip_gre.c:436
 erspan_xmit+0x409/0x13b0 net/ipv4/ip_gre.c:742
 __netdev_start_xmit include/linux/netdevice.h:4042 [inline]
 netdev_start_xmit include/linux/netdevice.h:4051 [inline]
 xmit_one net/core/dev.c:3003 [inline]
 dev_hard_start_xmit+0x24e/0xac0 net/core/dev.c:3019
 sch_direct_xmit+0x31d/0x6d0 net/sched/sch_generic.c:187
 __dev_xmit_skb net/core/dev.c:3201 [inline]
 __dev_queue_xmit+0x1ce2/0x2920 net/core/dev.c:3468
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x8d2/0x1500 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd10 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x277/0x1360 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:460 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1414
 ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1434
 icmp_push_reply+0x395/0x4f0 net/ipv4/icmp.c:394
 icmp_send+0x1148/0x19d0 net/ipv4/icmp.c:741
 ip_options_compile+0xc21/0x1a50 net/ipv4/ip_options.c:472
 ip_rcv_options net/ipv4/ip_input.c:284 [inline]
 ip_rcv_finish+0x80f/0x1e30 net/ipv4/ip_input.c:365
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ip_rcv+0xc5a/0x1840 net/ipv4/ip_input.c:493
 __netif_receive_skb_core+0x1a41/0x3460 net/core/dev.c:4473
 __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4538
 netif_receive_skb_internal+0x10b/0x670 net/core/dev.c:4611
 napi_frags_finish net/core/dev.c:5052 [inline]
 napi_gro_frags+0x58a/0xaf0 net/core/dev.c:5125
 tun_get_user+0x262e/0x3710 drivers/net/tun.c:1757
 tun_chr_write_iter+0xb9/0x160 drivers/net/tun.c:1800
 call_write_iter include/linux/fs.h:1772 [inline]
 do_iter_readv_writev+0x525/0x7f0 fs/read_write.c:653
 do_iter_write+0x154/0x540 fs/read_write.c:932
 vfs_writev+0x18a/0x340 fs/read_write.c:977
 do_writev+0xfc/0x2a0 fs/read_write.c:1012
 SYSC_writev fs/read_write.c:1085 [inline]
 SyS_writev+0x27/0x30 fs/read_write.c:1082
 entry_SYSCALL_64_fastpath+0x23/0x9a
RIP: 0033:0x444f50
RSP: 002b:00007ffc71764ad8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
RAX: ffffffffffffffda RBX: 00000000004a6852 RCX: 0000000000444f50
RDX: 0000000000000001 RSI: 00007ffc71764b10 RDI: 0000000000000003
RBP: 00007ffc71764c08 R08: 0000000000000023 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc71764c08
R13: 0000000000402520 R14: 0000000000000000 R15: 0000000000000000
syzkaller218108 (3657) used greatest stack depth: 11920 b

Crashes (1548):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro
ci-upstream-kasan-gce	2018/01/13 18:16	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce	2018/01/13 17:44	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce	2018/01/13 16:50	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce	2018/01/13 16:39	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce-386	2018/01/13 17:50	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce-386	2018/01/13 17:19	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce-386	2018/01/13 16:58	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce-386	2018/01/13 16:47	upstream	c92a9a461dff	c9e7aeae	.config	log	report	syz	C
ci-upstream-net-kasan-gce	2018/01/13 17:22	net-next	6bd39bc3da0f	c9e7aeae	.config	log	report	syz	C
ci-upstream-net-kasan-gce	2018/01/13 17:10	net-next	6bd39bc3da0f	c9e7aeae	.config	log	report	syz	C
ci-upstream-net-kasan-gce	2018/01/13 17:00	net-next	6bd39bc3da0f	c9e7aeae	.config	log	report	syz	C
ci-upstream-net-kasan-gce	2018/01/13 16:37	net-next	6bd39bc3da0f	c9e7aeae	.config	log	report	syz	C
ci-upstream-next-kasan-gce	2018/01/13 18:51	linux-next	3e53c7415294	c9e7aeae	.config	log	report	syz	C
ci-upstream-mmots-kasan-gce	2018/01/13 18:06	mmots	ce3c209f6733	c9e7aeae	.config	log	report	syz	C
ci-upstream-mmots-kasan-gce	2018/01/13 17:44	mmots	ce3c209f6733	c9e7aeae	.config	log	report	syz	C
ci-upstream-next-kasan-gce	2018/01/13 17:15	linux-next	3e53c7415294	c9e7aeae	.config	log	report	syz	C
ci-upstream-mmots-kasan-gce	2018/01/13 17:14	mmots	ce3c209f6733	c9e7aeae	.config	log	report	syz	C
ci-upstream-next-kasan-gce	2018/01/13 16:54	linux-next	3e53c7415294	c9e7aeae	.config	log	report	syz	C
ci-upstream-mmots-kasan-gce	2018/01/13 16:54	mmots	ce3c209f6733	c9e7aeae	.config	log	report	syz	C
ci-upstream-next-kasan-gce	2018/01/13 16:44	linux-next	3e53c7415294	c9e7aeae	.config	log	report	syz	C
ci-upstream-kasan-gce	2019/11/26 09:48	upstream	0be0ee71816b	f746151a	.config	log	report
ci-upstream-kasan-gce	2018/03/31 07:09	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce	2018/03/31 05:59	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce	2018/03/31 03:44	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce	2018/03/31 00:40	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce-root	2018/03/30 18:37	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce	2018/03/30 16:43	upstream	c2a9838452a4	d47f0ed6	.config	log	report
ci-upstream-kasan-gce-root	2018/03/30 15:29	upstream	c2a9838452a4	d47f0ed6	.config	log	report
ci-upstream-kasan-gce	2018/03/30 12:28	upstream	c2a9838452a4	d47f0ed6	.config	log	report
ci-upstream-kasan-gce	2018/03/30 10:21	upstream	c2a9838452a4	d47f0ed6	.config	log	report
ci-upstream-kasan-gce	2018/03/29 11:29	upstream	0b412605ef5f	d47f0ed6	.config	log	report
ci-upstream-kasan-gce	2018/03/24 08:46	upstream	99fec39e7725	2e9d9054	.config	log	report
ci-upstream-kasan-gce	2018/03/23 16:45	upstream	f36b7534b833	2e9d9054	.config	log	report
ci-upstream-kasan-gce	2018/02/27 00:35	upstream	4a3928c6f8a5	b370d4a7	.config	log	report
ci-upstream-kasan-gce	2018/02/26 10:31	upstream	c89be5242607	9fe8aa42	.config	log	report
ci-upstream-kasan-gce	2018/02/26 00:57	upstream	c89be5242607	9fe8aa42	.config	log	report
ci-upstream-kasan-gce	2018/02/25 13:18	upstream	3664ce2d9309	5c1e0207	.config	log	report
ci-upstream-kasan-gce	2018/02/24 23:30	upstream	9cb9c07d6b0c	5c1e0207	.config	log	report
ci-upstream-kasan-gce	2018/02/24 11:38	upstream	938e1426e262	5c1e0207	.config	log	report
ci-upstream-kasan-gce	2018/02/24 09:40	upstream	938e1426e262	5c1e0207	.config	log	report
ci-upstream-kasan-gce	2018/02/23 22:35	upstream	0f9da844d877	33464158	.config	log	report
ci-upstream-kasan-gce-386	2018/03/30 20:55	upstream	9dd2326890d8	8fbce0e4	.config	log	report
ci-upstream-kasan-gce-386	2018/03/15 20:21	upstream	0aa3fdb8b3a6	08dacaa0	.config	log	report
ci-upstream-kasan-gce-386	2018/03/08 16:06	upstream	1b88accf6a65	acd0caa5	.config	log	report
ci-upstream-kasan-gce-386	2018/03/06 13:22	upstream	094b58e1040a	aef0b792	.config	log	report
ci-upstream-net-kasan-gce	2019/09/21 06:17	net-next	b41dae061bbd	d96e88f3	.config	log	report
ci-upstream-net-kasan-gce	2018/04/03 12:06	net-next	159f02977b2f	676bd07e	.config	log	report
ci-upstream-net-kasan-gce	2018/04/03 10:46	net-next	159f02977b2f	676bd07e	.config	log	report
ci-upstream-net-kasan-gce	2018/04/02 21:33	net-next	159f02977b2f	676bd07e	.config	log	report
ci-upstream-net-kasan-gce	2018/04/01 12:52	net-next	06b19fe9a6df	0a78e248	.config	log	report
ci-upstream-net-kasan-gce	2018/04/01 03:13	net-next	8bde261e5352	0174c6c8	.config	log	report
ci-upstream-net-kasan-gce	2018/03/28 23:43	net-next	5d22d47b9ed9	bf5e585c	.config	log	report
ci-upstream-net-kasan-gce	2018/03/27 23:17	net-next	5d22d47b9ed9	bf5e585c	.config	log	report
ci-upstream-net-kasan-gce	2018/03/27 19:04	net-next	34fd03b9e6a6	bf5e585c	.config	log	report
ci-upstream-net-kasan-gce	2018/03/25 17:38	net-next	94cb54924092	e033c1f1	.config	log	report
ci-upstream-net-kasan-gce	2018/03/24 22:35	net-next	94cb54924092	2e9d9054	.config	log	report
ci-upstream-net-kasan-gce	2018/03/24 15:26	net-next	94cb54924092	2e9d9054	.config	log	report
ci-upstream-net-kasan-gce	2018/03/24 00:25	net-next	f452518c982e	2e9d9054	.config	log	report
ci-upstream-net-kasan-gce	2018/03/22 16:47	net-next	aa65f6365405	2e9d9054	.config	log	report
ci-upstream-net-kasan-gce	2018/03/22 07:14	net-next	454bfe97837a	95c88d7a	.config	log	report
ci-upstream-net-kasan-gce	2018/03/18 14:32	net-next	76f38f1f3cf8	08dacaa0	.config	log	report
ci-upstream-net-kasan-gce	2018/03/16 18:09	net-next	0aee4c259849	08dacaa0	.config	log	report
ci-upstream-net-kasan-gce	2018/03/16 05:17	net-next	80d9f3a0fdb8	08dacaa0	.config	log	report
ci-upstream-net-kasan-gce	2018/03/15 02:55	net-next	c292566a7779	08dacaa0	.config	log	report
ci-upstream-net-kasan-gce	2018/03/13 13:32	net-next	9ba32046fc2d	08dacaa0	.config	log	report
ci-upstream-net-kasan-gce	2018/03/13 02:42	net-next	129cf5f7f196	f505ca4b	.config	log	report
ci-upstream-net-kasan-gce	2018/03/12 22:12	net-next	129cf5f7f196	f505ca4b	.config	log	report
ci-upstream-net-kasan-gce	2018/03/09 10:41	net-next	fd372a7a9e5e	36d1c454	.config	log	report
ci-upstream-net-kasan-gce	2018/03/09 09:26	net-next	fd372a7a9e5e	36d1c454	.config	log	report
ci-upstream-net-kasan-gce	2018/03/08 04:08	net-next	a366e300ae9f	d50edb7e	.config	log	report
ci-upstream-net-kasan-gce	2018/03/05 00:36	net-next	efab163bbc19	2c6f473e	.config	log	report
ci-upstream-net-kasan-gce	2018/03/01 23:03	net-next	f1c02cfb7b30	2c6f473e	.config	log	report
ci-upstream-net-kasan-gce	2018/02/28 20:38	net-next	fb66cb077560	05b5a32c	.config	log	report
ci-upstream-net-kasan-gce	2018/02/27 11:15	net-next	3808b51911fe	05b5a32c	.config	log	report
ci-upstream-net-kasan-gce	2018/02/27 07:44	net-next	ba6056a41cb0	b370d4a7	.config	log	report
ci-upstream-net-kasan-gce	2018/02/27 03:43	net-next	ba6056a41cb0	b370d4a7	.config	log	report
ci-upstream-net-kasan-gce	2018/02/26 17:11	net-next	ba6056a41cb0	9fe8aa42	.config	log	report