syzbot


possible deadlock in sch_direct_xmit

Status: fixed on 2020/04/17 19:57
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+29cc278357da941e304e@syzkaller.appspotmail.com
Fix commit: 323ebb61e32b net: use listified RX for handling GRO_NORMAL skbs
First crash: 2458d, last: 1777d
Cause bisection: introduced by (bisect log) :
commit c84bed440e4e11a973e8c0254d0dfaccfca41fb0
Author: Xin Long <lucien.xin@gmail.com>
Date: Sun Oct 1 14:00:56 2017 +0000

  ip_gre: erspan device should keep dst

Crash: possible deadlock in sch_direct_xmit (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit 323ebb61e32b478e2432c5a3cbf9e2ca678a9609
Author: Edward Cree <ecree@solarflare.com>
Date: Tue Aug 6 13:53:55 2019 +0000

  net: use listified RX for handling GRO_NORMAL skbs

  
Discussions (3)
Title Replies (including bot) Last reply
possible deadlock in sch_direct_xmit 0 (3) 2019/12/28 03:16
Reminder: 99 open syzbot bugs in net subsystem 14 (14) 2019/07/31 15:13
Reminder: 94 open syzbot bugs in net subsystem 1 (1) 2019/06/25 05:48
Similar bugs (12)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in sch_direct_xmit (2) origin:lts-only C done 17 5d01h 271d 0/3 upstream: reported C repro on 2024/01/09 18:28
android-44 possible deadlock in sch_direct_xmit C 240 1769d 2006d 0/2 public: reported C repro on 2019/04/11 08:44
upstream possible deadlock in sch_direct_xmit (2) net C done unreliable 109 448d 1622d 0/28 auto-obsoleted due to no activity on 2024/01/14 06:05
linux-4.19 possible deadlock in sch_direct_xmit (2) C error 15 590d 1107d 0/1 upstream: reported C repro on 2021/09/26 01:30
linux-5.15 possible deadlock in sch_direct_xmit (2) origin:lts-only C error 12 35d 227d 0/3 upstream: reported C repro on 2024/02/22 19:25
linux-4.14 possible deadlock in sch_direct_xmit 1 1953d 1953d 0/1 auto-closed as invalid on 2019/10/25 08:40
upstream possible deadlock in sch_direct_xmit (4) net 1 162d 162d 25/28 fixed on 2024/06/05 13:52
linux-4.14 possible deadlock in sch_direct_xmit (2) 1 1786d 1786d 0/1 auto-closed as invalid on 2020/03/15 19:58
linux-4.19 possible deadlock in sch_direct_xmit 1 1954d 1954d 0/1 auto-closed as invalid on 2019/10/25 08:50
linux-5.15 possible deadlock in sch_direct_xmit 1 515d 515d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:09
linux-6.1 possible deadlock in sch_direct_xmit 2 522d 561d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:10
upstream possible deadlock in sch_direct_xmit (3) net 1 238d 238d 25/28 fixed on 2024/04/10 16:40
Fix bisection attempts (2)
Created Duration User Patch Repo Result
2019/12/27 03:08 1d00h bisect fix upstream OK (1) job log
2019/08/01 04:22 29m bisect fix upstream OK (0) job log log

Sample crash report:
============================================
WARNING: possible recursive locking detected
4.15.0-rc7+ #260 Not tainted
--------------------------------------------
syzkaller218108/3657 is trying to acquire lock:
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185

but task is already holding lock:
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(_xmit_ETHER#2);
  lock(_xmit_ETHER#2);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

10 locks held by syzkaller218108/3657:
 #0:  (&tfile->napi_mutex){+.+.}, at: [<00000000129cc53a>] tun_get_user+0xe5a/0x3710 drivers/net/tun.c:1636
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] arch_static_branch arch/x86/include/asm/jump_label.h:36 [inline]
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] static_key_false include/linux/jump_label.h:142 [inline]
 #1:  (rcu_read_lock){....}, at: [<000000007677140a>] netif_receive_skb_internal+0xa2/0x670 net/core/dev.c:4585
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] spin_trylock include/linux/spinlock.h:320 [inline]
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] icmp_xmit_lock net/ipv4/icmp.c:219 [inline]
 #2:  (k-slock-AF_INET){+...}, at: [<0000000095e11f1b>] icmp_send+0x75e/0x19d0 net/ipv4/icmp.c:668
 #3:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #3:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] ip_finish_output2+0x2b6/0x1500 net/ipv4/ip_output.c:213
 #4:  (rcu_read_lock_bh){....}, at: [<000000007d2deb0a>] __dev_queue_xmit+0x294/0x2920 net/core/dev.c:3434
 #5:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<00000000d2573a7c>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] spin_lock include/linux/spinlock.h:310 [inline]
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 #6:  (_xmit_ETHER#2){+.-.}, at: [<000000005c439601>] sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185
 #7:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] lwtunnel_xmit_redirect include/net/lwtunnel.h:92 [inline]
 #7:  (rcu_read_lock_bh){....}, at: [<0000000016d9d0cb>] ip_finish_output2+0x2b6/0x1500 net/ipv4/ip_output.c:213
 #8:  (rcu_read_lock_bh){....}, at: [<000000007d2deb0a>] __dev_queue_xmit+0x294/0x2920 net/core/dev.c:3434
 #9:  (dev->qdisc_running_key ?: &qdisc_running_key){+...}, at: [<00000000d2573a7c>] dev_queue_xmit+0x17/0x20 net/core/dev.c:3533

stack backtrace:
CPU: 0 PID: 3657 Comm: syzkaller218108 Not tainted 4.15.0-rc7+ #260
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 print_deadlock_bug kernel/locking/lockdep.c:1756 [inline]
 check_deadlock kernel/locking/lockdep.c:1800 [inline]
 validate_chain kernel/locking/lockdep.c:2396 [inline]
 __lock_acquire+0xe8f/0x3e00 kernel/locking/lockdep.c:3426
 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
 spin_lock include/linux/spinlock.h:310 [inline]
 __netif_tx_lock include/linux/netdevice.h:3537 [inline]
 sch_direct_xmit+0x280/0x6d0 net/sched/sch_generic.c:185
 __dev_xmit_skb net/core/dev.c:3201 [inline]
 __dev_queue_xmit+0x1ce2/0x2920 net/core/dev.c:3468
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x8d2/0x1500 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd10 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x277/0x1360 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:460 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 iptunnel_xmit+0x556/0x810 net/ipv4/ip_tunnel_core.c:91
 ip_tunnel_xmit+0x1780/0x3650 net/ipv4/ip_tunnel.c:786
 __gre_xmit+0x546/0x8b0 net/ipv4/ip_gre.c:436
 erspan_xmit+0x409/0x13b0 net/ipv4/ip_gre.c:742
 __netdev_start_xmit include/linux/netdevice.h:4042 [inline]
 netdev_start_xmit include/linux/netdevice.h:4051 [inline]
 xmit_one net/core/dev.c:3003 [inline]
 dev_hard_start_xmit+0x24e/0xac0 net/core/dev.c:3019
 sch_direct_xmit+0x31d/0x6d0 net/sched/sch_generic.c:187
 __dev_xmit_skb net/core/dev.c:3201 [inline]
 __dev_queue_xmit+0x1ce2/0x2920 net/core/dev.c:3468
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3533
 neigh_resolve_output+0x5e2/0xa00 net/core/neighbour.c:1350
 neigh_output include/net/neighbour.h:482 [inline]
 ip_finish_output2+0x8d2/0x1500 net/ipv4/ip_output.c:229
 ip_finish_output+0x864/0xd10 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x277/0x1360 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:460 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 ip_send_skb+0x3c/0xc0 net/ipv4/ip_output.c:1414
 ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1434
 icmp_push_reply+0x395/0x4f0 net/ipv4/icmp.c:394
 icmp_send+0x1148/0x19d0 net/ipv4/icmp.c:741
 ip_options_compile+0xc21/0x1a50 net/ipv4/ip_options.c:472
 ip_rcv_options net/ipv4/ip_input.c:284 [inline]
 ip_rcv_finish+0x80f/0x1e30 net/ipv4/ip_input.c:365
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ip_rcv+0xc5a/0x1840 net/ipv4/ip_input.c:493
 __netif_receive_skb_core+0x1a41/0x3460 net/core/dev.c:4473
 __netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4538
 netif_receive_skb_internal+0x10b/0x670 net/core/dev.c:4611
 napi_frags_finish net/core/dev.c:5052 [inline]
 napi_gro_frags+0x58a/0xaf0 net/core/dev.c:5125
 tun_get_user+0x262e/0x3710 drivers/net/tun.c:1757
 tun_chr_write_iter+0xb9/0x160 drivers/net/tun.c:1800
 call_write_iter include/linux/fs.h:1772 [inline]
 do_iter_readv_writev+0x525/0x7f0 fs/read_write.c:653
 do_iter_write+0x154/0x540 fs/read_write.c:932
 vfs_writev+0x18a/0x340 fs/read_write.c:977
 do_writev+0xfc/0x2a0 fs/read_write.c:1012
 SYSC_writev fs/read_write.c:1085 [inline]
 SyS_writev+0x27/0x30 fs/read_write.c:1082
 entry_SYSCALL_64_fastpath+0x23/0x9a
RIP: 0033:0x444f50
RSP: 002b:00007ffc71764ad8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
RAX: ffffffffffffffda RBX: 00000000004a6852 RCX: 0000000000444f50
RDX: 0000000000000001 RSI: 00007ffc71764b10 RDI: 0000000000000003
RBP: 00007ffc71764c08 R08: 0000000000000023 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc71764c08
R13: 0000000000402520 R14: 0000000000000000 R15: 0000000000000000
syzkaller218108 (3657) used greatest stack depth: 11920 b

Crashes (1548):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/01/13 18:16 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce
2018/01/13 17:44 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce
2018/01/13 16:50 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce
2018/01/13 16:39 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce
2018/01/13 17:50 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce-386
2018/01/13 17:19 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce-386
2018/01/13 16:58 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce-386
2018/01/13 16:47 upstream c92a9a461dff c9e7aeae .config console log report syz C ci-upstream-kasan-gce-386
2018/01/13 17:22 net-next-old 6bd39bc3da0f c9e7aeae .config console log report syz C ci-upstream-net-kasan-gce
2018/01/13 17:10 net-next-old 6bd39bc3da0f c9e7aeae .config console log report syz C ci-upstream-net-kasan-gce
2018/01/13 17:00 net-next-old 6bd39bc3da0f c9e7aeae .config console log report syz C ci-upstream-net-kasan-gce
2018/01/13 16:37 net-next-old 6bd39bc3da0f c9e7aeae .config console log report syz C ci-upstream-net-kasan-gce
2018/01/13 18:51 linux-next 3e53c7415294 c9e7aeae .config console log report syz C ci-upstream-next-kasan-gce
2018/01/13 18:06 mmots ce3c209f6733 c9e7aeae .config console log report syz C ci-upstream-mmots-kasan-gce
2018/01/13 17:44 mmots ce3c209f6733 c9e7aeae .config console log report syz C ci-upstream-mmots-kasan-gce
2018/01/13 17:15 linux-next 3e53c7415294 c9e7aeae .config console log report syz C ci-upstream-next-kasan-gce
2018/01/13 17:14 mmots ce3c209f6733 c9e7aeae .config console log report syz C ci-upstream-mmots-kasan-gce
2018/01/13 16:54 linux-next 3e53c7415294 c9e7aeae .config console log report syz C ci-upstream-next-kasan-gce
2018/01/13 16:54 mmots ce3c209f6733 c9e7aeae .config console log report syz C ci-upstream-mmots-kasan-gce
2018/01/13 16:44 linux-next 3e53c7415294 c9e7aeae .config console log report syz C ci-upstream-next-kasan-gce
2019/11/26 09:48 upstream 0be0ee71816b f746151a .config console log report ci-upstream-kasan-gce
2018/03/31 07:09 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce
2018/03/31 05:59 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce
2018/03/31 03:44 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce
2018/03/31 00:40 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce
2018/03/30 18:37 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce-root
2018/03/30 16:43 upstream c2a9838452a4 d47f0ed6 .config console log report ci-upstream-kasan-gce
2018/03/30 15:29 upstream c2a9838452a4 d47f0ed6 .config console log report ci-upstream-kasan-gce-root
2018/03/30 12:28 upstream c2a9838452a4 d47f0ed6 .config console log report ci-upstream-kasan-gce
2018/03/30 10:21 upstream c2a9838452a4 d47f0ed6 .config console log report ci-upstream-kasan-gce
2018/03/29 11:29 upstream 0b412605ef5f d47f0ed6 .config console log report ci-upstream-kasan-gce
2018/03/24 08:46 upstream 99fec39e7725 2e9d9054 .config console log report ci-upstream-kasan-gce
2018/03/23 16:45 upstream f36b7534b833 2e9d9054 .config console log report ci-upstream-kasan-gce
2018/02/27 00:35 upstream 4a3928c6f8a5 b370d4a7 .config console log report ci-upstream-kasan-gce
2018/02/26 10:31 upstream c89be5242607 9fe8aa42 .config console log report ci-upstream-kasan-gce
2018/02/26 00:57 upstream c89be5242607 9fe8aa42 .config console log report ci-upstream-kasan-gce
2018/02/25 13:18 upstream 3664ce2d9309 5c1e0207 .config console log report ci-upstream-kasan-gce
2018/02/24 23:30 upstream 9cb9c07d6b0c 5c1e0207 .config console log report ci-upstream-kasan-gce
2018/02/24 11:38 upstream 938e1426e262 5c1e0207 .config console log report ci-upstream-kasan-gce
2018/02/24 09:40 upstream 938e1426e262 5c1e0207 .config console log report ci-upstream-kasan-gce
2018/02/23 22:35 upstream 0f9da844d877 33464158 .config console log report ci-upstream-kasan-gce
2018/03/30 20:55 upstream 9dd2326890d8 8fbce0e4 .config console log report ci-upstream-kasan-gce-386
2018/03/15 20:21 upstream 0aa3fdb8b3a6 08dacaa0 .config console log report ci-upstream-kasan-gce-386
2018/03/08 16:06 upstream 1b88accf6a65 acd0caa5 .config console log report ci-upstream-kasan-gce-386
2018/03/06 13:22 upstream 094b58e1040a aef0b792 .config console log report ci-upstream-kasan-gce-386
2019/09/21 06:17 net-next-old b41dae061bbd d96e88f3 .config console log report ci-upstream-net-kasan-gce
2018/04/03 12:06 net-next-old 159f02977b2f 676bd07e .config console log report ci-upstream-net-kasan-gce
2018/04/03 10:46 net-next-old 159f02977b2f 676bd07e .config console log report ci-upstream-net-kasan-gce
2018/04/02 21:33 net-next-old 159f02977b2f 676bd07e .config console log report ci-upstream-net-kasan-gce
2018/04/01 12:52 net-next-old 06b19fe9a6df 0a78e248 .config console log report ci-upstream-net-kasan-gce
2018/04/01 03:13 net-next-old 8bde261e5352 0174c6c8 .config console log report ci-upstream-net-kasan-gce
2018/03/28 23:43 net-next-old 5d22d47b9ed9 bf5e585c .config console log report ci-upstream-net-kasan-gce
2018/03/27 23:17 net-next-old 5d22d47b9ed9 bf5e585c .config console log report ci-upstream-net-kasan-gce
2018/03/27 19:04 net-next-old 34fd03b9e6a6 bf5e585c .config console log report ci-upstream-net-kasan-gce
2018/03/25 17:38 net-next-old 94cb54924092 e033c1f1 .config console log report ci-upstream-net-kasan-gce
2018/03/24 22:35 net-next-old 94cb54924092 2e9d9054 .config console log report ci-upstream-net-kasan-gce
2018/03/24 15:26 net-next-old 94cb54924092 2e9d9054 .config console log report ci-upstream-net-kasan-gce
2018/03/24 00:25 net-next-old f452518c982e 2e9d9054 .config console log report ci-upstream-net-kasan-gce
2018/03/22 16:47 net-next-old aa65f6365405 2e9d9054 .config console log report ci-upstream-net-kasan-gce
2018/03/22 07:14 net-next-old 454bfe97837a 95c88d7a .config console log report ci-upstream-net-kasan-gce
2018/03/18 14:32 net-next-old 76f38f1f3cf8 08dacaa0 .config console log report ci-upstream-net-kasan-gce
2018/03/16 18:09 net-next-old 0aee4c259849 08dacaa0 .config console log report ci-upstream-net-kasan-gce
2018/03/16 05:17 net-next-old 80d9f3a0fdb8 08dacaa0 .config console log report ci-upstream-net-kasan-gce
2018/03/15 02:55 net-next-old c292566a7779 08dacaa0 .config console log report ci-upstream-net-kasan-gce
2018/03/13 13:32 net-next-old 9ba32046fc2d 08dacaa0 .config console log report ci-upstream-net-kasan-gce
2018/03/13 02:42 net-next-old 129cf5f7f196 f505ca4b .config console log report ci-upstream-net-kasan-gce
2018/03/12 22:12 net-next-old 129cf5f7f196 f505ca4b .config console log report ci-upstream-net-kasan-gce
2018/03/09 10:41 net-next-old fd372a7a9e5e 36d1c454 .config console log report ci-upstream-net-kasan-gce
2018/03/09 09:26 net-next-old fd372a7a9e5e 36d1c454 .config console log report ci-upstream-net-kasan-gce
2018/03/08 04:08 net-next-old a366e300ae9f d50edb7e .config console log report ci-upstream-net-kasan-gce
2018/03/05 00:36 net-next-old efab163bbc19 2c6f473e .config console log report ci-upstream-net-kasan-gce
2018/03/01 23:03 net-next-old f1c02cfb7b30 2c6f473e .config console log report ci-upstream-net-kasan-gce
2018/02/28 20:38 net-next-old fb66cb077560 05b5a32c .config console log report ci-upstream-net-kasan-gce
2018/02/27 11:15 net-next-old 3808b51911fe 05b5a32c .config console log report ci-upstream-net-kasan-gce
2018/02/27 07:44 net-next-old ba6056a41cb0 b370d4a7 .config console log report ci-upstream-net-kasan-gce
2018/02/27 03:43 net-next-old ba6056a41cb0 b370d4a7 .config console log report ci-upstream-net-kasan-gce
2018/02/26 17:11 net-next-old ba6056a41cb0 9fe8aa42 .config console log report ci-upstream-net-kasan-gce
* Struck through repros no longer work on HEAD.