syzbot


possible deadlock in j1939_sk_errqueue (2)

Status: fixed on 2024/03/26 17:39
Subsystems: can
[Documentation on labels]
Reported-by: syzbot+1591462f226d9cbf0564@syzkaller.appspotmail.com
Fix commit: 6cdedc18ba7b can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock
First crash: 302d, last: 71d
Cause bisection: introduced by (bisect log) :
commit 2030043e616cab40f510299f09b636285e0a3678
Author: Oleksij Rempel <o.rempel@pengutronix.de>
Date: Fri May 21 11:57:20 2021 +0000

  can: j1939: fix Use-after-Free, hold skb ref while in use

Crash: unregister_netdevice: waiting for DEV to become free (log)
Repro: C syz .config
  
Discussions (10)
Title Replies (including bot) Last reply
[PATCH net 1/3] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 3 (3) 2024/02/15 11:50
[syzbot] Monthly can report (Jan 2024) 0 (1) 2024/01/16 07:55
[PATCH] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 6 (7) 2023/11/17 08:10
[syzbot] [can?] possible deadlock in j1939_sk_errqueue (2) 0 (3) 2023/11/15 03:54
[syzbot] Monthly can report (Nov 2023) 0 (1) 2023/11/01 10:13
[syzbot] Monthly can report (Sep 2023) 0 (1) 2023/10/02 09:45
Re: [PATCH] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 1 (1) 2023/07/14 05:58
[PATCH] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 2 (3) 2023/07/13 22:23
Re: [PATCH] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 1 (1) 2023/07/07 04:39
Re: [PATCH] can: j1939: prevent deadlock by changing j1939_socks_lock to rwlock 3 (3) 2023/07/05 04:50
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 possible deadlock in j1939_sk_errqueue origin:upstream C error 6 98d 273d 0/3 upstream: reported C repro on 2023/07/28 13:47
linux-6.1 possible deadlock in j1939_sk_errqueue origin:upstream C inconclusive 6 64d 293d 0/3 upstream: reported C repro on 2023/07/09 02:39
upstream possible deadlock in j1939_sk_errqueue can C unreliable 227 395d 932d 22/26 fixed on 2023/06/08 14:41
Last patch testing requests (6)
Created Duration User Patch Repo Result
2024/02/15 16:36 20m retest repro git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci report log
2024/02/15 16:36 18m retest repro git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci report log
2023/11/26 21:44 25m retest repro upstream report log
2023/09/01 10:46 25m retest repro git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci OK log
2023/07/12 00:47 27m astrajoan@yahoo.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master OK log
2023/07/11 13:20 24m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git e40939bbfc68 OK log
Fix bisection attempts (2)
Created Duration User Patch Repo Result
2024/01/15 16:52 2h30m bisect fix upstream job log (0) log
2023/12/13 12:10 2h09m bisect fix upstream job log (0) log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.6.0-syzkaller-16176-g1b907d050735 #0 Not tainted
------------------------------------------------------
syz-executor139/5203 is trying to acquire lock:
ffff8880747810d0 (&priv->j1939_socks_lock){+.-.}-{2:2}, at: spin_lock_bh include/linux/spinlock.h:356 [inline]
ffff8880747810d0 (&priv->j1939_socks_lock){+.-.}-{2:2}, at: j1939_sk_errqueue+0x96/0x150 net/can/j1939/socket.c:1083

but task is already holding lock:
ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: spin_lock_bh include/linux/spinlock.h:356 [inline]
ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: j1939_session_list_lock net/can/j1939/transport.c:238 [inline]
ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: j1939_cancel_active_session+0x47/0x480 net/can/j1939/transport.c:2183

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&priv->active_session_list_lock){+.-.}-{2:2}:
       __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
       _raw_spin_lock_bh+0x35/0x50 kernel/locking/spinlock.c:178
       spin_lock_bh include/linux/spinlock.h:356 [inline]
       j1939_session_list_lock net/can/j1939/transport.c:238 [inline]
       j1939_session_activate+0x51/0x3e0 net/can/j1939/transport.c:1564
       j1939_sk_queue_activate_next_locked net/can/j1939/socket.c:181 [inline]
       j1939_sk_queue_activate_next+0x26c/0x3f0 net/can/j1939/socket.c:208
       j1939_session_deactivate_activate_next net/can/j1939/transport.c:1108 [inline]
       j1939_session_completed net/can/j1939/transport.c:1222 [inline]
       j1939_xtp_rx_eoma_one net/can/j1939/transport.c:1395 [inline]
       j1939_xtp_rx_eoma+0x323/0x530 net/can/j1939/transport.c:1410
       j1939_tp_cmd_recv net/can/j1939/transport.c:2099 [inline]
       j1939_tp_recv+0x88e/0x1050 net/can/j1939/transport.c:2144
       j1939_can_recv+0x732/0xb20 net/can/j1939/main.c:112
       deliver net/can/af_can.c:572 [inline]
       can_rcv_filter+0x349/0x7c0 net/can/af_can.c:606
       can_receive+0x327/0x480 net/can/af_can.c:663
       can_rcv+0x144/0x260 net/can/af_can.c:687
       __netif_receive_skb_one_core net/core/dev.c:5527 [inline]
       __netif_receive_skb+0x1ca/0x530 net/core/dev.c:5641
       process_backlog+0x385/0x760 net/core/dev.c:5969
       __napi_poll+0xc7/0x470 net/core/dev.c:6531
       napi_poll net/core/dev.c:6600 [inline]
       net_rx_action+0x78b/0x1010 net/core/dev.c:6733
       __do_softirq+0x2bf/0x93a kernel/softirq.c:553
       do_softirq+0x11b/0x1e0 kernel/softirq.c:454
       __local_bh_enable_ip+0x1b5/0x1f0 kernel/softirq.c:381
       spin_unlock_bh include/linux/spinlock.h:396 [inline]
       batadv_nc_purge_paths+0x309/0x3a0 net/batman-adv/network-coding.c:471
       batadv_nc_worker+0x328/0x610 net/batman-adv/network-coding.c:720
       process_one_work kernel/workqueue.c:2630 [inline]
       process_scheduled_works+0x90f/0x1400 kernel/workqueue.c:2703
       worker_thread+0xa5f/0xff0 kernel/workqueue.c:2784
       kthread+0x2d3/0x370 kernel/kthread.c:388
       ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:242

-> #1 (&jsk->sk_session_queue_lock){+.-.}-{2:2}:
       __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
       _raw_spin_lock_bh+0x35/0x50 kernel/locking/spinlock.c:178
       spin_lock_bh include/linux/spinlock.h:356 [inline]
       j1939_sk_queue_drop_all+0x3a/0x240 net/can/j1939/socket.c:139
       j1939_sk_netdev_event_netdown+0xed/0x160 net/can/j1939/socket.c:1282
       j1939_netdev_notify+0x20d/0x2e0 net/can/j1939/main.c:381
       notifier_call_chain+0x18c/0x3a0 kernel/notifier.c:93
       call_netdevice_notifiers_extack net/core/dev.c:2003 [inline]
       call_netdevice_notifiers net/core/dev.c:2017 [inline]
       dev_close_many+0x336/0x4b0 net/core/dev.c:1558
       unregister_netdevice_many_notify+0x520/0x1710 net/core/dev.c:10956
       rtnl_delete_link net/core/rtnetlink.c:3263 [inline]
       rtnl_dellink+0x523/0x8d0 net/core/rtnetlink.c:3315
       rtnetlink_rcv_msg+0x882/0x1030 net/core/rtnetlink.c:6558
       netlink_rcv_skb+0x1df/0x430 net/netlink/af_netlink.c:2545
       netlink_unicast_kernel net/netlink/af_netlink.c:1342 [inline]
       netlink_unicast+0x7e6/0x980 net/netlink/af_netlink.c:1368
       netlink_sendmsg+0xa37/0xd70 net/netlink/af_netlink.c:1910
       sock_sendmsg_nosec net/socket.c:730 [inline]
       __sock_sendmsg net/socket.c:745 [inline]
       ____sys_sendmsg+0x592/0x890 net/socket.c:2584
       ___sys_sendmsg net/socket.c:2638 [inline]
       __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
       do_syscall_x64 arch/x86/entry/common.c:51 [inline]
       do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
       entry_SYSCALL_64_after_hwframe+0x63/0x6b

-> #0 (&priv->j1939_socks_lock){+.-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
       validate_chain kernel/locking/lockdep.c:3868 [inline]
       __lock_acquire+0x39ff/0x7f70 kernel/locking/lockdep.c:5136
       lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
       __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
       _raw_spin_lock_bh+0x35/0x50 kernel/locking/spinlock.c:178
       spin_lock_bh include/linux/spinlock.h:356 [inline]
       j1939_sk_errqueue+0x96/0x150 net/can/j1939/socket.c:1083
       __j1939_session_release net/can/j1939/transport.c:294 [inline]
       kref_put include/linux/kref.h:65 [inline]
       j1939_session_put+0xff/0x460 net/can/j1939/transport.c:299
       j1939_session_deactivate_locked net/can/j1939/transport.c:1086 [inline]
       j1939_cancel_active_session+0x354/0x480 net/can/j1939/transport.c:2194
       j1939_netdev_notify+0x205/0x2e0 net/can/j1939/main.c:380
       notifier_call_chain+0x18c/0x3a0 kernel/notifier.c:93
       call_netdevice_notifiers_extack net/core/dev.c:2003 [inline]
       call_netdevice_notifiers net/core/dev.c:2017 [inline]
       dev_close_many+0x336/0x4b0 net/core/dev.c:1558
       unregister_netdevice_many_notify+0x520/0x1710 net/core/dev.c:10956
       rtnl_delete_link net/core/rtnetlink.c:3263 [inline]
       rtnl_dellink+0x523/0x8d0 net/core/rtnetlink.c:3315
       rtnetlink_rcv_msg+0x882/0x1030 net/core/rtnetlink.c:6558
       netlink_rcv_skb+0x1df/0x430 net/netlink/af_netlink.c:2545
       netlink_unicast_kernel net/netlink/af_netlink.c:1342 [inline]
       netlink_unicast+0x7e6/0x980 net/netlink/af_netlink.c:1368
       netlink_sendmsg+0xa37/0xd70 net/netlink/af_netlink.c:1910
       sock_sendmsg_nosec net/socket.c:730 [inline]
       __sock_sendmsg net/socket.c:745 [inline]
       ____sys_sendmsg+0x592/0x890 net/socket.c:2584
       ___sys_sendmsg net/socket.c:2638 [inline]
       __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
       do_syscall_x64 arch/x86/entry/common.c:51 [inline]
       do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
       entry_SYSCALL_64_after_hwframe+0x63/0x6b

other info that might help us debug this:

Chain exists of:
  &priv->j1939_socks_lock --> &jsk->sk_session_queue_lock --> &priv->active_session_list_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&priv->active_session_list_lock);
                               lock(&jsk->sk_session_queue_lock);
                               lock(&priv->active_session_list_lock);
  lock(&priv->j1939_socks_lock);

 *** DEADLOCK ***

2 locks held by syz-executor139/5203:
 #0: ffffffff8eb36e88 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8eb36e88 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x82c/0x1030 net/core/rtnetlink.c:6555
 #1: ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: spin_lock_bh include/linux/spinlock.h:356 [inline]
 #1: ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: j1939_session_list_lock net/can/j1939/transport.c:238 [inline]
 #1: ffff888074781088 (&priv->active_session_list_lock){+.-.}-{2:2}, at: j1939_cancel_active_session+0x47/0x480 net/can/j1939/transport.c:2183

stack backtrace:
CPU: 1 PID: 5203 Comm: syz-executor139 Not tainted 6.6.0-syzkaller-16176-g1b907d050735 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/09/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x375/0x4a0 kernel/locking/lockdep.c:2187
 check_prev_add kernel/locking/lockdep.c:3134 [inline]
 check_prevs_add kernel/locking/lockdep.c:3253 [inline]
 validate_chain kernel/locking/lockdep.c:3868 [inline]
 __lock_acquire+0x39ff/0x7f70 kernel/locking/lockdep.c:5136
 lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
 _raw_spin_lock_bh+0x35/0x50 kernel/locking/spinlock.c:178
 spin_lock_bh include/linux/spinlock.h:356 [inline]
 j1939_sk_errqueue+0x96/0x150 net/can/j1939/socket.c:1083
 __j1939_session_release net/can/j1939/transport.c:294 [inline]
 kref_put include/linux/kref.h:65 [inline]
 j1939_session_put+0xff/0x460 net/can/j1939/transport.c:299
 j1939_session_deactivate_locked net/can/j1939/transport.c:1086 [inline]
 j1939_cancel_active_session+0x354/0x480 net/can/j1939/transport.c:2194
 j1939_netdev_notify+0x205/0x2e0 net/can/j1939/main.c:380
 notifier_call_chain+0x18c/0x3a0 kernel/notifier.c:93
 call_netdevice_notifiers_extack net/core/dev.c:2003 [inline]
 call_netdevice_notifiers net/core/dev.c:2017 [inline]
 dev_close_many+0x336/0x4b0 net/core/dev.c:1558
 unregister_netdevice_many_notify+0x520/0x1710 net/core/dev.c:10956
 rtnl_delete_link net/core/rtnetlink.c:3263 [inline]
 rtnl_dellink+0x523/0x8d0 net/core/rtnetlink.c:3315
 rtnetlink_rcv_msg+0x882/0x1030 net/core/rtnetlink.c:6558
 netlink_rcv_skb+0x1df/0x430 net/netlink/af_netlink.c:2545
 netlink_unicast_kernel net/netlink/af_netlink.c:1342 [inline]
 netlink_unicast+0x7e6/0x980 net/netlink/af_netlink.c:1368
 netlink_sendmsg+0xa37/0xd70 net/netlink/af_netlink.c:1910
 sock_sendmsg_nosec net/socket.c:730 [inline]
 __sock_sendmsg net/socket.c:745 [inline]
 ____sys_sendmsg+0x592/0x890 net/socket.c:2584
 ___sys_sendmsg net/socket.c:2638 [inline]
 __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
 entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f959b6e2329
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 1c 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f959b679168 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007f959b772218 RCX: 00007f959b6e2329
RDX: 0000000000000000 RSI: 00000000200002c0 RDI: 0000000000000008
RBP: 00007f959b772210 R08: 00007f959b6796c0 R09: 00007f959b772218
R10: 00007f959b6796c0 R11: 0000000000000246 R12: 00007f959b77221c
R13: 000000000000006e R14: 00007ffdaabfee70 R15: 00007ffdaabfef58
 </TASK>

Crashes (24):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/11/12 19:33 upstream 1b907d050735 6d6dbf8a .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/20 07:03 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 0802e17d9aca 9bd8dcda .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in j1939_sk_errqueue
2024/01/19 19:57 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 0802e17d9aca 21772ce4 .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in j1939_sk_errqueue
2023/07/10 17:53 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci e40939bbfc68 52ae002a .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in j1939_sk_errqueue
2024/02/01 11:59 upstream 6764c317b6bb 81024119 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/02/01 09:03 upstream 6764c317b6bb 373b66cd .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/31 02:57 upstream 2a6526c4f389 7f400fcb .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/28 23:26 upstream 4854cf9c61d0 cc4a4020 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/27 16:59 upstream 3a5879d495b2 cc4a4020 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/25 20:44 upstream 6098d87eaf31 cc4a4020 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/25 17:35 upstream 6098d87eaf31 cc4a4020 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/23 18:53 upstream 7ed2632ec7d7 1e153dc8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/22 00:09 upstream 4fbbed787267 9bd8dcda .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2023/10/18 23:17 upstream dd72f9c7e512 342b9c55 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root possible deadlock in j1939_sk_errqueue
2024/01/21 05:07 upstream 125514880ddd 9bd8dcda .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 possible deadlock in j1939_sk_errqueue
2024/01/18 12:26 upstream 296455ade1fd 239abf84 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in j1939_sk_errqueue
2023/09/30 20:10 upstream 3b517966c561 8e26a358 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in j1939_sk_errqueue
2023/08/06 04:12 upstream f6a691685962 4ffcc9ef .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce possible deadlock in j1939_sk_errqueue
2023/07/09 02:46 upstream 8fc3b8f082cc 668cb1fa .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in j1939_sk_errqueue
2024/01/20 13:09 net dbc153fd3c14 9bd8dcda .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce possible deadlock in j1939_sk_errqueue
2023/12/14 03:44 net 2513974cc3e1 3222d10c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce possible deadlock in j1939_sk_errqueue
2023/10/01 18:55 net 6b09edc1b317 8e26a358 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce possible deadlock in j1939_sk_errqueue
2023/06/30 06:12 net-next ae230642190a 7b33cf8f .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce possible deadlock in j1939_sk_errqueue
2024/01/18 10:07 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 0802e17d9aca 915053c7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in j1939_sk_errqueue
* Struck through repros no longer work on HEAD.