syzbot


possible deadlock in sk_clone_lock

Status: upstream: reported C repro on 2021/01/19 15:48
Reported-by: syzbot+6383ce4b0b8ec575ad93@syzkaller.appspotmail.com
First crash: 1189d, last: 939d
Fix bisection: failed (error log, bisect log)
  
Discussions (1)
Title Replies (including bot) Last reply
possible deadlock in sk_clone_lock 18 (19) 2021/03/05 09:09
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in sk_clone_lock mm 1 1154d 1151d 0/26 auto-closed as invalid on 2021/04/25 05:11
linux-4.14 possible deadlock in sk_clone_lock C error 3 602d 1128d 0/1 upstream: reported C repro on 2021/03/22 00:35
Fix bisection attempts (9)
Created Duration User Patch Repo Result
2021/10/29 12:02 10m bisect fix linux-4.19.y error job log (0)
2021/09/26 13:14 26m bisect fix linux-4.19.y job log (0) log
2021/08/27 12:22 30m bisect fix linux-4.19.y job log (0) log
2021/07/28 11:49 26m bisect fix linux-4.19.y job log (0) log
2021/06/28 11:16 24m bisect fix linux-4.19.y job log (0) log
2021/05/29 10:19 25m bisect fix linux-4.19.y job log (0) log
2021/04/28 20:52 27m bisect fix linux-4.19.y job log (0) log
2021/03/29 17:05 22m bisect fix linux-4.19.y job log (0) log
2021/02/27 16:42 22m bisect fix linux-4.19.y job log (0) log

Sample crash report:
nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based  firewall rule not found. Use the iptables CT target to attach helpers instead.
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
4.19.189-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor187/8109 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
00000000ef5245fa (hugetlb_lock){+.+.}, at: spin_lock include/linux/spinlock.h:329 [inline]
00000000ef5245fa (hugetlb_lock){+.+.}, at: free_huge_page+0x482/0xd20 mm/hugetlb.c:1294

and this task is already holding:
00000000dac182a1 (slock-AF_INET){+.-.}, at: spin_lock include/linux/spinlock.h:329 [inline]
00000000dac182a1 (slock-AF_INET){+.-.}, at: tcp_close+0x5bd/0xfd0 net/ipv4/tcp.c:2434
which would create a new lock dependency:
 (slock-AF_INET){+.-.} -> (hugetlb_lock){+.+.}

but this new dependency connects a SOFTIRQ-irq-safe lock:
 (slock-AF_INET){+.-.}

... which became SOFTIRQ-irq-safe at:
  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
  _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
  spin_lock include/linux/spinlock.h:329 [inline]
  sk_clone_lock+0x40b/0x1430 net/core/sock.c:1671
  inet_csk_clone_lock+0x1f/0x3e0 net/ipv4/inet_connection_sock.c:824
  tcp_create_openreq_child+0x2c/0x19f0 net/ipv4/tcp_minisocks.c:452
  tcp_v4_syn_recv_sock+0xb6/0x1030 net/ipv4/tcp_ipv4.c:1429
  tcp_check_req+0x601/0x1710 net/ipv4/tcp_minisocks.c:789
  tcp_v4_rcv+0x1e3c/0x3b80 net/ipv4/tcp_ipv4.c:1773
  ip_local_deliver_finish+0x495/0xc00 net/ipv4/ip_input.c:215
  NF_HOOK include/linux/netfilter.h:289 [inline]
  ip_local_deliver+0x188/0x500 net/ipv4/ip_input.c:256
  dst_input include/net/dst.h:461 [inline]
  ip_rcv_finish+0x1ca/0x2e0 net/ipv4/ip_input.c:414
  NF_HOOK include/linux/netfilter.h:289 [inline]
  ip_rcv+0xca/0x3c0 net/ipv4/ip_input.c:524
  __netif_receive_skb_one_core+0x114/0x180 net/core/dev.c:4954
  __netif_receive_skb+0x27/0x1c0 net/core/dev.c:5066
  netif_receive_skb_internal+0xf0/0x3f0 net/core/dev.c:5156
  napi_skb_finish net/core/dev.c:5600 [inline]
  napi_gro_receive+0x2e6/0x450 net/core/dev.c:5631
  receive_buf+0xb8c/0x65c0 drivers/net/virtio_net.c:1084
  virtnet_receive drivers/net/virtio_net.c:1348 [inline]
  virtnet_poll+0x568/0xd70 drivers/net/virtio_net.c:1453
  napi_poll net/core/dev.c:6272 [inline]
  net_rx_action+0x4ac/0xfb0 net/core/dev.c:6338
  __do_softirq+0x265/0x980 kernel/softirq.c:292
  run_ksoftirqd+0x57/0x110 kernel/softirq.c:653
  smpboot_thread_fn+0x655/0x9e0 kernel/smpboot.c:164
  kthread+0x33f/0x460 kernel/kthread.c:259
  ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

to a SOFTIRQ-irq-unsafe lock:
 (hugetlb_lock){+.+.}

... which became SOFTIRQ-irq-unsafe at:
...
  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
  _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
  spin_lock include/linux/spinlock.h:329 [inline]
  hugetlb_overcommit_handler+0x2d4/0x460 mm/hugetlb.c:3041
  proc_sys_call_handler.isra.0+0x1f3/0x3b0 fs/proc/proc_sysctl.c:597
  __vfs_write+0xf7/0x770 fs/read_write.c:485
  vfs_write+0x1f3/0x540 fs/read_write.c:549
  ksys_write+0x12b/0x2a0 fs/read_write.c:599
  do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

 Possible interrupt unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(hugetlb_lock);
                               local_irq_disable();
                               lock(slock-AF_INET);
                               lock(hugetlb_lock);
  <Interrupt>
    lock(slock-AF_INET);

 *** DEADLOCK ***

3 locks held by syz-executor187/8109:
 #0: 00000000ea6c226e (&sb->s_type->i_mutex_key#13){+.+.}, at: inode_lock include/linux/fs.h:748 [inline]
 #0: 00000000ea6c226e (&sb->s_type->i_mutex_key#13){+.+.}, at: __sock_release+0x86/0x2a0 net/socket.c:578
 #1: 0000000006cfd8e5 (sk_lock-AF_INET){+.+.}, at: lock_sock include/net/sock.h:1510 [inline]
 #1: 0000000006cfd8e5 (sk_lock-AF_INET){+.+.}, at: tcp_close+0x25/0xfd0 net/ipv4/tcp.c:2344
 #2: 00000000dac182a1 (slock-AF_INET){+.-.}, at: spin_lock include/linux/spinlock.h:329 [inline]
 #2: 00000000dac182a1 (slock-AF_INET){+.-.}, at: tcp_close+0x5bd/0xfd0 net/ipv4/tcp.c:2434

the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (slock-AF_INET){+.-.} ops: 6629 {
   HARDIRQ-ON-W at:
                    __raw_spin_lock_bh include/linux/spinlock_api_smp.h:135 [inline]
                    _raw_spin_lock_bh+0x2f/0x40 kernel/locking/spinlock.c:168
                    spin_lock_bh include/linux/spinlock.h:334 [inline]
                    lock_sock_nested+0x3b/0x110 net/core/sock.c:2864
                    lock_sock include/net/sock.h:1510 [inline]
                    inet_autobind+0x1a/0x190 net/ipv4/af_inet.c:179
                    inet_dgram_connect+0x245/0x2d0 net/ipv4/af_inet.c:569
                    __sys_connect+0x265/0x2c0 net/socket.c:1663
                    __do_sys_connect net/socket.c:1674 [inline]
                    __se_sys_connect net/socket.c:1671 [inline]
                    __x64_sys_connect+0x6f/0xb0 net/socket.c:1671
                    do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
                    entry_SYSCALL_64_after_hwframe+0x49/0xbe
   IN-SOFTIRQ-W at:
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
                    spin_lock include/linux/spinlock.h:329 [inline]
                    sk_clone_lock+0x40b/0x1430 net/core/sock.c:1671
                    inet_csk_clone_lock+0x1f/0x3e0 net/ipv4/inet_connection_sock.c:824
                    tcp_create_openreq_child+0x2c/0x19f0 net/ipv4/tcp_minisocks.c:452
                    tcp_v4_syn_recv_sock+0xb6/0x1030 net/ipv4/tcp_ipv4.c:1429
                    tcp_check_req+0x601/0x1710 net/ipv4/tcp_minisocks.c:789
                    tcp_v4_rcv+0x1e3c/0x3b80 net/ipv4/tcp_ipv4.c:1773
                    ip_local_deliver_finish+0x495/0xc00 net/ipv4/ip_input.c:215
                    NF_HOOK include/linux/netfilter.h:289 [inline]
                    ip_local_deliver+0x188/0x500 net/ipv4/ip_input.c:256
                    dst_input include/net/dst.h:461 [inline]
                    ip_rcv_finish+0x1ca/0x2e0 net/ipv4/ip_input.c:414
                    NF_HOOK include/linux/netfilter.h:289 [inline]
                    ip_rcv+0xca/0x3c0 net/ipv4/ip_input.c:524
                    __netif_receive_skb_one_core+0x114/0x180 net/core/dev.c:4954
                    __netif_receive_skb+0x27/0x1c0 net/core/dev.c:5066
                    netif_receive_skb_internal+0xf0/0x3f0 net/core/dev.c:5156
                    napi_skb_finish net/core/dev.c:5600 [inline]
                    napi_gro_receive+0x2e6/0x450 net/core/dev.c:5631
                    receive_buf+0xb8c/0x65c0 drivers/net/virtio_net.c:1084
                    virtnet_receive drivers/net/virtio_net.c:1348 [inline]
                    virtnet_poll+0x568/0xd70 drivers/net/virtio_net.c:1453
                    napi_poll net/core/dev.c:6272 [inline]
                    net_rx_action+0x4ac/0xfb0 net/core/dev.c:6338
                    __do_softirq+0x265/0x980 kernel/softirq.c:292
                    run_ksoftirqd+0x57/0x110 kernel/softirq.c:653
                    smpboot_thread_fn+0x655/0x9e0 kernel/smpboot.c:164
                    kthread+0x33f/0x460 kernel/kthread.c:259
                    ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
   INITIAL USE at:
                   __raw_spin_lock_bh include/linux/spinlock_api_smp.h:135 [inline]
                   _raw_spin_lock_bh+0x2f/0x40 kernel/locking/spinlock.c:168
                   spin_lock_bh include/linux/spinlock.h:334 [inline]
                   lock_sock_nested+0x3b/0x110 net/core/sock.c:2864
                   lock_sock include/net/sock.h:1510 [inline]
                   inet_autobind+0x1a/0x190 net/ipv4/af_inet.c:179
                   inet_dgram_connect+0x245/0x2d0 net/ipv4/af_inet.c:569
                   __sys_connect+0x265/0x2c0 net/socket.c:1663
                   __do_sys_connect net/socket.c:1674 [inline]
                   __se_sys_connect net/socket.c:1671 [inline]
                   __x64_sys_connect+0x6f/0xb0 net/socket.c:1671
                   do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
                   entry_SYSCALL_64_after_hwframe+0x49/0xbe
 }
 ... key      at: [<ffffffff8dd74810>] af_family_slock_keys+0x10/0x1a0
 ... acquired at:
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
   spin_lock include/linux/spinlock.h:329 [inline]
   free_huge_page+0x482/0xd20 mm/hugetlb.c:1294
   __put_page+0xe2/0x3a0 mm/swap.c:112
   put_page include/linux/mm.h:963 [inline]
   __skb_frag_unref include/linux/skbuff.h:2833 [inline]
   skb_release_data+0x2f3/0x920 net/core/skbuff.c:577
   skb_release_all net/core/skbuff.c:640 [inline]
   __kfree_skb+0x46/0x60 net/core/skbuff.c:654
   sk_wmem_free_skb include/net/sock.h:1466 [inline]
   tcp_write_queue_purge+0x24d/0x800 net/ipv4/tcp.c:2543
   tcp_v4_destroy_sock+0x101/0x770 net/ipv4/tcp_ipv4.c:1989
   inet_csk_destroy_sock+0x189/0x400 net/ipv4/inet_connection_sock.c:875
   tcp_close+0x95f/0xfd0 net/ipv4/tcp.c:2498
   inet_release+0xd7/0x1e0 net/ipv4/af_inet.c:427
   __sock_release+0xcd/0x2a0 net/socket.c:579
   sock_close+0x15/0x20 net/socket.c:1140
   __fput+0x2ce/0x890 fs/file_table.c:278
   task_work_run+0x148/0x1c0 kernel/task_work.c:113
   exit_task_work include/linux/task_work.h:22 [inline]
   do_exit+0xbf3/0x2be0 kernel/exit.c:870
   do_group_exit+0x125/0x310 kernel/exit.c:967
   get_signal+0x3f2/0x1f70 kernel/signal.c:2589
   do_signal+0x8f/0x1670 arch/x86/kernel/signal.c:799
   exit_to_usermode_loop+0x204/0x2a0 arch/x86/entry/common.c:163
   prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
   syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
   do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
   entry_SYSCALL_64_after_hwframe+0x49/0xbe


the dependencies between the lock to be acquired
 and SOFTIRQ-irq-unsafe lock:
-> (hugetlb_lock){+.+.} ops: 38 {
   HARDIRQ-ON-W at:
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
                    spin_lock include/linux/spinlock.h:329 [inline]
                    hugetlb_overcommit_handler+0x2d4/0x460 mm/hugetlb.c:3041
                    proc_sys_call_handler.isra.0+0x1f3/0x3b0 fs/proc/proc_sysctl.c:597
                    __vfs_write+0xf7/0x770 fs/read_write.c:485
                    vfs_write+0x1f3/0x540 fs/read_write.c:549
                    ksys_write+0x12b/0x2a0 fs/read_write.c:599
                    do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
                    entry_SYSCALL_64_after_hwframe+0x49/0xbe
   SOFTIRQ-ON-W at:
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
                    spin_lock include/linux/spinlock.h:329 [inline]
                    hugetlb_overcommit_handler+0x2d4/0x460 mm/hugetlb.c:3041
                    proc_sys_call_handler.isra.0+0x1f3/0x3b0 fs/proc/proc_sysctl.c:597
                    __vfs_write+0xf7/0x770 fs/read_write.c:485
                    vfs_write+0x1f3/0x540 fs/read_write.c:549
                    ksys_write+0x12b/0x2a0 fs/read_write.c:599
                    do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
                    entry_SYSCALL_64_after_hwframe+0x49/0xbe
   INITIAL USE at:
                   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
                   spin_lock include/linux/spinlock.h:329 [inline]
                   hugetlb_overcommit_handler+0x2d4/0x460 mm/hugetlb.c:3041
                   proc_sys_call_handler.isra.0+0x1f3/0x3b0 fs/proc/proc_sysctl.c:597
                   __vfs_write+0xf7/0x770 fs/read_write.c:485
                   vfs_write+0x1f3/0x540 fs/read_write.c:549
                   ksys_write+0x12b/0x2a0 fs/read_write.c:599
                   do_syscall_64+0xf9/0x620 arch/x86/entry/common.c:293
                   entry_SYSCALL_64_after_hwframe+0x49/0xbe
 }
 ... key      at: [<ffffffff8a020cb8>] hugetlb_lock+0x18/0x17a0
 ... acquired at:
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
   spin_lock include/linux/spinlock.h:329 [inline]
   free_huge_page+0x482/0xd20 mm/hugetlb.c:1294
   __put_page+0xe2/0x3a0 mm/swap.c:112
   put_page include/linux/mm.h:963 [inline]
   __skb_frag_unref include/linux/skbuff.h:2833 [inline]
   skb_release_data+0x2f3/0x920 net/core/skbuff.c:577
   skb_release_all net/core/skbuff.c:640 [inline]
   __kfree_skb+0x46/0x60 net/core/skbuff.c:654
   sk_wmem_free_skb include/net/sock.h:1466 [inline]
   tcp_write_queue_purge+0x24d/0x800 net/ipv4/tcp.c:2543
   tcp_v4_destroy_sock+0x101/0x770 net/ipv4/tcp_ipv4.c:1989
   inet_csk_destroy_sock+0x189/0x400 net/ipv4/inet_connection_sock.c:875
   tcp_close+0x95f/0xfd0 net/ipv4/tcp.c:2498
   inet_release+0xd7/0x1e0 net/ipv4/af_inet.c:427
   __sock_release+0xcd/0x2a0 net/socket.c:579
   sock_close+0x15/0x20 net/socket.c:1140
   __fput+0x2ce/0x890 fs/file_table.c:278
   task_work_run+0x148/0x1c0 kernel/task_work.c:113
   exit_task_work include/linux/task_work.h:22 [inline]
   do_exit+0xbf3/0x2be0 kernel/exit.c:870
   do_group_exit+0x125/0x310 kernel/exit.c:967
   get_signal+0x3f2/0x1f70 kernel/signal.c:2589
   do_signal+0x8f/0x1670 arch/x86/kernel/signal.c:799
   exit_to_usermode_loop+0x204/0x2a0 arch/x86/entry/common.c:163
   prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
   syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
   do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
   entry_SYSCALL_64_after_hwframe+0x49/0xbe


stack backtrace:
CPU: 0 PID: 8109 Comm: syz-executor187 Not tainted 4.19.189-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
 print_bad_irq_dependency kernel/locking/lockdep.c:1573 [inline]
 check_usage.cold+0x7ea/0xbad kernel/locking/lockdep.c:1605
 check_irq_usage kernel/locking/lockdep.c:1661 [inline]
 check_prev_add_irq kernel/locking/lockdep_states.h:8 [inline]
 check_prev_add kernel/locking/lockdep.c:1871 [inline]
 check_prevs_add kernel/locking/lockdep.c:1979 [inline]
 validate_chain kernel/locking/lockdep.c:2420 [inline]
 __lock_acquire+0x1da1/0x3ff0 kernel/locking/lockdep.c:3416
 lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3908
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
 spin_lock include/linux/spinlock.h:329 [inline]
 free_huge_page+0x482/0xd20 mm/hugetlb.c:1294
 __put_page+0xe2/0x3a0 mm/swap.c:112
 put_page include/linux/mm.h:963 [inline]
 __skb_frag_unref include/linux/skbuff.h:2833 [inline]
 skb_release_data+0x2f3/0x920 net/core/skbuff.c:577
 skb_release_all net/core/skbuff.c:640 [inline]
 __kfree_skb+0x46/0x60 net/core/skbuff.c:654
 sk_wmem_free_skb include/net/sock.h:1466 [inline]
 tcp_write_queue_purge+0x24d/0x800 net/ipv4/tcp.c:2543
 tcp_v4_destroy_sock+0x101/0x770 net/ipv4/tcp_ipv4.c:1989
 inet_csk_destroy_sock+0x189/0x400 net/ipv4/inet_connection_sock.c:875
 tcp_close+0x95f/0xfd0 net/ipv4/tcp.c:2498
 inet_release+0xd7/0x1e0 net/ipv4/af_inet.c:427
 __sock_release+0xcd/0x2a0 net/socket.c:579
 sock_close+0x15/0x20 net/socket.c:1140
 __fput+0x2ce/0x890 fs/file_table.c:278
 task_work_run+0x148/0x1c0 kernel/task_work.c:113
 exit_task_work include/linux/task_work.h:22 [inline]
 do_exit+0xbf3/0x2be0 kernel/exit.c:870
 do_group_exit+0x125/0x310 kernel/exit.c:967
 get_signal+0x3f2/0x1f70 kernel/signal.c:2589
 do_signal+0x8f/0x1670 arch/x86/kernel/signal.c:799
 exit_to_usermode_loop+0x204/0x2a0 arch/x86/entry/common.c:163
 prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
 syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
 do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x44d639
Code: Bad RIP value.
RSP: 002b:00007f8a5f2bb2f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000004cb410 RCX: 000000000044d639
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00000000004cb418
RBP: 00000000004cb41c R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000049b6b0
R13: 000000000049b2e8 R14: 0000000020000900 R15: 00000000004cb418

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/04/29 09:59 linux-4.19.y 97a8651cadce 77e2b668 .config console log report syz C ci2-linux-4-19 possible deadlock in sk_clone_lock
2021/01/28 16:37 linux-4.19.y c4ff839de17f eefc07f2 .config console log report syz C ci2-linux-4-19 possible deadlock in sk_clone_lock
2021/01/19 16:16 linux-4.19.y c110fed0e606 63631df1 .config console log report syz C ci2-linux-4-19 possible deadlock in sk_clone_lock
2021/01/19 15:47 linux-4.19.y c110fed0e606 63631df1 .config console log report info ci2-linux-4-19 possible deadlock in sk_clone_lock
* Struck through repros no longer work on HEAD.