syzbot


panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb. (2)

Status: auto-closed as invalid on 2021/09/13 23:18
Reported-by: syzbot+5dfbe122c50e22d078bc@syzkaller.appspotmail.com
First crash: 1303d, last: 1254d
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb. (3) syz 7 302d 802d 0/2 upstream: reported syz repro on 2022/09/11 14:28
freebsd panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb. 3 1452d 1483d 0/2 auto-closed as invalid on 2021/02/28 18:58

Sample crash report:
panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3361

cpuid = 1
time = 46
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe005164f650
vpanic() at vpanic+0x1cc/frame 0xfffffe005164f6b0
panic() at panic+0x43/frame 0xfffffe005164f710
__mtx_lock_sleep() at __mtx_lock_sleep+0x68c/frame 0xfffffe005164f7a0
__mtx_lock_flags() at __mtx_lock_flags+0x16d/frame 0xfffffe005164f800
sctp_inpcb_free() at sctp_inpcb_free+0x4de/frame 0xfffffe005164f880
sctp_free_assoc() at sctp_free_assoc+0x288c/frame 0xfffffe005164f910
sctp_load_addresses_from_init() at sctp_load_addresses_from_init+0x14fd/frame 0xfffffe00516502a0
sctp_process_control() at sctp_process_control+0x969b/frame 0xfffffe0051650740
sctp_common_input_processing() at sctp_common_input_processing+0x83a/frame 0xfffffe00516508c0
sctp_input_with_port() at sctp_input_with_port+0x30a/frame 0xfffffe00516509b0
sctp_input() at sctp_input+0x1f/frame 0xfffffe00516509d0
ip_input() at ip_input+0x38b/frame 0xfffffe0051650a90
swi_net() at swi_net+0x20e/frame 0xfffffe0051650b10
ithread_loop() at ithread_loop+0x33f/frame 0xfffffe0051650bb0
fork_exit() at fork_exit+0xb3/frame 0xfffffe0051650bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0051650bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100033 ]
Stopped at      kdb_enter+0x67: movq    $0,0x164006e(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xffffffff81132800  vprintf+0x140
rdx                        0x1
rbx                          0
rsp         0xfffffe005164f630
rbp         0xfffffe005164f650
rsi                          0
rdi         0xffffffff81132836  vprintf+0x176
r8                           0
r9          0x8080808080808080
r10         0xfffffe005164f520
r11              0x1ffaefff59c
r12         0xffffffff82267d30  ddb_dbbe
r13         0xffffffff81992800
r14         0xffffffff81a73f45
r15         0xffffffff81a73f45
rip         0xffffffff81129897  kdb_enter+0x67
rflags                    0x86
kdb_enter+0x67: movq    $0,0x164006e(%rip)
db> show proc
Process 12 (intr) at 0xfffff80004c6fa60:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xffffffff8271ce70
 ABI: null
 flag: 0x10000284  flag2: 0
 reaper: 0xffffffff8271ce70 reapsubtree: 12
 sigparent: 20
 vmspace: 0xffffffff8271dad0
   (map 0xffffffff8271dad0)
   (map.pmap 0xffffffff8271db90)
   (pmap 0xffffffff8271dbf0)
 threads: 23
100011                   I                                   [swi5: fast taskq]
100014                   I                                   [swi6: task queue]
100016                   I                                   [swi6: Giant taskq]
100030                   I                                   [swi3: vm]
100031                   L      *sctp-tc 0xfffff80015a9aa80  [swi4: clock (0)]
100032                   I                                   [swi4: clock (1)]
100033                   Run     CPU 1                       [swi1: netisr 0]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq10: virtio_pci2]
100061                   I                                   [irq1: atkbd0]
100062                   I                                   [irq12: psm0]
100063                   I                                   [swi0: uart uart++]
100071                   I                                   [swi1: pf send]
100084                   I                                   [swi1: hpts]
100085                   I                                   [swi1: hpts]
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 9570   785   785     0  S       (threaded)                  syz-executor.1
114865                   S       nanslp  0xffffffff8273d0a0  syz-executor.1
115593                   S       sbwait  0xfffff80031ec7574  syz-executor.1
115594                   S       sbwait  0xfffff80031fb6cd4  syz-executor.1
115595                   S       sbwait  0xfffff8003cce3574  syz-executor.1
115596                   S       uwait   0xfffff8003ca6eb00  syz-executor.1
115597                   S       uwait   0xfffff80031d5e500  syz-executor.1
 9566   784   784     0  S       (threaded)                  syz-executor.0
113848                   S       nanslp  0xffffffff8273d0a0  syz-executor.0
115599                   S       connec  0xfffff80031fb6490  syz-executor.0
 8907     1   832     0  S       uwait   0xfffff8003ce6db00  syz-executor.3
 8903     1   832     0  S       uwait   0xfffff80015e96900  syz-executor.3
 8044     1   832     0  S       uwait   0xfffff8003cd9d900  syz-executor.3
 8041     1   832     0  S       uwait   0xfffff8003cd0a100  syz-executor.3
 8037     1   832     0  S       uwait   0xfffff8003ca6e300  syz-executor.3
 8031     1   832     0  S       uwait   0xfffff80031d5e900  syz-executor.3
 8029     1   832     0  S       uwait   0xfffff8004e247500  syz-executor.3
 7920     1   803     0  S       uwait   0xfffff80031e11300  syz-executor.2
 7913     1   803     0  S       uwait   0xfffff80031d5e100  syz-executor.2
 7908     1   803     0  S       uwait   0xfffff80031d5ef00  syz-executor.2
 7842     1   803     0  S       uwait   0xfffff8003cf32b00  syz-executor.2
 7840     1   803     0  S       uwait   0xfffff8003cd9dd00  syz-executor.2
 7605     1   832     0  S       uwait   0xfffff80031bb4d00  syz-executor.3
 7600     1   832     0  S       uwait   0xfffff8003ce6df00  syz-executor.3
 7598     1   832     0  S       uwait   0xfffff8003ce5d500  syz-executor.3
 7596     1   832     0  S       uwait   0xfffff8003ce5d300  syz-executor.3
 7595     1   832     0  S       uwait   0xfffff8003cd68100  syz-executor.3
 6549     1   803     0  S       uwait   0xfffff8003ca70900  syz-executor.2
 6541     1   803     0  S       uwait   0xfffff80031810300  syz-executor.2
 6540     1   784     0  S       umtxn   0xfffff80031810b00  syz-executor.0
 6529     1   803     0  S       uwait   0xfffff8003ca6ed00  syz-executor.2
 6528     1   784     0  S       uwait   0xfffff80031810100  syz-executor.0
 6519     1   784     0  S       uwait   0xfffff80031bb4f00  syz-executor.0
 6515     1   784     0  S       uwait   0xfffff8003cf32100  syz-executor.0
 6513     1   784     0  S       uwait   0xfffff8003ce5dd00  syz-executor.0
 5005     1   785     0  S       uwait   0xfffff8003cd0ad00  syz-executor.1
 4405     1   784     0  S       uwait   0xfffff8003cd0af00  syz-executor.0
 4403     1   803     0  S       uwait   0xfffff8003cd0a900  syz-executor.2
 4399     1   784     0  S       uwait   0xfffff8003cd9db00  syz-executor.0
 4396     1   803     0  S       uwait   0xfffff80015e77700  syz-executor.2
 4392     1   784     0  S       uwait   0xfffff8003ca6e500  syz-executor.0
 4388     1   803     0  S       uwait   0xfffff80031d5eb00  syz-executor.2
 4383     1  4376     0  S       uwait   0xfffff8003ca70d00  syz-executor.2
 4382     1   803     0  S       uwait   0xfffff80015e96300  syz-executor.2
 3854     1  3854    65  Ss      select  0xfffff800314ff440  dhclient
 3712     1   785     0  S       uwait   0xfffff8003cd0a500  syz-executor.1
 3711     1   785     0  S       uwait   0xfffff8003ce6d700  syz-executor.1
 3262     1   784     0  S       uwait   0xfffff80015e96700  syz-executor.0
 3261     1   784     0  S       uwait   0xfffff80015e96100  syz-executor.0
 3120     1  3120     0  Ss      select  0xfffff8003ce1e440  dhclient
 3117     1  3117     0  Ss      select  0xfffff80015dae2c0  dhclient
 3098     1  3098    65  Ss      select  0xfffff800315b3bc0  dhclient
 2327     1  2327     0  Ss      select  0xfffff8003ce1f2c0  dhclient
 2323     1  2323     0  Ss      select  0xfffff8003167f140  dhclient
 2281     1  2281    65  Ss      select  0xfffff8003ce1e5c0  dhclient
 2187     1   784     0  S       uwait   0xfffff8003cd9d100  syz-executor.0
 2184     1   784     0  S       uwait   0xfffff8003cd68900  syz-executor.0
 2182     1   784     0  S       uwait   0xfffff8003cd68700  syz-executor.0
 2180     1   784     0  S       uwait   0xfffff8003ce5d900  syz-executor.0
 2179     1   784     0  S       uwait   0xfffff8003cf32f00  syz-executor.0
 1837     1   785     0  S       uwait   0xfffff8003ce6dd00  syz-executor.1
 1836     1   785     0  S       uwait   0xfffff8003ca70b00  syz-executor.1
 1587     1  1587     0  Ss      select  0xfffff8003cd4b2c0  dhclient
 1584     1  1584     0  Ss      select  0xfffff8003ce1fbc0  dhclient
 1565     1  1565    65  Ss      select  0xfffff8003ce1f140  dhclient
 1184     1   784     0  S       uwait   0xfffff8003ca70f00  syz-executor.0
 1183     1   784     0  S       uwait   0xfffff8003ce6d100  syz-executor.0
  909     0     0     0  DL      aiordy  0xfffff8003cf04000  [aiod4]
  908     0     0     0  DL      aiordy  0xfffff8003cf04530  [aiod3]
  907     0     0     0  DL      aiordy  0xfffff8003cf04a60  [aiod2]
  906     0     0     0  DL      aiordy  0xfffff8003ceab000  [aiod1]
  832   780   832     0  Ss      piperd  0xfffff8003ccfe2e8  syz-executor.3
  822     1   822     0  Ss      select  0xfffff8003cd84bc0  dhclient
  817     1   817     0  Ss      select  0xfffff8003cc74a40  dhclient
  803   780   803     0  Ss      piperd  0xfffff8003ccfeba0  syz-executor.2
  785   780   785     0  Ss      nanslp  0xffffffff8273d0a0  syz-executor.1
  784   780   784     0  Ss      nanslp  0xffffffff8273d0a0  syz-executor.0
  780   778   778     0  S       (threaded)                  syz-fuzzer
100091                   S       uwait   0xfffff80015d43b00  syz-fuzzer
100118                   S       uwait   0xfffff80015e96b00  syz-fuzzer
100119                   S       uwait   0xfffff80015e96d00  syz-fuzzer
100120                   S       uwait   0xfffff80015e96f00  syz-fuzzer
100121                   S       uwait   0xfffff8003ca70300  syz-fuzzer
100122                   S       uwait   0xfffff8003ca70500  syz-fuzzer
100123                   S       uwait   0xfffff8003ca70700  syz-fuzzer
100124                   S       kqread  0xfffff8003c32b700  syz-fuzzer
102489                   S       uwait   0xfffff8003cf32d00  syz-fuzzer
  778   776   778     0  Ss      pause   0xfffff8003c4760b0  csh
  776   694   776     0  Ss      select  0xfffff8003c5935c0  sshd
  760     1   760     0  Ss+     ttyin   0xfffff800154804b0  getty
  759     1   759     0  Ss+     ttyin   0xfffff80015b054b0  getty
  758     1   758     0  Ss+     ttyin   0xfffff80015b05cb0  getty
  757     1   757     0  Ss+     ttyin   0xfffff80015b0a4b0  getty
  756     1   756     0  Ss+     ttyin   0xfffff80015b0acb0  getty
  755     1   755     0  Ss+     ttyin   0xfffff80015b0d4b0  getty
  754     1   754     0  Ss+     ttyin   0xfffff80015b0dcb0  getty
  753     1   753     0  Ss+     ttyin   0xfffff80015a964b0  getty
  752     1   752     0  Ss+     ttyin   0xfffff80015a96cb0  getty
  698     1   698     0  Ss      nanslp  0xffffffff8273d0a0  cron
  694     1   694     0  Ss      select  0xfffff80015da8d40  sshd
  507     1   507     0  Ss      select  0xfffff80015eefbc0  syslogd
  436     1   436     0  Ss      select  0xfffff80015777d40  devd
  435     1   435    65  Ss      select  0xfffff8001577d140  dhclient
  350     1   350     0  Ss      select  0xfffff80015dae8c0  dhclient
  347     1   347     0  Ss      select  0xfffff80015eefd40  dhclient
   23     0     0     0  DL      vlruwt  0xfffff80015aef000  [vnlru]
   22     0     0     0  DL      syncer  0xffffffff8282c450  [syncer]
   21     0     0     0  DL      (threaded)                  [bufdaemon]
100081                   D       qsleep  0xffffffff8282b520  [bufdaemon]
100088                   D       -       0xffffffff8220aec0  [bufspacedaemon-0]
100095                   D       sdflush 0xfffff80015e89ce8  [/ worker]
   20     0     0     0  DL      psleep  0xffffffff82853388  [vmdaemon]
   19     0     0     0  DL      (threaded)                  [pagedaemon]
100079                   D       psleep  0xffffffff828477f8  [dom0]
100086                   D       launds  0xffffffff82847804  [laundry: dom0]
100087                   D       umarcl  0xffffffff815c42c0  [uma]
   18     0     0     0  DL      -       0xffffffff82571458  [rand_harvestq]
   17     0     0     0  DL      waiting 0xffffffff82e338b8  [sctp_iterator]
   16     0     0     0  DL      pftm    0xffffffff83046410  [pf purge]
   15     0     0     0  DL      -       0xffffffff82828bdc  [soaiod4]
    9     0     0     0  DL      -       0xffffffff82828bdc  [soaiod3]
    8     0     0     0  DL      -       0xffffffff82828bdc  [soaiod2]
    7     0     0     0  DL      -       0xffffffff82828bdc  [soaiod1]
    6     0     0     0  DL      (threaded)                  [cam]
100044                   D       -       0xffffffff82448b40  [doneq0]
100045                   D       -       0xffffffff82448ac0  [async]
100078                   D       -       0xffffffff82448990  [scanner]
   14     0     0     0  DL      seqstat 0xfffff80004d84c88  [sequencer 00]
    5     0     0     0  DL      crypto_ 0xfffff80004d4dd80  [crypto returns 1]
    4     0     0     0  DL      crypto_ 0xfffff80004d4dd30  [crypto returns 0]
    3     0     0     0  DL      crypto_ 0xffffffff82844ce0  [crypto]
   13     0     0     0  DL      (threaded)                  [geom]
100035                   D       -       0xffffffff8271c8e0  [g_event]
100036                   D       -       0xffffffff8271c8e8  [g_up]
100037                   D       -       0xffffffff8271c8f0  [g_down]
    2     0     0     0  DL      (threaded)                  [KTLS]
100028                   D       -       0xfffff80004c71500  [thr_0]
100029                   D       -       0xfffff80004c71580  [thr_1]
   12     0     0     0  RL      (threaded)                  [intr]
100011                   I                                   [swi5: fast taskq]
100014                   I                                   [swi6: task queue]
100016                   I                                   [swi6: Giant taskq]
100030                   I                                   [swi3: vm]
100031                   L      *sctp-tc 0xfffff80015a9aa80  [swi4: clock (0)]
100032                   I                                   [swi4: clock (1)]
100033                   Run     CPU 1                       [swi1: netisr 0]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq10: virtio_pci2]
100061                   I                                   [irq1: atkbd0]
100062                   I                                   [irq12: psm0]
100063                   I                                   [swi0: uart uart++]
100071                   I                                   [swi1: pf send]
100084                   I                                   [swi1: hpts]
100085                   I                                   [swi1: hpts]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   Run     CPU 0                       [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffff80004bd7530  [init]
   10     0     0     0  DL      audit_w 0xffffffff828451f0  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff8271ce70  [swapper]
100005                   D       -       0xfffff80004c7a900  [if_io_tqg_0]
100006                   D       -       0xfffff80004c7a500  [if_io_tqg_1]
100007                   D       -       0xfffff80004c7a100  [if_config_tqg_0]
100008                   D       -       0xfffff80004c71d00  [softirq_0]
100009                   D       -       0xfffff80004c71900  [softirq_1]
100010                   D       -       0xfffff8000414f900  [aiod_kick taskq]
100012                   D       -       0xfffff8000414f100  [kqueue_ctx taskq]
100013                   D       -       0xfffff80004c69d00  [pci_hp taskq]
100015                   D       -       0xfffff80004c69500  [inm_free taskq]
100017                   D       -       0xfffff80004c66d00  [linuxkpi_irq_wq]
100018                   D       -       0xfffff80004c66900  [thread taskq]
100019                   D       -       0xfffff80004c66500  [in6m_free taskq]
100020                   D       -       0xfffff80004c66100  [linuxkpi_short_wq_0]
100021                   D       -       0xfffff80004c66100  [linuxkpi_short_wq_1]
100022                   D       -       0xfffff80004c66100  [linuxkpi_short_wq_2]
100023                   D       -       0xfffff80004c66100  [linuxkpi_short_wq_3]
100024                   D       -       0xfffff80004c5dd00  [linuxkpi_long_wq_0]
100025                   D       -       0xfffff80004c5dd00  [linuxkpi_long_wq_1]
100026                   D       -       0xfffff80004c5dd00  [linuxkpi_long_wq_2]
100027                   D       -       0xfffff80004c5dd00  [linuxkpi_long_wq_3]
100034                   D       -       0xfffff80004c5d900  [firmware taskq]
100038                   D       -       0xfffff80004c5d500  [crypto_0]
100039                   D       -       0xfffff80004c5d500  [crypto_1]
100055                   D       -       0xfffff800153e2d00  [vtnet0 rxq 0]
100056                   D       -       0xfffff800153e2900  [vtnet0 txq 0]
100057                   D       -       0xfffff800153e2500  [vtnet0 rxq 1]
100058                   D       -       0xfffff800153e2100  [vtnet0 txq 1]
100060                   D       vtbslp  0xfffff8001543c900  [virtio_balloon]
100064                   D       -       0xfffff800154bfd00  [mca taskq]
100066                   D       -       0xffffffff81e206e0  [deadlkres]
100074                   D       -       0xfffff80015a9ed00  [acpi_task_0]
100075                   D       -       0xfffff80015a9ed00  [acpi_task_1]
100076                   D       -       0xfffff80015a9ed00  [acpi_task_2]
100077                   D       -       0xfffff80004c5d100  [CAM taskq]
 5849     1   784     0  Z                                   syz-executor.0
 5850     1   784     0  Z                                   syz-executor.0
db> show all locks
Process 9570 (syz-executor.1) thread 0xfffffe00950a3900 (115595)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffff8003cce3518) locked @ /syzkaller/managers/main/kernel/sys/kern/uipc_sockbuf.c:467
Process 12 (intr) thread 0xfffffe00042be900 (100033)
exclusive sleep mutex sctp-inp (inp) r = 0 (0xfffff8004e0c8968) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3333
exclusive rw sctp-info (sctp-info) r = 0 (0xfffffe00041b0308) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3331
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffff8004e0c8988) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3330
exclusive sleep mutex sctp-tcb (tcb) r = 0 (0xfffffe009e366c70) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:1473
db> show malloc
              Type        InUse        MemUse     Requests
         sysctloid        34288        12858K        34355
           pf_hash            5        11560K            5
         sctp_stro          166         7828K         3872
            devbuf         4216         6982K         4244
          tcp_hpts            5         3219K            5
              kobj          326         2608K          486
               pcb          414         2335K        22004
             vtbuf           24         2064K           46
          filedesc          104         1668K        17037
            newblk           10         1037K        17871
          vfscache            3         1035K            3
           subproc          243         1034K         9710
            acpica         1674          649K        54806
          inodedep           56          575K         9345
           callout            2          528K            2
         ufs_quota            1          520K            1
          vfs_hash            1          520K            1
              intr            4          480K            4
               bus          986          377K         3469
         sctp_atcl          360          360K        13989
               BPF           46          337K           97
            linker          348          270K          441
           kdtrace          324          253K        25176
         sctp_atky          538          218K        18356
            DEVFS1          107          214K          124
              umtx          484          212K          484
            module          510          192K          510
         vnet_data            1          176K            1
           tidhash            3          164K            3
           pagedep           10          141K         8668
        tfo_ccache            1          136K            1
         sctp_athm          360          135K        14232
          sctp_map          332          125K         7704
               sem            4          120K            4
               UMA          270          102K          270
     audit_evclass          236           89K          294
          mtx_pool            2           80K            2
          routetbl          154           79K         1634
              temp           35           77K         3741
          syncache            1           76K            1
              vmem            3           72K            5
               msg            4           68K            4
            ifaddr           70           66K           74
          acpitask            1           64K            1
       ddb_capture            1           64K            1
            DEVFS3          126           63K          136
       ether_multi          152           57K          172
        gtaskqueue           18           57K           18
        DEVFS_RULE           56           54K           56
            kqueue          116           52K         9680
              kenv           95           52K           95
      eventhandler          133           50K          133
           ithread           99           43K           99
           pwddesc          110           42K         9586
              rman           84           42K          425
           lltable           47           41K          425
          pf_ifnet           70           39K          492
              ksem           31           38K          394
            KTRACE          100           38K          102
         taskqueue           60           36K           60
              proc            3           34K            3
         ufs_mount            5           34K            6
            bus-sc           33           34K         1684
           devstat            4           33K            4
         hostcache            1           32K            1
               tty           16           32K           16
               shm            1           32K           22
            DEVFSP           85           32K         1556
              cred           29           29K          372
              GEOM           60           29K          489
         in6_multi           65           29K           65
         sctp_timw           56           28K           56
            kbdmux            6           28K            6
             ifnet            7           25K            7
            plimit           24           24K          670
          kcovinfo           64           24K           68
            dirrem           45           23K         8816
         CAM queue            5           21K         1528
             selfd           55           21K       131983
           pf_osfp           54           21K           54
         pfs_nodes           20           20K           20
         proc-args           52           20K          722
             lockf           49           19K          602
              UART           12           18K           12
           session           35           18K           55
       inpcbpolicy           45           17K         8105
          freefile           44           17K         8744
         bmsafemap            2           17K         9209
               rpc            2           16K            2
             shmfd            1           16K           11
       pfs_vncache            1           16K            1
       ufs_dirhash           24           14K           24
            sglist            5           13K            5
               MCA           32           12K           32
           CAM DEV            3           12K          510
                vt           11           11K           11
               osd           29           11K         3749
           acpisem           28           11K           28
           CAM XPT           22           11K          543
            Unitno           27           11K          105
           uidinfo            4           10K           20
           memdesc            1            8K            1
         ipsec-saq            2            8K            2
             evdev            4            8K            4
           acpidev           20            8K           20
             hhook           15            8K           17
            select           19            8K          410
             mount           16            7K          272
         sctp_aadr           16            6K          131
       fpukern_ctx            3            6K            3
          terminal           11            6K           11
          sctp_ifa           14            6K           15
            ip6ndp           12            6K           13
       ipsecpolicy            2            5K            2
 encap_export_host           12            5K           12
             clone            9            5K            9
        local_apic            1            4K            1
           io_apic            1            4K            1
            sahead            1            4K            1
          secasvar            1            4K            1
             nhops            6            4K            8
          pci_link           10            4K           10
               msi            9            4K            9
             DEVFS            9            4K           10
               tun            7            4K            7
             ipsec            3            3K            3
           tcp_fsb            8            3K          873
         sctp_stri            3            3K          598
             mkdir            8            3K        16950
          in_multi            6            3K           10
          nexusdev            7            3K            7
             xform            7            3K         2852
            feeder            7            3K            7
          procdesc            5            3K           14
          freework            5            3K        11261
          sctp_ifn            6            3K           15
               mld            6            3K            6
              igmp            6            3K            6
         toponodes            6            3K            6
            prison            6            3K            6
            isadev            6            3K            6
           softdep            1            2K            1
          freeblks            4            2K         8829
       vnodemarker            2            2K          984
      NFSD session            1            2K            1
         newdirblk            5            2K         8475
             linux            5            2K            6
        CAM periph            4            2K          271
            soname            5            2K        15046
          filecaps            5            2K          113
            crypto            4            2K         1110
            diradd            4            2K         8858
              pfil            4            2K            4
          CAM path            4            2K         1034
filedesc_to_leader            4            2K           23
           tcpfunc            3            2K            3
        loginclass            3            2K            6
    chacha20random            1            1K            1
            vnodes            1            1K            2
           CAM SIM            2            1K            2
              ktls            1            1K           12
              cdev            2            1K            2
        aesni_data            2            1K            2
            ip6opt            2            1K          179
               iov            2            1K        22970
              cpus            2            1K            2
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
           entropy            2            1K           47
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
          pmchooks            1            1K            1
          sctp_vrf            1            1K            1
            apmdev            1            1K            1
 CAM I/O Scheduler            1            1K            1
       ip6_msource            1            1K            4
        ip_msource            1            1K            1
    vnet_data_free            1            1K            1
              vnet            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
          acpiintr            1            1K            1
               pmc            1            1K            1
        vegas data            0            0K           41
            mqdata            0            0K            0
          cdg data            0            0K           37
        dctcp data            0            0K           55
          chd data            0            0K           41
         htcp data            0            0K           17
        cubic data            0            0K           28
          pf_table            0            0K          282
           pf_rule            0            0K          445
           pf_altq            0            0K            0
           pf_temp            0            0K            0
            tcp_do            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         6498
         sctp_iter            0            0K           22
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            9
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           13
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
              iavf            0            0K            0
               ixl            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
             axgbe            0            0K            0
            DEVFS4            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xen_intr            0            0K            0
          xenstore            0            0K            0
         ciss_data            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            qpidrv            0            0K            0
         BACKLIGHT            0            0K            0
               xnb            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
          sysmouse            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            vtfont            0            0K            0
     vm_fictitious            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
               amr            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           UMAHash            0            0K            0
           scsi_cd            0            0K            0
       AHCI driver            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K         6526
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K          488
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K           32
        allocindir            0            0K            0
          indirdep            0            0K         3784
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               agp            0            0K            0
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
          ktls_ocf            0            0K            0
            twsbuf            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
            lkpifw            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
      ip6_moptions            0            0K            6
       in6_mfilter            0            0K           13
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
      twe_commands            0            0K            0
               LRO            0            0K            0
      newreno data            0            0K            2
       ip_moptions            0            0K           18
        in_mfilter            0            0K            4
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K           20
           VN POLL            0            0K            0
      twa_commands            0            0K            0
            statfs            0            0K         8861
     namei_tracker            0            0K            8
       export_host            0            0K            0
        cl_savebuf            0            0K          137
       tcp_log_dev            0            0K          287
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K        37878
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            4
               acl            0            0K            0
          tempbuff            0            0K            0
          mbuf_tag            0            0K          315
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
          ioctlops            0            0K         2069
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
              sbuf            0            0K          304
        md_sectors            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
           md_disk            0            0K            0
              SWAP            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
         sysctltmp            0            0K          875
            sysctl            0            0K            3
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
       aacraid_buf            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
             cache            0            0K            0
            iirbuf            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            3
               pwd            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
         scsi_pass            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8337     971      812916       0     254   38125568        0
        malloc-384    384   34761      59       71403       0      30   13370880        0
           tcp_log    416       0   15066      428144       0     254    6267456        0
       malloc-1024   1024    4126      22        5524       0      16    4247552        0
       malloc-8192   8192     439       6       10056       0       1    3645440        0
              mbuf    256    9135    2760     3084213       0     254    3045120        0
              pbuf   2624       0     957           0       0       2    2511168        0
       malloc-8192   8192     194     103       10212       0       1    2433024        0
        RADIX NODE    144   12638    1107      331597       0      62    1979280        0
          BUF TRIE    144     352   13116       24111       0      62    1939392        0
      malloc-32768  32768      52       6        8517       0       1    1900544        0
      mbuf_cluster   2048     735       1         735       0     254    1507328        0
       UMA Slabs 0    112   13312       5       13312       0     126    1491504        0
        malloc-384    384    2862     548      110303       0      30    1309440        0
    ertt_txseginfo     40    2050   21887     1346104       0     254     957480        0
       malloc-1024   1024     572     328       24516       0      16     921600        0
       tcp_bbr_map    128       0    6820      525379       0     126     872960        0
        malloc-384    384    1735     295       23629       0      30     779520        0
         sctp_asoc   2288     166     134        3852       0     254     686400        0
         FFS inode   1160     560      21        9305       0       8     673960        0
       malloc-2048   2048     164     128        3817       0       8     598016        0
        256 Bucket   2048     255      25       23006       0       8     573440        0
        malloc-512    512     261     843       28424       0      30     565248        0
       malloc-4096   4096      14     121       39058       0       3     552960        0
         VM OBJECT    264    1944      81      143534       0      30     534600        0
      malloc-65536  65536       0       8        3531       0       1     524288        0
           sctp_ep   1280     191     163        9683       0     254     453120        0
            THREAD   1808     209      33       15601       0       8     437536        0
         MAP ENTRY     96    4149     387      551493       0     126     435456        0
      malloc-16384  16384      23       2          34       0       1     409600        0
        malloc-384    384     748     232       16085       0      30     376320        0
       malloc-1024   1024       0     364        6667       0      16     372736        0
            lkpimm    160       1    2324           1       0      62     372000        0
          lkpicurr    160       2    2323           2       0      62     372000        0
       malloc-1024   1024     316      40       17726       0      16     364544        0
       malloc-1024   1024      68     288        9407       0      16     364544        0
            socket    944      99     249       19683       0     254     328512        0
      malloc-65536  65536       4       1         180       0       1     327680        0
      malloc-65536  65536       3       2          47       0       1     327680        0
      malloc-16384  16384      11       8          29       0       1     311296        0
             VNODE    448     598      77        9345       0      30     302400        0
       malloc-2048   2048     124      10         450       0       8     274432        0
        malloc-384    384     658      32       37897       0      30     264960        0
        malloc-384    384     623      67       19894       0      30     264960        0
           VMSPACE   2544      85      11        9548       0       4     244224        0
        sctp_raddr    736     190     140        4682       0     254     242880        0
            DEVCTL   1024       0     216         132       0       0     221184        0
             g_bio    408       0     500      120554       0      30     204000        0
        malloc-384    384     433      67      132966       0      30     192000        0
       mbuf_packet    256     178     557       44028       0     254     188160        0
         UMA Zones    768     242       2         242       0      16     187392        0
         vmem btag     56    3238      62        3238       0     254     184800        0
        malloc-512    512      65     295        8970       0      30     184320        0
       malloc-8192   8192      18       3         652       0       1     172032        0
              PROC   1328     113      16        9573       0       8     171312        0
        malloc-512    512     306      22        2407       0      30     167936        0
      malloc-16384  16384       6       4         268       0       1     163840        0
       S VFS Cache    104    1033     410       10058       0     126     150072        0
       FFS2 dinode    256     560      25        9305       0      62     149760        0
       malloc-1024   1024     131      13         252       0      16     147456        0
         filedesc0   1072     110      23        9586       0       8     142576        0
        sctp_chunk    152     154     730        3135       0     254     134368        0
      malloc-32768  32768       2       2          22       0       1     131072        0
      malloc-16384  16384       7       1          12       0       1     131072        0
            clpbuf   2624       0      48        2514       0      16     125952        0
        128 Bucket   1024      95      28        4992       0      16     125952        0
        malloc-512    512     114     118       21453       0      30     118784        0
          ksiginfo    112     114     930        1109       0     126     116928        0
       malloc-1024   1024      98       6         118       0      16     106496        0
      malloc-32768  32768       1       2           6       0       1      98304        0
      malloc-32768  32768       2       1         122       0       1      98304        0
       malloc-8192   8192       7       5          16       0       1      98304        0
             tcpcb   1080      25      66        3737       0     254      98280        0
         tcp_inpcb    488      28     156        3737       0     254      89792        0
          UMA Kegs    384     227       6         227       0      30      89472        0
        malloc-512    512     108      44        1109       0      30      77824        0
         32 Bucket    256     169     131       21494       0      62      76800        0
        malloc-512    512     132      12         506       0      30      73728        0
     udplite_inpcb    488       6     130        3684       0     254      66368        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2           8       0       1      65536        0
       malloc-8192   8192       2       6        8867       0       1      65536        0
       malloc-2048   2048      29       3          33       0       8      65536        0
       malloc-1024   1024      57       7         903       0      16      65536        0
      vtnet_tx_hdr     24       0    2672     1185810       0     254      64128        0
         64 Bucket    512      58      62       12293       0      30      61440        0
       malloc-8192   8192       2       5         159       0       1      57344        0
        malloc-384    384      69      71        3788       0      30      53760        0
       malloc-4096   4096       9       4        1404       0       2      53248        0
      malloc-16384  16384       0       3          13       0       1      49152        0
      malloc-16384  16384       1       2          12       0       1      49152        0
       malloc-2048   2048       7      17         540       0       8      49152        0
            pcpu-8      8    5661     483       16875       0     254      49152        0
             Files     80     319     281       40862       0     126      48000        0
        malloc-512    512      23      65        9759       0      30      45056        0
      tcp_rack_pcb    832       8      46         873       0      16      44928        0
              pipe    744      23      37         787       0      16      44640        0
       malloc-8192   8192       2       3          21       0       1      40960        0
           DIRHASH   1024      34       2          34       0      16      36864        0
             NAMEI   1024       0      36       55314       0      16      36864        0
       malloc-4096   4096       3       6          34       0       2      36864        0
       malloc-4096   4096       6       3          24       0       2      36864        0
         16 Bucket    144     134     118        3371       0      62      36288        0
         TURNSTILE    136     243       9         243       0      62      34272        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-16384  16384       1       1           5       0       1      32768        0
      malloc-16384  16384       2       0           2       0       1      32768        0
       malloc-4096   4096       2       6         585       0       2      32768        0
       malloc-2048   2048       1      15         106       0       8      32768        0
           pcpu-64     64     487      25         487       0     254      32768        0
       tcp_bbr_pcb    832       0      36        1842       0      16      29952        0
          4 Bucket     48      38     550        4855       0     254      28224        0
             KNOTE    160      28     147      143217       0      62      28000        0
             unpcb    256      21      84        1827       0     254      26880        0
        SLEEPQUEUE     88     243      45         243       0     126      25344        0
       malloc-4096   4096       4       2         304       0       2      24576        0
       malloc-4096   4096       3       3          23       0       2      24576        0
      tcp_rack_map    112      24     192        1721       0     126      24192        0
               PWD     32      56     700        8580       0     254      24192        0
            ttyinq    160     135      15         300       0      62      24000        0
          8 Bucket     80      63     237        2975       0     126      24000        0
           ttyoutq    256      72      18         160       0      62      23040        0
        malloc-512    512       4      36         595       0      30      20480        0
          2 Bucket     32     142     488        4370       0     254      20160        0
            itimer    352       0      55          55       0      30      19360        0
       Mountpoints   2752       2       5           4       0       4      19264        0
       malloc-4096   4096       0       4          19       0       2      16384        0
sctp_stream_msg_out    112      70      74         936       0     254      16128        0
             ripcb    488       5      27         263       0     254      15616        0
         udp_inpcb    488       6      26         421       0     254      15616        0
        sctp_laddr     48      76     176        2683       0     254      12096        0
              kenv    258      15      30        1042       0      30      11610        0
     routing nhops    256      27      18          37       0      62      11520        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-2048   2048       0       4          12       0       8       8192        0
       malloc-2048   2048       0       4        1363       0       8       8192        0
           pf tags    104       0      78          54       0     126       8112        0
           rtentry    176      30      16          37       0      62       8096        0
    tcp_log_bucket    176       0      46           5       0      62       8096        0
             tcptw     88       1      91          10       0     254       8096        0
              PGRP     88      36      56          63       0     126       8096        0
          rl_entry     40     112      90         112       0     254       8080        0
   sctp_asconf_ack     48       0     168          19       0     254       8064        0
             udpcb     32      12     240        4105       0     254       8064        0
              ertt     72      25      87        3737       0     126       8064        0
      ktls_session    128      11      51          11       0     126       7936        0
      tcp_log_node    120       0      66          25       0     126       7920        0
        sctp_readq    152       0      52         282       0     254       7904        0
               AIO    208       0      38          29       0      62       7904        0
           cryptop    280       0      28         211       0      30       7840        0
             AIOCB    552       0      14          37       0      16       7728        0
     FPU_save_area    832       1       8           1       0      16       7488        0
            cpuset    104       7      55         133       0     126       6448        0
         domainset     40       0     126          97       0     254       5040        0
 epoch_record pcpu    256       4      12           4       0      62       4096        0
       malloc-2048   2048       1       1           1       0       8       4096        0
           pcpu-16     16       7     249           7       0     254       4096        0
         hostcache     64       1      62           1       0     254       4032        0
          syncache    168       0      24           4       0     254       4032        0
               ipq     56       0      72           1       0     254       4032        0
              AIOP     32       4     122           4       0     254       4032        0
       UMA Slabs 1    176      11      11          11       0      62       3872        0
            AIOLIO    272       0      14           4       0      30       3808        0
            mqnode    416       3       6           3       0      30       3744        0
        KMAP ENTRY     96      12      27          12       0       0       3744        0
              vmem   1856       1       1           1       0       8       3712        0
           SMR CPU     32       3      60           3       0     254       2016        0
        SMR SHARED     24       3      60           3       0     254       1512        0
       FFS1 dinode    128       0       0           0       0     126          0        0
            da_ccb    544       0       0           0       0      16          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      62          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    304       0       0           0       0       0          0        0
          pf mtags     48       0       0           0       0     254          0        0
          tcpreass     48       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    584       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
  linux_dma_object     24       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/06/15 23:17 freebsd-src d63e6bc25601 990d3cbe console log report ci-freebsd-main panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.
2021/05/26 10:05 freebsd-src 962b3a092616 54f0bcf1 console log report ci-freebsd-main panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.
2021/04/27 23:42 freebsd-src 1b00608b264f 805b5003 console log report ci-freebsd-main panic: _mtx_lock_sleep: recursed on non-recursive mutex sctp-tcb @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.
* Struck through repros no longer work on HEAD.