syzbot


panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:LINE (2)

Status: auto-closed as invalid on 2021/09/27 05:56
Reported-by: syzbot+297fd4f82606e0465712@syzkaller.appspotmail.com
First crash: 1058d, last: 1057d
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:LINE 1 1206d 1206d 0/2 auto-closed as invalid on 2021/04/30 19:43
freebsd panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:LINE (3) 1 908d 908d 0/2 auto-closed as invalid on 2022/02/22 23:07

Sample crash report:
panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:1578
cpuid = 1
time = 11553
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe00517399a0
vpanic() at vpanic+0x1cc/frame 0xfffffe0051739a00
panic() at panic+0x43/frame 0xfffffe0051739a60
witness_unlock() at witness_unlock+0x4ae/frame 0xfffffe0051739ae0
__mtx_unlock_flags() at __mtx_unlock_flags+0x70/frame 0xfffffe0051739b20
sctp_iterator_worker() at sctp_iterator_worker+0x713/frame 0xfffffe0051739b90
sctp_iterator_thread() at sctp_iterator_thread+0x5e/frame 0xfffffe0051739bb0
fork_exit() at fork_exit+0xb3/frame 0xfffffe0051739bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0051739bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 17 tid 100072 ]
Stopped at      kdb_enter+0x67: movq    $0,0x1640a0e(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0xffffffff81131e60  vprintf+0x140
rdx                        0x1
rbx                          0
rsp         0xfffffe0051739980
rbp         0xfffffe00517399a0
rsi                          0
rdi         0xffffffff81131e96  vprintf+0x176
r8                           0
r9          0x8080808080808080
r10         0xfffffe0051739870
r11              0x1ffaefff59c
r12         0xffffffff82267d30  ddb_dbbe
r13         0xffffffff8194b400  cam_status_table+0x6bb0
r14         0xffffffff81a74f94
r15         0xffffffff81a74f94
rip         0xffffffff81128ef7  kdb_enter+0x67
rflags                    0x82
kdb_enter+0x67: movq    $0,0x1640a0e(%rip)
db> show proc
Process 17 (sctp_iterator) at 0xfffff80004cad000:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xffffffff8271ce70
 ABI: null
 flag: 0x10000204  flag2: 0
 reaper: 0xffffffff8271ce70 reapsubtree: 17
 sigparent: 20
 vmspace: 0xffffffff8271dad0
   (map 0xffffffff8271dad0)
   (map.pmap 0xffffffff8271db90)
   (pmap 0xffffffff8271dbf0)
 threads: 1
100072                   Run     CPU 1                       [sctp_iterator]
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
25059   786   786     0  R       (threaded)                  syz-executor.1
120745                   Run     CPU 0                       syz-executor.1
129641                   RunQ                                syz-executor.1
24876     1   794     0  S       uwait   0xfffff80030b35100  syz-executor.2
24874     1   794     0  S       uwait   0xfffff80030ed4b00  syz-executor.2
22710     1   786     0  S       uwait   0xfffff8003158dd00  syz-executor.1
22708     1   786     0  S       uwait   0xfffff80030ed4700  syz-executor.1
19771     0     0     0  DL      -       0xffffffff8271e5c4  [accounting]
18256     0     0     0  DL      (threaded)                  [zfskern]
106693                   D       t->zthr 0xfffff8005f5f7148  [arc_evict]
119867                   D       t->zthr 0xfffff80030e5a148  [arc_reap]
119868                   D       -       0xfffff8005fa35100  [dbu_evict]
119869                   D       dbuf_ev 0xffffffff839c76e8  [dbuf_evict_thread]
119872                   D       -       0xfffff8005fa35d00  [z_vdev_file_0]
119873                   D       -       0xfffff8005fa35d00  [z_vdev_file_1]
119874                   D       -       0xfffff8005fa35d00  [z_vdev_file_2]
119875                   D       -       0xfffff8005fa35d00  [z_vdev_file_3]
119876                   D       -       0xfffff8005fa35d00  [z_vdev_file_4]
119877                   D       -       0xfffff8005fa35d00  [z_vdev_file_5]
119878                   D       -       0xfffff8005fa35d00  [z_vdev_file_6]
119879                   D       -       0xfffff8005fa35d00  [z_vdev_file_7]
119880                   D       -       0xfffff8005fa35d00  [z_vdev_file_8]
119881                   D       -       0xfffff8005fa35d00  [z_vdev_file_9]
119882                   D       -       0xfffff8005fa35d00  [z_vdev_file_10]
119883                   D       -       0xfffff8005fa35d00  [z_vdev_file_11]
119884                   D       -       0xfffff8005fa35d00  [z_vdev_file_12]
119885                   D       -       0xfffff8005fa35d00  [z_vdev_file_13]
119886                   D       -       0xfffff8005fa35d00  [z_vdev_file_14]
119887                   D       -       0xfffff8005fa35d00  [z_vdev_file_15]
119888                   D       l2arc_f 0xffffffff83985ce0  [l2arc_feed_thread]
119889                   D       -       0xfffff8005f942100  [zfsvfs]
119890                   S       zevent_ 0xffffffff839c7d88  [sysevent]
18184     1   794     0  S       uwait   0xfffff80030ed4900  syz-executor.2
18181     1   794     0  S       uwait   0xfffff8003158d700  syz-executor.2
16891     1   785     0  S       uwait   0xfffff80030e2ed00  syz-executor.0
16883     1   785     0  S       uwait   0xfffff80037798f00  syz-executor.0
11648     1 11648    65  Ss      select  0xfffff80037881a40  dhclient
10975 10974   785     0  SV      uwait   0xfffff80030a5d500  syz-executor.0
10974     1   785     0  DV      ppwait  0xfffff80030c2ba10  syz-executor.0
 9528     1  9528     0  Ss      select  0xfffff800378fb140  dhclient
 9525     1  9525     0  Ss      select  0xfffff800378812c0  dhclient
 9506     1  9506    65  Ss      select  0xfffff800377b08c0  dhclient
 7090     1   820     0  SV      uwait   0xfffff8005f676500  syz-executor.3
 7080     1   820     0  SV      uwait   0xfffff8003037a500  syz-executor.3
 7073     1   820     0  SV      uwait   0xfffff80030e40700  syz-executor.3
 7061     1   820     0  SV      uwait   0xfffff80030e2eb00  syz-executor.3
 7052     1   820     0  SV      uwait   0xfffff80030b50700  syz-executor.3
 7047     1   820     0  SV      uwait   0xfffff8005f752b00  syz-executor.3
 7036     1   820     0  SV      uwait   0xfffff8005f752f00  syz-executor.3
 7028     1   820     0  SV      uwait   0xfffff80030e40100  syz-executor.3
 7022     1   820     0  SV      uwait   0xfffff80015ee0f00  syz-executor.3
 7009     1   820     0  SV      uwait   0xfffff8003041c900  syz-executor.3
 7004     1   820     0  SV      uwait   0xfffff8003158df00  syz-executor.3
 6995     1   820     0  SV      uwait   0xfffff80030ed4300  syz-executor.3
 6982     1   820     0  SV      uwait   0xfffff80030b38f00  syz-executor.3
 6977     1   820     0  SV      uwait   0xfffff800316c2f00  syz-executor.3
 6968     1  6968     0  Ss      select  0xfffff8003757e8c0  dhclient
 6965     1  6965     0  Ss      select  0xfffff80037881740  dhclient
 6946     1  6946    65  Ss      select  0xfffff800377bd5c0  dhclient
 6942     1   820     0  SV      uwait   0xfffff8005f752900  syz-executor.3
 6933     1   820     0  SV      uwait   0xfffff80030e40900  syz-executor.3
 6924     1   820     0  SV      uwait   0xfffff8003041c700  syz-executor.3
 6915     1   820     0  SV      uwait   0xfffff8005f752d00  syz-executor.3
 6906     1   820     0  SV      uwait   0xfffff80030a5d900  syz-executor.3
 6897     1   820     0  SV      uwait   0xfffff80015ee0b00  syz-executor.3
 5534     1   820     0  S       uwait   0xfffff80030b35900  syz-executor.3
 5530     1   820     0  S       umtxn   0xfffff80030e2e500  syz-executor.3
 4597  4591   794     0  SV      uwait   0xfffff800153d0900  syz-executor.2
 4591     1   794     0  DV      ppwait  0xfffff80015ec5a10  syz-executor.2
 4109     1  4109     0  Ss      select  0xfffff80015effbc0  dhclient
 4106     1  4106     0  Ss      select  0xfffff8005f11ba40  dhclient
 4086     1  4086    65  Ss      select  0xfffff80037777ec0  dhclient
 4010     1   794     0  S       uwait   0xfffff80030a5dd00  syz-executor.2
 4007     1   794     0  S       uwait   0xfffff80030b35700  syz-executor.2
 4005     1   794     0  S       uwait   0xfffff80030b38b00  syz-executor.2
 4003     1   794     0  S       uwait   0xfffff80030e2e900  syz-executor.2
 2246     1   786     0  SV      uwait   0xfffff80030b38700  syz-executor.1
 2241     1   786     0  SV      uwait   0xfffff80030b38300  syz-executor.1
 2232     1   786     0  SV      uwait   0xfffff80030e2e700  syz-executor.1
 2223     1   786     0  SV      uwait   0xfffff80030b35b00  syz-executor.1
 2214     1   786     0  SV      uwait   0xfffff80015ac1300  syz-executor.1
 2213     1   786     0  SV      uwait   0xfffff80030b38900  syz-executor.1
 1140     1  1140     0  Ss      select  0xfffff80030f03bc0  dhclient
 1137     1  1137     0  Ss      select  0xfffff80015efd740  dhclient
  889     0     0     0  DL      aiordy  0xfffff80030ce5530  [aiod4]
  888     0     0     0  DL      aiordy  0xfffff80030b47530  [aiod3]
  887     0     0     0  DL      aiordy  0xfffff80030af4530  [aiod2]
  886     0     0     0  DL      aiordy  0xfffff80030af4a60  [aiod1]
  882     1   785     0  S       uwait   0xfffff80015d12500  syz-executor.0
  880     1   785     0  S       uwait   0xfffff80030a5d700  syz-executor.0
  860   857   820     0  SV      uwait   0xfffff80030a5df00  syz-executor.3
  857     1   820     0  DV      ppwait  0xfffff80030b47f40  syz-executor.3
  820   780   820     0  Rs                                  syz-executor.3
  794   780   794     0  Ss      piperd  0xfffff8003043f000  syz-executor.2
  786   780   786     0  Ss      nanslp  0xffffffff8273d0a1  syz-executor.1
  785   780   785     0  Ss      piperd  0xfffff80015e678b8  syz-executor.0
  780   778   778     0  S       (threaded)                  syz-fuzzer
100113                   S       uwait   0xfffff80015d15d00  syz-fuzzer
100114                   S       uwait   0xfffff80015ee0900  syz-fuzzer
100115                   S       uwait   0xfffff80015d12b00  syz-fuzzer
100116                   S       uwait   0xfffff8003037a700  syz-fuzzer
100117                   S       uwait   0xfffff8003037a900  syz-fuzzer
100118                   S       uwait   0xfffff8003037ab00  syz-fuzzer
100119                   S       uwait   0xfffff80030a5db00  syz-fuzzer
100120                   S       uwait   0xfffff80015d15f00  syz-fuzzer
100121                   S       kqread  0xfffff8003041cb00  syz-fuzzer
  778   776   778     0  Ss      pause   0xfffff80015dcb5e0  csh
  776   694   776     0  Ss      select  0xfffff80015efe2c0  sshd
  760     1   760     0  Ss+     ttyin   0xfffff800154a6cb0  getty
  759     1   759     0  Ss+     ttyin   0xfffff80015b2b4b0  getty
  758     1   758     0  Ss+     ttyin   0xfffff80015b2bcb0  getty
  757     1   757     0  Ss+     ttyin   0xfffff80015b2f4b0  getty
  756     1   756     0  Ss+     ttyin   0xfffff80015b2fcb0  getty
  755     1   755     0  Ss+     ttyin   0xfffff80015b314b0  getty
  754     1   754     0  Ss+     ttyin   0xfffff80015b31cb0  getty
  753     1   753     0  Ss+     ttyin   0xfffff80015abc4b0  getty
  752     1   752     0  Ss+     ttyin   0xfffff80015abccb0  getty
  698     1   698     0  Ss      nanslp  0xffffffff8273d0a1  cron
  694     1   694     0  Ss      select  0xfffff80015efe5c0  sshd
  507     1   507     0  Ss      select  0xfffff8001574a2c0  syslogd
  436     1   436     0  Ss      select  0xfffff80015df55c0  devd
  435     1   435    65  Ss      select  0xfffff80015eff5c0  dhclient
  350     1   350     0  Ss      select  0xfffff80015df5440  dhclient
  347     1   347     0  Ss      select  0xfffff800302f9bc0  dhclient
   23     0     0     0  DL      vlruwt  0xfffff80015b12000  [vnlru]
   22     0     0     0  DL      syncer  0xffffffff8282c450  [syncer]
   21     0     0     0  DL      (threaded)                  [bufdaemon]
100081                   D       qsleep  0xffffffff8282b520  [bufdaemon]
100086                   D       -       0xffffffff8220aec0  [bufspacedaemon-0]
100097                   D       sdflush 0xfffff80004ddd4e8  [/ worker]
   20     0     0     0  DL      psleep  0xffffffff82853388  [vmdaemon]
   19     0     0     0  DL      (threaded)                  [pagedaemon]
100079                   D       psleep  0xffffffff828477f8  [dom0]
100087                   D       launds  0xffffffff82847804  [laundry: dom0]
100088                   D       umarcl  0xffffffff815c5070  [uma]
   18     0     0     0  DL      -       0xffffffff82571458  [rand_harvestq]
   17     0     0     0  RL      CPU 1                       [sctp_iterator]
   16     0     0     0  DL      pftm    0xffffffff82e05410  [pf purge]
   15     0     0     0  DL      -       0xffffffff82828bdc  [soaiod4]
    9     0     0     0  DL      -       0xffffffff82828bdc  [soaiod3]
    8     0     0     0  DL      -       0xffffffff82828bdc  [soaiod2]
    7     0     0     0  DL      -       0xffffffff82828bdc  [soaiod1]
    6     0     0     0  DL      (threaded)                  [cam]
100044                   D       -       0xffffffff82448b40  [doneq0]
100045                   D       -       0xffffffff82448ac0  [async]
100078                   D       -       0xffffffff82448990  [scanner]
   14     0     0     0  DL      seqstat 0xfffff80004d8a488  [sequencer 00]
    5     0     0     0  DL      crypto_ 0xfffff80004d53d80  [crypto returns 1]
    4     0     0     0  DL      crypto_ 0xfffff80004d53d30  [crypto returns 0]
    3     0     0     0  DL      crypto_ 0xffffffff82844ce0  [crypto]
   13     0     0     0  DL      (threaded)                  [geom]
100035                   D       -       0xffffffff8271c8e0  [g_event]
100036                   D       -       0xffffffff8271c8e8  [g_up]
100037                   D       -       0xffffffff8271c8f0  [g_down]
    2     0     0     0  DL      (threaded)                  [KTLS]
100028                   D       -       0xfffff80004c69100  [thr_0]
100029                   D       -       0xfffff80004c69180  [thr_1]
   12     0     0     0  LL      (threaded)                  [intr]
100011                   I                                   [swi5: fast taskq]
100014                   I                                   [swi6: task queue]
100016                   I                                   [swi6: Giant taskq]
100030                   I                                   [swi3: vm]
100031                   I                                   [swi4: clock (0)]
100032                   I                                   [swi4: clock (1)]
100033                   L      *sctp-in 0xfffff80004c71780  [swi1: netisr 0]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq10: virtio_pci2]
100061                   I                                   [irq1: atkbd0]
100062                   I                                   [irq12: psm0]
100063                   I                                   [swi0: uart uart++]
100071                   I                                   [swi1: pf send]
100084                   I                                   [swi1: hpts]
100085                   I                                   [swi1: hpts]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffff80004bda530  [init]
   10     0     0     0  DL      audit_w 0xffffffff828451f0  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff8271ce70  [swapper]
100005                   D       -       0xfffff8000443b500  [if_io_tqg_0]
100006                   D       -       0xfffff8000443b100  [if_io_tqg_1]
100007                   D       -       0xfffff80004c78d00  [if_config_tqg_0]
100008                   D       -       0xfffff80004c78900  [softirq_0]
100009                   D       -       0xfffff80004c78500  [softirq_1]
100010                   D       -       0xfffff80004c78100  [aiod_kick taskq]
100012                   D       -       0xfffff80004c75900  [kqueue_ctx taskq]
100013                   D       -       0xfffff80004c75500  [pci_hp taskq]
100015                   D       -       0xfffff80004c6ed00  [inm_free taskq]
100017                   D       -       0xfffff80004c6e500  [linuxkpi_irq_wq]
100018                   D       -       0xfffff80004c6e100  [thread taskq]
100019                   D       -       0xfffff80004c69d00  [in6m_free taskq]
100020                   D       -       0xfffff80004c69900  [linuxkpi_short_wq_0]
100021                   D       -       0xfffff80004c69900  [linuxkpi_short_wq_1]
100022                   D       -       0xfffff80004c69900  [linuxkpi_short_wq_2]
100023                   D       -       0xfffff80004c69900  [linuxkpi_short_wq_3]
100024                   D       -       0xfffff80004c69500  [linuxkpi_long_wq_0]
100025                   D       -       0xfffff80004c69500  [linuxkpi_long_wq_1]
100026                   D       -       0xfffff80004c69500  [linuxkpi_long_wq_2]
100027                   D       -       0xfffff80004c69500  [linuxkpi_long_wq_3]
100034                   D       -       0xfffff80004cd3d00  [firmware taskq]
100038                   D       -       0xfffff80004cd3900  [crypto_0]
100039                   D       -       0xfffff80004cd3900  [crypto_1]
100055                   D       -       0xfffff800153ecd00  [vtnet0 rxq 0]
100056                   D       -       0xfffff800153ec900  [vtnet0 txq 0]
100057                   D       -       0xfffff800153ec500  [vtnet0 rxq 1]
100058                   D       -       0xfffff800153ec100  [vtnet0 txq 1]
100060                   D       vtbslp  0xfffff80015463700  [virtio_balloon]
100064                   D       -       0xfffff80015469900  [mca taskq]
100066                   D       -       0xffffffff81e218a1  [deadlkres]
100074                   D       -       0xfffff80015ac5100  [acpi_task_0]
100075                   D       -       0xfffff80015ac5100  [acpi_task_1]
100076                   D       -       0xfffff80015ac5100  [acpi_task_2]
100077                   D       -       0xfffff80004cd3500  [CAM taskq]
119861                   D       -       0xfffff8005f8ce100  [system_taskq_0]
119862                   D       -       0xfffff8005f8ce100  [system_taskq_1]
119863                   D       -       0xfffff8005fa35500  [system_delay_taskq_]
119864                   D       -       0xfffff8005fa35500  [system_delay_taskq_]
119865                   D       -       0xfffff8005fa35900  [arc_prune_0]
119866                   D       -       0xfffff8005fa35900  [arc_prune_1]
25062   820   820     0  Z                                   syz-executor.3
db> show all locks
Process 25059 (syz-executor.1) thread 0xfffffe009e148e40 (120745)
exclusive sleep mutex umtxql (umtxql) r = 0 (0xffffffff8275b800) locked @ /syzkaller/managers/i386/kernel/sys/kern/kern_umtx.c:932
Process 17 (sctp_iterator) thread 0xfffffe0051e57560 (100072)
exclusive sleep mutex sctp-inp (inp) r = 0 (0xfffff8005fa89e68) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:1461
exclusive sleep mutex sctp-it (iterator) r = 0 (0xffffffff830de878) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:1458
shared rw sctp-info (sctp-info) r = 0 (0xfffffe00041b4390) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:1457
Process 12 (intr) thread 0xfffffe00042be900 (100033)
shared rw sctp-info (sctp-info) r = 0 (0xfffffe00041b4390) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_pcb.c:2130
db> show malloc
              Type        InUse        MemUse     Requests
         sysctloid        46272        17352K        46339
           pf_hash            5        11560K            5
               pcb         2240        10552K        35971
            devbuf         4216         6982K         4245
            linker          368         5883K          457
           solaris          130         4433K          200
          tcp_hpts            5         3219K            5
              kobj          326         2608K          486
         sctp_stro         1105         2470K         5141
         sctp_atcl         2210         2210K        20357
             vtbuf           24         2064K           46
          filedesc          108         1732K        48129
         sctp_atky         3335         1251K        26214
           subproc          245         1050K        25201
            newblk           14         1039K        30259
          vfscache            3         1035K            3
         sctp_athm         2210          829K        20561
          sctp_map         2210          829K        10164
            acpica         1674          649K        54806
          inodedep           28          547K        24447
           callout            2          528K            2
         ufs_quota            1          520K            1
          vfs_hash            1          520K            1
              intr            4          480K            4
               bus          986          377K         3469
               BPF           46          337K           94
           kdtrace          348          277K        54714
           pf_osfp          691          260K          691
              umtx          528          231K          528
            DEVFS1          108          216K          125
            module          514          193K          514
         vnet_data            1          176K            1
           tidhash            3          164K            3
           pagedep           10          141K        24100
               UMA          365          137K          365
        tfo_ccache            1          136K            1
               sem            4          120K            4
              vmem            3          104K            6
     audit_evclass          236           89K          294
          mtx_pool            2           80K            2
              temp           35           77K         5671
          syncache            1           76K            1
          routetbl          137           73K          433
               msg            4           68K            4
            ifaddr           71           67K           73
          acpitask            1           64K            1
       ddb_capture            1           64K            1
            DEVFS3          127           64K          137
       ether_multi          157           59K          167
        gtaskqueue           18           57K           18
        DEVFS_RULE           56           54K           56
           lltable           59           53K          454
            kqueue          117           52K        25186
              kenv           95           52K           95
      eventhandler          136           51K          136
         taskqueue           78           47K           78
           pwddesc          114           43K        25099
           ithread           99           43K           99
              rman           84           42K          425
            KTRACE          100           38K          104
        kstat_data           17           34K           17
              proc            3           34K            3
         ufs_mount            5           34K            6
            bus-sc           33           34K         1684
         sctp_timw           67           34K           67
           devstat            4           33K            4
         hostcache            1           32K            1
               tty           16           32K           16
               shm            1           32K           32
            DEVFSP           84           32K        15639
         in6_multi           71           31K           71
              GEOM           60           29K          493
            kbdmux            6           28K            6
              cred           26           26K         1638
             ifnet            7           25K            7
            plimit           24           24K          503
          kcovinfo           64           24K           68
       inpcbpolicy           63           24K        10232
         CAM queue            5           21K         1528
             selfd           55           21K       331118
               osd           53           20K         8110
         pfs_nodes           20           20K           20
         proc-args           52           20K          701
              UART           12           18K           12
           session           35           18K           54
              ksem            3           17K          233
         bmsafemap            2           17K        24391
               rpc            2           16K            2
             shmfd            1           16K           70
       pfs_vncache            1           16K            1
       ufs_dirhash           24           14K           24
            sglist            5           13K            5
             lockf           34           13K           81
               MCA           32           12K           32
           CAM DEV            3           12K          510
                vt           11           11K           11
           acpisem           28           11K           28
            Unitno           28           11K           65
           CAM XPT           22           11K          543
          pf_ifnet           11            9K           37
            dirrem           18            9K        24194
           uidinfo            3            9K          992
           memdesc            1            8K            1
         ipsec-saq            2            8K            2
             evdev            4            8K            4
           acpidev           20            8K           20
             hhook           15            8K           17
            select           19            8K          209
             mount           16            7K          126
          freefile           17            7K        24155
       fpukern_ctx            3            6K            3
          terminal           11            6K           11
          sctp_ifa           14            6K           15
            ip6ndp           12            6K           15
       ipsecpolicy            2            5K            2
 encap_export_host           12            5K           12
             clone            9            5K            9
        local_apic            1            4K            1
           io_apic            1            4K            1
            sahead            1            4K            1
          secasvar            1            4K            1
             nhops            6            4K            8
          pci_link           10            4K           10
               msi            9            4K            9
             DEVFS            9            4K           10
               tun            7            4K            7
             ipsec            3            3K            3
             mkdir            8            3K        48034
          in_multi            6            3K            8
       vnodemarker            3            3K          141
            prison            8            3K            8
           tcp_fsb            7            3K         3971
          nexusdev            7            3K            7
            feeder            7            3K            7
          procdesc            5            3K           18
          sctp_ifn            6            3K           15
             xform            6            3K        16769
               mld            6            3K            6
              igmp            6            3K            6
         toponodes            6            3K            6
            isadev            6            3K            6
           softdep            1            2K            1
      NFSD session            1            2K            1
         newdirblk            5            2K        24017
             linux            5            2K            6
        CAM periph            4            2K          271
          filecaps            5            2K          120
            crypto            4            2K            4
            diradd            4            2K        24235
          freefrag            4            2K           52
              pfil            4            2K            4
          CAM path            4            2K         1034
            soname            4            2K        28640
          freework            3            2K        25451
           tcpfunc            3            2K            3
        loginclass            3            2K            6
         sctp_stri            1            1K          516
          freeblks            2            1K        24140
    chacha20random            1            1K            1
            vnodes            1            1K            1
              ktls            1            1K            1
              cdev            2            1K            2
             taskq            2            1K            2
        aesni_data            2            1K            2
              cpus            2            1K            2
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
           entropy            2            1K           48
           CAM SIM            2            1K            2
             sigio            2            1K           39
filedesc_to_leader            2            1K            4
         sctp_iter            1            1K           15
         sctp_cpal            1            1K            5
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
          indirdep            1            1K         1073
          pmchooks            1            1K            1
          sctp_vrf            1            1K            1
            apmdev            1            1K            1
 CAM I/O Scheduler            1            1K            1
    vnet_data_free            1            1K            1
              vnet            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
          acpiintr            1            1K            1
               pmc            1            1K            1
         sfs_nodes            0            0K            0
        zones_data            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K        12570
         sctp_mvrf            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           10
         sctp_aadr            0            0K           28
        cubic data            0            0K           93
            tcp_do            0            0K            0
            mqdata            0            0K            0
          cdg data            0            0K           39
        dctcp data            0            0K           56
         htcp data            0            0K           31
        vegas data            0            0K           84
          chd data            0            0K           42
          pf_table            0            0K            0
           pf_rule            0            0K           88
           pf_altq            0            0K            0
           pf_temp            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
              iavf            0            0K            0
               ixl            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
             axgbe            0            0K            0
            DEVFS4            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xen_intr            0            0K            0
          xenstore            0            0K            0
         ciss_data            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            qpidrv            0            0K            0
         BACKLIGHT            0            0K            0
               xnb            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
          sysmouse            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            vtfont            0            0K            0
     vm_fictitious            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
               amr            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           UMAHash            0            0K            0
           scsi_cd            0            0K            0
       AHCI driver            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K        17510
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           68
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               agp            0            0K            0
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
          ktls_ocf            0            0K            0
            twsbuf            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
            lkpifw            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K          135
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            2
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
      twe_commands            0            0K            0
               LRO            0            0K            0
      newreno data            0            0K            4
        ip_msource            0            0K            0
       ip_moptions            0            0K           50
        in_mfilter            0            0K            2
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            2
      twa_commands            0            0K            0
            statfs            0            0K        24686
     namei_tracker            0            0K            9
       export_host            0            0K            0
        cl_savebuf            0            0K           11
       tcp_log_dev            0            0K         2094
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K         8082
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
            biobuf            0            0K            0
              aios            0            0K           16
               lio            0            0K          108
               acl            0            0K            0
          tempbuff            0            0K            0
          mbuf_tag            0            0K          387
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        23116
          ioctlops            0            0K         2009
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
              sbuf            0            0K          288
        md_sectors            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
           md_disk            0            0K            0
              SWAP            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
         sysctltmp            0            0K          885
            sysctl            0            0K            3
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
       aacraid_buf            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
             cache            0            0K            0
            iirbuf            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
         scsi_pass            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8385     922      891959       0     254   38121472        0
           tcp_log    416       0   69336      408128       0     254   28843776        0
        malloc-384    384   49698      32       61553       0      30   19096320        0
       malloc-8192   8192    1106      15       14906       0       1    9183232        0
       malloc-1024   1024    4179       9        4252       0      16    4288512        0
              mbuf    256   10483    3917     2479123       0     254    3686400        0
       malloc-8192   8192     328       2        1099       0       1    2703360        0
        malloc-384    384    6725      85      414634       0      30    2615040        0
         sctp_asoc   2288    1105      20        5082       0     254    2574000        0
              pbuf   2624       0     957           0       0       2    2511168        0
       malloc-2048   2048    1122      14        6432       0       8    2326528        0
       malloc-1024   1024    2216      32       21907       0      16    2301952        0
      malloc-32768  32768      55       7       24065       0       1    2031616        0
          BUF TRIE    144     218   13250        5076       0      62    1939392        0
        RADIX NODE    144   13061     208      559947       0      62    1910736        0
       malloc-1024   1024     139    1725       20524       0      16    1908736        0
       malloc-1024   1024     136    1684       25517       0      16    1863680        0
       UMA Slabs 0    112   15937       8       15937       0     126    1785840        0
           sctp_ep   1280    1104      51       14900       0     254    1478400        0
      mbuf_cluster   2048     705       1         705       0     254    1445888        0
       malloc-1024   1024    1347      41       44766       0      16    1421312        0
       malloc-8192   8192     117       6       25068       0       1    1007616        0
        malloc-512    512     156    1716       24840       0      30     958464        0
        malloc-384    384     618    1722      115634       0      30     898560        0
        256 Bucket   2048     417      17       27631       0       8     888832        0
        sctp_raddr    736    1117      16        5618       0     254     833888        0
             tcpcb   1080      24     676        8067       0     254     756000        0
            socket    944      97     671       27064       0     254     724992        0
        malloc-384    384    1730      30       53918       0      30     675840        0
         FFS inode   1160     556      25       24713       0       8     673960        0
       tcp_bbr_map    128       4    5235       79698       0     126     670592        0
      malloc-65536  65536       4       6         834       0       1     655360        0
         VM OBJECT    264    2092      83      357565       0      30     574200        0
        malloc-384    384     867     463       12191       0      30     510720        0
            THREAD   1808     232      32       29647       0       8     477312        0
      malloc-16384  16384      25       2          28       0       1     442368        0
        malloc-384    384     419     711      103873       0      30     433920        0
         MAP ENTRY     96    3891     435     1351168       0     126     415296        0
    ertt_txseginfo     40    1682    8519      668473       0     254     408040        0
        sctp_chunk    152    1100    1500        4562       0     254     395200        0
            lkpimm    160       1    2324           1       0      62     372000        0
          lkpicurr    160       2    2323           2       0      62     372000        0
         tcp_inpcb    488      26     678        8067       0     254     343552        0
      malloc-65536  65536       5       0           5       0       1     327680        0
        malloc-384    384     793      17       40338       0      30     311040        0
        malloc-512    512      36     556       31798       0      30     303104        0
             VNODE    448     594      72       24753       0      30     298368        0
       malloc-2048   2048     127       3         154       0       8     266240        0
         UMA Zones    768     337       2         337       0      16     260352        0
           VMSPACE   2544      81      21       25010       0       5     259488        0
         vmem btag     56    4429      95        4429       0     254     253344        0
       malloc-8192   8192      24       4          50       0       1     229376        0
            DEVCTL   1024       0     220         155       0       0     225280        0
      malloc-16384  16384      11       2         247       0       1     212992        0
       malloc-4096   4096       7      38        9043       0       2     184320        0
        malloc-384    384     414      66       43377       0      30     184320        0
       mbuf_packet    256     206     499       39354       0     254     180480        0
        malloc-512    512     195     141       52738       0      30     172032        0
              PROC   1328     114      15       25065       0       8     171312        0
      malloc-16384  16384       4       6          13       0       1     163840        0
        malloc-512    512     283      21       24628       0      30     155648        0
       S VFS Cache    104    1039     404       26661       0     126     150072        0
       FFS2 dinode    256     556      29       24712       0      62     149760        0
       malloc-1024   1024     142       2         152       0      16     147456        0
         filedesc0   1072     114      19       25099       0       8     142576        0
zio_buf_comb_131072 131072       0       1           1       0       1     131072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
      malloc-65536  65536       0       2         176       0       1     131072        0
            clpbuf   2624       0      48         223       0      16     125952        0
          UMA Kegs    384     322       1         322       0      30     124032        0
        128 Bucket   1024      82      33        3612       0      16     117760        0
          ksiginfo    112     114     930        1536       0     126     116928        0
      malloc-16384  16384       2       5          40       0       1     114688        0
      malloc-32768  32768       1       2         121       0       1      98304        0
         32 Bucket    256     250     125       31091       0      62      96000        0
        malloc-512    512     122      38       24892       0      30      81920        0
             g_bio    408       0     170       28404       0      30      69360        0
      malloc-65536  65536       0       1           1       0       1      65536        0
      malloc-32768  32768       1       1          17       0       1      65536        0
      malloc-16384  16384       2       2           7       0       1      65536        0
      malloc-16384  16384       3       1           5       0       1      65536        0
       malloc-8192   8192       3       5          41       0       1      65536        0
       malloc-8192   8192       2       6       24851       0       1      65536        0
       malloc-4096   4096      12       3         213       0       2      61440        0
       malloc-2048   2048      10      20         530       0       8      61440        0
         16 Bucket    144     276     144        6261       0      62      60480        0
      tcp_rack_pcb    832       7      65        3971       0      16      59904        0
       malloc-4096   4096       7       7          84       0       2      57344        0
              ertt     72      24     704        8067       0     126      52416        0
zio_data_buf_16384  16384       0       3           9       0       1      49152        0
      malloc-16384  16384       3       0           3       0       1      49152        0
      malloc-16384  16384       3       0           3       0       1      49152        0
        malloc-512    512      86      10         721       0      30      49152        0
             Files     80     301     299       82532       0     126      48000        0
            pcpu-8      8    5227     405        6329       0     254      45056        0
              pipe    744      26      34        1302       0      16      44640        0
       malloc-4096   4096       8       2          16       0       2      40960        0
       malloc-2048   2048      16       4          25       0       8      40960        0
        malloc-512    512      43      37        1367       0      30      40960        0
        malloc-512    512      24      56        1462       0      30      40960        0
        sctp_laddr     48     797      43        2494       0     254      40320        0
         TURNSTILE    136     265      29         265       0      62      39984        0
       tcp_bbr_pcb    832       2      43        1420       0      16      37440        0
           DIRHASH   1024      34       2          34       0      16      36864        0
             NAMEI   1024       0      36      133234       0      16      36864        0
           pcpu-64     64     492      84         492       0     254      36864        0
         64 Bucket    512      51      21       16230       0      30      36864        0
     zio_buf_16384  16384       0       2          10       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
       malloc-2048   2048       2      14         172       0       8      32768        0
sctp_stream_msg_out    112     195      57        1701       0     254      28224        0
               PWD     32      54     828       24107       0     254      28224        0
        SLEEPQUEUE     88     265      55         265       0     126      28160        0
             KNOTE    160      28     147      253498       0      62      28000        0
     udplite_inpcb    488      22      34         492       0     254      27328        0
         udp_inpcb    488       6      50         512       0     254      27328        0
       malloc-8192   8192       0       3          13       0       1      24576        0
       malloc-8192   8192       1       2          77       0       1      24576        0
       malloc-4096   4096       1       5          10       0       2      24576        0
      tcp_rack_map    112      21     195        6855       0     126      24192        0
            ttyinq    160     135      15         300       0      62      24000        0
          8 Bucket     80      53     247        5705       0     126      24000        0
             ripcb    488       9      39        1161       0     254      23424        0
             unpcb    256      20      70        1881       0     254      23040        0
           ttyoutq    256      72      18         160       0      62      23040        0
       malloc-4096   4096       3       2          80       0       2      20480        0
          2 Bucket     32     117     513        9865       0     254      20160        0
       Mountpoints   2752       2       5           2       0       4      19264        0
       malloc-4096   4096       2       2         791       0       2      16384        0
      vtnet_tx_hdr     24       0     668      536420       0     254      16032        0
       malloc-2048   2048       3       3         163       0       8      12288        0
       malloc-2048   2048       2       4           6       0       8      12288        0
            itimer    352       0      33          37       0      30      11616        0
              kenv    258      15      30        1044       0      30      11610        0
     routing nhops    256      27      18          35       0      62      11520        0
       malloc-2048   2048       4       0           4       0       8       8192        0
       malloc-1024   1024       1       7          25       0      16       8192        0
       malloc-1024   1024       0       8           6       0      16       8192        0
           rtentry    176      31      15          35       0      62       8096        0
    tcp_log_bucket    176       0      46           2       0      62       8096        0
              PGRP     88      35      57          56       0     126       8096        0
          rl_entry     40     120      82         120       0     254       8080        0
   sctp_asconf_ack     48       0     168          26       0     254       8064        0
             udpcb     32      28     224        1004       0     254       8064        0
          4 Bucket     48       1     167         424       0     254       8064        0
      tcp_log_node    120       0      66           4       0     126       7920        0
        sctp_readq    152       1      51          95       0     254       7904        0
               AIO    208       1      37         106       0      62       7904        0
             AIOCB    552       0      14         176       0      16       7728        0
            AIOLIO    272       0      28          54       0      30       7616        0
     FPU_save_area    832       1       8           1       0      16       7488        0
            cpuset    104       7      55          45       0     126       6448        0
         domainset     40       0     126          29       0     254       5040        0
 epoch_record pcpu    256       4      12           4       0      62       4096        0
       malloc-4096   4096       1       0           1       0       2       4096        0
           pcpu-16     16       7     249           7       0     254       4096        0
         hostcache     64       1      62           1       0     254       4032        0
          syncache    168       0      24           4       0     254       4032        0
              AIOP     32       4     122           4       0     254       4032        0
       UMA Slabs 1    176      10      12          10       0      62       3872        0
            mqnode    416       3       6           3       0      30       3744        0
        KMAP ENTRY     96      16      23          19       0       0       3744        0
              vmem   1856       1       1           1       0       8       3712        0
           SMR CPU     32       3      60           3       0     254       2016        0
        SMR SHARED     24       3      60           3       0     254       1512        0
   zfs_znode_cache    472       0       0           0       0      30          0        0
       sio_cache_2    168       0       0           0       0      62          0        0
       sio_cache_1    152       0       0           0       0      62          0        0
       sio_cache_0    136       0       0           0       0      62          0        0
     zil_zcw_cache     80       0       0           0       0     126          0        0
     zil_lwb_cache    360       0       0           0       0      30          0        0
    dmu_buf_impl_t    408       0       0           0       0      30          0        0
         arc_buf_t     64       0       0           0       0     254          0        0
arc_buf_hdr_t_l2only     96       0       0           0       0     126          0        0
arc_buf_hdr_t_full_crypt    424       0       0           0       0      30          0        0
arc_buf_hdr_t_full    360       0       0           0       0      30          0        0
           dnode_t   1040       0       0           0       0       8          0        0
          sa_cache    296       0       0           0       0      30          0        0
         abd_chunk   4096       0       0           0       0       2          0        0
zio_buf_comb_16777216 16777216       0       0           0       0       1          0        0
zio_buf_comb_14680064 14680064       0       0           0       0       1          0        0
zio_buf_comb_12582912 12582912       0       0           0       0       1          0        0
zio_buf_comb_10485760 10485760       0       0           0       0       1          0        0
zio_buf_comb_8388608 8388608       0       0           0       0       1          0        0
zio_buf_comb_7340032 7340032       0       0           0       0       1          0        0
zio_buf_comb_6291456 6291456       0       0           0       0       1          0        0
zio_buf_comb_5242880 5242880       0       0           0       0       1          0        0
zio_buf_comb_4194304 4194304       0       0           0       0       1          0        0
zio_buf_comb_3670016 3670016       0       0           0       0       1          0        0
zio_buf_comb_3145728 3145728       0       0           0       0       1          0        0
zio_buf_comb_2621440 2621440       0       0           0       0       1          0        0
zio_buf_comb_2097152 2097152       0       0           0       0       1          0        0
zio_buf_comb_1835008 1835008       0       0           0       0       1          0        0
zio_buf_comb_1572864 1572864       0       0           0       0       1          0        0
zio_buf_comb_1310720 1310720       0       0           0       0       1          0        0
zio_buf_comb_1048576 1048576       0       0           0       0       1          0        0
zio_buf_comb_917504 917504       0       0           0       0       1          0        0
zio_buf_comb_786432 786432       0       0           0       0       1          0        0
zio_buf_comb_655360 655360       0       0           0       0       1          0        0
zio_buf_comb_524288 524288       0       0           0       0       1          0        0
zio_buf_comb_458752 458752       0       0           0       0       1          0        0
zio_buf_comb_393216 393216       0       0           0       0       1          0        0
zio_buf_comb_327680 327680       0       0           0       0       1          0        0
zio_buf_comb_262144 262144       0       0           0       0       1          0        0
zio_buf_comb_229376 229376       0       0           0       0       1          0        0
zio_buf_comb_196608 196608       0       0           0       0       1          0        0
zio_buf_comb_163840 163840       0       0           0       0       1          0        0
zio_buf_comb_114688 114688       0       0           0       0       1          0        0
zio_buf_comb_98304  98304       0       0           0       0       1          0        0
zio_buf_comb_81920  81920       0       0           0       0       1          0        0
zio_buf_comb_65536  65536       0       0           0       0       1          0        0
zio_buf_comb_57344  57344       0       0           0       0       1          0        0
zio_buf_comb_49152  49152       0       0           0       0       1          0        0
zio_buf_comb_40960  40960       0       0           0       0       1          0        0
zio_buf_comb_32768  32768       0       0           0       0       1          0        0
zio_buf_comb_28672  28672       0       0           0       0       1          0        0
zio_buf_comb_24576  24576       0       0           0       0       1          0        0
zio_buf_comb_20480  20480       0       0           0       0       1          0        0
zio_data_buf_14336  14336       0       0           0       0       1          0        0
     zio_buf_14336  14336       0       0           0       0       1          0        0
zio_data_buf_12288  12288       0       0           0       0       1          0        0
     zio_buf_12288  12288       0       0           0       0       1          0        0
zio_data_buf_10240  10240       0       0           0       0       1          0        0
     zio_buf_10240  10240       0       0           0       0       1          0        0
 zio_data_buf_8192   8192       0       0           0       0       1          0        0
      zio_buf_8192   8192       0       0           0       0       1          0        0
 zio_data_buf_7168   7168       0       0           0       0       1          0        0
      zio_buf_7168   7168       0       0           0       0       1          0        0
 zio_data_buf_6144   6144       0       0           0       0       1          0        0
      zio_buf_6144   6144       0       0           0       0       1          0        0
 zio_data_buf_5120   5120       0       0           0       0       1          0        0
      zio_buf_5120   5120       0       0           0       0       1          0        0
 zio_data_buf_4096   4096       0       0           0       0       2          0        0
      zio_buf_4096   4096       0       0           0       0       2          0        0
 zio_data_buf_3584   3584       0       0           0       0       2          0        0
      zio_buf_3584   3584       0       0           0       0       2          0        0
 zio_data_buf_3072   3072       0       0           0       0       4          0        0
      zio_buf_3072   3072       0       0           0       0       4          0        0
 zio_data_buf_2560   2560       0       0           0       0       4          0        0
      zio_buf_2560   2560       0       0           0       0       4          0        0
 zio_data_buf_2048   2048       0       0           0       0       8          0        0
      zio_buf_2048   2048       0       0           0       0       8          0        0
 zio_data_buf_1536   1536       0       0           0       0       8          0        0
      zio_buf_1536   1536       0       0           0       0       8          0        0
 zio_data_buf_1024   1024       0       0           0       0      16          0        0
      zio_buf_1024   1024       0       0           0       0      16          0        0
  zio_data_buf_512    512       0       0           0       0      30          0        0
       zio_buf_512    512       0       0           0       0      30          0        0
    zio_link_cache     48       0       0           0       0     254          0        0
         zio_cache   1232       0       0           0       0       8          0        0
   ddt_entry_cache    392       0       0           0       0      30          0        0
         ddt_cache  24840       0       0           0       0       1          0        0
metaslab_alloc_trace_cache     72       0       0           0       0     126          0        0
zfs_btree_leaf_cache   4096       0       0           0       0       2          0        0
reference_history_cache      8       0       0           0       0     254          0        0
   reference_cache     40       0       0           0       0     254          0        0
        taskq_zone    192       0       0           0       0      62          0        0
       FFS1 dinode    128       0       0           0       0     126          0        0
            da_ccb    544       0       0           0       0      16          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      62          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    304       0       0           0       0     254          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags     48       0       0           0       0     254          0        0
          tcpreass     48       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     88       0       0           0       0     254          0        0
               ipq     56       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    584       0       0           0       0      16          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     24       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
      ktls_session    128       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/06/29 05:55 freebsd-src f94360971e64 9d2ab5df console log report ci-freebsd-i386 panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:LINE
2021/06/28 07:14 freebsd-src c7f048ab3532 9d2ab5df console log report ci-freebsd-i386 panic: lock (sleep mutex) sctp-tcb not locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:LINE
* Struck through repros no longer work on HEAD.