syzbot


panic: spin lock held too long (2)

Status: auto-closed as invalid on 2022/05/21 05:56
Reported-by: syzbot+273cf67dc87cdb943e22@syzkaller.appspotmail.com
First crash: 1005d, last: 1005d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd panic: spin lock held too long C 1 1467d 1467d 2/2 fixed on 2020/11/19 21:16

Sample crash report:
panic: spin lock held too long
cpuid = 0
time = 1645336436
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe009d80ca50
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe009d80cbb0
vpanic() at vpanic+0x2b8/frame 0xfffffe009d80cc90
panic() at panic+0xb5/frame 0xfffffe009d80cd60
_mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x12c/frame 0xfffffe009d80cd90
thread_lock_flags_() at thread_lock_flags_+0x2b1/frame 0xfffffe009d80ceb0
propagate_priority() at propagate_priority+0xde/frame 0xfffffe009d80cf00
turnstile_wait() at turnstile_wait+0x66d/frame 0xfffffe009d80cf70
__mtx_lock_sleep() at __mtx_lock_sleep+0x46b/frame 0xfffffe009d80d0b0
__mtx_lock_flags() at __mtx_lock_flags+0x231/frame 0xfffffe009d80d190
_sleep() at _sleep+0x65e/frame 0xfffffe009d80d2d0
sbwait() at sbwait+0xa9/frame 0xfffffe009d80d310
sctp_lower_sosend() at sctp_lower_sosend+0x2cc9/frame 0xfffffe009d80d700
sctp_sosend() at sctp_sosend+0x72c/frame 0xfffffe009d80d9a0
sosend() at sosend+0xfc/frame 0xfffffe009d80da10
kern_sendit() at kern_sendit+0x58a/frame 0xfffffe009d80db70
freebsd32_sendmsg() at freebsd32_sendmsg+0x4e6/frame 0xfffffe009d80dd30
ia32_syscall() at ia32_syscall+0x419/frame 0xfffffe009d80df30
int0x80_syscall_common() at int0x80_syscall_common+0x9c/frame 0xfbfdbf78
KDB: enter: panic
[ thread pid 3831 tid 103746 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x2708c3a(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0x5eca45a7cf1ca4db
rdx                    0x3ffff
rbx                          0
rsp         0xfffffe009d80cb90
rbp         0xfffffe009d80cbb0
rsi                    0x40001
rdi         0xffffffff817721da  vprintf+0x35a
r8                           0
r9                  0xffffffff
r10                          0
r11         0xfffffe0058df1c50
r12         0xfffffe009a89c000
r13         0xfffffe009d80cc01
r14         0xffffffff82ba1cc0  .str.18
r15         0xffffffff82ba1cc0  .str.18
rip         0xffffffff8176575b  kdb_enter+0x6b
rflags                0x200046  kernload+0x46
kdb_enter+0x6b: movq    $0,0x2708c3a(%rip)
db> show proc
Process 3831 (syz-executor.0) at 0xfffffe009a3e2548:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 779 at 0xfffffe00924a9548
 ABI: FreeBSD ELF32
 flag: 0x10000080  flag2: 0
 arguments: /root/syz-executor.0 exec
 reaper: 0xfffffe0053dda000 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffffe009a888000
   (map 0xfffffe009a888000)
   (map.pmap 0xfffffe009a8880c0)
   (pmap 0xfffffe009a888128)
 threads: 4
101742                   RunQ                                syz-executor.0
103743                   L      *sctp-tc 0xfffffe009a607b40  syz-executor.0
103746                   L      *so_snd  0xfffffe00078859c0  syz-executor.0
103747                   S       uwait   0xfffffe009a3fe600  syz-executor.0
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 3831   779   779     0  R       (threaded)                  syz-executor.0
101742                   RunQ                                syz-executor.0
103743                   L      *sctp-tc 0xfffffe009a607b40  syz-executor.0
103746                   L      *so_snd  0xfffffe00078859c0  syz-executor.0
103747                   S       uwait   0xfffffe009a3fe600  syz-executor.0
 3826   808   808     0  R       (threaded)                  syz-executor.3
102228                   RunQ                                syz-executor.3
103737                   S       connec  0xfffffe008fe3085a  syz-executor.3
103742                   S       uwait   0xfffffe0057446800  syz-executor.3
 3385     1   808     0  S       uwait   0xfffffe009a605980  syz-executor.3
 3377     1   790     0  S       umtxn   0xfffffe009a606300  syz-executor.2
 3373     1   808     0  S       uwait   0xfffffe009a3ff380  syz-executor.3
 3371     1   779     0  S       uwait   0xfffffe009a377500  syz-executor.0
 3367     1   790     0  S       uwait   0xfffffe009a605100  syz-executor.2
 3363     1   808     0  S       uwait   0xfffffe009a605200  syz-executor.3
 3361     1   779     0  S       umtxn   0xfffffe009a606600  syz-executor.0
 3357     1   780     0  S       uwait   0xfffffe009a605000  syz-executor.1
 3355     1   790     0  S       umtxn   0xfffffe009a376800  syz-executor.2
 3351     1   808     0  S       uwait   0xfffffe009a605400  syz-executor.3
 3349     1   779     0  S       uwait   0xfffffe009a3ff980  syz-executor.0
 3345     1   780     0  S       umtxn   0xfffffe0057446a00  syz-executor.1
 3343     1   790     0  S       uwait   0xfffffe0057446f00  syz-executor.2
 3339     1   808     0  S       uwait   0xfffffe009a605c80  syz-executor.3
 3335     1   780     0  S       uwait   0xfffffe0057446600  syz-executor.1
 3333     1   790     0  S       uwait   0xfffffe0057446c00  syz-executor.2
 3329     1   808     0  S       uwait   0xfffffe009a605700  syz-executor.3
 3325     1   790     0  S       uwait   0xfffffe009a376e80  syz-executor.2
 3319     1   790     0  S       uwait   0xfffffe0057879a80  syz-executor.2
 3274  3268  3274     0  Ss      select  0xfffffe009a605540  dhclient
 3271     1  3271     0  Ss      select  0xfffffe009a6054c0  dhclient
 3268  3261   430    65  S       select  0xfffffe009a376ac0  dhclient
 3261   430   430     0  S       wait    0xfffffe0092727a90  sh
 3251     1  3251    65  Ss      select  0xfffffe009a6056c0  dhclient
 3016     0     0     0  DL      -       0xffffffff83f5b740  [soaiod4]
 3015     0     0     0  DL      -       0xffffffff83f5b740  [soaiod3]
 3014     0     0     0  DL      -       0xffffffff83f5b740  [soaiod2]
 3013     0     0     0  DL      -       0xffffffff83f5b740  [soaiod1]
 2747     0     0     0  DL      mdwait  0xfffffe009a681000  [md32767]
 2300     1  2300     0  Ss      select  0xfffffe0057877040  dhclient
 2297     1  2297     0  Ss      select  0xfffffe009a605d40  dhclient
 2274     1  2274    65  Ss      select  0xfffffe00574466c0  dhclient
 2195     0     0     0  DL      aiordy  0xfffffe009a2fa548  [aiod4]
 2194     0     0     0  DL      aiordy  0xfffffe009a2f9548  [aiod3]
 2193     0     0     0  DL      aiordy  0xfffffe009a36d548  [aiod2]
 2192     0     0     0  DL      aiordy  0xfffffe009a36d000  [aiod1]
 1944     1   808     0  S       uwait   0xfffffe0057878780  syz-executor.3
 1942     1   808     0  S       uwait   0xfffffe009a377200  syz-executor.3
 1938     1   790     0  S       uwait   0xfffffe0057879e80  syz-executor.2
 1937     1   790     0  S       uwait   0xfffffe0057879c80  syz-executor.2
 1934     1   780     0  S       uwait   0xfffffe0057876c00  syz-executor.1
 1933     1   780     0  S       umtxn   0xfffffe0057876700  syz-executor.1
 1931     1   808     0  S       uwait   0xfffffe009a377100  syz-executor.3
 1926     1   808     0  S       uwait   0xfffffe0057879980  syz-executor.3
 1924     1   790     0  S       umtxn   0xfffffe0057876800  syz-executor.2
 1923     1   790     0  S       umtxn   0xfffffe0056f48880  syz-executor.2
 1920     1   780     0  S       uwait   0xfffffe0057876300  syz-executor.1
 1916     1   780     0  S       uwait   0xfffffe0057877600  syz-executor.1
 1915     1   808     0  S       uwait   0xfffffe0057877a00  syz-executor.3
 1912     1   808     0  S       uwait   0xfffffe0057879780  syz-executor.3
 1910     1   790     0  S       uwait   0xfffffe0057876500  syz-executor.2
 1909     1   790     0  S       uwait   0xfffffe0058df2100  syz-executor.2
 1907     1   779     0  S       uwait   0xfffffe0057877100  syz-executor.0
 1906     1   790     0  S       uwait   0xfffffe0058df3380  syz-executor.2
 1905     1   780     0  S       uwait   0xfffffe0057876200  syz-executor.1
 1904     1   808     0  S       uwait   0xfffffe0058df5300  syz-executor.3
 1903     1   790     0  S       uwait   0xfffffe0058df3b00  syz-executor.2
 1902     1   779     0  S       uwait   0xfffffe0058df4180  syz-executor.0
 1901     1   780     0  S       uwait   0xfffffe0058df2200  syz-executor.1
 1898     1   808     0  S       uwait   0xfffffe0057876900  syz-executor.3
 1895     1   780     0  S       umtxn   0xfffffe0058df3480  syz-executor.1
 1894     1   790     0  S       uwait   0xfffffe0057877800  syz-executor.2
 1893     1   779     0  S       uwait   0xfffffe0058df4e80  syz-executor.0
 1892     1   790     0  S       uwait   0xfffffe0058df3580  syz-executor.2
 1891     1   779     0  S       uwait   0xfffffe0058df4580  syz-executor.0
 1890     1   780     0  S       umtxn   0xfffffe0057877900  syz-executor.1
 1889     1   808     0  S       uwait   0xfffffe0057876a00  syz-executor.3
 1887     1   808     0  S       uwait   0xfffffe0057877300  syz-executor.3
 1883     1   790     0  S       uwait   0xfffffe0058df3280  syz-executor.2
 1882     1   780     0  S       uwait   0xfffffe0058df2e00  syz-executor.1
 1881     1   779     0  S       uwait   0xfffffe0057877700  syz-executor.0
 1880     1   780     0  S       uwait   0xfffffe0058df2000  syz-executor.1
 1879     1   790     0  S       uwait   0xfffffe0058df3780  syz-executor.2
 1878     1   808     0  S       uwait   0xfffffe0058df2f00  syz-executor.3
 1877     1   779     0  S       uwait   0xfffffe0057877200  syz-executor.0
 1876     1   808     0  S       uwait   0xfffffe0057878680  syz-executor.3
 1869     1   780     0  S       umtxn   0xfffffe0058df4780  syz-executor.1
 1868     1   780     0  S       uwait   0xfffffe0058df3c00  syz-executor.1
 1859     1   780     0  S       uwait   0xfffffe0058df3880  syz-executor.1
 1858     1   780     0  S       uwait   0xfffffe0057443480  syz-executor.1
 1301     1  1301     0  Ss      select  0xfffffe0057877f40  dhclient
 1298     1  1298     0  Ss      select  0xfffffe0058df21c0  dhclient
  808   774   808     0  Rs                                  syz-executor.3
  790   774   790     0  Ss      piperd  0xfffffe0092d8b000  syz-executor.2
  780   774   780     0  Ss      piperd  0xfffffe0058b2eba0  syz-executor.1
  779   774   779     0  Rs                                  syz-executor.0
  774   772   772     0  S       (threaded)                  syz-fuzzer
100096                   S       uwait   0xfffffe0057444780  syz-fuzzer
100115                   S       uwait   0xfffffe0058df5100  syz-fuzzer
100116                   S       uwait   0xfffffe0058df4d00  syz-fuzzer
100117                   S       kqread  0xfffffe000796f900  syz-fuzzer
100118                   S       uwait   0xfffffe0058df4280  syz-fuzzer
100119                   S       uwait   0xfffffe0058df4a80  syz-fuzzer
100120                   S       uwait   0xfffffe0058df4380  syz-fuzzer
100121                   S       uwait   0xfffffe0058df4480  syz-fuzzer
100122                   S       uwait   0xfffffe0058df3d80  syz-fuzzer
  772   770   772     0  Ss      pause   0xfffffe0058c095f8  csh
  770   688   770     0  Ss      select  0xfffffe0058df3ac0  sshd
  754     1   754     0  Ss+     ttyin   0xfffffe0057466cb0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe00579a6cb0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe00579a70b0  getty
  751     1   751     0  Ss+     ttyin   0xfffffe00579a74b0  getty
  750     1   750     0  Ss+     ttyin   0xfffffe00579a78b0  getty
  749     1   749     0  Ss+     ttyin   0xfffffe0057465cb0  getty
  748     1   748     0  Ss+     ttyin   0xfffffe00579a7cb0  getty
  747     1   747     0  Ss+     ttyin   0xfffffe00579a80b0  getty
  746     1   746     0  Ss+     ttyin   0xfffffe00579a84b0  getty
  692     1   692     0  Ss      nanslp  0xffffffff83e3ba81  cron
  688     1   688     0  Ss      select  0xfffffe0058df3d40  sshd
  501     1   501     0  Ss      select  0xfffffe00574440c0  syslogd
  430     1   430     0  Ss      wait    0xfffffe0058c09000  devd
  429     1   429    65  Ss      select  0xfffffe0058df3e40  dhclient
  344     1   344     0  Ss      select  0xfffffe0058df5040  dhclient
  341     1   341     0  Ss      select  0xfffffe0058df5240  dhclient
   17     0     0     0  DL      syncer  0xffffffff83f61260  [syncer]
   16     0     0     0  DL      vlruwt  0xfffffe0056f7ba90  [vnlru]
   15     0     0     0  DL      (threaded)                  [bufdaemon]
100080                   D       psleep  0xffffffff83f5f860  [bufdaemon]
100083                   D       -       0xffffffff83211f80  [bufspacedaemon-0]
100095                   D       sdflush 0xfffffe00574658e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83f932c0  [vmdaemon]
    8     0     0     0  DL      (threaded)                  [pagedaemon]
100078                   D       psleep  0xffffffff83f87178  [dom0]
100081                   D       launds  0xffffffff83f87184  [laundry: dom0]
100082                   D       umarcl  0xffffffff81e9b4a0  [uma]
    7     0     0     0  RL                                  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff84b65530  [pf purge]
    5     0     0     0  DL      waiting 0xffffffff847d5560  [sctp_iterator]
    4     0     0     0  DL      (threaded)                  [cam]
100045                   D       -       0xffffffff83a99940  [doneq0]
100046                   D       -       0xffffffff83a998c0  [async]
100077                   D       -       0xffffffff83a99740  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0053f88c88  [sequencer 00]
    3     0     0     0  DL      (threaded)                  [crypto]
100041                   D       crypto_ 0xffffffff83f829a0  [crypto]
100042                   D       crypto_ 0xfffffe0053c8ad30  [crypto returns 0]
100043                   D       crypto_ 0xfffffe0053c8ad80  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100036                   D       -       0xffffffff83e111c0  [g_event]
100037                   D       -       0xffffffff83e111e0  [g_up]
100038                   D       -       0xffffffff83e11200  [g_down]
    2     0     0     0  LL      (threaded)                  [clock]
100030                   L      *so_snd  0xfffffe00078859c0  [clock (0)]
100031                   I                                   [clock (1)]
   12     0     0     0  WL      (threaded)                  [intr]
100010                   I                                   [swi6: task queue]
100011                   I                                   [swi6: Giant taskq]
100018                   I                                   [swi5: fast taskq]
100029                   I                                   [swi1: netisr 0]
100032                   I                                   [swi3: busdma]
100033                   I                                   [swi1: hpts]
100034                   I                                   [swi1: hpts]
100047                   I                                   [irq24: virtio_pci0]
100048                   I                                   [irq25: virtio_pci0]
100049                   I                                   [irq26: virtio_pci0]
100050                   I                                   [irq27: virtio_pci0]
100051                   I                                   [irq28: virtio_pci1]
100052                   I                                   [irq29: virtio_pci1]
100053                   I                                   [irq30: virtio_pci1]
100054                   I                                   [irq31: virtio_pci1]
100055                   I                                   [irq32: virtio_pci1]
100060                   I                                   [irq33: virtio_pci2]
100061                   I                                   [irq34: virtio_pci2]
100062                   I                                   [irq35: virtio_pci2]
100064                   I                                   [irq1: atkbd0]
100065                   I                                   [irq12: psm0]
100066                   I                                   [swi0: uart uart++]
100070                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053dda000  [init]
   10     0     0     0  DL      audit_w 0xffffffff83f834a0  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff83e11c00  [swapper]
100005                   D       -       0xfffffe0053e01000  [if_config_tqg_0]
100006                   D       -       0xfffffe0053e00e00  [softirq_0]
100007                   D       -       0xfffffe0053e00d00  [softirq_1]
100008                   D       -       0xfffffe0053e00c00  [if_io_tqg_0]
100009                   D       -       0xfffffe0053e00b00  [if_io_tqg_1]
100012                   D       -       0xfffffe000796d200  [aiod_kick taskq]
100013                   D       -       0xfffffe000796d000  [inm_free taskq]
100014                   D       -       0xfffffe000796cd00  [linuxkpi_irq_wq]
100015                   D       -       0xfffffe000796cb00  [in6m_free taskq]
100016                   D       -       0xfffffe000796c900  [deferred_unmount ta]
100017                   D       -       0xfffffe000796c700  [thread taskq]
100019                   D       -       0xfffffe000796c300  [kqueue_ctx taskq]
100020                   D       -       0xfffffe000796c100  [pci_hp taskq]
100021                   D       -       0xfffffe000796be00  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe000796be00  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe000796be00  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe000796be00  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe000796b900  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe000796b900  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe000796b900  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe000796b900  [linuxkpi_long_wq_3]
100035                   D       -       0xfffffe0053ed0200  [firmware taskq]
100039                   D       -       0xfffffe0053ecfc00  [crypto_0]
100040                   D       -       0xfffffe0053ecfc00  [crypto_1]
100056                   D       -       0xfffffe0053eced00  [vtnet0 rxq 0]
100057                   D       -       0xfffffe0053ecec00  [vtnet0 txq 0]
100058                   D       -       0xfffffe0053eceb00  [vtnet0 rxq 1]
100059                   D       -       0xfffffe0053ecea00  [vtnet0 txq 1]
100063                   D       vtbslp  0xfffffe0053fde400  [virtio_balloon]
100067                   D       -       0xffffffff82ba77a1  [deadlkres]
100071                   D       -       0xfffffe005787d100  [acpi_task_0]
100072                   D       -       0xfffffe005787d100  [acpi_task_1]
100073                   D       -       0xfffffe005787d100  [acpi_task_2]
100074                   D       -       0xfffffe000796e200  [mca taskq]
100076                   D       -       0xfffffe0053ecf500  [CAM taskq]
db> show all locks
Process 3831 (syz-executor.0) thread 0xfffffe009a370c80 (103743)
exclusive sleep mutex so_snd (so_snd) r = 0 (0xfffffe008fe2e160) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_output.c:13066
Process 2 (clock) thread 0xfffffe0053e95c80 (100030)
exclusive sleep mutex sctp-send-tcb (tcbs) r = 0 (0xfffffe0097db4ab0) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_pcb.c:4716
exclusive sleep mutex sctp-tcb (tcb) r = 0 (0xfffffe0097db4a90) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctputil.c:1777
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            5        11524K            5
          tcp_hpts            6         4801K            6
            devbuf         4217         4323K         4246
         sysctloid        35423         2087K        35494
             vtbuf           24         1968K           46
              kobj          326         1304K          496
          filedesc          140         1115K         5737
               pcb          520         1097K         6417
            newblk            7         1026K         7361
          vfscache            3         1025K            3
           subproc          270          555K         3979
          inodedep           19          519K         3233
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
              vmem            3          274K            6
         sctp_stro          192          192K         1332
            acpica         1674          184K        56396
         vnet_data            1          168K            1
         sctp_atcl          441          166K         4616
           tidhash            3          141K            3
            linker          358          134K          386
           pagedep           10          131K         2899
        tfo_ccache            1          128K            1
            DEVFS1          110          110K          127
               sem            4          106K            4
               bus          988           81K         3441
          mtx_pool            2           72K            2
               BPF           38           71K           42
          syncache            1           68K            1
              umtx          528           66K          528
           kdtrace          352           65K         7582
          acpitask            1           64K            1
       ddb_capture            1           64K            1
            module          511           64K          511
              temp           35           35K         2523
            DEVFS3          129           33K          139
         hostcache            1           32K            1
               shm            1           32K            2
               msg            4           30K            4
         sctp_atky          647           27K         6225
        gtaskqueue           18           26K           18
            crypto           26           23K          164
            kbdmux            6           22K            6
            ifaddr           70           20K           72
        DEVFS_RULE           56           20K           56
          routetbl          144           19K          644
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
           ithread          100           16K          100
           lltable           50           16K          100
            bus-sc           34           15K         1651
         sctp_timw           56           14K           56
       ether_multi          157           13K          167
            KTRACE          100           13K          100
             ifnet            7           13K            7
              ksem           31           12K           85
              kenv           95           12K           95
              GEOM           68           12K          536
      eventhandler          133           12K          133
              rman           88           11K          429
            kqueue          127           11K         3868
         CAM queue            5           11K         1528
            DEVFSP          149           10K          607
         in6_multi           71            9K           71
         cryptodev          128            9K          808
           pf_rule            6            9K           42
         bmsafemap            2            9K         3212
              UART           12            9K           12
           devstat            4            9K            4
           filemon            1            8K           43
               rpc            2            8K            2
             shmfd            1            8K            1
       pfs_vncache            1            8K            1
           pwddesc          125            8K         3880
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          296
         sctp_athm          441            7K         4733
         taskqueue           63            7K           63
              cred           25            7K          233
            sglist            5            7K            5
          sctp_map          384            6K         2618
           CAM DEV            3            6K          510
            plimit           23            6K          394
          pf_ifnet           13            5K           88
       ufs_dirhash           24            5K           24
               UMA          273            5K          273
         proc-args          149            5K         5082
                vt           11            5K           11
           md_disk            2            5K            6
          pf_table            2            4K           54
           memdesc            1            4K            1
               MCA           32            4K           32
        md_sectors            1            4K            1
             evdev            4            4K            4
           session           31            4K           45
           acpisem           28            4K           28
             hhook           15            4K           17
          kcovinfo           52            4K           52
             lockf           28            3K           47
            CC Mem           46            3K         1075
             selfd           45            3K        57834
          terminal           11            3K           11
            dirrem           10            3K         3064
           uidinfo            3            3K            8
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         ipsec-saq            2            2K            2
            select           16            2K           76
            ip6ndp           12            2K           15
          sctp_ifa           14            2K           15
       inpcbpolicy           56            2K         1339
            Unitno           27            2K           97
           CAM XPT           22            2K          543
               msi           12            2K           12
          in_multi            6            2K            8
          freefile           10            2K         2963
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
             clone            9            2K            9
               tun            7            2K            7
               osd           50            2K         1030
           softdep            1            1K            1
            sahead            1            1K            1
          secasvar            1            1K            1
             nhops            6            1K            8
       vnodemarker            2            1K          278
      NFSD session            1            1K            1
             mkdir            7            1K         5642
        CAM periph            4            1K          271
          freework            4            1K         4274
             ipsec            3            1K            3
          sctp_ifn            6            1K           15
          freeblks            3            1K         3001
               mld            6            1K            6
              igmp            6            1K            6
         toponodes            6            1K            6
            isadev            6            1K            6
             mount           16            1K          446
          pci_link           10            1K           10
         newdirblk            5            1K         2821
 encap_export_host           12            1K           12
            diradd            4            1K         3101
              pfil            4            1K            4
          procdesc            4            1K           16
              cdev            2            1K            2
           pf_osfp            4            1K            4
    chacha20random            1            1K            1
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           10
            vnodes            1            1K            1
           CAM SIM            2            1K            2
            feeder            7            1K            7
           tcpfunc            3            1K            3
        loginclass            3            1K            6
            prison            6            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
            soname            5            1K         7098
           tcp_fsb            1            1K          315
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          pmchooks            1            1K            1
          sctp_vrf            1            1K            1
              vnet            1            1K            1
           entropy            2            1K           41
               pmc            1            1K            1
          acpiintr            1            1K            1
          filecaps            4            1K          104
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
               iov            1            1K        16876
          p1003.1b            1            1K            1
           pf_altq            0            0K            0
           pf_temp            0            0K            0
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
            tcp_do            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         2183
         sctp_iter            0            0K           41
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K           29
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           12
         sctp_aadr            0            0K           14
         sctp_stri            0            0K          254
            mqdata            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
               ixl            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
          xen_intr            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
            DEVFS4            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            DEVFS2            0            0K            0
            xenbus            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
     vm_fictitious            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
            vtfont            0            0K            0
           UMAHash            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K         2547
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K          136
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K            2
        allocindir            0            0K            0
          indirdep            0            0K         2307
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
       AHCI driver            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
          ktls_ocf            0            0K            0
               agp            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K           78
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
               LRO            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            1
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
           nvme_da            0            0K            0
            statfs            0            0K         3001
     namei_tracker            0            0K            0
       export_host            0            0K            0
        cl_savebuf            0            0K            7
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
       tcp_log_dev            0            0K           97
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
               PUC            0            0K            0
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            8
               acl            0            0K            0
          ppbusdev            0            0K            0
          mbuf_tag            0            0K          131
              ktls            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
          ioctlops            0            0K          262
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
              sbuf            0            0K          356
            mfibuf            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
              SWAP            0            0K            0
           malodev            0            0K            0
         sysctltmp            0            0K          751
            sysctl            0            0K            3
               LED            0            0K            0
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
                ix            0            0K            0
             cache            0            0K            0
            ipsbuf            0            0K            0
       aacraid_buf            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            1
filedesc_to_leader            0            0K           19
               pwd            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           78
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8320    1078      892525       0     254   38494208        0
           tcp_log    416    1589   13657      174006       0     254    6342336        0
              mbuf    256    9277     641     1963912       0     254    2539008        0
              pbuf   2624       0     778           0       0       2    2041472        0
        RADIX NODE    144   12670     877      116599       0      62    1950768        0
          BUF TRIE    144     270   11518        5181       0      62    1697472        0
        malloc-384    384    4117      53        4463       0      30    1601280        0
      mbuf_cluster   2048     762       0         762       0     254    1560576        0
        malloc-128    128   11750     123       11956       0     126    1519744        0
       malloc-4096   4096     326       2         509       0       2    1343488        0
      malloc-16384  16384      74       6        3046       0       1    1310720        0
       UMA Slabs 0    112   11181       9       11181       0     126    1253280        0
         sctp_asoc   2288     192     318        1309       0     254    1166880        0
         vmem btag     56   20180      91       20180       0     254    1135176        0
         VM OBJECT    264    2860     110       71135       0      30     784080        0
         FFS inode   1160     575      34        3538       0       8     706440        0
       tcp_bbr_map    128      57    5120      146675       0     126     662656        0
           sctp_ep   1208     249     261        3160       0     254     616080        0
        256 Bucket   2048     273      17        6222       0       8     593920        0
             tcpcb   1104      46     465        1019       0     254     564144        0
       malloc-2048   2048     252      20        3877       0       8     557056        0
       malloc-4096   4096     125       7        3836       0       2     540672        0
    ertt_txseginfo     40       0   12524      515213       0     254     500960        0
            socket    960     155     353        5798       0     254     487680        0
            THREAD   1808     226      38        3747       0       8     477312        0
         MAP ENTRY     96    4347     315      203217       0     126     447552        0
      malloc-65536  65536       4       2         288       0       1     393216        0
            lkpimm    168       1    2327           1       0      62     391104        0
          lkpicurr    168       2    2326           2       0      62     391104        0
        sctp_raddr    736     192     325        1506       0     254     380512        0
             VNODE    448     615     105        3580       0      30     322560        0
           VMSPACE   2552      99      15        3810       0       4     290928        0
         malloc-64     64    3966     381        4458       0     254     278208        0
        malloc-256    256     279     786        9417       0      62     272640        0
         malloc-16     16   14685     315       14876       0     254     240000        0
        malloc-256    256     312     618        8046       0      62     238080        0
     FPU_save_area    832     228      51        8238       0      16     232128        0
      malloc-32768  32768       0       7        2166       0       1     229376        0
            DEVCTL   1024      16     204         156       0       0     225280        0
       malloc-1024   1024     193      15        1302       0      16     212992        0
      malloc-65536  65536       1       2         110       0       1     196608        0
       mbuf_packet    256     155     607       13601       0     254     195072        0
              PROC   1352     124      20        3833       0       8     194688        0
         UMA Zones    768     245       4         245       0      16     191232        0
        malloc-384    384     441      39        4661       0      30     184320        0
         malloc-32     32    5413     257        5600       0     254     181440        0
        malloc-128    128    1257     138       26440       0     126     178560        0
       FFS2 dinode    256     575     115        3538       0      62     176640        0
        malloc-384    384      37     413        3377       0      30     172800        0
         filedesc0   1072     125      22        3880       0       8     157584        0
       malloc-1024   1024     126      18        1515       0      16     147456        0
        malloc-128    128    1042     105        2490       0     126     146816        0
       S VFS Cache    104    1011     393        4169       0     126     146016        0
        128 Bucket   1024      97      34        1710       0      16     134144        0
      malloc-65536  65536       2       0           2       0       1     131072        0
             unpcb    256      17     493        1266       0     254     130560        0
          ksiginfo    112     135     909        4749       0     126     116928        0
       malloc-2048   2048       4      52         535       0       8     114688        0
        malloc-128    128     518     257        3902       0     126      99200        0
       malloc-8192   8192       7       5         187       0       1      98304        0
       malloc-4096   4096      18       4         664       0       2      90112        0
          UMA Kegs    384     231       2         231       0      30      89472        0
             g_bio    408       0     210       42479       0      30      85680        0
            clpbuf   2624       0      32          40       0      16      83968        0
        malloc-128    128     101     550       14626       0     126      83328        0
        malloc-384    384     172      38         271       0      30      80640        0
        malloc-256    256      73     242        5052       0      62      80640        0
        malloc-256    256     257      58        3177       0      62      80640        0
        sctp_readq    152       0     520          97       0     254      79040        0
        sctp_chunk    152     262     258         697       0     254      79040        0
       malloc-8192   8192       9       0           9       0       1      73728        0
       malloc-4096   4096      18       0          18       0       2      73728        0
         64 Bucket    512     110      26        5463       0      30      69632        0
         malloc-64     64     859     212        5676       0     254      68544        0
         malloc-64     64     568     503        2635       0     254      68544        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2         127       0       1      65536        0
       malloc-1024   1024      14      50         522       0      16      65536        0
         32 Bucket    256      98     157       10169       0      62      65280        0
         tcp_inpcb    424      46     107        1019       0      30      64872        0
sctp_stream_msg_out    112      40     500         358       0     254      60480        0
         malloc-64     64     242     577       58160       0     254      52416        0
         malloc-64     64     466     353        8447       0     254      52416        0
             Files     80     460     190       16539       0     126      52000        0
        malloc-256    256      39     156        3326       0      62      49920        0
        malloc-256    256      51     144        7336       0      62      49920        0
           DIRHASH   1024      34      14          34       0      16      49152        0
             NAMEI   1024       0      48       27338       0      16      49152        0
       malloc-2048   2048      14      10         193       0       8      49152        0
       malloc-2048   2048       4      20          56       0       8      49152        0
       malloc-2048   2048      10      14          21       0       8      49152        0
      tcp_rack_pcb    896       1      53         315       0      16      48384        0
        malloc-384    384      95      25         104       0      30      46080        0
            pcpu-8      8    4783     849        6479       0     254      45056        0
       tcp_bbr_pcb    832      19      35         440       0      16      44928        0
          syncache    168       0     264           4       0     254      44352        0
         TURNSTILE    136     265      50         265       0      62      42840        0
       malloc-8192   8192       5       0           5       0       1      40960        0
       malloc-4096   4096       7       3          11       0       2      40960        0
       Mountpoints   2752       2      12           9       0       4      38528        0
             ripcb    424       4      86          75       0      30      38160        0
         udp_inpcb    424       6      84         232       0      30      38160        0
            da_ccb    544       0      70       10728       0      16      38080        0
              pipe    744      22      28         391       0      16      37200        0
         hostcache     64       6     561           6       0     254      36288        0
         malloc-64     64     192     375       17075       0     254      36288        0
         malloc-64     64       4     563          28       0     254      36288        0
         malloc-64     64       6     561          20       0     254      36288        0
         malloc-32     32     759     375        8030       0     254      36288        0
         16 Bucket    144      84     168         668       0      62      36288        0
        malloc-128    128      79     200         423       0     126      35712        0
        malloc-128    128       2     277         178       0     126      35712        0
        malloc-128    128      21     258         290       0     126      35712        0
     routing nhops    256      27     108          35       0      62      34560        0
           ttyoutq    256      72      63         160       0      62      34560        0
        malloc-256    256      51      84         329       0      62      34560        0
        malloc-256    256      19     116         885       0      62      34560        0
        SLEEPQUEUE     88     265     119         265       0     126      33792        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
       malloc-8192   8192       2       2           4       0       1      32768        0
       malloc-4096   4096       3       5        3008       0       2      32768        0
       malloc-2048   2048       3      13         129       0       8      32768        0
       malloc-2048   2048       3      13          10       0       8      32768        0
       malloc-2048   2048       1      15          14       0       8      32768        0
       malloc-1024   1024       4      28           7       0      16      32768        0
       malloc-1024   1024      19      13          40       0      16      32768        0
       malloc-1024   1024      23       9          37       0      16      32768        0
       malloc-1024   1024       0      32          47       0      16      32768        0
        malloc-512    512       0      64         262       0      30      32768        0
        malloc-512    512       4      60         216       0      30      32768        0
        malloc-512    512       9      55           9       0      30      32768        0
        malloc-512    512       2      62         283       0      30      32768        0
        malloc-512    512       8      56          13       0      30      32768        0
           pcpu-64     64     491      21         491       0     254      32768        0
             KNOTE    160      28     172       55261       0      62      32000        0
            ttyinq    160     135      65         300       0      62      32000        0
      tcp_rack_map    120       2     262        1057       0     126      31680        0
            cpuset    104       7     272          59       0     126      29016        0
        sctp_laddr     48      28     560        1422       0     254      28224        0
   tcp_inpcb ports     32       3     879         740       0     254      28224        0
               PWD     32      60     822        2884       0     254      28224        0
         malloc-32     32     342     540        4107       0     254      28224        0
          4 Bucket     48       7     581         123       0     254      28224        0
          2 Bucket     32      60     822        1059       0     254      28224        0
               AIO    208       0     133          36       0      62      27664        0
     udplite_inpcb    424       0      63          13       0      30      26712        0
       malloc-8192   8192       1       2           8       0       1      24576        0
           rtentry    176      31     107          35       0      62      24288        0
              PGRP     88      31     245          45       0     126      24288        0
          rl_entry     40      96     510          96       0     254      24240        0
         malloc-16     16     990     510        8883       0     254      24000        0
          8 Bucket     80      57     243        1206       0     126      24000        0
         domainset     40       0     567          39       0     254      22680        0
             udpcb     32       6     624         245       0     254      20160        0
udplite_inpcb ports     32       0     630           4       0     254      20160        0
   udp_inpcb ports     32       3     627          33       0     254      20160        0
              AIOP     32       4     626           4       0     254      20160        0
              ertt     72      46     234        1019       0     126      20160        0
         malloc-32     32      13     617         106       0     254      20160        0
         malloc-32     32      33     597        2872       0     254      20160        0
         malloc-32     32      12     618          25       0     254      20160        0
         malloc-32     32      66     564        1657       0     254      20160        0
         malloc-32     32      14     616          99       0     254      20160        0
           cryptop    280       0      70          14       0      30      19600        0
             AIOCB    552       0      35          36       0      16      19320        0
            AIOLIO    272       0      70           4       0      30      19040        0
 epoch_record pcpu    256       4      60           4       0      62      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
       malloc-8192   8192       2       0           2       0       1      16384        0
       malloc-8192   8192       0       2           7       0       1      16384        0
       malloc-1024   1024       8       8           8       0      16      16384        0
        malloc-512    512       0      32           7       0      30      16384        0
        malloc-512    512       0      32           1       0      30      16384        0
        malloc-512    512       1      31           1       0      30      16384        0
           SMR CPU     32       7     504           7       0     254      16352        0
           pf tags    104       1     155           1       0     126      16224        0
      vtnet_tx_hdr     24       0     668      667461       0     254      16032        0
         malloc-16     16     443     557        7812       0     254      16000        0
              kenv    258      15      45        1042       0      30      15480        0
            mqnode    416       3      33           3       0      30      14976        0
              vmem   1856       1       7           1       0       8      14848        0
        SMR SHARED     24       7     504           7       0     254      12264        0
         malloc-16     16      25     725         253       0     254      12000        0
         malloc-16     16      69     681       26666       0     254      12000        0
         malloc-16     16       8     742         154       0     254      12000        0
         malloc-16     16      62     688          97       0     254      12000        0
         malloc-16     16       5     745        1649       0     254      12000        0
        malloc-384    384       1      29           1       0      30      11520        0
        malloc-384    384      27       3          27       0      30      11520        0
       malloc-4096   4096       1       1           1       0       2       8192        0
       malloc-4096   4096       2       0           2       0       2       8192        0
           pcpu-16     16      14     498          14       0     254       8192        0
       UMA Slabs 1    176      10      12          10       0      62       3872        0
        KMAP ENTRY     96      12      27          14       0       0       3744        0
           md32767    512       0       0           0       0      30          0        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      62          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0     254          0        0
          pf mtags     48       0       0           0       0     254          0        0
               ipq     56       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     72       0       0           0       0     254          0        0
      tcp_log_node    120       0       0           0       0     126          0        0
    tcp_log_bucket    176       0       0           0       0      62          0        0
          tcpreass     48       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
       ripcb ports     32       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            itimer    352       0       0           0       0      30          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/02/20 05:55 freebsd-src a4a31271cc7a 3cd800e4 console log report ci-freebsd-i386 panic: spin lock held too long
* Struck through repros no longer work on HEAD.