syzbot


panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE (2)

Status: auto-closed as invalid on 2022/07/19 03:48
Reported-by: syzbot+975c4e6060f6b8b99ab8@syzkaller.appspotmail.com
First crash: 804d, last: 736d
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE (4) 2 18d 22d 0/2 upstream: reported on 2024/04/02 20:13
freebsd panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE (3) 1 161d 161d 0/2 auto-obsoleted due to no activity on 2024/02/14 01:12
freebsd panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE syz 19 1059d 1624d 0/2 closed as dup on 2021/09/07 21:35

Sample crash report:
panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:1187
cpuid = 1
time = 1650426459
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe009a6a72b0
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe009a6a7410
vpanic() at vpanic+0x2b8/frame 0xfffffe009a6a74f0
panic() at panic+0xb5/frame 0xfffffe009a6a75c0
_mtx_destroy() at _mtx_destroy+0x18d/frame 0xfffffe009a6a7600
sctp_inpcb_free() at sctp_inpcb_free+0x1f04/frame 0xfffffe009a6a7690
sctp_close() at sctp_close+0x1fb/frame 0xfffffe009a6a7770
soclose() at soclose+0x3fc/frame 0xfffffe009a6a7880
_fdrop() at _fdrop+0x58/frame 0xfffffe009a6a78b0
closef() at closef+0x689/frame 0xfffffe009a6a7a90
fdescfree() at fdescfree+0xaa4/frame 0xfffffe009a6a7c70
exit1() at exit1+0x882/frame 0xfffffe009a6a7d10
sys_exit() at sys_exit+0x28/frame 0xfffffe009a6a7d30
ia32_syscall() at ia32_syscall+0x419/frame 0xfffffe009a6a7f30
int0x80_syscall_common() at int0x80_syscall_common+0x9c/frame 0xffffd5e0
KDB: enter: panic
[ thread pid 12198 tid 106703 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x270578a(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0x9a6998f15d693966
rdx                        0x1
rbx                          0
rsp         0xfffffe009a6a73f0
rbp         0xfffffe009a6a7410
rsi                          0
rdi         0xffffffff8178169a  vprintf+0x35a
r8                           0
r9                  0xffffffff
r10                          0
r11         0xfffffe009a31afd0
r12         0xfffffe009a31aac0
r13         0xfffffe009a6a7401
r14         0xffffffff82bc7480  .str.26
r15         0xffffffff82bc7480  .str.26
rip         0xffffffff81774c0b  kdb_enter+0x6b
rflags                0x200046  kernload+0x46
kdb_enter+0x6b: movq    $0,0x270578a(%rip)
db> show proc
Process 12198 (syz-executor.2) at 0xfffffe009a44f000:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 793 at 0xfffffe0092d8a548
 ABI: FreeBSD ELF32
 flag: 0x10002000  flag2: 0
 arguments: /root/syz-executor.2 exec
 reaper: 0xfffffe0053ddc000 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffffe009a3399f8
   (map 0xfffffe009a3399f8)
   (map.pmap 0xfffffe009a339ab8)
   (pmap 0xfffffe009a339b20)
 threads: 1
106703                   Run     CPU 1                       syz-executor.2
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
12200   783   783     0  RE                                  syz-executor.0
12199   814   814     0  R       (threaded)                  syz-executor.3
111530                   RunQ                                syz-executor.3
111974                   S       uwait   0xfffffe009a372500  syz-executor.3
12198   793   793     0  RE      CPU 1                       syz-executor.2
12196   784   784     0  R       (threaded)                  syz-executor.1
106237                   RunQ                                syz-executor.1
111976                   S       uwait   0xfffffe009a371c80  syz-executor.1
11527 11512 11527     0  Ss      select  0xfffffe0053dd1dc0  dhclient
11518     1 11518     0  Ss      select  0xfffffe00078528c0  dhclient
11512 11491   430    65  S       select  0xfffffe0007852a40  dhclient
11491   430   430     0  S       wait    0xfffffe008fe97000  sh
11465     1 11465    65  Ss      select  0xfffffe0053dd00c0  dhclient
 6823     1   793     0  SV      uwait   0xfffffe00574c4100  syz-executor.2
 6610     1   793     0  SV      uwait   0xfffffe009a373580  syz-executor.2
 6606     1  6606     0  Ss      select  0xfffffe0053dd0040  dhclient
 6603     1  6603     0  Ss      select  0xfffffe0053dd01c0  dhclient
 6584     1  6584    65  Ss      select  0xfffffe0053dd0c40  dhclient
 6574     1   793     0  SV      uwait   0xfffffe00574c7d80  syz-executor.2
 6568     1   793     0  SV      uwait   0xfffffe0056f43880  syz-executor.2
 6558     1   793     0  SV      uwait   0xfffffe009a254380  syz-executor.2
 6548     1   793     0  SV      uwait   0xfffffe009a253700  syz-executor.2
 6543     1   793     0  SV      uwait   0xfffffe009a253a00  syz-executor.2
 6542     1   814     0  SV      uwait   0xfffffe009a253e00  syz-executor.3
 6533     1   793     0  SV      uwait   0xfffffe00574c7980  syz-executor.2
 6522     1   793     0  SV      uwait   0xfffffe009a253500  syz-executor.2
 6515     1   793     0  SV      uwait   0xfffffe009a253f00  syz-executor.2
 6503     1   793     0  SV      uwait   0xfffffe00574c4300  syz-executor.2
 6497     1   793     0  SV      uwait   0xfffffe00574c4580  syz-executor.2
 6488     1   793     0  SV      uwait   0xfffffe009a253600  syz-executor.2
 6478     1   793     0  SV      uwait   0xfffffe00574c4200  syz-executor.2
 6467     1   793     0  SV      uwait   0xfffffe00574c7c80  syz-executor.2
 6459     1   793     0  SV      uwait   0xfffffe0058dfa700  syz-executor.2
 6451     1   793     0  SV      uwait   0xfffffe00574c4c80  syz-executor.2
 6442     1   793     0  SV      uwait   0xfffffe00574c4000  syz-executor.2
 6433     1   793     0  SV      uwait   0xfffffe009a254180  syz-executor.2
 6424     1   793     0  SV      uwait   0xfffffe00574c5200  syz-executor.2
 6415     1   793     0  SV      uwait   0xfffffe009a254280  syz-executor.2
 6406     1   793     0  SV      uwait   0xfffffe009a254080  syz-executor.2
 6391  6390   783     0  SV      uwait   0xfffffe00574c7880  syz-executor.0
 6390  6389   783     0  DV      ppwait  0xfffffe009a240f88  syz-executor.0
 6389  6388   783     0  DV      ppwait  0xfffffe009a21e4f8  syz-executor.0
 6388  6387   783     0  DV      ppwait  0xfffffe009a2414f8  syz-executor.0
 6387  6386   783     0  DV      ppwait  0xfffffe009a21ea40  syz-executor.0
 6386  6385   783     0  DV      ppwait  0xfffffe009a21ef88  syz-executor.0
 6385  6384   783     0  DV      ppwait  0xfffffe009a241a40  syz-executor.0
 6384  6383   783     0  DV      ppwait  0xfffffe009a21f4f8  syz-executor.0
 6383  6382   783     0  DV      ppwait  0xfffffe009a241f88  syz-executor.0
 6382  6381   783     0  DV      ppwait  0xfffffe009a21fa40  syz-executor.0
 6381  6380   783     0  DV      ppwait  0xfffffe009a1e74f8  syz-executor.0
 6380  6379   783     0  DV      ppwait  0xfffffe009a21ff88  syz-executor.0
 6379  6378   783     0  DV      ppwait  0xfffffe009a1e7a40  syz-executor.0
 6378  6377   783     0  DV      ppwait  0xfffffe009a1e7f88  syz-executor.0
 6377  6376   783     0  DV      ppwait  0xfffffe009a2204f8  syz-executor.0
 6376  6375   783     0  DV      ppwait  0xfffffe009a1e84f8  syz-executor.0
 6375  6374   783     0  DV      ppwait  0xfffffe00965044f8  syz-executor.0
 6374  6373   783     0  DV      ppwait  0xfffffe009a1e8a40  syz-executor.0
 6373  6372   783     0  DV      ppwait  0xfffffe009a1e8f88  syz-executor.0
 6372  6371   783     0  DV      ppwait  0xfffffe0096493f88  syz-executor.0
 6371  6370   783     0  DV      ppwait  0xfffffe00929aaf88  syz-executor.0
 6370  6369   783     0  DV      ppwait  0xfffffe009a1e94f8  syz-executor.0
 6369  6368   783     0  DV      ppwait  0xfffffe0092d88a40  syz-executor.0
 6368  6367   783     0  DV      ppwait  0xfffffe009a1e9a40  syz-executor.0
 6367  6366   783     0  DV      ppwait  0xfffffe00965034f8  syz-executor.0
 6366  6365   783     0  DV      ppwait  0xfffffe0058bd24f8  syz-executor.0
 6365  6364   783     0  DV      ppwait  0xfffffe009a17ca40  syz-executor.0
 6364  6363   783     0  DV      ppwait  0xfffffe009a17cf88  syz-executor.0
 6363  6362   783     0  DV      ppwait  0xfffffe009a1a74f8  syz-executor.0
 6362  6361   783     0  DV      ppwait  0xfffffe009a1a7a40  syz-executor.0
 6361  6360   783     0  DV      ppwait  0xfffffe009a1a7f88  syz-executor.0
 6360  6359   783     0  DV      ppwait  0xfffffe009a1a84f8  syz-executor.0
 6359  6358   783     0  DV      ppwait  0xfffffe009a1a8a40  syz-executor.0
 6358  6357   783     0  DV      ppwait  0xfffffe009a1a8f88  syz-executor.0
 6357  6356   783     0  DV      ppwait  0xfffffe009a14df88  syz-executor.0
 6356  6355   783     0  DV      ppwait  0xfffffe0096503f88  syz-executor.0
 6355  6354   783     0  DV      ppwait  0xfffffe009a17a4f8  syz-executor.0
 6354  6353   783     0  DV      ppwait  0xfffffe009a17aa40  syz-executor.0
 6353  6352   783     0  DV      ppwait  0xfffffe0092d88f88  syz-executor.0
 6352  6351   783     0  DV      ppwait  0xfffffe0096504f88  syz-executor.0
 6351  6350   783     0  DV      ppwait  0xfffffe009a17af88  syz-executor.0
 6350  6349   783     0  DV      ppwait  0xfffffe009a17b4f8  syz-executor.0
 6349  6348   783     0  DV      ppwait  0xfffffe009a17ba40  syz-executor.0
 6348  6347   783     0  DV      ppwait  0xfffffe009a17bf88  syz-executor.0
 6347  6346   783     0  DV      ppwait  0xfffffe009a17c4f8  syz-executor.0
 6346  6345   783     0  DV      ppwait  0xfffffe009a14b4f8  syz-executor.0
 6345  6344   783     0  DV      ppwait  0xfffffe009a14ba40  syz-executor.0
 6344  6343   783     0  DV      ppwait  0xfffffe009a14bf88  syz-executor.0
 6343  6342   783     0  DV      ppwait  0xfffffe009a14c4f8  syz-executor.0
 6342  6341   783     0  DV      ppwait  0xfffffe009a14ca40  syz-executor.0
 6341  6340   783     0  DV      ppwait  0xfffffe009a14cf88  syz-executor.0
 6340  6339   783     0  DV      ppwait  0xfffffe009a14d4f8  syz-executor.0
 6339  6338   783     0  DV      ppwait  0xfffffe009a14da40  syz-executor.0
 6338  6337   783     0  DV      ppwait  0xfffffe008fe98a40  syz-executor.0
 6337  6336   783     0  DV      ppwait  0xfffffe009648ef88  syz-executor.0
 6336  6335   783     0  DV      ppwait  0xfffffe009648e4f8  syz-executor.0
 6335  6334   783     0  DV      ppwait  0xfffffe009648f4f8  syz-executor.0
 6334  6333   783     0  DV      ppwait  0xfffffe00964934f8  syz-executor.0
 6333  6332   783     0  DV      ppwait  0xfffffe008fe964f8  syz-executor.0
 6332  6330   783     0  DV      ppwait  0xfffffe0092d8af88  syz-executor.0
 6330  6328   783     0  DV      ppwait  0xfffffe0053ddaf88  syz-executor.0
 6328  6327   783     0  DV      ppwait  0xfffffe0096493a40  syz-executor.0
 6327  6326   783     0  DV      ppwait  0xfffffe0096503a40  syz-executor.0
 6326  6325   783     0  DV      ppwait  0xfffffe00929aa4f8  syz-executor.0
 6325     1   783     0  DV      ppwait  0xfffffe00929aaa40  syz-executor.0
 6315     1   783     0  SV      uwait   0xfffffe0058dfa300  syz-executor.0
 5261     0     0     0  DL      aiordy  0xfffffe009648fa90  [aiod4]
 5259     0     0     0  DL      aiordy  0xfffffe0092d89548  [aiod3]
 5256     0     0     0  DL      aiordy  0xfffffe008fe98a90  [aiod2]
 5255     0     0     0  DL      aiordy  0xfffffe009648e548  [aiod1]
 2657     1  2657     0  Ss      select  0xfffffe0053dd0ac0  dhclient
 2649     1  2649     0  Ss      select  0xfffffe0053dd0640  dhclient
  814   778   814     0  Rs                                  syz-executor.3
  793   778   793     0  Rs                                  syz-executor.2
  784   778   784     0  Rs                                  syz-executor.1
  783   778   783     0  Rs                                  syz-executor.0
  778   776   776     0  S       (threaded)                  syz-fuzzer
100112                   S       uwait   0xfffffe00574c7100  syz-fuzzer
100114                   S       uwait   0xfffffe0058dfba80  syz-fuzzer
100115                   S       uwait   0xfffffe0058dfbb80  syz-fuzzer
100116                   S       kqread  0xfffffe0007971d00  syz-fuzzer
100117                   S       uwait   0xfffffe0058dfae00  syz-fuzzer
100118                   S       uwait   0xfffffe0058dfbd80  syz-fuzzer
100119                   S       uwait   0xfffffe0058dfbe80  syz-fuzzer
100121                   S       uwait   0xfffffe0058dfaf00  syz-fuzzer
100123                   S       uwait   0xfffffe0058dfb080  syz-fuzzer
  776   774   776     0  Ss      pause   0xfffffe0092d89b40  csh
  774   688   774     0  Ss      select  0xfffffe0053dd1740  sshd
  754     1   754     0  Ss+     ttyin   0xfffffe0056feb4b0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe0056fe68b0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe0056fe6cb0  getty
  751     1   751     0  Ss+     ttyin   0xfffffe0056fe70b0  getty
  750     1   750     0  Ss+     ttyin   0xfffffe0056fe74b0  getty
  749     1   749     0  Ss+     ttyin   0xfffffe0056fe78b0  getty
  748     1   748     0  Ss+     ttyin   0xfffffe0056fe7cb0  getty
  747     1   747     0  Ss+     ttyin   0xfffffe0056fe90b0  getty
  746     1   746     0  Ss+     ttyin   0xfffffe0056fe94b0  getty
  692     1   692     0  Ss      nanslp  0xffffffff83e47a41  cron
  688     1   688     0  Ss      select  0xfffffe0053dd1c40  sshd
  501     1   501     0  Ss      select  0xfffffe0053dd20c0  syslogd
  430     1   430     0  Ss      wait    0xfffffe008fe97a90  devd
  429     1   429    65  Ss      select  0xfffffe0053dd24c0  dhclient
  344     1   344     0  Ss      select  0xfffffe0053dd2240  dhclient
  341     1   341     0  Ss      select  0xfffffe00570112c0  dhclient
   17     0     0     0  DL      syncer  0xffffffff83f6d260  [syncer]
   16     0     0     0  DL      vlruwt  0xfffffe0053ddb000  [vnlru]
   15     0     0     0  DL      (threaded)                  [bufdaemon]
100080                   D       psleep  0xffffffff83f6b860  [bufdaemon]
100083                   D       -       0xffffffff83211f80  [bufspacedaemon-0]
100095                   D       sdflush 0xfffffe0053f698e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83f9f380  [vmdaemon]
    8     0     0     0  DL      (threaded)                  [pagedaemon]
100078                   D       psleep  0xffffffff83f93238  [dom0]
100081                   D       launds  0xffffffff83f93244  [laundry: dom0]
100082                   D       umarcl  0xffffffff81eafd30  [uma]
    7     0     0     0  DL      -       0xffffffff83c03788  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff84978530  [pf purge]
    5     0     0     0  RL      CPU 0                       [sctp_iterator]
    4     0     0     0  DL      (threaded)                  [cam]
100045                   D       -       0xffffffff83aa56c0  [doneq0]
100046                   D       -       0xffffffff83aa5640  [async]
100077                   D       -       0xffffffff83aa54c0  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0056f1ac88  [sequencer 00]
    3     0     0     0  DL      (threaded)                  [crypto]
100041                   D       crypto_ 0xffffffff83f8ea40  [crypto]
100042                   D       crypto_ 0xfffffe0053effd30  [crypto returns 0]
100043                   D       crypto_ 0xfffffe0053effd80  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100036                   D       -       0xffffffff83e1d000  [g_event]
100037                   D       -       0xffffffff83e1d020  [g_up]
100038                   D       -       0xffffffff83e1d040  [g_down]
    2     0     0     0  WL      (threaded)                  [clock]
100030                   I                                   [clock (0)]
100031                   I                                   [clock (1)]
   12     0     0     0  WL      (threaded)                  [intr]
100010                   I                                   [swi6: Giant taskq]
100017                   I                                   [swi5: fast taskq]
100020                   I                                   [swi6: task queue]
100029                   I                                   [swi1: netisr 0]
100032                   I                                   [swi3: busdma]
100033                   I                                   [swi1: hpts]
100034                   I                                   [swi1: hpts]
100047                   I                                   [irq24: virtio_pci0]
100048                   I                                   [irq25: virtio_pci0]
100049                   I                                   [irq26: virtio_pci0]
100050                   I                                   [irq27: virtio_pci0]
100051                   I                                   [irq28: virtio_pci1]
100052                   I                                   [irq29: virtio_pci1]
100053                   I                                   [irq30: virtio_pci1]
100054                   I                                   [irq31: virtio_pci1]
100055                   I                                   [irq32: virtio_pci1]
100060                   I                                   [irq33: virtio_pci2]
100061                   I                                   [irq34: virtio_pci2]
100062                   I                                   [irq35: virtio_pci2]
100064                   I                                   [irq1: atkbd0]
100065                   I                                   [irq12: psm0]
100066                   I                                   [swi0: uart uart++]
100070                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053ddc000  [init]
   10     0     0     0  DL      audit_w 0xffffffff83f8f540  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff83e1da40  [swapper]
100005                   D       -       0xfffffe0007973100  [softirq_0]
100006                   D       -       0xfffffe0007973000  [softirq_1]
100007                   D       -       0xfffffe0007972e00  [if_io_tqg_0]
100008                   D       -       0xfffffe0007972d00  [if_io_tqg_1]
100009                   D       -       0xfffffe0007972c00  [if_config_tqg_0]
100011                   D       -       0xfffffe0007972a00  [aiod_kick taskq]
100012                   D       -       0xfffffe0007972900  [inm_free taskq]
100013                   D       -       0xfffffe0007972800  [linuxkpi_irq_wq]
100014                   D       -       0xfffffe0007972700  [in6m_free taskq]
100015                   D       -       0xfffffe0007972600  [deferred_unmount ta]
100016                   D       -       0xfffffe0007972500  [thread taskq]
100018                   D       -       0xfffffe0007972300  [pci_hp taskq]
100019                   D       -       0xfffffe0007972200  [kqueue_ctx taskq]
100021                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_3]
100035                   D       -       0xfffffe0053f31300  [firmware taskq]
100039                   D       -       0xfffffe0053f31100  [crypto_0]
100040                   D       -       0xfffffe0053f31100  [crypto_1]
100056                   D       -       0xfffffe0053f2e000  [vtnet0 rxq 0]
100057                   D       -       0xfffffe0007974e00  [vtnet0 txq 0]
100058                   D       -       0xfffffe0007974d00  [vtnet0 rxq 1]
100059                   D       -       0xfffffe0007974c00  [vtnet0 txq 1]
100063                   D       vtbslp  0xfffffe0057011800  [virtio_balloon]
100067                   D       -       0xffffffff82bcd301  [deadlkres]
100071                   D       -       0xfffffe0007973200  [mca taskq]
100072                   D       -       0xfffffe00574c3200  [acpi_task_0]
100073                   D       -       0xfffffe00574c3200  [acpi_task_1]
100074                   D       -       0xfffffe00574c3200  [acpi_task_2]
100076                   D       -       0xfffffe0053f30e00  [CAM taskq]
db> show all locks
Process 12200 (syz-executor.0) thread 0xfffffe009a411000 (111554)
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffffe009a4b8440) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_pcb.c:3338
Process 5 (sctp_iterator) thread 0xfffffe0056fd7020 (100068)
exclusive sleep mutex sctp-inp (inp) r = 0 (0xfffffe009626bbb8) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_output.c:6777
db> show malloc
              Type        InUse        MemUse     Requests
         sctp_stro          222        34767K         3404
           pf_hash            5        11524K            5
          tcp_hpts            7         4801K            7
            devbuf         4217         4323K         4245
         sysctloid        35322         2081K        35393
             vtbuf           24         1968K           46
          filedesc          196         1564K        22519
              kobj          328         1312K          489
          inodedep         1637         1126K        11271
               pcb          474         1045K        29784
            newblk           22         1030K        11746
          vfscache            3         1025K            3
           subproc          317          656K        12370
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
            dirrem         1614          404K        11185
              vmem            3          274K            6
          freefile         1614          202K        11158
            acpica         1674          184K        57552
         sctp_atcl          448          168K        17827
         vnet_data            1          168K            1
           tidhash            3          141K            3
            linker          358          134K          386
           pagedep           22          134K        11183
        tfo_ccache            1          128K            1
            DEVFS1          109          109K          126
               sem            4          106K            4
               bus          994           81K         5207
          mtx_pool            2           72K            2
               BPF           38           71K           38
           kdtrace          392           71K        24178
          syncache            1           68K            1
              umtx          528           66K          528
            module          513           65K          513
          acpitask            1           64K            1
       ddb_capture            1           64K            1
              temp           34           33K         5818
            DEVFS3          128           32K          138
         hostcache            1           32K            1
               shm            1           32K            3
         sctp_timw          120           30K          120
               msg            4           30K            4
         sctp_atky          670           28K        20798
        gtaskqueue           18           26K           18
            kbdmux            6           22K            6
            ifaddr           70           20K           72
        DEVFS_RULE           56           20K           56
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
           ithread          100           16K          100
          routetbl          137           16K          429
            bus-sc           34           15K         1681
           lltable           44           14K           57
       ether_multi          157           13K          181
            KTRACE          100           13K          100
             ifnet            7           13K            7
            kqueue          150           12K        12207
              kenv           95           12K           95
      eventhandler          134           12K          134
              rman           88           11K          431
              GEOM           61           11K          490
         CAM queue            5           11K         1528
           pwddesc          146           10K        12201
         in6_multi           71            9K           71
         bmsafemap            2            9K        11234
              UART           12            9K           12
           devstat            4            9K            4
              ksem            1            8K           49
               rpc            2            8K            2
             shmfd            1            8K            7
       pfs_vncache            1            8K            1
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          296
         sctp_athm          448            7K        17857
          sctp_map          444            7K         5930
         taskqueue           63            7K           63
              cred           26            7K          235
            sglist            5            7K            5
           CAM DEV            3            6K          510
            plimit           23            6K          409
         proc-args          177            5K        13468
       ufs_dirhash           24            5K           24
               UMA          272            5K          272
          pf_ifnet           10            5K           19
            DEVFSP           66            5K          315
                vt           11            5K           11
           memdesc            1            4K            1
               MCA           32            4K           32
             evdev            4            4K            4
           session           31            4K           46
           acpisem           28            4K           28
             hhook           15            4K           17
          kcovinfo           52            4K           52
             lockf           28            3K           47
             selfd           45            3K       167536
          terminal           11            3K           11
           uidinfo            3            3K            9
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         newdirblk           16            2K        11165
         ipsec-saq            2            2K            2
            select           16            2K           49
            ip6ndp           12            2K           15
          sctp_ifa           14            2K           15
            CC Mem           27            2K         1159
            Unitno           29            2K           47
           CAM XPT           22            2K          543
               msi           12            2K           12
          in_multi            6            2K           12
          freework            5            2K        11148
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
       inpcbpolicy           37            2K         1488
             clone            9            2K            9
               tun            7            2K            7
           softdep            1            1K            1
             mkdir            8            1K        22330
          freeblks            4            1K        11147
            sahead            1            1K            1
          secasvar            1            1K            1
             nhops            6            1K            8
       vnodemarker            2            1K           24
      NFSD session            1            1K            1
        CAM periph            4            1K          271
             ipsec            3            1K            3
          sctp_ifn            6            1K           15
               mld            6            1K            6
              igmp            6            1K            6
         toponodes            6            1K            6
               osd           31            1K         1170
            isadev            6            1K            6
             mount           16            1K           89
          pci_link           10            1K           10
            diradd            5            1K        11223
            crypto            4            1K            4
 encap_export_host           12            1K           12
              pfil            4            1K            4
          procdesc            4            1K           16
              cdev            2            1K            2
    chacha20random            1            1K            1
         sctp_iter            1            1K           59
         sctp_cpal            1            1K           47
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           10
            vnodes            1            1K            1
           CAM SIM            2            1K            2
            feeder            7            1K            7
           tcpfunc            3            1K            3
        loginclass            3            1K            6
            prison            6            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
         cryptodev            2            1K           48
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          pmchooks            1            1K            1
            soname            4            1K        10743
          sctp_vrf            1            1K            1
              vnet            1            1K            1
           entropy            2            1K           42
               pmc            1            1K            1
          acpiintr            1            1K            1
          filecaps            4            1K          108
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
            mqdata            0            0K            0
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
           filemon            0            0K          239
          pf_table            0            0K            0
           pf_rule            0            0K          339
           pf_altq            0            0K            0
           pf_osfp            0            0K            0
           pf_temp            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         6696
         sctp_mvrf            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           12
         sctp_aadr            0            0K           38
         sctp_stri            0            0K            0
            tcp_do            0            0K            0
           tcp_fsb            0            0K          463
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
               ixl            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
          xen_intr            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
            DEVFS4            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
     vm_fictitious            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
          xen_acpi            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
           UMAHash            0            0K            0
            vtfont            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K         7693
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K            9
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K            2
        allocindir            0            0K            0
          indirdep            0            0K           10
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
          ktls_ocf            0            0K            0
       AHCI driver            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K           12
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               LRO            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            2
        in_mfilter            0            0K            6
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
               agp            0            0K            0
            statfs            0            0K        11374
     namei_tracker            0            0K            0
       export_host            0            0K            0
        cl_savebuf            0            0K            2
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
       tcp_log_dev            0            0K           97
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            0
               acl            0            0K            0
          mbuf_tag            0            0K          119
              ktls            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        20741
          ioctlops            0            0K         1347
           eventfd            0            0K            3
           Witness            0            0K            0
             stack            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
              sbuf            0            0K          288
          mpr_user            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
            MPRSAS            0            0K            0
              SWAP            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
         sysctltmp            0            0K          750
            sysctl            0            0K            3
        md_sectors            0            0K            0
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
           md_disk            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
          ix_sriov            0            0K            0
             cache            0            0K            0
        aacraidcam            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            1
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
       aacraid_buf            0            0K            0
            aaccam            0            0K            0
         boottrace            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8320    1078     1184811       0     254   38494208        0
              mbuf    256    9125     538     1859273       0     254    2473728        0
           tcp_log    416    4061    1276       11135       0     254    2220192        0
        RADIX NODE    144   13577     642      339012       0      62    2047536        0
              pbuf   2624       0     778           0       0       2    2041472        0
       malloc-4096   4096     477       5       12776       0       2    1974272        0
          BUF TRIE    144     190   11598         706       0      62    1697472        0
      malloc-16384  16384      98       4       11258       0       1    1671168        0
        malloc-384    384    4165       5        4165       0      30    1601280        0
      mbuf_cluster   2048     762       0         762       0     254    1560576        0
        malloc-128    128   11640     233       11646       0     126    1519744        0
       UMA Slabs 0    112   11181       9       11181       0     126    1253280        0
         sctp_asoc   2256     222     288        2965       0     254    1150560        0
         vmem btag     56   19749      90       19752       0     254    1110984        0
        malloc-384    384    1636    1184       11408       0      30    1082880        0
       tcp_bbr_map    128    7202     238       10138       0     126     952320        0
           sctp_ep   1208     227     543       14862       0     254     930160        0
        malloc-256    256     640    2525       35323       0      62     810240        0
        malloc-256    256    1844    1201       12301       0      62     779520        0
            socket    960      63     701       17623       0     254     733440        0
         FFS inode   1160     539      35       11698       0       8     665840        0
         VM OBJECT    264    2403      57      186091       0      30     649440        0
    ertt_txseginfo     40   15197     559       37085       0     254     630240        0
       malloc-2048   2048     235      43       16144       0       9     569344        0
             tcpcb   1104      27     484        1159       0     254     564144        0
            THREAD   1808     243      21       11976       0       8     477312        0
          lkpicurr    168       2    2350           2       0      62     395136        0
        256 Bucket   2048     169      23        5079       0       8     393216        0
            lkpimm    168       1    2327           1       0      62     391104        0
        sctp_raddr    736     226     291        4254       0     254     380512        0
        malloc-128    128    1615    1299       11161       0     126     372992        0
         MAP ENTRY     96    3229     299      424953       0     126     338688        0
      malloc-65536  65536       5       0           5       0       1     327680        0
             VNODE    448     579     114       11740       0      30     310464        0
        malloc-384    384     619     101       18338       0      30     276480        0
       malloc-1024   1024     209      47        2971       0      16     262144        0
         malloc-64     64    3866     229        3868       0     254     262080        0
     FPU_save_area    832     245      43       17402       0      16     239616        0
         malloc-16     16   14608     142       14676       0     254     236000        0
              PROC   1352     147      21       12200       0       8     227136        0
            DEVCTL   1024      15     205         154       0       0     225280        0
      malloc-65536  65536       3       0           3       0       1     196608        0
       mbuf_packet    256     105     657       15459       0     254     195072        0
        malloc-128    128    1361     158       49851       0     126     194432        0
         UMA Zones    768     244       0         244       0      16     187392        0
           VMSPACE   2552      63       9       12097       0       4     183744        0
         malloc-32     32    5323     347        5332       0     254     181440        0
         filedesc0   1072     148      20       12201       0       8     180096        0
        malloc-256    256     111     579       12582       0      62     176640        0
       FFS2 dinode    256     539      91       11697       0      62     161280        0
       S VFS Cache    104    1022     382       12434       0     126     146016        0
      malloc-65536  65536       0       2          60       0       1     131072        0
      malloc-65536  65536       0       2         138       0       1     131072        0
       malloc-1024   1024     120       8         342       0      16     131072        0
        malloc-128    128     850     173        4162       0     126     130944        0
             unpcb    256      17     493        1240       0     254     130560        0
         64 Bucket    512     149      83       10403       0      30     118784        0
        128 Bucket   1024      71      44        1010       0      16     117760        0
          ksiginfo    112     153     891        5721       0     126     116928        0
        malloc-128    128     668     231       25738       0     126     115072        0
         malloc-64     64     920     655      182369       0     254     100800        0
          UMA Kegs    384     230       3         230       0      30      89472        0
             g_bio    408       0     210        6146       0      30      85680        0
         malloc-64     64    1011     312       17161       0     254      84672        0
            clpbuf   2624       0      32          20       0      16      83968        0
         32 Bucket    256     118     197        7453       0      62      80640        0
        sctp_chunk    152     222     298         570       0     254      79040        0
       malloc-8192   8192       2       7         242       0       1      73728        0
       malloc-8192   8192       9       0           9       0       1      73728        0
       malloc-8192   8192       7       2         110       0       1      73728        0
       malloc-4096   4096      15       3          42       0       2      73728        0
       malloc-4096   4096      18       0          18       0       2      73728        0
      malloc-65536  65536       0       1         144       0       1      65536        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2         120       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-16384  16384       4       0           4       0       1      65536        0
       malloc-1024   1024      21      43         547       0      16      65536        0
        sctp_laddr     48     860     484        7820       0     254      64512        0
              pipe    744      22      58         458       0      16      59520        0
         tcp_inpcb    424      27      99        1159       0      30      53424        0
             Files     80     234     416       40283       0     126      52000        0
         TURNSTILE    136     265     113         265       0      62      51408        0
           DIRHASH   1024      34      14          34       0      16      49152        0
             NAMEI   1024       0      48       62582       0      16      49152        0
       malloc-4096   4096      10       2         554       0       2      49152        0
       malloc-2048   2048      12      12          14       0       8      49152        0
       malloc-1024   1024      19      29        1181       0      16      49152        0
        SLEEPQUEUE     88     265     247         265       0     126      45056        0
          syncache    168       0     264           4       0     254      44352        0
         malloc-32     32     854     532       23679       0     254      44352        0
            pcpu-8      8    4688     432        6780       0     254      40960        0
       Mountpoints   2752       2      12           2       0       4      38528        0
             ripcb    424       4      86          73       0      30      38160        0
         udp_inpcb    424       6      84         236       0      30      38160        0
            da_ccb    544       0      70        1648       0      16      38080        0
               PWD     32      44    1090       11251       0     254      36288        0
         malloc-64     64      43     524       15870       0     254      36288        0
         malloc-64     64      26     541          68       0     254      36288        0
         malloc-64     64     175     392         983       0     254      36288        0
         malloc-64     64      11     556          74       0     254      36288        0
         malloc-64     64     154     413       12209       0     254      36288        0
        malloc-128    128      38     241         599       0     126      35712        0
        malloc-128    128      98     181         576       0     126      35712        0
        malloc-128    128      28     251          42       0     126      35712        0
     routing nhops    256      27     108          35       0      62      34560        0
           ttyoutq    256      72      63         160       0      62      34560        0
        malloc-384    384      55      35          77       0      30      34560        0
        malloc-256    256      25     110        1006       0      62      34560        0
        malloc-256    256      10     125       22296       0      62      34560        0
        malloc-256    256       7     128         319       0      62      34560        0
        malloc-256    256      76      59       11875       0      62      34560        0
        malloc-256    256      53      82       11229       0      62      34560        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-16384  16384       2       0          17       0       1      32768        0
      malloc-16384  16384       1       1         161       0       1      32768        0
       malloc-8192   8192       4       0           4       0       1      32768        0
       malloc-4096   4096       2       6       11380       0       2      32768        0
       malloc-2048   2048       7       9           7       0       8      32768        0
       malloc-2048   2048       1      15         181       0       8      32768        0
       malloc-2048   2048       1      15          25       0       8      32768        0
       malloc-2048   2048       2      14         643       0       8      32768        0
       malloc-2048   2048       3      13         344       0       8      32768        0
       malloc-1024   1024       2      30          45       0      16      32768        0
       malloc-1024   1024       4      28           5       0      16      32768        0
       malloc-1024   1024       3      29          11       0      16      32768        0
       malloc-1024   1024       1      31           7       0      16      32768        0
        malloc-512    512       0      64         118       0      30      32768        0
        malloc-512    512       2      62           2       0      30      32768        0
        malloc-512    512       3      61          25       0      30      32768        0
        malloc-512    512       0      64          14       0      30      32768        0
        malloc-512    512      10      54          60       0      30      32768        0
        malloc-512    512       0      64           6       0      30      32768        0
        malloc-512    512       9      55           9       0      30      32768        0
           pcpu-64     64     493      19         493       0     254      32768        0
      tcp_rack_pcb    896       0      36         463       0      16      32256        0
sctp_stream_msg_out    112       0     288         590       0     254      32256        0
             KNOTE    160      28     172      101775       0      62      32000        0
            ttyinq    160     135      65         300       0      62      32000        0
      tcp_rack_map    120       0     264         872       0     126      31680        0
       tcp_bbr_pcb    832      22      14          29       0      16      29952        0
            cpuset    104       7     272           7       0     126      29016        0
   tcp_inpcb ports     32       2     880         744       0     254      28224        0
         16 Bucket    144      73     123         492       0      62      28224        0
          4 Bucket     48       7     581         263       0     254      28224        0
     udplite_inpcb    424       0      63          20       0      30      26712        0
           rtentry    176      31     107          35       0      62      24288        0
              PGRP     88      31     245          46       0     126      24288        0
          rl_entry     40      53     553          53       0     254      24240        0
          8 Bucket     80      38     262        2342       0     126      24000        0
        malloc-384    384       0      60           5       0      30      23040        0
        malloc-384    384      43      17          83       0      30      23040        0
         hostcache     64       1     314           1       0     254      20160        0
             udpcb     32       6     624         256       0     254      20160        0
   udp_inpcb ports     32       3     627          33       0     254      20160        0
              AIOP     32       4     626           4       0     254      20160        0
              ertt     72      27     253        1159       0     126      20160        0
         malloc-32     32      34     596         907       0     254      20160        0
         malloc-32     32     103     527       11334       0     254      20160        0
         malloc-32     32     133     497        3625       0     254      20160        0
         malloc-32     32      81     549         542       0     254      20160        0
         malloc-32     32     110     520         112       0     254      20160        0
         malloc-32     32       7     623          13       0     254      20160        0
          2 Bucket     32      57     573        1427       0     254      20160        0
         malloc-16     16     454     796       17865       0     254      20000        0
             AIOCB    552       0      35           8       0      16      19320        0
 epoch_record pcpu    256       4      60           4       0      62      16384        0
       malloc-8192   8192       0       2          28       0       1      16384        0
       malloc-8192   8192       2       0           2       0       1      16384        0
       malloc-4096   4096       1       3           4       0       2      16384        0
       malloc-4096   4096       0       4         436       0       2      16384        0
       malloc-2048   2048       0       8          12       0       8      16384        0
        malloc-512    512       0      32           1       0      30      16384        0
           SMR CPU     32       7     504           7       0     254      16352        0
      vtnet_tx_hdr     24       0     668      600844       0     254      16032        0
         malloc-16     16     510     490        8632       0     254      16000        0
         malloc-16     16     517     483       33574       0     254      16000        0
               AIO    208       0      76           7       0      62      15808        0
              kenv    258      15      45        1050       0      30      15480        0
            mqnode    416       3      33           3       0      30      14976        0
              vmem   1856       1       7           1       0       8      14848        0
        SMR SHARED     24       7     504           7       0     254      12264        0
         malloc-16     16     151     599       22891       0     254      12000        0
         malloc-16     16      12     738          23       0     254      12000        0
         malloc-16     16      40     710         166       0     254      12000        0
        malloc-384    384      11      19          11       0      30      11520        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-4096   4096       0       2           2       0       2       8192        0
           pcpu-16     16      14     498          14       0     254       8192        0
         malloc-16     16       0     500           2       0     254       8000        0
       UMA Slabs 1    176       9      13           9       0      62       3872        0
        KMAP ENTRY     96      12      27          14       0       0       3744        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0     254          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0     254          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags     56       0       0           0       0     254          0        0
               ipq     56       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     72       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
        sctp_readq    152       0       0           0       0     254          0        0
   tcp_log_id_node    120       0       0           0       0     126          0        0
 tcp_log_id_bucket    176       0       0           0       0      62          0        0
          tcpreass     48       0       0           0       0     254          0        0
       ripcb ports     32       0       0           0       0     254          0        0
udplite_inpcb ports     32       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            itimer    352       0       0           0       0      30          0        0
            AIOLIO    272       0       0           0       0      30          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0           0       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/04/20 03:47 freebsd-src 1f568792c615 7c337266 console log report ci-freebsd-i386 panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE
2022/02/11 08:56 freebsd-src 74f7afdfd2f5 0b33604d console log report ci-freebsd-i386 panic: Assertion mtx_unowned(m) failed at /syzkaller/managers/i386/kernel/sys/kern/kern_mutex.c:LINE
* Struck through repros no longer work on HEAD.