syzbot


panic: witness_warn (3)

Status: upstream: reported on 2025/07/07 14:13
Reported-by: syzbot+2dc438cf9e06923ac94d@syzkaller.appspotmail.com
First crash: 54d, last: 54d
Similar bugs (2)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd panic: witness_warn (2) 2 29 1386d 1388d 2/2 fixed on 2021/11/13 06:29
freebsd panic: witness_warn 2 syz 1 1902d 1902d 2/2 fixed on 2020/07/01 21:53

Sample crash report:
panic: witness_warn
cpuid = 0
time = 291
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc6/frame 0xfffffe00571ae6d0
kdb_backtrace() at kdb_backtrace+0xd0/frame 0xfffffe00571ae830
vpanic() at vpanic+0x257/frame 0xfffffe00571ae9f0
panic() at panic+0xb5/frame 0xfffffe00571aeab0
witness_warn() at witness_warn+0x989/frame 0xfffffe00571aec10
userret() at userret+0x6bd/frame 0xfffffe00571aed10
amd64_syscall() at amd64_syscall+0x856/frame 0xfffffe00571aef30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00571aef30
--- syscall (0, FreeBSD ELF64, syscall), rip = 0x8245a62ca, rsp = 0x820fae2e8, rbp = 0x820fae330 ---
KDB: enter: panic
[ thread pid 1256 tid 100794 ]
Stopped at      kdb_enter+0x6e: movq    $0,0x25be2c7(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0xfffffe0002bf1850
rdx                          0
rbx         0xffffffff827bec80  .str.27
rsp         0xfffffe00571ae810
rbp         0xfffffe00571ae830
rsi                          0
rdi         0xffffffff81612349  printf+0x149
r8                           0
r9                  0xffffffff
r10                          0
r11                 0x33e1e8fa
r12         0xfffffe005419f000
r13         0xfffffffffffffffe
r14         0xffffffff827bec80  .str.27
r15                          0
rip         0xffffffff815fbece  kdb_enter+0x6e
rflags                    0x46
kdb_enter+0x6e: movq    $0,0x25be2c7(%rip)
db> show proc
Process 1256 (ifconfig) at 0xfffffe005419cae0:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 1241 at 0xfffffe00540045a0
 ABI: FreeBSD ELF64
 flag: 0x10004000  flag2: 0
 arguments: ifconfig tap5 inet6 fe80::05aa
 reaper: 0xfffffe0007809040 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffffe0054196248
   (map 0xfffffe0054196248)
   (map.pmap 0xfffffe00541962e8)
   (pmap 0xfffffe0054196358)
 threads: 1
100794                   Run     CPU 0                       ifconfig
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 1257     1   763     0  R                                   syz-executor
 1256  1241  1241     0  R       CPU 0                       ifconfig
 1253  1251   766     0  S       uwait   0xfffffe00584faa80  syz-executor
 1251   766   766     0  T       (threaded)                  syz-executor
100572                   s                                   syz-executor
100804                   RunQ                                syz-executor
100812                   D       ranged1 0xfffffe006e491a28  syz-executor
100813                   RunQ                                syz-executor
 1245     1   766     0  S       uwait   0xfffffe00584fcf00  syz-executor
 1241   762  1241     0  S       wait    0xfffffe00540045a0  syz-executor
 1230     1   764     0  T       (threaded)                  syz-executor
100761                   s                                   syz-executor
100762                   Run     CPU 1                       syz-executor
 1228     1   764     0  S       uwait   0xfffffe0077eb9700  syz-executor
 1223     1   766     0  S       uwait   0xfffffe0077eb7080  syz-executor
 1221     1  1073     0  S       uwait   0xfffffe0077eb8780  syz-executor
 1219     1   763     0  S       uwait   0xfffffe0077eb7780  syz-executor
 1217     1   763     0  S       uwait   0xfffffe00584fce00  syz-executor
 1215     1   766     0  S       uwait   0xfffffe0077eb7a80  syz-executor
 1211     1  1073     0  S       uwait   0xfffffe0077eb8880  syz-executor
 1206     1   766     0  S       uwait   0xfffffe0077eb8980  syz-executor
 1200     1  1073     0  S       uwait   0xfffffe0077eb8680  syz-executor
 1196     1   763     0  S       uwait   0xfffffe0077eb7180  syz-executor
 1186     1   764     0  S       uwait   0xfffffe0077eb9600  syz-executor
 1183     0     0     0  DL      mdwait  0xfffffe007c39e000  [md5]
 1180     1   763     0  S       uwait   0xfffffe006ddd3400  syz-executor
 1169     1   763     0  S       uwait   0xfffffe0077eb8480  syz-executor
 1164     1  1164     0  Ss+     ttyin   0xfffffe00586314b0  getty
 1163     1  1163     0  Ss+     ttyin   0xfffffe00586310b0  getty
 1162     1  1162     0  Ss+     ttyin   0xfffffe0058630cb0  getty
 1161     1  1161     0  Ss+     ttyin   0xfffffe00586308b0  getty
 1160     1  1160     0  Ss+     ttyin   0xfffffe00586304b0  getty
 1159     1  1159     0  Ss+     ttyin   0xfffffe00586300b0  getty
 1158     1  1158     0  Ss+     ttyin   0xfffffe005862fcb0  getty
 1157     1  1157     0  Ss+     ttyin   0xfffffe0007a7f0b0  getty
 1156     1  1156     0  Ss+     ttyin   0xfffffe005862f8b0  getty
 1153     1  1073    -1  S       uwait   0xfffffe0058274e00  syz-executor
 1144     1   766     0  S       uwait   0xfffffe0077eb9500  syz-executor
 1120     1   764     0  S       uwait   0xfffffe0077eb8000  syz-executor
 1117     1   763     0  S       uwait   0xfffffe00584fb900  syz-executor
 1116     1   763     0  S       uwait   0xfffffe006ddd3700  syz-executor
 1112     1   766     0  S       uwait   0xfffffe0077eb8380  syz-executor
 1107     1   766     0  S       uwait   0xfffffe0077eb7c80  syz-executor
 1106     1   766     0  S       uwait   0xfffffe0077eb7b80  syz-executor
 1103     1   766     0  S       uwait   0xfffffe0058274180  syz-executor
 1098     1  1073     0  S       uwait   0xfffffe00584f9680  syz-executor
 1092     1   766     0  S       uwait   0xfffffe0058277a00  syz-executor
 1091     1   763     0  S       uwait   0xfffffe0058274f00  syz-executor
 1080     1   764     0  S       uwait   0xfffffe0077eba300  syz-executor
 1078     1   764     0  S       uwait   0xfffffe0058277d00  syz-executor
 1073   762  1073     0  R                                   syz-executor
 1067     1   763     0  S       uwait   0xfffffe0077eb8280  syz-executor
 1064     1   763     0  S       uwait   0xfffffe0058274480  syz-executor
 1061     1   766     0  S       uwait   0xfffffe0058277e00  syz-executor
 1059     1   764     0  S       uwait   0xfffffe0077eb9900  syz-executor
 1058     1   766     0  S       uwait   0xfffffe0058274280  syz-executor
 1044     1   764     0  S       uwait   0xfffffe0058274780  syz-executor
 1042     1  1042     0  S       uwait   0xfffffe0058274d00  syz-executor
 1007     1   766     0  S       uwait   0xfffffe0077eba000  syz-executor
 1006     1   766     0  S       uwait   0xfffffe0077eb9400  syz-executor
  992     1   765     0  S       uwait   0xfffffe00584fa100  syz-executor
  990     1   765     0  S       uwait   0xfffffe0058277600  syz-executor
  988     1   766     0  S       uwait   0xfffffe00584f9080  syz-executor
  987     1   763     0  S       uwait   0xfffffe00584fa000  syz-executor
  985     1   766     0  S       uwait   0xfffffe0077eba200  syz-executor
  984     1   763     0  S       uwait   0xfffffe0077eb9300  syz-executor
  976     1   766     0  S       uwait   0xfffffe0058277300  syz-executor
  972     1   765     0  S       uwait   0xfffffe00584fa680  syz-executor
  970     1   765     0  S       uwait   0xfffffe00584f9e80  syz-executor
  965     1   763     0  S       uwait   0xfffffe0058277700  syz-executor
  964     0     0     0  DL      (threaded)                  [so_splice]
100300                   D       -       0xfffffe0077eb9a80  [thr_0]
100301                   D       -       0xfffffe0077eb9ac0  [thr_1]
  963     1   766     0  SV      uwait   0xfffffe0077eba100  syz-executor
  961     0     0     0  DL      -       0xffffffff83cab400  [soaiod4]
  960     0     0     0  DL      -       0xffffffff83cab400  [soaiod3]
  959     0     0     0  DL      -       0xffffffff83cab400  [soaiod2]
  958     0     0     0  DL      -       0xffffffff83cab400  [soaiod1]
  954     0     0     0  DL      mdwait  0xfffffe006e498000  [md4]
  949     1   764     0  S       uwait   0xfffffe00584f9280  syz-executor
  946     1   763     0  SV      uwait   0xfffffe0058277400  syz-executor
  940     0     0     0  DL      mdwait  0xfffffe00083f7000  [md3]
  939     1   765     0  SV      uwait   0xfffffe0058274380  syz-executor
  936     0     0     0  DL      mdwait  0xfffffe0077846000  [md2]
  926     0     0     0  DL      mdwait  0xfffffe0077847000  [md1]
  919     1   766     0  S       uwait   0xfffffe0058274080  syz-executor
  905     1   766     0  S       uwait   0xfffffe0058275480  syz-executor
  902     0     0     0  DL      (threaded)                  [KTLS]
100149                   D       -       0xfffffe00596e3600  [thr_0]
100189                   D       -       0xfffffe00596e3680  [thr_1]
100190                   D       -       0xffffffff83cacc28  [reclaim_0]
  865     1   763     0  S       uwait   0xfffffe00584fa200  syz-executor
  858     0     0     0  DL      mdwait  0xfffffe006e497000  [md0]
  848     1   764     0  S       uwait   0xfffffe00584fa480  syz-executor
  827     1   815     0  S       uwait   0xfffffe0058275080  syz-executor
  826     1   766     0  S       uwait   0xfffffe00584f9580  syz-executor
  825     1   766     0  S       uwait   0xfffffe00584f9480  syz-executor
  823     1   765     0  S       uwait   0xfffffe0058275e00  syz-executor
  820     0     0     0  RL                                  [aiod4]
  819     0     0     0  DL      aiordy  0xfffffe00540f5560  [aiod3]
  818     0     0     0  DL      aiordy  0xfffffe005410e5a0  [aiod2]
  817     0     0     0  DL      aiordy  0xfffffe00540f8060  [aiod1]
  766   762   766     0  S       wait    0xfffffe00540f5ac0  syz-executor
  762   760   760     0  R                                   syz-executor
  760   758   760     0  Ss      sigsusp 0xfffffe00540a0670  csh
  758     1   758     0  Ss      select  0xfffffe000778ed40  sshd
   16     0     0     0  DL      syncer  0xffffffff83cb8e20  [syncer]
   15     0     0     0  DL      vlruwt  0xfffffe005409d560  [vnlru]
   14     0     0     0  RL      (threaded)                  [bufdaemon]
100079                   RunQ                                [bufdaemon]
100082                   RunQ                                [bufspacedaemon-0]
100094                   D       sdflush 0xfffffe00077628e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83d02280  [vmdaemon]
    8     0     0     0  RL      (threaded)                  [pagedaemon]
100077                   RunQ                                [dom0]
100080                   D       launds  0xffffffff83ce8354  [laundry: dom0]
100081                   D       umarecl 0xffffffff83ce7420  [uma]
    7     0     0     0  DL      -       0xffffffff83918cd0  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff8450c980  [pf purge]
    5     0     0     0  DL      waiting 0xffffffff847e2700  [sctp_iterator]
    4     0     0     0  DL      (threaded)                  [cam]
100045                   D       -       0xffffffff838e3340  [doneq0]
100046                   D       -       0xffffffff838e32c0  [async]
100075                   D       -       0xffffffff838e3140  [scanner]
    3     0     0     0  DL      (threaded)                  [crypto]
100042                   D       crypto_ 0xffffffff83ce3c00  [crypto]
100043                   D       crypto_ 0xfffffe0057d72030  [crypto returns 0]
100044                   D       crypto_ 0xfffffe0057d72080  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100037                   D       -       0xffffffff83b43d00  [g_event]
100038                   D       -       0xffffffff83b43d20  [g_up]
100039                   D       -       0xffffffff83b43d40  [g_down]
    2     0     0     0  LL      (threaded)                  [clock]
100031                   L      *tcp_hpt 0xfffffe0007804540  [clock (0)]
100032                   I                                   [clock (1)]
   12     0     0     0  RL      (threaded)                  [intr]
100013                   I                                   [swi6: task queue]
100014                   I                                   [swi6: Giant taskq]
100016                   I                                   [swi5: fast taskq]
100033                   I                                   [swi1: netisr 0]
100034                   I                                   [swi1: hpts]
100035                   RunQ                                [swi1: hpts]
100047                   I                                   [irq24: virtio_pci0]
100048                   I                                   [irq25: virtio_pci0]
100049                   I                                   [irq26: virtio_pci0]
100050                   I                                   [irq27: virtio_pci0]
100051                   I                                   [irq28: virtio_pci1]
100052                   I                                   [irq29: virtio_pci1]
100053                   RunQ                                [irq30: virtio_pci1]
100054                   I                                   [irq31: virtio_pci1]
100055                   I                                   [irq32: virtio_pci1]
100060                   I                                   [irq10: virtio_pci2]
100062                   I                                   [irq1: atkbd0]
100063                   I                                   [irq12: psm0]
100064                   I                                   [swi0: uart uart++]
100068                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0007809040  [init]
   10     0     0     0  DL      audit_w 0xffffffff83ce46a0  [audit]
    0     0     0     0  RLs     (threaded)                  [kernel]
100000                   D       parked  0xffffffff84c35ff0  [swapper]
100005                   D       -       0xfffffe0007a8de00  [softirq_0]
100006                   D       -       0xfffffe0007a8dd00  [softirq_1]
100007                   D       -       0xfffffe0007a8dc00  [if_io_tqg_0]
100008                   D       -       0xfffffe0007a8db00  [if_io_tqg_1]
100009                   D       -       0xfffffe0007a8da00  [if_config_tqg_0]
100010                   D       -       0xfffffe000776d700  [kqueue_ctx taskq]
100011                   D       -       0xfffffe000776d500  [jail_remove taskq]
100012                   D       -       0xfffffe000776d300  [bus taskq]
100015                   D       -       0xfffffe000776cc00  [thread taskq]
100017                   D       -       0xfffffe000776c800  [aiod_kick taskq]
100018                   D       -       0xfffffe000776c600  [deferred_unmount ta]
100019                   D       -       0xfffffe000776c400  [inm_free taskq]
100020                   D       -       0xfffffe000776c200  [in6m_free taskq]
100021                   D       -       0xfffffe000776c000  [linuxkpi_irq_wq]
100022                   D       -       0xfffffe000776bd00  [linuxkpi_short_wq_0]
100023                   D       -       0xfffffe000776bd00  [linuxkpi_short_wq_1]
100024                   D       -       0xfffffe000776bd00  [linuxkpi_short_wq_2]
100025                   D       -       0xfffffe000776bd00  [linuxkpi_short_wq_3]
100026                   D       -       0xfffffe000776b800  [linuxkpi_long_wq_0]
100027                   D       -       0xfffffe000776b800  [linuxkpi_long_wq_1]
100028                   D       -       0xfffffe000776b800  [linuxkpi_long_wq_2]
100029                   D       -       0xfffffe000776b800  [linuxkpi_long_wq_3]
100036                   D       -       0xfffffe0053eef100  [firmware taskq]
100040                   D       -       0xfffffe0053eeeb00  [crypto_0]
100041                   D       -       0xfffffe0053eeeb00  [crypto_1]
100056                   D       -       0xfffffe0053eede00  [vtnet0 rxq 0]
100057                   D       -       0xfffffe0053eedd00  [vtnet0 txq 0]
100058                   D       -       0xfffffe0053eedc00  [vtnet0 rxq 1]
100059                   D       -       0xfffffe0053eedb00  [vtnet0 txq 1]
100061                   D       vtbslp  0xfffffe0007790680  [virtio_balloon]
100065                   D       -       0xffffffff827c4060  [deadlkres]
100069                   D       -       0xfffffe0058500100  [acpi_task_0]
100070                   D       -       0xfffffe0058500100  [acpi_task_1]
100071                   D       -       0xfffffe0058500100  [acpi_task_2]
100073                   RunQ                                [mca taskq]
100074                   D       -       0xfffffe0053eee500  [CAM taskq]
100076                   D       -       0xfffffe00584ff700  [ipsec_offload]
100330                   D       -       0xfffffe00584feb00  [system_taskq_0]
100331                   D       -       0xfffffe00584feb00  [system_taskq_1]
100332                   D       -       0xfffffe00596e3100  [system_delay_taskq_]
100333                   D       -       0xfffffe00596e3100  [system_delay_taskq_]
100334                   D       -       0xfffffe0058500c00  [zvol_tq-0_0]
100335                   D       -       0xfffffe0058500c00  [zvol_tq-0_1]
100336                   D       -       0xfffffe0058500c00  [zvol_tq-0_2]
100337                   D       -       0xfffffe0058500c00  [zvol_tq-0_3]
100338                   D       -       0xfffffe0058500c00  [zvol_tq-0_4]
100339                   D       -       0xfffffe0058500c00  [zvol_tq-0_5]
100340                   D       -       0xfffffe0058500c00  [zvol_tq-0_6]
100341                   D       -       0xfffffe0058500c00  [zvol_tq-0_7]
100342                   D       -       0xfffffe0058500c00  [zvol_tq-0_8]
100343                   D       -       0xfffffe0058500c00  [zvol_tq-0_9]
100344                   D       -       0xfffffe0058500c00  [zvol_tq-0_10]
100345                   D       -       0xfffffe0058500c00  [zvol_tq-0_11]
100346                   D       -       0xfffffe0058500c00  [zvol_tq-0_12]
100347                   D       -       0xfffffe0058500c00  [zvol_tq-0_13]
100348                   D       -       0xfffffe0058500c00  [zvol_tq-0_14]
100349                   D       -       0xfffffe0058500c00  [zvol_tq-0_15]
100350                   D       -       0xfffffe0058500c00  [zvol_tq-0_16]
100351                   D       -       0xfffffe0058500c00  [zvol_tq-0_17]
100352                   D       -       0xfffffe0058500c00  [zvol_tq-0_18]
100354                   D       -       0xfffffe0058500c00  [zvol_tq-0_19]
100355                   D       -       0xfffffe0058500c00  [zvol_tq-0_20]
100356                   D       -       0xfffffe0058500c00  [zvol_tq-0_21]
100357                   D       -       0xfffffe0058500c00  [zvol_tq-0_22]
100358                   D       -       0xfffffe0058500c00  [zvol_tq-0_23]
100359                   D       -       0xfffffe0058500c00  [zvol_tq-0_24]
100360                   D       -       0xfffffe0058500c00  [zvol_tq-0_25]
100361                   D       -       0xfffffe0058500c00  [zvol_tq-0_26]
100362                   D       -       0xfffffe0058500c00  [zvol_tq-0_27]
100363                   D       -       0xfffffe0058500c00  [zvol_tq-0_28]
100364                   D       -       0xfffffe0058500c00  [zvol_tq-0_29]
100365                   D       -       0xfffffe0058500c00  [zvol_tq-0_30]
100366                   D       -       0xfffffe0058500c00  [zvol_tq-0_31]
100368                   D       -       0xfffffe00584fd100  [arc_prune]
100369                   D       -       0xfffffe0053eefe00  [arc_flush_0]
100370                   D       -       0xfffffe0053eefe00  [arc_flush_1]
100387                   D       -       0xfffffe0079307b00  [dbu_evict]
100406                   D       -       0xfffffe00584fec00  [z_vdev_file_0]
100407                   D       -       0xfffffe00584fec00  [z_vdev_file_1]
100408                   D       -       0xfffffe00584fec00  [z_vdev_file_2]
100409                   D       -       0xfffffe00584fec00  [z_vdev_file_3]
100410                   D       -       0xfffffe00584fec00  [z_vdev_file_4]
100411                   D       -       0xfffffe00584fec00  [z_vdev_file_5]
100412                   D       -       0xfffffe00584fec00  [z_vdev_file_6]
100413                   D       -       0xfffffe00584fec00  [z_vdev_file_7]
100414                   D       -       0xfffffe00584fec00  [z_vdev_file_8]
100415                   D       -       0xfffffe00584fec00  [z_vdev_file_9]
100416                   D       -       0xfffffe00584fec00  [z_vdev_file_10]
100417                   D       -       0xfffffe00584fec00  [z_vdev_file_11]
100418                   D       -       0xfffffe00584fec00  [z_vdev_file_12]
100419                   D       -       0xfffffe00584fec00  [z_vdev_file_13]
100420                   D       -       0xfffffe00584fec00  [z_vdev_file_14]
100421                   D       -       0xfffffe00584fec00  [z_vdev_file_15]
100447                   D       -       0xfffffe000776fc00  [zfsvfs]
100488                   D       -       0xfffffe0079423900  [netlink_socket (PID]
db> show all locks
Process 1256 (ifconfig) thread 0xfffffe005419f000 (100794)
exclusive rw nd6 DAD queue (nd6 DAD queue) r = 0 (0xfffffe0007a5a178) locked @ /syzkaller/managers/main/kernel/sys/netinet6/nd6_nbr.c:1304
Process 1251 (syz-executor) thread 0xfffffe005416d780 (100813)
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0007db9518) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_bio.c:4022
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe006e491908) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_syscalls.c:1760
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0059ce7228) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_lookup.c:1169
Process 1230 (syz-executor) thread 0xfffffe00541a2000 (100762)
exclusive sleep mutex pmap (pmap) r = 0 (0xffffffff8380b5e0) locked @ /syzkaller/managers/main/kernel/sys/amd64/amd64/pmap.c:6402
Process 762 (syz-executor) thread 0xfffffe00540cf000 (100109)
shared sx vm map (user) (vm map (user)) r = 0 (0xfffffe0007810e10) locked @ /syzkaller/managers/main/kernel/sys/vm/vm_map.c:4998
Process 8 (pagedaemon) thread 0xfffffe0054007780 (100077)
shared rw UMA lock (UMA lock) r = 0 (0xffffffff830033c0) locked @ /syzkaller/managers/main/kernel/sys/vm/uma_core.c:3106
shared sx umareclaim (umareclaim) r = 0 (0xffffffff83ce7420) locked @ /syzkaller/managers/main/kernel/sys/vm/uma_core.c:5330
Process 2 (clock) thread 0xfffffe000781a780 (100031)
exclusive rw tcpinp (tcpinp) r = 0 (0xfffffe0077ca8aa0) locked @ /syzkaller/managers/main/kernel/sys/kern/kern_rwlock.c:190
Process 12 (intr) thread 0xfffffe0007820780 (100035)
exclusive sleep mutex tcp_hpts_lck (hpts) r = 0 (0xfffffe00083df900) locked @ /syzkaller/managers/main/kernel/sys/netinet/tcp_hpts.c:1403
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            6        12804K            6
            linker          425        12680K          659
          tcp_hpts            7         4801K            7
         sctp_stro            6         4613K           17
            devbuf         4187         4323K         4219
           solaris         2246         3597K         4472
         sysctloid        44910         2638K        45022
             vtbuf           24         1968K           46
              kobj          330         1320K          548
            newblk           45         1035K         3465
          filedesc          129         1030K          710
          vfscache            3         1025K            3
               pcb           56          694K          524
          inodedep           28          523K          728
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
           subproc          246          499K         1397
              intr            4          472K            4
              vmem            5          274K            9
         vnet_data            2          224K            2
          acpitask            1          224K            1
            KTRACE          102          201K        12613
            acpica         1674          184K        54426
           tidhash            3          141K            3
           pagedep           21          133K          360
        tfo_ccache            1          128K            1
          IP reass            1          128K            1
            DEVFS1          115          115K          136
               sem            4          106K            4
        gtaskqueue           18           98K           18
               bus          997           82K         5063
          mtx_pool            3           74K            3
              umtx          564           71K          564
           kdtrace          364           70K         2083
          syncache            1           68K            1
     NFSD srvcache            3           68K            3
        md_sectors           17           68K           17
            module          526           66K          528
           netlink           79           66K          154
       ddb_capture            1           64K            1
               shm            8           46K           31
            DEVFS3          134           34K          145
          routetbl          229           33K          578
         hostcache            1           32K            1
               msg            4           30K            4
            kbdmux            6           28K            6
           md_disk           23           25K           23
               LRO           24           25K           24
            ifaddr           84           22K           86
              temp           41           21K         2325
        DEVFS_RULE           56           20K           56
        kstat_data           19           19K           19
           lltable           61           19K           81
       ether_multi          213           17K          243
         ufs_mount            4           17K            5
              GEOM          103           17K          667
              proc            3           17K            3
             ifnet            9           17K           10
           filemon            2           16K           47
               tty           16           16K           16
           ithread           90           15K           90
              cred           33           15K          369
            bus-sc           34           15K         1647
      eventhandler          166           14K          166
           devstat            6           13K            6
         in6_multi           85           12K           87
              kenv           95           12K           95
            CC Mem           85           11K          275
         taskqueue           96           11K          129
         CAM queue            5           11K         1528
            kqueue          119           10K         1837
              rman           82           10K          437
             shmfd            4           10K           14
               rpc            8            9K            8
         sctp_atcl           23            9K          201
         bmsafemap            3            9K          602
              UART           12            9K           12
              ksem            1            8K            4
       pfs_vncache            1            8K            1
     audit_evclass          240            8K          303
            plimit           20            8K          786
           pwddesc          112            7K         1376
               UMA          340            7K          340
            sglist            6            7K            6
           CAM DEV            3            6K          510
         pfs_nodes           22            6K           22
          pf_ifnet           13            5K           26
            DEVFSP           71            5K          164
      tcp_fsb_rack            2            5K           10
                vt           11            5K           11
           memdesc            1            4K            1
               MCA           32            4K           32
       ufs_dirhash            8            4K           27
            crypto           11            4K           46
             evdev            4            4K            4
       inpcbpolicy          114            4K          676
           acpisem           28            4K           28
          kcovinfo           54            4K           54
             mount           23            4K         1105
               osd          147            3K          355
             lockf           26            3K          136
          terminal           11            3K           11
         proc-args          116            3K         2419
         sctp_timw           10            3K           10
         sctp_stri            5            3K           14
            ip6ndp           16            3K           18
           acpidev           20            3K           20
             hhook            8            3K           10
          in_multi            9            3K           13
               tun            6            3K            6
             clone            9            3K            9
           uidinfo            3            3K           20
          sctp_ifa           17            3K           18
           pf_rule            1            2K            1
        local_apic            1            2K            1
           io_apic            1            2K            1
         ipsec-saq            2            2K            2
         sctp_atky           33            2K          229
            Unitno           28            2K           81
         newdirblk           13            2K          311
            ip6opt           11            2K           74
           session           13            2K           67
           CAM XPT           22            2K          543
          indirdep            6            2K          839
               BPF            9            2K           29
       vnodemarker            3            2K           53
         toponodes            6            2K            6
       ipsecpolicy            2            2K            2
            dirrem            5            2K          541
             nhops            6            2K            9
               msi            9            2K            9
          sctp_ifn            8            1K           18
           softdep            1            1K            1
            diradd            8            1K          561
            sahead            1            1K            1
          secasvar            1            1K            1
               mld            8            1K            8
              igmp            8            1K            8
      NFSD session            1            1K            1
             mkdir            7            1K          622
        CAM periph            4            1K          271
         cryptodev           12            1K          180
             ipsec            3            1K            3
              pfil            6            1K            6
            isadev            6            1K            6
          pci_link           10            1K           10
 encap_export_host           12            1K           12
              cdev            2            1K            2
       lkpikmalloc            8            1K            9
             selfd            7            1K       297141
      counter_rate           14            1K           14
    chacha20random            1            1K            1
           VN POLL            3            1K            8
            biobuf            1            1K            1
filedesc_to_leader            6            1K           17
         sctp_athm           23            1K          207
              ktls            3            1K           36
      tcp_pcm_rack            1            1K            5
          freefile            2            1K          427
            vnodes            1            1K            5
               iov            4            1K        16198
            select            2            1K           63
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           11
           CAM SIM            2            1K            2
            prison            8            1K            8
            feeder            7            1K            7
             taskq            2            1K            2
          sctp_map           12            1K           34
           tcpfunc            3            1K            3
        ip_msource            3            1K           13
        loginclass            3            1K            5
         sctp_aadr            2            1K            5
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
          ktls_ocf            1            1K            5
       ip6_msource            2            1K           14
       in6_mfilter            2            1K           25
               aio            4            1K            4
          pmchooks            1            1K            1
          CAM path            4            1K         1034
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
        in_mfilter            2            1K           23
          sctp_vrf            1            1K            1
      ip6_moptions            2            1K           13
       ip_moptions            1            1K           12
              vnet            1            1K            1
               pmc            1            1K            1
             sigio            1            1K            3
           entropy            2            1K           35
          acpiintr            1            1K            1
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           fadvise            1            1K            9
           Per-cpu            1            1K            1
          freework            1            1K          978
            soname            1            1K         3784
          p1003.1b            1            1K            1
         sfs_nodes            0            0K            0
        zones_data            0            0K            0
        ext2_mount            0            0K            0
         ext2_node            0            0K            0
      ext2_extents            0            0K            0
       tcp_do_rack            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K           23
         sctp_iter            0            0K           15
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            1
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           14
          pf_table            0            0K            0
           pf_altq            0            0K            0
           pf_osfp            0            0K            0
     pf_krule_item            0            0K            0
           pf_temp            0            0K            0
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
            mqdata            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
               ixl            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
       fpukern_ctx            0            0K            0
          xen_intr            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
      NMI handlers            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
      amdiommu_dom            0            0K            0
      amdiommu_ctx            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
        aesni_data            0            0K            3
            xenbus            0            0K            0
     vm_fictitious            0            0K            0
           UMAHash            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K          593
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           26
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freeblks            0            0K          380
          freefrag            0            0K          130
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
       simple_attr            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
           lkpimhi            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
             frag6            0            0K           39
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K           23
            statfs            0            0K          203
     namei_tracker            0            0K            1
           inotify            0            0K            2
       export_host            0            0K            0
        cl_savebuf            0            0K           62
               lio            0            0K            7
               acl            0            0K            0
          mbuf_tag            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
           timerfd            0            0K            0
          procdesc            0            0K           10
          ioctlops            0            0K          223
           eventfd            0            0K            2
           Witness            0            0K            0
             stack            0            0K            0
              sbuf            0            0K          474
          firmware            0            0K            0
        compressor            0            0K            0
              SWAP            0            0K            0
         sysctltmp            0            0K          690
            sysctl            0            0K            3
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K          138
              rctl            0            0K            0
             cache            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
          filecaps            0            0K           84
               pwd            0            0K            0
       tty console            0            0K            0
         boottrace            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K          108
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
     tmpfs extattr            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
            DEVFS4            0            0K            2
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
               xnb            0            0K            0
          xen_acpi            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
            vtfont            0            0K            0
            pvscsi            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
            twsbuf            0            0K            0
       tcp_log_dev            0            0K            6
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
       SIIS driver            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
         mpi3mrbuf            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
          ix_sriov            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
         ciss_data            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
       AHCI driver            0            0K            0
               agp            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
         acpicmbat            0            0K            0
        aacraidcam            0            0K            0
       aacraid_buf            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K          357
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
         scsi_pass            0            0K            0
           scsi_da            0            0K           70
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
           nvme_da            0            0K            0
           CAM CCB            0            0K          523
     CAM ccb queue            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8342       3       17103       0      29   34181120        0
        RADIX NODE    152   25895     178     1517108       0      62    3963096        0
    ertt_txseginfo     40   96369     389      108095       0     254    3870320        0
              mbuf    256    9582     414      139483       0     254    2558976        0
        malloc-128    128   14693     125       15738       0     126    1896704        0
          BUF TRIE    152     312   11492        3154       0      62    1794208        0
        malloc-384    384    4118      12        4118       0      30    1585920        0
       malloc-4096   4096     333       0        1032       0       2    1363968        0
       UMA Slabs 0    112    9622    1532       11368       0     126    1249248        0
         vmem btag     56   17993    3502       21421       0     254    1203720        0
      malloc-16384  16384      66       1         542       0       1    1097728        0
         FFS inode   1168     601      22        1038       0       8     727664        0
      malloc-65536  65536      11       0          24       0       1     720896        0
        malloc-256    256    2451      69        4055       0      62     645120        0
      mbuf_cluster   2048     279       3         764       0      34     577536        0
              pbuf   2624       0     210           0       0       2     551040        0
            THREAD   1860     251      31         820       0       8     524520        0
       malloc-4096   4096     126       1        1282       0       2     520192        0
          lkpicurr    168       2    3094           2       0      62     520128        0
         VM OBJECT    248    1551     145       18914       0      62     420608        0
      malloc-65536  65536       6       0           6       0       1     393216        0
         malloc-64     64    5199      93        6725       0     254     338688        0
             VNODE    440     642      78        1083       0      30     316800        0
         malloc-16     16   18478     522       20423       0     254     304000        0
           tcp_log    416     415     314       38015       0      53     303264        0
         MAP ENTRY     96    2530     200       62980       0     126     262080        0
         malloc-32     32    7402     284        8245       0     254     245952        0
         UMA Zones    768     312       2         312       0      16     241152        0
     FPU_save_area    832     253      35        1125       0      16     239616        0
       malloc-2048   2048     106       2         345       0       8     221184        0
              PROC   1376     111      21        1261       0       8     181632        0
            socket   1024     132      44        2164       0      41     180224        0
            lkpimm     56       1    3095           1       0     254     173376        0
        malloc-128    128    1211      91       29315       0     126     166656        0
       FFS2 dinode    256     601      44        1038       0      62     165120        0
         tcp_inpcb   1304      85      38         275       0       8     160392        0
            DEVCTL   1024      47     101         172       0       0     151552        0
       S VFS Cache    104    1022     382        1599       0     126     146016        0
       malloc-4096   4096      35       0         267       0       2     143360        0
         filedesc0   1072     112      14        1376       0       8     135072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
      malloc-32768  32768       4       0           4       0       1     131072        0
        256 Bucket   2048      23      39        2376       0       8     126976        0
       malloc-1024   1024     120       0         141       0      16     122880        0
          UMA Kegs    384     298       5         298       0      30     116352        0
        malloc-128    128     786     113        1843       0     126     115072        0
        malloc-256    256     107     328        4069       0      62     111360        0
         malloc-64     64     219    1356      298593       0     254     100800        0
        malloc-128    128     610     134        1299       0     126      95232        0
       malloc-8192   8192      11       0          14       0       1      90112        0
       mbuf_packet    256      25     254         639       0     254      71424        0
        malloc-384    384     164      16         181       0      30      69120        0
               md1      8    8192       8        8192       0     254      65600        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-32768  32768       2       0          16       0       1      65536        0
        128 Bucket   1024       9      54         352       0      16      64512        0
        malloc-256    256      71     169        2658       0      62      61440        0
         64 Bucket    512      20     100       45628       0      30      61440        0
         32 Bucket    256      14     226      300232       0      62      61440        0
         malloc-64     64     521     424       20197       0     254      60480        0
        malloc-384    384      99      51         852       0      30      57600        0
       malloc-2048   2048      20       8        1102       0       8      57344        0
           VMSPACE    584      80      18        1222       0      16      57232        0
        malloc-256    256     210       0         260       0      62      53760        0
       malloc-4096   4096      13       0          55       0       2      53248        0
             Files     80     335     315       10579       0     126      52000        0
        malloc-128    128     302     101         345       0     126      51584        0
      malloc-16384  16384       3       0           4       0       1      49152        0
      malloc-16384  16384       3       0           3       0       1      49152        0
       malloc-2048   2048      16       8         244       0       8      49152        0
            pcpu-8      8    5456     688        5755       0     254      49152        0
        malloc-256    256      66     114        1525       0      62      46080        0
         TURNSTILE    136     283      53         283       0      62      45696        0
       malloc-4096   4096      11       0         108       0       2      45056        0
             ripcb    376      13      97         131       0      30      41360        0
       malloc-8192   8192       5       0         160       0       1      40960        0
       malloc-2048   2048      16       4          62       0       8      40960        0
        sctp_raddr    736      13      42          25       0      41      40480        0
              pipe    736      25      30         450       0      16      40480        0
           sctp_ep   1152      12      23         175       0      34      40320        0
         malloc-64     64     464     166         694       0     254      40320        0
        sctp_chunk    152     208      52         214       0      81      39520        0
        malloc-384    384      74      26        1183       0      30      38400        0
           pcpu-64     64     506      70         506       0     254      36864        0
         udp_inpcb    408      14      76         254       0      30      36720        0
         malloc-64     64     146     421        2088       0     254      36288        0
        malloc-256    256     134       1         697       0      62      34560        0
        malloc-256    256      55      80         650       0      62      34560        0
         sctp_asoc   2256       6       9          17       0      30      33840        0
      malloc-32768  32768       0       1         686       0       1      32768        0
       malloc-8192   8192       4       0           4       0       1      32768        0
       malloc-8192   8192       4       0           5       0       1      32768        0
       malloc-1024   1024      29       3          41       0      16      32768        0
         malloc-64     64     256     248         374       0     254      32256        0
            ttyinq    160     135      65        1110       0      62      32000        0
       tcp_bbr_map    128      97     151         616       0     126      31744        0
        malloc-128    128     134     114         727       0     126      31744        0
              PGRP    120      20     244          79       0     126      31680        0
        SLEEPQUEUE     88     283      69         283       0     126      30976        0
           ttyoutq    256      72      48         592       0      62      30720        0
        malloc-256    256      79      41         169       0      62      30720        0
       malloc-1024   1024      26       2        2183       0      16      28672        0
         malloc-32     32     336     546         434       0     254      28224        0
             NAMEI   1024       1      23       17229       0      16      24576        0
       malloc-8192   8192       3       0          50       0       1      24576        0
       malloc-1024   1024      18       6        1720       0      16      24576        0
               PWD     40      60     546         353       0     254      24240        0
         16 Bucket    144       6     162       34063       0      62      24192        0
        malloc-384    384      31      29         214       0      30      23040        0
            da_ccb    544       0      42        4873       0      16      22848        0
       Mountpoints   2816       2       6          12       0       4      22528        0
       malloc-2048   2048       8       2           8       0       8      20480        0
          ksiginfo    112     110      70         378       0     126      20160        0
         malloc-64     64      29     286         432       0     254      20160        0
          8 Bucket     80       9     241     1044929       0     126      20000        0
        malloc-128    128      46     109         526       0     126      19840        0
     routing nhops    256      35      40          44       0      62      19200        0
             unpcb    320       6      54        1261       0      53      19200        0
      malloc-16384  16384       1       0           1       0       1      16384        0
      malloc-16384  16384       1       0           2       0       1      16384        0
       malloc-8192   8192       2       0           5       0       1      16384        0
       malloc-2048   2048       5       3          86       0       8      16384        0
        malloc-512    512      13      19         179       0      30      16384        0
       tcp_bbr_pcb    896       2      16           9       0      16      16128        0
           rtentry    168      38      58          44       0      62      16128        0
        sctp_laddr     48      11     325          35       0     254      16128        0
          rl_entry     48      12     324          20       0     254      16128        0
         malloc-64     64      61     191         290       0     254      16128        0
         malloc-32     32      59     445         216       0     254      16128        0
         malloc-32     32     185     319         618       0     254      16128        0
         malloc-32     32     174     330         249       0     254      16128        0
         malloc-32     32      50     454        7626       0     254      16128        0
         malloc-32     32      38     466        1915       0     254      16128        0
         malloc-16     16     346     654         513       0     254      16000        0
        malloc-128    128      28      96         132       0     126      15872        0
       L VFS Cache    320       0      48          25       0      30      15360        0
     udplite_inpcb    408       2      34          16       0      30      14688        0
       malloc-2048   2048       3       3          40       0       8      12288        0
       malloc-1024   1024       9       3         233       0      16      12288        0
        malloc-512    512      10      14          26       0      30      12288        0
        malloc-512    512       9      15          16       0      30      12288        0
              ertt     72      85      83         275       0     126      12096        0
          2 Bucket     32       5     373         670       0     254      12096        0
         malloc-16     16     163     587         883       0     254      12000        0
         malloc-16     16      28     722          35       0     254      12000        0
         malloc-16     16     183     567        2909       0     254      12000        0
         malloc-16     16      19     731       29557       0     254      12000        0
         malloc-16     16      75     675         472       0     254      12000        0
              vmem   1856       2       3           2       0       8       9280        0
       malloc-4096   4096       0       2         106       0       2       8192        0
       malloc-1024   1024       8       0           9       0      16       8192        0
        malloc-512    512       2      14          13       0      30       8192        0
        malloc-512    512       6      10          29       0      30       8192        0
           pcpu-16     16       8     504           8       0     254       8192        0
         hostcache     64       3     123           3       0     254       8064        0
         malloc-32     32      37     215         723       0     254       8064        0
         malloc-16     16      10     490         217       0     254       8000        0
   tcp_log_id_node    120       0      66           9       0     126       7920        0
              kenv    258      17      13        1077       0      30       7740        0
      tcp_rack_pcb   1088       1       6           5       0       8       7616        0
            mqnode    416       5      13           5       0      30       7488        0
            cpuset    200       9      23          38       0      62       6400        0
 epoch_record pcpu    256       4      12           4       0      62       4096        0
       malloc-4096   4096       1       0           6       0       2       4096        0
       malloc-4096   4096       1       0          22       0       2       4096        0
       malloc-2048   2048       2       0           2       0       8       4096        0
       malloc-1024   1024       1       3         421       0      16       4096        0
        malloc-512    512       2       6         244       0      30       4096        0
           pf tags    104       1      38           1       0     126       4056        0
arc_buf_hdr_t_full    368       4       7           4       0      30       4048        0
 tcp_log_id_bucket    176       0      23           5       0      62       4048        0
   sctp_asconf_ack     48       1      83           3       0     254       4032        0
      vtnet_tx_hdr     24       1     166        3655       0     254       4008        0
             KNOTE    160       1      24         296       0      62       4000        0
            mqueue    248       2      14           2       0      62       3968        0
               AIO    208       1      18          68       0      62       3952        0
            itimer    352       1      10         234       0      30       3872        0
       UMA Slabs 1    176      11      11          11       0      62       3872        0
      ktls_session    256       1      14           7       0      62       3840        0
        malloc-384    384       1       9          44       0      30       3840        0
        KMAP ENTRY     96      18      21          36       0       0       3744        0
           SMR CPU     32       8      87           8       0     254       3040        0
tfo_ccache_entries     80       1      31           1       0     126       2560        0
        SMR SHARED     24       8      87           8       0     254       2280        0
   zfs_znode_cache    464       0       0           0       0      30          0        0
zap_attr_long_cache   1056       0       0           0       0       8          0        0
     zap_name_long   1096       0       0           0       0       8          0        0
    zap_attr_cache    288       0       0           0       0      30          0        0
          zap_name    328       0       0           0       0      30          0        0
       sio_cache_2    168       0       0           0       0      62          0        0
       sio_cache_1    152       0       0           0       0      62          0        0
       sio_cache_0    136       0       0           0       0      62          0        0
     zil_zcw_cache     80       0       0           0       0     126          0        0
     zil_lwb_cache    376       0       0           0       0      30          0        0
dbuf_dirty_record_t    368       0       0           0       0      30          0        0
    dmu_buf_impl_t    376       0       0           0       0      30          0        0
         arc_buf_t     32       0       0           0       0     254          0        0
arc_buf_hdr_t_l2only     96       0       0           0       0     126          0        0
           dnode_t    968       0       0           0       0      16          0        0
          sa_cache    272       0       0           0       0      30          0        0
         abd_chunk   4096       0       0           0       0       2          0        0
zio_buf_comb_16777216 16777216       0       0           0       0       1          0        0
zio_buf_comb_12582912 12582912       0       0           0       0       1          0        0
zio_buf_comb_8388608 8388608       0       0           0       0       1          0        0
zio_buf_comb_6291456 6291456       0       0           0       0       1          0        0
zio_buf_comb_4194304 4194304       0       0           0       0       1          0        0
zio_buf_comb_3145728 3145728       0       0           0       0       1          0        0
zio_buf_comb_2097152 2097152       0       0           0       0       1          0        0
zio_buf_comb_1572864 1572864       0       0           0       0       1          0        0
zio_buf_comb_1048576 1048576       0       0          15       0       1          0        0
zio_buf_comb_786432 786432       0       0           0       0       1          0        0
zio_buf_comb_524288 524288       0       0           0       0       1          0        0
zio_buf_comb_393216 393216       0       0           0       0       1          0        0
zio_buf_comb_262144 262144       0       0           0       0       1          0        0
zio_buf_comb_196608 196608       0       0           0       0       1          0        0
zio_buf_comb_131072 131072       0       0           1       0       1          0        0
zio_buf_comb_98304  98304       0       0           0       0       1          0        0
zio_buf_comb_65536  65536       0       0           0       0       1          0        0
zio_buf_comb_49152  49152       0       0           0       0       1          0        0
zio_buf_comb_32768  32768       0       0           0       0       1          0        0
zio_buf_comb_24576  24576       0       0           0       0       1          0        0
zio_data_buf_16384  16384       0       0           9       0       1          0        0
     zio_buf_16384  16384       0       0          10       0       1          0        0
zio_data_buf_12288  12288       0       0           0       0       1          0        0
     zio_buf_12288  12288       0       0           0       0       1          0        0
 zio_data_buf_8192   8192       0       0           0       0       1          0        0
      zio_buf_8192   8192       0       0           0       0       1          0        0
 zio_data_buf_6144   6144       0       0           0       0       1          0        0
      zio_buf_6144   6144       0       0           0       0       1          0        0
 zio_data_buf_4096   4096       0       0           0       0       2          0        0
      zio_buf_4096   4096       0       0           0       0       2          0        0
 zio_data_buf_3072   3072       0       0           0       0       4          0        0
      zio_buf_3072   3072       0       0           0       0       4          0        0
 zio_data_buf_2048   2048       0       0           0       0       8          0        0
      zio_buf_2048   2048       0       0           0       0       8          0        0
 zio_data_buf_1536   1536       0       0           0       0       8          0        0
      zio_buf_1536   1536       0       0           0       0       8          0        0
 zio_data_buf_1024   1024       0       0           0       0      16          0        0
      zio_buf_1024   1024       0       0           0       0      16          0        0
  zio_data_buf_512    512       0       0           0       0      30          0        0
       zio_buf_512    512       0       0           0       0      30          0        0
    zio_link_cache     48       0       0           0       0     254          0        0
         zio_cache   1216       0       0           0       0       8          0        0
ddt_log_entry_trad_cache    328       0       0           0       0      30          0        0
ddt_log_entry_flat_cache    144       0       0           0       0      62          0        0
ddt_entry_trad_cache    368       0       0           0       0      30          0        0
ddt_entry_flat_cache    184       0       0           0       0      62          0        0
         ddt_cache  29240       0       0           0       0       1          0        0
   brt_entry_cache    168       0       0           0       0      62          0        0
metaslab_alloc_trace_cache     72       0       0           0       0     126          0        0
zfs_btree_leaf_cache   4096       0       0           0       0       2          0        0
   reference_cache     48       0       0           0       0     254          0        0
        taskq_zone    160       0       0           0       0      62          0        0
            splice    184       0       0           2       0      62          0        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    152       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
  pf fragment node     72       0       0           0       0     126          0        0
          pf frags    232       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      81          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf UDP mappings    104       0       0           0       0     126          0        0
   pf source nodes    152       0       0           0       0      81          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    384       0       0           0       0      53          0        0
          pf mtags    184       0       0           0       0      62          0        0
      tcp_rack_map    128       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
          syncache    168       0       0           4       0      81          0        0
               ipq     56       0       0           1       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
sctp_stream_msg_out    112       0       0           4       0     137          0        0
        sctp_readq    152       0       0           0       0      81          0        0
          tcpreass     48       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            AIOLIO    272       0       0           7       0      30          0        0
             AIOCB    552       0       0         143       0      16          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
           DIRHASH   1024       0       0          36       0      16          0        0
        TMPFS node    240       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           8       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    152       0       0           0       0      62          0        0
             g_bio    408       0       0       18784       0      30          0        0
   IOMMU_MAP_ENTRY    112       0       0           0       0     126          0        0
            skbuff   1808       0       0           0       0       8          0        0
    mbuf_jumbo_16k  16384       0       0           0       0      28          0        0
     mbuf_jumbo_9k   9216       0       0           0       0      28          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0          14       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            clpbuf   2624       0       0         207       0       4          0        0
            vnpbuf   2624       0       0           0       0       4          0        0
           nfspbuf   2624       0       0           0       0       4          0        0
            swwbuf   2624       0       0           0       0       2          0        0
            swrbuf   2624       0       0           0       0       4          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    152       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0         110       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0          71       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0         122       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0          32       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           1       0       1          0        0
       malloc-8192   8192       0       0          23       0       1          0        0
       malloc-1024   1024       0       0           3       0      16          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-512    512       0       0          19       0      30          0        0
        malloc-384    384       0       0          16       0      30          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg     88       0       0           0       0     126          0        0
          4 Bucket     48       0       0          21       0     254          0        0
          UMA Hash    256       0       0           0       0      62          0        0
db> show all tcpcbs/bl
tcpcb at 0xfffffe0077ca8a80
  t_segq first: 0   t_segqlen: 0   t_dupacks: 0
  t_callout: 0xfffffe0077ca8c08   t_timers: 0xfffffe0077ca8c48
  t_state: 2 (TCPS_SYN_SENT)
  t_flags: 0x10000aa0 (TF_REQ_SCALE, TF_REQ_TSTMP, TF_SACK_PERMIT, TF_NEEDFIN, TF_LRD)
  t_flags2: 0x9020a (TF2_PLPMTU_PMTUD, TF2_LOG_AUTO, TF2_HPTS_CPU_SET, TF2_SUPPORTS_MBUFQ, TF2_CANNOT_DO_ECN)
  snd_una: 0x85282b98   snd_max: 0x85282b99   snd_nxt: 0x85282b99
  snd_up: 0x85282b98   snd_wl1: 0x00000000   snd_wl2: 0x00000000
  iss: 0x85282b98   irs: 0x00000000   rcv_nxt: 0x00000000
  rcv_adv: 0x0000ffff   rcv_wnd: 0   rcv_up: 0x00000000
  snd_wnd: 0   snd_cwnd: 1
  snd_ssthresh: 1073725440   snd_recover: 0x85282b99
  t_rcvtime: 2147430271   t_startime: 0
  t_rttime: 0   t_rtsq: 0x00000000
  t_rxtcur: 400   t_maxseg: 1220   t_srtt: 0
  t_rttvar: 400   t_rxtshift: 10   t_rttmin: 3
  t_rttupdated: 0   max_sndwnd: 0   t_softerror: 0
  t_oobflags: 0x0 ()   t_iobc: 0x00
  snd_scale: 0   rcv_scale: 0   request_r_scale: 6
  ts_recent: 0   ts_recent_age: 0
  ts_offset: 3483144382   last_ack_sent: 0x00000000   snd_cwnd_prev: 0
  snd_ssthresh_prev: 0   snd_recover_prev: 0x00000000   t_badrxtwin: 0
  snd_numholes: 0  snd_holes first: 0
  snd_fack: 0x00000000   rcv_numsacks: 0
  t_rttlow: 0   rfbuf_ts: 67283   rfbuf_cnt: 0
  t_fb.tfb_tcp_block_name: bbr
  t_cc.name: cubic
  _t_logstate: 1 (TCP_LOG_STATE_TAIL)
  t_lognum: 6   t_loglimit: 5000   t_logsn: 63
     0.000
    +0.000
    +0.000 > S 2234002328:2234002328(0) win 65535 <mss 1400,nop,wscale 6,sackOK,TS val 3483255719 ecr 0>
    +0.000
    +0.000
    +32.161 Processing Keepalive timer

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/07/07 14:12 freebsd-src e05999a8c5a9 4f67c4ae console log report ci-freebsd-main panic: witness_warn
* Struck through repros no longer work on HEAD.