syzbot


panic: Non-zero write count

Status: fixed on 2021/05/23 21:08
Reported-by: syzbot+6120ebbb354cd52e5107@syzkaller.appspotmail.com
Fix commit: 6f6cd1e8e8aa ktrace: Remove vrele() at the end of ktr_writerequest()
First crash: 1068d, last: 1067d

Sample crash report:
panic: Non-zero write count
cpuid = 1
time = 2000000019
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe0088780760
vpanic() at vpanic+0x1c7/frame 0xfffffe00887807c0
panic() at panic+0x43/frame 0xfffffe0088780820
freevnode() at freevnode+0x3e9/frame 0xfffffe0088780850
vput_final() at vput_final+0xef/frame 0xfffffe00887808c0
ktr_writerequest() at ktr_writerequest+0x735/frame 0xfffffe00887809b0
ktr_submitrequest() at ktr_submitrequest+0x5e/frame 0xfffffe00887809e0
dofilewrite() at dofilewrite+0x1d6/frame 0xfffffe0088780a30
sys_write() at sys_write+0x10c/frame 0xfffffe0088780ab0
amd64_syscall() at amd64_syscall+0x247/frame 0xfffffe0088780bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0088780bf0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x4ae72b, rsp = 0xc00066d500, rbp = 0xc00066d550 ---
KDB: enter: panic
[ thread pid 780 tid 100133 ]
Stopped at      kdb_enter+0x67: movq    $0,0x163736e(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx                       0x80
rdx         0xffffffff819c36cb
rbx                          0
rsp         0xfffffe0088780740
rbp         0xfffffe0088780760
rsi                        0x1
rdi                          0
r8                           0
r9          0x8080808080808080
r10         0xfffffe0088780630
r11              0x1ff77fff59c
r12         0xffffffff82267b90  ddb_dbbe
r13                          0
r14         0xffffffff81a74c1a
r15         0xffffffff81a74c1a
rip         0xffffffff81131a97  kdb_enter+0x67
rflags                    0x86
kdb_enter+0x67: movq    $0,0x163736e(%rip)
db> show proc
Process 780 (syz-fuzzer) at 0xfffff8003b305530:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 778 at 0xfffff8003b350000
 ABI: FreeBSD ELF64
 flag: 0x10004080  flag2: 0
 arguments: ./syz-fuzzer -executor=./syz-executor -name=vm-3 -arch=amd64 -manager=10.128.0.248:54351 -sandbox=none -procs=4 -cover=true -debug=false -test=false -optional=slowdown=1
 reaper: 0xfffff80004bc7530 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffffe005593d9f0
   (map 0xfffffe005593d9f0)
   (map.pmap 0xfffffe005593dab0)
   (pmap 0xfffffe005593db10)
 threads: 8
100105                   S       uwait   0xfffff80015d6df00  syz-fuzzer
100122                   S       uwait   0xfffff80015d4db00  syz-fuzzer
100123                   S       uwait   0xfffff80015d50300  syz-fuzzer
100124                   S       uwait   0xfffff80015d88d00  syz-fuzzer
100125                   S       uwait   0xfffff80015d7dd00  syz-fuzzer
100126                   S       uwait   0xfffff80015d4bf00  syz-fuzzer
100127                   S       kqread  0xfffff80015d49300  syz-fuzzer
100133                   Run     CPU 1                       syz-fuzzer
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
13992     1   791     0  S       uwait   0xfffff8004751ff00  syz-executor.0
13985     1   791     0  S       uwait   0xfffff80015d49100  syz-executor.0
13974     1   804     0  S       uwait   0xfffff80015d88900  syz-executor.2
13972     1   804     0  S       uwait   0xfffff800691ccf00  syz-executor.2
13967     1   791     0  S       uwait   0xfffff8005b03ab00  syz-executor.0
13959     1   791     0  S       uwait   0xfffff8003bfbc300  syz-executor.0
13936     1   792     0  S       uwait   0xfffff80015d4df00  syz-executor.1
13933     1   792     0  S       uwait   0xfffff80015d52900  syz-executor.1
13931     1   792     0  S       uwait   0xfffff8003bbad500  syz-executor.1
13929     1   792     0  S       uwait   0xfffff8003bc2a300  syz-executor.1
13922     1   792     0  S       uwait   0xfffff80015d50b00  syz-executor.1
13918     1   792     0  S       uwait   0xfffff8003bae4500  syz-executor.1
13917     1   791     0  S       uwait   0xfffff8005b03a300  syz-executor.0
13511     1   806     0  S       uwait   0xfffff8003bfbc900  syz-executor.3
13509     1   806     0  S       uwait   0xfffff8003bc2a500  syz-executor.3
13507     1   806     0  S       uwait   0xfffff8003bae4b00  syz-executor.3
13504     1   806     0  S       uwait   0xfffff8005b03a900  syz-executor.3
13488     1   792     0  S       uwait   0xfffff8003bc2a700  syz-executor.1
13487     1   792     0  S       uwait   0xfffff8003bae4700  syz-executor.1
13051     1   792     0  S       uwait   0xfffff8003bbad100  syz-executor.1
13050     1   792     0  S       uwait   0xfffff8003bbadb00  syz-executor.1
12931     1   791     0  S       uwait   0xfffff80015d52300  syz-executor.0
12923     1   791     0  S       uwait   0xfffff8005b03a100  syz-executor.0
12886     1   792     0  S       uwait   0xfffff80015d4b700  syz-executor.1
12884     1   792     0  S       uwait   0xfffff80015d49700  syz-executor.1
12439     1 12439    65  Ss      select  0xfffff800475fc2c0  dhclient
12077     1 12077     0  Ss      select  0xfffff8003b41b5c0  dhclient
12073     1 12073     0  Ss      select  0xfffff8006913eec0  dhclient
12048     1 12048    65  Ss      select  0xfffff80047ab8740  dhclient
11660     1   792     0  S       uwait   0xfffff8003bd75300  syz-executor.1
11657     1   792     0  S       uwait   0xfffff80047459f00  syz-executor.1
11630     1 11630     0  Ss      select  0xfffff8003be175c0  dhclient
11627     1 11627     0  Ss      select  0xfffff8003bd41bc0  dhclient
11608     1 11608    65  Ss      select  0xfffff8005b25e2c0  dhclient
11394     1   804     0  S       uwait   0xfffff80015d49500  syz-executor.2
11377     1   792     0  S       uwait   0xfffff8003bc2a100  syz-executor.1
11373     1   792     0  S       uwait   0xfffff8003bbad900  syz-executor.1
 8547     1   792     0  S       uwait   0xfffff80047459d00  syz-executor.1
 8541     1   792     0  S       uwait   0xfffff80047459500  syz-executor.1
 8504     1   806     0  S       uwait   0xfffff80015d7d500  syz-executor.3
 8503     1   806     0  S       uwait   0xfffff8003bc2af00  syz-executor.3
 8499     1   806     0  S       uwait   0xfffff80015d7d700  syz-executor.3
 8496     1   806     0  S       uwait   0xfffff80015d4b900  syz-executor.3
 8044     1  8044     0  Ss      select  0xfffff80047b0d5c0  dhclient
 8037     1  8037     0  Ss      select  0xfffff800476e55c0  dhclient
 7998     1  7998    65  Ss      select  0xfffff8003bf6c8c0  dhclient
 6085     1   792     0  SV      uwait   0xfffff80015d52500  syz-executor.1
 6084     1   792     0  SV      uwait   0xfffff8003bae4d00  syz-executor.1
 5590     1   806     0  S       uwait   0xfffff80015d50f00  syz-executor.3
 5582     1   806     0  S       uwait   0xfffff8003bae4900  syz-executor.3
 1715     1  1715     0  Ss      select  0xfffff8003be858c0  dhclient
 1706     1  1706     0  Ss      select  0xfffff8003be318c0  dhclient
 1305  1301   804     0  SV      uwait   0xfffff80015d4b300  syz-executor.2
 1301     1   804     0  RV                                  syz-executor.2
 1271     1   791     0  S       uwait   0xfffff80015d49f00  syz-executor.0
 1264     1   791     0  S       uwait   0xfffff80015d52b00  syz-executor.0
  806   780   806     0  Rs                                  syz-executor.3
  804   780   804     0  Ds      ktrace_ 0xffffffff8271e7e8  syz-executor.2
  792   780   792     0  Ds      getblk  0xfffffe000382a948  syz-executor.1
  791   780   791     0  Ds      getblk  0xfffffe000382a948  syz-executor.0
  780   778   778     0  R       (threaded)                  syz-fuzzer
100105                   S       uwait   0xfffff80015d6df00  syz-fuzzer
100122                   S       uwait   0xfffff80015d4db00  syz-fuzzer
100123                   S       uwait   0xfffff80015d50300  syz-fuzzer
100124                   S       uwait   0xfffff80015d88d00  syz-fuzzer
100125                   S       uwait   0xfffff80015d7dd00  syz-fuzzer
100126                   S       uwait   0xfffff80015d4bf00  syz-fuzzer
100127                   S       kqread  0xfffff80015d49300  syz-fuzzer
100133                   Run     CPU 1                       syz-fuzzer
  778   776   778     0  Ss      pause   0xfffff8003b3500b0  csh
  776   694   776     0  Ds      ktrace_ 0xffffffff8271e7e8  sshd
  760     1   760     0  Ss+     ttyin   0xfffff80015465cb0  getty
  759     1   759     0  Ss+     ttyin   0xfffff80015afe4b0  getty
  758     1   758     0  Ss+     ttyin   0xfffff80015afecb0  getty
  757     1   757     0  Ss+     ttyin   0xfffff80015a864b0  getty
  756     1   756     0  Ss+     ttyin   0xfffff80015a86cb0  getty
  755     1   755     0  Ss+     ttyin   0xfffff80015a8a4b0  getty
  754     1   754     0  Ss+     ttyin   0xfffff80015a8acb0  getty
  753     1   753     0  Ss+     ttyin   0xfffff80015a8d4b0  getty
  752     1   752     0  Ss+     ttyin   0xfffff80015a8dcb0  getty
  698     1   698     0  Ss      nanslp  0xffffffff8273c561  cron
  694     1   694     0  Ss      select  0xfffff8003b34d740  sshd
  507     1   507     0  Ss      select  0xfffff80015d7eec0  syslogd
  436     1   436     0  Ss      select  0xfffff80015ee0d40  devd
  435     1   435    65  Ss      select  0xfffff80015d4f740  dhclient
  350     1   350     0  Ss      select  0xfffff80015ee0140  dhclient
  347     1   347     0  Ss      select  0xfffff80015d4fa40  dhclient
   23     0     0     0  DL      vlruwt  0xfffff80015ca3530  [vnlru]
   22     0     0     0  DL      syncer  0xffffffff8282b9d0  [syncer]
   21     0     0     0  DL      (threaded)                  [bufdaemon]
100081                   D       qsleep  0xffffffff8282aa80  [bufdaemon]
100088                   D       -       0xffffffff8220ae80  [bufspacedaemon-0]
100097                   D       getblk  0xfffffe000382a948  [/ worker]
   20     0     0     0  DL      psleep  0xffffffff828528c8  [vmdaemon]
   19     0     0     0  DL      (threaded)                  [pagedaemon]
100079                   D       psleep  0xffffffff82846d38  [dom0]
100086                   D       launds  0xffffffff82846d44  [laundry: dom0]
100087                   D       umarcl  0xffffffff815cd2b0  [uma]
   18     0     0     0  DL      -       0xffffffff82570908  [rand_harvestq]
   17     0     0     0  DL      waiting 0xffffffff830e0828  [sctp_iterator]
   16     0     0     0  DL      pftm    0xffffffff82e973c0  [pf purge]
   15     0     0     0  DL      -       0xffffffff828280dc  [soaiod4]
    9     0     0     0  DL      -       0xffffffff828280dc  [soaiod3]
    8     0     0     0  DL      -       0xffffffff828280dc  [soaiod2]
    7     0     0     0  DL      -       0xffffffff828280dc  [soaiod1]
    6     0     0     0  RL      (threaded)                  [cam]
100044                   Run     CPU 0                       [doneq0]
100045                   D       -       0xffffffff82447d40  [async]
100078                   D       -       0xffffffff82447c10  [scanner]
   14     0     0     0  DL      seqstat 0xfffff80004dcbc88  [sequencer 00]
    5     0     0     0  DL      crypto_ 0xfffff80004d99d80  [crypto returns 1]
    4     0     0     0  DL      crypto_ 0xfffff80004d99d30  [crypto returns 0]
    3     0     0     0  DL      crypto_ 0xffffffff82844220  [crypto]
   13     0     0     0  DL      (threaded)                  [geom]
100035                   D       -       0xffffffff8271bda0  [g_event]
100036                   D       -       0xffffffff8271bda8  [g_up]
100037                   D       -       0xffffffff8271bdb0  [g_down]
    2     0     0     0  DL      (threaded)                  [KTLS]
100028                   D       -       0xfffff80004c39d00  [thr_0]
100029                   D       -       0xfffff80004c39d80  [thr_1]
   12     0     0     0  WL      (threaded)                  [intr]
100012                   I                                   [swi5: fast taskq]
100015                   I                                   [swi6: task queue]
100017                   I                                   [swi6: Giant taskq]
100030                   I                                   [swi4: clock (0)]
100031                   I                                   [swi4: clock (1)]
100032                   I                                   [swi1: netisr 0]
100033                   I                                   [swi3: vm]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq10: virtio_pci2]
100061                   I                                   [irq1: atkbd0]
100062                   I                                   [irq12: psm0]
100063                   I                                   [swi0: uart uart++]
100071                   I                                   [swi1: pf send]
100084                   I                                   [swi1: hpts]
100085                   I                                   [swi1: hpts]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffff80004bc7530  [init]
   10     0     0     0  DL      audit_w 0xffffffff82844730  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff8271c330  [swapper]
100005                   D       -       0xfffff80004c63d00  [if_config_tqg_0]
100006                   D       -       0xfffff80004c63900  [softirq_0]
100007                   D       -       0xfffff80004c63500  [softirq_1]
100008                   D       -       0xfffff80004c63100  [if_io_tqg_0]
100009                   D       -       0xfffff80004c61d00  [if_io_tqg_1]
100010                   D       -       0xfffff80004c5fd00  [in6m_free taskq]
100011                   D       -       0xfffff80004c5f900  [aiod_kick taskq]
100013                   D       -       0xfffff80004c5f100  [kqueue_ctx taskq]
100014                   D       -       0xfffff80004c58d00  [pci_hp taskq]
100016                   D       -       0xfffff80004c58500  [inm_free taskq]
100018                   D       -       0xfffff80004c53d00  [linuxkpi_irq_wq]
100019                   D       -       0xfffff80004c53900  [thread taskq]
100020                   D       -       0xfffff80004c53500  [linuxkpi_short_wq_0]
100021                   D       -       0xfffff80004c53500  [linuxkpi_short_wq_1]
100022                   D       -       0xfffff80004c53500  [linuxkpi_short_wq_2]
100023                   D       -       0xfffff80004c53500  [linuxkpi_short_wq_3]
100024                   D       -       0xfffff80004c53100  [linuxkpi_long_wq_0]
100025                   D       -       0xfffff80004c53100  [linuxkpi_long_wq_1]
100026                   D       -       0xfffff80004c53100  [linuxkpi_long_wq_2]
100027                   D       -       0xfffff80004c53100  [linuxkpi_long_wq_3]
100034                   D       -       0xfffff80004c39900  [firmware taskq]
100038                   D       -       0xfffff80004c39500  [crypto_0]
100039                   D       -       0xfffff80004c39500  [crypto_1]
100055                   D       -       0xfffff800153dc900  [vtnet0 rxq 0]
100056                   D       -       0xfffff800153dc500  [vtnet0 txq 0]
100057                   D       -       0xfffff800153dc100  [vtnet0 rxq 1]
100058                   D       -       0xfffff800153c6d00  [vtnet0 txq 1]
100060                   D       vtbslp  0xfffff80015429500  [virtio_balloon]
100064                   D       -       0xfffff800153c6900  [mca taskq]
100066                   D       -       0xffffffff81e21531  [deadlkres]
100074                   D       -       0xfffff80015a19900  [acpi_task_0]
100075                   D       -       0xfffff80015a19900  [acpi_task_1]
100076                   D       -       0xfffff80015a19900  [acpi_task_2]
100077                   D       -       0xfffff80004c39100  [CAM taskq]
db> show all locks
Process 806 (syz-executor.3) thread 0xfffffe0094fe5900 (100132)
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe00039370e8) locked @ /syzkaller/managers/main/kernel/sys/ufs/ffs/ffs_softdep.c:14702
exclusive lockmgr ufs (ufs) r = 0 (0xfffff8003bf3ee70) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_subr.c:3008
exclusive lockmgr ufs (ufs) r = 0 (0xfffff8003be49930) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_cache.c:4463
Process 792 (syz-executor.1) thread 0xfffffe0094f87ac0 (100111)
exclusive lockmgr ufs (ufs) r = 0 (0xfffff800473c1cb0) locked @ /syzkaller/managers/main/kernel/sys/ufs/ffs/ffs_vfsops.c:2029
exclusive lockmgr ufs (ufs) r = 0 (0xfffff8003be49cb0) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_cache.c:4463
Process 791 (syz-executor.0) thread 0xfffffe0094f85e40 (100115)
exclusive lockmgr ufs (ufs) r = 0 (0xfffff80015f17cb0) locked @ /syzkaller/managers/main/kernel/sys/ufs/ffs/ffs_vfsops.c:2029
exclusive lockmgr ufs (ufs) r = 0 (0xfffff8003b4d3070) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_cache.c:4463
Process 780 (syz-fuzzer) thread 0xfffffe0094ffa1e0 (100133)
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xfffff8003bb9f098) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_subr.c:3679
exclusive sx ktrace_sx (ktrace_sx) r = 0 (0xffffffff8271e7e8) locked @ /syzkaller/managers/main/kernel/sys/kern/kern_ktrace.c:416
Process 21 (bufdaemon) thread 0xfffffe0055940560 (100097)
exclusive lockmgr ufs (ufs) r = 0 (0xfffff8003be49770) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_subr.c:3008
Process 6 (cam) thread 0xfffffe00042d1900 (100044)
exclusive rw SUrw (SUrw) r = 0 (0xfffff80004dfc400) locked @ /syzkaller/managers/main/kernel/sys/ufs/ffs/ffs_softdep.c:11479
db> show malloc
              Type        InUse        MemUse     Requests
         sysctloid        34288        12858K        34355
           pf_hash            5        11560K            5
            devbuf         4216         6982K         4244
          tcp_hpts            5         3219K            5
              kobj          328         2624K          488
             vtbuf           24         2064K           46
          filedesc           98         1570K        26919
          inodedep          573         1092K        13564
            newblk           65         1064K        14185
          vfscache            3         1035K            3
           subproc          221          942K        14556
               pcb           56          740K         9391
            acpica         1674          649K        55230
           callout            2          528K            2
         ufs_quota            1          520K            1
          vfs_hash            1          520K            1
              intr            4          480K            4
           pf_rule          105          420K          121
            dirrem          792          396K        13478
               bus          990          378K         3499
               BPF           46          337K           46
          freefile          788          296K        13446
            linker          348          270K          441
           kdtrace          293          229K        29280
            DEVFS1          107          214K          124
            module          512          192K          512
              umtx          418          183K          418
         vnet_data            1          176K            1
           tidhash            3          164K            3
           pagedep           17          144K        13435
        tfo_ccache            1          136K            1
          savedino          126          126K        11744
               sem            4          120K            4
               UMA          270          102K          270
     audit_evclass          236           89K          294
          mtx_pool            2           80K            2
              temp           35           77K         2577
          syncache            1           76K            1
          routetbl          136           74K          425
              vmem            3           72K            5
               msg            4           68K            4
            ifaddr           70           66K           72
          acpitask            1           64K            1
       ddb_capture            1           64K            1
            DEVFS3          126           63K          136
         sctp_timw          114           57K          114
       ether_multi          152           57K          162
        gtaskqueue           18           57K           18
            KTRACE          104           55K         4488
        DEVFS_RULE           56           54K           56
              kenv           95           52K           95
      eventhandler          133           50K          133
            kqueue          105           48K        14475
              ksem           25           45K         2181
           ithread           99           43K           99
              rman           84           42K          425
           pwddesc          103           39K        14437
         taskqueue           60           36K           60
              proc            3           34K            3
         ufs_mount            5           34K            6
            bus-sc           33           34K         1710
           lltable           39           33K           51
           devstat            4           33K            4
            DEVFSP           86           33K         4900
         hostcache            1           32K            1
               tty           16           32K           16
               shm            1           32K           28
              GEOM           60           29K          489
         in6_multi           65           29K           65
            kbdmux            6           28K            6
         sctp_atcl           26           26K         6161
              cred           26           26K          322
             ifnet            7           25K            7
         sctp_stro           12           24K         2842
            plimit           24           24K          467
          kcovinfo           64           24K           68
           pf_osfp           60           23K           60
         CAM queue            5           21K         1528
           CAM CCB            5           20K         2367
         pfs_nodes           20           20K           20
         proc-args           52           20K          683
             selfd           48           18K       211140
              UART           12           18K           12
           session           35           18K           52
         bmsafemap            2           17K        13534
          freework           33           17K        13421
               rpc            2           16K            2
             shmfd            1           16K           30
       pfs_vncache            1           16K            1
         sctp_atky           38           15K         9052
       ufs_dirhash           24           14K           24
            sglist            5           13K            5
               MCA           32           12K           32
       inpcbpolicy           32           12K         5674
           CAM DEV            3           12K          510
             lockf           32           12K          108
            Unitno           30           12K           47
                vt           11           11K           11
           acpisem           28           11K           28
           CAM XPT           22           11K          543
          freeblks           20           10K        13408
         sctp_athm           26           10K         6182
               osd           24           10K         4828
          sctp_map           24            9K         5684
           uidinfo            3            9K           23
          pf_ifnet           10            9K           19
           memdesc            1            8K            1
         ipsec-saq            2            8K            2
             evdev            4            8K            4
           acpidev           20            8K           20
             hhook           15            8K           17
            select           19            8K           88
             mount           16            7K           90
       fpukern_ctx            3            6K            3
          terminal           11            6K           11
            ip6ndp           12            6K           14
          sctp_ifa           13            5K           14
         newdirblk           12            5K        13414
       ipsecpolicy            2            5K            2
 encap_export_host           12            5K           12
             clone            9            5K            9
          pf_table            1            4K            1
        local_apic            1            4K            1
           io_apic            1            4K            1
            sahead            1            4K            1
          secasvar            1            4K            1
             nhops            6            4K            8
          pci_link           10            4K           10
               msi            9            4K            9
             DEVFS            9            4K           10
               tun            7            4K            7
             ipsec            3            3K            3
             xform            8            3K         5345
          in_multi            6            3K            8
          nexusdev            7            3K            7
            feeder            7            3K            7
          procdesc            5            3K           18
          sctp_ifn            6            3K           14
               mld            6            3K            6
              igmp            6            3K            6
         toponodes            6            3K            6
            prison            6            3K            6
            isadev            6            3K            6
           softdep            1            2K            1
       vnodemarker            2            2K           36
      NFSD session            1            2K            1
             linux            5            2K            6
        CAM periph            4            2K          271
          filecaps            5            2K          118
            crypto            4            2K            6
              pfil            4            2K            4
          CAM path            4            2K         1034
            soname            4            2K        15309
           tcpfunc            3            2K            3
        loginclass            3            2K           10
    chacha20random            1            1K            1
            vnodes            1            1K            1
           CAM SIM            2            1K            2
              ktls            1            1K            1
              cdev            2            1K            2
        aesni_data            2            1K            2
              cpus            2            1K            2
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
             mkdir            2            1K        26828
           entropy            2            1K           44
             sigio            2            1K           11
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
          indirdep            1            1K           11
          pmchooks            1            1K            1
          sctp_vrf            1            1K            1
            apmdev            1            1K            1
 CAM I/O Scheduler            1            1K            1
            diradd            1            1K        13516
    vnet_data_free            1            1K            1
              vnet            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
          acpiintr            1            1K            1
               pmc            1            1K            1
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         4609
         sctp_iter            0            0K           11
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           11
         sctp_aadr            0            0K           71
         sctp_stri            0            0K           22
          chd data            0            0K            0
            tcp_do            0            0K            0
           tcp_fsb            0            0K           93
           pf_altq            0            0K            0
           pf_temp            0            0K            0
            mqdata            0            0K            0
        cubic data            0            0K            0
        dctcp data            0            0K            0
         htcp data            0            0K            0
          cdg data            0            0K            0
        vegas data            0            0K            2
       NFSD V4lock            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
      NFSD V4state            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
              iavf            0            0K            0
               ixl            0            0K            0
            DEVFS4            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
             axgbe            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         ciss_data            0            0K            0
         BACKLIGHT            0            0K            0
               xnb            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
            vtfont            0            0K            0
          xen_intr            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            qpidrv            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
               amr            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
     vm_fictitious            0            0K            0
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
       AHCI driver            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               agp            0            0K            0
           nvme_da            0            0K            0
           UMAHash            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           15
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K           49
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            twsbuf            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
            lkpifw            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K           39
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
      twe_commands            0            0K            0
               LRO            0            0K            0
      newreno data            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            8
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
      twa_commands            0            0K            0
            statfs            0            0K        14434
     namei_tracker            0            0K          812
       export_host            0            0K            0
        cl_savebuf            0            0K           14
       tcp_log_dev            0            0K         1110
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            0
               acl            0            0K            0
          tempbuff            0            0K            0
          mbuf_tag            0            0K          114
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        24758
          ioctlops            0            0K         3490
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
              sbuf            0            0K          288
        md_sectors            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
           md_disk            0            0K            0
              SWAP            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
         sysctltmp            0            0K          740
            sysctl            0            0K            3
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
       aacraid_buf            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
             cache            0            0K            0
            iirbuf            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
         scsi_pass            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8322     981     1424826       0     254   38105088        0
        malloc-384    384   34796      84       65657       0      30   13393920        0
       malloc-1024   1024    4154      10        4484       0      16    4263936        0
       malloc-1024   1024     640    3508       13704       0      16    4247552        0
        RADIX NODE    144   17574    8491      398798       0      63    3753360        0
       malloc-1024   1024     128    3252       12079       0      16    3461120        0
       malloc-8192   8192     329       2        1262       0       1    2711552        0
              pbuf   2624       0     973           0       0       2    2553152        0
              mbuf    256    8744     826     2772442       0     254    2449920        0
        malloc-384    384    2837    3303      157465       0      30    2357760        0
           tcp_log    416      96    5448       97467       0     254    2306304        0
        malloc-512    512     918    3250       41379       0      30    2134016        0
      malloc-32768  32768      52       8       13606       0       1    1966080        0
          BUF TRIE    144     196   13272         878       0      62    1939392        0
       UMA Slabs 0    112   14511      30       14511       0     126    1628592        0
      mbuf_cluster   2048     540       2         540       0     254    1110016        0
       malloc-8192   8192     112       8       14473       0       1     983040        0
         FFS inode   1160     543      24       13997       0       8     657720        0
        malloc-384    384    1559      61       50428       0      30     622080        0
         VM OBJECT    264    1814     106      204940       0      30     506880        0
       malloc-4096   4096     115       0         357       0       2     471040        0
      malloc-16384  16384      26       0          30       0       1     425984        0
        malloc-384    384    1010      50        8728       0      30     407040        0
         MAP ENTRY     96    3741     459      774496       0     126     403200        0
    ertt_txseginfo     40       0    9696      606609       0     254     387840        0
            THREAD   1808     189      20       14842       0       8     377872        0
            lkpimm    160       1    2324           1       0      62     372000        0
          lkpicurr    160       2    2323           2       0      62     372000        0
        256 Bucket   2048     141      17       19985       0       8     323584        0
       malloc-8192   8192      17      21        3311       0       1     311296        0
       tcp_bbr_map    128       4    2352       36815       0     126     301568        0
             VNODE    448     582      66       14037       0      30     290304        0
       malloc-1024   1024     250      30       15776       0      16     286720        0
        malloc-384    384     683      17         957       0      30     268800        0
      malloc-65536  65536       4       0           4       0       1     262144        0
         tcp_inpcb    488      39     465        4816       0     254     245952        0
      malloc-16384  16384      10       4         273       0       1     229376        0
           VMSPACE   2544      78      12       14411       0       4     228960        0
       malloc-2048   2048     107       3         124       0       8     225280        0
            DEVCTL   1024       0     216         132       0       0     221184        0
       malloc-1024   1024     185      31        7057       0      16     221184        0
      malloc-65536  65536       1       2         178       0       1     196608        0
         UMA Zones    768     242       2         242       0      16     187392        0
            socket    944      65     127       10583       0     254     181248        0
       malloc-4096   4096       8      36        2371       0       2     180224        0
        malloc-512    512     339       5        1156       0      30     176128        0
         sctp_asoc   2288      12      63        2842       0     254     171600        0
         vmem btag     56    2829     111        2829       0     254     164640        0
       malloc-8192   8192      16       4         170       0       1     163840        0
              PROC   1328     102      15       14436       0       8     155376        0
       S VFS Cache    104    1083     360       15672       0     126     150072        0
       FFS2 dinode    256     543      27       13997       0      62     145920        0
             tcpcb   1064      20     113        4816       0     254     141512        0
       mbuf_packet    256      28     512       16232       0     254     138240        0
         filedesc0   1072     103      23       14437       0       8     135072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
        malloc-384    384     273      47      229651       0      30     122880        0
          ksiginfo    112     101     943        1840       0     126     116928        0
         32 Bucket    256     295     125       29497       0      62     107520        0
        malloc-512    512     197      11        1347       0      30     106496        0
       malloc-2048   2048      20      30        3358       0       8     102400        0
      malloc-16384  16384       6       0           6       0       1      98304        0
        malloc-512    512     141      43        1328       0      30      94208        0
         16 Bucket    144     478     166        4851       0      62      92736        0
          UMA Kegs    384     227       6         227       0      30      89472        0
        128 Bucket   1024      55      32        2056       0      16      89088        0
       malloc-1024   1024      46      38        3345       0      16      86016        0
             g_bio    408      21     189        6915       0      30      85680        0
           sctp_ep   1280      14      52        3308       0     254      84480        0
            clpbuf   2624       0      32          21       0      16      83968        0
        malloc-384    384     147      63        7190       0      30      80640        0
        malloc-512    512     112      24       13871       0      30      69632        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-16384  16384       1       3          53       0       1      65536        0
        sctp_raddr    736      12      76        2847       0     254      64768        0
       malloc-8192   8192       2       5       14440       0       1      57344        0
       malloc-1024   1024      53       3          57       0      16      57344        0
       malloc-4096   4096      11       2        1018       0       2      53248        0
         64 Bucket    512      45      59       11145       0      30      53248        0
      malloc-16384  16384       3       0           3       0       1      49152        0
       malloc-8192   8192       4       2         534       0       1      49152        0
       malloc-4096   4096       5       7         544       0       2      49152        0
            pcpu-8      8    5545     599        5829       0     254      49152        0
       malloc-2048   2048      18       4          22       0       8      45056        0
        malloc-384    384      46      64        4889       0      30      42240        0
           DIRHASH   1024      34       6          34       0      16      40960        0
       malloc-4096   4096       8       2        1438       0       2      40960        0
       malloc-2048   2048       9      11        1541       0       8      40960        0
       tcp_bbr_pcb    832      12      33        3553       0      16      37440        0
             NAMEI   1024       2      34       79659       0      16      36864        0
      malloc-16384  16384       2       0           2       0       1      32768        0
           pcpu-64     64     486      26         486       0     254      32768        0
             Files     80     280     120       43565       0     126      32000        0
         TURNSTILE    136     210      21         210       0      62      31416        0
              pipe    744      23      17         739       0      16      29760        0
       malloc-4096   4096       1       6         282       0       2      28672        0
       malloc-4096   4096       7       0           7       0       2      28672        0
        malloc-512    512      47       9       14171       0      30      28672        0
        malloc-512    512      32      24        2205       0      30      28672        0
        malloc-512    512      29      27       14651       0      30      28672        0
             KNOTE    160      28     147      299399       0      62      28000        0
       malloc-8192   8192       2       1          49       0       1      24576        0
       malloc-2048   2048       9       3          93       0       8      24576        0
               PWD     32      38     718       13502       0     254      24192        0
            ttyinq    160     135      15         300       0      62      24000        0
          8 Bucket     80      42     258        3260       0     126      24000        0
           ttyoutq    256      72      18         160       0      62      23040        0
       malloc-2048   2048       9       1           9       0       8      20480        0
          2 Bucket     32     100     530        3240       0     254      20160        0
        SLEEPQUEUE     88     210      14         210       0     126      19712        0
       Mountpoints   2752       2       5           2       0       4      19264        0
      malloc-16384  16384       1       0           1       0       1      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
       malloc-2048   2048       4       4           8       0       8      16384        0
      vtnet_tx_hdr     24       0     668     1260652       0     254      16032        0
             ripcb    488       6      26         487       0     254      15616        0
             unpcb    256      20      40        1559       0     254      15360        0
      tcp_rack_pcb    832       0      18          93       0      16      14976        0
              ertt     72      20     148        4816       0     126      12096        0
     udplite_inpcb    488       0      24         101       0     254      11712        0
         udp_inpcb    488       6      18         270       0     254      11712        0
              kenv    258      15      30        1060       0      30      11610        0
       malloc-4096   4096       0       2        1899       0       2       8192        0
       malloc-2048   2048       0       4           9       0       8       8192        0
       malloc-1024   1024       0       8          95       0      16       8192        0
           pf tags    104      15      63          15       0     126       8112        0
           rtentry    176      30      16          34       0      62       8096        0
              PGRP     88      35      57          52       0     126       8096        0
          rl_entry     40      70     132          70       0     254       8080        0
sctp_stream_msg_out    112       4      68          22       0     254       8064        0
        sctp_laddr     48       8     160          60       0     254       8064        0
             udpcb     32       6     246         371       0     254       8064        0
               ipq     56       7     137           7       0     254       8064        0
          4 Bucket     48       7     161         340       0     254       8064        0
        sctp_chunk    152       4      48          38       0     254       7904        0
     routing nhops    256      27       3          34       0      62       7680        0
     FPU_save_area    832       1       8           1       0      16       7488        0
            cpuset    104       7      55           7       0     126       6448        0
 epoch_record pcpu    256       4      12           4       0      62       4096        0
           pcpu-16     16       7     249           7       0     254       4096        0
   sctp_asconf_ack     48       0      84           4       0     254       4032        0
         hostcache     64       1      62           1       0     254       4032        0
          syncache    168       0      24           4       0     254       4032        0
        sctp_readq    152       0      26           1       0     254       3952        0
            itimer    352       0      11           2       0      30       3872        0
       UMA Slabs 1    176      11      11          11       0      62       3872        0
            mqnode    416       3       6           3       0      30       3744        0
        KMAP ENTRY     96      12      27          12       0       0       3744        0
              vmem   1856       1       1           1       0       8       3712        0
           SMR CPU     32       3      60           3       0     254       2016        0
        SMR SHARED     24       3      60           3       0     254       1512        0
       FFS1 dinode    128       0       0           0       0     126          0        0
            da_ccb    544       0       0           0       0      16          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      62          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    304       0       0           0       0     254          0        0
          pf mtags     48       0       0           0       0     254          0        0
      tcp_rack_map    112       0       0           0       0     126          0        0
      tcp_log_node    120       0       0           0       0     126          0        0
    tcp_log_bucket    176       0       0           0       0      62          0        0
          tcpreass     48       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     88       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            AIOLIO    272       0       0           0       0      30          0        0
             AIOCB    552       0       0           0       0      16          0        0
              AIOP     32       0       0           0       0     254          0        0
               AIO    208       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
           NCLNODE    584       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     24       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
      ktls_session    192       0       0           0       0      62          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0           0       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-256    256       0       0           0       0      62          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
        malloc-128    128       0       0           0       0     126          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-64     64       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-32     32       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
         malloc-16     16       0       0           0       0     254          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (98):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/05/23 20:06 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 19:06 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 16:08 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 14:04 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 11:30 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:31 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:25 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:25 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:23 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:15 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:15 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:12 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:09 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:09 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:07 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:06 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:01 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 08:00 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:58 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:55 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:54 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:53 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:50 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:50 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:47 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:44 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:43 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:42 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:33 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 07:10 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-main panic: Non-zero write count
2021/05/23 18:02 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 15:04 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 12:47 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 10:20 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 09:18 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:34 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:30 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:27 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:25 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:13 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 08:09 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:50 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:49 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:48 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:46 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:45 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:38 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:34 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
2021/05/23 07:32 freebsd-src c50346bcf5c5 3c7fef33 console log report ci-freebsd-i386 panic: Non-zero write count
* Struck through repros no longer work on HEAD.