syzbot


panic: filesystem goof: vop_panic[vop_readdir]

Status: fixed on 2021/10/14 00:24
Reported-by: syzbot+4006732c69fb0f792b2c@syzkaller.appspotmail.com
Fix commit: 03d5820f738d mount: Check for !VDIR mount points before handling -o emptydir
First crash: 1142d, last: 1142d

Sample crash report:
panic: filesystem goof: vop_panic[vop_readdir]
cpuid = 0
time = 2066
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe00a1659dd0
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe00a1659f30
vpanic() at vpanic+0x2c5/frame 0xfffffe00a165a010
panic() at panic+0xb5/frame 0xfffffe00a165a0d0
vop_panic() at vop_panic+0x32/frame 0xfffffe00a165a0f0
VOP_READDIR_APV() at VOP_READDIR_APV+0xb0/frame 0xfffffe00a165a130
vfs_emptydir() at vfs_emptydir+0x2e4/frame 0xfffffe00a165a2d0
vfs_domount() at vfs_domount+0xa4c/frame 0xfffffe00a165a890
vfs_donmount() at vfs_donmount+0xf65/frame 0xfffffe00a165a9b0
kernel_mount() at kernel_mount+0x148/frame 0xfffffe00a165aab0
cd9660_cmount() at cd9660_cmount+0x2a4/frame 0xfffffe00a165ac50
sys_mount() at sys_mount+0x3b5/frame 0xfffffe00a165ad30
ia32_syscall() at ia32_syscall+0x41f/frame 0xfffffe00a165af30
int0x80_syscall_common() at int0x80_syscall_common+0x9c/frame 0xfbffcf78
KDB: enter: panic
[ thread pid 16182 tid 226645 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x28dd94a(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0xe05b8f14a3c45001
rdx                    0x3ffff
rbx                          0
rsp         0xfffffe00a1659f10
rbp         0xfffffe00a1659f30
rsi                    0x40001
rdi         0xffffffff817c047a  vprintf+0x35a
r8                           0
r9          0xfffffe00a1659a48
r10                          0
r11         0xfffffe00a0a9e530
r12         0xfffffe00a0a9e020
r13         0xfffffe00a1659f01
r14         0xffffffff82c84680  .str.18
r15         0xffffffff82c84680  .str.18
rip         0xffffffff817b3deb  kdb_enter+0x6b
rflags                0x200046  kernload+0x46
kdb_enter+0x6b: movq    $0,0x28dd94a(%rip)
db> show proc
Process 16182 (syz-executor.3) at 0xfffffe009e481000:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 46542 at 0xfffffe00a0b56538
 ABI: FreeBSD ELF32
 flag: 0x10000080  flag2: 0
 arguments: /root/syz-executor.3 exec
 reaper: 0xfffffe0053d5d538 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffffe00a0a263e0
   (map 0xfffffe00a0a263e0)
   (map.pmap 0xfffffe00a0a264a0)
   (pmap 0xfffffe00a0a26500)
 threads: 2
147465                   RunQ                                syz-executor.3
226645                   Run     CPU 0                       syz-executor.3
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
16188 10108 10108     0  R       (threaded)                  syz-executor.0
100103                   RunQ                                syz-executor.0
226649                   S       uwait   0xfffffe00a0a64d80  syz-executor.0
16184   783   783     0  R                                   syz-executor.1
16182 46542 46542     0  R       (threaded)                  syz-executor.3
147465                   RunQ                                syz-executor.3
226645                   Run     CPU 0                       syz-executor.3
15603     1 15603    65  Ss      select  0xfffffe00a0ac7ec0  dhclient
11044     1 11044     0  Ss      select  0xfffffe009e4232c0  dhclient
11036     1 11036     0  Ss      select  0xfffffe009e423140  dhclient
10108   780 10108     0  Rs                                  syz-executor.0
51486     1 51486    65  Ss      select  0xfffffe009e4233c0  dhclient
50869     1 50869     0  Ss      select  0xfffffe00a0ac7940  dhclient
50866     1 50866     0  Ss      select  0xfffffe009e4235c0  dhclient
49607     1 49607    65  Ss      select  0xfffffe00a0ac7b40  dhclient
46923     1 46923     0  Ss      select  0xfffffe00a0ac7f40  dhclient
46916     1 46916     0  Ss      select  0xfffffe00a0ac7a40  dhclient
46854     1 46854    65  Ss      select  0xfffffe00a0ac7d40  dhclient
46542   780 46542     0  Rs                                  syz-executor.3
45887   780 45887     0  Rs                                  syz-executor.2
45048     1 45048     0  Ss      select  0xfffffe005889d040  dhclient
45045     1 45045     0  Ss      select  0xfffffe009e4236c0  dhclient
33600     0     0     0  DL      (threaded)                  [zfskern]
107695                   D       t->zthr 0xfffffe00a0bfb348  [arc_evict]
133940                   D       t->zthr 0xfffffe00a0bfb048  [arc_reap]
133941                   D       -       0xfffffe00a0a6b500  [dbu_evict]
133942                   D       dbuf_ev 0xffffffff85aa0380  [dbuf_evict_thread]
133943                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_0]
133944                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_1]
133945                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_2]
133946                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_3]
133947                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_4]
133948                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_5]
133949                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_6]
133950                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_7]
133951                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_8]
133952                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_9]
133953                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_10]
133954                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_11]
133955                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_12]
133956                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_13]
133957                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_14]
133958                   D       -       0xfffffe00a0a6ba00  [z_vdev_file_15]
133959                   D       l2arc_f 0xffffffff85a8f380  [l2arc_feed_thread]
133960                   D       -       0xfffffe00a0a6b600  [zfsvfs]
133961                   S       zevent_ 0xffffffff85aa1340  [sysevent]
  842     0     0     0  DL      aiordy  0xfffffe00a0a8a000  [aiod4]
  841     0     0     0  DL      aiordy  0xfffffe00a0a8a538  [aiod3]
  840     0     0     0  DL      aiordy  0xfffffe00a0a8aa70  [aiod2]
  839     0     0     0  DL      aiordy  0xfffffe00588b2000  [aiod1]
  783   780   783     0  Rs                                  syz-executor.1
  780   778   778     0  R       (threaded)                  syz-fuzzer
100113                   Run     CPU 1                       syz-fuzzer
100120                   RunQ                                syz-fuzzer
100121                   S       uwait   0xfffffe0058770e00  syz-fuzzer
100122                   S       uwait   0xfffffe0058770f00  syz-fuzzer
100123                   S       uwait   0xfffffe009e470800  syz-fuzzer
100124                   RunQ                                syz-fuzzer
100125                   S       uwait   0xfffffe0057546180  syz-fuzzer
100127                   S       uwait   0xfffffe009e470300  syz-fuzzer
101018                   S       uwait   0xfffffe00a0a77880  syz-fuzzer
187584                   S       uwait   0xfffffe00a81d8300  syz-fuzzer
  778   776   778     0  Ss      pause   0xfffffe009e9e5b20  csh
  776   694   776     0  Ss      select  0xfffffe009e4234c0  sshd
  760     1   760     0  Ss+     ttyin   0xfffffe00081e98b0  getty
  759     1   759     0  Ss+     ttyin   0xfffffe00575f3cb0  getty
  758     1   758     0  Ss+     ttyin   0xfffffe00583f00b0  getty
  757     1   757     0  Ss+     ttyin   0xfffffe00583f04b0  getty
  756     1   756     0  Ss+     ttyin   0xfffffe00583f08b0  getty
  755     1   755     0  Ss+     ttyin   0xfffffe00583f0cb0  getty
  754     1   754     0  Ss+     ttyin   0xfffffe005746e0b0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe005746e4b0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe005746e8b0  getty
  698     1   698     0  Ss      nanslp  0xffffffff8405e641  cron
  694     1   694     0  Ss      select  0xfffffe009e4238c0  sshd
  507     1   507     0  Ss      select  0xfffffe009e3231c0  syslogd
  436     1   436     0  Ss      select  0xfffffe005889d1c0  devd
  435     1   435    65  Ss      select  0xfffffe009e423840  dhclient
  350     1   350     0  Ss      select  0xfffffe009e423740  dhclient
  347     1   347     0  Ss      select  0xfffffe005889d140  dhclient
   23     0     0     0  DL      syncer  0xffffffff841869e0  [syncer]
   22     0     0     0  DL      vlruwt  0xfffffe0058768a70  [vnlru]
   21     0     0     0  DL      (threaded)                  [bufdaemon]
100085                   D       qsleep  0xffffffff841846e0  [bufdaemon]
100088                   D       -       0xffffffff83411f80  [bufspacedaemon-0]
100099                   D       sdflush 0xfffffe0058968ce8  [/ worker]
   20     0     0     0  DL      psleep  0xffffffff841bcc00  [vmdaemon]
   19     0     0     0  DL      (threaded)                  [pagedaemon]
100083                   D       psleep  0xffffffff841b0738  [dom0]
100089                   D       launds  0xffffffff841b0744  [laundry: dom0]
100090                   D       umarcl  0xffffffff81eb7490  [uma]
   18     0     0     0  DL      -       0xffffffff83e14c60  [rand_harvestq]
   17     0     0     0  DL      waiting 0xffffffff8491c6e0  [sctp_iterator]
   16     0     0     0  DL      pftm    0xffffffff84da12f0  [pf purge]
   15     0     0     0  DL      -       0xffffffff8417f900  [soaiod4]
    9     0     0     0  DL      -       0xffffffff8417f900  [soaiod3]
    8     0     0     0  DL      -       0xffffffff8417f900  [soaiod2]
    7     0     0     0  DL      -       0xffffffff8417f900  [soaiod1]
    6     0     0     0  DL      (threaded)                  [cam]
100048                   D       -       0xffffffff83c9a440  [doneq0]
100049                   D       -       0xffffffff83c9a3c0  [async]
100082                   D       -       0xffffffff83c9a240  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0056ab6888  [sequencer 00]
    5     0     0     0  DL      crypto_ 0xfffffe00081f3c80  [crypto returns 1]
    4     0     0     0  DL      crypto_ 0xfffffe00081f3c30  [crypto returns 0]
    3     0     0     0  DL      crypto_ 0xffffffff841aa7e0  [crypto]
   13     0     0     0  DL      (threaded)                  [geom]
100039                   D       -       0xffffffff84031b40  [g_event]
100040                   D       -       0xffffffff84031b80  [g_up]
100041                   D       -       0xffffffff84031bc0  [g_down]
    2     0     0     0  DL      (threaded)                  [KTLS]
100029                   D       -       0xfffffe00081edc00  [thr_0]
100030                   D       -       0xfffffe00081edc80  [thr_1]
100031                   D       -       0xffffffff84180c88  [alloc_0]
   12     0     0     0  WL      (threaded)                  [intr]
100011                   I                                   [swi6: task queue]
100012                   I                                   [swi6: Giant taskq]
100014                   I                                   [swi5: fast taskq]
100032                   I                                   [swi1: netisr 0]
100033                   I                                   [swi3: vm]
100034                   I                                   [swi4: clock (0)]
100035                   I                                   [swi4: clock (1)]
100036                   I                                   [swi1: hpts]
100037                   I                                   [swi1: hpts]
100050                   I                                   [irq24: virtio_pci0]
100051                   I                                   [irq25: virtio_pci0]
100052                   I                                   [irq26: virtio_pci0]
100053                   I                                   [irq27: virtio_pci0]
100054                   I                                   [irq28: virtio_pci1]
100055                   I                                   [irq29: virtio_pci1]
100056                   I                                   [irq30: virtio_pci1]
100057                   I                                   [irq31: virtio_pci1]
100058                   I                                   [irq32: virtio_pci1]
100063                   I                                   [irq10: virtio_pci2]
100065                   I                                   [irq1: atkbd0]
100066                   I                                   [irq12: psm0]
100067                   I                                   [swi0: uart uart++]
100075                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053d5d538  [init]
   10     0     0     0  DL      audit_w 0xffffffff841ab780  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff84032a40  [swapper]
100005                   D       -       0xfffffe0053df5e00  [softirq_0]
100006                   D       -       0xfffffe0053df5d00  [softirq_1]
100007                   D       -       0xfffffe0053df5c00  [if_io_tqg_0]
100008                   D       -       0xfffffe0053df5b00  [if_io_tqg_1]
100009                   D       -       0xfffffe0053df5a00  [if_config_tqg_0]
100010                   D       -       0xfffffe0053df5900  [deferred_unmount ta]
100013                   D       -       0xfffffe0053df5600  [thread taskq]
100015                   D       -       0xfffffe0053df5400  [linuxkpi_irq_wq]
100016                   D       -       0xfffffe0053df5300  [kqueue_ctx taskq]
100017                   D       -       0xfffffe0053df5200  [pci_hp taskq]
100018                   D       -       0xfffffe0053df5100  [aiod_kick taskq]
100019                   D       -       0xfffffe0053df5000  [inm_free taskq]
100020                   D       -       0xfffffe0053e0de00  [in6m_free taskq]
100021                   D       -       0xfffffe0053e0dd00  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe0053e0dd00  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe0053e0dd00  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe0053e0dd00  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe0053e0dc00  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe0053e0dc00  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe0053e0dc00  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe0053e0dc00  [linuxkpi_long_wq_3]
100038                   D       -       0xfffffe0053e0db00  [firmware taskq]
100042                   D       -       0xfffffe0053e0da00  [crypto_0]
100043                   D       -       0xfffffe0053e0da00  [crypto_1]
100059                   D       -       0xfffffe0053e0d800  [vtnet0 rxq 0]
100060                   D       -       0xfffffe0053e0d700  [vtnet0 txq 0]
100061                   D       -       0xfffffe0053e0d600  [vtnet0 rxq 1]
100062                   D       -       0xfffffe0053e0d500  [vtnet0 txq 1]
100064                   D       vtbslp  0xfffffe0056fe3e80  [virtio_balloon]
100068                   D       -       0xfffffe0053e0d400  [mca taskq]
100073                   D       -       0xffffffff82c8b601  [deadlkres]
100077                   D       -       0xfffffe0057545b00  [acpi_task_0]
100078                   D       -       0xfffffe0057545b00  [acpi_task_1]
100079                   D       -       0xfffffe0057545b00  [acpi_task_2]
100081                   D       -       0xfffffe0053e0d900  [CAM taskq]
133934                   D       -       0xfffffe00a0a68b00  [system_taskq_0]
133935                   D       -       0xfffffe00a0a68b00  [system_taskq_1]
133936                   D       -       0xfffffe00a0a6b300  [system_delay_taskq_]
133937                   D       -       0xfffffe00a0a6b300  [system_delay_taskq_]
133938                   D       -       0xfffffe00a0a6b700  [arc_prune_0]
133939                   D       -       0xfffffe00a0a6b700  [arc_prune_1]
db> show all locks
Process 16182 (syz-executor.3) thread 0xfffffe00a0a9e020 (226645)
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe00a8308770) locked @ /syzkaller/managers/i386/kernel/sys/kern/vfs_subr.c:2993
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            5        11524K            5
            linker          401        10471K        18858
            devbuf         4213         4323K         4242
          tcp_hpts            6         3201K            6
           solaris          129         2804K          199
         sysctloid        47448         2790K        47552
             vtbuf           24         1968K           46
              kobj          326         1304K          483
            newblk            9         1026K       115748
          vfscache            3         1025K            3
          inodedep          206          589K       121426
               pcb           32          542K        34715
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
           subproc          155          301K       121219
              vmem            3          276K            6
            acpica         1674          184K        55237
         vnet_data            1          168K            1
             mount          738          151K        28547
           tidhash            3          141K            3
           pagedep           12          131K       114246
        tfo_ccache            1          128K            1
          filedesc           16          121K       228848
            DEVFS1          109          109K          231
               sem            4          106K            4
               BPF           46           88K        14164
               bus          986           80K         3474
          mtx_pool            2           72K            2
          syncache            1           68K            1
            module          514           65K          514
          routetbl         1667           65K         5185
          acpitask            1           64K            1
       ddb_capture            1           64K            1
              umtx          440           55K          440
           kdtrace          262           53K       242790
             sigio          812           51K         5268
            dirrem          191           48K       121146
              temp           36           35K        17249
            DEVFS3          128           32K          173
         hostcache            1           32K            1
               shm            1           32K          342
               msg            4           30K            4
        gtaskqueue           18           26K           18
          freefile          191           24K       121144
            kbdmux            6           22K            6
            ifaddr           71           21K          394
        DEVFS_RULE           56           20K           56
         sctp_timw           75           19K           75
        kstat_data           17           17K           17
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
           ithread           99           16K           99
            bus-sc           34           15K         1691
           lltable           45           14K          424
       ether_multi          157           13K         1157
            KTRACE          100           13K          100
             ifnet            7           13K           42
              kenv           95           12K           95
      eventhandler          136           12K          136
         CAM queue            5           11K         1528
              GEOM           61           10K          491
              rman           84           10K          425
         in6_multi           71            9K          421
         bmsafemap            3            9K       121485
         taskqueue           81            9K           81
              UART           12            9K           12
           devstat            4            9K            4
              ksem            1            8K         2625
               rpc            2            8K            2
             shmfd            1            8K            1
       pfs_vncache            1            8K            1
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          295
               UMA          365            7K          365
              cred           26            7K          331
            kqueue           69            7K       158058
            sglist            5            7K            5
           CAM DEV            3            6K          510
            plimit           24            6K          834
            DEVFSP           80            5K        10498
       ufs_dirhash           24            5K           24
           session           35            5K         1257
          pf_ifnet           10            5K          194
           pwddesc           67            5K       116140
                vt           11            5K           11
           memdesc            1            4K            1
               MCA           32            4K           32
             evdev            4            4K            4
          kcovinfo           64            4K          663
             lockf           36            4K          430
           acpisem           28            4K           28
             selfd           55            4K      1543057
             hhook           15            4K           17
         proc-args           52            3K         1838
          terminal           11            3K           11
            select           19            3K           85
           uidinfo            3            3K           12
         sctp_stro            2            2K         3128
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         ipsec-saq            2            2K            2
          sctp_ifa           15            2K           86
            ip6ndp           12            2K           86
            Unitno           27            2K          257
           CAM XPT           22            2K          543
         sctp_atcl            4            2K        18521
          in_multi            6            2K           45
       vnodemarker            3            2K          189
            diradd           10            2K       121328
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
               msi            9            2K            9
             clone            9            2K            9
               tun            7            2K           42
           softdep            1            1K            1
             mkdir            8            1K       228438
            sahead            1            1K            1
          secasvar            1            1K            1
             nhops            6            1K            8
      NFSD session            1            1K            1
        CAM periph            4            1K          271
               osd           33            1K        67287
          freework            4            1K       114891
             ipsec            3            1K            3
          sctp_ifn            6            1K           86
          freeblks            3            1K       114890
               mld            6            1K           41
              igmp            6            1K           41
         toponodes            6            1K            6
            isadev            6            1K            6
          pci_link           10            1K           10
            crypto            4            1K            4
 encap_export_host           12            1K           12
          procdesc            5            1K           30
       inpcbpolicy           17            1K        78347
         newdirblk            4            1K       114219
              pfil            4            1K            4
              cdev            2            1K            2
    chacha20random            1            1K            1
         sctp_atky            6            1K        21649
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           10
            vnodes            1            1K            1
           CAM SIM            2            1K            2
              ktls            1            1K            1
            prison            8            1K            8
            feeder            7            1K            7
             taskq            2            1K            2
           tcpfunc            3            1K            3
        loginclass            3            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
             xform            2            1K           42
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          pmchooks            1            1K            1
            soname            4            1K        24822
          filecaps            5            1K          157
          sctp_vrf            1            1K            1
         sctp_athm            4            1K        18521
          sctp_map            4            1K         6256
              vnet            1            1K            1
           entropy            2            1K           83
          acpiintr            1            1K            1
               pmc            1            1K            1
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
         sfs_nodes            0            0K            0
        zones_data            0            0K            0
        ext2_mount            0            0K            0
         ext2_node            0            0K            0
      ext2_extents            0            0K            0
          pf_table            0            0K            0
           pf_rule            0            0K            0
           pf_altq            0            0K            0
           pf_osfp            0            0K            0
           pf_temp            0            0K            0
            tcp_do            0            0K            0
           tcp_fsb            0            0K         1254
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         3459
         sctp_iter            0            0K          134
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K          134
         sctp_aadr            0            0K            2
         sctp_stri            0            0K            0
          chd data            0            0K            0
          cdg data            0            0K            0
            mqdata            0            0K            0
        cubic data            0            0K            0
        dctcp data            0            0K            0
         htcp data            0            0K            0
        vegas data            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
              iavf            0            0K            0
               ixl            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
             axgbe            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
            DEVFS4            0            0K            0
          xen_intr            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            qpidrv            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
          xenstore            0            0K            0
         ciss_data            0            0K            0
         BACKLIGHT            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
               xnb            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
              xbbd            0            0K            0
            xenbus            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
     vm_fictitious            0            0K            0
            vtfont            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
               amr            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
           UMAHash            0            0K            0
            ata_da            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K       102033
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           92
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K            8
        allocindir            0            0K            0
          indirdep            0            0K           33
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
       AHCI driver            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               agp            0            0K            0
           nvme_da            0            0K            0
          ktls_ocf            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
            lkpifw            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K         1576
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K           49
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
               LRO            0            0K            0
      newreno data            0            0K          672
        ip_msource            0            0K            0
       ip_moptions            0            0K            0
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
            twsbuf            0            0K            0
            statfs            0            0K       114601
     namei_tracker            0            0K          360
       export_host            0            0K            0
        cl_savebuf            0            0K            4
      twe_commands            0            0K            0
      twa_commands            0            0K            0
       tcp_log_dev            0            0K           80
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K        35884
               acl            0            0K            0
    osti_cacheable            0            0K            0
          mbuf_tag            0            0K         1186
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        43237
          ioctlops            0            0K         9618
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
              sbuf            0            0K         1782
        md_sectors            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
           md_disk            0            0K            0
              SWAP            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
         sysctltmp            0            0K         2851
            sysctl            0            0K            3
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
       aacraid_buf            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
             cache            0            0K            0
            iirbuf            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
         scsi_pass            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8320     975      188762       0     254   38072320        0
              pbuf   2624       0     957           0       0       2    2511168        0
              mbuf    256    8689     986      655148       0     254    2476800        0
        malloc-128    128   16137      45       18530       0     126    2071296        0
          BUF TRIE    144     261   11499        2606       0      62    1693440        0
        malloc-384    384    4168      12        4270       0      30    1605120        0
      mbuf_cluster   2048     705       1         705       0     254    1445888        0
       malloc-4096   4096     328       5        1118       0       2    1363968        0
       UMA Slabs 0    112   10994      19       10994       0     126    1233456        0
        malloc-384    384     296    2784      126817       0      30    1182720        0
        RADIX NODE    144    7373     630     2989513       0      64    1152432        0
         vmem btag     56   18747      92       18747       0     254    1054984        0
         sctp_asoc   2288       2     388        3128       0     254     892320        0
      malloc-65536  65536       5       8          25       0       1     851968        0
        malloc-256    256     340    2945      234300       0      62     840960        0
         FFS inode   1160     659      27      124129       0       8     795760        0
        malloc-256    256     199    2801      251720       0      62     768000        0
       malloc-2048   2048       3     313       15394       0       8     647168        0
            DEVCTL   1024       0     528         589       0       0     540672        0
           sctp_ep   1272       2     388       15393       0     254     496080        0
         VM OBJECT    264    1603     122     1640247       0      30     455400        0
            THREAD   1808     194      26      126649       0       8     397760        0
        malloc-128    128     229    2840      128397       0     126     392832        0
            lkpimm    168       1    2327           1       0      62     391104        0
          lkpicurr    168       2    2326           2       0      62     391104        0
            socket    960      41     363      115588       0     254     387840        0
        256 Bucket   2048     157      19       15099       0       8     360448        0
             VNODE    448     699      93      124206       0      30     354816        0
         malloc-64     64    5246     235      121494       0     254     350784        0
       malloc-4096   4096      73       8      116247       0       2     331776        0
       malloc-1024   1024       6     318        3291       0      16     331776        0
         malloc-16     16   19552     448       93385       0     254     320000        0
      malloc-32768  32768       2       7          18       0       1     294912        0
        sctp_raddr    736       2     394        3128       0     254     291456        0
      malloc-65536  65536       2       2         562       0       1     262144        0
         UMA Zones    768     337       2         337       0      16     260352        0
        malloc-384    384      28     642       18658       0      30     257280        0
         malloc-32     32    7418     520      121895       0     254     254016        0
      malloc-16384  16384       9       5      114421       0       1     229376        0
         MAP ENTRY     96    1626     642     6265804       0     127     217728        0
      malloc-65536  65536       1       2          20       0       1     196608        0
      malloc-65536  65536       2       1          10       0       1     196608        0
       FFS2 dinode    256     659     106      124129       0      62     195840        0
     FPU_save_area    832     196      38      145859       0      16     194688        0
        malloc-256    256     100     650      235454       0      62     192000        0
             tcpcb   1096       4     171       67246       0     254     191800        0
       mbuf_packet    256       5     700      118170       0     254     180480        0
       malloc-4096   4096      33      10        2048       0       2     176128        0
       S VFS Cache    104    1114     407      127603       0     126     158184        0
        malloc-128    128    1132      77       25702       0     126     154752        0
       malloc-1024   1024     126       6         453       0      16     135168        0
zio_buf_comb_131072 131072       0       1           1       0       1     131072        0
      malloc-65536  65536       0       2         136       0       1     131072        0
           VMSPACE   2544      39      12      116153       0       4     129744        0
            clpbuf   2624       0      48          56       0      16     125952        0
          UMA Kegs    384     322       1         322       0      30     124032        0
         64 Bucket    512     177      55       51339       0      30     118784        0
          ksiginfo    112      72     972       19643       0     126     116928        0
              PROC   1336      66      21      116139       0       8     116232        0
      malloc-16384  16384       7       0           7       0       1     114688        0
       malloc-2048   2048      53       3        2080       0       8     114688        0
        128 Bucket   1024      74      33       22909       0      16     109568        0
       malloc-2048   2048       6      42        1038       0       9      98304        0
             g_bio    408       0     240       19774       0      30      97920        0
         filedesc0   1072      67      24      116140       0       8      97552        0
         tcp_inpcb    496       6     178       67246       0     254      91264        0
        malloc-128    128     555      96        7030       0     126      83328        0
         32 Bucket    256     174     126       24661       0      62      76800        0
       malloc-4096   4096      14       4        1026       0       2      73728        0
        malloc-256    256      66     204      115623       0      62      69120        0
         malloc-64     64     945     126       10218       0     254      68544        0
         malloc-32     32    1993     149        2962       0     254      68544        0
        malloc-128    128     324     203      233570       0     126      67456        0
        malloc-128    128     193     334      236174       0     126      67456        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2         130       0       1      65536        0
       malloc-8192   8192       6       2          26       0       1      65536        0
       malloc-2048   2048      18      14         472       0       8      65536        0
             unpcb    256      21     219       21716       0     254      61440        0
        malloc-256    256     171      39        3910       0      62      53760        0
        malloc-256    256      81     129      150701       0      62      53760        0
       malloc-1024   1024      14      38         526       0      16      53248        0
zio_data_buf_16384  16384       0       3           9       0       1      49152        0
       malloc-8192   8192       4       2          32       0       1      49152        0
       malloc-8192   8192       6       0           6       0       1      49152        0
            pcpu-8      8    5224     920       79538       0     254      49152        0
       malloc-2048   2048       3      19        2887       0       8      45056        0
        malloc-512    512      22      66       10056       0      30      45056        0
         malloc-64     64     199     494      129972       0     254      44352        0
         malloc-64     64     188     505     1660815       0     254      44352        0
         malloc-64     64     512     181        1592       0     254      44352        0
           DIRHASH   1024      34       6          34       0      16      40960        0
       malloc-8192   8192       5       0           5       0       1      40960        0
       malloc-8192   8192       3       2         106       0       1      40960        0
              pipe    744      23      32         841       0      16      40920        0
             Files     80     233     267      320941       0     126      40000        0
            da_ccb    544       0      70        5113       0      16      38080        0
             NAMEI   1024       0      36      591066       0      16      36864        0
        malloc-512    512       8      64        7996       0      30      36864        0
           pcpu-64     64     495      81         495       0     254      36864        0
sctp_stream_msg_out    112       0     324        1480       0     254      36288        0
         malloc-64     64     336     231       47920       0     254      36288        0
         malloc-64     64     475      92         759       0     254      36288        0
         malloc-32     32     150     984       97975       0     254      36288        0
        malloc-128    128     236      43        2381       0     126      35712        0
        malloc-128    128      73     206         725       0     126      35712        0
        malloc-256    256      92      43         543       0      62      34560        0
         TURNSTILE    136     221      31         221       0      62      34272        0
     zio_buf_16384  16384       0       2          10       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-16384  16384       0       2         160       0       1      32768        0
       malloc-4096   4096       1       7      114603       0       2      32768        0
         16 Bucket    144      88     136        1304       0      62      32256        0
             ripcb    496       5      59        3473       0     254      31744        0
         udp_inpcb    496       6      58        2825       0     254      31744        0
       tcp_bbr_pcb    832       0      36        1670       0      16      29952        0
       malloc-1024   1024       3      25          57       0      16      28672        0
             KNOTE    160      27     148     1062967       0      62      28000        0
     udplite_inpcb    496       0      56        4803       0     254      27776        0
       malloc-1024   1024      19       5        5358       0      16      24576        0
       malloc-1024   1024      18       6          18       0      16      24576        0
    ertt_txseginfo     40       0     606      167251       0     254      24240        0
        sctp_laddr     48       0     504        2014       0     254      24192        0
               PWD     32      20     736      114387       0     254      24192        0
            ttyinq    160     135      15         300       0      62      24000        0
               AIO    208       0     114       18659       0      62      23712        0
             AIOCB    552       0      42      113871       0      16      23184        0
           ttyoutq    256      72      18         160       0      62      23040        0
        malloc-384    384      23      37        1224       0      30      23040        0
        malloc-384    384      53       7          53       0      30      23040        0
        SLEEPQUEUE     88     221      35         221       0     126      22528        0
      tcp_rack_pcb    832       0      27        1254       0      16      22464        0
           tcp_log    416       0      54        1669       0     254      22464        0
       malloc-2048   2048       5       5        3771       0       8      20480        0
        malloc-512    512       4      36         545       0      30      20480        0
         malloc-16     16     304     946       18997       0     254      20000        0
            itimer    352       0      55        7360       0      30      19360        0
       Mountpoints   2752       2       5           2       0       4      19264        0
      malloc-16384  16384       1       0           1       0       1      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
       malloc-8192   8192       2       0           2       0       1      16384        0
       malloc-8192   8192       2       0           2       0       1      16384        0
              ertt     72       4     220       67246       0     126      16128        0
         malloc-32     32     432      72       18616       0     254      16128        0
          2 Bucket     32      78     426       10890       0     254      16128        0
      vtnet_tx_hdr     24       0     668      171469       0     254      16032        0
         malloc-16     16      50     950       31464       0     254      16000        0
          8 Bucket     80      82     118       18222       0     126      16000        0
        sctp_chunk    152       2     102         111       0     254      15808        0
     routing nhops    256      27      33         179       0      62      15360        0
        malloc-384    384      20      20        2808       0      30      15360        0
       malloc-4096   4096       1       2           8       0       2      12288        0
           rtentry    176      31      38         179       0      62      12144        0
              PGRP     88      35     103        1257       0     126      12144        0
         malloc-32     32      27     351        3715       0     254      12096        0
              kenv    258      15      30        1155       0      30      11610        0
        malloc-256    256      19      26          21       0      62      11520        0
       malloc-2048   2048       0       4          50       0       8       8192        0
       malloc-2048   2048       1       3         192       0       8       8192        0
       malloc-1024   1024       6       2           6       0      16       8192        0
        malloc-512    512       0      16        2271       0      30       8192        0
        malloc-512    512       2      14          61       0      30       8192        0
          rl_entry     40     100     102         100       0     254       8080        0
             udpcb     32       6     246        7628       0     254       8064        0
               ipq     56       0     144          30       0     254       8064        0
         malloc-64     64       9     117         505       0     254       8064        0
         malloc-32     32      62     190       23279       0     254       8064        0
         malloc-32     32     107     145         921       0     254       8064        0
         malloc-32     32      12     240        3461       0     254       8064        0
          4 Bucket     48       7     161        2378       0     254       8064        0
         malloc-16     16      39     461       18563       0     254       8000        0
         malloc-16     16      52     448          95       0     254       8000        0
         malloc-16     16     216     284        2325       0     254       8000        0
         malloc-16     16       4     496           6       0     254       8000        0
         malloc-16     16      15     485       25081       0     254       8000        0
        malloc-384    384       0      20         340       0      30       7680        0
            AIOLIO    272       0      28       17926       0      30       7616        0
            cpuset    104       8      54           8       0     126       6448        0
         abd_chunk   4096       1       0           1       0       2       4096        0
 epoch_record pcpu    256       4      12           4       0      62       4096        0
       malloc-1024   1024       1       3           1       0      16       4096        0
        malloc-512    512       1       7           2       0      30       4096        0
        malloc-512    512       1       7           1       0      30       4096        0
           pcpu-16     16       7     249           7       0     254       4096        0
         hostcache     64       1      62           1       0     254       4032        0
          syncache    168       0      24           4       0     254       4032        0
              AIOP     32       4     122           4       0     254       4032        0
        sctp_readq    152       0      26           1       0     254       3952        0
       UMA Slabs 1    176      11      11          11       0      62       3872        0
        malloc-384    384       3       7           3       0      30       3840        0
            mqnode    416       3       6           3       0      30       3744        0
        KMAP ENTRY     96      21      18          30       0       0       3744        0
              vmem   1856       1       1           1       0       8       3712        0
           SMR CPU     32       3      60           3       0     254       2016        0
        SMR SHARED     24       3      60           3       0     254       1512        0
   zfs_znode_cache    464       0       0           0       0      30          0        0
       sio_cache_2    168       0       0           0       0      62          0        0
       sio_cache_1    152       0       0           0       0      62          0        0
       sio_cache_0    136       0       0           0       0      62          0        0
     zil_zcw_cache     80       0       0           0       0     126          0        0
     zil_lwb_cache    352       0       0           0       0      30          0        0
    dmu_buf_impl_t    408       0       0           0       0      30          0        0
         arc_buf_t     64       0       0           0       0     254          0        0
arc_buf_hdr_t_l2only     96       0       0           0       0     126          0        0
arc_buf_hdr_t_full_crypt    416       0       0           0       0      30          0        0
arc_buf_hdr_t_full    352       0       0           0       0      30          0        0
           dnode_t   1032       0       0           0       0       8          0        0
          sa_cache    296       0       0           0       0      30          0        0
zio_buf_comb_16777216 16777216       0       0           0       0       1          0        0
zio_buf_comb_14680064 14680064       0       0           0       0       1          0        0
zio_buf_comb_12582912 12582912       0       0           0       0       1          0        0
zio_buf_comb_10485760 10485760       0       0           0       0       1          0        0
zio_buf_comb_8388608 8388608       0       0           0       0       1          0        0
zio_buf_comb_7340032 7340032       0       0           0       0       1          0        0
zio_buf_comb_6291456 6291456       0       0           0       0       1          0        0
zio_buf_comb_5242880 5242880       0       0           0       0       1          0        0
zio_buf_comb_4194304 4194304       0       0           0       0       1          0        0
zio_buf_comb_3670016 3670016       0       0           0       0       1          0        0
zio_buf_comb_3145728 3145728       0       0           0       0       1          0        0
zio_buf_comb_2621440 2621440       0       0           0       0       1          0        0
zio_buf_comb_2097152 2097152       0       0           0       0       1          0        0
zio_buf_comb_1835008 1835008       0       0           0       0       1          0        0
zio_buf_comb_1572864 1572864       0       0           0       0       1          0        0
zio_buf_comb_1310720 1310720       0       0           0       0       1          0        0
zio_buf_comb_1048576 1048576       0       0           0       0       1          0        0
zio_buf_comb_917504 917504       0       0           0       0       1          0        0
zio_buf_comb_786432 786432       0       0           0       0       1          0        0
zio_buf_comb_655360 655360       0       0           0       0       1          0        0
zio_buf_comb_524288 524288       0       0           0       0       1          0        0
zio_buf_comb_458752 458752       0       0           0       0       1          0        0
zio_buf_comb_393216 393216       0       0           0       0       1          0        0
zio_buf_comb_327680 327680       0       0           0       0       1          0        0
zio_buf_comb_262144 262144       0       0           0       0       1          0        0
zio_buf_comb_229376 229376       0       0           0       0       1          0        0
zio_buf_comb_196608 196608       0       0           0       0       1          0        0
zio_buf_comb_163840 163840       0       0           0       0       1          0        0
zio_buf_comb_114688 114688       0       0           0       0       1          0        0
zio_buf_comb_98304  98304       0       0           0       0       1          0        0
zio_buf_comb_81920  81920       0       0           0       0       1          0        0
zio_buf_comb_65536  65536       0       0           0       0       1          0        0
zio_buf_comb_57344  57344       0       0           0       0       1          0        0
zio_buf_comb_49152  49152       0       0           0       0       1          0        0
zio_buf_comb_40960  40960       0       0           0       0       1          0        0
zio_buf_comb_32768  32768       0       0           0       0       1          0        0
zio_buf_comb_28672  28672       0       0           0       0       1          0        0
zio_buf_comb_24576  24576       0       0           0       0       1          0        0
zio_buf_comb_20480  20480       0       0           0       0       1          0        0
zio_data_buf_14336  14336       0       0           0       0       1          0        0
     zio_buf_14336  14336       0       0           0       0       1          0        0
zio_data_buf_12288  12288       0       0           0       0       1          0        0
     zio_buf_12288  12288       0       0           0       0       1          0        0
zio_data_buf_10240  10240       0       0           0       0       1          0        0
     zio_buf_10240  10240       0       0           0       0       1          0        0
 zio_data_buf_8192   8192       0       0           0       0       1          0        0
      zio_buf_8192   8192       0       0           0       0       1          0        0
 zio_data_buf_7168   7168       0       0           0       0       1          0        0
      zio_buf_7168   7168       0       0           0       0       1          0        0
 zio_data_buf_6144   6144       0       0           0       0       1          0        0
      zio_buf_6144   6144       0       0           0       0       1          0        0
 zio_data_buf_5120   5120       0       0           0       0       1          0        0
      zio_buf_5120   5120       0       0           0       0       1          0        0
 zio_data_buf_4096   4096       0       0           0       0       2          0        0
      zio_buf_4096   4096       0       0           0       0       2          0        0
 zio_data_buf_3584   3584       0       0           0       0       2          0        0
      zio_buf_3584   3584       0       0           0       0       2          0        0
 zio_data_buf_3072   3072       0       0           0       0       4          0        0
      zio_buf_3072   3072       0       0           0       0       4          0        0
 zio_data_buf_2560   2560       0       0           0       0       4          0        0
      zio_buf_2560   2560       0       0           0       0       4          0        0
 zio_data_buf_2048   2048       0       0           0       0       8          0        0
      zio_buf_2048   2048       0       0           0       0       8          0        0
 zio_data_buf_1536   1536       0       0           0       0       8          0        0
      zio_buf_1536   1536       0       0           0       0       8          0        0
 zio_data_buf_1024   1024       0       0           0       0      16          0        0
      zio_buf_1024   1024       0       0           0       0      16          0        0
  zio_data_buf_512    512       0       0           0       0      30          0        0
       zio_buf_512    512       0       0           0       0      30          0        0
    zio_link_cache     48       0       0           0       0     254          0        0
         zio_cache   1232       0       0           0       0       8          0        0
   ddt_entry_cache    392       0       0           0       0      30          0        0
         ddt_cache  24824       0       0           0       0       1          0        0
metaslab_alloc_trace_cache     72       0       0           0       0     126          0        0
zfs_btree_leaf_cache   4096       0       0           0       0       2          0        0
reference_history_cache      8       0       0           0       0     254          0        0
   reference_cache     40       0       0           0       0     254          0        0
        taskq_zone    192       0       0           0       0      62          0        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0      62          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0     254          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags     48       0       0           0       0     254          0        0
       tcp_bbr_map    128       0       0           0       0     126          0        0
      tcp_rack_map    112       0       0           0       0     126          0        0
      tcp_log_node    120       0       0           0       0     126          0        0
    tcp_log_bucket    176       0       0           0       0      62          0        0
          tcpreass     48       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     88       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    592       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
      ktls_session    192       0       0           0       0      62          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0           0       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
        malloc-512    512       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/10/08 19:05 freebsd-src 8ff2b5224140 efe0f24d console log report ci-freebsd-i386 panic: filesystem goof: vop_panic[vop_readdir]
* Struck through repros no longer work on HEAD.