syzbot


Fatal trap NUM: page fault in sctp_notify_stream_reset_add

Status: auto-obsoleted due to no activity on 2022/09/12 14:55
Reported-by: syzbot+98c1925ea1d3bfb7e2a2@syzkaller.appspotmail.com
First crash: 675d, last: 675d

Sample crash report:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x20c
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff847eef03
stack pointer	        = 0x0:0xfffffe0053b23c10
frame pointer	        = 0x0:0xfffffe0053b23c50
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, 
resume, IOPL = 0
current process		= 12 (swi1: netisr 0)
trap number		= 12
panic: page fault
cpuid = 0
time = 1655218484
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe0053b233f0
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe0053b23550
vpanic() at vpanic+0x254/frame 0xfffffe0053b23630
panic() at panic+0xb5/frame 0xfffffe0053b236f0
trap_fatal() at trap_fatal+0x6bd/frame 0xfffffe0053b23810
trap_pfault() at trap_pfault+0x182/frame 0xfffffe0053b23950
trap() at trap+0x5b3/frame 0xfffffe0053b23b40
calltrap() at calltrap+0x8/frame 0xfffffe0053b23b40
--- trap 0xc, rip = 0xffffffff847eef03, rsp = 0xfffffe0053b23c10, rbp = 0xfffffe0053b23c50 ---
sctp_notify_stream_reset_add() at sctp_notify_stream_reset_add+0x1c3/frame 0xfffffe0053b23c50
sctp_handle_stream_reset_response() at sctp_handle_stream_reset_response+0x87e/frame 0xfffffe0053b23d70
sctp_handle_stream_reset() at sctp_handle_stream_reset+0xa17/frame 0xfffffe0053b24170
sctp_process_control() at sctp_process_control+0x1bf8/frame 0xfffffe0053b24640
sctp_common_input_processing() at sctp_common_input_processing+0xb35/frame 0xfffffe0053b248a0
sctp6_input_with_port() at sctp6_input_with_port+0x597/frame 0xfffffe0053b24a70
sctp6_input() at sctp6_input+0x1f/frame 0xfffffe0053b24a90
ip6_input() at ip6_input+0x1f70/frame 0xfffffe0053b24cf0
swi_net() at swi_net+0x2ed/frame 0xfffffe0053b24d90
ithread_loop() at ithread_loop+0x4ee/frame 0xfffffe0053b24ef0
fork_exit() at fork_exit+0xd0/frame 0xfffffe0053b24f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0053b24f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100031 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x275b4ca(%rip)
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                           0
rax                       0x12
rcx         0xffffffff8172b5c6  printf+0xf6
rdx                        0x1
rbx                          0
rsp         0xfffffe0053b23530
rbp         0xfffffe0053b23550
rsi                          0
rdi         0xffffffff8172b628  printf+0x158
r8                           0
r9                  0xffffffff
r10                          0
r11                       0xbf
r12                          0
r13         0xfffffe0053ea8560
r14         0xffffffff82b5a980  .str.26
r15         0xffffffff82b5a980  .str.26
rip         0xffffffff8171b14b  kdb_enter+0x6b
rflags                    0x46
kdb_enter+0x6b: movq    $0,0x275b4ca(%rip)
db> show proc
Process 12 (intr) at 0xfffffe0053de2548:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xffffffff83e19cc0
 ABI: null
 flag: 0x10000284  flag2: 0
 reaper: 0xffffffff83e19cc0 reapsubtree: 12
 sigparent: 20
 vmspace: 0xffffffff83e1ac60
   (map 0xffffffff83e1ac60)
   (map.pmap 0xffffffff83e1ad20)
   (pmap 0xffffffff83e1ad88)
 threads: 22
100015                   I                                   [swi5: fast taskq]
100018                   I                                   [swi6: task queue]
100019                   I                                   [swi6: Giant taskq]
100031                   Run     CPU 0                       [swi1: netisr 0]
100032                   I                                   [swi1: hpts]
100033                   I                                   [swi1: hpts]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq33: virtio_pci2]
100060                   I                                   [irq34: virtio_pci2]
100061                   I                                   [irq35: virtio_pci2]
100063                   I                                   [irq1: atkbd0]
100064                   I                                   [irq12: psm0]
100065                   I                                   [swi0: uart uart++]
100069                   I                                   [swi1: pf send]
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
37364   778   778     0  R       (threaded)                  syz-executor.1
140200                   RunQ                                syz-executor.1
140217                   RunQ                                syz-executor.1
37336     1   777     0  S       uwait   0xfffffe005730ba00  syz-executor.0
37335     1   777     0  S       umtxn   0xfffffe00a70aec00  syz-executor.0
37326     1   777     0  S       uwait   0xfffffe005730ef00  syz-executor.0
37325     1   777     0  S       uwait   0xfffffe005730e900  syz-executor.0
37316     1   792     0  S       uwait   0xfffffe00a0d23580  syz-executor.3
37307     1   792     0  S       uwait   0xfffffe005730ed00  syz-executor.3
37297     1   792     0  S       uwait   0xfffffe00a70af180  syz-executor.3
37296     1   792     0  S       uwait   0xfffffe00a70af280  syz-executor.3
27538     1   778     0  S       uwait   0xfffffe0058de4100  syz-executor.1
27011     1   792     0  S       umtxn   0xfffffe0057877880  syz-executor.3
26871     1   792     0  S       uwait   0xfffffe00a0d23d80  syz-executor.3
26798     1   778     0  SV      uwait   0xfffffe0057878980  syz-executor.1
11645     1   792     0  T       uwait   0xfffffe00a0d24800  syz-executor.3
 7661     1  7661    65  Ss      select  0xfffffe00a70af940  dhclient
 6860     1   785     0  S       uwait   0xfffffe0057878a80  syz-executor.2
 6857     1   785     0  S       uwait   0xfffffe00a0d25080  syz-executor.2
 6853     1   785     0  S       uwait   0xfffffe0057877280  syz-executor.2
 6850     1   785     0  S       uwait   0xfffffe0057879b00  syz-executor.2
 6846     1   785     0  S       uwait   0xfffffe005787aa80  syz-executor.2
 6843     1   785     0  S       uwait   0xfffffe00a0d23c80  syz-executor.2
 6838     1   785     0  S       uwait   0xfffffe00a0d23a80  syz-executor.2
 6836     1   785     0  S       uwait   0xfffffe0057879300  syz-executor.2
 6831     1   785     0  S       uwait   0xfffffe0057877580  syz-executor.2
 6828     1   785     0  S       uwait   0xfffffe0057877980  syz-executor.2
 6109     0     0     0  DL      (threaded)                  [zfskern]
100123                   D       t->zthr 0xfffffe00a5ea7348  [arc_evict]
105555                   D       t->zthr 0xfffffe00a5ea7448  [arc_reap]
105556                   D       -       0xfffffe009eaa0500  [dbu_evict]
105557                   D       dbuf_ev 0xffffffff85837c60  [dbuf_evict_thread]
105558                   D       -       0xfffffe009eaa1200  [z_vdev_file_0]
105559                   D       -       0xfffffe009eaa1200  [z_vdev_file_1]
105560                   D       -       0xfffffe009eaa1200  [z_vdev_file_2]
105561                   D       -       0xfffffe009eaa1200  [z_vdev_file_3]
105562                   D       -       0xfffffe009eaa1200  [z_vdev_file_4]
105563                   D       -       0xfffffe009eaa1200  [z_vdev_file_5]
105564                   D       -       0xfffffe009eaa1200  [z_vdev_file_6]
105565                   D       -       0xfffffe009eaa1200  [z_vdev_file_7]
105566                   D       -       0xfffffe009eaa1200  [z_vdev_file_8]
105567                   D       -       0xfffffe009eaa1200  [z_vdev_file_9]
105568                   D       -       0xfffffe009eaa1200  [z_vdev_file_10]
105569                   D       -       0xfffffe009eaa1200  [z_vdev_file_11]
105570                   D       -       0xfffffe009eaa1200  [z_vdev_file_12]
105571                   D       -       0xfffffe009eaa1200  [z_vdev_file_13]
105572                   D       -       0xfffffe009eaa1200  [z_vdev_file_14]
105573                   D       -       0xfffffe009eaa1200  [z_vdev_file_15]
105574                   D       l2arc_f 0xffffffff857f6d80  [l2arc_feed_thread]
105575                   D       -       0xfffffe009eaa0400  [zfsvfs]
105576                   S       zevent_ 0xffffffff85838800  [sysevent]
 5930     0     0     0  DL      aiordy  0xfffffe0056f9a548  [aiod4]
 5929     0     0     0  DL      aiordy  0xfffffe00a0c00a90  [aiod3]
 5928     0     0     0  DL      aiordy  0xfffffe00a0cfd000  [aiod2]
 5925     0     0     0  DL      aiordy  0xfffffe00a0cfd548  [aiod1]
 5867     1  5867     0  Ss      select  0xfffffe0057877d40  dhclient
 5864     1  5864     0  Ss      select  0xfffffe0057877cc0  dhclient
 5836     1  5836    65  Ss      select  0xfffffe0058de2340  dhclient
 4837     1  4837     0  Ss      select  0xfffffe00a0d241c0  dhclient
 4834     1  4834     0  Ss      select  0xfffffe00a0d24140  dhclient
 4815     1  4815    65  Ss      select  0xfffffe0057878040  dhclient
 3381     1  3381     0  Ss      select  0xfffffe0057878240  dhclient
 3378     1  3378     0  Ss      select  0xfffffe0057877ec0  dhclient
 3351     1  3351    65  Ss      select  0xfffffe00a0d24340  dhclient
 1465     1  1465     0  Ss      select  0xfffffe00578781c0  dhclient
 1462     1  1462     0  Ss      select  0xfffffe0058de21c0  dhclient
  792   774   792     0  Rs                                  syz-executor.3
  785   774   785     0  Rs                                  syz-executor.2
  778   774   778     0  Ss      nanslp  0xffffffff83e43cc1  syz-executor.1
  777   774   777     0  Rs                                  syz-executor.0
  774   772   772     0  R       (threaded)                  syz-fuzzer
100091                   S       kqread  0xfffffe0053c93700  syz-fuzzer
100113                   S       nanslp  0xffffffff83e43cc1  syz-fuzzer
100114                   S       uwait   0xfffffe0057879700  syz-fuzzer
100115                   S       uwait   0xfffffe0057879800  syz-fuzzer
100116                   S       uwait   0xfffffe0057879900  syz-fuzzer
100117                   S       uwait   0xfffffe0053ee2700  syz-fuzzer
100118                   S       uwait   0xfffffe0053ee2800  syz-fuzzer
100119                   Run     CPU 1                       syz-fuzzer
100138                   S       uwait   0xfffffe0058de3880  syz-fuzzer
  772   770   772     0  Ss      pause   0xfffffe0056f99b40  csh
  770   688   770     0  Ss      select  0xfffffe0058de41c0  sshd
  754     1   754     0  Ss+     ttyin   0xfffffe0057463cb0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe00579b50b0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe00579b54b0  getty
  751     1   751     0  Ss+     ttyin   0xfffffe00574628b0  getty
  750     1   750     0  Ss+     ttyin   0xfffffe00579b58b0  getty
  749     1   749     0  Ss+     ttyin   0xfffffe0057462cb0  getty
  748     1   748     0  Ss+     ttyin   0xfffffe00579b5cb0  getty
  747     1   747     0  Ss+     ttyin   0xfffffe00579b60b0  getty
  746     1   746     0  Ss+     ttyin   0xfffffe00579b64b0  getty
  692     1   692     0  Ss      nanslp  0xffffffff83e43cc0  cron
  688     1   688     0  Ss      select  0xfffffe0058de44c0  sshd
  501     1   501     0  Ss      select  0xfffffe0058de4840  syslogd
  430     1   430     0  Ss      select  0xfffffe0058de46c0  devd
  429     1   429    65  Ss      select  0xfffffe0058de4540  dhclient
  344     1   344     0  Ss      select  0xfffffe0058de4640  dhclient
  341     1   341     0  Ss      select  0xfffffe0058de48c0  dhclient
   17     0     0     0  DL      vlruwt  0xfffffe0056f9b548  [vnlru]
   16     0     0     0  DL      syncer  0xffffffff83f694e0  [syncer]
   15     0     0     0  DL      (threaded)                  [bufdaemon]
100079                   D       psleep  0xffffffff83f67b00  [bufdaemon]
100082                   D       -       0xffffffff83211f80  [bufspacedaemon-0]
100093                   D       sdflush 0xfffffe00579488e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83f9b5c0  [vmdaemon]
    8     0     0     0  DL      (threaded)                  [pagedaemon]
100077                   D       psleep  0xffffffff83f8f478  [dom0]
100083                   D       launds  0xffffffff83f8f484  [laundry: dom0]
100084                   D       umarcl  0xffffffff81e4cca0  [uma]
    7     0     0     0  DL      -       0xffffffff83c002a8  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff84a08550  [pf purge]
    5     0     0     0  DL      waiting 0xffffffff8482d420  [sctp_iterator]
    4     0     0     0  DL      (threaded)                  [cam]
100044                   D       -       0xffffffff83aa3240  [doneq0]
100045                   D       -       0xffffffff83aa31c0  [async]
100076                   D       -       0xffffffff83aa3040  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0056f00c88  [sequencer 00]
    3     0     0     0  DL      (threaded)                  [crypto]
100040                   D       crypto_ 0xffffffff83f8aca0  [crypto]
100041                   D       crypto_ 0xfffffe0053ed8d30  [crypto returns 0]
100042                   D       crypto_ 0xfffffe0053ed8d80  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100035                   D       -       0xffffffff83e192c0  [g_event]
100036                   D       -       0xffffffff83e192e0  [g_up]
100037                   D       -       0xffffffff83e19300  [g_down]
    2     0     0     0  WL      (threaded)                  [clock]
100029                   I                                   [clock (0)]
100030                   I                                   [clock (1)]
   12     0     0     0  RL      (threaded)                  [intr]
100015                   I                                   [swi5: fast taskq]
100018                   I                                   [swi6: task queue]
100019                   I                                   [swi6: Giant taskq]
100031                   Run     CPU 0                       [swi1: netisr 0]
100032                   I                                   [swi1: hpts]
100033                   I                                   [swi1: hpts]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq33: virtio_pci2]
100060                   I                                   [irq34: virtio_pci2]
100061                   I                                   [irq35: virtio_pci2]
100063                   I                                   [irq1: atkbd0]
100064                   I                                   [irq12: psm0]
100065                   I                                   [swi0: uart uart++]
100069                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053de3000  [init]
   10     0     0     0  DL      audit_w 0xffffffff83f8b780  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff83e19cc0  [swapper]
100005                   D       -       0xfffffe0053e94100  [if_config_tqg_0]
100006                   D       -       0xfffffe0053e94000  [softirq_0]
100007                   D       -       0xfffffe0053e93e00  [softirq_1]
100008                   D       -       0xfffffe0053e93d00  [if_io_tqg_0]
100009                   D       -       0xfffffe0053e93c00  [if_io_tqg_1]
100010                   D       -       0xfffffe0007972600  [inm_free taskq]
100011                   D       -       0xfffffe0007972400  [linuxkpi_irq_wq]
100012                   D       -       0xfffffe0007972200  [in6m_free taskq]
100013                   D       -       0xfffffe0007972000  [deferred_unmount ta]
100014                   D       -       0xfffffe0007971d00  [thread taskq]
100016                   D       -       0xfffffe0007971900  [kqueue_ctx taskq]
100017                   D       -       0xfffffe0007971700  [pci_hp taskq]
100020                   D       -       0xfffffe0007971100  [aiod_kick taskq]
100021                   D       -       0xfffffe0007970e00  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe0007970e00  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe0007970e00  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe0007970e00  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe0007970900  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe0007970900  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe0007970900  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe0007970900  [linuxkpi_long_wq_3]
100034                   D       -       0xfffffe0053f06300  [firmware taskq]
100038                   D       -       0xfffffe0053f05d00  [crypto_0]
100039                   D       -       0xfffffe0053f05d00  [crypto_1]
100055                   D       -       0xfffffe0053f04e00  [vtnet0 rxq 0]
100056                   D       -       0xfffffe0053f04d00  [vtnet0 txq 0]
100057                   D       -       0xfffffe0053f04c00  [vtnet0 rxq 1]
100058                   D       -       0xfffffe0053f04b00  [vtnet0 txq 1]
100062                   D       vtbslp  0xfffffe005730da00  [virtio_balloon]
100066                   D       -       0xffffffff82b60801  [deadlkres]
100070                   D       -       0xfffffe0007973200  [mca taskq]
100071                   D       -       0xfffffe005787e000  [acpi_task_0]
100072                   D       -       0xfffffe005787e000  [acpi_task_1]
100073                   D       -       0xfffffe005787e000  [acpi_task_2]
100075                   D       -       0xfffffe0053f05600  [CAM taskq]
105550                   D       -       0xfffffe005787b600  [system_taskq_0]
105551                   D       -       0xfffffe005787b600  [system_taskq_1]
105552                   D       -       0xfffffe005787b500  [system_delay_taskq_]
105553                   D       -       0xfffffe005787b500  [system_delay_taskq_]
105554                   D       -       0xfffffe009eaa1b00  [arc_prune]
37358   777   777     0  Z                                   syz-executor.0
37362   792   792     0  Z                                   syz-executor.3
db> show all locks
Process 774 (syz-fuzzer) thread 0xfffffe009ef95ac0 (100119)
exclusive sleep mutex pipe mutex (pipe mutex) r = 0 (0xfffffe0058bbe2c0) locked @ /syzkaller/managers/i386/kernel/sys/kern/sys_pipe.c:1095
Process 12 (intr) thread 0xfffffe0053ea8560 (100031)
exclusive sleep mutex sctp-tcb (tcb) r = 0 (0xfffffe00a0da7a50) locked @ /syzkaller/managers/i386/kernel/sys/netinet/sctp_pcb.c:2138
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            5        11524K            5
            linker          380         9604K          416
          tcp_hpts            7         4801K            7
            devbuf         4217         4323K         4246
           solaris          130         2804K          200
         sysctloid        47619         2801K        47723
             vtbuf           24         1968K           46
              kobj          328         1312K          490
            newblk          121         1054K        37594
          vfscache            3         1025K            3
               pcb          275          820K       105557
          inodedep          103          551K        36897
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
          filedesc           58          457K        72729
           subproc          181          373K        37465
              vmem            3          274K            5
            acpica         1674          184K        57552
         vnet_data            1          168K            1
           tidhash            3          141K            3
           pagedep           14          132K        36396
        tfo_ccache            1          128K            1
         sctp_stro          120          120K        20176
            DEVFS1          110          110K          127
               sem            4          106K            4
         sctp_atcl          247           93K        74964
               BPF           46           88K          303
               bus          995           81K         5208
          mtx_pool            2           72K            2
         sctp_timw          275           69K          275
          syncache            1           68K            1
            module          518           65K          518
          acpitask            1           64K            1
       ddb_capture            1           64K            1
              umtx          484           61K          484
           kdtrace          285           56K        77583
              temp           34           33K        11009
            DEVFS3          129           33K          139
         hostcache            1           32K            1
               shm            1           32K           36
               msg            4           30K            4
        gtaskqueue           18           26K           18
            dirrem           88           22K        36796
            kbdmux            6           22K            6
            ifaddr           71           21K           73
        DEVFS_RULE           56           20K           56
        kstat_data           17           17K           17
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
          routetbl          138           16K          430
           ithread           97           16K           97
           pf_osfp          128           16K          128
         sctp_atky          367           16K        99463
           lltable           46           15K          165
            bus-sc           34           15K         1681
            KTRACE          101           13K          123
       ether_multi          157           13K          182
             ifnet            7           13K            7
      eventhandler          139           12K          139
              kenv           95           12K           95
          freefile           87           11K        36768
              rman           88           11K          431
              GEOM           61           11K          494
         CAM queue            5           11K         1528
         in6_multi           71            9K           71
         taskqueue           81            9K           81
         bmsafemap            2            9K        36923
              UART           12            9K           12
           devstat            4            9K            4
              ksem            1            8K          261
               rpc            2            8K            2
             shmfd            1            8K           45
       pfs_vncache            1            8K            1
            kqueue           86            8K        37511
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          296
               UMA          367            7K          367
              cred           26            7K          281
            sglist            5            7K            5
           CAM DEV            3            6K          510
            plimit           24            6K          475
           pwddesc           82            6K        37531
          pf_ifnet           13            5K           55
       ufs_dirhash           24            5K           24
            DEVFSP           72            5K         2789
           session           35            5K           53
                vt           11            5K           11
           memdesc            1            4K            1
               MCA           32            4K           32
             evdev            4            4K            4
         sctp_athm          247            4K        77603
          sctp_map          240            4K        39872
         proc-args          112            4K        38758
             lockf           34            4K          312
           acpisem           28            4K           28
             selfd           55            4K       493932
             hhook           15            4K           17
          kcovinfo           52            4K           52
          terminal           11            3K           11
            select           19            3K          144
           uidinfo            3            3K           14
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         ipsec-saq            2            2K            2
            ip6ndp           12            2K           15
          sctp_ifa           14            2K           15
            Unitno           29            2K           53
           CAM XPT           22            2K          543
               msi           12            2K           12
          in_multi            6            2K           11
       vnodemarker            3            2K           87
          indirdep            5            2K          347
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
         newdirblk            9            2K        36376
             clone            9            2K            9
               tun            7            2K            7
          freework            5            2K        36667
           softdep            1            1K            1
             mkdir            8            1K        72752
          freeblks            4            1K        36466
            sahead            1            1K            1
          secasvar            1            1K            1
             nhops            6            1K            8
               osd           42            1K         2633
      NFSD session            1            1K            1
            CC Mem           14            1K         2616
        CAM periph            4            1K          271
       inpcbpolicy           26            1K        16578
          sctp_ifn            6            1K           15
             ipsec            3            1K            3
               mld            6            1K            6
              igmp            6            1K            6
         toponodes            6            1K            6
            isadev            6            1K            6
             mount           16            1K           89
          pci_link           10            1K           10
            crypto            4            1K            4
 encap_export_host           12            1K           12
          procdesc            5            1K           18
         sctp_stri            1            1K         5201
            diradd            4            1K        36836
              pfil            4            1K            4
              cdev            2            1K            2
         cryptodev            6            1K         2426
    chacha20random            1            1K            1
           tcp_fsb            2            1K          988
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           10
            vnodes            1            1K            1
           CAM SIM            2            1K            2
            prison            8            1K            8
            feeder            7            1K            7
             taskq            2            1K            2
           tcpfunc            3            1K            3
        loginclass            3            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          pmchooks            1            1K            1
            soname            4            1K        46255
          filecaps            5            1K          116
          sctp_vrf            1            1K            1
              vnet            1            1K            1
           entropy            2            1K           47
               pmc            1            1K            1
          acpiintr            1            1K            1
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
         sfs_nodes            0            0K            0
        zones_data            0            0K            0
            mqdata            0            0K            0
          pf_table            0            0K            0
           pf_rule            0            0K           17
           pf_altq            0            0K            0
           pf_temp            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K        31424
         sctp_iter            0            0K          258
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K          246
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K           12
         sctp_aadr            0            0K            0
           filemon            0            0K          326
            tcp_do            0            0K            0
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
               ixl            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
          xen_intr            0            0K            0
      NFSD V4state            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            DEVFS4            0            0K            0
     vm_fictitious            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
          xen_acpi            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           UMAHash            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K        21480
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           40
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K           55
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            vtfont            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
          ktls_ocf            0            0K            0
       AHCI driver            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K           40
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               LRO            0            0K            0
        ip_msource            0            0K            1
       ip_moptions            0            0K            3
        in_mfilter            0            0K            4
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
               agp            0            0K            0
            statfs            0            0K        36586
     namei_tracker            0            0K          267
       export_host            0            0K            0
        cl_savebuf            0            0K            2
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
       tcp_log_dev            0            0K           48
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
            biobuf            0            0K            0
              aios            0            0K            7
               lio            0            0K           86
               acl            0            0K            0
          mbuf_tag            0            0K          169
              ktls            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        31132
          ioctlops            0            0K          829
           eventfd            0            0K           83
           Witness            0            0K            0
             stack            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
              sbuf            0            0K          288
          mpr_user            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
            MPRSAS            0            0K            0
              SWAP            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
         sysctltmp            0            0K          929
            sysctl            0            0K            3
        md_sectors            0            0K            0
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
           md_disk            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
          ix_sriov            0            0K            0
             cache            0            0K            0
        aacraidcam            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K           17
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
       aacraid_buf            0            0K            0
            aaccam            0            0K            0
         boottrace            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8339    1059     1554878       0     254   38494208        0
      malloc-32768  32768       0     101         162       0       1    3309568        0
              mbuf    256    9018    2160     2799294       0     254    2861568        0
              pbuf   2624       0     989           0       0       2    2595136        0
        RADIX NODE    144   16754     405      926839       0      64    2470896        0
           tcp_log    416     581    4756       43297       0     254    2220192        0
        malloc-128    128   15597     306       89758       0     126    2035584        0
         sctp_asoc   2256     120     645       19936       0     254    1725840        0
          BUF TRIE    144     238   11550        1815       0      62    1697472        0
       malloc-4096   4096     412       2       37939       0       2    1695744        0
        malloc-384    384    4117      53        4117       0      30    1601280        0
      mbuf_cluster   2048     762       0         762       0     254    1560576        0
       UMA Slabs 0    112   11356      14       11356       0     126    1273440        0
           sctp_ep   1208     126     894       52427       0     254    1232160        0
         vmem btag     56   20407      80       20407       0     254    1147272        0
        malloc-384    384     162    2598       37099       0      30    1059840        0
        sctp_raddr    736     138    1138       21798       0     254     939136        0
       malloc-2048   2048     127     273       52430       0       8     819200        0
            socket    960      57     707       71865       0     254     733440        0
        malloc-256    256     296    2494       74739       0      62     714240        0
         FFS inode   1160     543      45       37321       0       8     682080        0
         VM OBJECT    264    2443      77      543075       0      30     665280        0
      malloc-16384  16384      32       6       36541       0       1     622592        0
        malloc-256    256      64    2231       59491       0      62     587520        0
       malloc-8192   8192       4      67         330       0       1     581632        0
             tcpcb   1104      14     497        2593       0     254     564144        0
        256 Bucket   2048     253      21        6753       0       8     561152        0
        malloc-128    128    1382    2679      106027       0     126     519808        0
            THREAD   1808     200      42       40217       0       8     437536        0
       malloc-1024   1024     128     256       20496       0      16     393216        0
            lkpimm    168       1    2327           1       0      62     391104        0
          lkpicurr    168       2    2326           2       0      62     391104        0
         malloc-64     64    5482     629       80954       0     254     391104        0
         MAP ENTRY     96    3241     287     1159787       0     126     338688        0
         malloc-16     16   19681     569       81114       0     254     324000        0
             VNODE    448     584     109       37364       0      30     310464        0
        malloc-384    384     247     503       74970       0      30     288000        0
      malloc-65536  65536       4       0           4       0       1     262144        0
         malloc-32     32    7532     658       26854       0     254     262080        0
         UMA Zones    768     339       0         339       0      16     260352        0
            DEVCTL   1024       0     220         156       0       0     225280        0
     FPU_save_area    832     202      59       49640       0      16     217152        0
       tcp_bbr_map    128       0    1643       19932       0     126     210304        0
        malloc-256    256     425     385       53513       0      62     207360        0
    ertt_txseginfo     40       1    5150      155029       0     254     206040        0
      malloc-32768  32768       1       5         321       0       1     196608        0
       mbuf_packet    256     122     640       74936       0     254     195072        0
        sctp_chunk    152     155    1119        6574       0     254     193648        0
        malloc-128    128    1277     118        3838       0     126     178560        0
           VMSPACE   2552      60       9       37348       0       4     176088        0
      malloc-32768  32768       0       5         175       0       1     163840        0
       FFS2 dinode    256     543      87       37321       0      62     161280        0
       S VFS Cache    104    1025     496       38274       0     126     158184        0
        128 Bucket   1024     101      30        4080       0      16     134144        0
zio_buf_comb_131072 131072       0       1           1       0       1     131072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
      malloc-65536  65536       0       2         488       0       1     131072        0
      malloc-65536  65536       0       2          12       0       1     131072        0
      malloc-65536  65536       2       0           2       0       1     131072        0
       malloc-1024   1024     115      13         132       0      16     131072        0
             unpcb    256      20     490        2800       0     254     130560        0
              PROC   1352      83      13       37364       0       8     129792        0
          UMA Kegs    384     325       8         325       0      30     127872        0
         64 Bucket    512     154      78       22535       0      30     118784        0
          ksiginfo    112      87     957        9728       0     126     116928        0
        malloc-256    256     252     183       46064       0      62     111360        0
       malloc-4096   4096      23       3         571       0       2     106496        0
         filedesc0   1072      82      16       37531       0       8     105056        0
              pipe    744      23     107         989       0      16      96720        0
             ripcb    424       6     210       13147       0      30      91584        0
         malloc-64     64     250    1073      535063       0     254      84672        0
       malloc-4096   4096      14       6       36614       0       2      81920        0
         32 Bucket    256     189     126        9432       0      62      80640        0
        sctp_readq    152       0     520        1375       0     254      79040        0
       malloc-8192   8192       7       2          22       0       1      73728        0
             g_bio    408       0     180       18040       0      30      73440        0
         malloc-64     64     608     463       18946       0     254      68544        0
        malloc-128    128     299     228         534       0     126      67456        0
      malloc-65536  65536       0       1           8       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
       malloc-8192   8192       5       3         134       0       1      65536        0
       malloc-8192   8192       4       4          19       0       1      65536        0
       malloc-2048   2048       8      24         714       0       8      65536        0
        malloc-256    256      37     218       37188       0      62      65280        0
        malloc-256    256      67     188       36696       0      62      65280        0
     udplite_inpcb    424       0     153         168       0      30      64872        0
         udp_inpcb    424       6     147         670       0      30      64872        0
             Files     80     278     522      127730       0     126      64000        0
sctp_stream_msg_out    112      42     498        2685       0     254      60480        0
        malloc-384    384     117      33         157       0      30      57600        0
         tcp_inpcb    424      14     112        2593       0      30      53424        0
        sctp_laddr     48       1    1091       22904       0     254      52416        0
         malloc-64     64     130     689       28174       0     254      52416        0
        malloc-128    128     123     280         217       0     126      51584        0
        malloc-128    128      10     393        1018       0     126      51584        0
        malloc-256    256     159      36         214       0      62      49920        0
        malloc-256    256      71     124       39270       0      62      49920        0
zio_data_buf_16384  16384       0       3           9       0       1      49152        0
           DIRHASH   1024      35      13          35       0      16      49152        0
             NAMEI   1024       0      48      169231       0      16      49152        0
      malloc-16384  16384       3       0           3       0       1      49152        0
       malloc-4096   4096      11       1          11       0       2      49152        0
       malloc-2048   2048      10      14         568       0       8      49152        0
       malloc-1024   1024      23      25         221       0      16      49152        0
      tcp_rack_pcb    896       2      52         988       0      16      48384        0
            pcpu-8      8    5263     369        7840       0     254      45056        0
          syncache    168       0     264           4       0     254      44352        0
         malloc-32     32     271    1115       74271       0     254      44352        0
         TURNSTILE    136     243      72         243       0      62      42840        0
            clpbuf   2624       0      16          38       0      16      41984        0
       malloc-8192   8192       3       2          71       0       1      40960        0
       Mountpoints   2752       2      12           2       0       4      38528        0
            da_ccb    544       0      70        4659       0      16      38080        0
               PWD     32      39    1095       36411       0     254      36288        0
         malloc-64     64      29     538         129       0     254      36288        0
         malloc-64     64     148     419        2868       0     254      36288        0
         malloc-64     64     139     428         203       0     254      36288        0
         malloc-64     64     487      80         709       0     254      36288        0
        malloc-128    128      75     204         301       0     126      35712        0
        malloc-128    128      96     183       36597       0     126      35712        0
     routing nhops    256      27     108          35       0      62      34560        0
           ttyoutq    256      72      63         160       0      62      34560        0
        malloc-384    384      53      37         540       0      30      34560        0
        malloc-384    384      31      59          37       0      30      34560        0
        SLEEPQUEUE     88     243     141         243       0     126      33792        0
     zio_buf_16384  16384       0       2          10       0       1      32768        0
      malloc-16384  16384       2       0           2       0       1      32768        0
       malloc-4096   4096       3       5          97       0       2      32768        0
       malloc-2048   2048       1      15         269       0       8      32768        0
       malloc-2048   2048       8       8           8       0       8      32768        0
       malloc-2048   2048       2      14          43       0       8      32768        0
       malloc-2048   2048       5      11         275       0       8      32768        0
       malloc-1024   1024       0      32          12       0      16      32768        0
       malloc-1024   1024      10      22        1506       0      16      32768        0
       malloc-1024   1024      18      14          22       0      16      32768        0
        malloc-512    512       9      55          25       0      30      32768        0
        malloc-512    512       2      62         269       0      30      32768        0
        malloc-512    512       3      61          88       0      30      32768        0
        malloc-512    512      12      52         263       0      30      32768        0
        malloc-512    512       1      63        5202       0      30      32768        0
           pcpu-64     64     498      14         498       0     254      32768        0
             KNOTE    160      28     172      284647       0      62      32000        0
            ttyinq    160     135      65         300       0      62      32000        0
      tcp_rack_map    120       6     258        2516       0     126      31680        0
       tcp_bbr_pcb    832       0      36         333       0      16      29952        0
            cpuset    104       7     272          28       0     126      29016        0
   tcp_inpcb ports     32       6     876        1703       0     254      28224        0
         malloc-32     32     224     658       39551       0     254      28224        0
         malloc-32     32     295     587       30531       0     254      28224        0
         16 Bucket    144      82     114         635       0      62      28224        0
          4 Bucket     48       6     582         822       0     254      28224        0
       malloc-8192   8192       3       0           3       0       1      24576        0
       malloc-8192   8192       2       1           4       0       1      24576        0
           rtentry    176      31     107          35       0      62      24288        0
              PGRP     88      35     241          53       0     126      24288        0
          rl_entry     40     101     505         171       0     254      24240        0
         malloc-16     16     603     897       41373       0     254      24000        0
         malloc-16     16     258    1242       78037       0     254      24000        0
          8 Bucket     80      80     220        7165       0     126      24000        0
         domainset     40       0     567          18       0     254      22680        0
         hostcache     64       1     314           1       0     254      20160        0
             udpcb     32       6     624         838       0     254      20160        0
udplite_inpcb ports     32       0     630           6       0     254      20160        0
   udp_inpcb ports     32       3     627          37       0     254      20160        0
              AIOP     32       4     626           4       0     254      20160        0
              ertt     72      14     266        2593       0     126      20160        0
         malloc-32     32      51     579          78       0     254      20160        0
         malloc-32     32       0     630         268       0     254      20160        0
         malloc-32     32     131     499         372       0     254      20160        0
         malloc-32     32      40     590         738       0     254      20160        0
          2 Bucket     32     150     480        4168       0     254      20160        0
             AIOCB    552       0      35          32       0      16      19320        0
            AIOLIO    272       0      70          43       0      30      19040        0
 epoch_record pcpu    256       4      60           4       0      62      16384        0
       malloc-4096   4096       2       2           7       0       2      16384        0
       malloc-4096   4096       0       4           5       0       2      16384        0
       malloc-2048   2048       1       7           1       0       8      16384        0
       malloc-1024   1024       8       8           9       0      16      16384        0
       malloc-1024   1024       9       7          77       0      16      16384        0
        malloc-512    512       2      30           2       0      30      16384        0
        malloc-512    512       2      30           3       0      30      16384        0
           SMR CPU     32       7     504           7       0     254      16352        0
               ipq     56       0     288           1       0     254      16128        0
          tcpreass     48       0     336           1       0     254      16128        0
      vtnet_tx_hdr     24       0     668      809259       0     254      16032        0
               AIO    208       0      76          28       0      62      15808        0
              kenv    258      15      45        1055       0      30      15480        0
            mqnode    416       3      33           3       0      30      14976        0
              vmem   1856       1       7           1       0       8      14848        0
        SMR SHARED     24       7     504           7       0     254      12264        0
         malloc-16     16      56     694        2678       0     254      12000        0
         malloc-16     16      18     732          99       0     254      12000        0
         malloc-16     16     185     565        3103       0     254      12000        0
         malloc-16     16      37     713       53748       0     254      12000        0
arc_buf_hdr_t_full    352       4      29           4       0      30      11616        0
            itimer    352       0      33           3       0      30      11616        0
        malloc-384    384       2      28           3       0      30      11520        0
           pcpu-16     16      14     498          14       0     254       8192        0
         malloc-16     16       1     499           1       0     254       8000        0
       UMA Slabs 1    176      11      11          11       0      62       3872        0
        KMAP ENTRY     96      16      23          23       0       0       3744        0
            pcpu-4      4       1     511          18       0     254       2048        0
   zfs_znode_cache    472       0       0           0       0      30          0        0
       sio_cache_2    168       0       0           0       0      62          0        0
       sio_cache_1    152       0       0           0       0      62          0        0
       sio_cache_0    136       0       0           0       0      62          0        0
     zil_zcw_cache     80       0       0           0       0     126          0        0
     zil_lwb_cache    352       0       0           0       0      30          0        0
    dmu_buf_impl_t    408       0       0           0       0      30          0        0
         arc_buf_t     64       0       0           0       0     254          0        0
arc_buf_hdr_t_l2only     96       0       0           0       0     126          0        0
arc_buf_hdr_t_full_crypt    416       0       0           0       0      30          0        0
           dnode_t   1032       0       0           0       0       8          0        0
          sa_cache    296       0       0           0       0      30          0        0
         abd_chunk   4096       0       0           0       0       2          0        0
zio_buf_comb_16777216 16777216       0       0           0       0       1          0        0
zio_buf_comb_14680064 14680064       0       0           0       0       1          0        0
zio_buf_comb_12582912 12582912       0       0           0       0       1          0        0
zio_buf_comb_10485760 10485760       0       0           0       0       1          0        0
zio_buf_comb_8388608 8388608       0       0           0       0       1          0        0
zio_buf_comb_7340032 7340032       0       0           0       0       1          0        0
zio_buf_comb_6291456 6291456       0       0           0       0       1          0        0
zio_buf_comb_5242880 5242880       0       0           0       0       1          0        0
zio_buf_comb_4194304 4194304       0       0           0       0       1          0        0
zio_buf_comb_3670016 3670016       0       0           0       0       1          0        0
zio_buf_comb_3145728 3145728       0       0           0       0       1          0        0
zio_buf_comb_2621440 2621440       0       0           0       0       1          0        0
zio_buf_comb_2097152 2097152       0       0           0       0       1          0        0
zio_buf_comb_1835008 1835008       0       0           0       0       1          0        0
zio_buf_comb_1572864 1572864       0       0           0       0       1          0        0
zio_buf_comb_1310720 1310720       0       0           0       0       1          0        0
zio_buf_comb_1048576 1048576       0       0           0       0       1          0        0
zio_buf_comb_917504 917504       0       0           0       0       1          0        0
zio_buf_comb_786432 786432       0       0           0       0       1          0        0
zio_buf_comb_655360 655360       0       0           0       0       1          0        0
zio_buf_comb_524288 524288       0       0           0       0       1          0        0
zio_buf_comb_458752 458752       0       0           0       0       1          0        0
zio_buf_comb_393216 393216       0       0           0       0       1          0        0
zio_buf_comb_327680 327680       0       0           0       0       1          0        0
zio_buf_comb_262144 262144       0       0           0       0       1          0        0
zio_buf_comb_229376 229376       0       0           0       0       1          0        0
zio_buf_comb_196608 196608       0       0           0       0       1          0        0
zio_buf_comb_163840 163840       0       0           0       0       1          0        0
zio_buf_comb_114688 114688       0       0           0       0       1          0        0
zio_buf_comb_98304  98304       0       0           0       0       1          0        0
zio_buf_comb_81920  81920       0       0           0       0       1          0        0
zio_buf_comb_65536  65536       0       0           0       0       1          0        0
zio_buf_comb_57344  57344       0       0           0       0       1          0        0
zio_buf_comb_49152  49152       0       0           0       0       1          0        0
zio_buf_comb_40960  40960       0       0           0       0       1          0        0
zio_buf_comb_32768  32768       0       0           0       0       1          0        0
zio_buf_comb_28672  28672       0       0           0       0       1          0        0
zio_buf_comb_24576  24576       0       0           0       0       1          0        0
zio_buf_comb_20480  20480       0       0           0       0       1          0        0
zio_data_buf_14336  14336       0       0           0       0       1          0        0
     zio_buf_14336  14336       0       0           0       0       1          0        0
zio_data_buf_12288  12288       0       0           0       0       1          0        0
     zio_buf_12288  12288       0       0           0       0       1          0        0
zio_data_buf_10240  10240       0       0           0       0       1          0        0
     zio_buf_10240  10240       0       0           0       0       1          0        0
 zio_data_buf_8192   8192       0       0           0       0       1          0        0
      zio_buf_8192   8192       0       0           0       0       1          0        0
 zio_data_buf_7168   7168       0       0           0       0       1          0        0
      zio_buf_7168   7168       0       0           0       0       1          0        0
 zio_data_buf_6144   6144       0       0           0       0       1          0        0
      zio_buf_6144   6144       0       0           0       0       1          0        0
 zio_data_buf_5120   5120       0       0           0       0       1          0        0
      zio_buf_5120   5120       0       0           0       0       1          0        0
 zio_data_buf_4096   4096       0       0           0       0       2          0        0
      zio_buf_4096   4096       0       0           0       0       2          0        0
 zio_data_buf_3584   3584       0       0           0       0       2          0        0
      zio_buf_3584   3584       0       0           0       0       2          0        0
 zio_data_buf_3072   3072       0       0           0       0       4          0        0
      zio_buf_3072   3072       0       0           0       0       4          0        0
 zio_data_buf_2560   2560       0       0           0       0       4          0        0
      zio_buf_2560   2560       0       0           0       0       4          0        0
 zio_data_buf_2048   2048       0       0           0       0       8          0        0
      zio_buf_2048   2048       0       0           0       0       8          0        0
 zio_data_buf_1536   1536       0       0           0       0       8          0        0
      zio_buf_1536   1536       0       0           0       0       8          0        0
 zio_data_buf_1024   1024       0       0           0       0      16          0        0
      zio_buf_1024   1024       0       0           0       0      16          0        0
  zio_data_buf_512    512       0       0           0       0      30          0        0
       zio_buf_512    512       0       0           0       0      30          0        0
    zio_link_cache     48       0       0           0       0     254          0        0
         zio_cache   1232       0       0           0       0       8          0        0
   ddt_entry_cache    392       0       0           0       0      30          0        0
         ddt_cache  24824       0       0           0       0       1          0        0
metaslab_alloc_trace_cache     72       0       0           0       0     126          0        0
zfs_btree_leaf_cache   4096       0       0           0       0       2          0        0
reference_history_cache      8       0       0           0       0     254          0        0
   reference_cache     40       0       0           0       0     254          0        0
        taskq_zone    192       0       0           0       0      62          0        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0     254          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0       0          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags    184       0       0           0       0      62          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     72       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   tcp_log_id_node    120       0       0           0       0     126          0        0
 tcp_log_id_bucket    176       0       0           0       0      62          0        0
       ripcb ports     32       0       0           0       0     254          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/06/14 14:55 freebsd-src 975715b78819 9ce2c85a console log report ci-freebsd-i386 Fatal trap NUM: page fault in sctp_notify_stream_reset_add
* Struck through repros no longer work on HEAD.