syzbot


Fatal trap NUM: page fault in __mtx_assert

Status: upstream: reported C repro on 2022/06/03 10:06
Reported-by: syzbot+6e348d00590179004961@syzkaller.appspotmail.com
First crash: 686d, last: 48d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
freebsd Fatal trap 12: page fault in __mtx_assert (5) 7 1026d 1059d 0/2 auto-closed as invalid on 2021/09/26 00:34

Sample crash report:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x1b8
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff81603131
stack pointer	        = 0x28:0xfffffe0053aa8cd0
frame pointer	        = 0x28:0xfffffe0053aa8d10
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 0)

rdi:              1b8 rsi: ffffffff836cc400 rdx:       
FreeBSD/amd64         0
 (ci-freebsd-maircx: fffffe00033eee30  r8: fffff78000000000  r9:      20000000000
n-2.c.syzkaller.rax: fffffe00033eee30 rbx:              1a0 rbp: fffffe0053aa8d10
r10:                0 r11:               bf r12:                1
internal) (ttyu0r13:                4 r14:              1b8 r15: fffffe0053e8b560
)

login: trap number		= 12
panic: page fault
cpuid = 0
time = 1662858102
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe0053aa84b0
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe0053aa8610
vpanic() at vpanic+0x254/frame 0xfffffe0053aa86f0
panic() at panic+0xb5/frame 0xfffffe0053aa87b0
trap_fatal() at trap_fatal+0x7ed/frame 0xfffffe0053aa88d0
trap_pfault() at trap_pfault+0x182/frame 0xfffffe0053aa8a10
trap() at trap+0x5e1/frame 0xfffffe0053aa8c00
calltrap() at calltrap+0x8/frame 0xfffffe0053aa8c00
--- trap 0xc, rip = 0xffffffff81603131, rsp = 0xfffffe0053aa8cd0, rbp = 0xfffffe0053aa8d10 ---
__mtx_assert() at __mtx_assert+0x111/frame 0xfffffe0053aa8d10
sorwakeup_locked() at sorwakeup_locked+0x36/frame 0xfffffe0053aa8d50
sctp_add_to_readq() at sctp_add_to_readq+0x717/frame 0xfffffe0053aa8df0
sctp_queue_data_to_stream() at sctp_queue_data_to_stream+0x77a/frame 0xfffffe0053aa8f90
sctp_process_data() at sctp_process_data+0x2bfc/frame 0xfffffe0053aa9680
sctp_common_input_processing() at sctp_common_input_processing+0x168d/frame 0xfffffe0053aa98a0
sctp6_input_with_port() at sctp6_input_with_port+0x597/frame 0xfffffe0053aa9a70
sctp6_input() at sctp6_input+0x1f/frame 0xfffffe0053aa9a90
ip6_input() at ip6_input+0x1ebf/frame 0xfffffe0053aa9cf0
swi_net() at swi_net+0x2ed/frame 0xfffffe0053aa9d90
ithread_loop() at ithread_loop+0x4ee/frame 0xfffffe0053aa9ef0
fork_exit() at fork_exit+0xd0/frame 0xfffffe0053aa9f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0053aa9f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100031 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x27566aa(%rip)
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xfffffe00033eee30
rdx         0xdffff7c000000000
rbx                          0
rsp         0xfffffe0053aa85f0
rbp         0xfffffe0053aa8610
rsi                        0x1
rdi                          0
r8                         0x3
r9                  0xffffffff
r10                          0
r11                       0xbf
r12                          0
r13         0xfffffe0053e8b560
r14         0xffffffff82b6d3e0  .str.26
r15         0xffffffff82b6d3e0  .str.26
rip         0xffffffff8171caeb  kdb_enter+0x6b
rflags                    0x46
kdb_enter+0x6b: movq    $0,0x27566aa(%rip)
db> show proc
Process 12 (intr) at 0xfffffe0053dda560:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xffffffff83e167e0
 ABI: null
 flag: 0x10000284  flag2: 0
 reaper: 0xffffffff83e167e0 reapsubtree: 12
 sigparent: 20
 vmspace: 0xffffffff83e17780
   (map 0xffffffff83e17780)
   (map.pmap 0xffffffff83e17840)
   (pmap 0xffffffff83e178b0)
 threads: 22
100013                   I                                   [swi6: Giant taskq]
100015                   I                                   [swi5: fast taskq]
100018                   I                                   [swi6: task queue]
100031                   Run     CPU 0                       [swi1: netisr 0]
100032                   Run     CPU 1                       [swi1: hpts]
100033                   I                                   [swi1: hpts]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq33: virtio_pci2]
100060                   I                                   [irq34: virtio_pci2]
100061                   I                                   [irq35: virtio_pci2]
100063                   I                                   [irq1: atkbd0]
100064                   I                                   [irq12: psm0]
100065                   RunQ                                [swi0: uart uart++]
100069                   I                                   [swi1: pf send]
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 7325   803   799     0  LE     *sctp-in 0xfffffe0057886900  syz-executor2212783
 7324   802   799     0  LE     *sctp-in 0xfffffe0057886900  syz-executor2212783
 7323   805   799     0  LE     *sctp-tc 0xfffffe0057886000  syz-executor2212783
 7322   804   799     0  LE     *sctp-in 0xfffffe0057886900  syz-executor2212783
  805   801   799     0  S       nanslp  0xffffffff83e40941  syz-executor2212783
  804   801   799     0  S       nanslp  0xffffffff83e40941  syz-executor2212783
  803   801   799     0  S       nanslp  0xffffffff83e40941  syz-executor2212783
  802   801   799     0  S       nanslp  0xffffffff83e40941  syz-executor2212783
  801   799   799     0  S       nanslp  0xffffffff83e40941  syz-executor2212783
  799   797   799     0  Ss      pause   0xfffffe00579a00c8  csh
  797   688   797     0  Ss      select  0xfffffe0092b57040  sshd
  754     1   754     0  Ss+     ttyin   0xfffffe0056f5acb0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe00579c1cb0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe00579c20b0  getty
  751     1   751     0  Ss+     ttyin   0xfffffe00579c24b0  getty
  750     1   750     0  Ss+     ttyin   0xfffffe00579c28b0  getty
  749     1   749     0  Ss+     ttyin   0xfffffe00574900b0  getty
  748     1   748     0  Ss+     ttyin   0xfffffe00579c2cb0  getty
  747     1   747     0  Ss+     ttyin   0xfffffe00579c30b0  getty
  746     1   746     0  Ss+     ttyin   0xfffffe00579c34b0  getty
  692     1   692     0  Ss      nanslp  0xffffffff83e40940  cron
  688     1   688     0  Ss      select  0xfffffe0007980540  sshd
  501     1   501     0  Ds      biowr   0xfffffe00079966c8  syslogd
  430     1   430     0  Ss      select  0xfffffe00079807c0  devd
  429     1   429    65  Ss      select  0xfffffe0007980640  dhclient
  344     1   344     0  Ss      select  0xfffffe0007980740  dhclient
  341     1   341     0  Ss      select  0xfffffe0007980cc0  dhclient
   17     0     0     0  DL      syncer  0xffffffff83f65de0  [syncer]
   16     0     0     0  DL      vlruwt  0xfffffe0056fa4010  [vnlru]
   15     0     0     0  DL      (threaded)                  [bufdaemon]
100079                   D       psleep  0xffffffff83f64400  [bufdaemon]
100082                   D       -       0xffffffff83212100  [bufspacedaemon-0]
100094                   D       sdflush 0xfffffe005748f4e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83f97e80  [vmdaemon]
    8     0     0     0  DL      (threaded)                  [pagedaemon]
100077                   D       psleep  0xffffffff83f8bd38  [dom0]
100080                   D       launds  0xffffffff83f8bd44  [laundry: dom0]
100081                   D       umarcl  0xffffffff81e5d9f0  [uma]
    7     0     0     0  DL      -       0xffffffff83bfcde8  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff84b62330  [pf purge]
    5     0     0     0  DL      waiting 0xffffffff84743640  [sctp_iterator]
    4     0     0     0  RL      (threaded)                  [cam]
100044                   RunQ                                [doneq0]
100045                   D       -       0xffffffff83a9f2c0  [async]
100076                   D       -       0xffffffff83a9f140  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0056efcc88  [sequencer 00]
    3     0     0     0  DL      (threaded)                  [crypto]
100040                   D       crypto_ 0xffffffff83f87460  [crypto]
100041                   D       crypto_ 0xfffffe0053f6c030  [crypto returns 0]
100042                   D       crypto_ 0xfffffe0053f6c080  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100035                   D       -       0xffffffff83e15dc0  [g_event]
100036                   D       -       0xffffffff83e15de0  [g_up]
100037                   D       -       0xffffffff83e15e00  [g_down]
    2     0     0     0  WL      (threaded)                  [clock]
100029                   I                                   [clock (0)]
100030                   I                                   [clock (1)]
   12     0     0     0  RL      (threaded)                  [intr]
100013                   I                                   [swi6: Giant taskq]
100015                   I                                   [swi5: fast taskq]
100018                   I                                   [swi6: task queue]
100031                   Run     CPU 0                       [swi1: netisr 0]
100032                   Run     CPU 1                       [swi1: hpts]
100033                   I                                   [swi1: hpts]
100046                   I                                   [irq24: virtio_pci0]
100047                   I                                   [irq25: virtio_pci0]
100048                   I                                   [irq26: virtio_pci0]
100049                   I                                   [irq27: virtio_pci0]
100050                   I                                   [irq28: virtio_pci1]
100051                   I                                   [irq29: virtio_pci1]
100052                   I                                   [irq30: virtio_pci1]
100053                   I                                   [irq31: virtio_pci1]
100054                   I                                   [irq32: virtio_pci1]
100059                   I                                   [irq33: virtio_pci2]
100060                   I                                   [irq34: virtio_pci2]
100061                   I                                   [irq35: virtio_pci2]
100063                   I                                   [irq1: atkbd0]
100064                   I                                   [irq12: psm0]
100065                   RunQ                                [swi0: uart uart++]
100069                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053ddb010  [init]
   10     0     0     0  DL      audit_w 0xffffffff83f88000  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff83e167e0  [swapper]
100005                   D       -       0xfffffe0053c82700  [softirq_0]
100006                   D       -       0xfffffe0053c82600  [softirq_1]
100007                   D       -       0xfffffe0053c82500  [if_io_tqg_0]
100008                   D       -       0xfffffe0053c82400  [if_io_tqg_1]
100009                   D       -       0xfffffe0053c82300  [if_config_tqg_0]
100010                   D       -       0xfffffe000817bd00  [in6m_free taskq]
100011                   D       -       0xfffffe000817bc00  [linuxkpi_irq_wq]
100012                   D       -       0xfffffe000817bb00  [deferred_unmount ta]
100014                   D       -       0xfffffe000817b900  [thread taskq]
100016                   D       -       0xfffffe000817b700  [kqueue_ctx taskq]
100017                   D       -       0xfffffe000817b600  [pci_hp taskq]
100019                   D       -       0xfffffe000817b400  [inm_free taskq]
100020                   D       -       0xfffffe000817b300  [aiod_kick taskq]
100021                   D       -       0xfffffe000817b200  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe000817b200  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe000817b200  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe000817b200  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe000817b100  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe000817b100  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe000817b100  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe000817b100  [linuxkpi_long_wq_3]
100034                   D       -       0xfffffe000817b000  [firmware taskq]
100038                   D       -       0xfffffe000817ae00  [crypto_0]
100039                   D       -       0xfffffe000817ae00  [crypto_1]
100055                   D       -       0xfffffe000817ac00  [vtnet0 rxq 0]
100056                   D       -       0xfffffe000817ab00  [vtnet0 txq 0]
100057                   D       -       0xfffffe000817aa00  [vtnet0 rxq 1]
100058                   D       -       0xfffffe000817a900  [vtnet0 txq 1]
100062                   D       vtbslp  0xfffffe0007982100  [virtio_balloon]
100066                   D       -       0xffffffff82b73440  [deadlkres]
100071                   D       -       0xfffffe0057901c00  [acpi_task_0]
100072                   D       -       0xfffffe0057901c00  [acpi_task_1]
100073                   D       -       0xfffffe0057901c00  [acpi_task_2]
100074                   D       -       0xfffffe000817c100  [mca taskq]
100075                   D       -       0xfffffe000817ad00  [CAM taskq]
db> show all locks
Process 7325 (syz-executor2212783) thread 0xfffffe0058bcd560 (100093)
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffffe0092bb4a00) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3338
Process 7324 (syz-executor2212783) thread 0xfffffe0058df8560 (100098)
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffffe0092bb88f8) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3338
Process 7323 (syz-executor2212783) thread 0xfffffe0092e443a0 (102346)
exclusive sleep mutex sctp-inp (inp) r = 0 (0xfffffe0092bb5420) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3340
exclusive rw sctp-info (sctp-info) r = 0 (0xfffffe00081d6f58) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3339
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffffe0092bb5440) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3338
Process 7322 (syz-executor2212783) thread 0xfffffe0092e221e0 (102123)
exclusive sleep mutex sctp-create (inp_create) r = 0 (0xfffffe0092bb4548) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:3338
Process 501 (syslogd) thread 0xfffffe0058df7e40 (100099)
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0007996748) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_bio.c:3988
exclusive lockmgr ufs (ufs) r = 0 (0xfffffe0092658cb0) locked @ /syzkaller/managers/main/kernel/sys/kern/vfs_syscalls.c:3562
Process 12 (intr) thread 0xfffffe0053e8b560 (100031)
exclusive sleep mutex so_rcv (so_rcv) r = 0 (0xfffffe008fe53560) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctputil.c:4847
exclusive sleep mutex sctp-read (inpr) r = 0 (0xfffffe0092bb5460) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctputil.c:4875
exclusive sleep mutex sctp-tcb (tcb) r = 0 (0xfffffe0092daebf0) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:2138
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            5        11524K            5
          tcp_hpts            7         4801K            7
            devbuf         4217         4323K         4245
         sysctloid        35217         2075K        35288
             vtbuf           24         1968K           46
              kobj          329         1316K          493
            newblk            9         1026K         7102
          vfscache            3         1025K            3
               pcb           27          546K        13036
          inodedep           61          535K         6592
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
           subproc          112          206K         7393
            acpica         1674          184K        58126
              vmem            3          146K            5
           tidhash            3          141K            3
            linker          358          134K          386
           pagedep            9          130K         6527
        tfo_ccache            1          128K            1
          IP reass            1          128K            1
         vnet_data            1          112K            1
               sem            4          106K            4
            DEVFS1          105          105K          114
               bus          998           82K         5213
          mtx_pool            2           72K            2
          syncache            1           68K            1
            module          516           65K          516
          acpitask            1           64K            1
       ddb_capture            1           64K            1
              umtx          330           42K          330
           kdtrace          187           39K        21556
              temp           17           33K         1741
         hostcache            1           32K            1
               shm            1           32K            1
            DEVFS3          124           31K          134
               msg            4           30K            4
        gtaskqueue           18           26K           18
         sctp_timw           94           24K           94
            kbdmux            6           22K            6
        DEVFS_RULE           56           20K           56
               BPF           10           18K           10
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
           ithread           97           16K           97
            bus-sc           34           15K         1682
            dirrem           52           13K         6531
            KTRACE          100           13K          100
      eventhandler          151           13K          151
              kenv           95           12K           95
            ifaddr           30           12K           32
              rman           88           11K          431
              GEOM           61           11K          483
          routetbl           50           11K          176
         CAM queue            5           11K         1528
         bmsafemap            2            9K         6557
              UART           12            9K           12
           devstat            4            9K            4
              ksem            1            8K            1
               rpc            2            8K            2
             shmfd            1            8K            1
       pfs_vncache            1            8K            1
              cred           30            8K          364
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          296
         taskqueue           63            7K           63
          freefile           52            7K         6521
            sglist            5            7K            5
           CAM DEV            3            6K          510
            plimit           20            5K          557
       ufs_dirhash           24            5K           24
         sctp_atcl           12            5K        19495
               UMA          271            5K          271
                vt           11            5K           11
             ifnet            3            5K            3
         sctp_stro            4            4K         6499
           memdesc            1            4K            1
               MCA           32            4K           32
          filedesc            1            4K            1
             evdev            4            4K            4
           acpisem           28            4K           28
             hhook           15            4K           17
       ether_multi           40            4K           50
           lltable           11            4K           12
          pf_ifnet            5            3K            6
         in6_multi           25            3K           25
            kqueue           45            3K         7328
          terminal           11            3K           11
           pwddesc           41            3K         7326
           session           19            3K           45
           uidinfo            3            3K           19
         proc-args           64            3K         8374
         sctp_stri            4            2K        12994
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         ipsec-saq            2            2K            2
             lockf           16            2K           44
            Unitno           27            2K           39
           CAM XPT           22            2K          543
               msi           12            2K           12
             selfd           22            2K        93531
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
             clone            9            2K            9
          freework            5            2K         6521
           softdep            1            1K            1
             mkdir            8            1K        13022
          freeblks            4            1K         6520
            sahead            1            1K            1
          secasvar            1            1K            1
       vnodemarker            2            1K          144
      NFSD session            1            1K            1
        CAM periph            4            1K          271
            select            7            1K           29
         sctp_atky           16            1K        32491
             ipsec            3            1K            3
             nhops            6            1K            6
         toponodes            6            1K            6
            isadev            6            1K            6
             mount           16            1K           89
          pci_link           10            1K           10
          sctp_ifa            5            1K            6
            crypto            4            1K        19489
            ip6ndp            4            1K            5
 encap_export_host           12            1K           12
         newdirblk            4            1K         6511
            diradd            4            1K         6552
          in_multi            2            1K            4
              pfil            4            1K            4
              cdev            2            1K            2
       inpcbpolicy           14            1K        13137
            CC Mem            7            1K        13005
               osd           11            1K        13016
    chacha20random            1            1K            1
          sctp_ifn            2            1K            6
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
            DEVFSP            4            1K         6508
             DEVFS            9            1K           10
               mld            2            1K            2
              igmp            2            1K            2
            vnodes            1            1K            1
           CAM SIM            2            1K            2
            feeder            7            1K            7
         sctp_athm           12            1K        25992
           tcpfunc            3            1K            3
        loginclass            3            1K            7
            prison            6            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
         cryptodev            2            1K        19538
          sctp_map            8            1K        12998
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          procdesc            1            1K            6
          pmchooks            1            1K            1
            soname            4            1K        48943
               tun            3            1K            3
          sctp_vrf            1            1K            1
              vnet            1            1K            1
           entropy            2            1K           44
               pmc            1            1K            1
          acpiintr            1            1K            1
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
          filecaps            1            1K           82
          pf_table            0            0K            0
           pf_rule            0            0K            0
           pf_altq            0            0K            0
           pf_osfp            0            0K            0
           pf_temp            0            0K            0
            mqdata            0            0K            0
            tcp_do            0            0K            0
           tcp_fsb            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K         6499
         sctp_iter            0            0K            4
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K            4
         sctp_aadr            0            0K            0
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
           filemon            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
               ixl            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
          xen_intr            0            0K            0
      NFSD V4state            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            DEVFS4            0            0K            0
     vm_fictitious            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
          xen_acpi            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           UMAHash            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K         6107
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K           24
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefrag            0            0K            2
        allocindir            0            0K            0
          indirdep            0            0K            3
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
            vtfont            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
          ktls_ocf            0            0K            0
       AHCI driver            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K            3
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               LRO            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            0
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
               agp            0            0K            0
            statfs            0            0K         6738
     namei_tracker            0            0K            0
       export_host            0            0K            0
        cl_savebuf            0            0K            3
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
       tcp_log_dev            0            0K            0
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
            biobuf            0            0K            0
               aio            0            0K            0
               lio            0            0K            0
               acl            0            0K            0
          mbuf_tag            0            0K            0
              ktls            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        20593
          ioctlops            0            0K           86
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
              sbuf            0            0K          288
          mpr_user            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
            MPRSAS            0            0K            0
              SWAP            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
         sysctltmp            0            0K          665
            sysctl            0            0K            3
        md_sectors            0            0K            0
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
           md_disk            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
          ix_sriov            0            0K            0
             cache            0            0K            0
        aacraidcam            0            0K            0
          kcovinfo            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            1
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
       aacraid_buf            0            0K            0
            aaccam            0            0K            0
         boottrace            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8320    1078       16888       0     254   38494208        0
              mbuf    256    8585    1077     5663070       0     254    2473472        0
              pbuf   2624       0     794           0       0       2    2083456        0
          BUF TRIE    144     169   11591         664       0      62    1693440        0
       malloc-4096   4096     393       7        8390       0       2    1638400        0
        malloc-384    384    4187      13        4188       0      30    1612800        0
        malloc-128    128   11600     273       18125       0     126    1519744        0
       UMA Slabs 0    112   10604      10       10604       0     126    1188768        0
         sctp_asoc   2256       4     506        6499       0     254    1150560        0
      mbuf_cluster   2048     508       0         508       0     254    1040384        0
         vmem btag     56   16850     109       16850       0     254     949704        0
         FFS inode   1160     501      31        7022       0       8     617120        0
           sctp_ep   1208       4     506        6499       0     254     616080        0
             tcpcb   1104       7     504       13005       0     254     564144        0
        RADIX NODE    144    3340     211      106761       0      62     511344        0
            socket    960      26     482       20830       0     254     487680        0
      malloc-65536  65536       7       0           7       0       1     458752        0
        256 Bucket   2048     170      40       17161       0       8     430080        0
         VM OBJECT    264    1447     173      156895       0      30     427680        0
            lkpimm    168       1    2327           1       0      62     391104        0
          lkpicurr    168       2    2326           2       0      62     391104        0
        sctp_raddr    736       4     513        6499       0     254     380512        0
    ertt_txseginfo     40    7391     285     5453338       0     254     307040        0
            THREAD   1824     141      24       14229       0       8     300960        0
             VNODE    448     531      99        7054       0      30     282240        0
         malloc-64     64    4066     281       30581       0     254     278208        0
         malloc-16     16   14609     641       53674       0     254     244000        0
            DEVCTL   1024       0     220         124       0       0     225280        0
         MAP ENTRY     96    1718     424      344618       0     126     205632        0
        malloc-128    128    1249     270       37019       0     126     194432        0
         malloc-32     32    5365     557       32038       0     254     189504        0
         UMA Zones    768     243       1         243       0      16     187392        0
        malloc-256    256     127     563       20310       0      62     176640        0
        malloc-384    384      81     369        6643       0      30     172800        0
        malloc-256    256     122     508       20033       0      62     161280        0
     FPU_save_area    832     143      37       14327       0      16     149760        0
       tcp_bbr_map    128     876     271     1423062       0     126     146816        0
        malloc-128    128     675     472        7283       0     126     146816        0
       S VFS Cache    104    1032     372        7585       0     126     146016        0
       FFS2 dinode    256     501      69        7022       0      62     145920        0
      malloc-65536  65536       0       2         136       0       1     131072        0
      malloc-65536  65536       0       2          54       0       1     131072        0
       malloc-2048   2048       6      58         522       0       8     131072        0
       malloc-1024   1024     105      23         114       0      16     131072        0
             unpcb    256       7     503        1177       0     254     130560        0
       mbuf_packet    256       7     501       45590       0     254     130048        0
        128 Bucket   1024      59      56        8207       0      16     117760        0
          ksiginfo    112      55     989         140       0     126     116928        0
           VMSPACE   2560      28      14        7310       0       4     107520        0
              PROC   1368      44      22        7325       0       8      90288        0
       malloc-8192   8192       9       2         137       0       1      90112        0
          UMA Kegs    384     229       4         229       0      30      89472        0
        malloc-256    256     160     155       14906       0      62      80640        0
        sctp_readq    152       1     519        5203       0     254      79040        0
        sctp_chunk    152       4     516       11699       0     254      79040        0
         filedesc0   1072      45      25        7326       0       8      75040        0
             g_bio    408       4     176        6011       0      30      73440        0
         64 Bucket    512      90      46        3936       0      30      69632        0
         malloc-64     64     561     510       22607       0     254      68544        0
        malloc-128    128     292     235         304       0     126      67456        0
        malloc-128    128     347     180        1226       0     126      67456        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2         120       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-16384  16384       3       1         163       0       1      65536        0
         32 Bucket    256      80     175        3318       0      62      65280        0
sctp_stream_msg_out    112       0     540        6496       0     254      60480        0
       malloc-8192   8192       7       0           7       0       1      57344        0
         tcp_inpcb    424       7     119       13005       0      30      53424        0
        malloc-128    128     141     262        7146       0     126      51584        0
        malloc-256    256     128      67         141       0      62      49920        0
        malloc-256    256      70     125        7012       0      62      49920        0
        malloc-256    256     118      77         760       0      62      49920        0
        malloc-256    256       9     186       26301       0      62      49920        0
           DIRHASH   1024      35      13          35       0      16      49152        0
             NAMEI   1024       0      48       52414       0      16      49152        0
      malloc-16384  16384       3       0           3       0       1      49152        0
       malloc-2048   2048       7      17        6528       0       8      49152        0
       malloc-1024   1024      34      14          42       0      16      49152        0
       malloc-1024   1024       9      39        7028       0      16      49152        0
        malloc-512    512      14      82       13004       0      30      49152        0
        malloc-384    384      39      81       19522       0      30      46080        0
       tcp_bbr_pcb    832       4      50        6498       0      16      44928        0
          syncache    168       0     264           5       0     254      44352        0
            clpbuf   2624       0      16          21       0      16      41984        0
       malloc-8192   8192       5       0           5       0       1      40960        0
       Mountpoints   2752       2      12           2       0       4      38528        0
         udp_inpcb    424       6      84         128       0      30      38160        0
            da_ccb    544       1      69        1617       0      16      38080        0
            pcpu-8      8    4223     385        4251       0     254      36864        0
         malloc-64     64     118     449        6639       0     254      36288        0
         malloc-64     64     111     456       27725       0     254      36288        0
         malloc-64     64      89     478       93630       0     254      36288        0
         malloc-64     64     491      76         696       0     254      36288        0
         malloc-64     64      20     547         293       0     254      36288        0
        malloc-128    128      20     259          21       0     126      35712        0
        malloc-128    128      64     215       13153       0     126      35712        0
     routing nhops    256      10     125          17       0      62      34560        0
           ttyoutq    256      72      63         160       0      62      34560        0
        malloc-384    384      53      37         393       0      30      34560        0
        malloc-256    256      16     119          49       0      62      34560        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-16384  16384       1       1          16       0       1      32768        0
       malloc-4096   4096       3       5        6741       0       2      32768        0
       malloc-4096   4096       8       0           8       0       2      32768        0
       malloc-2048   2048       3      13           3       0       8      32768        0
       malloc-2048   2048       4      12          67       0       8      32768        0
       malloc-2048   2048       7       9         530       0       8      32768        0
       malloc-2048   2048       7       9         198       0       8      32768        0
       malloc-1024   1024       6      26         209       0      16      32768        0
       malloc-1024   1024       4      28           4       0      16      32768        0
       malloc-1024   1024       6      26         964       0      16      32768        0
        malloc-512    512       5      59         315       0      30      32768        0
        malloc-512    512       3      61           3       0      30      32768        0
           pcpu-64     64     494      18         494       0     254      32768        0
            ttyinq    160     135      65         300       0      62      32000        0
             Files     80      75     325       46052       0     126      32000        0
            cpuset    104       7     272           7       0     126      29016        0
        sctp_laddr     48       0     588           4       0     254      28224        0
   tcp_inpcb ports     32       5     877        6500       0     254      28224        0
               PWD     32      14     868        6628       0     254      28224        0
         16 Bucket    144      57     139         408       0      62      28224        0
          4 Bucket     48       7     581         162       0     254      28224        0
             ripcb    424       1      62           4       0      30      26712        0
         TURNSTILE    136     166      23         166       0      62      25704        0
              PGRP     88      19     257          45       0     126      24288        0
          rl_entry     40      38     568          38       0     254      24240        0
           rtentry    168      13     131          17       0      62      24192        0
              pipe    728       6      27         316       0      16      24024        0
          8 Bucket     80      50     250        1566       0     126      24000        0
        SLEEPQUEUE     88     166      90         166       0     126      22528        0
         hostcache     64       1     314           1       0     254      20160        0
             udpcb     32       6     624         128       0     254      20160        0
   udp_inpcb ports     32       3     627          40       0     254      20160        0
              ertt     72       7     273       13005       0     126      20160        0
         malloc-64     64       2     313          29       0     254      20160        0
         malloc-32     32       6     624          56       0     254      20160        0
         malloc-32     32     287     343         489       0     254      20160        0
         malloc-32     32      74     556          86       0     254      20160        0
         malloc-32     32      41     589         737       0     254      20160        0
         malloc-32     32      41     589       28829       0     254      20160        0
         malloc-32     32      51     579        7775       0     254      20160        0
          2 Bucket     32      44     586         447       0     254      20160        0
 epoch_record pcpu    256       4      60           4       0      62      16384        0
      malloc-16384  16384       1       0           1       0       1      16384        0
       malloc-8192   8192       1       1           3       0       1      16384        0
       malloc-4096   4096       3       1          87       0       2      16384        0
       malloc-2048   2048       1       7           1       0       8      16384        0
       malloc-1024   1024       8       8           9       0      16      16384        0
        malloc-512    512       0      32           1       0      30      16384        0
        malloc-512    512       0      32          10       0      30      16384        0
           SMR CPU     32       7     504           7       0     254      16352        0
      vtnet_tx_hdr     24       0     668     5461249       0     254      16032        0
         malloc-16     16     304     696         500       0     254      16000        0
              kenv    258      15      45        1038       0      30      15480        0
            mqnode    416       3      33           3       0      30      14976        0
              vmem   1856       1       7           1       0       8      14848        0
        SMR SHARED     24       7     504           7       0     254      12264        0
         malloc-32     32      80     298          93       0     254      12096        0
             KNOTE    160       0      75           8       0      62      12000        0
         malloc-16     16      19     731       13024       0     254      12000        0
         malloc-16     16      18     732         132       0     254      12000        0
         malloc-16     16      23     727          25       0     254      12000        0
         malloc-16     16     185     565        2994       0     254      12000        0
         malloc-16     16      15     735       52675       0     254      12000        0
         malloc-16     16      42     708       14388       0     254      12000        0
        malloc-384    384       1      29           2       0      30      11520        0
        malloc-384    384      20      10          21       0      30      11520        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-4096   4096       0       2           7       0       2       8192        0
       malloc-4096   4096       0       2           2       0       2       8192        0
           pcpu-16     16      14     498          14       0     254       8192        0
       UMA Slabs 1    176       9      13           9       0      62       3872        0
        KMAP ENTRY     96      12      27          14       0       0       3744        0
            pcpu-4      4       1     511           1       0     254       2048        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0     254          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0     254          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags    184       0       0           0       0      62          0        0
      tcp_rack_pcb    896       0       0           0       0      16          0        0
      tcp_rack_map    120       0       0           0       0     126          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     72       0       0           0       0     254          0        0
               ipq     56       0       0           0       0     254          0        0
   tcp_log_id_node    120       0       0           0       0     126          0        0
 tcp_log_id_bucket    176       0       0           0       0      62          0        0
           tcp_log    416       0       0           0       0     254          0        0
          tcpreass     48       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
       ripcb ports     32       0       0           0       0     254          0        0
udplite_inpcb ports     32       0       0           0       0     254          0        0
     udplite_inpcb    424       0       0           0       0      30          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            itimer    352       0       0           0       0      30          0        0
            AIOLIO    272       0       0           0       0      30          0        0
             AIOCB    552       0       0           0       0      16          0        0
               AIO    208       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    104       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0           0       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
       malloc-2048   2048       0       0           0       0       8          0        0
       malloc-1024   1024       0       0           0       0      16          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-384    384       0       0           0       0      30          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (242):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/09/11 01:49 freebsd-src 13604a80f7a4 356d8217 console log report syz C ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2022/08/28 14:44 freebsd-src b875d4f5ddcb 07177916 console log report syz ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2024/03/02 10:52 freebsd-src 2956f5885cf4 25905f5d console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2024/02/27 08:32 freebsd-src bc4c7e78f845 05e69c83 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2024/02/23 13:20 freebsd-src 09cb8031b43c 8d446f15 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2024/02/21 22:03 freebsd-src 2f4cbf459d4a 9e510875 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2024/02/09 00:54 freebsd-src d5677b0b8b8a 7f07e9b0 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/09/24 19:35 freebsd-src 0a5d2802b41f 0b6a67ac console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/09/14 13:35 freebsd-src 79278872ad96 0b6a67ac console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/09/05 14:18 freebsd-src e8fc6a9980d9 0b6286dc console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/09/01 02:49 freebsd-src d88983333447 696ea0d2 console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/08/28 00:06 freebsd-src 315ee00fa961 7ba13a15 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/08/19 06:36 freebsd-src 57a3b81785c0 d216d8a0 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/08/18 05:25 freebsd-src a4aaee2120ce 74b106b6 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/08/17 12:46 freebsd-src d1d7a273707a 74b106b6 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/30 12:36 freebsd-src 38f5cb1bfbe1 92476829 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/30 09:25 freebsd-src 38f5cb1bfbe1 92476829 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/29 07:09 freebsd-src 38588749af45 92476829 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/25 00:07 freebsd-src 92103adbebd9 9a0ddda3 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/24 21:24 freebsd-src 92103adbebd9 9a0ddda3 console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/20 13:02 freebsd-src a52f23f4c49e d922ca7e console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/07/15 15:45 freebsd-src 2a0c0aea4209 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/28 10:32 freebsd-src 6927176113ee ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/28 03:17 freebsd-src 2e1665ff3454 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/21 03:24 freebsd-src 05963ea4d130 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/14 12:14 freebsd-src 246aec9a3ce3 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/09 12:42 freebsd-src 5253d74e4650 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/06 02:06 freebsd-src d66540e829ec ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/06/05 01:07 freebsd-src 6fa30de4d3ca ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/05/29 17:13 freebsd-src eb98f7791040 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/05/28 23:26 freebsd-src 27234ad6999b ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/05/19 01:16 freebsd-src 0f62290e7f74 ee50e71c console log report ci-freebsd-main Fatal trap NUM: page fault in __mtx_assert
2023/08/20 03:48 freebsd-src a129642ced9e d216d8a0 console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/08/16 16:50 freebsd-src 77a999ef2f0c 1b4c2821 console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/08/06 21:53 freebsd-src 66f5f4bffed9 4ffcc9ef console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/07/14 11:31 freebsd-src 0a5e35a7b181 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/07/13 15:28 freebsd-src 6fc7fc2dbb2b ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/07/08 07:20 freebsd-src bfa51318dee8 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/07/08 04:50 freebsd-src be30fd3ab2e8 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/07/05 21:51 freebsd-src 544deacc90f4 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/06/27 11:33 freebsd-src e495600ce3ab ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/06/06 20:19 freebsd-src 82aa1470c94c ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/06/05 20:50 freebsd-src 9d6ae1e3c26a ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/06/04 07:28 freebsd-src e5b3cd6100e4 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/05/21 14:48 freebsd-src 63ad5ce7335b ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2023/05/17 04:05 freebsd-src 0282f87530b3 ee50e71c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
2022/06/03 10:05 freebsd-src 1326017849ee eee80d3c console log report ci-freebsd-i386 Fatal trap NUM: page fault in __mtx_assert
* Struck through repros no longer work on HEAD.