syzbot


panic: Queues are not empty when handling SHUTDOWN-COMPLETE

Status: fixed on 2022/05/29 00:41
Reported-by: syzbot+341e9ebd9d24ca7dc62a@syzkaller.appspotmail.com
Fix commit: 64b297e803bd sctp: improve handling of send() when association is shutdown
First crash: 817d, last: 287d

Sample crash report:
login: panic: Queues are not empty when handling SHUTDOWN-COMPLETE
cpuid = 0
time = 1650518396
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0xc7/frame 0xfffffe0053bf9e30
kdb_backtrace() at kdb_backtrace+0xd3/frame 0xfffffe0053bf9f90
vpanic() at vpanic+0x2b8/frame 0xfffffe0053bfa070
panic() at panic+0xb5/frame 0xfffffe0053bfa140
sctp_handle_shutdown_complete() at sctp_handle_shutdown_complete+0x1ab/frame 0xfffffe0053bfa170
sctp_process_control() at sctp_process_control+0x2fd3/frame 0xfffffe0053bfa640
sctp_common_input_processing() at sctp_common_input_processing+0xb35/frame 0xfffffe0053bfa8a0
sctp6_input_with_port() at sctp6_input_with_port+0x597/frame 0xfffffe0053bfaa70
sctp6_input() at sctp6_input+0x1f/frame 0xfffffe0053bfaa90
ip6_input() at ip6_input+0x1f40/frame 0xfffffe0053bfacf0
swi_net() at swi_net+0x2f0/frame 0xfffffe0053bfad90
ithread_loop() at ithread_loop+0x4f1/frame 0xfffffe0053bfaef0
fork_exit() at fork_exit+0xd0/frame 0xfffffe0053bfaf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0053bfaf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 12 tid 100029 ]
Stopped at      kdb_enter+0x6b: movq    $0,0x270578a(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xd5b0c8e962dfaf01
rdx         0xdffff7c000000000
rbx                          0
rsp         0xfffffe0053bf9f70
rbp         0xfffffe0053bf9f90
rsi                        0x1
rdi                          0
r8                         0x3
r9                  0xffffffff
r10                          0
r11                       0xbf
r12         0xfffffe0053ec4ac0
r13         0xfffffe0053bf9f01
r14         0xffffffff82bc74a0  .str.26
r15         0xffffffff82bc74a0  .str.26
rip         0xffffffff81774c0b  kdb_enter+0x6b
rflags                    0x46
kdb_enter+0x6b: movq    $0,0x270578a(%rip)
db> show proc
Process 12 (intr) at 0xfffffe0053ddb548:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xffffffff83e1da40
 ABI: null
 flag: 0x10000284  flag2: 0
 reaper: 0xffffffff83e1da40 reapsubtree: 12
 sigparent: 20
 vmspace: 0xffffffff83e1e9e0
   (map 0xffffffff83e1e9e0)
   (map.pmap 0xffffffff83e1eaa0)
   (pmap 0xffffffff83e1eb08)
 threads: 23
100010                   I                                   [swi6: Giant taskq]
100017                   I                                   [swi5: fast taskq]
100020                   I                                   [swi6: task queue]
100029                   Run     CPU 0                       [swi1: netisr 0]
100032                   I                                   [swi3: busdma]
100033                   I                                   [swi1: hpts]
100034                   I                                   [swi1: hpts]
100047                   I                                   [irq24: virtio_pci0]
100048                   I                                   [irq25: virtio_pci0]
100049                   I                                   [irq26: virtio_pci0]
100050                   I                                   [irq27: virtio_pci0]
100051                   I                                   [irq28: virtio_pci1]
100052                   I                                   [irq29: virtio_pci1]
100053                   I                                   [irq30: virtio_pci1]
100054                   I                                   [irq31: virtio_pci1]
100055                   I                                   [irq32: virtio_pci1]
100060                   I                                   [irq33: virtio_pci2]
100061                   I                                   [irq34: virtio_pci2]
100062                   I                                   [irq35: virtio_pci2]
100064                   I                                   [irq1: atkbd0]
100065                   I                                   [irq12: psm0]
100066                   I                                   [swi0: uart uart++]
100070                   I                                   [swi1: pf send]
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
 1197   779   777     0  RE      CPU 1                       syz-executor8581159
  779   777   777     0  S       nanslp  0xffffffff83e47a41  syz-executor8581159
  777   775   777     0  Ss      pause   0xfffffe0058bb75f8  csh
  775   688   775     0  Ss      select  0xfffffe0053dd22c0  sshd
  754     1   754     0  Ss+     ttyin   0xfffffe0056feb4b0  getty
  753     1   753     0  Ss+     ttyin   0xfffffe0056fe68b0  getty
  752     1   752     0  Ss+     ttyin   0xfffffe0056fe6cb0  getty
  751     1   751     0  Ss+     ttyin   0xfffffe0056fe70b0  getty
  750     1   750     0  Ss+     ttyin   0xfffffe0056fe74b0  getty
  749     1   749     0  Ss+     ttyin   0xfffffe0056fe78b0  getty
  748     1   748     0  Ss+     ttyin   0xfffffe0056fe7cb0  getty
  747     1   747     0  Ss+     ttyin   0xfffffe0056fe90b0  getty
  746     1   746     0  Ss+     ttyin   0xfffffe0056fe94b0  getty
  744     1    18     0  S+      piperd  0xfffffe0058b4f2e8  logger
  743   742    18     0  S+      nanslp  0xffffffff83e47a40  sleep
  742     1    18     0  S+      wait    0xfffffe0053ee9a90  sh
  692     1   692     0  Ss      nanslp  0xffffffff83e47a41  cron
  688     1   688     0  Ss      select  0xfffffe00570367c0  sshd
  501     1   501     0  Ss      select  0xfffffe00570368c0  syslogd
  430     1   430     0  Ss      select  0xfffffe00570110c0  devd
  429     1   429    65  Ss      select  0xfffffe0057036940  dhclient
  344     1   344     0  Ss      select  0xfffffe0057011040  dhclient
  341     1   341     0  Ss      select  0xfffffe0057036b40  dhclient
   17     0     0     0  DL      syncer  0xffffffff83f6d260  [syncer]
   16     0     0     0  DL      vlruwt  0xfffffe00587b6a90  [vnlru]
   15     0     0     0  DL      (threaded)                  [bufdaemon]
100080                   D       psleep  0xffffffff83f6b860  [bufdaemon]
100083                   D       -       0xffffffff83211f80  [bufspacedaemon-0]
100095                   D       sdflush 0xfffffe0053f698e8  [/ worker]
    9     0     0     0  DL      psleep  0xffffffff83f9f380  [vmdaemon]
    8     0     0     0  DL      (threaded)                  [pagedaemon]
100078                   D       psleep  0xffffffff83f93238  [dom0]
100081                   D       launds  0xffffffff83f93244  [laundry: dom0]
100082                   D       umarcl  0xffffffff81eafd30  [uma]
    7     0     0     0  DL      -       0xffffffff83c03788  [rand_harvestq]
    6     0     0     0  DL      pftm    0xffffffff84855530  [pf purge]
    5     0     0     0  DL      waiting 0xffffffff846904a0  [sctp_iterator]
    4     0     0     0  DL      (threaded)                  [cam]
100045                   D       -       0xffffffff83aa56c0  [doneq0]
100046                   D       -       0xffffffff83aa5640  [async]
100077                   D       -       0xffffffff83aa54c0  [scanner]
   14     0     0     0  DL      seqstat 0xfffffe0056f1ac88  [sequencer 00]
    3     0     0     0  DL      (threaded)                  [crypto]
100041                   D       crypto_ 0xffffffff83f8ea40  [crypto]
100042                   D       crypto_ 0xfffffe0053effd30  [crypto returns 0]
100043                   D       crypto_ 0xfffffe0053effd80  [crypto returns 1]
   13     0     0     0  DL      (threaded)                  [geom]
100036                   D       -       0xffffffff83e1d000  [g_event]
100037                   D       -       0xffffffff83e1d020  [g_up]
100038                   D       -       0xffffffff83e1d040  [g_down]
    2     0     0     0  WL      (threaded)                  [clock]
100030                   I                                   [clock (0)]
100031                   I                                   [clock (1)]
   12     0     0     0  RL      (threaded)                  [intr]
100010                   I                                   [swi6: Giant taskq]
100017                   I                                   [swi5: fast taskq]
100020                   I                                   [swi6: task queue]
100029                   Run     CPU 0                       [swi1: netisr 0]
100032                   I                                   [swi3: busdma]
100033                   I                                   [swi1: hpts]
100034                   I                                   [swi1: hpts]
100047                   I                                   [irq24: virtio_pci0]
100048                   I                                   [irq25: virtio_pci0]
100049                   I                                   [irq26: virtio_pci0]
100050                   I                                   [irq27: virtio_pci0]
100051                   I                                   [irq28: virtio_pci1]
100052                   I                                   [irq29: virtio_pci1]
100053                   I                                   [irq30: virtio_pci1]
100054                   I                                   [irq31: virtio_pci1]
100055                   I                                   [irq32: virtio_pci1]
100060                   I                                   [irq33: virtio_pci2]
100061                   I                                   [irq34: virtio_pci2]
100062                   I                                   [irq35: virtio_pci2]
100064                   I                                   [irq1: atkbd0]
100065                   I                                   [irq12: psm0]
100066                   I                                   [swi0: uart uart++]
100070                   I                                   [swi1: pf send]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffffe0053ddc000  [init]
   10     0     0     0  DL      audit_w 0xffffffff83f8f540  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff83e1da40  [swapper]
100005                   D       -       0xfffffe0007973100  [softirq_0]
100006                   D       -       0xfffffe0007973000  [softirq_1]
100007                   D       -       0xfffffe0007972e00  [if_io_tqg_0]
100008                   D       -       0xfffffe0007972d00  [if_io_tqg_1]
100009                   D       -       0xfffffe0007972c00  [if_config_tqg_0]
100011                   D       -       0xfffffe0007972a00  [aiod_kick taskq]
100012                   D       -       0xfffffe0007972900  [inm_free taskq]
100013                   D       -       0xfffffe0007972800  [linuxkpi_irq_wq]
100014                   D       -       0xfffffe0007972700  [in6m_free taskq]
100015                   D       -       0xfffffe0007972600  [deferred_unmount ta]
100016                   D       -       0xfffffe0007972500  [thread taskq]
100018                   D       -       0xfffffe0007972300  [pci_hp taskq]
100019                   D       -       0xfffffe0007972200  [kqueue_ctx taskq]
100021                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_0]
100022                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_1]
100023                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_2]
100024                   D       -       0xfffffe0007972000  [linuxkpi_short_wq_3]
100025                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_0]
100026                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_1]
100027                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_2]
100028                   D       -       0xfffffe0007971e00  [linuxkpi_long_wq_3]
100035                   D       -       0xfffffe0053f31300  [firmware taskq]
100039                   D       -       0xfffffe0053f31100  [crypto_0]
100040                   D       -       0xfffffe0053f31100  [crypto_1]
100056                   D       -       0xfffffe0053f2e000  [vtnet0 rxq 0]
100057                   D       -       0xfffffe0007974e00  [vtnet0 txq 0]
100058                   D       -       0xfffffe0007974d00  [vtnet0 rxq 1]
100059                   D       -       0xfffffe0007974c00  [vtnet0 txq 1]
100063                   D       vtbslp  0xfffffe0057011800  [virtio_balloon]
100067                   D       -       0xffffffff82bcd321  [deadlkres]
100071                   D       -       0xfffffe0007973200  [mca taskq]
100072                   D       -       0xfffffe00574c3200  [acpi_task_0]
100073                   D       -       0xfffffe00574c3200  [acpi_task_1]
100074                   D       -       0xfffffe00574c3200  [acpi_task_2]
100076                   D       -       0xfffffe0053f30e00  [CAM taskq]
db> show all locks
Process 12 (intr) thread 0xfffffe0053ec4ac0 (100029)
exclusive sleep mutex sctp-tcb (tcb) r = 0 (0xfffffe0092f56180) locked @ /syzkaller/managers/main/kernel/sys/netinet/sctp_pcb.c:2138
db> show malloc
              Type        InUse        MemUse     Requests
           pf_hash            5        11524K            5
          tcp_hpts            7         4801K            7
            devbuf         4217         4323K         4242
         sysctloid        35322         2081K        35393
             vtbuf           24         1968K           46
              kobj          328         1312K          489
            newblk          707         1201K          719
          vfscache            3         1025K            3
               pcb           29          548K         2546
          inodedep           48          530K           71
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
           callout            2          512K            2
              intr            4          472K            4
            acpica         1674          184K        57552
           subproc           90          183K         1247
         vnet_data            1          168K            1
           tidhash            3          141K            3
              vmem            3          138K            4
            linker          358          134K          386
           pagedep           14          132K           18
        tfo_ccache            1          128K            1
               sem            4          106K            4
            DEVFS1          105          105K          114
               bus          994           81K         5207
          mtx_pool            2           72K            2
          syncache            1           68K            1
            module          513           65K          513
          acpitask            1           64K            1
       ddb_capture            1           64K            1
              temp           17           33K         1606
           kdtrace          159           33K         1316
         hostcache            1           32K            1
               shm            1           32K            1
            DEVFS3          124           31K          134
              umtx          242           31K          242
               msg            4           30K            4
        gtaskqueue           18           26K           18
            kbdmux            6           22K            6
        DEVFS_RULE           56           20K           56
               BPF           10           18K           10
         sctp_timw           70           18K           70
         ufs_mount            4           17K            5
              proc            3           17K            3
               tty           16           16K           16
           ithread          100           16K          100
            bus-sc           34           15K         1681
            KTRACE          100           13K          100
              kenv           95           12K           95
      eventhandler          134           12K          134
            ifaddr           30           12K           32
              rman           88           11K          431
              GEOM           61           11K          490
          routetbl           50           11K          176
         CAM queue            5           11K         1528
              cred           36            9K          234
         bmsafemap            4            9K           40
              UART           12            9K           12
           devstat            4            9K            4
              ksem            1            8K            1
               rpc            2            8K            2
             shmfd            1            8K            1
       pfs_vncache            1            8K            1
         pfs_nodes           20            8K           20
     audit_evclass          237            8K          296
         taskqueue           63            7K           63
            sglist            5            7K            5
           CAM DEV            3            6K          510
       ufs_dirhash           24            5K           24
               UMA          272            5K          272
            dirrem           17            5K           28
            plimit           17            5K          322
                vt           11            5K           11
             ifnet            3            5K            3
           memdesc            1            4K            1
               MCA           32            4K           32
          filedesc            1            4K            1
             evdev            4            4K            4
           acpisem           28            4K           28
             hhook           15            4K           17
       ether_multi           40            4K           50
            diradd           25            4K           36
           lltable           11            4K           11
          pf_ifnet            5            3K            6
         in6_multi           25            3K           25
          terminal           11            3K           11
         sctp_atcl            7            3K         3344
            kqueue           41            3K         1200
           session           20            3K           31
           pwddesc           40            3K         1198
           uidinfo            3            3K            8
         proc-args           63            3K         2138
        local_apic            1            2K            1
           io_apic            1            2K            1
       fpukern_ctx            2            2K            2
         ipsec-saq            2            2K            2
             selfd           27            2K        16425
            Unitno           27            2K           39
           CAM XPT           22            2K          543
             lockf           15            2K           22
               msi           12            2K           12
       ipsecpolicy            2            2K            2
           acpidev           20            2K           20
             clone            9            2K            9
         sctp_stro            1            1K          836
           softdep            1            1K            1
            sahead            1            1K            1
          secasvar            1            1K            1
       vnodemarker            2            1K           10
      NFSD session            1            1K            1
        CAM periph            4            1K          271
            select            7            1K           29
             ipsec            3            1K            3
          indirdep            3            1K            3
             nhops            6            1K            6
         toponodes            6            1K            6
            isadev            6            1K            6
             mount           16            1K           89
          pci_link           10            1K           10
          sctp_ifa            5            1K            6
            crypto            4            1K            4
            ip6ndp            4            1K            5
 encap_export_host           12            1K           12
         sctp_stri            1            1K          836
         newdirblk            4            1K            8
             mkdir            4            1K           16
          in_multi            2            1K            4
              pfil            4            1K            4
              cdev            2            1K            2
    chacha20random            1            1K            1
               osd            7            1K           18
         sctp_atky            9            1K         3762
       inpcbpolicy           10            1K          139
          sctp_ifn            2            1K            6
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
            DEVFSP            4            1K            9
             DEVFS            9            1K           10
          freework            1            1K           26
               mld            2            1K            2
              igmp            2            1K            2
            vnodes            1            1K            1
           CAM SIM            2            1K            2
            feeder            7            1K            7
           tcpfunc            3            1K            3
            CC Mem            3            1K            7
        loginclass            3            1K            7
            prison            6            1K            6
       lkpikmalloc            5            1K            6
        aesni_data            2            1K            2
         cryptodev            2            1K           49
          nexusdev            8            1K            8
            apmdev            1            1K            1
          atkbddev            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1034
          procdesc            1            1K            6
          pmchooks            1            1K            1
         sctp_athm            7            1K         3344
            soname            4            1K         4725
          filecaps            4            1K           66
               tun            3            1K            3
          sctp_vrf            1            1K            1
              vnet            1            1K            1
           entropy            2            1K           35
               pmc            1            1K            1
          acpiintr            1            1K            1
          sctp_map            2            1K         1672
              cpus            2            1K            2
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
          p1003.1b            1            1K            1
            ipcomp            0            0K            0
               esp            0            0K            0
                ah            0            0K            0
            mqdata            0            0K            0
            tcp_do            0            0K            0
           tcp_fsb            0            0K            0
           filemon            0            0K            0
          pf_table            0            0K            0
           pf_rule            0            0K            0
           pf_altq            0            0K            0
           pf_osfp            0            0K            0
           pf_temp            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K            0
         sctp_iter            0            0K            4
         sctp_mvrf            0            0K            0
         sctp_cpal            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_a_it            0            0K            4
         sctp_aadr            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
        madt_table            0            0K            2
          smartpqi            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
               ixl            0            0K            0
      NFSCL diroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        ice-resmgr            0            0K            0
         ice-osdep            0            0K            0
               ice            0            0K            0
              iavf            0            0K            0
             axgbe            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
          xen_intr            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            bounce            0            0K            0
            busdma            0            0K            0
            qpidrv            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
              isci            0            0K            0
      iommu_dmamap            0            0K            0
            DEVFS4            0            0K            0
     hyperv_socket            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
     vm_fictitious            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
          xen_acpi            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
           UMAHash            0            0K            0
            vtfont            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K           15
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K            3
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefile            0            0K            9
          freeblks            0            0K           25
          freefrag            0            0K            1
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
         BACKLIGHT            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
            pvscsi            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
          ktls_ocf            0            0K            0
       AHCI driver            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
      MLX5E_TLS_RX            0            0K            0
        MLX5EEPROM            0            0K            0
         MLX5E_TLS            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
            MLX5EN            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          MLX5DUMP            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
        MLX5EEPROM            0            0K            0
          seq_file            0            0K            0
           lkpiskb            0            0K            0
             radix            0            0K            0
               idr            0            0K            0
          lkpindev            0            0K            0
            lkpifw            0            0K            0
         lkpi80211            0            0K            0
               NLM            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K            3
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
        tcp_hwpace            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
               LRO            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            0
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
           VN POLL            0            0K            0
               agp            0            0K            0
            statfs            0            0K          195
     namei_tracker            0            0K            0
       export_host            0            0K            0
        cl_savebuf            0            0K            4
           nvme_da            0            0K            0
           acpipwr            0            0K            0
         acpi_perf            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
       tcp_log_dev            0            0K            0
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K          523
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            0
               acl            0            0K            0
          mbuf_tag            0            0K           27
              ktls            0            0K            0
               PUC            0            0K            0
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
          tempbuff            0            0K            0
          tempbuff            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
     CAM ccb queue            0            0K            0
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        14046
          ioctlops            0            0K           86
           eventfd            0            0K            0
           Witness            0            0K            0
             stack            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
               mps            0            0K            0
              sbuf            0            0K          288
          mpr_user            0            0K            0
          firmware            0            0K            0
        compressor            0            0K            0
            MPRSAS            0            0K            0
              SWAP            0            0K            0
               mpr            0            0K            0
            mfibuf            0            0K            0
         sysctltmp            0            0K          658
            sysctl            0            0K            3
        md_sectors            0            0K            0
              ekcd            0            0K            0
            dumper            0            0K            0
          sendfile            0            0K            0
              rctl            0            0K            0
           md_disk            0            0K            0
           malodev            0            0K            0
               LED            0            0K            0
          ix_sriov            0            0K            0
             cache            0            0K            0
        aacraidcam            0            0K            0
          kcovinfo            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            1
filedesc_to_leader            0            0K            0
               pwd            0            0K            0
       tty console            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
       aacraid_buf            0            0K            0
            aaccam            0            0K            0
         boottrace            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            XZ_DEC            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         tmpfs dir            0            0K            0
        tmpfs name            0            0K            0
       tmpfs mount            0            0K            0
           NFS FHA            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
db> show uma
              Zone   Size    Used    Free    Requests  Sleeps  Bucket  Total Mem    XFree
   mbuf_jumbo_page   4096    8320    1078       14727       0     254   38494208        0
              mbuf    256    8579    1083       29867       0     254    2473472        0
              pbuf   2624       0     794           0       0       2    2083456        0
          BUF TRIE    144     185   11603         500       0      62    1697472        0
        malloc-384    384    4165       5        4165       0      30    1601280        0
       malloc-4096   4096     370       2        1773       0       2    1523712        0
        malloc-128    128   11636      82       11642       0     126    1499904        0
       UMA Slabs 0    112   10595      19       10595       0     126    1188768        0
         sctp_asoc   2256       1     509         836       0     254    1150560        0
      mbuf_cluster   2048     508       0         508       0     254    1040384        0
         vmem btag     56   16545     126       16545       0     254     933576        0
           sctp_ep   1208       6     504        1254       0     254     616080        0
         FFS inode   1160     499      19         508       0       8     600880        0
             tcpcb   1104       3     508           7       0     254     564144        0
        RADIX NODE    144    3346     205       24612       0      62     511344        0
            socket    960      20     488        2589       0     254     487680        0
         VM OBJECT    264    1418      82       27797       0      30     396000        0
          lkpicurr    168       2    2350           2       0      62     395136        0
            lkpimm    168       1    2327           1       0      62     391104        0
        sctp_raddr    736       1     516         836       0     254     380512        0
      malloc-65536  65536       5       0           5       0       1     327680        0
        256 Bucket   2048     147       5        1024       0       8     311296        0
         malloc-64     64    3866     229        3868       0     254     262080        0
             VNODE    448     529      47         540       0      30     258048        0
         malloc-16     16   14608     392       14668       0     254     240000        0
            DEVCTL   1024       0     220         126       0       0     225280        0
            THREAD   1808     117       4         117       0       8     218768        0
        malloc-256    256     747      63         852       0      62     207360        0
      malloc-65536  65536       3       0           3       0       1     196608        0
        malloc-128    128    1324     195       27484       0     126     194432        0
         MAP ENTRY     96    1580     436       92704       0     126     193536        0
         UMA Zones    768     244       0         244       0      16     187392        0
         malloc-32     32    5323     347        5332       0     254     181440        0
       malloc-2048   2048      14      74        2536       0       8     180224        0
       FFS2 dinode    256     499      71         508       0      62     145920        0
      malloc-65536  65536       0       2          46       0       1     131072        0
      malloc-65536  65536       0       2         144       0       1     131072        0
       malloc-1024   1024     116      12         282       0      16     131072        0
             unpcb    256       7     503        1179       0     254     130560        0
       mbuf_packet    256       2     506        2928       0     254     130048        0
       S VFS Cache    104     966     204        1007       0     126     121680        0
     FPU_save_area    832     119      25         132       0      16     119808        0
          ksiginfo    112      39    1005          55       0     126     116928        0
        malloc-128    128     564     211        3870       0     126      99200        0
        malloc-128    128     626     149        1164       0     126      99200        0
        malloc-256    256     299      76        2173       0      62      96000        0
          UMA Kegs    384     230       3         230       0      30      89472        0
        128 Bucket   1024      45      38         478       0      16      84992        0
         malloc-64     64     771     552       19093       0     254      84672        0
           VMSPACE   2552      24       9        1182       0       4      84216        0
        sctp_readq    152       1     519         418       0     254      79040        0
        sctp_chunk    152       2     518        1672       0     254      79040        0
         filedesc0   1072      41      29        1198       0       8      75040        0
       malloc-8192   8192       9       0           9       0       1      73728        0
       malloc-8192   8192       7       2         110       0       1      73728        0
         64 Bucket    512      65      71        1869       0      30      69632        0
         malloc-64     64     570     501        4507       0     254      68544        0
      malloc-65536  65536       1       0           1       0       1      65536        0
      malloc-32768  32768       0       2         120       0       1      65536        0
      malloc-32768  32768       2       0           2       0       1      65536        0
      malloc-16384  16384       4       0           4       0       1      65536        0
       malloc-4096   4096      15       1          26       0       2      65536        0
       malloc-1024   1024      21      43         546       0      16      65536        0
        malloc-256    256     191      64         847       0      62      65280        0
              PROC   1352      40       8        1197       0       8      64896        0
             g_bio    408       0     150        4518       0      30      61200        0
sctp_stream_msg_out    112       0     540         799       0     254      60480        0
         udp_inpcb    424       6     120         128       0      30      53424        0
         32 Bucket    256      67     128        6988       0      62      49920        0
           DIRHASH   1024      35      13          35       0      16      49152        0
             NAMEI   1024       0      48       12227       0      16      49152        0
      malloc-16384  16384       1       2         161       0       1      49152        0
       malloc-4096   4096      10       2         553       0       2      49152        0
       malloc-1024   1024      19      29         887       0      16      49152        0
       malloc-1024   1024       3      45         842       0      16      49152        0
        malloc-384    384      58      62        3735       0      30      46080        0
          syncache    168       0     264           5       0     254      44352        0
            clpbuf   2624       0      16          22       0      16      41984        0
       Mountpoints   2752       2      12           2       0       4      38528        0
            pcpu-8      8    4221     387        4249       0     254      36864        0
         malloc-64     64      32     535       13602       0     254      36288        0
         malloc-64     64      26     541          42       0     254      36288        0
         malloc-64     64     116     451         141       0     254      36288        0
         malloc-64     64      10     557          42       0     254      36288        0
         malloc-64     64      48     519        1206       0     254      36288        0
        malloc-128    128      28     251         157       0     126      35712        0
        malloc-128    128       1     278          11       0     126      35712        0
        malloc-128    128      95     184         492       0     126      35712        0
        malloc-128    128      20     259          25       0     126      35712        0
     routing nhops    256      10     125          17       0      62      34560        0
           ttyoutq    256      72      63         160       0      62      34560        0
        malloc-384    384      47      43          89       0      30      34560        0
        malloc-256    256      23     112         863       0      62      34560        0
        malloc-256    256       2     133          52       0      62      34560        0
        malloc-256    256       6     129         288       0      62      34560        0
        malloc-256    256      70      65         476       0      62      34560        0
        malloc-256    256      35     100          40       0      62      34560        0
      malloc-32768  32768       1       0           1       0       1      32768        0
      malloc-16384  16384       2       0          17       0       1      32768        0
       malloc-8192   8192       4       0           4       0       1      32768        0
       malloc-2048   2048      11       5          12       0       8      32768        0
       malloc-2048   2048       1      15          25       0       8      32768        0
       malloc-2048   2048       2      14          11       0       8      32768        0
       malloc-1024   1024       2      30          45       0      16      32768        0
       malloc-1024   1024       4      28           5       0      16      32768        0
       malloc-1024   1024       3      29           7       0      16      32768        0
        malloc-512    512       0      64         118       0      30      32768        0
        malloc-512    512       2      62           2       0      30      32768        0
        malloc-512    512       3      61          11       0      30      32768        0
        malloc-512    512       0      64           9       0      30      32768        0
        malloc-512    512      10      54          60       0      30      32768        0
        malloc-512    512       1      63         837       0      30      32768        0
        malloc-512    512       3      61           3       0      30      32768        0
           pcpu-64     64     493      19         493       0     254      32768        0
            ttyinq    160     135      65         300       0      62      32000        0
            cpuset    104       7     272           7       0     126      29016        0
        sctp_laddr     48       2     586         840       0     254      28224        0
         malloc-32     32     383     499        6565       0     254      28224        0
         16 Bucket    144      50     146         241       0      62      28224        0
          4 Bucket     48       6     582          58       0     254      28224        0
         tcp_inpcb    424       3      60           7       0      30      26712        0
            da_ccb    544       0      49        1250       0      16      26656        0
              pipe    744       7      28         284       0      16      26040        0
         TURNSTILE    136     122      67         122       0      62      25704        0
       malloc-8192   8192       3       0           3       0       1      24576        0
       malloc-4096   4096       6       0           6       0       2      24576        0
           rtentry    176      13     125          17       0      62      24288        0
              PGRP     88      20     256          31       0     126      24288        0
    ertt_txseginfo     40       0     606         662       0     254      24240        0
          rl_entry     40      32     574          32       0     254      24240        0
             Files     80      73     227        7834       0     126      24000        0
          8 Bucket     80      40     260         289       0     126      24000        0
        malloc-384    384      10      50          13       0      30      23040        0
        malloc-384    384      54       6          64       0      30      23040        0
        SLEEPQUEUE     88     122     134         122       0     126      22528        0
         hostcache     64       1     314           1       0     254      20160        0
             udpcb     32       6     624         128       0     254      20160        0
   udp_inpcb ports     32       3     627          40       0     254      20160        0
              ertt     72       3     277           7       0     126      20160        0
               PWD     32      10     620         100       0     254      20160        0
         malloc-32     32      35     595         613       0     254      20160        0
         malloc-32     32       7     623          60       0     254      20160        0
         malloc-32     32     133     497        2646       0     254      20160        0
         malloc-32     32      31     599          56       0     254      20160        0
         malloc-32     32      29     601          31       0     254      20160        0
         malloc-32     32       7     623          10       0     254      20160        0
          2 Bucket     32      42     588         288       0     254      20160        0
 epoch_record pcpu    256       4      60           4       0      62      16384        0
       malloc-8192   8192       0       2          28       0       1      16384        0
       malloc-8192   8192       2       0           2       0       1      16384        0
       malloc-4096   4096       1       3         197       0       2      16384        0
       malloc-2048   2048       0       8          12       0       8      16384        0
       malloc-2048   2048       3       5           3       0       8      16384        0
       malloc-2048   2048       2       6           2       0       8      16384        0
       malloc-2048   2048       3       5           3       0       8      16384        0
       malloc-1024   1024       1      15           1       0      16      16384        0
           SMR CPU     32       7     504           7       0     254      16352        0
         malloc-16     16     510     490        3505       0     254      16000        0
              kenv    258      15      45        1037       0      30      15480        0
             ripcb    424       1      35           4       0      30      15264        0
            mqnode    416       3      33           3       0      30      14976        0
              vmem   1856       1       7           1       0       8      14848        0
        SMR SHARED     24       7     504           7       0     254      12264        0
   tcp_inpcb ports     32       1     377           1       0     254      12096        0
      vtnet_tx_hdr     24       0     501        2426       0     254      12024        0
             KNOTE    160       0      75           8       0      62      12000        0
         malloc-16     16      46     704        1356       0     254      12000        0
         malloc-16     16       9     741          14       0     254      12000        0
         malloc-16     16      16     734          65       0     254      12000        0
         malloc-16     16      51     699       28164       0     254      12000        0
         malloc-16     16      13     737        3352       0     254      12000        0
        malloc-384    384       0      30           1       0      30      11520        0
        malloc-384    384      11      19          11       0      30      11520        0
       malloc-8192   8192       1       0           1       0       1       8192        0
       malloc-4096   4096       0       2           2       0       2       8192        0
       malloc-4096   4096       0       2           1       0       2       8192        0
           pcpu-16     16      14     498          14       0     254       8192        0
         malloc-16     16       0     500           2       0     254       8000        0
       UMA Slabs 1    176       8      14           8       0      62       3872        0
        KMAP ENTRY     96      12      27          14       0       0       3744        0
       FFS1 dinode    128       0       0           0       0     126          0        0
           ada_ccb    272       0       0           0       0      30          0        0
             swblk    136       0       0           0       0      62          0        0
          swpctrie    144       0       0           0       0      62          0        0
   cdg_qdiffsample     16       0       0           0       0     254          0        0
   pf state scrubs     40       0       0           0       0     254          0        0
   pf frag entries     40       0       0           0       0     254          0        0
          pf frags    248       0       0           0       0      62          0        0
  pf table entries    160       0       0           0       0     254          0        0
pf table entry counters     64       0       0           0       0     254          0        0
   pf source nodes    136       0       0           0       0     254          0        0
     pf state keys     88       0       0           0       0     126          0        0
         pf states    312       0       0           0       0     254          0        0
           pf tags    104       0       0           0       0     126          0        0
          pf mtags     56       0       0           0       0     254          0        0
       tcp_bbr_pcb    832       0       0           0       0      16          0        0
       tcp_bbr_map    128       0       0           0       0     126          0        0
      tcp_rack_pcb    896       0       0           0       0      16          0        0
      tcp_rack_map    120       0       0           0       0     126          0        0
               ipq     56       0       0           0       0     254          0        0
tfo_ccache_entries     80       0       0           0       0     126          0        0
               tfo      4       0       0           0       0     254          0        0
          sackhole     32       0       0           0       0     254          0        0
             tcptw     72       0       0           0       0     254          0        0
   sctp_asconf_ack     48       0       0           0       0     254          0        0
       sctp_asconf     40       0       0           0       0     254          0        0
   tcp_log_id_node    120       0       0           0       0     126          0        0
 tcp_log_id_bucket    176       0       0           0       0      62          0        0
           tcp_log    416       0       0           0       0     254          0        0
          tcpreass     48       0       0           0       0     254          0        0
       ripcb ports     32       0       0           0       0     254          0        0
udplite_inpcb ports     32       0       0           0       0     254          0        0
     udplite_inpcb    424       0       0           0       0      30          0        0
    IPsec SA lft_c     16       0       0           0       0     254          0        0
            itimer    352       0       0           0       0      30          0        0
            AIOLIO    272       0       0           0       0      30          0        0
             AIOCB    552       0       0           0       0      16          0        0
              AIOP     32       0       0           0       0     254          0        0
               AIO    208       0       0           0       0      62          0        0
        mqnotifier    216       0       0           0       0      62          0        0
            mvdata     64       0       0           0       0     254          0        0
            mqueue    248       0       0           0       0      62          0        0
        TMPFS node    224       0       0           0       0      62          0        0
           NCLNODE    608       0       0           0       0      16          0        0
     LTS VFS Cache    360       0       0           0       0      30          0        0
       L VFS Cache    320       0       0           0       0      30          0        0
     STS VFS Cache    144       0       0           0       0      62          0        0
           cryptop    280       0       0           0       0      30          0        0
  linux_dma_object     32       0       0           0       0     254          0        0
  linux_dma_pctrie    144       0       0           0       0      62          0        0
   IOMMU_MAP_ENTRY    120       0       0           0       0     126          0        0
    mbuf_jumbo_16k  16384       0       0           0       0     254          0        0
     mbuf_jumbo_9k   9216       0       0           0       0     254          0        0
      audit_record   1280       0       0           0       0       8          0        0
         domainset     40       0       0           0       0     254          0        0
        MAC labels     40       0       0           0       0     254          0        0
            vnpbuf   2624       0       0           0       0      64          0        0
            mdpbuf   2624       0       0           0       0       3          0        0
           nfspbuf   2624       0       0           0       0      16          0        0
            swwbuf   2624       0       0           0       0       8          0        0
            swrbuf   2624       0       0           0       0      16          0        0
          umtx_shm     88       0       0           0       0     126          0        0
           umtx pi     96       0       0           0       0     126          0        0
rangeset pctrie nodes    144       0       0           0       0      62          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-65536  65536       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-32768  32768       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
      malloc-16384  16384       0       0           0       0       1          0        0
       malloc-8192   8192       0       0           0       0       1          0        0
       malloc-4096   4096       0       0           0       0       2          0        0
        malloc-512    512       0       0           0       0      30          0        0
        malloc-384    384       0       0           0       0      30          0        0
           pcpu-32     32       0       0           0       0     254          0        0
            pcpu-4      4       0       0           0       0     254          0        0
            fakepg    104       0       0           0       0     126          0        0
          UMA Hash    256       0       0           0       0      62          0        0

Crashes (17):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Title
ci-freebsd-main 2022/04/21 05:22 freebsd-src b91a48693a53 7c337266 console log report syz C panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/02/18 14:57 freebsd-src 2ed689a674c3 14052202 console log report syz panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/02/14 22:43 freebsd-src 25c6318c7906 98682e5e console log report syz panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2020/12/11 20:59 freebsd 519a1a7fed29 ba24ffcd console log report syz
ci-freebsd-main 2020/11/07 16:41 freebsd c4f7f2c1ece8 64069d48 console log report syz
ci-freebsd-i386 2020/11/07 17:03 freebsd 2c7fa4443f7f 64069d48 console log report syz
ci-freebsd-main 2022/04/21 04:59 freebsd-src b91a48693a53 7c337266 console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/11/06 18:47 freebsd-src acdfc0963946 4c1be0be console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/07/05 01:08 freebsd-src 4060e77f49d1 55aa55c2 console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/04/15 01:20 freebsd-src 8e84712d01eb fcdb12ba console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/02/18 14:37 freebsd-src 2ed689a674c3 14052202 console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2021/02/14 22:23 freebsd-src 25c6318c7906 98682e5e console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-i386 2021/02/18 23:48 freebsd-src b9cbc85d7272 14052202 console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-i386 2021/02/10 09:59 freebsd-src f44e67d120ad 9c8b8541 console log report panic: Queues are not empty when handling SHUTDOWN-COMPLETE
ci-freebsd-main 2020/12/11 20:44 freebsd 519a1a7fed29 ba24ffcd console log report
ci-freebsd-main 2020/11/27 22:50 freebsd 219284e64fa7 486f93ef console log report
ci-freebsd-main 2020/11/07 16:22 freebsd c4f7f2c1ece8 64069d48 console log report
* Struck through repros no longer work on HEAD.