syzbot


panic: mutex pcbinfohash not owned at /syzkaller/managers/main/kernel/sys/netinet6/in6_pcb.c:LINE

Status: fixed on 2019/11/11 15:39
Reported-by: syzbot+0b0488ca537e20cb2429@syzkaller.appspotmail.com
Fix commit: c17cd08f5302 It is unclear why in6_pcblookup_local() would require write access to the PCB hash. The function doesn't modify the hash. It always asserted write lock historically, but with epoch conversion this fails in some special cases.
First crash: 1623d, last: 1622d

Sample crash report:
login: panic: mutex pcbinfohash not owned at /syzkaller/managers/main/kernel/sys/netinet6/in6_pcb.c:717
cpuid = 0
time = 1573258495
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x47/frame 0xfffffe00242b5260
vpanic() at vpanic+0x1c7/frame 0xfffffe00242b52d0
panic() at panic+0x43/frame 0xfffffe00242b5330
__mtx_assert() at __mtx_assert+0x18b/frame 0xfffffe00242b5370
in6_pcblookup_local() at in6_pcblookup_local+0x53/frame 0xfffffe00242b53c0
in_pcb_lport() at in_pcb_lport+0x3fd/frame 0xfffffe00242b5450
in_pcbbind_setup() at in_pcbbind_setup+0x28b/frame 0xfffffe00242b54f0
in_pcbconnect_setup() at in_pcbconnect_setup+0x4aa/frame 0xfffffe00242b55b0
udp_send() at udp_send+0xee4/frame 0xfffffe00242b56c0
udp6_send() at udp6_send+0x4e8/frame 0xfffffe00242b5870
sosend_dgram() at sosend_dgram+0x54f/frame 0xfffffe00242b58e0
sosend() at sosend+0xc6/frame 0xfffffe00242b5950
kern_sendit() at kern_sendit+0x32d/frame 0xfffffe00242b5a00
sendit() at sendit+0x226/frame 0xfffffe00242b5a60
sys_sendto() at sys_sendto+0x5c/frame 0xfffffe00242b5ac0
amd64_syscall() at amd64_syscall+0x473/frame 0xfffffe00242b5bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00242b5bf0
--- syscall (0, FreeBSD ELF64, nosys), rip = 0x41c3aa, rsp = 0x7fffffffead8, rbp = 0x7fffffffeb40 ---
KDB: enter: panic
[ thread pid 759 tid 100101 ]
Stopped at      kdb_enter+0x67: movq    $0,0x14698f6(%rip)
db> 
db> set $lines = 0
db> set $maxwidth = 0
db> show registers
cs                        0x20
ds                        0x3b  ll+0x1a
es                        0x3b  ll+0x1a
fs                        0x13
gs                        0x1b
ss                        0x28  ll+0x7
rax                       0x12
rcx                       0x80  ll+0x5f
rdx         0xffffffff81875593
rbx                          0
rsp         0xfffffe00242b5240
rbp         0xfffffe00242b5260
rsi                          0
rdi                          0
r8                           0
r9                  0xffffffff
r10                 0x7d64a6a7
r11                 0x233205a0
r12         0xffffffff82068cf0  ddb_dbbe
r13                          0
r14         0xffffffff81912c59
r15         0xffffffff81912c59
rip         0xffffffff810a9cf7  kdb_enter+0x67
rflags                    0x86  ll+0x65
kdb_enter+0x67: movq    $0,0x14698f6(%rip)
db> show proc
Process 759 (syz-executor3952699) at 0xfffff80036545000:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 757 at 0xfffff80036545530
 ABI: FreeBSD ELF64
 arguments: ./syz-executor395269985
 reaper: 0xfffff800031f9530 reapsubtree: 1
 sigparent: 20
 vmspace: 0xfffff800366be000
   (map 0xfffff800366be000)
   (map.pmap 0xfffff800366be0d0)
   (pmap 0xfffff800366be130)
 threads: 1
100101                   Run     CPU 0                       syz-executor3952699
db> ps
  pid  ppid  pgrp   uid  state   wmesg   wchan               cmd
  759   757   757     0  R       CPU 0                       syz-executor3952699
  757   755   757     0  Ss      pause   0xfffff800365455d8  csh
  755   668   755     0  Rs      CPU 1                       sshd
  734     1   734     0  Ss+     ttyin   0xfffff800036e40b0  getty
  733     1   733     0  Ss+     ttyin   0xfffff80003a040b0  getty
  732     1   732     0  Ss+     ttyin   0xfffff80003a048b0  getty
  731     1   731     0  Ss+     ttyin   0xfffff80003a050b0  getty
  730     1   730     0  Ss+     ttyin   0xfffff80003a058b0  getty
  729     1   729     0  Ss+     ttyin   0xfffff80003a060b0  getty
  728     1   728     0  Ss+     ttyin   0xfffff80003a068b0  getty
  727     1   727     0  Ss+     ttyin   0xfffff80003a070b0  getty
  726     1   726     0  Ss+     ttyin   0xfffff800036fb0b0  getty
  724     1    22     0  S+      piperd  0xfffff800362808e8  logger
  723   722    22     0  S+      nanslp  0xffffffff824fc1e1  sleep
  722     1    22     0  S+      wait    0xfffff80003b5d000  sh
  672     1   672     0  Ss      nanslp  0xffffffff824fc1e1  cron
  668     1   668     0  Ss      select  0xfffff800363f90c0  sshd
  481     1   481     0  Ss      select  0xfffff80003074440  syslogd
  410     1   410     0  Ss      select  0xfffff800039febc0  devd
  409     1   409    65  Ss      select  0xfffff800030746c0  dhclient
  324     1   324     0  Ss      select  0xfffff800039fed40  dhclient
  321     1   321     0  Ss      select  0xfffff800039fee40  dhclient
   21     0     0     0  DL      syncer  0xffffffff825d26b0  [syncer]
   20     0     0     0  DL      vlruwt  0xfffff80003429000  [vnlru]
   19     0     0     0  DL      (threaded)                  [bufdaemon]
100063                   D       qsleep  0xffffffff825d1b58  [bufdaemon]
100065                   D       -       0xffffffff8200a900  [bufspacedaemon-0]
100078                   D       sdflush 0xfffff800039024e8  [/ worker]
   18     0     0     0  DL      psleep  0xffffffff825ed008  [vmdaemon]
   17     0     0     0  DL      (threaded)                  [pagedaemon]
100061                   D       psleep  0xffffffff82618d98  [dom0]
100067                   D       launds  0xffffffff82618da4  [laundry: dom0]
100068                   D       umarcl  0xffffffff81529b80  [uma]
   16     0     0     0  DL      -       0xffffffff82357e20  [rand_harvestq]
   15     0     0     0  DL      waiting 0xffffffff8265e370  [sctp_iterator]
    9     0     0     0  DL      -       0xffffffff825d155c  [soaiod4]
    8     0     0     0  DL      -       0xffffffff825d155c  [soaiod3]
    7     0     0     0  DL      -       0xffffffff825d155c  [soaiod2]
    6     0     0     0  DL      -       0xffffffff825d155c  [soaiod1]
    5     0     0     0  DL      (threaded)                  [cam]
100031                   D       -       0xffffffff822331c0  [doneq0]
100060                   D       -       0xffffffff82233088  [scanner]
    4     0     0     0  DL      crypto_ 0xfffff800031dbc90  [crypto returns 1]
    3     0     0     0  DL      crypto_ 0xfffff800031dbc30  [crypto returns 0]
    2     0     0     0  DL      crypto_ 0xffffffff825e7648  [crypto]
   14     0     0     0  DL      seqstat 0xfffff80003254888  [sequencer 00]
   13     0     0     0  DL      (threaded)                  [geom]
100022                   D       -       0xffffffff82617390  [g_event]
100023                   D       -       0xffffffff826173a0  [g_up]
100024                   D       -       0xffffffff82617398  [g_down]
   12     0     0     0  WL      (threaded)                  [intr]
100005                   I                                   [swi6: Giant taskq]
100007                   I                                   [swi5: fast taskq]
100011                   I                                   [swi6: task queue]
100017                   I                                   [swi4: clock (0)]
100018                   I                                   [swi4: clock (1)]
100019                   I                                   [swi3: vm]
100020                   I                                   [swi1: netisr 0]
100032                   I                                   [irq24: virtio_pci0]
100033                   I                                   [irq25: virtio_pci0]
100034                   I                                   [irq26: virtio_pci0]
100035                   I                                   [irq27: virtio_pci0]
100036                   I                                   [irq28: virtio_pci1]
100037                   I                                   [irq29: virtio_pci1]
100038                   I                                   [irq30: virtio_pci1]
100039                   I                                   [irq31: virtio_pci1]
100040                   I                                   [irq32: virtio_pci1]
100045                   I                                   [irq1: atkbd0]
100046                   I                                   [irq12: psm0]
100047                   I                                   [swi0: uart uart++]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
    1     0     1     0  SLs     wait    0xfffff800031f9530  [init]
   10     0     0     0  DL      audit_w 0xffffffff8265f000  [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin  0xffffffff82606b68  [swapper]
100006                   D       -       0xfffff8000322ce00  [thread taskq]
100008                   D       -       0xfffff8000322cc00  [config_0]
100009                   D       -       0xfffff8000322cb00  [kqueue_ctx taskq]
100010                   D       -       0xfffff8000322ca00  [aiod_kick taskq]
100012                   D       -       0xfffff8000322c800  [softirq_0]
100013                   D       -       0xfffff8000322c700  [softirq_1]
100014                   D       -       0xfffff8000322c600  [if_io_tqg_0]
100015                   D       -       0xfffff8000322c500  [if_io_tqg_1]
100016                   D       -       0xfffff8000322c400  [if_config_tqg_0]
100021                   D       -       0xfffff8000322c300  [firmware taskq]
100026                   D       -       0xfffff8000322c200  [crypto_0]
100027                   D       -       0xfffff8000322c200  [crypto_1]
100041                   D       -       0xfffff8000322bd00  [vtnet0 rxq 0]
100042                   D       -       0xfffff8000322bc00  [vtnet0 txq 0]
100043                   D       -       0xfffff8000322bb00  [vtnet0 rxq 1]
100044                   D       -       0xfffff8000322ba00  [vtnet0 txq 1]
100048                   D       -       0xfffff8000322b900  [mca taskq]
100051                   D       -       0xffffffff824faf61  [deadlkres]
100055                   D       -       0xfffff800039d9100  [acpi_task_0]
100056                   D       -       0xfffff800039d9100  [acpi_task_1]
100057                   D       -       0xfffff800039d9100  [acpi_task_2]
100059                   D       -       0xfffff8000322c100  [CAM taskq]
db> show all locks
Process 759 (syz-executor3952699) thread 0xfffff8003658d6e0 (100101)
exclusive rw udpinp (udpinp) r = 0 (0xfffff80003d3ad78) locked @ /syzkaller/managers/main/kernel/sys/netinet/udp_usrreq.c:1125
db> show malloc
              Type        InUse        MemUse     Requests
            devbuf         4200         4764K         4224
             vtbuf           24         1968K           46
           callout            3         1672K            3
              kobj          332         1328K          488
            newblk          506         1151K          524
          vfscache            4         1025K            4
               pcb           21          537K           75
          inodedep           48          536K           71
         ufs_quota            1          512K            1
          vfs_hash            1          512K            1
              intr            4          388K            4
           subproc          108          208K          824
            acpica         1674          185K        47809
         vnet_data            1          168K            1
           pagedep           14          132K           18
        tfo_ccache            1          128K            1
         sysctloid         2038          107K         2098
               sem            4          106K            4
            DEVFS1          101          101K          110
               bus          948           77K         2897
            linker          205           76K          219
          mtx_pool            2           72K            2
          syncache            1           68K            1
           UMAHash            2           65K            2
          acpitask            1           64K            1
       ddb_capture            1           64K            1
            module          493           62K          494
        gtaskqueue           22           34K           22
         hostcache            1           32K            1
               shm            1           32K            1
               msg            4           30K            4
            DEVFS3          120           30K          130
              umtx          212           27K          212
        DEVFS_RULE           56           27K           56
           kdtrace          136           26K         1568
            kbdmux            6           22K            6
              vmem            3           19K            4
               BPF           10           18K           10
              temp           17           17K         1489
         ufs_mount            3           17K            4
              proc            3           17K            3
               tty           16           16K           16
           tidhash            1           16K            1
           ithread           87           15K           87
            ifaddr           32           13K           32
            bus-sc           26           13K         1128
            KTRACE          100           13K          100
              kenv           95           12K           99
      eventhandler          122           11K          122
         pfs_nodes           20           10K           20
              GEOM           60           10K          487
              rman           77            9K          418
         bmsafemap            4            9K           41
           devstat            4            9K            4
              UART           12            9K           12
               rpc            2            8K            2
             shmfd            1            8K            1
       pfs_vncache            1            8K            1
     audit_evclass          230            8K          288
              cred           27            7K          233
           CAM DEV            3            6K          508
                vt           11            6K           11
            sglist            5            6K            5
         CAM queue            5            6K         1522
       ufs_dirhash           24            5K           24
         taskqueue           42            5K           42
            plimit           17            5K          322
             ifnet            3            5K            3
           memdesc            1            4K            1
               MCA           32            4K           32
             evdev            4            4K            4
          filedesc            1            4K            1
          routetbl           28            4K           32
           lltable           11            4K           11
             hhook           13            4K           13
       ether_multi           40            4K           45
            diradd           25            4K           36
         in6_multi           25            3K           25
          terminal           11            3K           11
            kqueue           44            3K          762
           acpisem           20            3K           20
            select           20            3K           20
           session           20            3K           31
              pgrp           20            3K           31
           uidinfo            4            3K            4
            dirrem           17            3K           28
        local_apic            1            2K            1
           io_apic            1            2K            1
         ipsec-saq            2            2K            2
         proc-args           39            2K          460
           CAM XPT           22            2K          542
             lockf           15            2K           22
            Unitno           25            2K           37
           acpidev           20            2K           20
            crypto            2            2K            2
               msi            9            2K            9
       ipsecpolicy            1            1K            1
            sahead            1            1K            1
          secasvar            1            1K            1
             clone            8            1K            8
              cdev            4            1K            4
      NFSD session            1            1K            1
        CAM periph            4            1K          270
          indirdep            3            1K            3
         toponodes            6            1K            6
            isadev            6            1K            6
             mount           16            1K           86
          pci_link           10            1K           10
            ip6ndp            4            1K            5
          sctp_ifa            5            1K            5
           CAM SIM            2            1K            2
           softdep            1            1K            1
             mkdir            4            1K           16
          in_multi            2            1K            3
    chacha20random            1            1K            1
             epoch            4            1K            4
 encap_export_host            8            1K            8
              pfil            3            1K            3
               osd            3            1K            9
         newdirblk            4            1K            8
               mld            2            1K            2
          sctp_ifn            2            1K            2
              igmp            2            1K            2
            vnodes            1            1K            1
      NFSD lckfile            1            1K            1
     NFSD V4client            1            1K            1
             DEVFS            9            1K           10
       inpcbpolicy            7            1K          132
            feeder            7            1K            7
        loginclass            3            1K            3
            soname            5            1K         5489
            apmdev            1            1K            1
          atkbddev            2            1K            2
          pmchooks            1            1K            1
            prison            4            1K            4
            DEVFSP            2            1K            2
     CAM dev queue            2            1K            2
 CAM I/O Scheduler            1            1K            1
          CAM path            4            1K         1030
          filecaps            4            1K           63
               tun            3            1K            3
          nexusdev            5            1K            5
           tcpfunc            1            1K            1
          sctp_vrf            1            1K            1
              vnet            1            1K            1
          acpiintr            1            1K            1
               pmc            1            1K            1
              cpus            2            1K            2
          freework            1            1K           26
    vnet_data_free            1            1K            1
           Per-cpu            1            1K            1
           entropy            1            1K           34
          p1003.1b            1            1K            1
          ppbusdev            0            0K            0
agtiapi_MemAlloc malloc            0            0K            0
    osti_cacheable            0            0K            0
        madt_table            0            0K            2
          tempbuff            0            0K            0
          tempbuff            0            0K            0
          smartpqi            0            0K            0
ag_tgt_map_t malloc            0            0K            0
ag_slr_map_t malloc            0            0K            0
lDevFlags * malloc            0            0K            0
tiDeviceHandle_t * malloc            0            0K            0
ag_portal_data_t malloc            0            0K            0
ag_device_t malloc            0            0K            0
     STLock malloc            0            0K            0
          CCB List            0            0K            0
              iavf            0            0K            0
               ixl            0            0K            0
            sr_iov            0            0K            0
               OCS            0            0K            0
               OCS            0            0K            0
              nvme            0            0K            0
               nvd            0            0K            0
            netmap            0            0K            0
            mwldev            0            0K            0
        MVS driver            0            0K            0
       fpukern_ctx            0            0K            0
          xen_intr            0            0K            0
     CAM ccb queue            0            0K            0
           xen_hvm            0            0K            0
         legacydrv            0            0K            0
            qpidrv            0            0K            0
          mrsasbuf            0            0K            0
          mpt_user            0            0K            0
      dmar_idpgtbl            0            0K            0
          dmar_dom            0            0K            0
          dmar_ctx            0            0K            0
       dmar_dmamap            0            0K            0
          mps_user            0            0K            0
            MPSSAS            0            0K            0
              isci            0            0K            0
           bxe_ilt            0            0K            0
            xenbus            0            0K            0
     vm_fictitious            0            0K            0
               mps            0            0K            0
          mpr_user            0            0K            0
            MPRSAS            0            0K            0
         vm_pgdata            0            0K            0
           jblocks            0            0K            0
          savedino            0            0K           13
          sentinel            0            0K            0
            jfsync            0            0K            0
            jtrunc            0            0K            0
             sbdep            0            0K            2
           jsegdep            0            0K            0
              jseg            0            0K            0
         jfreefrag            0            0K            0
          jfreeblk            0            0K            0
           jnewblk            0            0K            0
            jmvref            0            0K            0
           jremref            0            0K            0
           jaddref            0            0K            0
           freedep            0            0K            0
          freefile            0            0K            9
          freeblks            0            0K           25
          freefrag            0            0K            7
        allocindir            0            0K            0
       allocdirect            0            0K            0
          ufs_trim            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_pipeent            0            0K            0
        audit_pipe            0            0K            0
      audit_evname            0            0K            0
         audit_bsm            0            0K            0
      audit_gidset            0            0K            0
        audit_text            0            0K            0
        audit_path            0            0K            0
        audit_data            0            0K            0
        audit_cred            0            0K            0
             xform            0            0K            0
               NLM            0            0K            0
 nfsclient_nlminfo            0            0K            0
    nfsclient_lock            0            0K            0
           NFS FHA            0            0K            0
    ipsec-spdcache            0            0K            0
         ipsec-reg            0            0K            0
        ipsec-misc            0            0K            0
      ipsecrequest            0            0K            0
            ip6opt            0            0K            3
       ip6_msource            0            0K            0
      ip6_moptions            0            0K            0
       in6_mfilter            0            0K            0
             frag6            0            0K            0
            tcplog            0            0K            0
               LRO            0            0K            0
        sctp_mcore            0            0K            0
        sctp_socko            0            0K            0
         sctp_iter            0            0K            4
         sctp_mvrf            0            0K            0
         sctp_timw            0            0K            0
         sctp_cpal            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_stre            0            0K            0
         sctp_athi            0            0K            0
         sctp_athm            0            0K            0
         sctp_atky            0            0K            0
         sctp_atcl            0            0K            0
         sctp_a_it            0            0K            4
         sctp_aadr            0            0K            0
         sctp_stro            0            0K            0
         sctp_stri            0            0K            0
          sctp_map            0            0K            0
      newreno data            0            0K            0
        ip_msource            0            0K            0
       ip_moptions            0            0K            0
        in_mfilter            0            0K            0
              ipid            0            0K            0
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
       80211nodeie            0            0K            0
         80211node            0            0K            0
      80211mesh_gt            0            0K            0
      80211mesh_rt            0            0K            0
         80211perr            0            0K            0
         80211prep            0            0K            0
         80211preq            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
             iflib            0            0K            0
              vlan            0            0K            0
               gif            0            0K            0
           ifdescr            0            0K            0
              zlib            0            0K            0
           fadvise            0            0K            0
       vnodemarker            0            0K            4
               mpr            0            0K            0
            statfs            0            0K          195
       export_host            0            0K            0
        cl_savebuf            0            0K            3
            biobuf            0            0K            0
              aios            0            0K            0
               lio            0            0K            0
               acl            0            0K            0
            mfibuf            0            0K            0
          mbuf_tag            0            0K           25
              accf            0            0K            0
               pts            0            0K            0
               iov            0            0K        12739
          ioctlops            0            0K           84
           Witness            0            0K            0
             stack            0            0K            0
        md_sectors            0            0K            0
              sbuf            0            0K          364
           md_disk            0            0K            0
        compressor            0            0K            0
           malodev            0            0K            0
              SWAP            0            0K            0
               LED            0            0K            0
         sysctltmp            0            0K          545
            sysctl            0            0K            1
              ekcd            0            0K            0
            dumper            0            0K            0
              rctl            0            0K            0
          ix_sriov            0            0K            0
        aacraidcam            0            0K            0
                ix            0            0K            0
            ipsbuf            0            0K            0
            iirbuf            0            0K            0
             cache            0            0K            0
       aacraid_buf            0            0K            0
          kcovinfo            0            0K            0
      prison_racct            0            0K            0
       Fail Points            0            0K            0
             sigio            0            0K            1
filedesc_to_leader            0            0K            0
       tty console            0            0K            0
            aaccam            0            0K            0
            aacbuf            0            0K            0
              zstd            0            0K            0
            nvlist            0            0K            0
          SCSI ENC            0            0K            0
           SCSI sa            0            0K            0
        isofs_node            0            0K            0
       isofs_mount            0            0K            0
     tr_raid5_data            0            0K            0
    tr_raid1e_data            0            0K            0
     tr_raid1_data            0            0K            0
     tr_raid0_data            0            0K            0
    tr_concat_data            0            0K            0
       md_sii_data            0            0K            0
   md_promise_data            0            0K            0
    md_nvidia_data            0            0K            0
   md_jmicron_data            0            0K            0
     md_intel_data            0            0K            0
       md_ddf_data            0            0K            0
         raid_data            0            0K           72
     geom_flashmap            0            0K            0
         newnfsmnt            0            0K            0
  newnfsclient_req            0            0K            0
   NFSCL layrecall            0            0K            0
     NFSCL session            0            0K            0
     NFSCL sockreq            0            0K            0
     NFSCL devinfo            0            0K            0
     NFSCL flayout            0            0K            0
      NFSCL layout            0            0K            0
     NFSD rollback            0            0K            0
NFSCL diroffdiroff            0            0K            0
       NEWdirectio            0            0K            0
        NEWNFSnode            0            0K            0
         NFSCL lck            0            0K            0
      NFSCL lckown            0            0K            0
      NFSCL client            0            0K            0
       NFSCL deleg            0            0K            0
        NFSCL open            0            0K            0
       NFSCL owner            0            0K            0
            NFS fh            0            0K            0
           NFS req            0            0K            0
     NFSD usrgroup            0            0K            0
       NFSD string            0            0K            0
       NFSD V4lock            0            0K            0
      NFSD V4state            0            0K            0
     NFSD srvcache            0            0K            0
       msdosfs_fat            0            0K            0
     msdosfs_mount            0            0K            0
      msdosfs_node            0            0K            0
            DEVFS4            0            0K            0
            DEVFS2            0            0K            0
            gntdev            0            0K            0
       privcmd_dev            0            0K            0
        evtchn_dev            0            0K            0
          xenstore            0            0K            0
         scsi_pass            0            0K            0
         ciss_data            0            0K            0
               xnb            0            0K            0
              xbbd            0            0K            0
               xbd            0            0K            0
           Balloon            0            0K            0
          sysmouse            0            0K            0
            vtfont            0            0K            0
           ath_hal            0            0K            0
            athdev            0            0K            0
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            0            0K            0
               amr            0            0K            0
           scsi_da            0            0K           69
            ata_da            0            0K            0
           scsi_ch            0            0K            0
           scsi_cd            0            0K            0
            USBdev            0            0K            0
               USB            0            0K            0
       AHCI driver            0            0K            0
               agp            0            0K            0
           nvme_da            0            0K            0
           acpipwr            0            0K            0
            twsbuf            0            0K            0
      twe_commands            0            0K            0
      twa_commands            0            0K            0
       tcp_log_dev            0            0K            0
      midi buffers            0            0K            0
             mixer            0            0K            0
              ac97            0            0K            0
             hdacc            0            0K            0
              hdac            0            0K            0
              hdaa            0            0K            0
         acpi_perf            0            0K            0
         acpicmbat            0            0K            0
       SIIS driver            0            0K            0
           CAM CCB            0            0K         1780
               PUC            0            0K            0
db> show ktr
No such command; use "help" to list available commands

Crashes (5):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/11/09 00:18 freebsd d12922a9f2b2 dc438b91 console log report syz C ci-freebsd-main
2019/11/08 21:14 freebsd d12922a9f2b2 dc438b91 console log report syz C ci-freebsd-main
2019/11/09 23:31 freebsd 76b010eaf2eb dc438b91 console log report ci-freebsd-main
2019/11/09 10:02 freebsd b740365e0f05 dc438b91 console log report ci-freebsd-main
2019/11/08 19:59 freebsd d12922a9f2b2 dc438b91 console log report ci-freebsd-main
* Struck through repros no longer work on HEAD.