syzbot


uvm_fault: pckbc_start (2)

Status: fixed on 2019/06/03 23:49
Reported-by: syzbot+fe74fc50c630bfa26302@syzkaller.appspotmail.com
Fix commit: bc79b6e32eb3 Prevent corruption of the pckbc command queue. If multiple synchronous commands are in flight and all corresponding threads are sleeping waiting for a response, the first command to timeout will clear the command queue. The remaining threads once awake will then try to remove a dequeued command from the queue, leading to corruption. Instead, remove commands from the queue before waking up the sleeping thread. A quirk is still needed to handle the case where tsleep() returns successfully during suspend.
First crash: 1290d, last: 1290d
similar bugs (1):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
openbsd uvm_fault: pckbc_start 1 1383d 1383d 3/3 fixed on 2019/02/19 14:52
Patch testing requests:
Created Duration User Patch Repo Result
2019/06/02 16:24 15m anton@basename.se https://github.com/mptre/openbsd-src pckbc OK
2019/05/30 21:06 16m anton@basename.se https://github.com/mptre/openbsd-src pckbc OK
2019/05/30 10:17 15m anton@basename.se https://github.com/mptre/openbsd-src pckbc report log
2019/05/28 15:25 15m anton@basename.se https://github.com/mptre/openbsd-src pckbc OK
2019/05/27 20:02 9m anton@basename.se https://github.com/mptre/openbsd-src cd6858bee94 report log
2019/05/27 18:24 9m anton@basename.se https://github.com/mptre/openbsd-src b50fe85dab5 report log
2019/05/27 18:08 8m anton@basename.se https://github.com/mptre/openbsd-src c77fcae412c report log
2019/05/27 17:32 11m anton@basename.se https://github.com/mptre/openbsd-src 50ca04f8b6d report log
2019/05/25 11:25 14m anton@basename.se https://github.com/mptre/openbsd-src.git bcbc3a82a68f0522eac31ab9060119194f065d13 report log

Sample crash report:
uvm_fault(0xfffffd807f00c870, 0x1c, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at      pckbc_start+0x170:      movsxdq 0x1c(%r14),%rax
ddb{0}> 
ddb{0}> set $lines = 0
ddb{0}> set $maxwidth = 0
ddb{0}> show panic
kernel page fault
uvm_fault(0xfffffd807f00c870, 0x1c, 0, 1) -> e
pckbc_start(ffff800000026e00,1) at pckbc_start+0x170 sys/dev/ic/pckbc.c:809
end trace frame: 0xffff800020c36ae0, count: 0
ddb{0}> trace
pckbc_start(ffff800000026e00,1) at pckbc_start+0x170 sys/dev/ic/pckbc.c:809
pckbc_enqueue_cmd(ffff800000026e00,1,ffff800020c36afe,2,0,1) at pckbc_enqueue_cmd+0x25a sys/dev/ic/pckbc.c:918
pms_ioctl_mouse(ffff80000066c200,80045721,ffff800020c36ea0,42,ffff800020b384c8) at pms_ioctl_mouse+0x137
wsmouse_do_ioctl(ffff80000064dc00,80045721,ffff800020c36ea0,42,ffff800020b384c8) at wsmouse_do_ioctl+0x2e3 sys/dev/wscons/wsmouse.c:530
wsmousedoioctl(ffff80000064dc00,80045721,ffff800020c36ea0,42,ffff800020b384c8) at wsmousedoioctl+0x51 sys/dev/wscons/wsmouse.c:432
wsmux_do_ioctl(ffff800000026d00,80045721,ffff800020c36ea0,42,ffff800020b384c8) at wsmux_do_ioctl+0x5b3 sys/dev/wscons/wsmux.c:544
VOP_IOCTL(fffffd807078c6f8,80045721,ffff800020c36ea0,42,fffffd807f7c6b40,ffff800020b384c8) at VOP_IOCTL+0x88 sys/kern/vfs_vops.c:290
vn_ioctl(fffffd806d04dac0,80045721,ffff800020c36ea0,ffff800020b384c8) at vn_ioctl+0xb6 sys/kern/vfs_vnops.c:512
sys_ioctl(ffff800020b384c8,ffff800020c36fc8,ffff800020c37030) at sys_ioctl+0x5b8
syscall(ffff800020c370a0) at syscall+0x552 mi_syscall sys/sys/syscall_mi.h:99 [inline]
syscall(ffff800020c370a0) at syscall+0x552 sys/arch/amd64/amd64/trap.c:574
Xsyscall(6,0,ffffffffffffff39,0,3,e7d5117c010) at Xsyscall+0x128
end of kernel
end trace frame: 0xe7f9f2c8080, count: -11
ddb{0}> show registers
rdi                                0
rsi                                0
rbp               0xffff800020c36a40
rbx                                0
rdx                              0x2
rcx                                0
rax                              0x1
r8                                 0
r9                               0x1
r10               0x7fc9d5263e6512bb
r11               0x3acb7f87347620a3
r12               0xffff800000026e00
r13               0xffff80000066c400
r14                                0
r15                              0x1
rip               0xffffffff817c2a20    pckbc_start+0x170
cs                               0x8
rflags                       0x10246    __ALIGN_SIZE+0xf246
rsp               0xffff800020c369e0
ss                              0x10
pckbc_start+0x170:      movsxdq 0x1c(%r14),%rax
ddb{0}> show proc
PROC (syz-executor.0) pid=8027 stat=onproc
    flags process=0 proc=4000000<THREAD>
    pri=52, usrpri=52, nice=20
    forw=0xffffffffffffffff, list=0xffff800020b38e28,0xffffffff8237f6f8
    process=0xffff800020b8c9f0 user=0xffff800020c32000, vmspace=0xfffffd807f00c870
    estcpu=8, cpticks=1, pctcpu=0.0
    user=0, sys=1, intr=0
ddb{0}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
 68979  223726  30061      0  2           0                syz-executor.0
*68979    8027  30061      0  7   0x4000000                syz-executor.0
 51083   91150  87283      0  3        0x82  nanosleep     syz-executor.1
 30061  406734  87283      0  3        0x82  nanosleep     syz-executor.0
 87283   11925  11838      0  3        0x82  thrsleep      syz-execprog
 87283  298187  11838      0  3   0x4000082  nanosleep     syz-execprog
 87283  387730  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  223584  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  101209  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  409225  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  324024  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  283126  11838      0  3   0x4000082  thrsleep      syz-execprog
 87283  105447  11838      0  3   0x4000082  kqread        syz-execprog
 11838  512979  98557      0  3    0x10008a  pause         ksh
 98557  348053  38114      0  3        0x92  select        sshd
 27594  219225      1      0  3    0x100083  ttyin         getty
 38114  115717      1      0  3        0x80  select        sshd
 62237  352477  37177     74  3    0x100092  bpf           pflogd
 37177  170818      1      0  3        0x80  netio         pflogd
 36146  373113  11712     73  2    0x100090                syslogd
 11712   13278      1      0  3    0x100082  netio         syslogd
  1110  179094      1     77  3    0x100090  poll          dhclient
 32167  147612      1      0  3        0x80  poll          dhclient
 79811  373370      0      0  2     0x14200                zerothread
 14002  122343      0      0  3     0x14200  aiodoned      aiodoned
 47952  475147      0      0  3     0x14200  syncer        update
 57307   60310      0      0  3     0x14200  cleaner       cleaner
 83433  225231      0      0  7     0x14200                reaper
 55766  252947      0      0  3     0x14200  pgdaemon      pagedaemon
 19853  263592      0      0  3     0x14200  bored         crynlk
 91976  267339      0      0  3     0x14200  bored         crypto
 16238  304075      0      0  3  0x40014200  acpi0         acpi0
 50733   40334      0      0  3  0x40014200                idle1
 47547  264176      0      0  3     0x14200  bored         softnet
 77315   34027      0      0  3     0x14200  bored         systqmp
   763  127276      0      0  3     0x14200  bored         systq
 87299  492243      0      0  3  0x40014200  bored         softclock
 40028  492858      0      0  3  0x40014200                idle0
 10833  454173      0      0  3     0x14200  bored         smr
     1  477969      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper
ddb{0}> show all locks
Process 68979 (syz-executor.0) thread 0xffff800020b384c8 (8027)
shared rwlock wsmuxlk r = 0 (0xffff800000026dd0)
#0  witness_lock+0x52e sys/kern/subr_witness.c:1161
#1  wsmux_do_ioctl+0x521
#2  VOP_IOCTL+0x88 sys/kern/vfs_vops.c:290
#3  vn_ioctl+0xb6 sys/kern/vfs_vnops.c:512
#4  sys_ioctl+0x5b8
#5  syscall+0x552 mi_syscall sys/sys/syscall_mi.h:99 [inline]
#5  syscall+0x552 sys/arch/amd64/amd64/trap.c:574
#6  Xsyscall+0x128
exclusive kernel_lock &kernel_lock r = 1 (0xffffffff82396f70)
#0  witness_lock+0x52e sys/kern/subr_witness.c:1161
#1  syscall+0x43a mi_syscall sys/sys/syscall_mi.h:91 [inline]
#1  syscall+0x43a sys/arch/amd64/amd64/trap.c:574
#2  Xsyscall+0x128
ddb{0}> show malloc
           Type InUse  MemUse  HighUse   Limit  Requests Type Lim Kern Lim
         devbuf  9461   6395K    6395K  78643K     10548        0        0
            pcb    25      9K       9K  78643K        61        0        0
         rtable   105      3K       3K  78643K       201        0        0
         ifaddr    38     10K      10K  78643K        39        0        0
       counters    39     33K      33K  78643K        39        0        0
       ioctlops     0      0K       4K  78643K      1469        0        0
          mount     1      1K       1K  78643K         1        0        0
         vnodes  1176     74K      74K  78643K      1746        0        0
      UFS quota     1     32K      32K  78643K         1        0        0
      UFS mount     5     36K      36K  78643K         5        0        0
            shm     2      1K       1K  78643K         2        0        0
         VM map     2      1K       1K  78643K         2        0        0
            sem     2      0K       0K  78643K         2        0        0
        dirhash    12      2K       2K  78643K        12        0        0
           ACPI  1808    196K     290K  78643K     12628        0        0
      file desc     4     12K      24K  78643K       389        0        0
           proc    52     50K      83K  78643K       359        0        0
    NFS srvsock     1      0K       0K  78643K         1        0        0
     NFS daemon     1     16K      16K  78643K         1        0        0
       in_multi    33      2K       2K  78643K        33        0        0
    ether_multi     1      0K       0K  78643K         1        0        0
    ISOFS mount     1     32K      32K  78643K         1        0        0
  MSDOSFS mount     1     16K      16K  78643K         1        0        0
           ttys    18     79K      79K  78643K        18        0        0
           exec     0      0K       1K  78643K       212        0        0
        pagedep     1      8K       8K  78643K         1        0        0
       inodedep     1     32K      32K  78643K         1        0        0
         newblk     1      0K       0K  78643K         1        0        0
        VM swap     7     26K      26K  78643K         7        0        0
       UVM amap    84     20K      20K  78643K      2054        0        0
       UVM aobj     2      2K       2K  78643K         2        0        0
        memdesc     1      4K       4K  78643K         1        0        0
    crypto data     1      1K       1K  78643K         1        0        0
            NDP     6      0K       0K  78643K        10        0        0
           temp    55   2714K    2778K  78643K      3981        0        0
      SYN cache     2     16K      16K  78643K         2        0        0
ddb{0}> show all pools
Name      Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
arp         64        6    0        0     1     0     1     1     0     8    0
inpcbpl    280       37    0       31     1     0     1     1     0     8    0
plimitpl   152       16    0        8     1     0     1     1     0     8    0
plcache    128       20    0        0     1     0     1     1     0     8    0
rtentry    112       45    0        1     2     0     2     2     0     8    0
syncache   264        5    0        5     1     1     0     1     0     8    0
tcpcb      544        8    0        5     1     0     1     1     0     8    0
nd6         48        6    0        0     1     0     1     1     0     8    0
pfosfp      40      846    0      423     5     0     5     5     0     8    0
pfosfpen   112     1428    0      714    21     0    21    21     0     8    0
pfstitem    24       11    0        2     1     0     1     1     0     8    0
pfstkey    112       11    0        2     1     0     1     1     0     8    0
pfstate    328       11    0        2     1     0     1     1     0     8    0
pfrule     1360      21    0       16     2     1     1     2     0     8    0
art_heap8  4096       1    0        0     1     0     1     1     0     8    0
art_heap4  256      187    0        0    12     0    12    12     0     8    0
art_table   32      188    0        0     2     0     2     2     0     8    0
art_node    16       44    0        4     1     0     1     1     0     8    0
dirhash    1024      17    0        0     3     0     3     3     0     8    0
dino1pl    128     2722    0     1296    47     0    47    47     0     8    0
ffsino     272     2722    0     1296    96     0    96    96     0     8    0
nchpl      144     3304    0     1660    61     0    61    61     0     8    0
uvmvnodes   72     2732    0        0    50     0    50    50     0     8    0
vnodes     200     2732    0        0   144     0   144   144     0     8    0
namei      1024    8265    0     8265     2     1     1     1     0     8    1
percpumem   16       30    0        0     1     0     1     1     0     8    0
scxspl     192     7045    0     7045     8     7     1     6     0     8    1
sigapl     432      594    0      579     3     0     3     3     0     8    1
futexpl     56     1726    0     1726     1     0     1     1     0     8    1
knotepl    112       51    0       34     1     0     1     1     0     8    0
kqueuepl   104        2    0        0     1     0     1     1     0     8    0
pipepl     112      162    0      143     2     1     1     1     0     8    0
fdescpl    488      595    0      579     3     0     3     3     0     8    0
filepl     152     2584    0     2516     3     0     3     3     0     8    0
lockfpl    104     1422    0     1420     2     1     1     1     0     8    0
lockfspl    48      359    0      358     2     1     1     1     0     8    0
sessionpl  112       20    0        9     1     0     1     1     0     8    0
pgrppl      48       20    0        9     1     0     1     1     0     8    0
ucredpl     96       52    0       43     1     0     1     1     0     8    0
zombiepl   144      579    0      578     2     1     1     1     0     8    0
processpl  840      610    0      578     4     0     4     4     0     8    0
procpl     600     1300    0     1259     4     0     4     4     0     8    0
sockpl     384       85    0       67     3     0     3     3     0     8    1
mcl4k      4096       2    0        0     1     0     1     1     0     8    0
mcl2k      2048      80    0        0    10     0    10    10     0     8    0
mtagpl      80        1    0        0     1     0     1     1     0     8    0
mbufpl     256      143    0        0     8     0     8     8     0     8    0
bufpl      256     5754    0     1188   286     0   286   286     0     8    0
anonpl      16    43131    0    41064    15     5    10    13     0   125    1
amapchunkpl 152    2680    0     2590     6     0     6     6     0   158    2
amappl16   192     1664    0     1600     4     0     4     4     0     8    0
amappl15   184        1    0        0     1     0     1     1     0     8    0
amappl14   176       52    0       46     2     1     1     1     0     8    0
amappl13   168      177    0      175     1     0     1     1     0     8    0
amappl12   160        5    0        5     1     1     0     1     0     8    0
amappl11   152      241    0      221     1     0     1     1     0     8    0
amappl10   144       79    0       74     1     0     1     1     0     8    0
amappl9    136      444    0      441     1     0     1     1     0     8    0
amappl8    128      135    0      124     1     0     1     1     0     8    0
amappl7    120       34    0       30     1     0     1     1     0     8    0
amappl6    112      242    0      233     1     0     1     1     0     8    0
amappl5    104      119    0      106     1     0     1     1     0     8    0
amappl4     96      771    0      749     1     0     1     1     0     8    0
amappl3     88      190    0      179     1     0     1     1     0     8    0
amappl2     80     4032    0     3949     3     1     2     3     0     8    0
amappl1     72    22180    0    21715    25    10    15    20     0     8    5
amappl      80     1545    0     1507     1     0     1     1     0    84    0
dma4096    4096       1    0        1     1     1     0     1     0     8    0
dma256     256        6    0        6     1     1     0     1     0     8    0
dma64       64      259    0      259     1     1     0     1     0     8    0
dma32       32        7    0        7     1     1     0     1     0     8    0
dma16       16       17    0       17     1     1     0     1     0     8    0
aobjpl      64        1    0        0     1     0     1     1     0     8    0
uaddrrnd    24      595    0      579     1     0     1     1     0     8    0
uaddrbest   32        2    0        0     1     0     1     1     0     8    0
uaddr       24      595    0      579     1     0     1     1     0     8    0
vmmpekpl   168     8632    0     8609     2     0     2     2     0     8    0
vmmpepl    168    61600    0    60499    89    14    75    78     0   357   25
vmsppl     360      594    0      578     2     0     2     2     0     8    0
pdppl      4096    1197    0     1156     6     0     6     6     0     8    0
pvpl        32   167724    0   163195   105     6    99   102     0   265   61
pmappl     232      594    0      578     2     0     2     2     0     8    1
extentpl    40       41    0       26     1     0     1     1     0     8    0
phpool     112      430    0        4    13     0    13    13     0     8    0

Crashes (2):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-openbsd-multicore 2019/05/22 20:22 openbsd f537473e237b 84b9d384 .config log report syz
ci-openbsd-multicore 2019/05/22 18:34 openbsd f537473e237b 84b9d384 .config log report
* Struck through repros no longer work on HEAD.