syzbot


witness: userret: ioctl (3)

Status: fixed on 2021/12/23 16:59
Reported-by: syzbot+6dde3fda33074a256318@syzkaller.appspotmail.com
Fix commit: 41a45098bac2 Disk lock was held when returning to userland. Add a missing unlock in vnd ioctl error path. Reported-by: syzbot+6dde3fda33074a256318@syzkaller.appspotmail.com OK jsg@ anton@
First crash: 345d, last: 345d
similar bugs (2):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
openbsd witness: userret: ioctl C 3 772d 772d 3/3 fixed on 2020/10/22 20:39
openbsd witness: userret: ioctl (2) 1 360d 360d 3/3 fixed on 2021/12/08 00:40

Sample crash report:
login: witness: userret: returning with the following locks held:
exclusive rwlock dklk r = 0 (0xffff8000006b6068)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  vndioctl+0xcde sys/dev/vnd.c:498
#3  VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
#4  vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
#5  sys_ioctl+0x4a2
#6  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#6  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#7  Xsyscall+0x128
panic: witness_warn
Stopped at      db_enter+0x18:  addq    $0x8,%rsp
    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
*411516   3882      0           0  0x4000000    1  syz-executor.0
 445713  44828     73    0x100010          0    0  syslogd
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff82446f88) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e witness_debugger sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e sys/kern/subr_witness.c:1473
userret(ffff8000212547e0) at userret+0x253 sys/kern/kern_sig.c:1950
syscall(ffff8000212b8710) at syscall+0x57e mi_syscall_return sys/sys/syscall_mi.h:129 [inline]
syscall(ffff8000212b8710) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0xcd714983470, count: 9
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports.  Insufficient info makes it difficult to find and fix bugs.
ddb{1}> 
ddb{1}> set $lines = 0
ddb{1}> set $maxwidth = 0
ddb{1}> show panic
*cpu1: witness_warn
ddb{1}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff82446f88) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e witness_debugger sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e sys/kern/subr_witness.c:1473
userret(ffff8000212547e0) at userret+0x253 sys/kern/kern_sig.c:1950
syscall(ffff8000212b8710) at syscall+0x57e mi_syscall_return sys/sys/syscall_mi.h:129 [inline]
syscall(ffff8000212b8710) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0xcd714983470, count: -6
ddb{1}> show registers
rdi                                0
rsi                              0x1
rbp               0xffff8000212b8470
rbx               0xffff800020d39bff
rdx                            0x3fd
rcx                                0
rax                             0x14
r8                 0x101010101010101
r9                0x8080808080808080
r10               0xc83d85b0359851cf
r11               0x1f1fca05086808ff
r12               0xffff800020d39a00
r13                                0
r14                                0
r15                              0x1
rip               0xffffffff81a3ecd8    db_enter+0x18
cs                               0x8
rflags                         0x246
rsp               0xffff8000212b8460
ss                                 0
db_enter+0x18:  addq    $0x8,%rsp
ddb{1}> show proc
PROC (syz-executor.0) pid=411516 stat=onproc
    flags process=0 proc=4000000<THREAD>
    pri=32, usrpri=86, nice=20
    forw=0xffffffffffffffff, list=0xffff800021254a80,0xffff800021254550
    process=0xffff8000211f1900 user=0xffff8000212b3000, vmspace=0xfffffd807efffcf0
    estcpu=36, cpticks=4, pctcpu=0.0
    user=0, sys=4, intr=0
ddb{1}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
  3882  281232  76065      0  2           0                syz-executor.0
  3882  305584  76065      0  3   0x4000080  fsleep        syz-executor.0
  3882  193809  76065      0  2   0x4000000                syz-executor.0
* 3882  411516  76065      0  7   0x4000000                syz-executor.0
  3882  379592  76065      0  3   0x4000080  dklk          syz-executor.0
 26363   74012  68691      0  2           0                syz-executor.1
 26363   99127  68691      0  3   0x4000080  fsleep        syz-executor.1
 26363  254575  68691      0  3   0x4000080  fsleep        syz-executor.1
 26363  260652  68691      0  3   0x4000080  fsleep        syz-executor.1
 26363  119598  68691      0  2   0x4000000                syz-executor.1
 68691  289740  77754      0  2       0x482                syz-executor.1
 76065  291676  77754      0  3        0x82  nanoslp       syz-executor.0
 77754  336187   7802      0  3        0x82  thrsleep      syz-execprog
 77754  136443   7802      0  2   0x4000482                syz-execprog
 77754   61685   7802      0  3   0x4000082  thrsleep      syz-execprog
 77754  417558   7802      0  3   0x4000082  thrsleep      syz-execprog
 77754  510775   7802      0  3   0x4000082  kqread        syz-execprog
 77754  261341   7802      0  3   0x4000082  thrsleep      syz-execprog
 77754  220644   7802      0  3   0x4000082  thrsleep      syz-execprog
  7802  471913  35007      0  3    0x10008a  sigsusp       ksh
 35007  334843  43343      0  3        0x9a  poll          sshd
   901   81551      1      0  3    0x100083  ttyin         getty
 43343  393831      1      0  3        0x88  poll          sshd
 90752  523986  13682     74  3    0x100092  bpf           pflogd
 13682  222458      1      0  3        0x80  netio         pflogd
 44828  445713  94012     73  7    0x100010                syslogd
 94012  291113      1      0  3    0x100082  netio         syslogd
 85697  420356      1      0  3    0x100080  kqread        resolvd
 80848    2158  10262     77  3    0x100092  kqread        dhcpleased
 27118  261534  10262     77  3    0x100092  kqread        dhcpleased
 10262  205462      1      0  3        0x80  kqread        dhcpleased
 64886   79503      0      0  3     0x14200  bored         smr
 69158  176912      0      0  2     0x14200                zerothread
 80060  143973      0      0  3     0x14200  aiodoned      aiodoned
 33368   40635      0      0  3     0x14200  syncer        update
  5026  486437      0      0  3     0x14200  cleaner       cleaner
 90575  126035      0      0  3     0x14200  reaper        reaper
 16400  428579      0      0  3     0x14200  pgdaemon      pagedaemon
 24937  340394      0      0  3     0x14200  bored         viomb
 56245  329564      0      0  3  0x40014200  acpi0         acpi0
 46660  289873      0      0  3  0x40014200                idle1
 20424  112215      0      0  3     0x14200  bored         softnet
 57044  515606      0      0  3     0x14200  bored         systqmp
 87324  404494      0      0  3     0x14200  bored         systq
 45570  478615      0      0  3  0x40014200  bored         softclock
 50345  251344      0      0  3  0x40014200                idle0
     1  409544      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper
ddb{1}> show all locks
Process 3882 (syz-executor.0) thread 0xffff8000212547e0 (411516)
exclusive rwlock dklk r = 0 (0xffff8000006b6068)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  vndioctl+0xcde sys/dev/vnd.c:498
#3  VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
#4  vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
#5  sys_ioctl+0x4a2
#6  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#6  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#7  Xsyscall+0x128
Process 44828 (syslogd) thread 0xffff8000ffff6d30 (445713)
exclusive rrwlock inode r = 0 (0xfffffd806e786700)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  vn_lock+0x84 sys/kern/vfs_vnops.c:579
#5  sys_fsync+0xf5 sys/kern/vfs_syscalls.c:2888
#6  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#6  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#7  Xsyscall+0x128
ddb{1}> show malloc
           Type InUse  MemUse  HighUse   Limit  Requests Type Lim
         devbuf 10124   6457K    6457K  78643K     11240        0
            pcb    13      8K       8K  78643K        13        0
         rtable   106      3K       3K  78643K       172        0
         ifaddr    44     10K      10K  78643K        45        0
       counters    44     34K      34K  78643K        44        0
       ioctlops     0      0K       4K  78643K      1481        0
          mount     1      1K       1K  78643K         1        0
            log     0      0K       0K  78643K         5        0
         vnodes  1179     74K      74K  78643K      1185        0
      UFS quota     1     32K      32K  78643K         1        0
      UFS mount     5     36K      36K  78643K         5        0
            shm     2      1K       1K  78643K         2        0
         VM map     2      1K       1K  78643K         2        0
            sem     2      0K       0K  78643K         2        0
        dirhash    12      2K       2K  78643K        12        0
           ACPI  1697    195K     286K  78643K     12598        0
      file desc     6     17K      25K  78643K        94        0
           proc    67     87K     111K  78643K       338        0
    NFS srvsock     1      0K       0K  78643K         1        0
     NFS daemon     1     16K      16K  78643K         1        0
       in_multi    33      2K       2K  78643K        33        0
    ether_multi     1      0K       0K  78643K         1        0
    ISOFS mount     1     32K      32K  78643K         1        0
  MSDOSFS mount     1     16K      16K  78643K         1        0
           ttys    25    122K     122K  78643K        25        0
           exec     0      0K       2K  78643K       413        0
            tdb     3      0K       0K  78643K         3        0
        pagedep     1      8K       8K  78643K         1        0
       inodedep     1     32K      32K  78643K         1        0
         newblk     1      0K       0K  78643K         1        0
        VM swap     7     26K      26K  78643K         7        0
       UVM amap   173     24K      24K  78643K      2898        0
       UVM aobj     3      2K       2K  78643K         3        0
        memdesc     1      4K       4K  78643K         1        0
    crypto data     1      1K       1K  78643K         1        0
            NDP     6      0K       0K  78643K        10        0
           temp    33   4182K    4246K  78643K      3515        0
         kqueue    10     14K      14K  78643K        10        0
      SYN cache     2     16K      16K  78643K         2        0
ddb{1}> show all pools
Name      Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
plcache    128       22    0        0     1     0     1     1     0     8    0
rtpcb      120       21    0       18     1     0     1     1     0     8    0
rtentry    112       45    0        1     2     0     2     2     0     8    0
unpcb      128       35    0       20     1     0     1     1     0     8    0
syncache   296        5    0        5     2     1     1     1     0     8    1
tcpcb      736        8    0        5     1     0     1     1     0     8    0
arp        120        6    0        0     1     0     1     1     0     8    0
inpcb      304       40    0       34     1     0     1     1     0     8    0
nd6         48        6    0        0     1     0     1     1     0     8    0
pfosfp      40     1428    0     1005     5     0     5     5     0     8    0
pfosfpen   112     1428    0      714    21     0    21    21     0     8    0
pfstitem    24       11    0        2     1     0     1     1     0     8    0
pfstkey    112       11    0        2     1     0     1     1     0     8    0
pfstate    320       11    0        2     1     0     1     1     0     8    0
pfrule     1360      21    0       15     2     0     2     2     0     8    0
art_heap8  4096       1    0        0     1     0     1     1     0     8    0
art_heap4  256      188    0        0    12     0    12    12     0     8    0
art_table   32      189    0        0     2     0     2     2     0     8    0
art_node    16       44    0        4     1     0     1     1     0     8    0
dirhash    1024      17    0        0     3     0     3     3     0     8    0
dino2pl    256     1555    0      145    89     0    89    89     0     8    0
ffsino     272     1555    0      145    95     0    95    95     0     8    0
nchpl      144     1921    0      299    61     0    61    61     0     8    0
uvmvnodes   80     1566    0        0    32     0    32    32     0     8    0
vnodes     224     1566    0        0    93     0    93    93     0     8    0
namei      1024    5291    0     5291     2     1     1     1     0     8    1
percpumem   16       34    0        0     1     0     1     1     0     8    0
scxspl     216     5772    0     5772    10     9     1     8     0     8    1
plimitpl   152       18    0        9     1     0     1     1     0     8    0
sigapl     424      348    0      315     4     0     4     4     0     8    0
futexpl     64      326    0      322     1     0     1     1     0     8    0
knotepl    112       52    0        0     2     0     2     2     0     8    0
kqueuepl   216        6    0        0     1     0     1     1     0     8    0
pipepl     336       87    0       77     2     1     1     1     0     8    0
fdescpl    496      334    0      315     3     0     3     3     0     8    0
filepl     152     1419    0     1336     4     0     4     4     0     8    0
lockfpl    104        6    0        4     1     0     1     1     0     8    0
lockfspl    48        4    0        2     1     0     1     1     0     8    0
sessionpl  144       20    0        9     1     0     1     1     0     8    0
pgrppl      48       20    0        9     1     0     1     1     0     8    0
ucredpl     96      108    0       95     1     0     1     1     0     8    0
zombiepl   144      315    0      315     2     1     1     1     0     8    1
processpl  1064     348    0      315     3     0     3     3     0     8    0
procpl     672      551    0      504     4     0     4     4     0     8    0
sockpl     480       96    0       72     5     1     4     4     0     8    0
mcl8k      8192       2    0        0     1     0     1     1     0     8    0
mcl4k      4096       2    0        0     1     0     1     1     0     8    0
mcl2k      2048      57    0        0     8     0     8     8     0     8    0
mtagpl      96        1    0        0     1     0     1     1     0     8    0
mbufpl     256      157    0        0     9     0     9     9     0     8    0
bufpl      288     3651    0      133   252     0   252   252     0     8    0
anonpl      24    61599    0    56991    35     3    32    32     0   186    4
amapchunkpl 152    6245    0     5847    17     1    16    16     0   158    0
amappl16   200      238    0      176     4     0     4     4     0     8    0
amappl15   192       84    0       78     1     0     1     1     0     8    0
amappl14   184        2    0        1     1     0     1     1     0     8    0
amappl13   176       61    0       58     2     1     1     1     0     8    0
amappl12   168       89    0       82     1     0     1     1     0     8    0
amappl11   160       46    0       32     1     0     1     1     0     8    0
amappl10   152       17    0       13     1     0     1     1     0     8    0
amappl9    144      441    0      439     1     0     1     1     0     8    0
amappl8    136      313    0      296     2     1     1     1     0     8    0
amappl7    128       71    0       62     1     0     1     1     0     8    0
amappl6    120      150    0      135     1     0     1     1     0     8    0
amappl5    112      169    0      155     1     0     1     1     0     8    0
amappl4    104      603    0      580     1     0     1     1     0     8    0
amappl3     96      189    0      170     1     0     1     1     0     8    0
amappl2     88      438    0      398     2     0     2     2     0     8    1
amappl1     80     9729    0     9255    15     2    13    13     0     8    3
amappl      88     2595    0     2458     4     0     4     4     0    92    0
dma4096    4096       1    0        1     1     1     0     1     0     8    0
dma1024    1024       1    0        0     1     0     1     1     0     8    0
dma256     256        6    0        6     1     1     0     1     0     8    0
dma128     128      253    0      253     1     1     0     1     0     8    0
dma64       64        6    0        6     1     1     0     1     0     8    0
dma32       32        7    0        7     1     1     0     1     0     8    0
dma16       16       18    0       17     1     0     1     1     0     8    0
aobjpl      72        2    0        0     1     0     1     1     0     8    0
uaddrrnd    24      334    0      315     1     0     1     1     0     8    0
uaddrbest   32        2    0        0     1     0     1     1     0     8    0
uaddr       24      334    0      315     1     0     1     1     0     8    0
vmmpekpl   168     7589    0     7569     2     0     2     2     0     8    0
vmmpepl    168    32548    0    31237    67     3    64    64     0   357    7
vmsppl     368      333    0      315     2     0     2     2     0     8    0
rwobjpl     56    11483    0     9119    35     0    35    35     0     8    1
pdppl      4096     676    0      630    68    18    50    50     0     8    4
pvpl        32   193153    0   185450   147     6   141   141     0   265   78
pmappl     248      333    0      315     2     0     2     2     0     8    0
extentpl    40       57    0       38     1     0     1     1     0     8    0
phpool     112      551    0       22    16     0    16    16     0     8    0
ddb{1}> machine ddbcpu 0
Stopped at      x86_ipi_db+0x1a:        addq    $0x8,%rsp
x86_ipi_db(ffffffff82813ff0) at x86_ipi_db+0x1a sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_lock(ffffffff82952170) at __mp_lock+0x122 __mp_lock_spin sys/kern/kern_lock.c:116 [inline]
__mp_lock(ffffffff82952170) at __mp_lock+0x122 sys/kern/kern_lock.c:147
__mp_acquire_count(ffffffff82952170,1) at __mp_acquire_count+0x48 sys/kern/kern_lock.c:227
mi_switch() at mi_switch+0x3d3 sys/kern/sched_bsd.c:416
sleep_finish(ffff800021206240,1) at sleep_finish+0x1b2 sys/kern/kern_synch.c:433
tsleep(fffffd806e98e450,11,ffffffff82462ccd,0) at tsleep+0x12c sys/kern/kern_synch.c:158
vwaitforio(fffffd806e98e3d0,0,ffffffff82462ccd,ffffffffffffffff) at vwaitforio+0x98 sys/kern/vfs_subr.c:1897
ffs_fsync(ffff800021206390) at ffs_fsync+0x38e sys/ufs/ffs/ffs_vnops.c:498
VOP_FSYNC(fffffd806e98e3d0,fffffd807f7d8a80,1,ffff8000ffff6d30) at VOP_FSYNC+0xce sys/kern/vfs_vops.c:327
sys_fsync(ffff8000ffff6d30,ffff800021206480,ffff8000212064e0) at sys_fsync+0x109 sys/kern/vfs_syscalls.c:2888
syscall(ffff800021206550) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff800021206550) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffe0410, count: 1
ddb{0}> trace
x86_ipi_db(ffffffff82813ff0) at x86_ipi_db+0x1a sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_lock(ffffffff82952170) at __mp_lock+0x122 __mp_lock_spin sys/kern/kern_lock.c:116 [inline]
__mp_lock(ffffffff82952170) at __mp_lock+0x122 sys/kern/kern_lock.c:147
__mp_acquire_count(ffffffff82952170,1) at __mp_acquire_count+0x48 sys/kern/kern_lock.c:227
mi_switch() at mi_switch+0x3d3 sys/kern/sched_bsd.c:416
sleep_finish(ffff800021206240,1) at sleep_finish+0x1b2 sys/kern/kern_synch.c:433
tsleep(fffffd806e98e450,11,ffffffff82462ccd,0) at tsleep+0x12c sys/kern/kern_synch.c:158
vwaitforio(fffffd806e98e3d0,0,ffffffff82462ccd,ffffffffffffffff) at vwaitforio+0x98 sys/kern/vfs_subr.c:1897
ffs_fsync(ffff800021206390) at ffs_fsync+0x38e sys/ufs/ffs/ffs_vnops.c:498
VOP_FSYNC(fffffd806e98e3d0,fffffd807f7d8a80,1,ffff8000ffff6d30) at VOP_FSYNC+0xce sys/kern/vfs_vops.c:327
sys_fsync(ffff8000ffff6d30,ffff800021206480,ffff8000212064e0) at sys_fsync+0x109 sys/kern/vfs_syscalls.c:2888
syscall(ffff800021206550) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff800021206550) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffe0410, count: -14
ddb{0}> machine ddbcpu 1
Stopped at      db_enter+0x18:  addq    $0x8,%rsp
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff82446f88) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e witness_debugger sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e sys/kern/subr_witness.c:1473
userret(ffff8000212547e0) at userret+0x253 sys/kern/kern_sig.c:1950
syscall(ffff8000212b8710) at syscall+0x57e mi_syscall_return sys/sys/syscall_mi.h:129 [inline]
syscall(ffff8000212b8710) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0xcd714983470, count: 9
ddb{1}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff82446f88) at panic+0x177 sys/kern/subr_prf.c:202
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e witness_debugger sys/kern/subr_witness.c:2505 [inline]
witness_warn(2,0,ffffffff824cde14) at witness_warn+0x65e sys/kern/subr_witness.c:1473
userret(ffff8000212547e0) at userret+0x253 sys/kern/kern_sig.c:1950
syscall(ffff8000212b8710) at syscall+0x57e mi_syscall_return sys/sys/syscall_mi.h:129 [inline]
syscall(ffff8000212b8710) at syscall+0x57e sys/arch/amd64/amd64/trap.c:607
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0xcd714983470, count: -6
ddb{1}> 

Crashes (2):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-openbsd-multicore 2021/12/22 22:08 openbsd 5845290fe6de 6caa12e4 .config log report syz witness: userret: ioctl
ci-openbsd-multicore 2021/12/22 21:54 openbsd 5845290fe6de 6caa12e4 .config log report witness: userret: ioctl
* Struck through repros no longer work on HEAD.