syzbot


witness: reversal: pf_lock netlock

Status: fixed on 2022/04/08 05:01
Reported-by: syzbot+b22ec16c5bf937578937@syzkaller.appspotmail.com
Fix commit: 82d8999861ab Release PF und NET lock before calling copyin and copyout for DIOCXBEGIN. OK bluhm@ OK sashan@ Reported-by: syzbot+b22ec16c5bf937578937@syzkaller.appspotmail.com
First crash: 301d, last: 249d
similar bugs (2):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
openbsd witness: reversal: pf_lock netlock (2) C 2 237d 237d 3/3 fixed on 2022/04/10 04:45
openbsd witness: reversal: pf_lock netlock (3) syz 25 217d 229d 3/3 fixed on 2022/04/29 14:56

Sample crash report:
login: witness: lock order reversal:
 1st 0xffffffff82981660 pf_lock (pf_lock)
 2nd 0xffffffff8291bdd0 netlock (netlock)
lock order "netlock"(rwlock) -> "pf_lock"(rwlock) first seen at:
#0  rw_enter_write+0x5b sys/kern/kern_rwlock.c:128
#1  pfioctl+0x3fba sys/net/pf_ioctl.c:2966
#2  VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
#3  vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
#4  sys_ioctl+0x4a2
#5  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#5  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#6  Xsyscall+0x128
lock order "pf_lock"(rwlock) -> "netlock"(rwlock) first seen at:
#0  rw_enter_write+0x5b sys/kern/kern_rwlock.c:128
#1  uvn_io+0x418 sys/uvm/uvm_vnode.c:1243
#2  uvn_get+0x1df sys/uvm/uvm_vnode.c:1076
#3  uvm_fault_lower+0x36c sys/uvm/uvm_fault.c:1275
#4  uvm_fault+0x248
#5  kpageflttrap+0x209
#6  kerntrap+0xef sys/arch/amd64/amd64/trap.c:318
#7  alltraps_kern_meltdown+0x7b
#8  copyin+0x53
#9  VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
#10 vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
#11 sys_ioctl+0x4a2
#12 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#12 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#13 Xsyscall+0x128
Stopped at      db_enter+0x18:  addq    $0x8,%rsp
ddb{0}> 
ddb{0}> set $lines = 0
ddb{0}> set $maxwidth = 0
ddb{0}> show panic
the kernel did not panic
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
witness_checkorder(ffffffff8291bdd0,9,0) at witness_checkorder+0x10b7 witness_debugger sys/kern/subr_witness.c:2502 [inline]
witness_checkorder(ffffffff8291bdd0,9,0) at witness_checkorder+0x10b7 sys/kern/subr_witness.c:1105
rw_enter_write(ffffffff8291bdc0) at rw_enter_write+0x5b sys/kern/kern_rwlock.c:128
uvn_io(fffffd806d66d790,ffff8000211cd938,1,202,0) at uvn_io+0x418 sys/uvm/uvm_vnode.c:1243
uvn_get(fffffd806d66d790,1000,ffff8000211cd9f8,ffff8000211cd9c8,0,1,71fb8b939eed8b75,0) at uvn_get+0x1df sys/uvm/uvm_vnode.c:1076
uvm_fault_lower(ffff8000211cdb60,ffff8000211cdb98,ffff8000211cdae0,0) at uvm_fault_lower+0x36c sys/uvm/uvm_fault.c:1275
uvm_fault(fffffd806f1efe68,20001000,0,1) at uvm_fault+0x248
kpageflttrap(ffff8000211cdcf0,20001000) at kpageflttrap+0x209
kerntrap(ffff8000211cdcf0) at kerntrap+0xef sys/arch/amd64/amd64/trap.c:318
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
copyin() at copyin+0x53
VOP_IOCTL(fffffd806f8159c8,c0104451,ffff8000211ce0d0,2,fffffd807f7d8960,ffff8000ffff6a90) at VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
vn_ioctl(fffffd806e51a860,c0104451,ffff8000211ce0d0,ffff8000ffff6a90) at vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
sys_ioctl(ffff8000ffff6a90,ffff8000211ce1e8,ffff8000211ce240) at sys_ioctl+0x4a2
syscall(ffff8000211ce2b0) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff8000211ce2b0) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffbec30, count: -16
ddb{0}> show registers
rdi                              0x3
rsi               0xffffffff8294d790    __sancov_gen_cov_switch_values.134
rbp               0xffff8000211cd700
rbx                              0x3
rdx                            0x3fd
rcx                                0
rax                              0x1
r8                0xffff8000211cd670
r9                0x8080808080808080
r10                0x38bdf05f4bfbb50
r11               0x17ac1df0a447fe87
r12               0xffffffff82adbf70    w_lodata+0x52800
r13                                0
r14               0xffffffff82ad89d0    w_lodata+0x4f260
r15               0xfffffd8002f60780
rip               0xffffffff81395b28    db_enter+0x18
cs                               0x8
rflags                         0x246
rsp               0xffff8000211cd6f0
ss                              0x10
db_enter+0x18:  addq    $0x8,%rsp
ddb{0}> show proc
PROC (syz-executor3073947665) pid=134615 stat=onproc
    flags process=2<EXEC> proc=0
    pri=17, usrpri=50, nice=20
    forw=0xffffffffffffffff, list=0xffff8000ffff62b0,0xffffffff82af99a8
    process=0xffff8000211a0010 user=0xffff8000211c9000, vmspace=0xfffffd806f1efe68
    estcpu=0, cpticks=0, pctcpu=0.0
    user=0, sys=0, intr=0
ddb{0}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
*64312  134615  31587      0  7         0x2                syz-executor3073947665
 31587  289449  44521      0  3    0x10008a  sigsusp       ksh
 44521  297979  22700      0  3        0x9a  kqread        sshd
 17075   25490      1      0  3    0x100083  ttyin         getty
 22700  175702      1      0  3        0x88  kqread        sshd
 14436  278851  62179     74  3   0x1100092  bpf           pflogd
 62179  369652      1      0  3        0x80  netio         pflogd
 78047  417736  38822     73  3   0x1100090  kqread        syslogd
 38822  174128      1      0  3    0x100082  netio         syslogd
 71221  218432      1      0  3    0x100080  kqread        resolvd
 84822  272576  73574     77  3    0x100092  kqread        dhcpleased
  6677  237660  73574     77  3    0x100092  kqread        dhcpleased
 73574  344477      1      0  3        0x80  kqread        dhcpleased
 68020  305756      0      0  3     0x14200  bored         smr
 34372  179053      0      0  3     0x14200  pgzero        zerothread
 32195  425231      0      0  3     0x14200  aiodoned      aiodoned
 14149  376999      0      0  3     0x14200  syncer        update
 72890   77747      0      0  3     0x14200  cleaner       cleaner
 43277  383154      0      0  3     0x14200  reaper        reaper
 37625  457917      0      0  3     0x14200  pgdaemon      pagedaemon
 86144  197469      0      0  3     0x14200  bored         viomb
 38268  469731      0      0  3  0x40014200  acpi0         acpi0
 97997  267316      0      0  7  0x40014200                idle1
 38004  330148      0      0  3     0x14200  bored         softnet
 61483  190397      0      0  3     0x14200  bored         systqmp
 59157   92360      0      0  3     0x14200  bored         systq
 48065   42089      0      0  3  0x40014200  bored         softclock
 78209   39519      0      0  3  0x40014200                idle0
     1  152114      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper
ddb{0}> show all locks
Process 64312 (syz-executor3073947665) thread 0xffff8000ffff6a90 (134615)
exclusive rwlock pf_lock r = 0 (0xffffffff82981660)
#0  witness_lock+0x44d
#1  pfioctl+0x413b sys/net/pf_ioctl.c:2476
#2  VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
#3  vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
#4  sys_ioctl+0x4a2
#5  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#5  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#6  Xsyscall+0x128
exclusive kernel_lock &kernel_lock r = 0 (0xffffffff82a17948)
#0  witness_lock+0x44d
#1  vn_ioctl+0x41 sys/kern/vfs_vnops.c:514
#2  sys_ioctl+0x4a2
#3  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#3  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#4  Xsyscall+0x128
ddb{0}> show malloc
           Type InUse  MemUse  HighUse   Limit  Requests Type Lim
         devbuf 10149   6454K    6455K  78643K     11239        0
            pcb    13      8K       8K  78643K        13        0
         rtable    62      2K       2K  78643K       112        0
         ifaddr    29      8K       8K  78643K        32        0
       counters    40     33K      33K  78643K        40        0
       ioctlops     0      0K       4K  78643K      1479        0
            iov     0      0K       0K  78643K         1        0
          mount     1      1K       1K  78643K         1        0
            log     0      0K       0K  78643K         5        0
         vnodes  1167     73K      73K  78643K      1180        0
      UFS quota     1     32K      32K  78643K         1        0
      UFS mount     5     36K      36K  78643K         5        0
            shm     2      1K       1K  78643K         2        0
         VM map     2      1K       1K  78643K         2        0
            sem     2      0K       0K  78643K         2        0
        dirhash    12      2K       2K  78643K        12        0
           ACPI  1697    195K     286K  78643K     12548        0
      file desc     1      0K       0K  78643K         1        0
           proc    67     87K      87K  78643K       282        0
    NFS srvsock     1      0K       0K  78643K         1        0
     NFS daemon     1     16K      16K  78643K         1        0
       in_multi    11      0K       0K  78643K        11        0
    ether_multi     1      0K       0K  78643K         1        0
    ISOFS mount     1     32K      32K  78643K         1        0
  MSDOSFS mount     1     16K      16K  78643K         1        0
           ttys    25    122K     122K  78643K        25        0
           exec     0      0K       2K  78643K       432        0
            tdb     3      0K       0K  78643K         3        0
        pagedep     1      8K       8K  78643K         1        0
       inodedep     1     32K      32K  78643K         1        0
         newblk     1      0K       0K  78643K         1        0
        VM swap     7     26K      26K  78643K         7        0
       UVM amap    82      3K       5K  78643K      2179        0
       UVM aobj     3      2K       2K  78643K         3        0
        memdesc     1      4K       4K  78643K         1        0
    crypto data     1      1K       1K  78643K         1        0
            NDP     4      0K       0K  78643K         4        0
           temp    26   4698K    4757K  78643K      3281        0
         kqueue    11     16K      18K  78643K        24        0
      SYN cache     2     16K      16K  78643K         2        0
ddb{0}> show all pools
Name      Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
plcache    128       22    0        0     1     0     1     1     0     8    0
rtpcb      120       17    0       14     1     0     1     1     0     8    0
rtentry    112       23    0        1     1     0     1     1     0     8    0
unpcb      136       35    0       20     1     0     1     1     0     8    0
syncache   296        5    0        5     1     0     1     1     0     8    1
tcpcb      736        8    0        5     1     0     1     1     0     8    0
arp        120        2    0        0     1     0     1     1     0     8    0
inpcb      304       32    0       26     1     0     1     1     0     8    0
pfosfp      40     1428    0     1005     5     0     5     5     0     8    0
pfosfpen   112     1428    0      714    21     0    21    21     0     8    0
pfstitem    24        9    0        0     1     0     1     1     0     8    0
pfstkey    112        9    0        0     1     0     1     1     0     8    0
pfstate    320        9    0        0     1     0     1     1     0     8    0
pfrule     1360      21    0       16     2     1     1     2     0     8    0
art_heap8  4096       1    0        0     1     0     1     1     0     8    0
art_heap4  256       96    0        0     6     0     6     6     0     8    0
art_table   32       97    0        0     1     0     1     1     0     8    0
art_node    16       22    0        2     1     0     1     1     0     8    0
dirhash    1024      17    0        0     3     0     3     3     0     8    0
dino2pl    256     1425    0       39    87     0    87    87     0     8    0
ffsino     272     1425    0       39    93     0    93    93     0     8    0
nchpl      144     1600    0       49    58     0    58    58     0     8    0
uvmvnodes   80     1435    0        0    30     0    30    30     0     8    0
vnodes     224     1435    0        0    85     0    85    85     0     8    0
namei      1024    4425    0     4425     1     0     1     1     0     8    1
percpumem   16       32    0        0     1     0     1     1     0     8    0
scxspl     216     3824    0     3822     2     1     1     2     0     8    0
plimitpl   152       16    0        9     1     0     1     1     0     8    0
sigapl     424      320    0      291     4     0     4     4     0     8    0
knotepl    120       52    0        0     2     0     2     2     0     8    0
kqueuepl   216       20    0       13     1     0     1     1     0     8    0
pipepl     336       86    0       83     1     0     1     1     0     8    0
fdescpl    496      306    0      291     3     0     3     3     0     8    0
filepl     152     1148    0     1087     3     0     3     3     0     8    0
lockfpl    104        6    0        4     1     0     1     1     0     8    0
lockfspl    48        4    0        2     1     0     1     1     0     8    0
sessionpl  144       18    0        9     1     0     1     1     0     8    0
pgrppl      48       18    0        9     1     0     1     1     0     8    0
ucredpl     96       69    0       57     1     0     1     1     0     8    0
zombiepl   144      291    0      291     1     0     1     1     0     8    1
processpl  1064     320    0      291     3     0     3     3     0     8    0
procpl     672      320    0      291     3     0     3     3     0     8    0
sockpl     480       84    0       60     4     0     4     4     0     8    0
mcl8k      8192       3    0        0     1     0     1     1     0     8    0
mcl4k      4096       2    0        0     1     0     1     1     0     8    0
mcl2k      2048      57    0        0     8     0     8     8     0     8    0
mtagpl      96        1    0        0     1     0     1     1     0     8    0
mbufpl     256      105    0        0     7     0     7     7     0     8    0
bufpl      288     2024    0       93   138     0   138   138     0     8    0
anonpl      24    41240    0    38925    17     2    15    17     0   186    1
amapchunkpl 152    3965    0     3822     7     1     6     7     0   158    0
amappl16   200       34    0       33     2     1     1     1     0     8    0
amappl15   192       67    0       64     1     0     1     1     0     8    0
amappl13   176       34    0       33     1     0     1     1     0     8    0
amappl12   168       19    0       18     1     0     1     1     0     8    0
amappl11   160       49    0       35     1     0     1     1     0     8    0
amappl10   152        2    0        0     1     0     1     1     0     8    0
amappl9    144      442    0      439     1     0     1     1     0     8    0
amappl8    136      370    0      367     1     0     1     1     0     8    0
amappl7    128       65    0       62     1     0     1     1     0     8    0
amappl6    120      116    0      103     1     0     1     1     0     8    0
amappl5    112      189    0      175     1     0     1     1     0     8    0
amappl4    104      642    0      622     1     0     1     1     0     8    0
amappl3     96      121    0      112     1     0     1     1     0     8    0
amappl2     88      369    0      331     1     0     1     1     0     8    0
amappl1     80     8804    0     8423     9     0     9     9     0     8    0
amappl      88     1898    0     1834     2     0     2     2     0    92    0
dma4096    4096       1    0        1     1     1     0     1     0     8    0
dma1024    1024       1    0        0     1     0     1     1     0     8    0
dma256     256        6    0        6     1     1     0     1     0     8    0
dma128     128      253    0      253     1     1     0     1     0     8    0
dma64       64        6    0        6     1     1     0     1     0     8    0
dma32       32        7    0        7     1     1     0     1     0     8    0
dma16       16       18    0       17     1     0     1     1     0     8    0
aobjpl      72        2    0        0     1     0     1     1     0     8    0
uaddrrnd    24      306    0      291     1     0     1     1     0     8    0
uaddrbest   32        2    0        0     1     0     1     1     0     8    0
uaddr       24      306    0      291     1     0     1     1     0     8    0
vmmpekpl   168     6250    0     6235     1     0     1     1     0     8    0
vmmpepl    168    26576    0    25733    41     0    41    41     0   357    3
vmsppl     368      305    0      291     2     0     2     2     0     8    0
rwobjpl     56     9581    0     7588    29     0    29    29     0     8    0
pdppl      4096     619    0      582    51    14    37    45     0     8    0
pvpl        32   134486    0   130337    37     0    37    37     0   265    2
pmappl     248      305    0      291     2     0     2     2     0     8    0
extentpl    40       58    0       38     1     0     1     1     0     8    0
phpool     112      406    0       17    12     0    12    12     0     8    0
ddb{0}> machine ddbcpu 0
Invalid cpu 0
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
witness_checkorder(ffffffff8291bdd0,9,0) at witness_checkorder+0x10b7 witness_debugger sys/kern/subr_witness.c:2502 [inline]
witness_checkorder(ffffffff8291bdd0,9,0) at witness_checkorder+0x10b7 sys/kern/subr_witness.c:1105
rw_enter_write(ffffffff8291bdc0) at rw_enter_write+0x5b sys/kern/kern_rwlock.c:128
uvn_io(fffffd806d66d790,ffff8000211cd938,1,202,0) at uvn_io+0x418 sys/uvm/uvm_vnode.c:1243
uvn_get(fffffd806d66d790,1000,ffff8000211cd9f8,ffff8000211cd9c8,0,1,71fb8b939eed8b75,0) at uvn_get+0x1df sys/uvm/uvm_vnode.c:1076
uvm_fault_lower(ffff8000211cdb60,ffff8000211cdb98,ffff8000211cdae0,0) at uvm_fault_lower+0x36c sys/uvm/uvm_fault.c:1275
uvm_fault(fffffd806f1efe68,20001000,0,1) at uvm_fault+0x248
kpageflttrap(ffff8000211cdcf0,20001000) at kpageflttrap+0x209
kerntrap(ffff8000211cdcf0) at kerntrap+0xef sys/arch/amd64/amd64/trap.c:318
alltraps_kern_meltdown() at alltraps_kern_meltdown+0x7b
copyin() at copyin+0x53
VOP_IOCTL(fffffd806f8159c8,c0104451,ffff8000211ce0d0,2,fffffd807f7d8960,ffff8000ffff6a90) at VOP_IOCTL+0x96 sys/kern/vfs_vops.c:264
vn_ioctl(fffffd806e51a860,c0104451,ffff8000211ce0d0,ffff8000ffff6a90) at vn_ioctl+0xbc sys/kern/vfs_vnops.c:531
sys_ioctl(ffff8000ffff6a90,ffff8000211ce1e8,ffff8000211ce240) at sys_ioctl+0x4a2
syscall(ffff8000211ce2b0) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff8000211ce2b0) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffbec30, count: -16
ddb{0}> machine ddbcpu 1
Stopped at      x86_ipi_db+0x1a:        addq    $0x8,%rsp
ddb{1}> trace
x86_ipi_db(ffff800020ce8ff0) at x86_ipi_db+0x1a sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x312 sys/dev/acpi/acpicpu.c:1206
sched_idle(ffff800020ce8ff0) at sched_idle+0x417 sys/kern/kern_sched.c:178
end trace frame: 0x0, count: -5
ddb{1}> 

Crashes (19):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-openbsd-multicore 2022/03/15 20:25 openbsd 4141bb4bac68 9e8eaa75 .config log report syz C witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/01 02:15 openbsd 8a504047818d 45a13a73 .config log report syz C witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/02/24 22:28 openbsd 83bd653d9a39 b28851a4 .config log report syz witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/02/04 15:02 openbsd ceda8ed5a346 a3e470b2 .config log report syz witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/28 11:55 openbsd c0f44e363efa ee339263 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/27 20:02 openbsd 814fd41b4cce 89bc8608 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/27 13:20 openbsd a8034bb6cd6c 89bc8608 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/24 23:02 openbsd 2569963f91dd 89bc8608 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/24 21:28 openbsd 475618162a2c 89bc8608 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/24 00:26 openbsd 2426d74d418a 5ff41e94 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/22 23:39 openbsd bf088e2b2bca 5ff41e94 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/16 18:10 openbsd f1075cc4acbf 46cc3b21 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/16 15:19 openbsd f1075cc4acbf 46cc3b21 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/05 15:34 openbsd 8043fa601940 45a13a73 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/04 10:20 openbsd 5c205e9bf3d5 45a13a73 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/03 19:58 openbsd 0d6a9aa9baac 45a13a73 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/03/01 01:54 openbsd 8a504047818d 45a13a73 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/02/24 22:07 openbsd 83bd653d9a39 b28851a4 .config log report witness: reversal: pf_lock netlock
ci-openbsd-multicore 2022/02/04 14:42 openbsd ceda8ed5a346 a3e470b2 .config log report witness: reversal: pf_lock netlock
* Struck through repros no longer work on HEAD.