syzbot


panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock (5)

Status: fixed on 2022/03/22 22:28
Reported-by: syzbot+ebe3f03a472fecf5e42e@syzkaller.appspotmail.com
Fix commit: d25fea59a0de For raw IP packets rip_input() traverses the loop of all PCBs. From there it calls sbappendaddr() while holding the raw table mutex. This ends in sorwakeup() where we finally grab the kernel lock while holding a mutex. Witness detects this misuse. Use the same solution as for PCB notify. Collect the affected PCBs in a temporary list. The list is protected by exclusive net lock. syzbot+ebe3f03a472fecf5e42e@syzkaller.appspotmail.com OK claudio@
First crash: 250d, last: 250d
similar bugs (5):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
openbsd panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock (3) C 2 448d 451d 3/3 fixed on 2021/09/07 20:47
openbsd panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock (4) C 249 250d 251d 3/3 fixed on 2022/03/22 10:24
openbsd panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock 33 1188d 1193d 3/3 fixed on 2019/08/27 19:45
openbsd panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock (2) C 152 452d 1173d 3/3 fixed on 2021/09/01 16:54
openbsd panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock (6) C 80 249d 250d 3/3 fixed on 2022/03/24 08:53

Sample crash report:
login: panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
Stopped at      db_enter+0x18:  addq    $0x8,%rsp
    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
*165426  30709      0           0          0    0  syz-executor1476446240
 516859  70260      0           0          0    1  syz-executor1476446240
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff825a2db5) at panic+0x177 sys/kern/subr_prf.c:202
witness_checkorder(ffffffff829f3ad8,9,0) at witness_checkorder+0x116d sys/kern/subr_witness.c:833
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 read_rflags machine/cpufunc.h:195 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 intr_disable machine/cpufunc.h:216 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 sys/kern/kern_lock.c:142
selwakeup(fffffd807f5474f0) at selwakeup+0x16 klist_empty sys/sys/event.h:361 [inline]
selwakeup(fffffd807f5474f0) at selwakeup+0x16 sys/kern/sys_generic.c:885
sorwakeup(fffffd807f5473d8) at sorwakeup+0xc9 sys/kern/uipc_socket.c:1699
rip_input(ffff80002120fef8,ffff80002120ff04,0,2) at rip_input+0x3b0 sys/netinet/raw_ip.c:188
ip_deliver(ffff80002120fef8,ffff80002120ff04,0,2) at ip_deliver+0x322 sys/netinet/ip_input.c:657
ip_ours(ffff80002120fef8,ffff80002120ff04,fffffd806d2926bc,0) at ip_ours+0x3ba sys/netinet/ip_input.c:616
ip_input_if(ffff80002120fef8,ffff80002120ff04,4,0,ffff800000689000) at ip_input_if+0x2a1
ipv4_input(ffff800000689000,fffffd806d292600) at ipv4_input+0x48 sys/netinet/ip_input.c:242
if_input_local(ffff800000689000,fffffd806d292600,2) at if_input_local+0x10e sys/net/if.c:774
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 ip_mloopback sys/netinet/ip_output.c:1791 [inline]
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 sys/netinet/ip_output.c:332
rip_output(fffffd806d292a00,fffffd807f547978,ffff800021210150,1) at rip_output+0x2cb sys/netinet/raw_ip.c:302
end trace frame: 0xffff8000212101d0, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports.  Insufficient info makes it difficult to find and fix bugs.
ddb{0}> 
ddb{0}> set $lines = 0
ddb{0}> set $maxwidth = 0
ddb{0}> show panic
*cpu0: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff825a2db5) at panic+0x177 sys/kern/subr_prf.c:202
witness_checkorder(ffffffff829f3ad8,9,0) at witness_checkorder+0x116d sys/kern/subr_witness.c:833
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 read_rflags machine/cpufunc.h:195 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 intr_disable machine/cpufunc.h:216 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 sys/kern/kern_lock.c:142
selwakeup(fffffd807f5474f0) at selwakeup+0x16 klist_empty sys/sys/event.h:361 [inline]
selwakeup(fffffd807f5474f0) at selwakeup+0x16 sys/kern/sys_generic.c:885
sorwakeup(fffffd807f5473d8) at sorwakeup+0xc9 sys/kern/uipc_socket.c:1699
rip_input(ffff80002120fef8,ffff80002120ff04,0,2) at rip_input+0x3b0 sys/netinet/raw_ip.c:188
ip_deliver(ffff80002120fef8,ffff80002120ff04,0,2) at ip_deliver+0x322 sys/netinet/ip_input.c:657
ip_ours(ffff80002120fef8,ffff80002120ff04,fffffd806d2926bc,0) at ip_ours+0x3ba sys/netinet/ip_input.c:616
ip_input_if(ffff80002120fef8,ffff80002120ff04,4,0,ffff800000689000) at ip_input_if+0x2a1
ipv4_input(ffff800000689000,fffffd806d292600) at ipv4_input+0x48 sys/netinet/ip_input.c:242
if_input_local(ffff800000689000,fffffd806d292600,2) at if_input_local+0x10e sys/net/if.c:774
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 ip_mloopback sys/netinet/ip_output.c:1791 [inline]
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 sys/netinet/ip_output.c:332
rip_output(fffffd806d292a00,fffffd807f547978,ffff800021210150,1) at rip_output+0x2cb sys/netinet/raw_ip.c:302
rip_usrreq(fffffd807f547978,9,fffffd806d292a00,0,0,ffff80002119ed20) at rip_usrreq+0x49c sys/netinet/raw_ip.c:554
sosend(fffffd807f547978,0,ffff8000212103e0,0,0,0) at sosend+0x632 sys/kern/uipc_socket.c:582
dofilewritev(ffff80002119ed20,3,ffff8000212103e0,0,ffff8000212104e0) at dofilewritev+0x19c sys/kern/sys_generic.c:381
sys_writev(ffff80002119ed20,ffff800021210488,ffff8000212104e0) at sys_writev+0xa7 sys/kern/sys_generic.c:328
syscall(ffff800021210550) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff800021210550) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffd4c80, count: -20
ddb{0}> show registers
rdi                                0
rsi                              0x1
rbp               0xffff80002120fa80
rbx               0xffffffff8298dbff    cpu_info_full_primary+0x2bff
rdx                            0x3fd
rcx                                0
rax                             0x68
r8                 0x101010101010101
r9                0x8080808080808080
r10               0x66d36eb50989c84f
r11               0x80c3ad1c49cfcd3d
r12               0xffffffff8298da00    cpu_info_full_primary+0x2a00
r13                                0
r14                                0
r15                              0x1
rip               0xffffffff811e4c58    db_enter+0x18
cs                               0x8
rflags                         0x246
rsp               0xffff80002120fa70
ss                              0x10
db_enter+0x18:  addq    $0x8,%rsp
ddb{0}> show proc
PROC (syz-executor1476446240) pid=165426 stat=onproc
    flags process=0 proc=0
    pri=50, usrpri=50, nice=20
    forw=0xffffffffffffffff, list=0xffff80002119f500,0xffff80002119efd0
    process=0xffff8000ffffa998 user=0xffff80002120b000, vmspace=0xfffffd806cc0e5d0
    estcpu=0, cpticks=1, pctcpu=0.0
    user=0, sys=1, intr=0
ddb{0}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
 24913  196049  70260      0  2           0                syz-executor1476446240
 94350  451574  95346      0  2           0                syz-executor1476446240
*30709  165426  39723      0  7           0                syz-executor1476446240
 96279     507  89009      0  2           0                syz-executor1476446240
 91902  132793  97861      0  3           0  biowait       syz-executor1476446240
 80289   16918  97861      0  3           0  biowait       syz-executor1476446240
 95346  305639  97861      0  3        0x80  nanoslp       syz-executor1476446240
 89009   91194  97861      0  3        0x80  nanoslp       syz-executor1476446240
 39723  133206  97861      0  3        0x80  nanoslp       syz-executor1476446240
 14233  103357  97861      0  3           0  biowait       syz-executor1476446240
 41614     174  97861      0  3           0  biowait       syz-executor1476446240
 70260  516859  97861      0  7           0                syz-executor1476446240
 97861  293920   2990      0  3        0x82  nanoslp       syz-executor1476446240
  2990  431273  88953      0  3    0x10008a  sigsusp       ksh
 88953  480109  50642      0  2        0x12                sshd
 78913  428607      1      0  3    0x100083  ttyin         getty
 50642  378402      1      0  3        0x88  kqread        sshd
 16928  199573  62992     74  3   0x1100092  bpf           pflogd
 62992  405173      1      0  3        0x80  netio         pflogd
 80236  178160  57747     73  3   0x1100090  kqread        syslogd
 57747  187692      1      0  3    0x100082  netio         syslogd
 19598  386295      1      0  3    0x100080  kqread        resolvd
 72408  290208  97513     77  3    0x100092  kqread        dhcpleased
 87904  457670  97513     77  3    0x100092  kqread        dhcpleased
 97513  316175      1      0  3        0x80  kqread        dhcpleased
 27711  309851      0      0  3     0x14200  bored         smr
  4608  251054      0      0  2     0x14200                zerothread
 72731   13509      0      0  3     0x14200  aiodoned      aiodoned
  2608  509137      0      0  3     0x14200  syncer        update
 50105  451851      0      0  3     0x14200  cleaner       cleaner
 98319  495471      0      0  3     0x14200  reaper        reaper
 10222   13113      0      0  3     0x14200  pgdaemon      pagedaemon
 54125   87219      0      0  3     0x14200  bored         viomb
 48831  340993      0      0  3  0x40014200  acpi0         acpi0
 64245  204713      0      0  3  0x40014200                idle1
 98731  267150      0      0  3     0x14200  bored         softnet
 78123  220788      0      0  3     0x14200  bored         systqmp
 42388   16751      0      0  3     0x14200  bored         systq
 65358   81752      0      0  3  0x40014200  bored         softclock
 26152  263897      0      0  3  0x40014200                idle0
     1  202524      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper
ddb{0}> show all locks
CPU 0:
exclusive mutex &table->inpt_mtx r = 0 (0xffffffff82b03010)
#0  witness_lock+0x44d
#1  mtx_enter_try+0x100
#2  mtx_enter+0x4b sys/kern/kern_lock.c:266
#3  rip_input+0x135
#4  ip_deliver+0x322 sys/netinet/ip_input.c:657
#5  ip_ours+0x3ba sys/netinet/ip_input.c:616
#6  ip_input_if+0x2a1
#7  ipv4_input+0x48 sys/netinet/ip_input.c:242
#8  if_input_local+0x10e sys/net/if.c:774
#9  ip_output+0xb05 ip_mloopback sys/netinet/ip_output.c:1791 [inline]
#9  ip_output+0xb05 sys/netinet/ip_output.c:332
#10 rip_output+0x2cb sys/netinet/raw_ip.c:302
#11 rip_usrreq+0x49c sys/netinet/raw_ip.c:554
#12 sosend+0x632 sys/kern/uipc_socket.c:582
#13 dofilewritev+0x19c sys/kern/sys_generic.c:381
#14 sys_writev+0xa7 sys/kern/sys_generic.c:328
#15 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#15 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#16 Xsyscall+0x128
Process 30709 (syz-executor1476446240) thread 0xffff80002119ed20 (165426)
exclusive rwlock netlock r = 0 (0xffffffff82904160)
#0  witness_lock+0x44d
#1  solock+0x86 sys/kern/uipc_socket2.c:295
#2  sosend+0x517 sys/kern/uipc_socket.c:570
#3  dofilewritev+0x19c sys/kern/sys_generic.c:381
#4  sys_writev+0xa7 sys/kern/sys_generic.c:328
#5  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#5  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#6  Xsyscall+0x128
exclusive mutex &table->inpt_mtx r = 0 (0xffffffff82b03010)
#0  witness_lock+0x44d
#1  mtx_enter_try+0x100
#2  mtx_enter+0x4b sys/kern/kern_lock.c:266
#3  rip_input+0x135
#4  ip_deliver+0x322 sys/netinet/ip_input.c:657
#5  ip_ours+0x3ba sys/netinet/ip_input.c:616
#6  ip_input_if+0x2a1
#7  ipv4_input+0x48 sys/netinet/ip_input.c:242
#8  if_input_local+0x10e sys/net/if.c:774
#9  ip_output+0xb05 ip_mloopback sys/netinet/ip_output.c:1791 [inline]
#9  ip_output+0xb05 sys/netinet/ip_output.c:332
#10 rip_output+0x2cb sys/netinet/raw_ip.c:302
#11 rip_usrreq+0x49c sys/netinet/raw_ip.c:554
#12 sosend+0x632 sys/kern/uipc_socket.c:582
#13 dofilewritev+0x19c sys/kern/sys_generic.c:381
#14 sys_writev+0xa7 sys/kern/sys_generic.c:328
#15 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#15 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#16 Xsyscall+0x128
Process 91902 (syz-executor1476446240) thread 0xffff80002119fce0 (132793)
exclusive rrwlock inode r = 0 (0xfffffd806cc2c3c8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  ufs_ihashins+0x42 sys/ufs/ufs/ufs_ihash.c:140
#5  ffs_vget+0x141 sys/ufs/ffs/ffs_vfsops.c:1347
#6  ffs_inode_alloc+0x1be sys/ufs/ffs/ffs_alloc.c:394
#7  ufs_mkdir+0xf4 sys/ufs/ufs/ufs_vnops.c:1162
#8  VOP_MKDIR+0xbf sys/kern/vfs_vops.c:404
#9  domkdirat+0x121 sys/kern/vfs_syscalls.c:3101
#10 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#10 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#11 Xsyscall+0x128
exclusive rrwlock inode r = 0 (0xfffffd806d4111b8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  vn_lock+0x84 sys/kern/vfs_vnops.c:579
#5  vfs_lookup+0xd1 sys/kern/vfs_lookup.c:413
#6  namei+0x36a sys/kern/vfs_lookup.c:245
#7  domkdirat+0x75 sys/kern/vfs_syscalls.c:3086
#8  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#8  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#9  Xsyscall+0x128
Process 80289 (syz-executor1476446240) thread 0xffff800021143268 (16918)
exclusive rrwlock inode r = 0 (0xfffffd806cc2c6f8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  ufs_ihashins+0x42 sys/ufs/ufs/ufs_ihash.c:140
#5  ffs_vget+0x141 sys/ufs/ffs/ffs_vfsops.c:1347
#6  ffs_inode_alloc+0x1be sys/ufs/ffs/ffs_alloc.c:394
#7  ufs_mkdir+0xf4 sys/ufs/ufs/ufs_vnops.c:1162
#8  VOP_MKDIR+0xbf sys/kern/vfs_vops.c:404
#9  domkdirat+0x121 sys/kern/vfs_syscalls.c:3101
#10 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#10 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#11 Xsyscall+0x128
exclusive rrwlock inode r = 0 (0xfffffd806d4115f8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  vn_lock+0x84 sys/kern/vfs_vnops.c:579
#5  vfs_lookup+0xd1 sys/kern/vfs_lookup.c:413
#6  namei+0x36a sys/kern/vfs_lookup.c:245
#7  domkdirat+0x75 sys/kern/vfs_syscalls.c:3086
#8  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#8  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#9  Xsyscall+0x128
Process 14233 (syz-executor1476446240) thread 0xffff800021142548 (103357)
exclusive rrwlock inode r = 0 (0xfffffd806cc2cd58)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  ufs_ihashins+0x42 sys/ufs/ufs/ufs_ihash.c:140
#5  ffs_vget+0x141 sys/ufs/ffs/ffs_vfsops.c:1347
#6  ffs_inode_alloc+0x1be sys/ufs/ffs/ffs_alloc.c:394
#7  ufs_mkdir+0xf4 sys/ufs/ufs/ufs_vnops.c:1162
#8  VOP_MKDIR+0xbf sys/kern/vfs_vops.c:404
#9  domkdirat+0x121 sys/kern/vfs_syscalls.c:3101
#10 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#10 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#11 Xsyscall+0x128
exclusive rrwlock inode r = 0 (0xfffffd806d4110a8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  vn_lock+0x84 sys/kern/vfs_vnops.c:579
#5  vfs_lookup+0xd1 sys/kern/vfs_lookup.c:413
#6  namei+0x36a sys/kern/vfs_lookup.c:245
#7  domkdirat+0x75 sys/kern/vfs_syscalls.c:3086
#8  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#8  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#9  Xsyscall+0x128
Process 41614 (syz-executor1476446240) thread 0xffff8000ffff7cf0 (174)
exclusive rrwlock inode r = 0 (0xfffffd806d411e78)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  ufs_ihashins+0x42 sys/ufs/ufs/ufs_ihash.c:140
#5  ffs_vget+0x141 sys/ufs/ffs/ffs_vfsops.c:1347
#6  ffs_inode_alloc+0x1be sys/ufs/ffs/ffs_alloc.c:394
#7  ufs_mkdir+0xf4 sys/ufs/ufs/ufs_vnops.c:1162
#8  VOP_MKDIR+0xbf sys/kern/vfs_vops.c:404
#9  domkdirat+0x121 sys/kern/vfs_syscalls.c:3101
#10 syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#10 syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#11 Xsyscall+0x128
exclusive rrwlock inode r = 0 (0xfffffd806d4113d8)
#0  witness_lock+0x44d
#1  rw_enter+0x3e1 sys/kern/kern_rwlock.c:310
#2  rrw_enter+0x8b sys/kern/kern_rwlock.c:461
#3  VOP_LOCK+0x87 sys/kern/vfs_vops.c:534
#4  vn_lock+0x84 sys/kern/vfs_vnops.c:579
#5  vfs_lookup+0xd1 sys/kern/vfs_lookup.c:413
#6  namei+0x36a sys/kern/vfs_lookup.c:245
#7  domkdirat+0x75 sys/kern/vfs_syscalls.c:3086
#8  syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
#8  syscall+0x489 sys/arch/amd64/amd64/trap.c:585
#9  Xsyscall+0x128
Process 70260 (syz-executor1476446240) thread 0xffff8000ffff67f0 (516859)
exclusive kernel_lock &kernel_lock r = 0 (0xffffffff829f3ad8)
#0  witness_lock+0x44d
#1  syscall+0x3ef mi_syscall sys/sys/syscall_mi.h:93 [inline]
#1  syscall+0x3ef sys/arch/amd64/amd64/trap.c:585
#2  Xsyscall+0x128
ddb{0}> show malloc
           Type InUse  MemUse  HighUse   Limit  Requests Type Lim
         devbuf 10149   6454K    6455K  78643K     11239        0
            pcb    13      8K       8K  78643K        17        0
         rtable    62      2K       2K  78643K       112        0
         ifaddr    29      8K       8K  78643K        32        0
       counters    40     33K      33K  78643K        40        0
       ioctlops     0      0K       4K  78643K      1479        0
          mount     1      1K       1K  78643K         1        0
            log     0      0K       0K  78643K         5        0
         vnodes  1167     73K      73K  78643K      1180        0
      UFS quota     1     32K      32K  78643K         1        0
      UFS mount     5     36K      36K  78643K         5        0
            shm     2      1K       1K  78643K         2        0
         VM map     2      1K       1K  78643K         2        0
            sem     2      0K       0K  78643K         2        0
        dirhash    12      2K       2K  78643K        12        0
           ACPI  1697    195K     286K  78643K     12548        0
      file desc     1      0K       0K  78643K         1        0
           proc    67     87K      87K  78643K       282        0
    NFS srvsock     1      0K       0K  78643K         1        0
     NFS daemon     1     16K      16K  78643K         1        0
       in_multi    11      0K       0K  78643K        11        0
    ether_multi     1      0K       0K  78643K         1        0
    ISOFS mount     1     32K      32K  78643K         1        0
  MSDOSFS mount     1     16K      16K  78643K         1        0
           ttys    25    122K     122K  78643K        25        0
           exec     0      0K       2K  78643K       432        0
            tdb     3      0K       0K  78643K         3        0
        pagedep     1      8K       8K  78643K         1        0
       inodedep     1     32K      32K  78643K         1        0
         newblk     1      0K       0K  78643K         1        0
        VM swap     7     26K      26K  78643K         7        0
       UVM amap   102      5K       5K  78643K      2203        0
       UVM aobj     3      2K       2K  78643K         3        0
        memdesc     1      4K       4K  78643K         1        0
    crypto data     1      1K       1K  78643K         1        0
            NDP     4      0K       0K  78643K         4        0
           temp    24   4694K    4757K  78643K      3175        0
         kqueue    11     16K      18K  78643K        24        0
      SYN cache     2     16K      16K  78643K         2        0
ddb{0}> show all pools
Name      Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
plcache    128       22    0        0     1     0     1     1     0     8    0
rtpcb      120       17    0       14     1     0     1     1     0     8    0
rtentry    112       23    0        1     1     0     1     1     0     8    0
unpcb      136       35    0       20     1     0     1     1     0     8    0
syncache   296        5    0        5     1     0     1     1     0     8    1
tcpqe       32       13    0       13     1     1     0     1     0     8    0
tcpcb      736        8    0        5     1     0     1     1     0     8    0
arp        120        2    0        0     1     0     1     1     0     8    0
inpcb      312       35    0       26     1     0     1     1     0     8    0
pfosfp      40     1428    0     1005     5     0     5     5     0     8    0
pfosfpen   112     1428    0      714    21     0    21    21     0     8    0
pfstitem    24        8    0        0     1     0     1     1     0     8    0
pfstkey    112        8    0        0     1     0     1     1     0     8    0
pfstate    320        8    0        0     1     0     1     1     0     8    0
pfrule     1360      21    0       16     2     1     1     2     0     8    0
art_heap8  4096       1    0        0     1     0     1     1     0     8    0
art_heap4  256       97    0        0     7     0     7     7     0     8    0
art_table   32       98    0        0     1     0     1     1     0     8    0
art_node    16       22    0        2     1     0     1     1     0     8    0
dirhash    1024      17    0        0     3     0     3     3     0     8    0
dino2pl    256     1440    0       39    88     0    88    88     0     8    0
ffsino     272     1440    0       39    94     0    94    94     0     8    0
nchpl      144     1618    0       49    59     0    59    59     0     8    0
uvmvnodes   80     1450    0        0    30     0    30    30     0     8    0
vnodes     224     1450    0        0    86     0    86    86     0     8    0
namei      1024    4457    0     4453     2     0     2     2     0     8    0
percpumem   16       32    0        0     1     0     1     1     0     8    0
scxspl     216     3888    0     3884     2     0     2     2     0     8    1
plimitpl   152       16    0        9     1     0     1     1     0     8    0
sigapl     424      332    0      291     5     0     5     5     0     8    0
knotepl    120       47    0        0     2     0     2     2     0     8    0
kqueuepl   216       20    0       13     1     0     1     1     0     8    0
pipepl     336       86    0       83     1     0     1     1     0     8    0
fdescpl    496      318    0      291     4     0     4     4     0     8    0
filepl     152     1148    0     1087     3     0     3     3     0     8    0
lockfpl    104        6    0        4     1     0     1     1     0     8    0
lockfspl    48        4    0        2     1     0     1     1     0     8    0
sessionpl  144       18    0        9     1     0     1     1     0     8    0
pgrppl      48       18    0        9     1     0     1     1     0     8    0
ucredpl     96       69    0       57     1     0     1     1     0     8    0
zombiepl   144      291    0      291     1     0     1     1     0     8    1
processpl  1064     332    0      291     3     0     3     3     0     8    0
procpl     672      332    0      291     4     0     4     4     0     8    0
sockpl     480       87    0       60     4     0     4     4     0     8    0
mcl8k      8192       5    0        0     1     0     1     1     0     8    0
mcl4k      4096       2    0        0     1     0     1     1     0     8    0
mcl2k      2048      81    0        0    11     0    11    11     0     8    0
mtagpl      96        1    0        0     1     0     1     1     0     8    0
mbufpl     256      130    0        0     9     0     9     9     0     8    0
bufpl      288     2034    0       93   139     0   139   139     0     8    0
anonpl      24    41514    0    38951    19     2    17    17     0   186    1
amapchunkpl 152    4017    0     3844     8     1     7     7     0   158    0
amappl16   200       29    0       29     1     1     0     1     0     8    0
amappl15   192       67    0       64     1     0     1     1     0     8    0
amappl13   176       34    0       33     1     0     1     1     0     8    0
amappl12   168       11    0       11     1     0     1     1     0     8    1
amappl11   160       48    0       33     1     0     1     1     0     8    0
amappl10   152        4    0        1     1     0     1     1     0     8    0
amappl9    144      454    0      452     1     0     1     1     0     8    0
amappl8    136      370    0      367     1     0     1     1     0     8    0
amappl7    128       66    0       63     1     0     1     1     0     8    0
amappl6    120      127    0      101     1     0     1     1     0     8    0
amappl5    112      180    0      168     1     0     1     1     0     8    0
amappl4    104      638    0      616     1     0     1     1     0     8    0
amappl3     96      121    0      112     1     0     1     1     0     8    0
amappl2     88      370    0      332     1     0     1     1     0     8    0
amappl1     80     8795    0     8361     9     0     9     9     0     8    0
amappl      88     1921    0     1838     2     0     2     2     0    92    0
dma4096    4096       1    0        1     1     1     0     1     0     8    0
dma1024    1024       1    0        0     1     0     1     1     0     8    0
dma256     256        6    0        6     1     1     0     1     0     8    0
dma128     128      253    0      253     1     1     0     1     0     8    0
dma64       64        6    0        6     1     1     0     1     0     8    0
dma32       32        7    0        7     1     1     0     1     0     8    0
dma16       16       18    0       17     1     0     1     1     0     8    0
aobjpl      72        2    0        0     1     0     1     1     0     8    0
uaddrrnd    24      318    0      291     1     0     1     1     0     8    0
uaddrbest   32        2    0        0     1     0     1     1     0     8    0
uaddr       24      318    0      291     1     0     1     1     0     8    0
vmmpekpl   168     6234    0     6219     1     0     1     1     0     8    0
vmmpepl    168    26718    0    25653    47     0    47    47     0   357    0
vmsppl     368      317    0      291     3     0     3     3     0     8    0
rwobjpl     56     9557    0     7522    29     0    29    29     0     8    0
pdppl      4096     643    0      582    73    12    61    61     0     8    0
pvpl        32   135126    0   130420    40     1    39    39     0   265    1
pmappl     248      317    0      291     2     0     2     2     0     8    0
extentpl    40       58    0       38     1     0     1     1     0     8    0
phpool     112      440    0       15    13     0    13    13     0     8    0
ddb{0}> machine ddbcpu 0
Invalid cpu 0
ddb{0}> trace
db_enter() at db_enter+0x18 sys/arch/amd64/amd64/db_interface.c:440
panic(ffffffff825a2db5) at panic+0x177 sys/kern/subr_prf.c:202
witness_checkorder(ffffffff829f3ad8,9,0) at witness_checkorder+0x116d sys/kern/subr_witness.c:833
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 read_rflags machine/cpufunc.h:195 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 intr_disable machine/cpufunc.h:216 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0xa1 sys/kern/kern_lock.c:142
selwakeup(fffffd807f5474f0) at selwakeup+0x16 klist_empty sys/sys/event.h:361 [inline]
selwakeup(fffffd807f5474f0) at selwakeup+0x16 sys/kern/sys_generic.c:885
sorwakeup(fffffd807f5473d8) at sorwakeup+0xc9 sys/kern/uipc_socket.c:1699
rip_input(ffff80002120fef8,ffff80002120ff04,0,2) at rip_input+0x3b0 sys/netinet/raw_ip.c:188
ip_deliver(ffff80002120fef8,ffff80002120ff04,0,2) at ip_deliver+0x322 sys/netinet/ip_input.c:657
ip_ours(ffff80002120fef8,ffff80002120ff04,fffffd806d2926bc,0) at ip_ours+0x3ba sys/netinet/ip_input.c:616
ip_input_if(ffff80002120fef8,ffff80002120ff04,4,0,ffff800000689000) at ip_input_if+0x2a1
ipv4_input(ffff800000689000,fffffd806d292600) at ipv4_input+0x48 sys/netinet/ip_input.c:242
if_input_local(ffff800000689000,fffffd806d292600,2) at if_input_local+0x10e sys/net/if.c:774
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 ip_mloopback sys/netinet/ip_output.c:1791 [inline]
ip_output(fffffd806d292a00,0,fffffd806f687420,20,0,fffffd806f6873a8,46b983d8eac4981a) at ip_output+0xb05 sys/netinet/ip_output.c:332
rip_output(fffffd806d292a00,fffffd807f547978,ffff800021210150,1) at rip_output+0x2cb sys/netinet/raw_ip.c:302
rip_usrreq(fffffd807f547978,9,fffffd806d292a00,0,0,ffff80002119ed20) at rip_usrreq+0x49c sys/netinet/raw_ip.c:554
sosend(fffffd807f547978,0,ffff8000212103e0,0,0,0) at sosend+0x632 sys/kern/uipc_socket.c:582
dofilewritev(ffff80002119ed20,3,ffff8000212103e0,0,ffff8000212104e0) at dofilewritev+0x19c sys/kern/sys_generic.c:381
sys_writev(ffff80002119ed20,ffff800021210488,ffff8000212104e0) at sys_writev+0xa7 sys/kern/sys_generic.c:328
syscall(ffff800021210550) at syscall+0x489 mi_syscall sys/sys/syscall_mi.h:102 [inline]
syscall(ffff800021210550) at syscall+0x489 sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffd4c80, count: -20
ddb{0}> machine ddbcpu 1
Stopped at      x86_ipi_db+0x1a:        addq    $0x8,%rsp
x86_ipi_db(ffff800020ce8ff0) at x86_ipi_db+0x1a sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_lock(ffffffff829f38d0) at __mp_lock+0x122 __mp_lock_spin sys/kern/kern_lock.c:116 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0x122 sys/kern/kern_lock.c:147
syscall(ffff8000211d4830) at syscall+0x3ef mi_syscall sys/sys/syscall_mi.h:93 [inline]
syscall(ffff8000211d4830) at syscall+0x3ef sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffd4c80, count: 9
ddb{1}> trace
x86_ipi_db(ffff800020ce8ff0) at x86_ipi_db+0x1a sys/arch/amd64/amd64/db_interface.c:393
x86_ipi_handler() at x86_ipi_handler+0xb7 sys/arch/amd64/amd64/ipi.c:106
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
__mp_lock(ffffffff829f38d0) at __mp_lock+0x122 __mp_lock_spin sys/kern/kern_lock.c:116 [inline]
__mp_lock(ffffffff829f38d0) at __mp_lock+0x122 sys/kern/kern_lock.c:147
syscall(ffff8000211d4830) at syscall+0x3ef mi_syscall sys/sys/syscall_mi.h:93 [inline]
syscall(ffff8000211d4830) at syscall+0x3ef sys/arch/amd64/amd64/trap.c:585
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffd4c80, count: -6
ddb{1}> 

Crashes (104):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-openbsd-multicore 2022/03/22 17:58 openbsd 63abc0ec39b5 d88ef0c5 .config log report syz C panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:46 openbsd 63abc0ec39b5 d88ef0c5 .config log report syz C panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 10:38 openbsd 63abc0ec39b5 d88ef0c5 .config log report syz C panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 19:17 openbsd 63abc0ec39b5 d88ef0c5 .config log report syz panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 21:53 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 21:39 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 20:20 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 18:59 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 17:13 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 16:48 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:21 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:20 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:17 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:14 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:11 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:09 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:08 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:06 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 15:06 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:59 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:56 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:55 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:48 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:16 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:14 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 14:11 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:47 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:42 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:33 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:31 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:19 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:15 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:07 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 13:01 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:56 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:53 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:39 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:37 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:35 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:32 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 12:14 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 11:26 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 11:25 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 11:22 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 11:20 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
ci-openbsd-multicore 2022/03/22 11:08 openbsd 63abc0ec39b5 d88ef0c5 .config log report panic: acquiring blockable sleep lock with spinlock or critical section held (kernel_lock) &kernel_lock
* Struck through repros no longer work on HEAD.