syzbot


panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex: (2)

Status: fixed on 2022/12/16 09:36
Fix commit: bc440b67fce0 Don't hold nic.mu when calling n.linkResQueue.cancel.
First crash: 713d, last: 711d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
gvisor panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex: C 1246 713d 719d 14/26 fixed on 2022/12/13 21:58

Sample crash report:
panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
goroutine 268 [running]:
gvisor.dev/gvisor/pkg/sync/locking.(*ancestorsAtomicPtrMap).RangeRepeatable(0xc0002d8140, 0xc000ba9290)
	bazel-out/k8-fastbuild-ST-3dcbe13c9b87/bin/pkg/sync/locking/atomicptrmap_ancestors_unsafe.go:441 +0x1dc
gvisor.dev/gvisor/pkg/sync/locking.checkLock(0xc0002d82d0, 0xc0002d8120, {0x0, 0x0, 0x0})
	pkg/sync/locking/lockdep.go:98 +0x373
gvisor.dev/gvisor/pkg/sync/locking.AddGLock(0xc0002d82d0, 0xffffffffffffffff)
	pkg/sync/locking/lockdep.go:126 +0x28c
gvisor.dev/gvisor/pkg/tcpip/stack.(*packetsPendingLinkResolutionMutex).Lock(0xc0008a8dc0)
	bazel-out/k8-fastbuild-ST-3dcbe13c9b87/bin/pkg/tcpip/stack/packets_pending_link_resolution_mutex.go:33 +0x45
gvisor.dev/gvisor/pkg/tcpip/stack.(*packetsPendingLinkResolution).cancel(0xc0008a8db8)
	pkg/tcpip/stack/pending_packets.go:75 +0x5c
gvisor.dev/gvisor/pkg/tcpip/stack.(*nic).remove(0xc0008a8c00)
	pkg/tcpip/stack/nic.go:314 +0x1a5
gvisor.dev/gvisor/pkg/tcpip/stack.(*Stack).removeNICLocked(0xc0003fb500, 0x3)
	pkg/tcpip/stack/stack.go:976 +0x1dd
gvisor.dev/gvisor/pkg/tcpip/stack.(*Stack).RemoveNIC(0xc0003fb500, 0x11334f9?)
	pkg/tcpip/stack/stack.go:949 +0xad
gvisor.dev/gvisor/pkg/tcpip/link/tun.(*tunEndpoint).DecRef.func1()
	pkg/tcpip/link/tun/device.go:347 +0x89
gvisor.dev/gvisor/pkg/tcpip/link/tun.(*tunEndpointRefs).DecRef(0xc000733100, 0xc000ba98f0)
	bazel-out/k8-fastbuild-ST-3dcbe13c9b87/bin/pkg/tcpip/link/tun/tun_endpoint_refs.go:131 +0x5f
gvisor.dev/gvisor/pkg/tcpip/link/tun.(*tunEndpoint).DecRef(0xc000733100, {0xc000666560?, 0xc000664160?})
	pkg/tcpip/link/tun/device.go:345 +0x4d
gvisor.dev/gvisor/pkg/tcpip/link/tun.(*Device).Release(0xc000556368, {0x79c878, 0xc000277500})
	pkg/tcpip/link/tun/device.go:82 +0x1c6
gvisor.dev/gvisor/pkg/sentry/devices/tundev.(*tunFD).Release(0xc000556300, {0x79c878, 0xc000277500})
	pkg/sentry/devices/tundev/tundev.go:117 +0x47
gvisor.dev/gvisor/pkg/sentry/vfs.(*FileDescription).DecRef.func1()
	pkg/sentry/vfs/file_description.go:192 +0x348
gvisor.dev/gvisor/pkg/sentry/vfs.(*FileDescriptionRefs).DecRef(0xc000556300, 0xc000ba9b68)
	bazel-out/k8-fastbuild-ST-3dcbe13c9b87/bin/pkg/sentry/vfs/file_description_refs.go:131 +0x5f
gvisor.dev/gvisor/pkg/sentry/vfs.(*FileDescription).DecRef(0xc000556300, {0x79c878, 0xc000277500})
	pkg/sentry/vfs/file_description.go:161 +0x6f
gvisor.dev/gvisor/pkg/sentry/kernel.(*FDTable).drop(0xc0001f7f10?, {0x79c878, 0xc000277500}, 0xc000556300)
	pkg/sentry/kernel/fd_table.go:137 +0x1ed
gvisor.dev/gvisor/pkg/sentry/kernel.(*FDTable).RemoveIf(0xc0001f7f00, {0x79c878, 0xc000277500}, 0x4d4bc0)
	pkg/sentry/kernel/fd_table.go:501 +0x150
gvisor.dev/gvisor/pkg/sentry/kernel.(*FDTable).DecRef.func1()
	pkg/sentry/kernel/fd_table.go:152 +0x54
gvisor.dev/gvisor/pkg/sentry/kernel.(*FDTableRefs).DecRef(0xc0001f7f00, 0xc000ba9d88)
	bazel-out/k8-fastbuild-ST-3dcbe13c9b87/bin/pkg/sentry/kernel/fd_table_refs.go:131 +0x5f
gvisor.dev/gvisor/pkg/sentry/kernel.(*FDTable).DecRef(0xc0001f7f00, {0x79c878, 0xc000277500})
	pkg/sentry/kernel/fd_table.go:151 +0x6f
gvisor.dev/gvisor/pkg/sentry/kernel.(*runExitMain).execute(0xc000277500, 0xc000277500)
	pkg/sentry/kernel/task_exit.go:281 +0x797
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc000277500, 0xd)
	pkg/sentry/kernel/task_run.go:94 +0x3a2
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start
	pkg/sentry/kernel/task_start.go:377 +0x1ad

Crashes (53):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/12/14 20:53 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:42 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:41 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:32 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:32 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:20 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:18 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:09 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:04 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 19:38 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 19:34 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 19:27 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 19:16 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 18:56 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 18:20 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 17:37 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 17:36 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 16:43 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 14:34 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 14:23 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:53 gvisor 2deb308638ae b18f0a64 .config console log report syz C ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/16 02:44 gvisor 14b5ff5a2f32 6f9c033e .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 19:18 gvisor 14b5ff5a2f32 6f9c033e .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 18:13 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 16:12 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 12:08 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 10:00 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 06:43 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 06:38 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 05:54 gvisor 4c7561da986f 6f9c033e .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 01:32 gvisor 4c7561da986f b18f0a64 .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/15 01:26 gvisor 4c7561da986f b18f0a64 .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 23:33 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 22:24 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 22:20 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 21:14 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-1-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 20:54 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 18:38 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 13:47 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:55 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:40 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:33 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:20 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 12:16 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-2-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 11:39 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 11:28 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
2022/12/14 11:25 gvisor 2deb308638ae b18f0a64 .config console log report info ci-gvisor-ptrace-1-race-cover panic: WARNING: circular locking detected: stack.nicRWMutex -> stack.packetsPendingLinkResolutionMutex:
* Struck through repros no longer work on HEAD.