syzbot

 sign-in | mailing list | source | docs
🐞 Open [1225]
≡ Subsystems
🐞 Fixed [5626]
🐞 Invalid [13553]
⬇ Missing Backports [97]
Kernel Health Bugs/Month Bug Lifetimes Fuzzing Crashes
Total Repo Heatmap Subsystems Heatmap
💬 Send us feedback

possible deadlock in htab_lock_bucket (2)

Status: upstream: reported C repro on 2024/08/05 10:05
Subsystems: bpf
[Documentation on labels]
Reported-by: syzbot+ee7551b0640c5471e610@syzkaller.appspotmail.com
First crash: 38d, last: 10d
Cause bisection: failed (error log, bisect log)
  
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [bpf?] possible deadlock in htab_lock_bucket (2) 0 (1) 2024/08/05 10:05
▶ ▼ Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in htab_lock_bucket origin:upstream C 2 1d23h 148d 0/3 upstream: reported C repro on 2024/04/17 13:40
▶ ▼ Last patch testing requests (1)
Created Duration User Patch Repo Result
2024/08/19 11:04 59m retest repro bpf-next report log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.11.0-rc4-syzkaller-gb408473ea01b #0 Not tainted
------------------------------------------------------
strace-static-x/5228 is trying to acquire lock:
ffff88806cea70f8 (&htab->lockdep_key#29){-.-.}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167

but task is already holding lock:
ffff88806cea5458 (&htab->lockdep_key#28){-.-.}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&htab->lockdep_key#28){-.-.}-{2:2}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5759
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
       bpf_prog_6f5f05285f674219+0x42/0x4b
       bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
       __bpf_prog_run include/linux/filter.h:691 [inline]
       bpf_prog_run include/linux/filter.h:698 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
       queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
       bpf_prog_6f5f05285f674219+0x42/0x4b
       bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
       __bpf_prog_run include/linux/filter.h:691 [inline]
       bpf_prog_run include/linux/filter.h:698 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0xf5/0x120 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:610 [inline]
       __mutex_lock+0x147/0xd70 kernel/locking/mutex.c:752
       pipe_write+0x1c9/0x1a40 fs/pipe.c:455
       new_sync_write fs/read_write.c:497 [inline]
       vfs_write+0xa72/0xc90 fs/read_write.c:590
       ksys_write+0x1a0/0x2c0 fs/read_write.c:643
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&htab->lockdep_key#29){-.-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3133 [inline]
       check_prevs_add kernel/locking/lockdep.c:3252 [inline]
       validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3868
       __lock_acquire+0x137a/0x2040 kernel/locking/lockdep.c:5142
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5759
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
       bpf_prog_6f5f05285f674219+0x42/0x4b
       bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
       __bpf_prog_run include/linux/filter.h:691 [inline]
       bpf_prog_run include/linux/filter.h:698 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
       queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
       bpf_prog_6f5f05285f674219+0x42/0x4b
       bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
       __bpf_prog_run include/linux/filter.h:691 [inline]
       bpf_prog_run include/linux/filter.h:698 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0xf5/0x120 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:610 [inline]
       __mutex_lock+0x147/0xd70 kernel/locking/mutex.c:752
       pipe_write+0x1c9/0x1a40 fs/pipe.c:455
       new_sync_write fs/read_write.c:497 [inline]
       vfs_write+0xa72/0xc90 fs/read_write.c:590
       ksys_write+0x1a0/0x2c0 fs/read_write.c:643
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&htab->lockdep_key#28);
                               lock(&htab->lockdep_key#29);
                               lock(&htab->lockdep_key#28);
  lock(&htab->lockdep_key#29);

 *** DEADLOCK ***

4 locks held by strace-static-x/5228:
 #0: ffff88807e6b7c68 (&pipe->mutex){+.+.}-{3:3}, at: pipe_write+0x1c9/0x1a40 fs/pipe.c:455
 #1: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:326 [inline]
 #1: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:838 [inline]
 #1: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2405 [inline]
 #1: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x1fc/0x540 kernel/trace/bpf_trace.c:2447
 #2: ffff88806cea5458 (&htab->lockdep_key#28){-.-.}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 #3: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:326 [inline]
 #3: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:838 [inline]
 #3: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2405 [inline]
 #3: ffffffff8e938320 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x1fc/0x540 kernel/trace/bpf_trace.c:2447

stack backtrace:
CPU: 1 UID: 0 PID: 5228 Comm: strace-static-x Not tainted 6.11.0-rc4-syzkaller-gb408473ea01b #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:93 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:119
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2186
 check_prev_add kernel/locking/lockdep.c:3133 [inline]
 check_prevs_add kernel/locking/lockdep.c:3252 [inline]
 validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3868
 __lock_acquire+0x137a/0x2040 kernel/locking/lockdep.c:5142
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5759
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
 bpf_prog_6f5f05285f674219+0x42/0x4b
 bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
 __bpf_prog_run include/linux/filter.h:691 [inline]
 bpf_prog_run include/linux/filter.h:698 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
 bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
 __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
 trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
 __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
 queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
 queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
 do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
 htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1462
 bpf_prog_6f5f05285f674219+0x42/0x4b
 bpf_dispatcher_nop_func include/linux/bpf.h:1243 [inline]
 __bpf_prog_run include/linux/filter.h:691 [inline]
 bpf_prog_run include/linux/filter.h:698 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2406 [inline]
 bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2447
 __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
 trace_contention_begin+0xf5/0x120 include/trace/events/lock.h:95
 __mutex_lock_common kernel/locking/mutex.c:610 [inline]
 __mutex_lock+0x147/0xd70 kernel/locking/mutex.c:752
 pipe_write+0x1c9/0x1a40 fs/pipe.c:455
 new_sync_write fs/read_write.c:497 [inline]
 vfs_write+0xa72/0xc90 fs/read_write.c:590
 ksys_write+0x1a0/0x2c0 fs/read_write.c:643
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x4e8593
Code: c7 c2 a8 ff ff ff f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 48 89 54 24 18
RSP: 002b:00007ffd888fed08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000000000000002c RCX: 00000000004e8593
RDX: 000000000000002c RSI: 0000000009621140 RDI: 0000000000000002
RBP: 0000000009621140 R08: 0000000000000003 R09: 00000000000016a8
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000002c
R13: 000000000063f460 R14: 000000000000002c R15: 0000000000000001
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/02 00:16 bpf b408473ea01b 1eda0d14 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce possible deadlock in htab_lock_bucket
2024/08/05 09:53 bpf-next 3d650ab5e7d9 1786a2a8 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-bpf-next-kasan-gce possible deadlock in htab_lock_bucket
* Struck through repros no longer work on HEAD.