syzbot

 sign-in | mailing list | source | docs
🐞 Open [1413]
≡ Subsystems
🐞 Fixed [5827]
🐞 Invalid [14217]
⬇ Missing Backports [92]
≡ Crashes
Kernel Health Bugs/Month Bug Lifetimes Fuzzing
Total Repo Heatmap Subsystems Heatmap
💬 Send us feedback

possible deadlock in htab_lock_bucket (2)

Status: upstream: reported C repro on 2024/08/05 10:05
Subsystems: bpf
[Documentation on labels]
Reported-by: syzbot+ee7551b0640c5471e610@syzkaller.appspotmail.com
First crash: 107d, last: 10d
Cause bisection: failed (error log, bisect log)
  
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [bpf?] possible deadlock in htab_lock_bucket (2) 0 (1) 2024/08/05 10:05
▶ ▼ Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 possible deadlock in htab_lock_bucket origin:upstream C error 2 71d 217d 0/3 upstream: reported C repro on 2024/04/17 13:40
▶ ▼ Last patch testing requests (4)
Created Duration User Patch Repo Result
2024/11/10 18:10 2h03m retest repro net report log
2024/10/06 01:05 19m retest repro bpf-next report log
2024/09/16 00:30 18m retest repro bpf report log
2024/08/19 11:04 59m retest repro bpf-next report log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.12.0-rc4-syzkaller-00168-ge31a8219fbfc #0 Not tainted
------------------------------------------------------
syz-executor494/6571 is trying to acquire lock:
ffff888026e911d0 (&htab->lockdep_key#2){....}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167

but task is already holding lock:
ffff88807bc5b1d0 (&htab->lockdep_key#5){....}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&htab->lockdep_key#5){....}-{2:2}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
       0xffffffffa0003923
       bpf_dispatcher_nop_func include/linux/bpf.h:1257 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2318 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2359
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
       queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1257 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2318 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2359
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0xf5/0x120 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:610 [inline]
       __mutex_lock+0x147/0xd70 kernel/locking/mutex.c:752
       uprobe_clear_state+0x54/0x290 kernel/events/uprobes.c:1598
       __mmput+0x5f/0x390 kernel/fork.c:1343
       exit_mm+0x220/0x310 kernel/exit.c:571
       do_exit+0x9b2/0x28e0 kernel/exit.c:926
       do_group_exit+0x207/0x2c0 kernel/exit.c:1088
       __do_sys_exit_group kernel/exit.c:1099 [inline]
       __se_sys_exit_group kernel/exit.c:1097 [inline]
       __x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1097
       x64_sys_call+0x2634/0x2640 arch/x86/include/generated/asm/syscalls_64.h:232
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&htab->lockdep_key#2){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3161 [inline]
       check_prevs_add kernel/locking/lockdep.c:3280 [inline]
       validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
       __lock_acquire+0x1384/0x2050 kernel/locking/lockdep.c:5202
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1257 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2318 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2359
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
       queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
       htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
       0xffffffffa0003923
       bpf_dispatcher_nop_func include/linux/bpf.h:1257 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2318 [inline]
       bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2359
       __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
       trace_contention_begin+0xf5/0x120 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:610 [inline]
       __mutex_lock+0x147/0xd70 kernel/locking/mutex.c:752
       tracepoint_probe_unregister+0x32/0x990 kernel/tracepoint.c:548
       bpf_raw_tp_link_release+0x45/0x70 kernel/bpf/syscall.c:3475
       bpf_link_free+0xf5/0x250 kernel/bpf/syscall.c:3005
       bpf_link_put_direct kernel/bpf/syscall.c:3045 [inline]
       bpf_link_release+0x78/0x90 kernel/bpf/syscall.c:3052
       __fput+0x23f/0x880 fs/file_table.c:431
       task_work_run+0x24f/0x310 kernel/task_work.c:239
       exit_task_work include/linux/task_work.h:43 [inline]
       do_exit+0xa2f/0x28e0 kernel/exit.c:939
       do_group_exit+0x207/0x2c0 kernel/exit.c:1088
       __do_sys_exit_group kernel/exit.c:1099 [inline]
       __se_sys_exit_group kernel/exit.c:1097 [inline]
       __x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1097
       x64_sys_call+0x2634/0x2640 arch/x86/include/generated/asm/syscalls_64.h:232
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&htab->lockdep_key#5);
                               lock(&htab->lockdep_key#2);
                               lock(&htab->lockdep_key#5);
  lock(&htab->lockdep_key#2);

 *** DEADLOCK ***

4 locks held by syz-executor494/6571:
 #0: ffffffff8e98a548 (tracepoints_mutex){+.+.}-{3:3}, at: tracepoint_probe_unregister+0x32/0x990 kernel/tracepoint.c:548
 #1: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #1: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #1: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2317 [inline]
 #1: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x1fc/0x540 kernel/trace/bpf_trace.c:2359
 #2: ffff88807bc5b1d0 (&htab->lockdep_key#5){....}-{2:2}, at: htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 #3: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #3: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #3: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2317 [inline]
 #3: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run2+0x1fc/0x540 kernel/trace/bpf_trace.c:2359

stack backtrace:
CPU: 1 UID: 0 PID: 6571 Comm: syz-executor494 Not tainted 6.12.0-rc4-syzkaller-00168-ge31a8219fbfc #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 print_circular_bug+0x13a/0x1b0 kernel/locking/lockdep.c:2074
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2206
 check_prev_add kernel/locking/lockdep.c:3161 [inline]
 check_prevs_add kernel/locking/lockdep.c:3280 [inline]
 validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
 __lock_acquire+0x1384/0x2050 kernel/locking/lockdep.c:5202
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
 bpf_prog_6f5f05285f674219+0x43/0x4c
 bpf_dispatcher_nop_func include/linux/bpf.h:1257 [inline]
 __bpf_prog_run include/linux/filter.h:701 [inline]
 bpf_prog_run include/linux/filter.h:708 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2318 [inline]
 bpf_trace_run2+0x2ec/0x540 kernel/trace/bpf_trace.c:2359
 __traceiter_contention_begin+0x7b/0xb0 include/trace/events/lock.h:95
 trace_contention_begin+0x117/0x140 include/trace/events/lock.h:95
 __pv_queued_spin_lock_slowpath+0x114/0xdb0 kernel/locking/qspinlock.c:402
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:584 [inline]
 queued_spin_lock_slowpath+0x42/0x50 arch/x86/include/asm/qspinlock.h:51
 queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
 do_raw_spin_lock+0x272/0x370 kernel/locking/spinlock_debug.c:116
 htab_lock_bucket+0x1a4/0x370 kernel/bpf/hashtab.c:167
 htab_lru_map_delete_elem+0x1f1/0x700 kernel/bpf/hashtab.c:1466
 </TASK>

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/10/27 15:34 net e31a8219fbfc 65e8686b .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce possible deadlock in htab_lock_bucket
2024/09/02 00:16 bpf b408473ea01b 1eda0d14 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce possible deadlock in htab_lock_bucket
2024/09/21 18:25 bpf-next 5277d130947b 6f888b75 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-bpf-next-kasan-gce possible deadlock in htab_lock_bucket
2024/08/05 09:53 bpf-next 3d650ab5e7d9 1786a2a8 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-bpf-next-kasan-gce possible deadlock in htab_lock_bucket
* Struck through repros no longer work on HEAD.