syzbot


possible deadlock in pcpu_alloc

Status: upstream: reported on 2024/04/05 23:38
Reported-by: syzbot+29ce28af963e0eb23843@syzkaller.appspotmail.com
First crash: 28d, last: 27d

Sample crash report:
=====================================================
WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
6.1.84-syzkaller #0 Not tainted
-----------------------------------------------------
kworker/0:7/3620 [HC0[0]:SC0[2]:HE0:SE0] is trying to acquire:
ffff88807ac38240 (&stab->lock){+...}-{2:2}, at: __sock_map_delete net/core/sock_map.c:416 [inline]
ffff88807ac38240 (&stab->lock){+...}-{2:2}, at: sock_map_delete_elem+0x97/0x130 net/core/sock_map.c:448

and this task is already holding:
ffffffff8d1e9458 (pcpu_lock){-.-.}-{2:2}, at: free_percpu+0xab/0xea0 mm/percpu.c:2277
which would create a new lock dependency:
 (pcpu_lock){-.-.}-{2:2} -> (&stab->lock){+...}-{2:2}

but this new dependency connects a HARDIRQ-irq-safe lock:
 (pcpu_lock){-.-.}-{2:2}

... which became HARDIRQ-irq-safe at:
  lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
  __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
  _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
  pcpu_alloc+0x320/0x18f0 mm/percpu.c:1780
  __alloc kernel/bpf/memalloc.c:135 [inline]
  alloc_bulk+0x614/0x8d0 kernel/bpf/memalloc.c:174
  irq_work_single+0xd5/0x230 kernel/irq_work.c:211
  irq_work_run_list kernel/irq_work.c:242 [inline]
  irq_work_run+0x187/0x350 kernel/irq_work.c:251
  __sysvec_irq_work+0xbb/0x360 arch/x86/kernel/irq_work.c:22
  sysvec_irq_work+0x89/0xb0 arch/x86/kernel/irq_work.c:17
  asm_sysvec_irq_work+0x16/0x20 arch/x86/include/asm/idtentry.h:679
  htab_unlock_bucket kernel/bpf/hashtab.c:180 [inline]
  __htab_percpu_map_update_elem+0x6d2/0x7e0 kernel/bpf/hashtab.c:1294
  bpf_percpu_hash_update+0x134/0x1f0 kernel/bpf/hashtab.c:2336
  bpf_map_update_value+0x282/0x6f0 kernel/bpf/syscall.c:200
  map_update_elem+0x503/0x680 kernel/bpf/syscall.c:1448
  __sys_bpf+0x337/0x6c0 kernel/bpf/syscall.c:4993
  __do_sys_bpf kernel/bpf/syscall.c:5109 [inline]
  __se_sys_bpf kernel/bpf/syscall.c:5107 [inline]
  __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:5107
  do_syscall_x64 arch/x86/entry/common.c:51 [inline]
  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:81
  entry_SYSCALL_64_after_hwframe+0x63/0xcd

to a HARDIRQ-irq-unsafe lock:
 (&stab->lock){+...}-{2:2}

... which became HARDIRQ-irq-unsafe at:
...
  lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
  __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
  _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:178
  sock_map_update_common+0x1b6/0x5b0 net/core/sock_map.c:492
  sock_map_update_elem_sys+0x55b/0x910 net/core/sock_map.c:581
  map_update_elem+0x503/0x680 kernel/bpf/syscall.c:1448
  __sys_bpf+0x337/0x6c0 kernel/bpf/syscall.c:4993
  __do_sys_bpf kernel/bpf/syscall.c:5109 [inline]
  __se_sys_bpf kernel/bpf/syscall.c:5107 [inline]
  __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:5107
  do_syscall_x64 arch/x86/entry/common.c:51 [inline]
  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:81
  entry_SYSCALL_64_after_hwframe+0x63/0xcd

other info that might help us debug this:

 Possible interrupt unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&stab->lock);
                               local_irq_disable();
                               lock(pcpu_lock);
                               lock(&stab->lock);
  <Interrupt>
    lock(pcpu_lock);

 *** DEADLOCK ***

4 locks held by kworker/0:7/3620:
 #0: ffff888012470938 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc90004d97d20 ((work_completion)(&aux->work)){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #2: ffffffff8d1e9458 (pcpu_lock){-.-.}-{2:2}, at: free_percpu+0xab/0xea0 mm/percpu.c:2277
 #3: ffffffff8d12a980 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #3: ffffffff8d12a980 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #3: ffffffff8d12a980 (rcu_read_lock){....}-{1:2}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2272 [inline]
 #3: ffffffff8d12a980 (rcu_read_lock){....}-{1:2}, at: bpf_trace_run3+0x146/0x440 kernel/trace/bpf_trace.c:2313

the dependencies between HARDIRQ-irq-safe lock and the holding lock:
-> (pcpu_lock){-.-.}-{2:2} {
   IN-HARDIRQ-W at:
                    lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
                    __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                    _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
                    pcpu_alloc+0x320/0x18f0 mm/percpu.c:1780
                    __alloc kernel/bpf/memalloc.c:135 [inline]
                    alloc_bulk+0x614/0x8d0 kernel/bpf/memalloc.c:174
                    irq_work_single+0xd5/0x230 kernel/irq_work.c:211
                    irq_work_run_list kernel/irq_work.c:242 [inline]
                    irq_work_run+0x187/0x350 kernel/irq_work.c:251
                    __sysvec_irq_work+0xbb/0x360 arch/x86/kernel/irq_work.c:22
                    sysvec_irq_work+0x89/0xb0 arch/x86/kernel/irq_work.c:17
                    asm_sysvec_irq_work+0x16/0x20 arch/x86/include/asm/idtentry.h:679
                    htab_unlock_bucket kernel/bpf/hashtab.c:180 [inline]
                    __htab_percpu_map_update_elem+0x6d2/0x7e0 kernel/bpf/hashtab.c:1294
                    bpf_percpu_hash_update+0x134/0x1f0 kernel/bpf/hashtab.c:2336
                    bpf_map_update_value+0x282/0x6f0 kernel/bpf/syscall.c:200
                    map_update_elem+0x503/0x680 kernel/bpf/syscall.c:1448
                    __sys_bpf+0x337/0x6c0 kernel/bpf/syscall.c:4993
                    __do_sys_bpf kernel/bpf/syscall.c:5109 [inline]
                    __se_sys_bpf kernel/bpf/syscall.c:5107 [inline]
                    __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:5107
                    do_syscall_x64 arch/x86/entry/common.c:51 [inline]
                    do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:81
                    entry_SYSCALL_64_after_hwframe+0x63/0xcd
   IN-SOFTIRQ-W at:
                    lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
                    __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                    _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
                    free_percpu+0xab/0xea0 mm/percpu.c:2277
                    free_vfsmnt+0xe8/0x120 fs/namespace.c:612
                    rcu_do_batch kernel/rcu/tree.c:2296 [inline]
                    rcu_core+0xad4/0x17e0 kernel/rcu/tree.c:2556
                    __do_softirq+0x2e9/0xa4c kernel/softirq.c:571
                    invoke_softirq kernel/softirq.c:445 [inline]
                    __irq_exit_rcu+0x155/0x240 kernel/softirq.c:650
                    irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
                    sysvec_apic_timer_interrupt+0x91/0xb0 arch/x86/kernel/apic/apic.c:1106
                    asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:653
                    check_kcov_mode kernel/kcov.c:175 [inline]
                    write_comp_data kernel/kcov.c:236 [inline]
                    __sanitizer_cov_trace_const_cmp4+0x30/0x80 kernel/kcov.c:304
                    isdigit include/linux/ctype.h:45 [inline]
                    update_event_printk kernel/trace/trace_events.c:2629 [inline]
                    trace_event_eval_update+0x3d9/0xfc0 kernel/trace/trace_events.c:2788
                    process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
                    worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
                    kthread+0x28d/0x320 kernel/kthread.c:376
                    ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
   INITIAL USE at:
                   lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
                   __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                   _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
                   pcpu_stats_chunk_alloc mm/percpu-internal.h:211 [inline]
                   pcpu_setup_first_chunk+0xdd2/0x172c mm/percpu.c:2775
                   pcpu_embed_first_chunk+0xb24/0xbd2 mm/percpu.c:3158
                   setup_per_cpu_areas+0xd4/0xbcb arch/x86/kernel/setup_percpu.c:156
                   start_kernel+0xc3/0x53f init/main.c:965
                   secondary_startup_64_no_verify+0xcf/0xdb
 }
 ... key      at: [<ffffffff8d1e9458>] pcpu_lock+0x18/0x160

the dependencies between the lock to be acquired
 and HARDIRQ-irq-unsafe lock:
-> (&stab->lock){+...}-{2:2} {
   HARDIRQ-ON-W at:
                    lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
                    __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
                    _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:178
                    sock_map_update_common+0x1b6/0x5b0 net/core/sock_map.c:492
                    sock_map_update_elem_sys+0x55b/0x910 net/core/sock_map.c:581
                    map_update_elem+0x503/0x680 kernel/bpf/syscall.c:1448
                    __sys_bpf+0x337/0x6c0 kernel/bpf/syscall.c:4993
                    __do_sys_bpf kernel/bpf/syscall.c:5109 [inline]
                    __se_sys_bpf kernel/bpf/syscall.c:5107 [inline]
                    __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:5107
                    do_syscall_x64 arch/x86/entry/common.c:51 [inline]
                    do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:81
                    entry_SYSCALL_64_after_hwframe+0x63/0xcd
   INITIAL USE at:
                   lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
                   __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
                   _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:178
                   sock_map_update_common+0x1b6/0x5b0 net/core/sock_map.c:492
                   sock_map_update_elem_sys+0x55b/0x910 net/core/sock_map.c:581
                   map_update_elem+0x503/0x680 kernel/bpf/syscall.c:1448
                   __sys_bpf+0x337/0x6c0 kernel/bpf/syscall.c:4993
                   __do_sys_bpf kernel/bpf/syscall.c:5109 [inline]
                   __se_sys_bpf kernel/bpf/syscall.c:5107 [inline]
                   __x64_sys_bpf+0x78/0x90 kernel/bpf/syscall.c:5107
                   do_syscall_x64 arch/x86/entry/common.c:51 [inline]
                   do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:81
                   entry_SYSCALL_64_after_hwframe+0x63/0xcd
 }
 ... key      at: [<ffffffff920b1320>] sock_map_alloc.__key+0x0/0x20
 ... acquired at:
   lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
   __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
   _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:178
   __sock_map_delete net/core/sock_map.c:416 [inline]
   sock_map_delete_elem+0x97/0x130 net/core/sock_map.c:448
   bpf_prog_2c29ac5cdc6b1842+0x3a/0x3e
   bpf_dispatcher_nop_func include/linux/bpf.h:989 [inline]
   __bpf_prog_run include/linux/filter.h:603 [inline]
   bpf_prog_run include/linux/filter.h:610 [inline]
   __bpf_trace_run kernel/trace/bpf_trace.c:2273 [inline]
   bpf_trace_run3+0x231/0x440 kernel/trace/bpf_trace.c:2313
   __traceiter_percpu_free_percpu+0x78/0xd0 include/trace/events/percpu.h:54
   trace_percpu_free_percpu+0x1d6/0x260 include/trace/events/percpu.h:54
   free_percpu+0x91b/0xea0 mm/percpu.c:2304
   __bpf_prog_free+0xe7/0x120 kernel/bpf/core.c:269
   process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
   worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
   kthread+0x28d/0x320 kernel/kthread.c:376
   ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307


stack backtrace:
CPU: 0 PID: 3620 Comm: kworker/0:7 Not tainted 6.1.84-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events bpf_prog_free_deferred
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 print_bad_irq_dependency kernel/locking/lockdep.c:2604 [inline]
 check_irq_usage kernel/locking/lockdep.c:2843 [inline]
 check_prev_add kernel/locking/lockdep.c:3094 [inline]
 check_prevs_add kernel/locking/lockdep.c:3209 [inline]
 validate_chain+0x4d16/0x5950 kernel/locking/lockdep.c:3825
 __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5049
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:126 [inline]
 _raw_spin_lock_bh+0x31/0x40 kernel/locking/spinlock.c:178
 __sock_map_delete net/core/sock_map.c:416 [inline]
 sock_map_delete_elem+0x97/0x130 net/core/sock_map.c:448
 bpf_prog_2c29ac5cdc6b1842+0x3a/0x3e
 bpf_dispatcher_nop_func include/linux/bpf.h:989 [inline]
 __bpf_prog_run include/linux/filter.h:603 [inline]
 bpf_prog_run include/linux/filter.h:610 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2273 [inline]
 bpf_trace_run3+0x231/0x440 kernel/trace/bpf_trace.c:2313
 __traceiter_percpu_free_percpu+0x78/0xd0 include/trace/events/percpu.h:54
 trace_percpu_free_percpu+0x1d6/0x260 include/trace/events/percpu.h:54
 free_percpu+0x91b/0xea0 mm/percpu.c:2304
 __bpf_prog_free+0xe7/0x120 kernel/bpf/core.c:269
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>
------------[ cut here ]------------
raw_local_irq_restore() called with IRQs enabled
WARNING: CPU: 0 PID: 3620 at kernel/locking/irqflag-debug.c:10 warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:10
Modules linked in:
CPU: 0 PID: 3620 Comm: kworker/0:7 Not tainted 6.1.84-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Workqueue: events bpf_prog_free_deferred
RIP: 0010:warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:10
Code: 24 48 c7 c7 00 bc ea 8a e8 6c f5 fd ff 80 3d 2f 5b d5 03 00 74 01 c3 c6 05 25 5b d5 03 01 48 c7 c7 60 e6 eb 8a e8 23 64 c8 f6 <0f> 0b c3 41 56 53 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 44
RSP: 0018:ffffc90004d97a58 EFLAGS: 00010246
RAX: 549faa56e3193600 RBX: 1ffff920009b2f50 RCX: ffff88807d455940
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90004d97af0 R08: ffffffff81527eae R09: fffff520009b2ead
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: 1ffff920009b2f4c R14: ffffc90004d97a80 R15: 0000000000000246
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff2abba8000 CR3: 000000000ce8e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:151 [inline]
 _raw_spin_unlock_irqrestore+0x118/0x130 kernel/locking/spinlock.c:194
 spin_unlock_irqrestore include/linux/spinlock.h:406 [inline]
 free_percpu+0x92c/0xea0 mm/percpu.c:2306
 __bpf_prog_free+0xe7/0x120 kernel/bpf/core.c:269
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/04/06 21:46 linux-6.1.y 347385861c50 ca620dd8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in pcpu_alloc
2024/04/05 23:37 linux-6.1.y 347385861c50 77230c29 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-perf possible deadlock in pcpu_alloc
* Struck through repros no longer work on HEAD.