bisecting fixing commit since d6765985a42a660f078896d5c5b27f97c580a490
building syzkaller on 9d2ab5dfe7727dfea4b9b279f4edf731acb386ef
testing commit d6765985a42a660f078896d5c5b27f97c580a490
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2
kernel signature: 65241d6ac4131102e3bb9c9a61f5d375abcc730a52f149fb3b1cdca04a394b59
run #0: crashed: INFO: rcu detected stall in ieee80211_tasklet_handler
run #1: crashed: BUG: soft lockup in mac80211_hwsim_beacon
run #2: crashed: INFO: rcu detected stall in net_tx_action
run #3: crashed: INFO: rcu detected stall in wg_packet_handshake_send_worker
run #4: crashed: INFO: rcu detected stall in ipv6_rcv
run #5: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #6: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #7: crashed: INFO: rcu detected stall in do_sys_ftruncate
run #8: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #9: crashed: INFO: rcu detected stall in net_tx_action
run #10: crashed: INFO: rcu detected stall in ieee80211_tasklet_handler
run #11: crashed: BUG: soft lockup in tc_modify_qdisc
run #12: crashed: INFO: rcu detected stall in net_tx_action
run #13: crashed: INFO: rcu detected stall in dst_destroy
run #14: crashed: INFO: rcu detected stall in smp_call_function
run #15: crashed: BUG: soft lockup in mac80211_hwsim_beacon
run #16: crashed: BUG: sleeping function called from invalid context in lock_sock_nested
run #17: crashed: INFO: rcu detected stall in ieee80211_tasklet_handler
run #18: crashed: INFO: rcu detected stall in smp_call_function
run #19: crashed: BUG: soft lockup in mac80211_hwsim_beacon
testing current HEAD 19fa0887c57d35b57bfb895e6caf8e72d9601ec0
testing commit 19fa0887c57d35b57bfb895e6caf8e72d9601ec0
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2
kernel signature: cdce0b8aaa479c97587583458af51a3ace848d507357c6794fd2ed2d4258ecb1
run #0: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #3: crashed: INFO: rcu detected stall in ext4_end_io_rsv_work
run #4: crashed: INFO: rcu detected stall in wg_packet_handshake_receive_worker
run #5: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #6: crashed: INFO: rcu detected stall in net_tx_action
run #7: crashed: INFO: rcu detected stall in batadv_iv_send_outstanding_bat_ogm_packet
run #8: crashed: BUG: soft lockup in smp_call_function
run #9: crashed: INFO: rcu detected stall in neigh_periodic_work
revisions tested: 2, total time: 26m44.584360892s (build: 12m51.597907473s, test: 13m10.823468643s)
the crash still happens on HEAD
commit msg: MAINTAINERS: please remove myself from the Prestera driver
crash: INFO: rcu detected stall in neigh_periodic_work
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...!: (1 GPs behind) idle=faf/1/0x4000000000000000 softirq=9851/9852 fqs=0
(t=12364 jiffies g=8421 q=1012)
rcu: rcu_preempt kthread timer wakeup didn't happen for 12363 jiffies! g8421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: Possible timer handling issue on cpu=1 timer-softirq=3763
rcu: rcu_preempt kthread starved for 12364 jiffies! g8421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:I stack:28760 pid: 14 ppid: 2 flags:0x00004000
Call Trace:
context_switch kernel/sched/core.c:4940 [inline]
__schedule+0x90d/0x26c0 kernel/sched/core.c:6287
schedule+0xd3/0x270 kernel/sched/core.c:6366
schedule_timeout+0x11d/0x250 kernel/time/timer.c:1881
rcu_gp_fqs_loop+0x186/0x800 kernel/rcu/tree.c:1957
rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2130
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 3010 Comm: kworker/1:4 Not tainted 5.15.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_power_efficient neigh_periodic_work
RIP: 0010:arch_atomic_read arch/x86/include/asm/atomic.h:29 [inline]
RIP: 0010:atomic_read include/linux/atomic/atomic-instrumented.h:28 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:330 [inline]
RIP: 0010:rcu_is_watching+0x69/0xc0 kernel/rcu/tree.c:1121
Code: 48 03 1c ed 80 88 61 8a be 04 00 00 00 48 89 df e8 7c fa 4e 00 48 89 da 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 0f b6 14 02 <48> 89 d8 83 e0 07 83 c0 03 38 d0 7c 04 84 d2 75 19 8b 03 83 e0 01
RSP: 0018:ffffc90000dc0db0 EFLAGS: 00000806
RAX: dffffc0000000000 RBX: ffff8880b9f329c8 RCX: ffffffff8158b624
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8880b9f329c8
RBP: 0000000000000001 R08: 0000000000000000 R09: ffff8880b9f329cb
R10: ffffed10173e6539 R11: 0000000000000001 R12: ffff88801a508620
R13: 000000000000228a R14: ffff88807e2c5ae8 R15: ffff88801a508600
FS: 0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faedd865028 CR3: 000000007c735000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
rcu_read_lock include/linux/rcupdate.h:688 [inline]
advance_sched+0x3ea/0x920 net/sched/sch_taprio.c:763
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x4d7/0xb00 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:current_gfp_context include/linux/sched/mm.h:158 [inline]
RIP: 0010:fs_reclaim_acquire+0x1/0x160 mm/page_alloc.c:4549
Code: 7a ff ff ff 89 da 80 e2 7f a9 00 00 04 00 0f 45 da eb df e8 c1 03 09 00 e9 57 ff ff ff e8 b7 03 09 00 eb 92 0f 1f 44 00 00 55 <65> 48 8b 04 25 40 f0 01 00 48 89 e5 41 56 49 89 c6 53 89 fb 48 8d
RSP: 0018:ffffc900020ff8d8 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000000028 RCX: 0000000000000001
RDX: 1ffff9200041ff36 RSI: 0000000000012b20 RDI: 0000000000012b20
RBP: ffff88800fc4f8c0 R08: ffff88800fc4f8c0 R09: ffff8880b9f329cb
R10: ffffed10173e6539 R11: 000000000007a089 R12: dead000000000100
R13: ffffffff838b06d4 R14: 0000000000012b20 R15: 0000000000012b20
might_alloc include/linux/sched/mm.h:198 [inline]
slab_pre_alloc_hook mm/slab.h:492 [inline]
slab_alloc_node mm/slub.c:3127 [inline]
slab_alloc mm/slub.c:3221 [inline]
kmem_cache_alloc+0x3e/0x390 mm/slub.c:3226
kmem_cache_zalloc include/linux/slab.h:711 [inline]
fill_pool+0x264/0x5c0 lib/debugobjects.c:171
__debug_object_init+0x7a/0xd10 lib/debugobjects.c:565
debug_object_init lib/debugobjects.c:620 [inline]
debug_object_activate+0x32c/0x3e0 lib/debugobjects.c:706
debug_rcu_head_queue kernel/rcu/rcu.h:176 [inline]
kvfree_call_rcu+0x32/0x990 kernel/rcu/tree.c:3543
neigh_periodic_work+0x4ba/0x890 net/core/neighbour.c:941
process_one_work+0x87f/0x1450 kernel/workqueue.c:2297
worker_thread+0x598/0x1040 kernel/workqueue.c:2444
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
NMI backtrace for cpu 0
CPU: 0 PID: 10 Comm: kworker/u4:1 Not tainted 5.15.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106
nmi_cpu_backtrace.cold+0x30/0xc0 lib/nmi_backtrace.c:105
nmi_trigger_cpumask_backtrace+0x11a/0x160 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:343
print_cpu_stall kernel/rcu/tree_stall.h:627 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:711 [inline]
rcu_pending kernel/rcu/tree.c:3880 [inline]
rcu_sched_clock_irq.cold+0x9d/0x746 kernel/rcu/tree.c:2599
update_process_times+0x13b/0x1c0 kernel/time/timer.c:1785
tick_sched_handle+0x6f/0x130 kernel/time/tick-sched.c:226
tick_sched_timer+0x132/0x210 kernel/time/tick-sched.c:1421
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x18a/0xb00 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x22e/0x9d0 kernel/smp.c:969
Code: 38 d0 7c 08 84 d2 0f 85 2b 05 00 00 8b 43 08 a8 01 74 2e 48 89 ca 49 89 cf 48 c1 ea 03 41 83 e7 07 4c 01 e2 41 83 c7 03 f3 90 <0f> b6 02 41 38 c7 7c 08 84 c0 0f 85 d4 04 00 00 8b 43 08 a8 01 75
RSP: 0018:ffffc90000cf7a58 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff8880b9f37c00 RCX: ffff8880b9f37c08
RDX: ffffed10173e6f81 RSI: ffff8880b9e32b08 RDI: ffffffff8a618888
RBP: ffff8880b9e32b00 R08: 0000000000000001 R09: ffffffff8ee07927
R10: 0000000000000001 R11: 0000000000000001 R12: dffffc0000000000
R13: ffff8880b9e32b08 R14: ffffed10173c6560 R15: 0000000000000003
on_each_cpu_cond_mask+0x3f/0x70 kernel/smp.c:1135
on_each_cpu include/linux/smp.h:71 [inline]
text_poke_sync arch/x86/kernel/alternative.c:929 [inline]
text_poke_bp_batch+0x1b3/0x560 arch/x86/kernel/alternative.c:1114
text_poke_flush arch/x86/kernel/alternative.c:1268 [inline]
text_poke_flush arch/x86/kernel/alternative.c:1265 [inline]
text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1275
arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
static_key_disable_cpuslocked+0x100/0x160 kernel/jump_label.c:207
static_key_disable+0x11/0x20 kernel/jump_label.c:215
toggle_allocation_gate mm/kfence/core.c:640 [inline]
toggle_allocation_gate+0x156/0x310 mm/kfence/core.c:618
process_one_work+0x87f/0x1450 kernel/workqueue.c:2297
worker_thread+0x598/0x1040 kernel/workqueue.c:2444
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
----------------
Code disassembly (best guess):
0: 48 03 1c ed 80 88 61 add -0x759e7780(,%rbp,8),%rbx
7: 8a
8: be 04 00 00 00 mov $0x4,%esi
d: 48 89 df mov %rbx,%rdi
10: e8 7c fa 4e 00 callq 0x4efa91
15: 48 89 da mov %rbx,%rdx
18: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
1f: fc ff df
22: 48 c1 ea 03 shr $0x3,%rdx
26: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
* 2a: 48 89 d8 mov %rbx,%rax <-- trapping instruction
2d: 83 e0 07 and $0x7,%eax
30: 83 c0 03 add $0x3,%eax
33: 38 d0 cmp %dl,%al
35: 7c 04 jl 0x3b
37: 84 d2 test %dl,%dl
39: 75 19 jne 0x54
3b: 8b 03 mov (%rbx),%eax
3d: 83 e0 01 and $0x1,%eax