bisecting fixing commit since d6765985a42a660f078896d5c5b27f97c580a490
building syzkaller on 9d2ab5dfe7727dfea4b9b279f4edf731acb386ef
testing commit d6765985a42a660f078896d5c5b27f97c580a490
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1
kernel signature: e7bc24bc964c806449309effc3928e701cc5fc7a31787c556d885a5393c73270
run #0: crashed: BUG: sleeping function called from invalid context in corrupted
run #1: crashed: BUG: sleeping function called from invalid context in lock_sock_nested
run #2: crashed: INFO: rcu detected stall in net_tx_action
run #3: crashed: INFO: rcu detected stall in net_tx_action
run #4: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #5: crashed: BUG: soft lockup in tc_modify_qdisc
run #6: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #7: crashed: INFO: rcu detected stall in batadv_iv_send_outstanding_bat_ogm_packet
run #8: crashed: INFO: rcu detected stall in smp_call_function
run #9: crashed: INFO: rcu detected stall in ieee80211_iface_work
run #10: crashed: INFO: rcu detected stall in ieee80211_iface_work
run #11: crashed: INFO: rcu detected stall in net_tx_action
run #12: crashed: INFO: rcu detected stall in fill_pool
run #13: crashed: BUG: soft lockup in tc_modify_qdisc
run #14: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #15: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #16: crashed: INFO: rcu detected stall in smp_call_function
run #17: crashed: INFO: rcu detected stall in ieee80211_tasklet_handler
run #18: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #19: crashed: BUG: soft lockup in mac80211_hwsim_beacon
testing current HEAD 73367f05b25dbd064061aee780638564d15b01d1
testing commit 73367f05b25dbd064061aee780638564d15b01d1
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.1
kernel signature: b49682b344a77c28738d3017a6304e7cff0a78b1a49aef94396153df37a5f692
run #0: crashed: INFO: rcu detected stall in ieee80211_iface_work
run #1: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #2: crashed: INFO: rcu detected stall in mac80211_hwsim_beacon
run #3: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #4: crashed: BUG: soft lockup in tc_modify_qdisc
run #5: crashed: BUG: soft lockup in mac80211_hwsim_beacon
run #6: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #7: crashed: INFO: rcu detected stall in smp_call_function
run #8: crashed: BUG: soft lockup in net_tx_action
run #9: crashed: INFO: rcu detected stall in ieee80211_ibss_work
revisions tested: 2, total time: 29m43.768056723s (build: 14m18.339420841s, test: 14m43.973275857s)
the crash still happens on HEAD
commit msg: Merge tag 'nfsd-5.14-1' of git://linux-nfs.org/~bfields/linux
crash: INFO: rcu detected stall in ieee80211_ibss_work
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...0: (2 ticks this GP) idle=426/1/0x4000000000000000 softirq=11145/11145 fqs=3
(t=11539 jiffies g=9017 q=1513)
rcu: rcu_preempt kthread timer wakeup didn't happen for 5550 jiffies! g9017 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: Possible timer handling issue on cpu=1 timer-softirq=3904
rcu: rcu_preempt kthread starved for 5551 jiffies! g9017 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:I stack:28928 pid: 14 ppid: 2 flags:0x00004000
Call Trace:
context_switch kernel/sched/core.c:4681 [inline]
__schedule+0x907/0x26b0 kernel/sched/core.c:5938
schedule+0xd3/0x270 kernel/sched/core.c:6017
schedule_timeout+0x11d/0x250 kernel/time/timer.c:1881
rcu_gp_fqs_loop kernel/rcu/tree.c:1996 [inline]
rcu_gp_kthread+0xd34/0x1980 kernel/rcu/tree.c:2169
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 35 Comm: kworker/u4:2 Not tainted 5.14.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x22e/0x9d0 kernel/smp.c:967
Code: 38 d0 7c 08 84 d2 0f 85 2b 05 00 00 8b 43 08 a8 01 74 2e 48 89 ca 49 89 cf 48 c1 ea 03 41 83 e7 07 4c 01 e2 41 83 c7 03 f3 90 <0f> b6 02 41 38 c7 7c 08 84 c0 0f 85 d4 04 00 00 8b 43 08 a8 01 75
RSP: 0018:ffffc90000e8fa68 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff8880b9e3c8c0 RCX: ffff8880b9e3c8c8
RDX: ffffed10173c7919 RSI: ffff8880b9f36bc8 RDI: ffffffff8a5f3880
RBP: ffff8880b9f36bc0 R08: 0000000000000000 R09: ffffffff8edb98e7
R10: 0000000000000001 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff8880b9f36bc8 R14: ffffed10173e6d78 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000051e370 CR3: 000000000a88e000 CR4: 00000000001506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
on_each_cpu_cond_mask+0x3f/0x70 kernel/smp.c:1133
on_each_cpu include/linux/smp.h:71 [inline]
text_poke_sync arch/x86/kernel/alternative.c:929 [inline]
text_poke_bp_batch+0x47f/0x560 arch/x86/kernel/alternative.c:1183
text_poke_flush arch/x86/kernel/alternative.c:1268 [inline]
text_poke_flush arch/x86/kernel/alternative.c:1265 [inline]
text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1275
arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x15f/0x220 kernel/jump_label.c:177
static_key_enable+0x11/0x20 kernel/jump_label.c:190
toggle_allocation_gate mm/kfence/core.c:623 [inline]
toggle_allocation_gate+0xe3/0x310 mm/kfence/core.c:615
process_one_work+0x84c/0x13d0 kernel/workqueue.c:2276
worker_thread+0x598/0x1040 kernel/workqueue.c:2422
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
NMI backtrace for cpu 0
CPU: 0 PID: 8405 Comm: kworker/u4:8 Not tainted 5.14.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: phy10 ieee80211_iface_work
Call Trace:
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x57/0x7d lib/dump_stack.c:105
nmi_cpu_backtrace.cold+0x30/0x99 lib/nmi_backtrace.c:105
nmi_trigger_cpumask_backtrace+0x11f/0x170 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:342
print_cpu_stall kernel/rcu/tree_stall.h:625 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:700 [inline]
rcu_pending kernel/rcu/tree.c:3922 [inline]
rcu_sched_clock_irq.cold+0x9f/0x747 kernel/rcu/tree.c:2641
update_process_times+0x13b/0x1c0 kernel/time/timer.c:1785
tick_sched_handle+0x6f/0x130 kernel/time/tick-sched.c:226
tick_sched_timer+0x132/0x210 kernel/time/tick-sched.c:1421
__run_hrtimer kernel/time/hrtimer.c:1537 [inline]
__hrtimer_run_queues+0x18a/0xb00 kernel/time/hrtimer.c:1601
hrtimer_interrupt+0x2f9/0x920 kernel/time/hrtimer.c:1663
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1106
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1100
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:console_unlock+0x559/0x8d0 kernel/printk/printk.c:2653
Code: a8 8a e8 9a 25 fe ff e8 65 2c 00 00 4d 85 f6 0f 85 b8 01 00 00 9c 58 f6 c4 02 0f 85 9f 02 00 00 4d 85 f6 74 01 fb 8b 54 24 30 <85> d2 0f 84 00 fc ff ff 31 d2 be 5e 0a 00 00 48 c7 c7 80 e4 cb 88
RSP: 0018:ffffc90002cef960 EFLAGS: 00000206
RAX: 0000000000000002 RBX: dffffc0000000000 RCX: 1ffffffff1db740a
RDX: 0000000000000000 RSI: ffffffff88cb28c0 RDI: ffffffff891ffc00
RBP: ffffc90002cef9b8 R08: 0000000000000001 R09: ffffffff8edb98e7
R10: 0000000000000001 R11: 615f6e616d746162 R12: 0000000000000000
R13: ffffffff8b7ba610 R14: 0000000000000200 R15: 0000000000000000
vprintk_emit+0x110/0x330 kernel/printk/printk.c:2174
printk+0xad/0xde kernel/printk/printk.c:2216
ieee80211_sta_find_ibss net/mac80211/ibss.c:1482 [inline]
ieee80211_ibss_work.cold+0x233/0x4c6 net/mac80211/ibss.c:1708
process_one_work+0x84c/0x13d0 kernel/workqueue.c:2276
worker_thread+0x598/0x1040 kernel/workqueue.c:2422
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
watchdog: BUG: soft lockup - CPU#1 stuck for 184s! [kworker/u4:2:35]
Modules linked in:
irq event stamp: 916376
hardirqs last enabled at (916375): [] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
hardirqs last disabled at (916376): [] sysvec_apic_timer_interrupt+0xb/0xc0 arch/x86/kernel/apic/apic.c:1100
softirqs last enabled at (877106): [] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last enabled at (877106): [] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
softirqs last disabled at (877089): [] invoke_softirq kernel/softirq.c:432 [inline]
softirqs last disabled at (877089): [] __irq_exit_rcu+0x16e/0x1c0 kernel/softirq.c:636
CPU: 1 PID: 35 Comm: kworker/u4:2 Not tainted 5.14.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x231/0x9d0 kernel/smp.c:967
Code: 08 84 d2 0f 85 2b 05 00 00 8b 43 08 a8 01 74 2e 48 89 ca 49 89 cf 48 c1 ea 03 41 83 e7 07 4c 01 e2 41 83 c7 03 f3 90 0f b6 02 <41> 38 c7 7c 08 84 c0 0f 85 d4 04 00 00 8b 43 08 a8 01 75 e7 e9 4a
RSP: 0018:ffffc90000e8fa68 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffff8880b9e3c8c0 RCX: ffff8880b9e3c8c8
RDX: ffffed10173c7919 RSI: ffff8880b9f36bc8 RDI: ffffffff8a5f3880
RBP: ffff8880b9f36bc0 R08: 0000000000000000 R09: ffffffff8edb98e7
R10: 0000000000000001 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff8880b9f36bc8 R14: ffffed10173e6d78 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000051e370 CR3: 000000000a88e000 CR4: 00000000001506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
on_each_cpu_cond_mask+0x3f/0x70 kernel/smp.c:1133
on_each_cpu include/linux/smp.h:71 [inline]
text_poke_sync arch/x86/kernel/alternative.c:929 [inline]
text_poke_bp_batch+0x47f/0x560 arch/x86/kernel/alternative.c:1183
text_poke_flush arch/x86/kernel/alternative.c:1268 [inline]
text_poke_flush arch/x86/kernel/alternative.c:1265 [inline]
text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1275
arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x15f/0x220 kernel/jump_label.c:177
static_key_enable+0x11/0x20 kernel/jump_label.c:190
toggle_allocation_gate mm/kfence/core.c:623 [inline]
toggle_allocation_gate+0xe3/0x310 mm/kfence/core.c:615
process_one_work+0x84c/0x13d0 kernel/workqueue.c:2276
worker_thread+0x598/0x1040 kernel/workqueue.c:2422
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 8405 Comm: kworker/u4:8 Not tainted 5.14.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: phy10 ieee80211_iface_work
RIP: 0010:check_region_inline mm/kasan/generic.c:175 [inline]
RIP: 0010:kasan_check_range+0x27/0x180 mm/kasan/generic.c:189
Code: 0f 1f 00 48 85 f6 0f 84 70 01 00 00 49 89 f9 41 54 44 0f b6 c2 49 01 f1 55 53 0f 82 18 01 00 00 48 b8 ff ff ff ff ff 7f ff ff <48> 39 c7 0f 86 05 01 00 00 49 83 e9 01 48 89 fd 48 b8 00 00 00 00
RSP: 0018:ffffc90000007d80 EFLAGS: 00000086
RAX: ffff7fffffffffff RBX: 1ffff92000000fb5 RCX: ffffffff815361e0
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffc90000007dc8
RBP: ffff8880b9e26400 R08: 0000000000000001 R09: ffffc90000007dcc
R10: ffffed10173c4c80 R11: 0000000000000001 R12: ffff8880b9e26408
R13: ffff8880b9e26410 R14: 0000000000000001 R15: ffff8880b9e26400
FS: 0000000000000000(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fba32dff000 CR3: 000000000a88e000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
instrument_atomic_read_write include/linux/instrumented.h:101 [inline]
atomic_try_cmpxchg_acquire include/asm-generic/atomic-instrumented.h:512 [inline]
queued_spin_lock include/asm-generic/qspinlock.h:82 [inline]
do_raw_spin_lock+0x120/0x2b0 kernel/locking/spinlock_debug.c:113
__run_hrtimer kernel/time/hrtimer.c:1541 [inline]
__hrtimer_run_queues+0x1dd/0xb00 kernel/time/hrtimer.c:1601
hrtimer_interrupt+0x2f9/0x920 kernel/time/hrtimer.c:1663
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1106
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1100
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:console_unlock+0x559/0x8d0 kernel/printk/printk.c:2653
Code: a8 8a e8 9a 25 fe ff e8 65 2c 00 00 4d 85 f6 0f 85 b8 01 00 00 9c 58 f6 c4 02 0f 85 9f 02 00 00 4d 85 f6 74 01 fb 8b 54 24 30 <85> d2 0f 84 00 fc ff ff 31 d2 be 5e 0a 00 00 48 c7 c7 80 e4 cb 88
RSP: 0018:ffffc90002cef960 EFLAGS: 00000206
RAX: 0000000000000002 RBX: dffffc0000000000 RCX: 1ffffffff1db740a
RDX: 0000000000000000 RSI: ffffffff88cb28c0 RDI: ffffffff891ffc00
RBP: ffffc90002cef9b8 R08: 0000000000000001 R09: ffffffff8edb98e7
R10: 0000000000000001 R11: 615f6e616d746162 R12: 0000000000000000
R13: ffffffff8b7ba610 R14: 0000000000000200 R15: 0000000000000000
vprintk_emit+0x110/0x330 kernel/printk/printk.c:2174
printk+0xad/0xde kernel/printk/printk.c:2216
ieee80211_sta_find_ibss net/mac80211/ibss.c:1482 [inline]
ieee80211_ibss_work.cold+0x233/0x4c6 net/mac80211/ibss.c:1708
process_one_work+0x84c/0x13d0 kernel/workqueue.c:2276
worker_thread+0x598/0x1040 kernel/workqueue.c:2422
kthread+0x38b/0x460 kernel/kthread.c:319
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
----------------
Code disassembly (best guess):
0: 38 d0 cmp %dl,%al
2: 7c 08 jl 0xc
4: 84 d2 test %dl,%dl
6: 0f 85 2b 05 00 00 jne 0x537
c: 8b 43 08 mov 0x8(%rbx),%eax
f: a8 01 test $0x1,%al
11: 74 2e je 0x41
13: 48 89 ca mov %rcx,%rdx
16: 49 89 cf mov %rcx,%r15
19: 48 c1 ea 03 shr $0x3,%rdx
1d: 41 83 e7 07 and $0x7,%r15d
21: 4c 01 e2 add %r12,%rdx
24: 41 83 c7 03 add $0x3,%r15d
28: f3 90 pause
* 2a: 0f b6 02 movzbl (%rdx),%eax <-- trapping instruction
2d: 41 38 c7 cmp %al,%r15b
30: 7c 08 jl 0x3a
32: 84 c0 test %al,%al
34: 0f 85 d4 04 00 00 jne 0x50e
3a: 8b 43 08 mov 0x8(%rbx),%eax
3d: a8 01 test $0x1,%al
3f: 75 .byte 0x75