bisecting fixing commit since 79b69a83705e621b258ac6d8ae6d3bfdb4b930aa
building syzkaller on 6b3c5e64ee67d74980d229cd1332aa038d4ce936
testing commit 79b69a83705e621b258ac6d8ae6d3bfdb4b930aa
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2
kernel signature: 5f521a3f897e066d0f48e955c097037d0d8d6d8fcb75a1a6c7b2a435a1482201
run #0: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #1: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #2: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #3: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #4: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #5: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #6: crashed: BUG: soft lockup in smp_call_function
run #7: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #10: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #11: crashed: BUG: soft lockup in smp_call_function
run #12: crashed: BUG: soft lockup in smp_call_function
run #13: crashed: BUG: soft lockup in smp_call_function
run #14: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #15: crashed: BUG: soft lockup in smp_call_function
run #16: crashed: BUG: soft lockup in smp_call_function
run #17: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #18: crashed: INFO: rcu detected stall in tc_modify_qdisc
run #19: crashed: INFO: rcu detected stall in tc_modify_qdisc
testing current HEAD 7fb0269720d7d5359bc8349eef908a1c96866b65
testing commit 7fb0269720d7d5359bc8349eef908a1c96866b65
compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2
kernel signature: a65a97549d1660290430900c80899d1357e701928e8fbed8281868358953d3ab
all runs: crashed: INFO: rcu detected stall in tc_modify_qdisc
revisions tested: 2, total time: 24m59.526604434s (build: 12m6.286558421s, test: 12m10.753988403s)
the crash still happens on HEAD
commit msg: Merge tag 'for-net-2022-05-23' of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth
crash: INFO: rcu detected stall in tc_modify_qdisc
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 1-...!: (10500 ticks this GP) idle=415/1/0x4000000000000000 softirq=7211/7216 fqs=0
(t=10501 jiffies g=6413 q=120)
rcu: rcu_preempt kthread timer wakeup didn't happen for 10500 jiffies! g6413 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: Possible timer handling issue on cpu=0 timer-softirq=2237
rcu: rcu_preempt kthread starved for 10501 jiffies! g6413 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:I stack:28624 pid: 16 ppid: 2 flags:0x00004000
Call Trace:
context_switch kernel/sched/core.c:5073 [inline]
__schedule+0xa5a/0x4c80 kernel/sched/core.c:6388
schedule+0xd2/0x1f0 kernel/sched/core.c:6460
schedule_timeout+0x11d/0x250 kernel/time/timer.c:1884
rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1971
rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2144
kthread+0x299/0x340 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 4085 Comm: syz-executor.0 Not tainted 5.18.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:lookup_chain_cache kernel/locking/lockdep.c:3697 [inline]
RIP: 0010:lookup_chain_cache_add kernel/locking/lockdep.c:3716 [inline]
RIP: 0010:validate_chain kernel/locking/lockdep.c:3771 [inline]
RIP: 0010:__lock_acquire+0x1642/0x5660 kernel/locking/lockdep.c:5029
Code: db 74 58 48 83 eb 08 74 52 48 ba 00 00 00 00 00 fc ff df eb 06 48 83 eb 08 74 40 48 8d 7b 18 48 89 f8 48 c1 e8 03 80 3c 10 00 <0f> 85 c3 2f 00 00 48 8b 43 18 49 39 c7 0f 84 84 f6 ff ff 48 8d 7b
RSP: 0018:ffffc90000007af0 EFLAGS: 00000046
RAX: 1ffffffff1c904e7 RBX: ffffffff8e482720 RCX: ffffffff8154b85c
RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffffffff8e482738
RBP: 0000000000002c9b R08: 0000000000000000 R09: ffffffff8f12b907
R10: fffffbfff1e25720 R11: 0000000000000001 R12: ffff88801efe2818
R13: ffff88801efe1d40 R14: 0000000000000000 R15: 5198900cfc7cfb74
FS: 00007f16a3c29700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000600 CR3: 0000000019aa3000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
lock_acquire kernel/locking/lockdep.c:5641 [inline]
lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5606
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162
debug_object_deactivate lib/debugobjects.c:735 [inline]
debug_object_deactivate+0x101/0x300 lib/debugobjects.c:723
debug_hrtimer_deactivate kernel/time/hrtimer.c:425 [inline]
debug_deactivate kernel/time/hrtimer.c:481 [inline]
__run_hrtimer kernel/time/hrtimer.c:1653 [inline]
__hrtimer_run_queues+0x337/0xb00 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 5a cf c1 f8 48 89 ef e8 a2 4e c2 f8 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 43 70 b5 f8 65 8b 05 2c 3a 6f 77 85 c0 74 0a 5b 5d c3 e8 f0 07
RSP: 0018:ffffc90002ecf290 EFLAGS: 00000206
RAX: 0000000000000012 RBX: 0000000000000200 RCX: 1ffffffff1e2e766
RDX: 0000000000000000 RSI: ffffffff88eb9060 RDI: 0000000000000001
RBP: ffff88806efca2e8 R08: 0000000000000001 R09: ffffffff8f12b9bf
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000246
R13: 16f1ff9a5d3e57a0 R14: 0000000000000000 R15: ffff88806efca0f0
spin_unlock_irqrestore include/linux/spinlock.h:404 [inline]
taprio_change+0x25a5/0x3a80 net/sched/sch_taprio.c:1607
qdisc_create.constprop.0+0x39c/0xdc0 net/sched/sch_api.c:1253
tc_modify_qdisc+0x3cc/0x1680 net/sched/sch_api.c:1660
rtnetlink_rcv_msg+0x31d/0x8d0 net/core/rtnetlink.c:5993
netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2502
netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline]
netlink_unicast+0x433/0x710 net/netlink/af_netlink.c:1345
netlink_sendmsg+0x770/0xc20 net/netlink/af_netlink.c:1921
sock_sendmsg_nosec net/socket.c:705 [inline]
sock_sendmsg+0xab/0xe0 net/socket.c:725
____sys_sendmsg+0x5b9/0x7a0 net/socket.c:2413
___sys_sendmsg+0xd3/0x150 net/socket.c:2467
__sys_sendmsg+0xb2/0x140 net/socket.c:2496
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0x80 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f16a2a88e99
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f16a3c29168 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007f16a2b9bf60 RCX: 00007f16a2a88e99
RDX: 0000000000000000 RSI: 00000000200007c0 RDI: 0000000000000004
RBP: 00007f16a2ae2ff1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffd60c8b48f R14: 00007f16a3c29300 R15: 0000000000022000
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 4085 Comm: syz-executor.0 Not tainted 5.18.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:mark_lock.part.0+0x53/0x19a0 kernel/locking/lockdep.c:4566
Code: 24 48 c7 44 24 38 b3 8a b5 41 48 c1 eb 03 48 c7 44 24 40 c8 2f 69 8a 48 8d 04 13 48 c7 44 24 48 20 84 54 81 c7 00 f1 f1 f1 f1 40 04 f1 f1 04 f2 c7 40 08 00 f2 f2 f2 c7 40 0c 00 f2 f2 f2 c7
RSP: 0018:ffffc90000007a10 EFLAGS: 00000802
RAX: fffff52000000f49 RBX: 1ffff92000000f49 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: ffff88801efe27f0 RDI: ffff88801efe1d40
RBP: 0000000000000002 R08: dffffc0000000000 R09: ffffffff8f12b907
R10: fffffbfff1e25720 R11: 0000000000000001 R12: ffff88801efe27f0
R13: 0000000000000008 R14: 0000000000000002 R15: 0000000000000040
FS: 00007f16a3c29700(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000600 CR3: 0000000019aa3000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
mark_lock kernel/locking/lockdep.c:4571 [inline]
mark_usage kernel/locking/lockdep.c:4528 [inline]
__lock_acquire+0x8d2/0x5660 kernel/locking/lockdep.c:4983
lock_acquire kernel/locking/lockdep.c:5641 [inline]
lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5606
rcu_lock_acquire include/linux/rcupdate.h:268 [inline]
rcu_read_lock include/linux/rcupdate.h:694 [inline]
advance_sched+0x3e5/0x920 net/sched/sch_taprio.c:768
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x4d7/0xb00 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 5a cf c1 f8 48 89 ef e8 a2 4e c2 f8 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 43 70 b5 f8 65 8b 05 2c 3a 6f 77 85 c0 74 0a 5b 5d c3 e8 f0 07
RSP: 0018:ffffc90002ecf290 EFLAGS: 00000206
RAX: 0000000000000012 RBX: 0000000000000200 RCX: 1ffffffff1e2e766
RDX: 0000000000000000 RSI: ffffffff88eb9060 RDI: 0000000000000001
RBP: ffff88806efca2e8 R08: 0000000000000001 R09: ffffffff8f12b9bf
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000246
R13: 16f1ff9a5d3e57a0 R14: 0000000000000000 R15: ffff88806efca0f0
spin_unlock_irqrestore include/linux/spinlock.h:404 [inline]
taprio_change+0x25a5/0x3a80 net/sched/sch_taprio.c:1607
qdisc_create.constprop.0+0x39c/0xdc0 net/sched/sch_api.c:1253
tc_modify_qdisc+0x3cc/0x1680 net/sched/sch_api.c:1660
rtnetlink_rcv_msg+0x31d/0x8d0 net/core/rtnetlink.c:5993
netlink_rcv_skb+0x118/0x370 net/netlink/af_netlink.c:2502
netlink_unicast_kernel net/netlink/af_netlink.c:1319 [inline]
netlink_unicast+0x433/0x710 net/netlink/af_netlink.c:1345
netlink_sendmsg+0x770/0xc20 net/netlink/af_netlink.c:1921
sock_sendmsg_nosec net/socket.c:705 [inline]
sock_sendmsg+0xab/0xe0 net/socket.c:725
____sys_sendmsg+0x5b9/0x7a0 net/socket.c:2413
___sys_sendmsg+0xd3/0x150 net/socket.c:2467
__sys_sendmsg+0xb2/0x140 net/socket.c:2496
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0x80 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f16a2a88e99
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f16a3c29168 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00007f16a2b9bf60 RCX: 00007f16a2a88e99
RDX: 0000000000000000 RSI: 00000000200007c0 RDI: 0000000000000004
RBP: 00007f16a2ae2ff1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffd60c8b48f R14: 00007f16a3c29300 R15: 0000000000022000
NMI backtrace for cpu 1
CPU: 1 PID: 633 Comm: kworker/u4:4 Not tainted 5.18.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106
nmi_cpu_backtrace.cold+0x30/0xc0 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x140/0x170 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x262/0x3f0 kernel/rcu/tree_stall.h:343
print_cpu_stall kernel/rcu/tree_stall.h:639 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:723 [inline]
rcu_pending kernel/rcu/tree.c:3923 [inline]
rcu_sched_clock_irq.cold+0xa1/0x8d6 kernel/rcu/tree.c:2625
update_process_times+0x13b/0x1c0 kernel/time/timer.c:1788
tick_sched_handle+0x6f/0x130 kernel/time/tick-sched.c:243
tick_sched_timer+0xaa/0xd0 kernel/time/tick-sched.c:1473
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x18a/0xb00 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0x291/0xa50 kernel/smp.c:969
Code: 38 d0 7c 08 84 d2 0f 85 71 05 00 00 8b 43 08 a8 01 74 2e 48 89 ca 49 89 cf 48 c1 ea 03 41 83 e7 07 4c 01 e2 41 83 c7 03 f3 90 <0f> b6 02 41 38 c7 7c 08 84 c0 0f 85 b1 04 00 00 8b 43 08 a8 01 75
RSP: 0018:ffffc9000281fa40 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff8880b9e42340 RCX: ffff8880b9e42348
RDX: ffffed10173c8469 RSI: ffff8880b9f3b088 RDI: ffffffff8a8838e0
RBP: ffff8880b9f3b080 R08: 0000000000000000 R09: ffffffff8f12b907
R10: 0000000000000001 R11: 0000000000000001 R12: dffffc0000000000
R13: ffff8880b9f3b088 R14: ffffed10173e7610 R15: 0000000000000003
on_each_cpu_cond_mask+0x3f/0x70 kernel/smp.c:1135
on_each_cpu include/linux/smp.h:71 [inline]
text_poke_sync arch/x86/kernel/alternative.c:1184 [inline]
text_poke_bp_batch+0x22c/0x6c0 arch/x86/kernel/alternative.c:1372
text_poke_flush arch/x86/kernel/alternative.c:1542 [inline]
text_poke_flush arch/x86/kernel/alternative.c:1539 [inline]
text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1549
arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x15f/0x220 kernel/jump_label.c:177
static_key_enable+0x11/0x20 kernel/jump_label.c:190
toggle_allocation_gate mm/kfence/core.c:774 [inline]
toggle_allocation_gate+0xe3/0x310 mm/kfence/core.c:766
process_one_work+0x865/0x13d0 kernel/workqueue.c:2289
worker_thread+0x598/0xec0 kernel/workqueue.c:2436
kthread+0x299/0x340 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
----------------
Code disassembly (best guess):
0: db 74 58 48 (bad) 0x48(%rax,%rbx,2)
4: 83 eb 08 sub $0x8,%ebx
7: 74 52 je 0x5b
9: 48 ba 00 00 00 00 00 movabs $0xdffffc0000000000,%rdx
10: fc ff df
13: eb 06 jmp 0x1b
15: 48 83 eb 08 sub $0x8,%rbx
19: 74 40 je 0x5b
1b: 48 8d 7b 18 lea 0x18(%rbx),%rdi
1f: 48 89 f8 mov %rdi,%rax
22: 48 c1 e8 03 shr $0x3,%rax
26: 80 3c 10 00 cmpb $0x0,(%rax,%rdx,1)
* 2a: 0f 85 c3 2f 00 00 jne 0x2ff3 <-- trapping instruction
30: 48 8b 43 18 mov 0x18(%rbx),%rax
34: 49 39 c7 cmp %rax,%r15
37: 0f 84 84 f6 ff ff je 0xfffff6c1
3d: 48 rex.W
3e: 8d .byte 0x8d
3f: 7b .byte 0x7b