syzbot


INFO: rcu detected stall in handle_softirqs

Status: auto-obsoleted due to no activity on 2025/02/26 20:58
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+afcbef13b9fa6ae41f9a@syzkaller.appspotmail.com
First crash: 325d, last: 116d
Cause bisection: introduced by (bisect log) :
commit d15121be7485655129101f3960ae6add40204463
Author: Paolo Abeni <pabeni@redhat.com>
Date: Mon May 8 06:17:44 2023 +0000

  Revert "softirq: Let ksoftirqd do its job"

Crash: INFO: rcu detected stall in corrupted (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit 5f6bd380c7bdbe10f7b4e8ddcceed60ce0714c6d
Author: Peter Zijlstra <peterz@infradead.org>
Date: Mon May 27 12:06:55 2024 +0000

  sched/rt: Remove default bandwidth control

  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [bpf?] [net?] INFO: rcu detected stall in handle_softirqs 1 (4) 2025/01/07 14:37
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in handle_softirqs 2 246d 259d 0/3 auto-obsoleted due to no activity on 2024/11/01 18:35
Last patch testing requests (4)
Created Duration User Patch Repo Result
2024/11/24 13:45 23m retest repro upstream OK log
2024/08/15 19:48 36m retest repro upstream report log
2024/05/25 23:08 1h35m retest repro upstream report log
2024/05/11 11:06 27m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ee5b455b0ada report log
Fix bisection attempts (2)
Created Duration User Patch Repo Result
2025/01/07 07:54 6h42m bisect fix upstream OK (1) job log
2024/09/01 01:58 1h59m bisect fix upstream OK (0) job log log

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P7687/1:b..l
rcu: 	(detected by 1, t=10502 jiffies, g=33925, q=633 ncpus=2)
task:syz-executor    state:R  running task     stack:20184 pid:7687  tgid:7687  ppid:7666   flags:0x00000000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0x1850/0x4c30 kernel/sched/core.c:6756
 preempt_schedule_common+0x84/0xd0 kernel/sched/core.c:6935
 preempt_schedule+0xe1/0xf0 kernel/sched/core.c:6959
 preempt_schedule_thunk+0x1a/0x30 arch/x86/entry/thunk.S:12
 __raw_spin_unlock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_unlock+0x3e/0x50 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:391 [inline]
 copy_pte_range mm/memory.c:1200 [inline]
 copy_pmd_range+0x74aa/0x7eb0 mm/memory.c:1261
 copy_pud_range mm/memory.c:1298 [inline]
 copy_p4d_range mm/memory.c:1322 [inline]
 copy_page_range+0x99f/0xe90 mm/memory.c:1420
 dup_mmap kernel/fork.c:751 [inline]
 dup_mm kernel/fork.c:1695 [inline]
 copy_mm+0x11c7/0x1f90 kernel/fork.c:1744
 copy_process+0x1845/0x3d50 kernel/fork.c:2395
 kernel_clone+0x226/0x8e0 kernel/fork.c:2807
 __do_sys_clone kernel/fork.c:2950 [inline]
 __se_sys_clone kernel/fork.c:2934 [inline]
 __x64_sys_clone+0x258/0x2a0 kernel/fork.c:2934
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fdeed377093
RSP: 002b:00007fdeed66fcb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdeed377093
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
R10: 00005555851117d0 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000000386a1 R14: 000000000003801e R15: 00007fdeed66fe40
 </TASK>
rcu: rcu_preempt kthread starved for 4203 jiffies! g33925 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:25392 pid:17    tgid:17    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0x1850/0x4c30 kernel/sched/core.c:6756
 __schedule_loop kernel/sched/core.c:6833 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6848
 schedule_timeout+0x15a/0x290 kernel/time/sleep_timeout.c:99
 rcu_gp_fqs_loop+0x2df/0x1330 kernel/rcu/tree.c:2045
 rcu_gp_kthread+0xa7/0x3b0 kernel/rcu/tree.c:2247
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.12.0-syzkaller-12113-gbcc8eda6d349 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:__kasan_check_read+0xa/0x20 mm/kasan/shadow.c:31
Code: db eb d0 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 89 f6 48 8b 0c 24 <31> d2 e9 6f e6 ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
RSP: 0018:ffffc90000007c58 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff817ad2f0
RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffffffff901945f0
RBP: ffffc90000007d90 R08: ffffffff942b888f R09: 1ffffffff2857111
R10: dffffc0000000000 R11: fffffbfff2857112 R12: 1ffff92000000f98
R13: ffffffff818d8c9d R14: ffffc90000007dc0 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f38d6e448e5 CR3: 000000000e736000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 instrument_atomic_read include/linux/instrumented.h:68 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 cpumask_test_cpu include/linux/cpumask.h:570 [inline]
 cpu_online include/linux/cpumask.h:1117 [inline]
 trace_lock_release include/trace/events/lock.h:69 [inline]
 lock_release+0xb0/0xa30 kernel/locking/lockdep.c:5860
 seqcount_lockdep_reader_access+0x10f/0x220 include/linux/seqlock.h:71
 ktime_get_update_offsets_now+0x2d/0x3b0 kernel/time/timekeeping.c:2452
 hrtimer_update_base kernel/time/hrtimer.c:625 [inline]
 hrtimer_interrupt+0x131/0xa40 kernel/time/hrtimer.c:1847
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
 __sysvec_apic_timer_interrupt+0x110/0x420 arch/x86/kernel/apic/apic.c:1055
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xd8/0x140 kernel/locking/spinlock.c:194
Code: 9c 8f 44 24 20 42 80 3c 23 00 74 08 4c 89 f7 e8 fe 48 37 f6 f6 44 24 21 02 75 52 41 f7 c7 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> d3 72 9a f5 65 8b 05 84 c5 30 74 85 c0 74 43 48 c7 04 24 0e 36
RSP: 0018:ffffc90000157940 EFLAGS: 00000206
RAX: 57e115ce37691c00 RBX: 1ffff9200002af2c RCX: ffffffff817b239a
RDX: dffffc0000000000 RSI: ffffffff8c0a9680 RDI: 0000000000000001
RBP: ffffc900001579c8 R08: ffffffff942b89df R09: 1ffffffff285713b
R10: dffffc0000000000 R11: fffffbfff285713c R12: dffffc0000000000
R13: 1ffff9200002af28 R14: ffffc90000157960 R15: 0000000000000246
 qdisc_watchdog_schedule_ns include/net/pkt_sched.h:81 [inline]
 tbf_dequeue+0x65e/0xd40 net/sched/sch_tbf.c:310
 dequeue_skb net/sched/sch_generic.c:293 [inline]
 qdisc_restart net/sched/sch_generic.c:398 [inline]
 __qdisc_run+0x272/0x2180 net/sched/sch_generic.c:416
 qdisc_run+0xda/0x270 include/net/pkt_sched.h:127
 net_tx_action+0x896/0xa50 net/core/dev.c:5334
 handle_softirqs+0x2d4/0x9b0 kernel/softirq.c:554
 run_ksoftirqd+0xca/0x130 kernel/softirq.c:943
 smpboot_thread_fn+0x544/0xa30 kernel/smpboot.c:164
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (34):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/12/01 15:57 upstream bcc8eda6d349 68914665 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/11/10 13:36 upstream de2f378f2b77 6b856513 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/10/29 02:08 upstream 819837584309 9efb3cc7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/10/24 18:12 upstream c2ee9f594da8 9fc8fe02 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in handle_softirqs
2024/10/24 09:45 upstream c2ee9f594da8 15fa2979 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in handle_softirqs
2024/10/02 23:39 upstream f23aa4c0761a a4c7fd36 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in handle_softirqs
2024/07/29 04:34 upstream 5437f30d3458 46eb10b7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in handle_softirqs
2024/07/27 20:22 upstream 3a7e02c040b1 46eb10b7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in handle_softirqs
2024/07/10 12:06 upstream 34afb82a3c67 e7213be3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/07/05 18:55 upstream 661e504db04c 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/07/05 13:00 upstream 661e504db04c 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/06/23 21:53 upstream 7c16f0a4ed1c edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in handle_softirqs
2024/05/11 18:06 upstream cf87f46fd34d 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/05/07 07:21 upstream ee5b455b0ada c035c6de .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in handle_softirqs
2024/10/25 19:51 net-next 6d858708d465 65e8686b .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/10/17 13:43 net-next 6d858708d465 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/10/02 00:16 net-next 44badc908f2c ea2b66a6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: rcu detected stall in handle_softirqs
2024/08/01 19:08 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing d483f034f032 1e9c4cf3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/31 14:41 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 8400291e289e 6fde257d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/29 22:36 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 1722389b0d86 5187fc86 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/29 20:05 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 1722389b0d86 5187fc86 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/22 12:42 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 933069701c1b df655b64 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/20 21:44 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 3c3ff7be9729 b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/20 08:04 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 3c3ff7be9729 b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/20 06:54 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 3c3ff7be9729 b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/19 22:54 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing b72749301112 890ce4f3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/17 02:21 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing b72749301112 215bec2d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/08 06:14 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 179264157bba 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/07 10:25 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 179264157bba 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/07 00:55 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 179264157bba 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/07/04 16:35 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 179264157bba 3f2748a3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/06/30 09:07 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 2ea6d07efe53 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/06/25 09:47 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 819984a0dd36 04bd2a30 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
2024/06/21 18:12 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 819984a0dd36 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in handle_softirqs
* Struck through repros no longer work on HEAD.