syzbot

 sign-in | mailing list | source | docs
🐞 Open [569] 🐞 Fixed [66] 🐞 Invalid [207] ⬇ Missing Backports [40] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

INFO: rcu detected stall in kauditd_thread (2)

Status: upstream: reported on 2023/12/11 16:51
Reported-by: syzbot+4d37ebcdb19e4c39aac3@syzkaller.appspotmail.com
First crash: 144d, last: 38d
▶ ▼ Similar bugs (6)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in kauditd_thread (3) kernel 2 163d 196d 0/26 auto-obsoleted due to no activity on 2024/02/20 23:02
upstream INFO: rcu detected stall in kauditd_thread kernel 2 832d 887d 0/26 closed as invalid on 2022/02/08 10:00
upstream INFO: rcu detected stall in kauditd_thread (4) audit bpf C 5 8d07h 37d 0/26 upstream: reported C repro on 2024/03/27 18:39
linux-6.1 INFO: rcu detected stall in kauditd_thread 2 258d 344d 0/3 auto-obsoleted due to no activity on 2023/11/27 06:59
linux-5.15 INFO: rcu detected stall in kauditd_thread 2 133d 150d 0/3 auto-obsoleted due to no activity on 2024/03/31 22:45
android-5-15 BUG: soft lockup in kauditd_thread origin:lts C 2 14d 29d 0/2 upstream: reported C repro on 2024/04/04 20:01

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	1-...!: (1 GPs behind) idle=b0dc/1/0x4000000000000000 softirq=48543/48544 fqs=175
	(detected by 0, t=10502 jiffies, g=62421, q=530 ncpus=2)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 27 Comm: kauditd Not tainted 6.1.82-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
RIP: 0010:arch_atomic_read arch/x86/include/asm/atomic.h:29 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs include/linux/context_tracking.h:122 [inline]
RIP: 0010:rcu_is_watching+0x56/0xb0 kernel/rcu/tree.c:721
Code: f0 48 c1 e8 03 42 80 3c 38 00 74 08 4c 89 f7 e8 f0 bb 6e 00 48 c7 c3 a8 56 03 00 49 03 1e 48 89 d8 48 c1 e8 03 42 0f b6 04 38 <84> c0 75 1e 8b 03 65 ff 0d 65 32 8f 7e 74 0c 83 e0 04 c1 e8 02 5b
RSP: 0018:ffffc900001e0b60 EFLAGS: 00000802
RAX: 0000000000000000 RBX: ffff8880b99356a8 RCX: ffffffff816a7857
RDX: 0000000000000000 RSI: ffffffff8b3d2b20 RDI: ffffffff8b3d2ae0
RBP: ffffc900001e0cb0 R08: dffffc0000000000 R09: fffffbfff1ce6c56
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff9200003c17c
R13: ffffffff88b3cb80 R14: ffffffff8cb0e858 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f101494affc CR3: 00000000619bb000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 trace_lock_release include/trace/events/lock.h:69 [inline]
 lock_release+0xd6/0xa20 kernel/locking/lockdep.c:5673
 rcu_lock_release include/linux/rcupdate.h:324 [inline]
 rcu_read_unlock include/linux/rcupdate.h:793 [inline]
 advance_sched+0x800/0x970 net/sched/sch_taprio.c:755
 __run_hrtimer kernel/time/hrtimer.c:1686 [inline]
 __hrtimer_run_queues+0x5e5/0xe50 kernel/time/hrtimer.c:1750
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1812
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x156/0x580 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:653
RIP: 0010:console_emit_next_record+0xc69/0xea0 kernel/printk/printk.c:2751
Code: 00 e8 bb 31 1c 00 44 0f b6 74 24 1f 4d 85 e4 75 07 e8 ab 31 1c 00 eb 06 e8 a4 31 1c 00 fb 48 c7 84 24 a0 00 00 00 0e 36 e0 45 <42> c7 04 2b 00 00 00 00 4a c7 44 2b 0a 00 00 00 00 4a c7 44 2b 12
RSP: 0018:ffffc90000a3f760 EFLAGS: 00000293
RAX: ffffffff816e51ec RBX: dffffc0000000000 RCX: ffff888013a6d940
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000a3fa10 R08: ffffffff816e51c7 R09: fffffbfff2092c45
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000200
R13: 1ffff92000147f00 R14: 0000000000000001 R15: 0000000000000000
 console_unlock+0x278/0x7c0 kernel/printk/printk.c:2871
 vprintk_emit+0x523/0x740 kernel/printk/printk.c:2268
 _printk+0xd1/0x111 kernel/printk/printk.c:2293
 kauditd_printk_skb kernel/audit.c:547 [inline]
 kauditd_hold_skb+0x1b8/0x200 kernel/audit.c:582
 kauditd_send_queue+0x2a3/0x2f0 kernel/audit.c:767
 kauditd_thread+0x758/0x990 kernel/audit.c:891
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>
rcu: rcu_preempt kthread starved for 10151 jiffies! g62421 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:25528 pid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5245 [inline]
 __schedule+0x142d/0x4550 kernel/sched/core.c:6558
 schedule+0xbf/0x180 kernel/sched/core.c:6634
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1935
 rcu_gp_fqs_loop+0x2d2/0x1120 kernel/rcu/tree.c:1706
 rcu_gp_kthread+0xa3/0x3a0 kernel/rcu/tree.c:1905
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 0 PID: 3637 Comm: kworker/0:9 Not tainted 6.1.82-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
Workqueue: events jump_label_update_timeout
RIP: 0010:csd_lock_wait kernel/smp.c:424 [inline]
RIP: 0010:smp_call_function_many_cond+0x1fc0/0x3460 kernel/smp.c:998
Code: e5 01 49 bd 00 00 00 00 00 fc ff df 75 0a e8 77 3f 0b 00 e9 1b ff ff ff f3 90 42 0f b6 04 2b 84 c0 75 14 41 f7 07 01 00 00 00 <0f> 84 fe fe ff ff e8 55 3f 0b 00 eb e1 44 89 f9 80 e1 07 80 c1 03
RSP: 0018:ffffc900052ef6a0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 1ffff110173281b1 RCX: ffff8880771c1dc0
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900052efa80 R08: ffffffff817f4404 R09: fffffbfff2092c45
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000800000000
R13: dffffc0000000000 R14: 0000000000000001 R15: ffff8880b9940d88
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2d227000 CR3: 000000000ce8e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x3b/0x80 kernel/smp.c:1166
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1334 [inline]
 text_poke_bp_batch+0x2bb/0x940 arch/x86/kernel/alternative.c:1534
 text_poke_flush arch/x86/kernel/alternative.c:1725 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1732
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 __static_key_slow_dec_cpuslocked+0x107/0x160 kernel/jump_label.c:248
 __static_key_slow_dec kernel/jump_label.c:255 [inline]
 jump_label_update_timeout+0x1a/0x20 kernel/jump_label.c:263
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>
watchdog: BUG: soft lockup - CPU#0 stuck for 246s! [kworker/0:9:3637]
Modules linked in:
irq event stamp: 1962846
hardirqs last  enabled at (1962845): [<ffffffff8aa00d46>] asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:653
hardirqs last disabled at (1962846): [<ffffffff8a897c7a>] sysvec_apic_timer_interrupt+0xa/0xb0 arch/x86/kernel/apic/apic.c:1106
softirqs last  enabled at (1912180): [<ffffffff81541015>] invoke_softirq kernel/softirq.c:445 [inline]
softirqs last  enabled at (1912180): [<ffffffff81541015>] __irq_exit_rcu+0x155/0x240 kernel/softirq.c:650
softirqs last disabled at (1912131): [<ffffffff81541015>] invoke_softirq kernel/softirq.c:445 [inline]
softirqs last disabled at (1912131): [<ffffffff81541015>] __irq_exit_rcu+0x155/0x240 kernel/softirq.c:650
CPU: 0 PID: 3637 Comm: kworker/0:9 Not tainted 6.1.82-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
Workqueue: events jump_label_update_timeout
RIP: 0010:csd_lock_wait kernel/smp.c:424 [inline]
RIP: 0010:smp_call_function_many_cond+0x1fb0/0x3460 kernel/smp.c:998
Code: 2f 44 89 ee 83 e6 01 31 ff e8 ec 42 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 0a e8 77 3f 0b 00 e9 1b ff ff ff f3 90 <42> 0f b6 04 2b 84 c0 75 14 41 f7 07 01 00 00 00 0f 84 fe fe ff ff
RSP: 0018:ffffc900052ef6a0 EFLAGS: 00000293
RAX: ffffffff817f443b RBX: 1ffff110173281b1 RCX: ffff8880771c1dc0
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900052efa80 R08: ffffffff817f4404 R09: fffffbfff2092c45
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000800000000
R13: dffffc0000000000 R14: 0000000000000001 R15: ffff8880b9940d88
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2d227000 CR3: 000000000ce8e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x3b/0x80 kernel/smp.c:1166
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1334 [inline]
 text_poke_bp_batch+0x2bb/0x940 arch/x86/kernel/alternative.c:1534
 text_poke_flush arch/x86/kernel/alternative.c:1725 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1732
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 __static_key_slow_dec_cpuslocked+0x107/0x160 kernel/jump_label.c:248
 __static_key_slow_dec kernel/jump_label.c:255 [inline]
 jump_label_update_timeout+0x1a/0x20 kernel/jump_label.c:263
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 27 Comm: kauditd Not tainted 6.1.82-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
RIP: 0010:lookup_chain_cache_add kernel/locking/lockdep.c:3738 [inline]
RIP: 0010:validate_chain+0x183/0x5950 kernel/locking/lockdep.c:3793
Code: 35 90 48 89 d8 48 c1 e8 03 48 89 44 24 58 42 80 3c 20 00 74 08 48 89 df e8 da ec 76 00 48 89 5c 24 28 48 8b 1b 48 85 db 74 48 <48> 83 c3 f8 74 42 4c 8d 7b 18 4c 89 f8 48 c1 e8 03 42 80 3c 20 00
RSP: 0018:ffffc900001e07c0 EFLAGS: 00000082
RAX: 1ffffffff208a594 RBX: ffffffff90497c68 RCX: ffffffff816b0b72
RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffff90496228
RBP: ffffc900001e0a70 R08: dffffc0000000000 R09: fffffbfff2092c46
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff888013a6e468 R14: abe92a66ad81a2ce R15: 1ffff1100274dc8d
FS:  0000000000000000(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f101494affc CR3: 00000000619bb000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __lock_acquire+0x125b/0x1f80 kernel/locking/lockdep.c:5049
 lock_acquire+0x1f8/0x5a0 kernel/locking/lockdep.c:5662
 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
 _raw_spin_lock_irq+0xcf/0x110 kernel/locking/spinlock.c:170
 __run_hrtimer kernel/time/hrtimer.c:1690 [inline]
 __hrtimer_run_queues+0x6d3/0xe50 kernel/time/hrtimer.c:1750
 hrtimer_interrupt+0x392/0x980 kernel/time/hrtimer.c:1812
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x156/0x580 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:653
RIP: 0010:console_emit_next_record+0xc69/0xea0 kernel/printk/printk.c:2751
Code: 00 e8 bb 31 1c 00 44 0f b6 74 24 1f 4d 85 e4 75 07 e8 ab 31 1c 00 eb 06 e8 a4 31 1c 00 fb 48 c7 84 24 a0 00 00 00 0e 36 e0 45 <42> c7 04 2b 00 00 00 00 4a c7 44 2b 0a 00 00 00 00 4a c7 44 2b 12
RSP: 0018:ffffc90000a3f760 EFLAGS: 00000293
RAX: ffffffff816e51ec RBX: dffffc0000000000 RCX: ffff888013a6d940
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000a3fa10 R08: ffffffff816e51c7 R09: fffffbfff2092c45
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000200
R13: 1ffff92000147f00 R14: 0000000000000001 R15: 0000000000000000
 console_unlock+0x278/0x7c0 kernel/printk/printk.c:2871
 vprintk_emit+0x523/0x740 kernel/printk/printk.c:2268
 _printk+0xd1/0x111 kernel/printk/printk.c:2293
 kauditd_printk_skb kernel/audit.c:547 [inline]
 kauditd_hold_skb+0x1b8/0x200 kernel/audit.c:582
 kauditd_send_queue+0x2a3/0x2f0 kernel/audit.c:767
 kauditd_thread+0x758/0x990 kernel/audit.c:891
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:307
 </TASK>

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/03/26 19:28 linux-6.1.y d7543167affd 454571b6 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in kauditd_thread
2024/02/13 21:20 linux-6.1.y f1bb70486c9c e66542d7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in kauditd_thread
2023/12/11 16:50 linux-6.1.y e7cddbb41b63 28b24332 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: rcu detected stall in kauditd_thread
* Struck through repros no longer work on HEAD.