syzbot


INFO: rcu detected stall in sys_timerfd_settime

Status: closed as invalid on 2022/02/08 09:50
Subsystems: fs
[Documentation on labels]
First crash: 934d, last: 867d
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in sys_timerfd_settime (2) net 1 436d 436d 0/26 auto-obsoleted due to no activity on 2023/04/25 06:19
linux-5.15 INFO: rcu detected stall in sys_timerfd_settime 1 288d 288d 0/3 auto-obsoleted due to no activity on 2023/09/22 02:06
linux-6.1 INFO: rcu detected stall in sys_timerfd_settime 1 280d 280d 0/3 auto-obsoleted due to no activity on 2023/09/29 23:03

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (10499 ticks this GP) idle=46d/1/0x4000000000000000 softirq=213400/213400 fqs=0 
	(t=10500 jiffies g=406221 q=43)
rcu: rcu_preempt kthread timer wakeup didn't happen for 10499 jiffies! g406221 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: 	Possible timer handling issue on cpu=0 timer-softirq=165853
rcu: rcu_preempt kthread starved for 10500 jiffies! g406221 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:28904 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4969 [inline]
 __schedule+0xa9a/0x4940 kernel/sched/core.c:6250
 schedule+0xd2/0x260 kernel/sched/core.c:6323
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x186/0x810 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0x1de/0x320 kernel/rcu/tree.c:2128
 kthread+0x405/0x4f0 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 2978 Comm: systemd-udevd Not tainted 5.15.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:arch_atomic_read arch/x86/include/asm/atomic.h:29 [inline]
RIP: 0010:rcu_dynticks_curr_cpu_in_eqs kernel/rcu/tree.c:330 [inline]
RIP: 0010:rcu_is_watching+0x69/0xb0 kernel/rcu/tree.c:1121
Code: a0 28 35 8b 48 b8 00 00 00 00 00 fc ff df 48 89 da 48 c1 ea 03 0f b6 14 02 48 89 d8 83 e0 07 83 c0 03 38 d0 7c 04 84 d2 75 19 <8b> 03 83 e0 01 65 ff 0d 8b 0f a0 7e 74 03 5b 5d c3 0f 1f 44 00 00
RSP: 0018:ffffc90000007e18 EFLAGS: 00000046
RAX: 0000000000000003 RBX: ffff8880b9c3aad0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000010003 RDI: ffffffff8b3528a0
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff8166660f R11: 0000000000000000 R12: 0000000000000001
R13: ffff8880b9c2a500 R14: ffff8880b9c2a400 R15: ffffffff8755c590
FS:  00007f75005638c0(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe88850e018 CR3: 000000002411f000 CR4: 00000000003526f0
DR0: 0010000010000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 rcu_read_lock_held_common kernel/rcu/update.c:108 [inline]
 rcu_read_lock_sched_held+0x1c/0x70 kernel/rcu/update.c:123
 trace_hrtimer_expire_entry include/trace/events/timer.h:232 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1682 [inline]
 __hrtimer_run_queues+0x9b0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:202
Code: 0f 1f 44 00 00 55 48 8b 74 24 08 48 89 fd 48 83 c7 18 e8 de 39 1e f8 48 89 ef e8 26 b0 1e f8 e8 d1 6e 3f f8 fb bf 01 00 00 00 <e8> a6 7a 11 f8 65 8b 05 8f 9a c4 76 85 c0 74 02 5d c3 0f 1f 44 00
RSP: 0018:ffffc90001b07d90 EFLAGS: 00000202
RAX: 00000000031dfe0b RBX: ffff88801d10cc00 RCX: 1ffffffff1ae1081
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffff88801d10cc88 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff817d4468 R11: 0000000000000000 R12: ffff88801d10cc88
R13: 0000000000000000 R14: 000002619777b118 R15: 0000000000000001
 spin_unlock_irq include/linux/spinlock.h:399 [inline]
 do_timerfd_settime+0xa15/0x11f0 fs/timerfd.c:521
 __do_sys_timerfd_settime fs/timerfd.c:567 [inline]
 __se_sys_timerfd_settime fs/timerfd.c:558 [inline]
 __x64_sys_timerfd_settime+0x136/0x220 fs/timerfd.c:558
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f74ff3e37ba
Code: 48 8b 0d e1 f6 2a 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 1e 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ae f6 2a 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc8f437fd8 EFLAGS: 00000246 ORIG_RAX: 000000000000011e
RAX: ffffffffffffffda RBX: 000055a8a1d93b48 RCX: 00007f74ff3e37ba
RDX: 00007ffc8f437ff0 RSI: 0000000000000001 RDI: 000000000000000d
RBP: 000000009c0e331f R08: 00000000000000e2 R09: 0000000000000018
R10: 0000000000000000 R11: 0000000000000246 R12: 000000009bff12f8
R13: 000055a8a1d93b20 R14: 000055a8a1d93b00 R15: 0000000000000000
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 2978 Comm: systemd-udevd Not tainted 5.15.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:__lock_acquire+0xe57/0x54a0 kernel/locking/lockdep.c:5054
Code: 00 00 00 fc ff df 48 8b 5c 24 30 48 c7 04 03 00 00 00 00 48 8b 84 24 e0 00 00 00 65 48 2b 04 25 28 00 00 00 0f 85 1e 2f 00 00 <48> 81 c4 e8 00 00 00 44 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8
RSP: 0018:ffffc90000007b90 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 1ffff92000000f83 RCX: ffffffff815bc4ff
RDX: 1ffff1100fe8a983 RSI: 0000000000000001 RDI: ffffffff8f2e78b8
RBP: 000000000000e274 R08: 0000000000000000 R09: ffffffff8fd60bff
R10: fffffbfff1fac17f R11: 0000000000000000 R12: ffff88807f454c20
R13: ffff88807f4541c0 R14: 0000000000000001 R15: 9d5a1b447a0ae661
FS:  00007f75005638c0(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe88850e018 CR3: 000000002411f000 CR4: 00000000003526f0
DR0: 0010000010000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 lock_acquire kernel/locking/lockdep.c:5637 [inline]
 lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5602
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:349 [inline]
 advance_sched+0x53/0x9a0 net/sched/sch_taprio.c:714
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x609/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:202
Code: 0f 1f 44 00 00 55 48 8b 74 24 08 48 89 fd 48 83 c7 18 e8 de 39 1e f8 48 89 ef e8 26 b0 1e f8 e8 d1 6e 3f f8 fb bf 01 00 00 00 <e8> a6 7a 11 f8 65 8b 05 8f 9a c4 76 85 c0 74 02 5d c3 0f 1f 44 00
RSP: 0018:ffffc90001b07d90 EFLAGS: 00000202
RAX: 00000000031dfe0b RBX: ffff88801d10cc00 RCX: 1ffffffff1ae1081
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffff88801d10cc88 R08: 0000000000000001 R09: 0000000000000001
R10: ffffffff817d4468 R11: 0000000000000000 R12: ffff88801d10cc88
R13: 0000000000000000 R14: 000002619777b118 R15: 0000000000000001
 spin_unlock_irq include/linux/spinlock.h:399 [inline]
 do_timerfd_settime+0xa15/0x11f0 fs/timerfd.c:521
 __do_sys_timerfd_settime fs/timerfd.c:567 [inline]
 __se_sys_timerfd_settime fs/timerfd.c:558 [inline]
 __x64_sys_timerfd_settime+0x136/0x220 fs/timerfd.c:558
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f74ff3e37ba
Code: 48 8b 0d e1 f6 2a 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 1e 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ae f6 2a 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc8f437fd8 EFLAGS: 00000246 ORIG_RAX: 000000000000011e
RAX: ffffffffffffffda RBX: 000055a8a1d93b48 RCX: 00007f74ff3e37ba
RDX: 00007ffc8f437ff0 RSI: 0000000000000001 RDI: 000000000000000d
RBP: 000000009c0e331f R08: 00000000000000e2 R09: 0000000000000018
R10: 0000000000000000 R11: 0000000000000246 R12: 000000009bff12f8
R13: 000055a8a1d93b20 R14: 000055a8a1d93b00 R15: 0000000000000000
 </TASK>
NMI backtrace for cpu 1
CPU: 1 PID: 14792 Comm: syz-executor.0 Not tainted 5.15.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x1ae/0x220 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:343
 print_cpu_stall kernel/rcu/tree_stall.h:627 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:711 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq.cold+0x9d/0x746 kernel/rcu/tree.c:2597
 update_process_times+0x16d/0x200 kernel/time/timer.c:1785
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:226
 tick_sched_timer+0x1b0/0x2d0 kernel/time/tick-sched.c:1421
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:check_kcov_mode kernel/kcov.c:166 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x7/0x60 kernel/kcov.c:200
Code: 45 00 5d be 03 00 00 00 e9 06 98 63 02 66 0f 1f 44 00 00 48 8b be b0 01 00 00 e8 b4 ff ff ff 31 c0 c3 90 65 8b 05 09 70 8b 7e <89> c1 48 8b 34 24 81 e1 00 01 00 00 65 48 8b 14 25 40 70 02 00 a9
RSP: 0018:ffffc900035d7c00 EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffe8ffffc0b640 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff888074ff0140 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff816b86a6 R11: 0000000000000000 R12: fffff91ffff816c9
R13: 0000000000000000 R14: ffffe8ffffc0b648 R15: 0000000000000001
 rep_nop arch/x86/include/asm/vdso/processor.h:13 [inline]
 cpu_relax arch/x86/include/asm/vdso/processor.h:18 [inline]
 csd_lock_wait kernel/smp.c:440 [inline]
 smp_call_function_many_cond+0x450/0xc20 kernel/smp.c:969
 clock_was_set+0x599/0x790 kernel/time/hrtimer.c:974
 do_settimeofday64 kernel/time/timekeeping.c:1327 [inline]
 do_settimeofday64+0x3e2/0x5c0 kernel/time/timekeeping.c:1293
 do_sys_settimeofday64 kernel/time/time.c:195 [inline]
 do_sys_settimeofday64+0x1de/0x260 kernel/time/time.c:169
 __do_sys_clock_settime kernel/time/posix-timers.c:1079 [inline]
 __se_sys_clock_settime kernel/time/posix-timers.c:1067 [inline]
 __x64_sys_clock_settime+0x1a1/0x280 kernel/time/posix-timers.c:1067
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f1aef123ae9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f1aec699188 EFLAGS: 00000246 ORIG_RAX: 00000000000000e3
RAX: ffffffffffffffda RBX: 00007f1aef236f60 RCX: 00007f1aef123ae9
RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000000
RBP: 00007f1aef17df6d R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffc9699addf R14: 00007f1aec699300 R15: 0000000000022000
 </TASK>
----------------
Code disassembly (best guess), 4 bytes skipped:
   0:	48 b8 00 00 00 00 00 	movabs $0xdffffc0000000000,%rax
   7:	fc ff df
   a:	48 89 da             	mov    %rbx,%rdx
   d:	48 c1 ea 03          	shr    $0x3,%rdx
  11:	0f b6 14 02          	movzbl (%rdx,%rax,1),%edx
  15:	48 89 d8             	mov    %rbx,%rax
  18:	83 e0 07             	and    $0x7,%eax
  1b:	83 c0 03             	add    $0x3,%eax
  1e:	38 d0                	cmp    %dl,%al
  20:	7c 04                	jl     0x26
  22:	84 d2                	test   %dl,%dl
  24:	75 19                	jne    0x3f
* 26:	8b 03                	mov    (%rbx),%eax <-- trapping instruction
  28:	83 e0 01             	and    $0x1,%eax
  2b:	65 ff 0d 8b 0f a0 7e 	decl   %gs:0x7ea00f8b(%rip)        # 0x7ea00fbd
  32:	74 03                	je     0x37
  34:	5b                   	pop    %rbx
  35:	5d                   	pop    %rbp
  36:	c3                   	retq
  37:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/11/12 13:35 upstream 5833291ab6de 75b04091 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in sys_timerfd_settime
2021/11/04 05:15 upstream ce840177930f 4c1be0be .config console log report info ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in sys_timerfd_settime
2021/09/05 23:22 upstream 0319b848b155 d236a457 .config console log report info ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in sys_timerfd_settime
* Struck through repros no longer work on HEAD.