syzbot


INFO: rcu detected stall in gc_worker

Status: upstream: reported syz repro on 2022/04/22 15:43
Reported-by: syzbot+b970eb44855cb1e48606@syzkaller.appspotmail.com
First crash: 728d, last: 728d
Fix bisection: failed (error log, bisect log)
  
Similar bugs (5)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in gc_worker (2) netfilter C unreliable 4 824d 862d 0/26 closed as invalid on 2022/02/08 10:33
upstream INFO: rcu detected stall in gc_worker (3) netfilter C done done 47 81d 761d 0/26 upstream: reported C repro on 2022/03/20 12:02
upstream INFO: rcu detected stall in gc_worker netfilter 8 1829d 1914d 0/26 auto-closed as invalid on 2019/10/14 16:34
linux-6.1 INFO: rcu detected stall in gc_worker origin:upstream C 7 26d 291d 0/3 upstream: reported C repro on 2023/07/03 02:12
linux-5.15 INFO: rcu detected stall in gc_worker origin:upstream C error 6 68d 257d 0/3 upstream: reported C repro on 2023/08/06 21:00

Sample crash report:
IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
wlan0: Created IBSS using preconfigured BSSID 50:50:50:50:50:50
wlan0: Creating new IBSS network, BSSID 50:50:50:50:50:50
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	0-...!: (1 ticks this GP) idle=10a/1/0x4000000000000002 softirq=55819/55819 fqs=0 
rcu: 	 (t=104705 jiffies g=68533 q=246)
rcu: rcu_preempt kthread starved for 104705 jiffies! g68533 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: RCU grace-period kthread stack dump:
rcu_preempt     I29208    10      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2828 [inline]
 __schedule+0x887/0x2040 kernel/sched/core.c:3517
 schedule+0x8d/0x1b0 kernel/sched/core.c:3561
 schedule_timeout+0x4cf/0xfe0 kernel/time/timer.c:1818
 rcu_gp_kthread+0xdad/0x21c0 kernel/rcu/tree.c:2202
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
NMI backtrace for cpu 0
CPU: 0 PID: 9365 Comm: kworker/0:4 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_power_efficient gc_worker
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x63/0xa2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x1a6/0x1f0 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x15f/0x19c kernel/rcu/tree.c:1340
 print_cpu_stall kernel/rcu/tree.c:1478 [inline]
 check_cpu_stall kernel/rcu/tree.c:1550 [inline]
 __rcu_pending kernel/rcu/tree.c:3293 [inline]
 rcu_pending kernel/rcu/tree.c:3336 [inline]
 rcu_check_callbacks.cold+0x62d/0xe19 kernel/rcu/tree.c:2682
 update_process_times+0x2a/0x70 kernel/time/timer.c:1650
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:168
 tick_sched_timer+0xfc/0x290 kernel/time/tick-sched.c:1278
 __run_hrtimer kernel/time/hrtimer.c:1465 [inline]
 __hrtimer_run_queues+0x3f6/0xe60 kernel/time/hrtimer.c:1527
 hrtimer_interrupt+0x326/0x9e0 kernel/time/hrtimer.c:1585
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1071 [inline]
 smp_apic_timer_interrupt+0x10c/0x550 arch/x86/kernel/apic/apic.c:1096
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:894
 </IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:789 [inline]
RIP: 0010:seqcount_lockdep_reader_access include/linux/seqlock.h:83 [inline]
RIP: 0010:read_seqcount_begin include/linux/seqlock.h:164 [inline]
RIP: 0010:nf_conntrack_get_ht include/net/netfilter/nf_conntrack.h:302 [inline]
RIP: 0010:gc_worker+0x85a/0xe90 net/netfilter/nf_conntrack_core.c:1214
Code: 82 f1 89 48 c1 e8 03 42 80 3c 30 00 0f 85 a7 05 00 00 48 83 3d fe b3 4c 03 00 0f 84 7d 03 00 00 e8 db b6 bd fa 48 89 ef 57 9d <0f> 1f 44 00 00 e9 62 fa ff ff e8 c7 b6 bd fa 48 8b 44 24 10 48 8d
RSP: 0018:ffff88808f4bfc58 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: ffff8880a4a0a000 RBX: fffffbfff13e3050 RCX: 1ffff1101494151a
RDX: 0000000000000000 RSI: ffffffff86a4ce95 RDI: 0000000000000293
RBP: 0000000000000293 R08: ffffffff8cd50238 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000200
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000010000
 process_one_work+0x864/0x1570 kernel/workqueue.c:2153
 worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 13277 Comm: systemd-udevd Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:logarithmic_accumulation kernel/time/timekeeping.c:2034 [inline]
RIP: 0010:timekeeping_advance+0x317/0x9b0 kernel/time/timekeeping.c:2097
Code: 47 a9 cf 0b 49 d3 e7 4c 89 fe 48 89 df 48 89 1d 37 a9 cf 0b e8 3a d6 0d 00 4c 39 fb 72 50 e8 20 d5 0d 00 48 8b 05 89 a9 cf 0b <4c> 8d 70 01 e8 10 d5 0d 00 4c 29 fb 4c 89 ff 48 89 de e8 12 d6 0d
RSP: 0018:ffff8880ba107cc0 EFLAGS: 00000006
RAX: 000000000028c165 RBX: 003ab5e840000000 RCX: ffffffff8154b046
RDX: 0000000000010000 RSI: ffffffff8154b050 RDI: 0000000000000006
RBP: ffff8880ba107d80 R08: 00989b29342e177c R09: 003ab5e840000000
R10: 0000000000000006 R11: ffffffff8c66505b R12: 7450dfba3d419a78
R13: 00004c4d94a47a80 R14: 00000000628b82ca R15: 001dcd6500000000
FS:  00007f1a4aba08c0(0000) GS:ffff8880ba100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005591896108c8 CR3: 00000000b2d50000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 tick_do_update_jiffies64.part.0+0x188/0x290 kernel/time/tick-sched.c:101
 tick_do_update_jiffies64 kernel/time/tick-sched.c:67 [inline]
 tick_sched_do_timer kernel/time/tick-sched.c:139 [inline]
 tick_sched_timer+0x220/0x290 kernel/time/tick-sched.c:1271
 __run_hrtimer kernel/time/hrtimer.c:1465 [inline]
 __hrtimer_run_queues+0x3f6/0xe60 kernel/time/hrtimer.c:1527
 hrtimer_interrupt+0x326/0x9e0 kernel/time/hrtimer.c:1585
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1071 [inline]
 smp_apic_timer_interrupt+0x10c/0x550 arch/x86/kernel/apic/apic.c:1096
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:894
 </IRQ>
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x50/0x80 kernel/locking/spinlock.c:192
Code: c0 98 82 f1 89 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 31 48 83 3d 01 31 d8 01 00 74 25 fb 66 0f 1f 44 00 00 <bf> 01 00 00 00 e8 26 1b 28 f9 65 8b 05 9f 8d e8 77 85 c0 74 02 5d
RSP: 0018:ffff8880a15f7e88 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff13e3053 RBX: 0000000000000000 RCX: 0000000000000000
RDX: dffffc0000000000 RSI: 0000000000000001 RDI: ffff88809ea20c04
RBP: ffff88809ea20b48 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88809ea20ad8
R13: dffffc0000000000 R14: ffff88809ea20380 R15: 0000000000000000
 task_work_run+0x11c/0x1c0 kernel/task_work.c:106
 tracehook_notify_resume include/linux/tracehook.h:193 [inline]
 exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
 prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
 syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
 do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f1a499a8a4b
Code: 48 8b 34 24 eb c5 66 0f 1f 84 00 00 00 00 00 48 c7 c0 ff ff ff ff eb c8 0f 1f 80 00 00 00 00 48 63 7f 70 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 f3 c3 0f 1f 00 48 8b 15 19 74 32 00 f7 d8
RSP: 002b:00007fff987d6598 EFLAGS: 00000206 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 000055918961fab0 RCX: 00007f1a499a8a4b
RDX: 00007f1a49ccc900 RSI: 0000000000000001 RDI: 000000000000000f
RBP: 00007f1a49ccd440 R08: 00007f1a4aba08c0 R09: 000055918961fab0
R10: 00007f1a4aba08c0 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000001000 R15: 00007fff987d6770
systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)!
systemd[1]: systemd-udevd.service: Killing process 4699 (systemd-udevd) with signal SIGABRT.
systemd[1]: systemd-udevd.service: Main process exited, code=killed, status=6/ABRT
systemd[1]: systemd-udevd.service: Killing process 13268 (systemd-udevd) with signal SIGKILL.
systemd[1]: systemd-udevd.service: Killing process 13277 (systemd-udevd) with signal SIGKILL.
systemd[1]: systemd-timesyncd.service: State 'stop-sigabrt' timed out. Terminating.
systemd[1]: systemd-journald.service: State 'stop-sigabrt' timed out. Terminating.
systemd[1]: systemd-udevd.service: Processes still around after final SIGKILL. Entering failed mode.
systemd[1]: systemd-udevd.service: Unit entered failed state.
systemd[1]: systemd-udevd.service: Failed with result 'watchdog'.
syz-executor.4 calls setitimer() with new_value NULL pointer. Misfeature support will be removed
----------------
Code disassembly (best guess), 1 bytes skipped:
   0:	f1                   	icebp
   1:	89 48 c1             	mov    %ecx,-0x3f(%rax)
   4:	e8 03 42 80 3c       	callq  0x3c80420c
   9:	30 00                	xor    %al,(%rax)
   b:	0f 85 a7 05 00 00    	jne    0x5b8
  11:	48 83 3d fe b3 4c 03 	cmpq   $0x0,0x34cb3fe(%rip)        # 0x34cb417
  18:	00
  19:	0f 84 7d 03 00 00    	je     0x39c
  1f:	e8 db b6 bd fa       	callq  0xfabdb6ff
  24:	48 89 ef             	mov    %rbp,%rdi
  27:	57                   	push   %rdi
  28:	9d                   	popfq
* 29:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1) <-- trapping instruction
  2e:	e9 62 fa ff ff       	jmpq   0xfffffa95
  33:	e8 c7 b6 bd fa       	callq  0xfabdb6ff
  38:	48 8b 44 24 10       	mov    0x10(%rsp),%rax
  3d:	48                   	rex.W
  3e:	8d                   	.byte 0x8d

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/04/22 15:42 linux-4.19.y 3f8a27f9e27b 2738b391 .config console log report syz ci2-linux-4-19 INFO: rcu detected stall in gc_worker
* Struck through repros no longer work on HEAD.