syzbot


INFO: rcu detected stall in sys_sendmmsg (3)

Status: closed as invalid on 2022/02/08 10:00
Reported-by: syzbot+@syzkaller.appspotmail.com
First crash: 430d, last: 330d
similar bugs (3):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in sys_sendmmsg (2) 2 1095d 1096d 0/24 closed as invalid on 2019/12/04 14:14
upstream INFO: rcu detected stall in sys_sendmmsg 2 1179d 1180d 14/24 fixed on 2019/10/09 10:54
upstream INFO: rcu detected stall in sys_sendmmsg (4) 1 129d 129d 0/24 auto-obsoleted due to no activity on 2022/10/25 00:28

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 GPs behind) idle=5df/1/0x4000000000000000 softirq=124258/124259 fqs=2 
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P2972/1:b..l
	(detected by 1, t=10502 jiffies, g=206389, q=172)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 17326 Comm: syz-executor.3 Not tainted 5.16.0-rc8-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:advance_sched+0x150/0x930 net/sched/sch_taprio.c:722
Code: 42 80 3c 20 00 74 08 48 89 df e8 6b 2a 7e f9 48 8b 03 48 89 44 24 18 31 ff 48 89 ee e8 49 bc 33 f9 48 85 ed 0f 84 9c 00 00 00 <e8> 8b b7 33 f9 e9 e5 00 00 00 e8 11 c8 b9 01 89 c3 31 ff 89 c6 e8
RSP: 0018:ffffc90000007c80 EFLAGS: 00000082
RAX: 0000000000010002 RBX: ffff888147d2a338 RCX: ffff8880151e3a00
RDX: ffff8880151e3a00 RSI: ffff88807296f080 RDI: 0000000000000000
RBP: ffff88807296f080 R08: ffffffff8850c677 R09: 0000000000000003
R10: fffff52000000f81 R11: 0000000000000004 R12: dffffc0000000000
R13: ffff888147d2a340 R14: ffff888147d2a2e8 R15: ffff888147d2a000
FS:  00007f3592988700(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055aa00697b08 CR3: 0000000040cab000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x50b/0xa60 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x3b3/0x1040 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0xf9/0x270 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:22 [inline]
RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline]
RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:132 [inline]
RIP: 0010:preempt_schedule_irq+0xfd/0x1c0 kernel/sched/core.c:6669
Code: 08 00 00 00 0f 84 98 00 00 00 bf 01 00 00 00 e8 39 e2 4d f7 e8 74 d7 7e f7 fb bf 01 00 00 00 e8 f9 e3 ff ff 43 80 7c 3d 00 00 <74> 08 4c 89 f7 e8 29 ae c2 f7 48 c7 44 24 20 00 00 00 00 9c 8f 44
RSP: 0018:ffffc9000ae1f6e0 EFLAGS: 00000246
RAX: dcba546af2f80100 RBX: 1ffff920015c3ee4 RCX: 1ffff920015c3ec4
RDX: dffffc0000000000 RSI: 0000000000000001 RDI: ffff8880151e4f80
RBP: ffffc9000ae1f798 R08: dffffc0000000000 R09: fffffbfff1ff3dfc
R10: fffffbfff1ff3dfc R11: 0000000000000000 R12: 1ffff920015c3edc
R13: 1ffff920015c3ee0 R14: ffffc9000ae1f700 R15: dffffc0000000000
 irqentry_exit+0x56/0x90 kernel/entry/common.c:425
 asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x4/0x60 kernel/kcov.c:199
Code: 00 00 0f 1f 40 00 be 0d 00 00 00 48 c7 c7 e0 af b5 8c e8 8f 66 4a 00 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc 48 8b 04 24 <65> 48 8b 0c 25 c0 6f 02 00 65 8b 15 a4 f1 7d 7e f7 c2 00 01 ff 00
RSP: 0018:ffffc9000ae1f860 EFLAGS: 00000246
RAX: ffffffff83defdbb RBX: ffff888077ee2000 RCX: ffff88814afc7808
RDX: ffffc9000be12000 RSI: 00000000000208e3 RDI: ffff88801a2bdb68
RBP: ffff88801a2bdb18 R08: ffffffff83dddd1f R09: fffffbfff1ff3dfc
R10: fffffbfff1ff3dfc R11: 0000000000000000 R12: ffff88802af30000
R13: dffffc0000000000 R14: ffff88801a2bdb68 R15: ffff88801a2bd800
 hmac_init+0x1b/0x1c0 crypto/hmac.c:101
 crypto_ahash_init include/crypto/hash.h:535 [inline]
 hash_sendmsg+0x8dd/0xb50 crypto/algif_hash.c:81
 sock_sendmsg_nosec net/socket.c:704 [inline]
 sock_sendmsg net/socket.c:724 [inline]
 ____sys_sendmsg+0x5b9/0x910 net/socket.c:2409
 ___sys_sendmsg net/socket.c:2463 [inline]
 __sys_sendmmsg+0x3ba/0x750 net/socket.c:2549
 __do_sys_sendmmsg net/socket.c:2578 [inline]
 __se_sys_sendmmsg net/socket.c:2575 [inline]
 __x64_sys_sendmmsg+0x9c/0xb0 net/socket.c:2575
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x44/0xd0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f3594012e99
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f3592988168 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f3594125f60 RCX: 00007f3594012e99
RDX: 0000000000000500 RSI: 0000000020002280 RDI: 0000000000000005
RBP: 00007f359406cff1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffc4acf294f R14: 00007f3592988300 R15: 0000000000022000
 </TASK>
task:udevd           state:R  running task     stack:22192 pid: 2972 ppid:     1 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4972 [inline]
 __schedule+0xb72/0x1460 kernel/sched/core.c:6253
 preempt_schedule_common kernel/sched/core.c:6419 [inline]
 preempt_schedule+0x14d/0x190 kernel/sched/core.c:6444
 preempt_schedule_thunk+0x16/0x18
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
 _raw_spin_unlock_irqrestore+0x128/0x130 kernel/locking/spinlock.c:194
 spin_unlock_irqrestore include/linux/spinlock.h:404 [inline]
 __wake_up_common_lock kernel/sched/wait.c:140 [inline]
 __wake_up_sync_key+0x124/0x1c0 kernel/sched/wait.c:205
 sock_def_readable+0x106/0x200 net/core/sock.c:3136
 __netlink_sendskb net/netlink/af_netlink.c:1266 [inline]
 netlink_sendskb+0x8d/0x130 net/netlink/af_netlink.c:1272
 netlink_unicast+0x633/0x9f0 net/netlink/af_netlink.c:1360
 netlink_sendmsg+0xaea/0xe60 net/netlink/af_netlink.c:1921
 sock_sendmsg_nosec net/socket.c:704 [inline]
 sock_sendmsg net/socket.c:724 [inline]
 ____sys_sendmsg+0x5b9/0x910 net/socket.c:2409
 ___sys_sendmsg net/socket.c:2463 [inline]
 __sys_sendmsg+0x280/0x370 net/socket.c:2492
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x44/0xd0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f53e369f163
RSP: 002b:00007ffc174b6528 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 000055aa0068dd60 RCX: 00007f53e369f163
RDX: 0000000000000000 RSI: 00007ffc174b6538 RDI: 0000000000000004
RBP: 000055aa006aea60 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000020 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000000000bb R14: 0000000000000000 R15: 0000000000000000
 </TASK>
rcu: rcu_preempt kthread starved for 10498 jiffies! g206389 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:26288 pid:   14 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:4972 [inline]
 __schedule+0xb72/0x1460 kernel/sched/core.c:6253
 schedule+0x12b/0x1f0 kernel/sched/core.c:6326
 schedule_timeout+0x1b9/0x300 kernel/time/timer.c:1881
 rcu_gp_fqs_loop+0x1fd/0x770 kernel/rcu/tree.c:1955
 rcu_gp_kthread+0xa5/0x350 kernel/rcu/tree.c:2128
 kthread+0x468/0x490 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 1
CPU: 1 PID: 15423 Comm: kworker/u4:0 Not tainted 5.16.0-rc8-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1dc/0x2d8 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x45f/0x490 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x16a/0x280 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_check_gp_kthread_starvation+0x1ff/0x270 kernel/rcu/tree_stall.h:481
 print_other_cpu_stall kernel/rcu/tree_stall.h:586 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:729 [inline]
 rcu_pending kernel/rcu/tree.c:3878 [inline]
 rcu_sched_clock_irq+0x23d5/0x2bc0 kernel/rcu/tree.c:2597
 update_process_times+0x197/0x200 kernel/time/timer.c:1785
 tick_sched_handle kernel/time/tick-sched.c:226 [inline]
 tick_sched_timer+0x27d/0x420 kernel/time/tick-sched.c:1428
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x4cb/0xa60 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x3b3/0x1040 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0xf9/0x270 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8c/0xb0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:csd_lock_wait kernel/smp.c:440 [inline]
RIP: 0010:smp_call_function_many_cond+0xbb7/0xda0 kernel/smp.c:969
Code: e8 ee b1 0b 00 41 83 e7 01 49 bf 00 00 00 00 00 fc ff df 75 0e e8 c9 ad 0b 00 eb 4c 0f 1f 80 00 00 00 00 f3 90 42 0f b6 04 3b <84> c0 75 11 41 f7 45 00 01 00 00 00 74 2b e8 a6 ad 0b 00 eb e4 44
RSP: 0018:ffffc90004eaf820 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 1ffff1101734831d RCX: ffff888018688000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90004eaf960 R08: ffffffff8178d032 R09: ffffffff8442f810
R10: 0000000000000009 R11: ffff888018688000 R12: 0000000000000000
R13: ffff8880b9a418e8 R14: ffff8880b9b3a640 R15: dffffc0000000000
 on_each_cpu_cond_mask+0x3b/0x80 kernel/smp.c:1135
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1112 [inline]
 text_poke_bp_batch+0x310/0x940 arch/x86/kernel/alternative.c:1297
 text_poke_flush arch/x86/kernel/alternative.c:1451 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1458
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x12d/0x250 kernel/jump_label.c:177
 static_key_enable+0x16/0x20 kernel/jump_label.c:190
 toggle_allocation_gate+0xbf/0x460 mm/kfence/core.c:733
 process_one_work+0x853/0x1140 kernel/workqueue.c:2298
 worker_thread+0xac1/0x1320 kernel/workqueue.c:2445
 kthread+0x468/0x490 kernel/kthread.c:327
 ret_from_fork+0x1f/0x30
 </TASK>
----------------
Code disassembly (best guess):
   0:	42 80 3c 20 00       	cmpb   $0x0,(%rax,%r12,1)
   5:	74 08                	je     0xf
   7:	48 89 df             	mov    %rbx,%rdi
   a:	e8 6b 2a 7e f9       	callq  0xf97e2a7a
   f:	48 8b 03             	mov    (%rbx),%rax
  12:	48 89 44 24 18       	mov    %rax,0x18(%rsp)
  17:	31 ff                	xor    %edi,%edi
  19:	48 89 ee             	mov    %rbp,%rsi
  1c:	e8 49 bc 33 f9       	callq  0xf933bc6a
  21:	48 85 ed             	test   %rbp,%rbp
  24:	0f 84 9c 00 00 00    	je     0xc6
* 2a:	e8 8b b7 33 f9       	callq  0xf933b7ba <-- trapping instruction
  2f:	e9 e5 00 00 00       	jmpq   0x119
  34:	e8 11 c8 b9 01       	callq  0x1b9c84a
  39:	89 c3                	mov    %eax,%ebx
  3b:	31 ff                	xor    %edi,%edi
  3d:	89 c6                	mov    %eax,%esi
  3f:	e8                   	.byte 0xe8

Crashes (3):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-kasan-gce-smack-root 2022/01/06 13:20 upstream 75acfdb6fd92 6acc789a .config log report info INFO: rcu detected stall in sys_sendmmsg
ci-upstream-kasan-gce-smack-root 2021/12/09 14:13 upstream 2a987e65025e b54aa474 .config log report info INFO: rcu detected stall in sys_sendmmsg
ci-upstream-kasan-gce-selinux-root 2021/09/28 16:28 upstream d33bec7b3dfa d82cb927 .config log report info INFO: rcu detected stall in sys_sendmmsg
* Struck through repros no longer work on HEAD.