syzbot


INFO: rcu detected stall in task_numa_work

Status: auto-obsoleted due to no activity on 2023/11/28 14:56
Subsystems: mm
[Documentation on labels]
First crash: 250d, last: 249d
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in task_numa_work 1 885d 885d 0/1 auto-closed as invalid on 2022/04/01 16:00

Sample crash report:
bridge0: received packet on veth0_to_bridge with own address as source address (addr:b6:eb:ab:21:85:8c, vlan:0)
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P7551/1:b..l P4468/1:b..l
rcu: 	(detected by 1, t=10503 jiffies, g=262185, q=287 ncpus=2)
task:syslogd         state:R  running task     stack:25536 pid:4468  ppid:1      flags:0x00004002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5382 [inline]
 __schedule+0xee1/0x59f0 kernel/sched/core.c:6695
 preempt_schedule_irq+0x52/0x90 kernel/sched/core.c:7007
 irqentry_exit+0x35/0x80 kernel/entry/common.c:432
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:lock_acquire+0x1ef/0x510 kernel/locking/lockdep.c:5729
Code: c1 05 f5 1e 9b 7e 83 f8 01 0f 85 b0 02 00 00 9c 58 f6 c4 02 0f 85 9b 02 00 00 48 85 ed 74 01 fb 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 8b 84 24
RSP: 0018:ffffc9000316f9d8 EFLAGS: 00000206
RAX: dffffc0000000000 RBX: 1ffff9200062df3d RCX: 0000000000000001
RDX: 1ffff1100fba4158 RSI: ffffffff8a6c8bc0 RDI: ffffffff8ac862e0
RBP: 0000000000000200 R08: 0000000000000000 R09: fffffbfff22f2fd0
R10: ffffffff91797e87 R11: 0000000000094000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffffff8c9a7d20 R15: 0000000000000000
 rcu_lock_acquire include/linux/rcupdate.h:303 [inline]
 rcu_read_lock include/linux/rcupdate.h:749 [inline]
 page_ext_get+0x3c/0x310 mm/page_ext.c:167
 __reset_page_owner+0x2f/0x190 mm/page_owner.c:145
 reset_page_owner include/linux/page_owner.h:24 [inline]
 free_pages_prepare mm/page_alloc.c:1161 [inline]
 free_unref_page_prepare+0x508/0xb90 mm/page_alloc.c:2348
 free_unref_page+0x33/0x3b0 mm/page_alloc.c:2443
 qlink_free mm/kasan/quarantine.c:166 [inline]
 qlist_free_all+0x6a/0x170 mm/kasan/quarantine.c:185
 kasan_quarantine_reduce+0x18b/0x1d0 mm/kasan/quarantine.c:292
 ____kasan_kmalloc mm/kasan/common.c:340 [inline]
 __kasan_kmalloc+0x86/0xb0 mm/kasan/common.c:383
 kmalloc include/linux/slab.h:582 [inline]
 kzalloc include/linux/slab.h:703 [inline]
 task_numa_work+0xc0c/0x1290 kernel/sched/fair.c:3252
 task_work_run+0x14d/0x240 kernel/task_work.c:179
 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
 exit_to_user_mode_prepare+0x210/0x240 kernel/entry/common.c:204
 __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
 syscall_exit_to_user_mode+0x1d/0x60 kernel/entry/common.c:296
 do_syscall_64+0x44/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f2c846a3b6a
RSP: 002b:00007fff14dd29e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: 00000000000000a0 RBX: 0000000000000002 RCX: 00007f2c846a3b6a
RDX: 00000000000000ff RSI: 0000564513928950 RDI: 0000000000000000
RBP: 0000564513928910 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001000 R11: 0000000000000246 R12: 00005645139289b0
R13: 0000564513928950 R14: 0000000000000000 R15: 00007f2c84880a80
 </TASK>
task:udevd           state:R  running task     stack:25376 pid:7551  ppid:4486   flags:0x00004002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5382 [inline]
 __schedule+0xee1/0x59f0 kernel/sched/core.c:6695
 preempt_schedule_common+0x45/0xc0 kernel/sched/core.c:6864
 preempt_schedule_thunk+0x1a/0x30 arch/x86/entry/thunk_64.S:45
 unwind_next_frame+0x16b4/0x2020 arch/x86/kernel/unwind_orc.c:672
 arch_stack_walk+0x8b/0xf0 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x96/0xd0 kernel/stacktrace.c:122
 save_stack+0x160/0x1f0 mm/page_owner.c:128
 __reset_page_owner+0x5a/0x190 mm/page_owner.c:149
 reset_page_owner include/linux/page_owner.h:24 [inline]
 free_pages_prepare mm/page_alloc.c:1161 [inline]
 free_unref_page_prepare+0x508/0xb90 mm/page_alloc.c:2348
 free_unref_page_list+0xe6/0xb30 mm/page_alloc.c:2489
 release_pages+0x32a/0x14e0 mm/swap.c:1042
 tlb_batch_pages_flush+0x9a/0x190 mm/mmu_gather.c:97
 tlb_flush_mmu_free mm/mmu_gather.c:292 [inline]
 tlb_flush_mmu mm/mmu_gather.c:299 [inline]
 tlb_finish_mmu+0x14b/0x7e0 mm/mmu_gather.c:391
 exit_mmap+0x2db/0x960 mm/mmap.c:3215
 __mmput+0x12a/0x4d0 kernel/fork.c:1356
 mmput+0x62/0x70 kernel/fork.c:1378
 exit_mm kernel/exit.c:567 [inline]
 do_exit+0x9b4/0x2a20 kernel/exit.c:861
 do_group_exit+0xd4/0x2a0 kernel/exit.c:1024
 __do_sys_exit_group kernel/exit.c:1035 [inline]
 __se_sys_exit_group kernel/exit.c:1033 [inline]
 __x64_sys_exit_group+0x3e/0x50 kernel/exit.c:1033
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x38/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fb175afca90
RSP: 002b:00007ffce9c92f98 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb175afca90
RDX: 00000000000000e7 RSI: 000000000000003c RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000007 R09: 1912f7679b496566
R10: 00000000ffffffff R11: 0000000000000206 R12: 00005564f3522590
R13: 00007ffce9c92fd8 R14: 0000000000000001 R15: 00005564f3500910
 </TASK>
rcu: rcu_preempt kthread starved for 1204 jiffies! g262185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28240 pid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5382 [inline]
 __schedule+0xee1/0x59f0 kernel/sched/core.c:6695
 schedule+0xe7/0x1b0 kernel/sched/core.c:6771
 schedule_timeout+0x157/0x2c0 kernel/time/timer.c:2167
 rcu_gp_fqs_loop+0x1ec/0xa50 kernel/rcu/tree.c:1613
 rcu_gp_kthread+0x249/0x380 kernel/rcu/tree.c:1812
 kthread+0x33a/0x430 kernel/kthread.c:389
 ret_from_fork+0x2c/0x70 arch/x86/kernel/process.c:145
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:304
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 1 PID: 5052 Comm: syz-fuzzer Not tainted 6.5.0-syzkaller-01207-g1c59d383390f #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
RIP: 0010:ip6t_do_table+0x261/0x1d20 net/ipv6/netfilter/ip6_tables.c:282
Code: 48 c7 c7 00 d3 92 8b 65 8b 05 bb 8b 01 77 83 c0 01 83 e0 01 41 89 c5 e8 dd 0f 31 01 65 44 01 2d a5 8b 01 77 f0 83 44 24 fc 00 <48> 8b 44 24 40 48 8d 78 18 48 b8 00 00 00 00 00 fc ff df 48 89 fa
RSP: 0000:ffffc90003e5ea60 EFLAGS: 00000286
RAX: 0000000000000001 RBX: 0000000000000010 RCX: 0000000000000100
RDX: ffff888028b18000 RSI: ffffffff8b92d300 RDI: ffffffff8ac862e0
RBP: ffff888051604000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000010 R11: 0200000000000000 R12: 1ffff920007cbd6b
R13: 0000000000000001 R14: 0000000000000004 R15: ffff888044644dc0
FS:  000000c000bfc890(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005564f350d040 CR3: 00000000228f4000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 ip6table_mangle_hook+0xc4/0x780 net/ipv6/netfilter/ip6table_mangle.c:72
 nf_hook_entry_hookfn include/linux/netfilter.h:143 [inline]
 nf_hook_slow+0xbf/0x1e0 net/netfilter/core.c:626
 nf_hook include/linux/netfilter.h:258 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 br_nf_post_routing+0xa32/0x15c0 net/bridge/br_netfilter_hooks.c:856
 nf_hook_entry_hookfn include/linux/netfilter.h:143 [inline]
 nf_hook_slow+0xbf/0x1e0 net/netfilter/core.c:626
 nf_hook include/linux/netfilter.h:258 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 br_forward_finish+0x266/0x480 net/bridge/br_forward.c:66
 br_nf_hook_thresh+0x2ff/0x410 net/bridge/br_netfilter_hooks.c:1048
 br_nf_forward_finish+0x431/0xa70 net/bridge/br_netfilter_hooks.c:567
 NF_HOOK include/linux/netfilter.h:303 [inline]
 NF_HOOK include/linux/netfilter.h:297 [inline]
 br_nf_forward_ip+0xf6c/0x1760 net/bridge/br_netfilter_hooks.c:637
 nf_hook_entry_hookfn include/linux/netfilter.h:143 [inline]
 nf_hook_slow+0xbf/0x1e0 net/netfilter/core.c:626
 nf_hook include/linux/netfilter.h:258 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 __br_forward+0x2d9/0x900 net/bridge/br_forward.c:115
 deliver_clone net/bridge/br_forward.c:131 [inline]
 br_flood+0x39e/0x640 net/bridge/br_forward.c:244
 br_handle_frame_finish+0xfcb/0x1dd0 net/bridge/br_input.c:210
 br_nf_hook_thresh+0x2ff/0x410 net/bridge/br_netfilter_hooks.c:1048
 br_nf_pre_routing_finish_ipv6+0x683/0xf20 net/bridge/br_netfilter_ipv6.c:148
 NF_HOOK include/linux/netfilter.h:303 [inline]
 br_nf_pre_routing_ipv6+0x41b/0x850 net/bridge/br_netfilter_ipv6.c:178
 br_nf_pre_routing+0x8d8/0x1950 net/bridge/br_netfilter_hooks.c:508
 nf_hook_entry_hookfn include/linux/netfilter.h:143 [inline]
 nf_hook_bridge_pre net/bridge/br_input.c:272 [inline]
 br_handle_frame+0x9da/0x16d0 net/bridge/br_input.c:417
 __netif_receive_skb_core.constprop.0+0xa78/0x3df0 net/core/dev.c:5346
 __netif_receive_skb_one_core+0xaf/0x180 net/core/dev.c:5450
 __netif_receive_skb+0x1f/0x1b0 net/core/dev.c:5566
 process_backlog+0x101/0x6c0 net/core/dev.c:5894
 __napi_poll.constprop.0+0xb4/0x530 net/core/dev.c:6460
 napi_poll net/core/dev.c:6527 [inline]
 net_rx_action+0x956/0xe90 net/core/dev.c:6660
 __do_softirq+0x218/0x965 kernel/softirq.c:553
 invoke_softirq kernel/softirq.c:427 [inline]
 __irq_exit_rcu kernel/softirq.c:632 [inline]
 irq_exit_rcu+0xb7/0x120 kernel/softirq.c:644
 sysvec_apic_timer_interrupt+0x47/0xc0 arch/x86/kernel/apic/apic.c:1109
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0033:0x4a2bca
Code: 8b 4c 24 18 48 8b 7c 24 20 48 8b 74 24 28 e9 4d fe ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc 49 3b 66 10 0f 86 92 00 00 00 <48> 83 ec 30 48 89 6c 24 28 48 8d 6c 24 28 48 89 44 24 38 48 89 5c
RSP: 002b:000000c000025298 EFLAGS: 00000202
RAX: 0000000000a32220 RBX: 000000c0027205e8 RCX: 0000000000000099
RDX: 0000000000000099 RSI: 0000000000000018 RDI: 0000000000000005
RBP: 000000c0000253f8 R08: 0000000000000099 R09: 00000000004aa400
R10: 0000000000d3da98 R11: 0000000000a32220 R12: 0000000000d3da98
R13: 0000000000000041 R14: 000000c000750b60 R15: 000000c000bfc800
 </TASK>
net_ratelimit: 10766 callbacks suppressed
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on veth0_to_bridge with own address as source address (addr:b6:eb:ab:21:85:8c, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
net_ratelimit: 12100 callbacks suppressed
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on veth0_to_bridge with own address as source address (addr:b6:eb:ab:21:85:8c, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)
bridge0: received packet on bridge_slave_0 with own address as source address (addr:aa:aa:aa:aa:aa:1b, vlan:0)

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/08/29 11:16 upstream 1c59d383390f 7ba13a15 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in task_numa_work
2023/08/30 14:45 net e4da8c78973c 84803932 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: rcu detected stall in task_numa_work
* Struck through repros no longer work on HEAD.