INFO: rcu detected stall in sys

Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in sys_lsetxattr fs	C	done		2	748d	747d	22/26	fixed on 2023/02/24 13:51
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
rcu: 	1-....: (12318 ticks this GP) idle=5f2/1/0x4000000000000002 softirq=154847/156905 fqs=5249 
rcu: 	(detected by 0, t=10502 jiffies, g=13293, q=1)
Sending NMI from CPU 0 to CPUs 1:
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.205 msecs
NMI backtrace for cpu 1
CPU: 1 PID: 7111 Comm: udevd Not tainted 4.19.73 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:__sanitizer_cov_trace_const_cmp1+0x15/0x20 kernel/kcov.c:174
Code: 00 48 89 e5 48 8b 4d 08 e8 18 ff ff ff 5d c3 66 0f 1f 44 00 00 55 40 0f b6 d6 40 0f b6 f7 bf 01 00 00 00 48 89 e5 48 8b 4d 08 <e8> f6 fe ff ff 5d c3 0f 1f 40 00 55 0f b7 d6 0f b7 f7 bf 03 00 00
RSP: 0018:ffff8880ae906ad8 EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff88808d1c6738 RCX: ffffffff858ebde6
RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff8880ae906ad8 R08: ffff88805854c000 R09: 0000000000000006
R10: ffff88805854c970 R11: 0000000090ba6091 R12: dffffc0000000000
R13: ffff88808d1c6480 R14: 0000000000000001 R15: 0000000000000000
FS:  00007fa6a1fb77a0(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000187b828 CR3: 000000008eff8000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 __list_del_entry include/linux/list.h:117 [inline]
 list_move_tail include/linux/list.h:182 [inline]
 hhf_dequeue+0x5b6/0xa00 net/sched/sch_hhf.c:438
 dequeue_skb net/sched/sch_generic.c:277 [inline]
 qdisc_restart net/sched/sch_generic.c:380 [inline]
 __qdisc_run+0x1e7/0x1960 net/sched/sch_generic.c:398
 __dev_xmit_skb net/core/dev.c:3501 [inline]
 __dev_queue_xmit+0x165c/0x2fe0 net/core/dev.c:3811
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3876
 br_dev_queue_push_xmit+0x405/0x5d0 net/bridge/br_forward.c:56
 br_nf_dev_queue_xmit+0x365/0x1610 net/bridge/br_netfilter_hooks.c:779
 NF_HOOK include/linux/netfilter.h:289 [inline]
 NF_HOOK include/linux/netfilter.h:283 [inline]
 br_nf_post_routing+0x76f/0x12f0 net/bridge/br_netfilter_hooks.c:825
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 br_forward_finish+0x215/0x400 net/bridge/br_forward.c:69
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
 br_nf_forward_finish+0x356/0x700 net/bridge/br_netfilter_hooks.c:553
 NF_HOOK include/linux/netfilter.h:289 [inline]
 NF_HOOK include/linux/netfilter.h:283 [inline]
 br_nf_forward_ip net/bridge/br_netfilter_hooks.c:620 [inline]
 br_nf_forward_ip+0x6b4/0x17c0 net/bridge/br_netfilter_hooks.c:564
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 __br_forward+0x393/0xb30 net/bridge/br_forward.c:113
 deliver_clone+0x61/0xc0 net/bridge/br_forward.c:129
 br_flood+0x5fa/0x710 net/bridge/br_forward.c:238
 br_handle_frame_finish+0xf72/0x1510 net/bridge/br_input.c:167
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
 br_nf_pre_routing_finish_ipv6+0x6ed/0xd70 net/bridge/br_netfilter_ipv6.c:210
 NF_HOOK include/linux/netfilter.h:289 [inline]
 br_nf_pre_routing_ipv6+0x4a5/0x8ca net/bridge/br_netfilter_ipv6.c:240
 br_nf_pre_routing+0x1016/0x159c net/bridge/br_netfilter_hooks.c:494
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 br_handle_frame+0x985/0x1480 net/bridge/br_input.c:306
 __netif_receive_skb_core+0xac1/0x3570 net/core/dev.c:4861
 __netif_receive_skb_one_core+0xa8/0x1a0 net/core/dev.c:4930
 __netif_receive_skb+0x2c/0x1d0 net/core/dev.c:5044
 process_backlog+0x23a/0x7a0 net/core/dev.c:5855
 napi_poll net/core/dev.c:6278 [inline]
 net_rx_action+0x4f5/0x1070 net/core/dev.c:6344
 __do_softirq+0x25c/0x921 kernel/softirq.c:292
 invoke_softirq kernel/softirq.c:372 [inline]
 irq_exit+0x180/0x1d0 kernel/softirq.c:412
 exiting_irq arch/x86/include/asm/apic.h:536 [inline]
 smp_apic_timer_interrupt+0x13b/0x550 arch/x86/kernel/apic/apic.c:1094
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:893
 </IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:789 [inline]
RIP: 0010:kmem_cache_free+0xab/0x260 mm/slab.c:3766
Code: e8 1a 32 00 00 84 c0 74 77 41 f7 c5 00 02 00 00 74 4f e8 08 df ce ff 48 83 3d c8 88 d0 06 00 0f 84 ab 01 00 00 4c 89 ef 57 9d <0f> 1f 44 00 00 4c 8b 6d 08 0f 1f 44 00 00 65 8b 05 d0 c3 5f 7e 83
RSP: 0018:ffff88809372fbe0 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000007 RBX: ffff8880a4a1e500 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000282
RBP: ffff88809372fc00 R08: ffff88805854c000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88821bc47980
R13: 0000000000000282 R14: ffffffff81af18ef R15: ffff88809372fe70
 putname+0xef/0x130 fs/namei.c:261
 filename_lookup+0x28f/0x410 fs/namei.c:2357
 user_path_at_empty+0x43/0x50 fs/namei.c:2608
 user_path_at include/linux/namei.h:57 [inline]
 path_setxattr+0xae/0x1b0 fs/xattr.c:464
 __do_sys_lsetxattr fs/xattr.c:491 [inline]
 __se_sys_lsetxattr fs/xattr.c:487 [inline]
 __x64_sys_lsetxattr+0xc1/0x150 fs/xattr.c:487
 do_syscall_64+0xfd/0x620 arch/x86/entry/common.c:293
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7fa6a16c9ffa
Code: 48 8b 0d 41 9e 2a 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90 90 90 90 90 90 90 90 90 90 90 49 89 ca b8 bd 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0e 9e 2a 00 31 d2 48 29 c2 64
RSP: 002b:00007fff37970328 EFLAGS: 00000206 ORIG_RAX: 00000000000000bd
RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007fa6a16c9ffa
RDX: 000000000176cf80 RSI: 00007fa6a1b98f69 RDI: 00007fff37970d80
RBP: 00007fff37970d80 R08: 0000000000000000 R09: 0000000000001fff
R10: 000000000000001e R11: 0000000000000206 R12: 000000000187a7c0
R13: 00000000ffffffea R14: 00007fff379704e0 R15: 00007fff379708e0
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	1-....: (1 GPs behind) idle=5f2/1/0x4000000000000002 softirq=156904/156905 fqs=5248 
rcu: 	(detected by 0, t=10502 jiffies, g=222837, q=1539)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 7111 Comm: udevd Not tainted 4.19.73 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:__list_add include/linux/list.h:64 [inline]
RIP: 0010:list_add_tail include/linux/list.h:93 [inline]
RIP: 0010:list_move_tail include/linux/list.h:183 [inline]
RIP: 0010:hhf_dequeue+0x695/0xa00 net/sched/sch_hhf.c:438
Code: e8 00 a4 dc fb 4c 89 f0 48 c1 e8 03 42 80 3c 20 00 0f 85 fb 02 00 00 48 89 d8 49 89 9d 58 03 00 00 48 c1 e8 03 42 80 3c 20 00 <0f> 85 a4 02 00 00 48 8b 45 c8 48 8d 7b 08 48 89 03 48 89 f8 48 c1
RSP: 0018:ffff8880ae906ae8 EFLAGS: 00000246
RAX: 1ffff11011a38ce7 RBX: ffff88808d1c6738 RCX: ffffffff858ebe8e
RDX: 0000000000000100 RSI: ffffffff858ebea0 RDI: 0000000000000001
RBP: ffff8880ae906b38 R08: ffff88805854c000 R09: 0000000000000006
R10: ffff88805854c970 R11: 0000000090ba6091 R12: dffffc0000000000
R13: ffff88808d1c6480 R14: ffff88808d1c67d8 R15: ffff88808d1c67d0
FS:  00007fa6a1fb77a0(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000187b828 CR3: 000000008eff8000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 dequeue_skb net/sched/sch_generic.c:277 [inline]
 qdisc_restart net/sched/sch_generic.c:380 [inline]
 __qdisc_run+0x1e7/0x1960 net/sched/sch_generic.c:398
 __dev_xmit_skb net/core/dev.c:3501 [inline]
 __dev_queue_xmit+0x165c/0x2fe0 net/core/dev.c:3811
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3876
 br_dev_queue_push_xmit+0x405/0x5d0 net/bridge/br_forward.c:56
 br_nf_dev_queue_xmit+0x365/0x1610 net/bridge/br_netfilter_hooks.c:779
 NF_HOOK include/linux/netfilter.h:289 [inline]
 NF_HOOK include/linux/netfilter.h:283 [inline]
 br_nf_post_routing+0x76f/0x12f0 net/bridge/br_netfilter_hooks.c:825
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 br_forward_finish+0x215/0x400 net/bridge/br_forward.c:69
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
 br_nf_forward_finish+0x356/0x700 net/bridge/br_netfilter_hooks.c:553
 NF_HOOK include/linux/netfilter.h:289 [inline]
 NF_HOOK include/linux/netfilter.h:283 [inline]
 br_nf_forward_ip net/bridge/br_netfilter_hooks.c:620 [inline]
 br_nf_forward_ip+0x6b4/0x17c0 net/bridge/br_netfilter_hooks.c:564
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 __br_forward+0x393/0xb30 net/bridge/br_forward.c:113
 deliver_clone+0x61/0xc0 net/bridge/br_forward.c:129
 br_flood+0x5fa/0x710 net/bridge/br_forward.c:238
 br_handle_frame_finish+0xf72/0x1510 net/bridge/br_input.c:167
 br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
 br_nf_pre_routing_finish_ipv6+0x6ed/0xd70 net/bridge/br_netfilter_ipv6.c:210
 NF_HOOK include/linux/netfilter.h:289 [inline]
 br_nf_pre_routing_ipv6+0x4a5/0x8ca net/bridge/br_netfilter_ipv6.c:240
 br_nf_pre_routing+0x1016/0x159c net/bridge/br_netfilter_hooks.c:494
 nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
 nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
 nf_hook include/linux/netfilter.h:244 [inline]
 NF_HOOK include/linux/netfilter.h:287 [inline]
 br_handle_frame+0x985/0x1480 net/bridge/br_input.c:306
 __netif_receive_skb_core+0xac1/0x3570 net/core/dev.c:4861
 __netif_receive_skb_one_core+0xa8/0x1a0 net/core/dev.c:4930
 __netif_receive_skb+0x2c/0x1d0 net/core/dev.c:5044
 process_backlog+0x23a/0x7a0 net/core/dev.c:5855
 napi_poll net/core/dev.c:6278 [inline]
 net_rx_action+0x4f5/0x1070 net/core/dev.c:6344
 __do_softirq+0x25c/0x921 kernel/softirq.c:292
 invoke_softirq kernel/softirq.c:372 [inline]
 irq_exit+0x180/0x1d0 kernel/softirq.c:412
 exiting_irq arch/x86/include/asm/apic.h:536 [inline]
 smp_apic_timer_interrupt+0x13b/0x550 arch/x86/kernel/apic/apic.c:1094
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:893
 </IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:789 [inline]
RIP: 0010:kmem_cache_free+0xab/0x260 mm/slab.c:3766
Code: e8 1a 32 00 00 84 c0 74 77 41 f7 c5 00 02 00 00 74 4f e8 08 df ce ff 48 83 3d c8 88 d0 06 00 0f 84 ab 01 00 00 4c 89 ef 57 9d <0f> 1f 44 00 00 4c 8b 6d 08 0f 1f 44 00 00 65 8b 05 d0 c3 5f 7e 83
RSP: 0018:ffff88809372fbe0 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000007 RBX: ffff8880a4a1e500 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000282
RBP: ffff88809372fc00 R08: ffff88805854c000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88821bc47980
R13: 0000000000000282 R14: ffffffff81af18ef R15: ffff88809372fe70
 putname+0xef/0x130 fs/namei.c:261
 filename_lookup+0x28f/0x410 fs/namei.c:2357
 user_path_at_empty+0x43/0x50 fs/namei.c:2608
 user_path_at include/linux/namei.h:57 [inline]
 path_setxattr+0xae/0x1b0 fs/xattr.c:464
 __do_sys_lsetxattr fs/xattr.c:491 [inline]
 __se_sys_lsetxattr fs/xattr.c:487 [inline]
 __x64_sys_lsetxattr+0xc1/0x150 fs/xattr.c:487
 do_syscall_64+0xfd/0x620 arch/x86/entry/common.c:293
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7fa6a16c9ffa
Code: 48 8b 0d 41 9e 2a 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90 90 90 90 90 90 90 90 90 90 90 49 89 ca b8 bd 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0e 9e 2a 00 31 d2 48 29 c2 64
RSP: 002b:00007fff37970328 EFLAGS: 00000206 ORIG_RAX: 00000000000000bd
RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007fa6a16c9ffa
RDX: 000000000176cf80 RSI: 00007fa6a1b98f69 RDI: 00007fff37970d80
RBP: 00007fff37970d80 R08: 0000000000000000 R09: 0000000000001fff
R10: 000000000000001e R11: 0000000000000206 R12: 000000000187a7c0
R13: 00000000ffffffea R14: 00007fff379704e0 R15: 00007fff379708e0