syzbot

 sign-in | mailing list | source | docs
🐞 Open [717] 🐞 Fixed [181] 🐞 Invalid [640] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

INFO: rcu detected stall in corrupted

Status: fixed on 2019/12/05 23:59
Reported-by: syzbot+d3a6c6c6579d03ba70d7@syzkaller.appspotmail.com
Fix commit: cc243e2427ce sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero
First crash: 1702d, last: 1680d
Fix bisection: fixed by (bisect log) :
commit cc243e2427cef2a5dd7367cb0e0b846503350ffe
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Sun Sep 8 20:40:51 2019 +0000

  sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero

  
▶ ▼ Similar bugs (10)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 INFO: rcu detected stall in corrupted origin:upstream C 138 2h29m 403d 0/3 upstream: reported C repro on 2023/03/13 04:03
android-414 INFO: rcu detected stall in corrupted C 3 1763d 1835d 0/1 public: reported C repro on 2019/04/10 16:04
android-49 INFO: rcu detected stall in corrupted C 27 1647d 1832d 0/3 public: reported C repro on 2019/04/14 00:00
upstream INFO: rcu detected stall in corrupted (3) kernel C done 1 1851d 1847d 0/26 closed as invalid on 2019/03/30 00:13
linux-4.14 INFO: rcu detected stall in corrupted (2) C error 4 1405d 1567d 0/1 upstream: reported C repro on 2020/01/04 08:54
upstream INFO: rcu detected stall in corrupted (4) bpf C done inconclusive 416 4d08h 1322d 0/26 upstream: reported C repro on 2020/09/04 18:49
upstream INFO: rcu detected stall in corrupted net syz 1 2160d 2159d 8/26 fixed on 2018/07/09 18:05
linux-4.19 INFO: rcu detected stall in corrupted C error 2 819d 1527d 0/1 upstream: reported C repro on 2020/02/13 05:06
upstream INFO: rcu detected stall in corrupted (2) kernel syz 1 2091d 2091d 0/26 closed as invalid on 2018/07/29 11:55
linux-5.15 INFO: rcu detected stall in corrupted origin:lts-only C error 4 34d 179d 0/3 upstream: reported C repro on 2023/10/23 02:41

Sample crash report:
IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
IPv6: ADDRCONF(NETDEV_CHANGE): team0: link becomes ready
IPv6: ADDRCONF(NETDEV_UP): vxcan1: link is not ready
8021q: adding VLAN 0 to HW filter on device batadv0
INFO: rcu_preempt self-detected stall on CPU
	1-...: (10499 ticks this GP) idle=b02/140000000000001/0 softirq=13713/13713 fqs=25 
	 (t=10501 jiffies g=1138 c=1137 q=194)
rcu_preempt kthread starved for 10451 jiffies! g1138 c1137 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu_preempt     I
	1-...: (10499 ticks this GP) idle=b02/140000000000001/0 softirq=13713/13713 fqs=25 
29824     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
	
(detected by 0, t=10502 jiffies, g=1138, c=1137, q=194)
Sending NMI from CPU 0 to CPUs 1:
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
NMI backtrace for cpu 1
CPU: 1 PID: 6943 Comm: kworker/1:3 Not tainted 4.14.143 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
task: ffff888097e2c640 task.stack: ffff888091280000
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:60
RSP: 0018:ffff8880aef07680 EFLAGS: 00000082
RAX: fffffffffffffffd RBX: 0000000000000006 RCX: 0000000000000001
RDX: 0000000000010000 RSI: 0000000000000002 RDI: ffffffff8707aaa6
RBP: ffff8880aef07788 R08: ffffffff866c00a4 R09: ffffffff88c857b0
R10: ffff888097e2d028 R11: ffff888097e2c640 R12: 0000000000000006
R13: 0000000000000006 R14: dffffc0000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880aef00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dab10 CR3: 000000000766a000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 vsnprintf+0x1eb/0x1560 lib/vsprintf.c:2247
 snprintf+0xa5/0xd0 lib/vsprintf.c:2314
 print_time kernel/printk/printk.c:1222 [inline]
 print_prefix+0x23c/0x3a0 kernel/printk/printk.c:1247
 msg_print_text+0x83/0x1b0 kernel/printk/printk.c:1270
 console_unlock+0x302/0xed0 kernel/printk/printk.c:2368
 vprintk_emit kernel/printk/printk.c:1917 [inline]
 vprintk_emit+0x1f9/0x600 kernel/printk/printk.c:1882
 vprintk_default+0x28/0x30 kernel/printk/printk.c:1957
 vprintk_func+0x5d/0x159 kernel/printk/printk_safe.c:401
 printk+0x9e/0xbc kernel/printk/printk.c:1990
 printk_stack_address arch/x86/kernel/dumpstack.c:70 [inline]
 show_trace_log_lvl+0x22d/0x26b arch/x86/kernel/dumpstack.c:202
 show_stack+0x39/0x3b arch/x86/kernel/dumpstack.c:237
 sched_show_task kernel/sched/core.c:5167 [inline]
 sched_show_task.cold+0x2f0/0x351 kernel/sched/core.c:5142
 rcu_check_gp_kthread_starvation+0x28e/0x2c7 kernel/rcu/tree.c:1374
 print_cpu_stall kernel/rcu/tree.c:1540 [inline]
 check_cpu_stall kernel/rcu/tree.c:1610 [inline]
 __rcu_pending kernel/rcu/tree.c:3390 [inline]
 rcu_pending kernel/rcu/tree.c:3452 [inline]
 rcu_check_callbacks.cold+0x435/0xd0a kernel/rcu/tree.c:2792
 update_process_times+0x31/0x70 kernel/time/timer.c:1588
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
 tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
 __run_hrtimer kernel/time/hrtimer.c:1220 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:__list_add_valid+0x48/0xa0 lib/list_debug.c:26
RSP: 0018:ffff888091287528 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff888090125438 RCX: 0000000000000000
RDX: ffff8880901254d0 RSI: ffff8880901254d0 RDI: ffff8880901254d8
RBP: ffff888091287540 R08: 0000000000000000 R09: ffff888097e2cff8
R10: ffff888097e2cfd8 R11: ffff888097e2c640 R12: ffff8880901254d0
R13: ffff888090125438 R14: ffff8880901254d0 R15: ffff8880901254c0
 __list_add include/linux/list.h:60 [inline]
 list_add_tail include/linux/list.h:93 [inline]
 list_move_tail include/linux/list.h:183 [inline]
 hhf_dequeue+0x61b/0xa60 net/sched/sch_hhf.c:438
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip6_finish_output2+0x9ab/0x21b0 net/ipv6/ip6_output.c:120
 ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:462 [inline]
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
 ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
 addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: 00 00 48 c7 c7 80 99 6e 86 4c 89 35 eb 4f be 07 41 be f4 ff ff ff e8 13 3c ee ff 48 c7 05 d5 4f be 07 00 00 00 00 e9 2f ec ff ff <65> 48 8b 04 25 40 ee 01 00 48 85 c0 74 1a 65 8b 15 4b 0b a5 7e 
rcu_preempt kthread starved for 10452 jiffies! g1138 c1137 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
rcu_preempt     I
29824     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 kthread+0x319/0x430 kernel/kthread.c:232
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 1
CPU: 1 PID: 6943 Comm: kworker/1:3 Not tainted 4.14.143 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
Call Trace:
 kthread+0x319/0x430 kernel/kthread.c:232
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x138/0x197 lib/dump_stack.c:53
 nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
 nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
 rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
 print_cpu_stall kernel/rcu/tree.c:1542 [inline]
 check_cpu_stall kernel/rcu/tree.c:1610 [inline]
 __rcu_pending kernel/rcu/tree.c:3390 [inline]
 rcu_pending kernel/rcu/tree.c:3452 [inline]
 rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
 update_process_times+0x31/0x70 kernel/time/timer.c:1588
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
 tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
 __run_hrtimer kernel/time/hrtimer.c:1220 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:__list_add_valid+0x48/0xa0 lib/list_debug.c:26
RSP: 0018:ffff888091287528 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff888090125438 RCX: 0000000000000000
RDX: ffff8880901254d0 RSI: ffff8880901254d0 RDI: ffff8880901254d8
RBP: ffff888091287540 R08: 0000000000000000 R09: ffff888097e2cff8
R10: ffff888097e2cfd8 R11: ffff888097e2c640 R12: ffff8880901254d0
R13: ffff888090125438 R14: ffff8880901254d0 R15: ffff8880901254c0
 __list_add include/linux/list.h:60 [inline]
 list_add_tail include/linux/list.h:93 [inline]
 list_move_tail include/linux/list.h:183 [inline]
 hhf_dequeue+0x61b/0xa60 net/sched/sch_hhf.c:438
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip6_finish_output2+0x9ab/0x21b0 net/ipv6/ip6_output.c:120
 ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:462 [inline]
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
 ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
 addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
INFO: rcu_sched detected stalls on CPUs/tasks:
	1-...: (10501 ticks this GP) idle=b02/140000000000000/0 softirq=13713/13713 fqs=25 
	(detected by 0, t=10502 jiffies, g=788, c=787, q=7)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 6943 Comm: kworker/1:3 Not tainted 4.14.143 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
task: ffff888097e2c640 task.stack: ffff888091280000
RIP: 0010:__read_once_size include/linux/compiler.h:183 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x1d/0x60 kernel/kcov.c:67
RSP: 0018:ffff888091287548 EFLAGS: 00000246
RAX: ffff888097e2c640 RBX: ffff888090125438 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8880901254d0 RDI: ffff8880901254d8
RBP: ffff8880912875a0 R08: 0000000000000000 R09: ffff888097e2cff8
R10: ffff888097e2cfd8 R11: ffff888097e2c640 R12: dffffc0000000000
R13: ffff888090125240 R14: ffff8880901254d0 R15: ffff8880901254c0
FS:  0000000000000000(0000) GS:ffff8880aef00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dab10 CR3: 000000000766a000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip6_finish_output2+0x9ab/0x21b0 net/ipv6/ip6_output.c:120
 ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
 dst_output include/net/dst.h:462 [inline]
 NF_HOOK include/linux/netfilter.h:250 [inline]
 ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
 ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
 addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: 05 d5 4f be 07 00 00 00 00 e9 2f ec ff ff 65 48 8b 04 25 40 ee 01 00 48 85 c0 74 1a 65 8b 15 4b 0b a5 7e 81 e2 00 01 1f 00 75 0b <8b> 90 c8 12 00 00 83 fa 01 74 01 c3 55 48 89 e5 48 8b 75 08 48 
rcu_sched kthread starved for 10452 jiffies! g788 c787 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
rcu_sched       I29824     9      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/13 06:08 linux-4.14.y e2cd24b62938 40fa42bc .config console log report syz C ci2-linux-4-14
2019/09/08 09:40 linux-4.14.y 414510bc00a5 a60cb4cd .config console log report syz C ci2-linux-4-14
2019/08/22 06:56 linux-4.14.y 45f092f9e9cb 4ea67ff8 .config console log report syz ci2-linux-4-14
* Struck through repros no longer work on HEAD.