bisecting fixing commit since f4cc0ed9b2c72687303b035379c5824a02224354
building syzkaller on 7509bf360eba1461ac6059e4cacfbc29c9d2d4c7
testing commit f4cc0ed9b2c72687303b035379c5824a02224354 with gcc (GCC) 8.1.0
kernel signature: fb07a4326455eeaf5525bdbe4dd256a6e50fd0d92c6babd7c3693ffad8adb5f0
run #0: crashed: BUG: workqueue lockup
run #1: crashed: BUG: workqueue lockup
run #2: crashed: BUG: workqueue lockup
run #3: crashed: BUG: workqueue lockup
run #4: crashed: BUG: workqueue lockup
run #5: crashed: BUG: workqueue lockup
run #6: crashed: BUG: workqueue lockup
run #7: crashed: BUG: workqueue lockup
run #8: crashed: BUG: workqueue lockup
run #9: crashed: INFO: rcu detected stall in corrupted
testing current HEAD 2d2791fce891fc20709232d49a6bae075b9a77f8
testing commit 2d2791fce891fc20709232d49a6bae075b9a77f8 with gcc (GCC) 8.1.0
kernel signature: c2683140db00519e576046ad0af3236af9b89f3bf7752507921e6d2d8d0c62ae
run #0: crashed: BUG: workqueue lockup
run #1: crashed: BUG: workqueue lockup
run #2: crashed: BUG: workqueue lockup
run #3: crashed: BUG: workqueue lockup
run #4: crashed: BUG: workqueue lockup
run #5: crashed: BUG: workqueue lockup
run #6: crashed: BUG: workqueue lockup
run #7: crashed: BUG: workqueue lockup
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in corrupted
revisions tested: 2, total time: 25m19.624757184s (build: 16m3.25883072s, test: 8m44.997966445s)
the crash still happens on HEAD
commit msg: Linux 4.14.217
crash: INFO: rcu detected stall in corrupted
IPv6: ADDRCONF(NETDEV_UP): veth0_to_bond: link is not ready
IPv6: ADDRCONF(NETDEV_UP): veth1_to_bond: link is not ready
IPv6: ADDRCONF(NETDEV_UP): veth0_to_team: link is not ready
IPv6: ADDRCONF(NETDEV_UP): veth1_to_team: link is not ready
INFO: rcu_sched detected stalls on CPUs/tasks:
0-...: (10501 ticks this GP) idle=e96/140000000000000/0 softirq=8523/8529 fqs=6
(detected by 1, t=10503 jiffies, g=190, c=189, q=4)
INFO: rcu_preempt self-detected stall on CPU
Sending NMI from CPU 1 to CPUs 0:
0-...: (10499 ticks this GP) idle=e96/140000000000001/0 softirq=8529/8529 fqs=4
NMI backtrace for cpu 0
CPU: 0 PID: 6463 Comm: syz-executor.0 Not tainted 4.14.217-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
task: ffff8881ef452000 task.stack: ffff8881e1178000
RIP: 0010:number+0x142/0x9d0 lib/vsprintf.c:410
RSP: 0018:ffff8881f6807678 EFLAGS: 00000097
RAX: 0000000000000007 RBX: 0000000000000001 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 1ffff1103ed00e01 RDI: ffff8881f68076fd
RBP: ffff8881f68077a0 R08: ffff8881f6807778 R09: ffff8881f6807778
R10: 0000000000000000 R11: dffffc0000000000 R12: 0000000000000000
R13: ffff8881f68076f8 R14: ffff8881f6807908 R15: dffffc0000000000
FS: 00007fc865e35700(0000) GS:ffff8881f6800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000758080 CR3: 00000001e0c68003 CR4: 00000000001606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
vsnprintf+0xddd/0x1780 lib/vsprintf.c:2247
snprintf+0x88/0xa0 lib/vsprintf.c:2314
print_time kernel/printk/printk.c:1228 [inline]
print_prefix+0x2ea/0x370 kernel/printk/printk.c:1253
msg_print_text+0x69/0x170 kernel/printk/printk.c:1276
console_unlock+0x323/0xe30 kernel/printk/printk.c:2377
vprintk_emit+0x1b4/0x4e0 kernel/printk/printk.c:1923
vprintk_default+0x1a/0x20 kernel/printk/printk.c:1963
vprintk_func+0x49/0x130 kernel/printk/printk_safe.c:401
printk+0x91/0xab kernel/printk/printk.c:1996
print_cpu_stall_info+0x3ce/0x850 kernel/rcu/tree_plugin.h:1700
print_cpu_stall kernel/rcu/tree.c:1531 [inline]
check_cpu_stall kernel/rcu/tree.c:1610 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold.77+0x56/0xced kernel/rcu/tree.c:2792
update_process_times+0x2a/0x60 kernel/time/timer.c:1591
tick_sched_handle+0x7b/0x140 kernel/time/tick-sched.c:165
tick_sched_timer+0x34/0xf0 kernel/time/tick-sched.c:1223
__run_hrtimer kernel/time/hrtimer.c:1223 [inline]
__hrtimer_run_queues+0x297/0xaf0 kernel/time/hrtimer.c:1287
hrtimer_interrupt+0x1ae/0x600 kernel/time/hrtimer.c:1321
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11f/0x5d0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x9a/0xa0 arch/x86/entry/entry_64.S:793
RIP: 0010:init_sync_kiocb include/linux/fs.h:1926 [inline]
RIP: 0010:do_iter_readv_writev+0x2c1/0x900 fs/read_write.c:666
RSP: 0018:ffff8881e117f7d0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000000 RBX: ffff8881e117f878 RCX: 0000000000000001
RDX: ffff8881e117faa0 RSI: ffff8881e117fa28 RDI: ffff8881cb946378
RBP: ffff8881e117f8a0 R08: 0000000000000000 R09: 1ffff11039728c6f
R10: 0000000000000000 R11: 1ffff1103c99043a R12: ffff8881eadb2c80
R13: 0000000000000000 R14: 1ffff1103c22feff R15: ffff8881cb9462b0
do_iter_write+0x12b/0x520 fs/read_write.c:954
vfs_iter_write+0x5b/0xb0 fs/read_write.c:967
iter_file_splice_write+0x540/0xc20 fs/splice.c:749
do_splice_from fs/splice.c:851 [inline]
direct_splice_actor+0x104/0x1c0 fs/splice.c:1018
splice_direct_to_actor+0x27c/0x750 fs/splice.c:973
do_splice_direct+0x144/0x250 fs/splice.c:1061
do_sendfile+0x463/0xd00 fs/read_write.c:1441
SYSC_sendfile64 fs/read_write.c:1496 [inline]
SyS_sendfile64+0x97/0x110 fs/read_write.c:1488
do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x46/0xbb
RIP: 0033:0x459599
RSP: 002b:00007fc865e34c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 0000000000459599
RDX: 0000000020000000 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 00008080fffffffe R11: 0000000000000246 R12: 00007fc865e356d4
R13: 00000000004c6b70 R14: 00000000004dbde8 R15: 00000000ffffffff
Code: 0c 10 49 8d 45 07 48 89 c6 48 c1 ee 03 0f b6 14 16 84 c9 40 0f 9e c6 0f 95 c1 40 84 ce 0f 85 e6 07 00 00 83 e0 07 38 c2 0f 9e c1 <84> d2 0f 95 c0 84 c1 0f 85 d1 07 00 00 41 8b 41 80 c1 f8 08 89
rcu_sched kthread starved for 10490 jiffies! g190 c189 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
rcu_sched R
(t=10500 jiffies g=1003 c=1002 q=408)
running task 29832 9 2 0x80000000
rcu_preempt kthread starved for 10490 jiffies! g1003 c1002 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
Call Trace:
rcu_preempt R
context_switch kernel/sched/core.c:2808 [inline]
__schedule+0x780/0x1d20 kernel/sched/core.c:3384
running task 29720 8 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2808 [inline]
__schedule+0x780/0x1d20 kernel/sched/core.c:3384
schedule+0x7f/0x1b0 kernel/sched/core.c:3428
schedule_timeout+0x40c/0xcc0 kernel/time/timer.c:1747
schedule+0x7f/0x1b0 kernel/sched/core.c:3428
schedule_timeout+0x40c/0xcc0 kernel/time/timer.c:1747
rcu_gp_kthread+0xbbf/0x1e00 kernel/rcu/tree.c:2255
kthread+0x338/0x400 kernel/kthread.c:232
rcu_gp_kthread+0xbbf/0x1e00 kernel/rcu/tree.c:2255
kthread+0x338/0x400 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 0
CPU: 0 PID: 6463 Comm: syz-executor.0 Not tainted 4.14.217-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0xf7/0x137 lib/dump_stack.c:58
nmi_cpu_backtrace.cold.4+0x3e/0x76 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0xe6/0x120 lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
rcu_dump_cpu_stacks+0x172/0x1bf kernel/rcu/tree.c:1396
print_cpu_stall kernel/rcu/tree.c:1542 [inline]
check_cpu_stall kernel/rcu/tree.c:1610 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold.77+0x415/0xced kernel/rcu/tree.c:2792
update_process_times+0x2a/0x60 kernel/time/timer.c:1591
tick_sched_handle+0x7b/0x140 kernel/time/tick-sched.c:165
tick_sched_timer+0x34/0xf0 kernel/time/tick-sched.c:1223
__run_hrtimer kernel/time/hrtimer.c:1223 [inline]
__hrtimer_run_queues+0x297/0xaf0 kernel/time/hrtimer.c:1287
hrtimer_interrupt+0x1ae/0x600 kernel/time/hrtimer.c:1321
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11f/0x5d0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x9a/0xa0 arch/x86/entry/entry_64.S:793
RIP: 0010:init_sync_kiocb include/linux/fs.h:1926 [inline]
RIP: 0010:do_iter_readv_writev+0x2c1/0x900 fs/read_write.c:666
RSP: 0018:ffff8881e117f7d0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000000 RBX: ffff8881e117f878 RCX: 0000000000000001
RDX: ffff8881e117faa0 RSI: ffff8881e117fa28 RDI: ffff8881cb946378
RBP: ffff8881e117f8a0 R08: 0000000000000000 R09: 1ffff11039728c6f
R10: 0000000000000000 R11: 1ffff1103c99043a R12: ffff8881eadb2c80
R13: 0000000000000000 R14: 1ffff1103c22feff R15: ffff8881cb9462b0
do_iter_write+0x12b/0x520 fs/read_write.c:954
vfs_iter_write+0x5b/0xb0 fs/read_write.c:967
iter_file_splice_write+0x540/0xc20 fs/splice.c:749
do_splice_from fs/splice.c:851 [inline]
direct_splice_actor+0x104/0x1c0 fs/splice.c:1018
splice_direct_to_actor+0x27c/0x750 fs/splice.c:973
do_splice_direct+0x144/0x250 fs/splice.c:1061
do_sendfile+0x463/0xd00 fs/read_write.c:1441
SYSC_sendfile64 fs/read_write.c:1496 [inline]
SyS_sendfile64+0x97/0x110 fs/read_write.c:1488
do_syscall_64+0x1c7/0x5b0 arch/x86/entry/common.c:292
entry_SYSCALL_64_after_hwframe+0x46/0xbb
RIP: 0033:0x459599
RSP: 002b:00007fc865e34c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 0000000000459599
RDX: 0000000020000000 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 00008080fffffffe R11: 0000000000000246 R12: 00007fc865e356d4
R13: 00000000004c6b70 R14: 00000000004dbde8 R15: 00000000ffffffff