bisecting fixing commit since fc3abb53250a90ba2150eebd182137c136f4d25a
building syzkaller on abf9ba4fc75d9b29af15625d44dcfc1360fad3b7
testing commit fc3abb53250a90ba2150eebd182137c136f4d25a with gcc (GCC) 8.1.0
kernel signature: dfcab66d04c760212847cd7829140c9a2e69b4f0fb48ee520ff4be31f381ca6f
run #0: crashed: BUG: workqueue lockup
run #1: crashed: INFO: rcu detected stall in do_idle
run #2: crashed: BUG: workqueue lockup
run #3: crashed: BUG: workqueue lockup
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: BUG: workqueue lockup
run #7: crashed: INFO: rcu detected stall in do_idle
run #8: crashed: BUG: workqueue lockup
run #9: crashed: no output from test machine
testing current HEAD 0477e92881850d44910a7e94fc2c46f96faa131f
testing commit 0477e92881850d44910a7e94fc2c46f96faa131f with gcc (GCC) 8.1.0
kernel signature: 496b616e365f9b4a62bd826fee8cf8b8443000c55cb4c18a0f0be7c0af9384d4
run #0: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #1: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #2: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #3: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #4: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #5: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #6: boot failed: create image operation failed: &{Code:ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS Location: Message:The zone 'projects/syzkaller/zones/us-central1-c' does not have enough resources available to fulfill the request. '(resource type:compute)'. ForceSendFields:[] NullFields:[]}.
run #7: crashed: BUG: workqueue lockup
run #8: crashed: BUG: workqueue lockup
run #9: crashed: BUG: workqueue lockup
revisions tested: 2, total time: 24m21.465990931s (build: 10m8.447327094s, test: 13m40.906912533s)
the crash still happens on HEAD
commit msg: Linux 5.10-rc7
crash: BUG: workqueue lockup
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 91s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
in-flight: 7435:linkwatch_event
pending: free_work
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=12/256 refcnt=13
pending: kfree_rcu_monitor, nsim_dev_trap_report_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, psi_avgs_work, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, vmstat_shepherd, cache_reap
workqueue events_long: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5
pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...!: (1 GPs behind) idle=432/0/0x3 softirq=8755/8773 fqs=0
(t=14713 jiffies g=3409 q=647)
rcu: rcu_preempt kthread starved for 14713 jiffies! g3409 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:I stack:14328 pid: 10 ppid: 2 flags:0x00004000
Call Trace:
context_switch kernel/sched/core.c:3779 [inline]
__schedule+0x404/0x890 kernel/sched/core.c:4528
schedule+0x38/0xe0 kernel/sched/core.c:4606
schedule_timeout+0x1be/0x2e0 kernel/time/timer.c:1871
rcu_gp_fqs_loop kernel/rcu/tree.c:1925 [inline]
rcu_gp_kthread+0x707/0xc60 kernel/rcu/tree.c:2099
kthread+0x145/0x170 kernel/kthread.c:292
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296
NMI backtrace for cpu 0
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.0-rc7-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0xa3/0xc8 lib/dump_stack.c:118
nmi_cpu_backtrace.cold.8+0x53/0x6d lib/nmi_backtrace.c:105
nmi_trigger_cpumask_backtrace+0xd5/0xf0 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0xa2/0xce kernel/rcu/tree_stall.h:331
print_cpu_stall kernel/rcu/tree_stall.h:563 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:637 [inline]
rcu_pending kernel/rcu/tree.c:3694 [inline]
rcu_sched_clock_irq.cold.95+0x61/0x5d5 kernel/rcu/tree.c:2567
update_process_times+0x50/0x80 kernel/time/timer.c:1709
tick_sched_handle.isra.24+0x1a/0x50 kernel/time/tick-sched.c:176
tick_sched_timer+0x6c/0x80 kernel/time/tick-sched.c:1328
__run_hrtimer kernel/time/hrtimer.c:1519 [inline]
__hrtimer_run_queues+0x1e3/0x4f0 kernel/time/hrtimer.c:1583
hrtimer_interrupt+0xf9/0x210 kernel/time/hrtimer.c:1645
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1080 [inline]
__sysvec_apic_timer_interrupt+0x8e/0x290 arch/x86/kernel/apic/apic.c:1097
run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:91 [inline]
sysvec_apic_timer_interrupt+0x52/0xf0 arch/x86/kernel/apic/apic.c:1091
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:631
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:653 [inline]
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x32/0x70 kernel/locking/spinlock.c:191
Code: 53 48 89 f3 48 8b 74 24 10 e8 0a 74 15 fe 48 89 ef e8 62 b8 15 fe f6 c7 02 75 2c 48 83 3d 0d ed 1e 01 00 74 31 48 89 df 57 9d <0f> 1f 44 00 00 bf 01 00 00 00 e8 0f 17 12 fe 65 8b 05 28 a8 f3 7c
RSP: 0018:ffffc90000003e08 EFLAGS: 00000282
RAX: 0000000000046b1a RBX: 0000000000000282 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffffffff83f3143b RDI: 0000000000000282
RBP: ffff888237c2d380 R08: 0000000000000001 R09: 0000000000000001
R10: ffffc90000003c10 R11: ffffc90000003c08 R12: 0000000000000282
R13: ffffffff83eac151 R14: 00000000ffffd5da R15: ffff888237c31e00
show_workqueue_state.cold.58+0xa4/0x2da kernel/workqueue.c:4788
wq_watchdog_timer_fn+0x224/0x230 kernel/workqueue.c:5801
call_timer_fn+0xa5/0x300 kernel/time/timer.c:1410
expire_timers kernel/time/timer.c:1455 [inline]
__run_timers kernel/time/timer.c:1747 [inline]
run_timer_softirq+0x4d2/0x570 kernel/time/timer.c:1762
__do_softirq+0xe9/0x52f kernel/softirq.c:298
asm_call_irq_on_stack+0xf/0x20
__run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
do_softirq_own_stack+0x7c/0xa0 arch/x86/kernel/irq_64.c:77
invoke_softirq kernel/softirq.c:393 [inline]
__irq_exit_rcu kernel/softirq.c:423 [inline]
irq_exit_rcu+0xea/0x110 kernel/softirq.c:435
sysvec_apic_timer_interrupt+0x57/0xf0 arch/x86/kernel/apic/apic.c:1091
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:631
RIP: 0010:native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:61
Code: 5b c3 65 48 8b 04 25 c0 7e 01 00 f0 80 48 02 20 48 8b 00 a8 08 75 c3 e9 7c ff ff ff e9 07 00 00 00 0f 00 2d 0c 45 54 00 fb f4 90 e9 07 00 00 00 0f 00 2d fc 44 54 00 f4 c3 cc cc e8 cb 07 ff
RSP: 0018:ffffffff84203e38 EFLAGS: 00000286
RAX: 0000000000045261 RBX: ffff888103fd5c00 RCX: 00000000ffffffff
RDX: 0000000000000000 RSI: ffffffff83f3143b RDI: ffffffff83f6e406
RBP: ffffffff84529b80 R08: 0000000000000001 R09: 0000000000000001
R10: ffff888237c2c404 R11: ffff888237c2c3e4 R12: 0000000000000001
R13: ffff888100ca5864 R14: 0000000000000001 R15: 0000000000000000
arch_safe_halt arch/x86/include/asm/paravirt.h:150 [inline]
acpi_safe_halt drivers/acpi/processor_idle.c:111 [inline]
acpi_idle_do_entry+0x50/0x90 drivers/acpi/processor_idle.c:517
acpi_idle_enter+0xa0/0xf0 drivers/acpi/processor_idle.c:648
cpuidle_enter_state+0x94/0x520 drivers/cpuidle/cpuidle.c:237
cpuidle_enter+0x24/0x40 drivers/cpuidle/cpuidle.c:351
call_cpuidle kernel/sched/idle.c:158 [inline]
cpuidle_idle_call kernel/sched/idle.c:239 [inline]
do_idle+0x2dc/0x350 kernel/sched/idle.c:299
cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:395
start_kernel+0x4eb/0x50a init/main.c:1061
secondary_startup_64_no_verify+0xb0/0xbb