bisecting fixing commit since d573e8a79f70404ba08623d1de7ea617d55092ac building syzkaller on d96e88f3207d7ac7ad65e13b896f702ad04c46f7 testing commit d573e8a79f70404ba08623d1de7ea617d55092ac with gcc (GCC) 8.4.1 20210217 kernel signature: 8cd36744f313f19bedd7fd44232c01b543cd2d77ef5bd867e5c82061cffeb01a run #0: crashed: BUG: workqueue lockup run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in pie_timer run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in pie_timer run #8: crashed: INFO: rcu detected stall in pie_timer run #9: crashed: INFO: rcu detected stall in pie_timer run #10: crashed: INFO: rcu detected stall in pie_timer run #11: crashed: INFO: rcu detected stall in corrupted run #12: crashed: INFO: rcu detected stall in corrupted run #13: crashed: INFO: rcu detected stall in run_timer_softirq run #14: crashed: INFO: rcu detected stall in pie_timer run #15: crashed: INFO: rcu detected stall in lock_is_held_type run #16: crashed: INFO: rcu detected stall in pie_timer run #17: crashed: INFO: rcu detected stall in pie_timer run #18: crashed: INFO: rcu detected stall in run_timer_softirq run #19: crashed: INFO: rcu detected stall in pie_timer testing current HEAD 2d19be4653f5e74ed95560b69f94eb6791d49af3 testing commit 2d19be4653f5e74ed95560b69f94eb6791d49af3 with gcc (GCC) 8.4.1 20210217 kernel signature: f8dc1014604717d22203fe49bdbb81e8174e233bb2c89bf10e6b3bb656232a86 run #0: crashed: INFO: rcu detected stall in pie_timer run #1: crashed: BUG: workqueue lockup run #2: crashed: BUG: workqueue lockup run #3: crashed: BUG: workqueue lockup run #4: crashed: BUG: workqueue lockup run #5: crashed: BUG: workqueue lockup run #6: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "/tmp/syz-executor353622682" "root@10.128.10.32:./syz-executor353622682"]: exit status 1 ssh: connect to host 10.128.10.32 port 22: Connection timed out lost connection run #7: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "/tmp/syz-executor634179889" "root@10.128.10.37:./syz-executor634179889"]: exit status 1 ssh: connect to host 10.128.10.37 port 22: Connection timed out lost connection run #8: crashed: BUG: workqueue lockup run #9: crashed: INFO: rcu detected stall in run_timer_softirq revisions tested: 2, total time: 34m29.263720437s (build: 15m3.397756723s, test: 18m12.684343741s) the crash still happens on HEAD commit msg: Linux 4.19.177 crash: INFO: rcu detected stall in run_timer_softirq IPVS: ftp: loaded support on port[0] = 21 rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 0-...!: (1 GPs behind) idle=0fa/1/0x4000000000000004 softirq=48888/48889 fqs=1423 rcu: (t=10501 jiffies g=76221 q=5252) rcu: rcu_preempt kthread starved for 7655 jiffies! g76221 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 rcu: RCU grace-period kthread stack dump: rcu_preempt R running task 29208 10 2 0x80000000 Call Trace: context_switch kernel/sched/core.c:2828 [inline] __schedule+0xdb5/0x1d40 kernel/sched/core.c:3517 schedule+0x7f/0x1b0 kernel/sched/core.c:3561 schedule_timeout+0x3ab/0xe00 kernel/time/timer.c:1818 rcu_gp_kthread+0xce6/0x23d0 kernel/rcu/tree.c:2202 kthread+0x347/0x410 kernel/kthread.c:259 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415 NMI backtrace for cpu 0 CPU: 0 PID: 6747 Comm: syz-executor.5 Not tainted 4.19.177-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x123/0x171 lib/dump_stack.c:118 nmi_cpu_backtrace.cold.4+0x3e/0x76 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0xe6/0x120 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x175/0x1c2 kernel/rcu/tree.c:1340 print_cpu_stall kernel/rcu/tree.c:1478 [inline] check_cpu_stall kernel/rcu/tree.c:1550 [inline] __rcu_pending kernel/rcu/tree.c:3293 [inline] rcu_pending kernel/rcu/tree.c:3336 [inline] rcu_check_callbacks.cold.81+0x5b1/0xd89 kernel/rcu/tree.c:2682 update_process_times+0x2a/0x60 kernel/time/timer.c:1650 tick_sched_handle+0x77/0x140 kernel/time/tick-sched.c:168 tick_sched_timer+0x37/0xf0 kernel/time/tick-sched.c:1278 __run_hrtimer kernel/time/hrtimer.c:1401 [inline] __hrtimer_run_queues+0x2f7/0xab0 kernel/time/hrtimer.c:1463 hrtimer_interrupt+0x2df/0x760 kernel/time/hrtimer.c:1521 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1071 [inline] smp_apic_timer_interrupt+0x114/0x540 arch/x86/kernel/apic/apic.c:1096 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:894 RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline] RIP: 0010:_raw_spin_unlock_irq+0x56/0x80 kernel/locking/spinlock.c:192 Code: 12 88 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 32 48 83 3d 6b e8 89 01 00 74 1f fb 66 0f 1f 44 00 00 01 00 00 00 e8 50 f2 b9 fa 65 8b 05 89 d5 79 79 85 c0 74 05 5b RSP: 0018:ffff8881f6807df0 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 RAX: dffffc0000000000 RBX: ffff8881f6823a00 RCX: 1ffff11035a3cd95 RDX: 1ffffffff1024013 RSI: ffff8881ad1e6cb0 RDI: ffffffff88120098 RBP: ffff8881f6807df8 R08: ffff8881ad1e6ca8 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881f6807e70 R13: dffffc0000000000 R14: ffffffff85288c50 R15: ffff8881f6823a00 expire_timers kernel/time/timer.c:1374 [inline] __run_timers kernel/time/timer.c:1696 [inline] __run_timers kernel/time/timer.c:1664 [inline] run_timer_softirq+0xb58/0x1180 kernel/time/timer.c:1709 __do_softirq+0x25f/0x919 kernel/softirq.c:292 invoke_softirq kernel/softirq.c:372 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:412 exiting_irq arch/x86/include/asm/apic.h:535 [inline] smp_apic_timer_interrupt+0x13e/0x540 arch/x86/kernel/apic/apic.c:1098 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:894 RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:789 [inline] RIP: 0010:lock_acquire+0x1f1/0x3d0 kernel/locking/lockdep.c:3910 Code: 00 00 00 00 00 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 aa 01 00 00 48 83 3d 83 85 c6 06 00 0f 84 31 01 00 00 48 8b 7d c8 57 9d <0f> 1f 44 00 00 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 8b RSP: 0018:ffff8881a93c7630 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 RAX: dffffc0000000000 RBX: ffff8881ad1e6400 RCX: 1ffff11035a3cd9a RDX: 1ffffffff1024011 RSI: 0000000000000000 RDI: 0000000000000282 RBP: ffff8881a93c7678 R08: ffff8881ad1e6cd0 R09: 0000000000000001 R10: ffff8881ad1e6cb0 R11: 0000000000000001 R12: 0000000000000000 R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000 rcu_lock_acquire include/linux/rcupdate.h:242 [inline] rcu_read_lock include/linux/rcupdate.h:627 [inline] lock_page_memcg+0x39/0x240 mm/memcontrol.c:1908 page_remove_file_rmap mm/rmap.c:1215 [inline] page_remove_rmap+0x35d/0xd20 mm/rmap.c:1300 zap_pte_range mm/memory.c:1350 [inline] zap_pmd_range mm/memory.c:1452 [inline] zap_pud_range mm/memory.c:1481 [inline] zap_p4d_range mm/memory.c:1502 [inline] unmap_page_range+0xb47/0x20c0 mm/memory.c:1523 unmap_single_vma+0x121/0x300 mm/memory.c:1568 unmap_vmas+0xe7/0x1a0 mm/memory.c:1598 exit_mmap+0x251/0x470 mm/mmap.c:3093 __mmput kernel/fork.c:1015 [inline] mmput+0x107/0x3f0 kernel/fork.c:1036 exit_mm kernel/exit.c:549 [inline] do_exit+0x9ce/0x2cf0 kernel/exit.c:857 do_group_exit+0xf4/0x2f0 kernel/exit.c:967 get_signal+0x316/0x19e0 kernel/signal.c:2589 do_signal+0x87/0x1960 arch/x86/kernel/signal.c:821 exit_to_usermode_loop+0x114/0x200 arch/x86/entry/common.c:163 prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline] syscall_return_slowpath arch/x86/entry/common.c:271 [inline] do_syscall_64+0x413/0x4e0 arch/x86/entry/common.c:296 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x461209 Code: Bad RIP value. RSP: 002b:00007fc08a58a1a8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a RAX: 0000000000000006 RBX: 000000000052bf00 RCX: 0000000000461209 RDX: ffffffffffffffff RSI: 0000000000000000 RDI: 000000002001d000 RBP: 0000000000000005 R08: 0000000000000000 R09: 0000000000000000 R10: ffffffffffffffff R11: 0000000000000246 R12: 00000000ffffffff R13: 00000000004f42a8 R14: 00000000004b240a R15: 00007fc08a58a6bc BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 77s! BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 77s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=6/256 refcnt=7 pending: delayed_fput, linkwatch_event, defense_work_handler, defense_work_handler, defense_work_handler, check_corruption pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=7/256 refcnt=8 pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler, vmstat_shepherd, cache_reap workqueue events_freezable: flags=0x4 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: update_balloon_stats_func workqueue events_power_efficient: flags=0x80 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: check_lifetime pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: gc_worker, neigh_periodic_work, neigh_periodic_work workqueue mm_percpu_wq: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4e pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wb_workfn workqueue dm_bufio_cache: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: work_fn workqueue ipv6_addrconf: flags=0x40008 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 pending: addrconf_verify_work workqueue bat_events: flags=0xe000a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=38 pending: batadv_nc_worker delayed: batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_purge_orig, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_purge_orig, batadv_purge_orig, batadv_purge_orig, batadv_purge_orig, batadv_purge_orig, batadv_tt_purge, batadv_tt_purge, batadv_tt_purge, batadv_tt_purge, batadv_tt_purge, batadv_tt_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_bla_periodic_work, batadv_dat_purge, batadv_dat_purge