bisecting fixing commit since ae4b064e2a616b545acf02b8f50cc513b32c7522 building syzkaller on 08003f6440deafc4e193b159c4acece64f7864b1 testing commit ae4b064e2a616b545acf02b8f50cc513b32c7522 with gcc (GCC) 8.4.1 20210217 kernel signature: 79450ac766e6ee1c6d674c4246108ce8e17a02768744d4e7b53c5a30c04b9f53 run #0: crashed: BUG: workqueue lockup run #1: crashed: BUG: workqueue lockup run #2: crashed: BUG: workqueue lockup run #3: crashed: INFO: rcu detected stall in chrdev_open run #4: crashed: INFO: rcu detected stall in chrdev_open run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: BUG: workqueue lockup run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: BUG: workqueue lockup run #10: crashed: INFO: rcu detected stall in chrdev_open run #11: crashed: INFO: rcu detected stall in chrdev_open run #12: crashed: INFO: rcu detected stall in chrdev_open run #13: crashed: BUG: workqueue lockup run #14: crashed: INFO: rcu detected stall in corrupted run #15: crashed: INFO: rcu detected stall in chrdev_open run #16: crashed: INFO: rcu detected stall in chrdev_open run #17: crashed: INFO: rcu detected stall in corrupted run #18: crashed: no output from test machine run #19: crashed: INFO: rcu detected stall in corrupted testing current HEAD 6905b1dc3c32a094f0da61bd656a740f0a97d592 testing commit 6905b1dc3c32a094f0da61bd656a740f0a97d592 with gcc (GCC) 10.2.1 20210217 kernel signature: 16490ab2bfb4e2a24c156e25af88d013398cdd45915da113d97eb285470a6451 run #0: crashed: BUG: workqueue lockup run #1: crashed: BUG: workqueue lockup run #2: crashed: BUG: workqueue lockup run #3: crashed: BUG: workqueue lockup run #4: crashed: BUG: workqueue lockup run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in chrdev_open run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in chrdev_open run #9: crashed: INFO: rcu detected stall in chrdev_open revisions tested: 2, total time: 28m7.456799433s (build: 12m6.260235851s, test: 14m56.19129792s) the crash still happens on HEAD commit msg: Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm crash: INFO: rcu detected stall in chrdev_open rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 0-...!: (10500 ticks this GP) idle=20e/1/0x4000000000000000 softirq=8682/8682 fqs=0 (t=10501 jiffies g=6197 q=408) rcu: rcu_preempt kthread timer wakeup didn't happen for 10501 jiffies! g6197 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 rcu: Possible timer handling issue on cpu=0 timer-softirq=1790 rcu: rcu_preempt kthread starved for 10504 jiffies! g6197 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0 rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. rcu: RCU grace-period kthread stack dump: task:rcu_preempt state:I stack:28744 pid: 13 ppid: 2 flags:0x00004000 Call Trace: context_switch kernel/sched/core.c:4322 [inline] __schedule+0xf86/0x2180 kernel/sched/core.c:5073 schedule+0xdb/0x280 kernel/sched/core.c:5152 schedule_timeout+0x133/0x220 kernel/time/timer.c:1892 rcu_gp_fqs_loop kernel/rcu/tree.c:2005 [inline] rcu_gp_kthread+0xcf6/0x2200 kernel/rcu/tree.c:2178 kthread+0x37d/0x460 kernel/kthread.c:292 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294 rcu: Stack dump where RCU GP kthread last ran: NMI backtrace for cpu 0 CPU: 0 PID: 6981 Comm: syz-executor.5 Not tainted 5.12.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x114/0x14e lib/dump_stack.c:120 nmi_cpu_backtrace.cold+0x30/0x9c lib/nmi_backtrace.c:105 nmi_trigger_cpumask_backtrace+0x120/0x170 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:39 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_check_gp_kthread_starvation+0x1a1/0x1b8 kernel/rcu/tree_stall.h:478 print_cpu_stall kernel/rcu/tree_stall.h:620 [inline] check_cpu_stall kernel/rcu/tree_stall.h:697 [inline] rcu_pending kernel/rcu/tree.c:3830 [inline] rcu_sched_clock_irq.cold+0x677/0x11b6 kernel/rcu/tree.c:2650 update_process_times+0x139/0x1b0 kernel/time/timer.c:1796 tick_sched_handle+0x77/0x140 kernel/time/tick-sched.c:226 tick_sched_timer+0x142/0x230 kernel/time/tick-sched.c:1369 __run_hrtimer kernel/time/hrtimer.c:1537 [inline] __hrtimer_run_queues+0x1b7/0xc30 kernel/time/hrtimer.c:1601 hrtimer_interrupt+0x2f3/0x950 kernel/time/hrtimer.c:1663 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline] __sysvec_apic_timer_interrupt+0x143/0x530 arch/x86/kernel/apic/apic.c:1106 sysvec_apic_timer_interrupt+0x98/0xc0 arch/x86/kernel/apic/apic.c:1100 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632 RIP: 0010:unwind_next_frame.part.0+0x123/0x750 arch/x86/kernel/unwind_frame.c:295 Code: 1d 02 00 00 49 8d 46 d8 49 39 c5 0f 84 02 03 00 00 48 8d 7b 50 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 80 3c 01 00 <0f> 85 62 04 00 00 4c 8b 73 50 4d 85 f6 0f 84 a3 02 00 00 48 c7 43 RSP: 0018:ffffc90003d97438 EFLAGS: 00000246 RAX: dffffc0000000000 RBX: ffffc90003d974a8 RCX: 1ffff920007b2e9f RDX: ffff88810890e400 RSI: ffffffff83b8122a RDI: ffffc90003d974f8 RBP: ffffc90003d97480 R08: 1ffff920007b2e9d R09: ffffc90003d974d0 R10: ffffc90003d974e0 R11: 0000000000008400 R12: 0000000000000000 R13: ffffc90003d97868 R14: ffffc90003d97f58 R15: ffffc90003d97500 unwind_next_frame+0x32/0x50 arch/x86/kernel/unwind_frame.c:259 arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25 stack_trace_save+0x8f/0xc0 kernel/stacktrace.c:121 kasan_save_stack+0x23/0x50 mm/kasan/common.c:38 kasan_set_track mm/kasan/common.c:46 [inline] set_alloc_info mm/kasan/common.c:427 [inline] ____kasan_kmalloc mm/kasan/common.c:506 [inline] __kasan_kmalloc+0x7f/0xa0 mm/kasan/common.c:515 kasan_kmalloc include/linux/kasan.h:233 [inline] kmem_cache_alloc_trace+0x20b/0x400 mm/slab.c:3570 kmalloc include/linux/slab.h:554 [inline] tty_alloc_file drivers/tty/tty_io.c:186 [inline] tty_open+0x14a/0xf60 drivers/tty/tty_io.c:2145 chrdev_open+0x237/0x6a0 fs/char_dev.c:414 do_dentry_open+0x42d/0xf90 fs/open.c:826 vfs_open+0x9a/0xc0 fs/open.c:940 do_open fs/namei.c:3365 [inline] path_openat+0xa73/0x25d0 fs/namei.c:3498 do_filp_open+0x17d/0x3b0 fs/namei.c:3525 do_sys_openat2+0x120/0x3d0 fs/open.c:1187 do_sys_open fs/open.c:1203 [inline] __do_sys_openat fs/open.c:1219 [inline] __se_sys_openat fs/open.c:1214 [inline] __x64_sys_openat+0x124/0x200 fs/open.c:1214 do_syscall_64+0x32/0x80 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x462209 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f2a08f1d1a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 RAX: ffffffffffffffda RBX: 000000000052bf00 RCX: 0000000000462209 RDX: 0000000000000000 RSI: 0000000020000380 RDI: ffffffffffffff9c RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 00000000004f8a08 R14: 00000000004b41d5 R15: 00007f2a08f1d6bc NMI backtrace for cpu 0 CPU: 0 PID: 6981 Comm: syz-executor.5 Not tainted 5.12.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x114/0x14e lib/dump_stack.c:120 nmi_cpu_backtrace.cold+0x30/0x9c lib/nmi_backtrace.c:105 nmi_trigger_cpumask_backtrace+0x120/0x170 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:39 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x1cc/0x252 kernel/rcu/tree_stall.h:341 print_cpu_stall kernel/rcu/tree_stall.h:622 [inline] check_cpu_stall kernel/rcu/tree_stall.h:697 [inline] rcu_pending kernel/rcu/tree.c:3830 [inline] rcu_sched_clock_irq.cold+0x67c/0x11b6 kernel/rcu/tree.c:2650 update_process_times+0x139/0x1b0 kernel/time/timer.c:1796 tick_sched_handle+0x77/0x140 kernel/time/tick-sched.c:226 tick_sched_timer+0x142/0x230 kernel/time/tick-sched.c:1369 __run_hrtimer kernel/time/hrtimer.c:1537 [inline] __hrtimer_run_queues+0x1b7/0xc30 kernel/time/hrtimer.c:1601 hrtimer_interrupt+0x2f3/0x950 kernel/time/hrtimer.c:1663 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline] __sysvec_apic_timer_interrupt+0x143/0x530 arch/x86/kernel/apic/apic.c:1106 sysvec_apic_timer_interrupt+0x98/0xc0 arch/x86/kernel/apic/apic.c:1100 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632 RIP: 0010:unwind_next_frame.part.0+0x123/0x750 arch/x86/kernel/unwind_frame.c:295 Code: 1d 02 00 00 49 8d 46 d8 49 39 c5 0f 84 02 03 00 00 48 8d 7b 50 48 b8 00 00 00 00 00 fc ff df 48 89 f9 48 c1 e9 03 80 3c 01 00 <0f> 85 62 04 00 00 4c 8b 73 50 4d 85 f6 0f 84 a3 02 00 00 48 c7 43 RSP: 0018:ffffc90003d97438 EFLAGS: 00000246 RAX: dffffc0000000000 RBX: ffffc90003d974a8 RCX: 1ffff920007b2e9f RDX: ffff88810890e400 RSI: ffffffff83b8122a RDI: ffffc90003d974f8 RBP: ffffc90003d97480 R08: 1ffff920007b2e9d R09: ffffc90003d974d0 R10: ffffc90003d974e0 R11: 0000000000008400 R12: 0000000000000000 R13: ffffc90003d97868 R14: ffffc90003d97f58 R15: ffffc90003d97500 unwind_next_frame+0x32/0x50 arch/x86/kernel/unwind_frame.c:259 arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25 stack_trace_save+0x8f/0xc0 kernel/stacktrace.c:121 kasan_save_stack+0x23/0x50 mm/kasan/common.c:38 kasan_set_track mm/kasan/common.c:46 [inline] set_alloc_info mm/kasan/common.c:427 [inline] ____kasan_kmalloc mm/kasan/common.c:506 [inline] __kasan_kmalloc+0x7f/0xa0 mm/kasan/common.c:515 kasan_kmalloc include/linux/kasan.h:233 [inline] kmem_cache_alloc_trace+0x20b/0x400 mm/slab.c:3570 kmalloc include/linux/slab.h:554 [inline] tty_alloc_file drivers/tty/tty_io.c:186 [inline] tty_open+0x14a/0xf60 drivers/tty/tty_io.c:2145 chrdev_open+0x237/0x6a0 fs/char_dev.c:414 do_dentry_open+0x42d/0xf90 fs/open.c:826 vfs_open+0x9a/0xc0 fs/open.c:940 do_open fs/namei.c:3365 [inline] path_openat+0xa73/0x25d0 fs/namei.c:3498 do_filp_open+0x17d/0x3b0 fs/namei.c:3525 do_sys_openat2+0x120/0x3d0 fs/open.c:1187 do_sys_open fs/open.c:1203 [inline] __do_sys_openat fs/open.c:1219 [inline] __se_sys_openat fs/open.c:1214 [inline] __x64_sys_openat+0x124/0x200 fs/open.c:1214 do_syscall_64+0x32/0x80 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x462209 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f2a08f1d1a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 RAX: ffffffffffffffda RBX: 000000000052bf00 RCX: 0000000000462209 RDX: 0000000000000000 RSI: 0000000020000380 RDI: ffffffffffffff9c RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 00000000004f8a08 R14: 00000000004b41d5 R15: 00007f2a08f1d6bc sched: RT throttling activated BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 198s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=14/256 refcnt=15 in-flight: 5895:nsim_fib_event_work, 37:nsim_dev_trap_report_work, 6898:nsim_fib_event_work pending: nsim_dev_trap_report_work, psi_avgs_work, rht_deferred_worker, psi_avgs_work, ovs_dp_masks_rebalance, psi_avgs_work, cache_reap, psi_avgs_work, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=19/256 refcnt=20 in-flight: 1315:nsim_fib_event_work, 6900:delayed_fput, 6887:nsim_fib_event_work nsim_fib_event_work pending: linkwatch_event, nsim_dev_trap_report_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, perf_sched_delayed, psi_avgs_work, ovs_dp_masks_rebalance, ovs_dp_masks_rebalance, psi_avgs_work, ovs_dp_masks_rebalance, psi_avgs_work, check_corruption, vmstat_shepherd, cache_reap workqueue events_long: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: defense_work_handler, defense_work_handler, defense_work_handler workqueue events_freezable: flags=0x4 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: update_balloon_stats_func workqueue events_power_efficient: flags=0x80 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: gc_worker, neigh_periodic_work pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 pending: fb_flashcursor, reg_check_chans_work, check_lifetime, neigh_periodic_work workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: vmstat_update pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: vmstat_update workqueue cgroup_pidlist_destroy: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 pending: cgroup_pidlist_destroy_work_fn workqueue writeback: flags=0x4a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 in-flight: 24:wb_workfn workqueue dm_bufio_cache: flags=0x8 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: work_fn workqueue ipv6_addrconf: flags=0x40008 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=87 pending: addrconf_dad_work delayed: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work , addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=62 pending: addrconf_dad_work delayed: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work , addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_verify_work workqueue bat_events: flags=0xe000a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=38 in-flight: 5827:batadv_nc_worker delayed: batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_mcast_mla_update, batadv_purge_orig, batadv_mcast_mla_update, batadv_purge_orig, batadv_purge_orig, batadv_purge_orig, batadv_tt_purge, batadv_tt_purge, batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_tt_purge, batadv_purge_orig, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_mcast_mla_update, batadv_purge_orig, batadv_tt_purge, batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=192s workers=9 idle: 6873 5 3437 6907 6875 11 pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=198s workers=10 idle: 6862 6905 6945 6944 2452 6946 19 pool 4: cpus=0-1 flags=0x4 nice=0 hung=192s workers=6 idle: 133 176 7 36