bisecting fixing commit since ae4b064e2a616b545acf02b8f50cc513b32c7522
building syzkaller on 08003f6440deafc4e193b159c4acece64f7864b1
testing commit ae4b064e2a616b545acf02b8f50cc513b32c7522 with gcc (GCC) 8.4.1 20210217
kernel signature: 79450ac766e6ee1c6d674c4246108ce8e17a02768744d4e7b53c5a30c04b9f53
run #0: crashed: BUG: workqueue lockup
run #1: crashed: BUG: workqueue lockup
run #2: crashed: BUG: workqueue lockup
run #3: crashed: BUG: workqueue lockup
run #4: crashed: BUG: workqueue lockup
run #5: crashed: BUG: workqueue lockup
run #6: crashed: BUG: workqueue lockup
run #7: crashed: INFO: rcu detected stall in chrdev_open
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in chrdev_open
run #10: crashed: INFO: rcu detected stall in chrdev_open
run #11: crashed: INFO: rcu detected stall in corrupted
run #12: crashed: INFO: rcu detected stall in corrupted
run #13: crashed: INFO: rcu detected stall in chrdev_open
run #14: crashed: INFO: rcu detected stall in chrdev_open
run #15: crashed: INFO: rcu detected stall in chrdev_open
run #16: crashed: INFO: rcu detected stall in corrupted
run #17: crashed: INFO: rcu detected stall in chrdev_open
run #18: crashed: INFO: rcu detected stall in chrdev_open
run #19: crashed: BUG: workqueue lockup
testing current HEAD fe07bfda2fb9cdef8a4d4008a409bb02f35f1bd8
testing commit fe07bfda2fb9cdef8a4d4008a409bb02f35f1bd8 with gcc (GCC) 10.2.1 20210217
kernel signature: 54899f2a395fba70dcf8d622faed9eead13c6ede1a0d5ea65558b56bd6f44fc4
run #0: crashed: BUG: workqueue lockup
run #1: crashed: BUG: workqueue lockup
run #2: crashed: INFO: rcu detected stall in chrdev_open
run #3: crashed: INFO: rcu detected stall in chrdev_open
run #4: crashed: BUG: workqueue lockup
run #5: crashed: BUG: workqueue lockup
run #6: crashed: BUG: workqueue lockup
run #7: crashed: BUG: workqueue lockup
run #8: crashed: INFO: rcu detected stall in chrdev_open
run #9: OK
revisions tested: 2, total time: 33m11.503073407s (build: 12m19.32380769s, test: 19m55.091140194s)
the crash still happens on HEAD
commit msg: Linux 5.12-rc1
crash: INFO: rcu detected stall in chrdev_open
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	0-...!: (9650 ticks this GP) idle=542/1/0x4000000000000000 softirq=41533/41533 fqs=0 
	(t=10500 jiffies g=68813 q=43)
rcu: rcu_preempt kthread starved for 10500 jiffies! g68813 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:28744 pid:   13 ppid:     2 flags:0x00004000
Call Trace:
 context_switch kernel/sched/core.c:4324 [inline]
 __schedule+0xf86/0x2180 kernel/sched/core.c:5075
 preempt_schedule_common+0x4a/0xe0 kernel/sched/core.c:5235
 preempt_schedule+0x4b/0x60 kernel/sched/core.c:5260
 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:35
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
 _raw_spin_unlock_irqrestore+0x79/0x8b kernel/locking/spinlock.c:191
 prepare_to_swait_event+0x152/0x5b0 kernel/sched/swait.c:120
 rcu_gp_fqs_loop kernel/rcu/tree.c:2005 [inline]
 rcu_gp_kthread+0xd0d/0x2200 kernel/rcu/tree.c:2178
 kthread+0x37d/0x460 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 18890 Comm: syz-executor047 Not tainted 5.12.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:rcu_lockdep_current_cpu_online kernel/rcu/tree.c:1219 [inline]
RIP: 0010:rcu_lockdep_current_cpu_online+0x89/0x150 kernel/rcu/tree.c:1208
Code: 00 00 00 48 03 1c c5 20 d9 3f 89 48 b8 00 00 00 00 00 fc ff df 48 8d 7b 18 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 90 00 00 00 <48> 8d 7b 20 4c 8b 63 18 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48
RSP: 0000:ffffc90000178dc0 EFLAGS: 00000046
RAX: dffffc0000000000 RBX: ffff8881f6737240 RCX: 1ffffffff127fb25
RDX: 1ffff1103ece6e4b RSI: ffffffff8850b0a0 RDI: ffff8881f6737258
RBP: ffffc90000178dd8 R08: 0000000000000000 R09: ffffffff8a7f1a0f
R10: fffffbfff14fe341 R11: 0000000000000001 R12: 0000000000000001
R13: ffff8881f6727280 R14: ffff8881f6727300 R15: dffffc0000000000
FS:  00007feebfcb3700(0000) GS:ffff8881f6700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffc183e0970 CR3: 000000010bc85003 CR4: 00000000001706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 rcu_read_lock_held_common kernel/rcu/update.c:112 [inline]
 rcu_read_lock_held_common kernel/rcu/update.c:102 [inline]
 rcu_read_lock_sched_held+0x2c/0xb0 kernel/rcu/update.c:123
 trace_hrtimer_expire_entry include/trace/events/timer.h:232 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1516 [inline]
 __hrtimer_run_queues+0xa01/0xc30 kernel/time/hrtimer.c:1583
 hrtimer_interrupt+0x2f6/0x900 kernel/time/hrtimer.c:1645
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x143/0x530 arch/x86/kernel/apic/apic.c:1106
 sysvec_apic_timer_interrupt+0x98/0xc0 arch/x86/kernel/apic/apic.c:1100
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
RIP: 0010:kfree+0x240/0x2d0 mm/slab.c:3798
Code: 89 e7 e8 73 01 dd 01 48 8b 55 08 e9 df fe ff ff e8 c5 f0 c9 ff 48 83 3d bd e4 cb 07 00 0f 84 90 00 00 00 fb 66 0f 1f 44 00 00 <5b> 41 5c 41 5d 41 5e 5d c3 0f 0b 0f 0b e9 d5 fd ff ff 0f 0b 48 8b
RSP: 0000:ffffc900007ff848 EFLAGS: 00000286
RAX: 0000000000d4bb41 RBX: 0000000000000200 RCX: 1ffffffff14fdba1
RDX: dffffc0000000000 RSI: ffffffff880b2420 RDI: ffffffff8850b120
RBP: ffffc900007ff868 R08: 0000000000000001 R09: 0000000000000001
R10: ffff8881265b2e88 R11: 0000000000006800 R12: ffff88810f6cf740
R13: ffff888100040100 R14: ffffffff83b6a5a6 R15: 00000000fffffff5
 tty_free_file drivers/tty/tty_io.c:219 [inline]
 tty_open+0x676/0xf60 drivers/tty/tty_io.c:2155
 chrdev_open+0x237/0x6a0 fs/char_dev.c:414
 do_dentry_open+0x42d/0xf90 fs/open.c:826
 vfs_open+0x9a/0xc0 fs/open.c:940
 do_open fs/namei.c:3365 [inline]
 path_openat+0xa73/0x25d0 fs/namei.c:3498
 do_filp_open+0x17d/0x3b0 fs/namei.c:3525
 do_sys_openat2+0x120/0x3d0 fs/open.c:1187
 do_sys_open fs/open.c:1203 [inline]
 __do_sys_openat fs/open.c:1219 [inline]
 __se_sys_openat fs/open.c:1214 [inline]
 __x64_sys_openat+0x124/0x200 fs/open.c:1214
 do_syscall_64+0x32/0x80 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x445919
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 11 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007feebfcb3318 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00000000004ca408 RCX: 0000000000445919
RDX: 0000000000000000 RSI: 0000000020000380 RDI: ffffffffffffff9c
RBP: 00000000004ca400 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 7974742f7665642f
R13: 00007ffc1831ffbf R14: 00007feebfcb3400 R15: 0000000000022000
NMI backtrace for cpu 0
CPU: 0 PID: 18895 Comm: syz-executor047 Not tainted 5.12.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0x114/0x14e lib/dump_stack.c:120
 nmi_cpu_backtrace.cold+0x30/0x9c lib/nmi_backtrace.c:105
 nmi_trigger_cpumask_backtrace+0x120/0x170 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:39
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x1cc/0x252 kernel/rcu/tree_stall.h:341
 print_cpu_stall kernel/rcu/tree_stall.h:622 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:697 [inline]
 rcu_pending kernel/rcu/tree.c:3830 [inline]
 rcu_sched_clock_irq.cold+0x67c/0x11b6 kernel/rcu/tree.c:2650
 update_process_times+0x139/0x1b0 kernel/time/timer.c:1796
 tick_sched_handle+0x77/0x140 kernel/time/tick-sched.c:226
 tick_sched_timer+0x142/0x230 kernel/time/tick-sched.c:1369
 __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
 __hrtimer_run_queues+0x1b7/0xc30 kernel/time/hrtimer.c:1583
 hrtimer_interrupt+0x2f6/0x900 kernel/time/hrtimer.c:1645
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
 __sysvec_apic_timer_interrupt+0x143/0x530 arch/x86/kernel/apic/apic.c:1106
 sysvec_apic_timer_interrupt+0x98/0xc0 arch/x86/kernel/apic/apic.c:1100
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
RIP: 0010:lock_acquire+0x0/0x8c0 kernel/locking/lockdep.c:5478
Code: ff 48 c7 c7 a8 4b 7f 8a e8 5d 65 53 00 e9 64 fe ff ff 48 c7 c7 a8 4b 7f 8a e8 4c 65 53 00 e9 44 fd ff ff 0f 1f 80 00 00 00 00 <48> b8 00 00 00 00 00 fc ff df 55 48 89 e5 41 57 41 56 4c 8d 7d d8
RSP: 0018:ffffc900009b77f0 EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000080 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff899726e0
RBP: ffffc900009b7810 R08: 0000000000000001 R09: 0000000000000000
R10: fffffbfff165057e R11: 0000000000008c00 R12: ffff88811b80e080
R13: 0000000000000cc0 R14: 0000000000000020 R15: 00000000fffffff5
 might_alloc include/linux/sched/mm.h:193 [inline]
 slab_pre_alloc_hook mm/slab.h:497 [inline]
 slab_alloc mm/slab.c:3306 [inline]
 kmem_cache_alloc_trace+0x38/0x400 mm/slab.c:3568
 kmalloc include/linux/slab.h:554 [inline]
 tty_alloc_file drivers/tty/tty_io.c:186 [inline]
 tty_open+0x14a/0xf60 drivers/tty/tty_io.c:2146
 chrdev_open+0x237/0x6a0 fs/char_dev.c:414
 do_dentry_open+0x42d/0xf90 fs/open.c:826
 vfs_open+0x9a/0xc0 fs/open.c:940
 do_open fs/namei.c:3365 [inline]
 path_openat+0xa73/0x25d0 fs/namei.c:3498
 do_filp_open+0x17d/0x3b0 fs/namei.c:3525
 do_sys_openat2+0x120/0x3d0 fs/open.c:1187
 do_sys_open fs/open.c:1203 [inline]
 __do_sys_openat fs/open.c:1219 [inline]
 __se_sys_openat fs/open.c:1214 [inline]
 __x64_sys_openat+0x124/0x200 fs/open.c:1214
 do_syscall_64+0x32/0x80 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x445919
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 11 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007feebfcb3318 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00000000004ca408 RCX: 0000000000445919
RDX: 0000000000000000 RSI: 0000000020000380 RDI: ffffffffffffff9c
RBP: 00000000004ca400 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 7974742f7665642f
R13: 00007ffc1831ffbf R14: 00007feebfcb3400 R15: 0000000000022000
sched: RT throttling activated