BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 143s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=76/256 refcnt=77
pending: nsim_dev_trap_report_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, nsim_dev_trap_report_work, defense_work_handler, defense_work_handler, defense_work_handler, vmstat_shepherd, cache_reap, defense_work_handler, psi_avgs_work, psi_avgs_work, psi_avgs_work, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, check_corruption, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred
, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred, rt6_probe_deferred
======================================================
WARNING: possible circular locking dependency detected
5.4.0-rc7-next-20191115 #0 Not tainted
------------------------------------------------------
syz-executor.1/11333 is trying to acquire lock:
ffffffff88fa2080 (console_owner){-.-.}, at: log_next kernel/printk/printk.c:516 [inline]
ffffffff88fa2080 (console_owner){-.-.}, at: console_unlock+0x415/0xf00 kernel/printk/printk.c:2460
but task is already holding lock:
ffff8880ae836b58 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state kernel/workqueue.c:4767 [inline]
ffff8880ae836b58 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold+0x156/0x802 kernel/workqueue.c:4740
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #4 (&(&pool->lock)->rlock){-.-.}:
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
__queue_work+0x285/0x1280 kernel/workqueue.c:1444
queue_work_on+0x19f/0x210 kernel/workqueue.c:1513
queue_work include/linux/workqueue.h:494 [inline]
schedule_work include/linux/workqueue.h:552 [inline]
put_pwq kernel/workqueue.c:1113 [inline]
put_pwq+0x178/0x1d0 kernel/workqueue.c:1098
put_pwq_unlocked.part.0+0x34/0x70 kernel/workqueue.c:1130
put_pwq_unlocked kernel/workqueue.c:1124 [inline]
apply_wqattrs_cleanup.part.0+0xf6/0x160 kernel/workqueue.c:3878
apply_wqattrs_cleanup kernel/workqueue.c:4017 [inline]
apply_workqueue_attrs_locked+0xeb/0x140 kernel/workqueue.c:4015
apply_workqueue_attrs+0x31/0x50 kernel/workqueue.c:4046
padata_setup_cpumasks kernel/padata.c:365 [inline]
padata_alloc_pd+0x298/0xb60 kernel/padata.c:436
padata_alloc kernel/padata.c:996 [inline]
padata_alloc_possible+0x1b6/0x480 kernel/padata.c:1042
pcrypt_init_padata+0x20/0x105 crypto/pcrypt.c:311
pcrypt_init+0x76/0x11b crypto/pcrypt.c:342
do_one_initcall+0x120/0x81a init/main.c:938
do_initcall_level init/main.c:1006 [inline]
do_initcalls init/main.c:1014 [inline]
do_basic_setup init/main.c:1031 [inline]
kernel_init_freeable+0x4ca/0x5b9 init/main.c:1191
kernel_init+0x12/0x1bf init/main.c:1109
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
-> #3 (&pool->lock/1){..-.}:
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
__queue_work+0x285/0x1280 kernel/workqueue.c:1444
queue_work_on+0x19f/0x210 kernel/workqueue.c:1513
queue_work include/linux/workqueue.h:494 [inline]
tty_schedule_flip drivers/tty/tty_buffer.c:413 [inline]
tty_flip_buffer_push+0xc5/0x100 drivers/tty/tty_buffer.c:556
pty_write+0x1a6/0x200 drivers/tty/pty.c:125
n_tty_write+0xb1d/0x1080 drivers/tty/n_tty.c:2356
do_tty_write drivers/tty/tty_io.c:962 [inline]
tty_write+0x496/0x7f0 drivers/tty/tty_io.c:1046
__vfs_write+0x8a/0x110 fs/read_write.c:494
vfs_write+0x268/0x5d0 fs/read_write.c:558
ksys_write+0x14f/0x290 fs/read_write.c:611
__do_sys_write fs/read_write.c:623 [inline]
__se_sys_write fs/read_write.c:620 [inline]
__x64_sys_write+0x73/0xb0 fs/read_write.c:620
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #2 (&(&port->lock)->rlock){-.-.}:
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x95/0xcd kernel/locking/spinlock.c:159
tty_port_tty_get+0x22/0x80 drivers/tty/tty_port.c:287
tty_port_default_wakeup+0x16/0x40 drivers/tty/tty_port.c:47
tty_port_tty_wakeup+0x57/0x70 drivers/tty/tty_port.c:387
uart_write_wakeup+0x46/0x70 drivers/tty/serial/serial_core.c:104
serial8250_tx_chars+0x495/0xaf0 drivers/tty/serial/8250/8250_port.c:1761
serial8250_handle_irq.part.0+0x2a2/0x330 drivers/tty/serial/8250/8250_port.c:1834
serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1820 [inline]
serial8250_default_handle_irq+0xc0/0x150 drivers/tty/serial/8250/8250_port.c:1850
serial8250_interrupt+0xf1/0x1a0 drivers/tty/serial/8250/8250_core.c:126
__handle_irq_event_percpu+0x15d/0x970 kernel/irq/handle.c:149
handle_irq_event_percpu+0x74/0x160 kernel/irq/handle.c:189
handle_irq_event+0xa7/0x134 kernel/irq/handle.c:206
handle_edge_irq+0x25e/0x8d0 kernel/irq/chip.c:830
generic_handle_irq_desc include/linux/irqdesc.h:156 [inline]
do_IRQ+0xde/0x280 arch/x86/kernel/irq.c:250
ret_from_intr+0x0/0x36
arch_local_irq_restore arch/x86/include/asm/paravirt.h:752 [inline]
__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
_raw_spin_unlock_irqrestore+0x90/0xe0 kernel/locking/spinlock.c:191
spin_unlock_irqrestore include/linux/spinlock.h:393 [inline]
uart_write+0x3b6/0x6f0 drivers/tty/serial/serial_core.c:613
process_output_block drivers/tty/n_tty.c:595 [inline]
n_tty_write+0x40e/0x1080 drivers/tty/n_tty.c:2333
do_tty_write drivers/tty/tty_io.c:962 [inline]
tty_write+0x496/0x7f0 drivers/tty/tty_io.c:1046
redirected_tty_write+0xb2/0xc0 drivers/tty/tty_io.c:1067
__vfs_write+0x8a/0x110 fs/read_write.c:494
vfs_write+0x268/0x5d0 fs/read_write.c:558
ksys_write+0x14f/0x290 fs/read_write.c:611
__do_sys_write fs/read_write.c:623 [inline]
__se_sys_write fs/read_write.c:620 [inline]
__x64_sys_write+0x73/0xb0 fs/read_write.c:620
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #1 (&port_lock_key){-.-.}:
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x95/0xcd kernel/locking/spinlock.c:159
serial8250_console_write+0x253/0x9a0 drivers/tty/serial/8250/8250_port.c:3142
univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:587
call_console_drivers kernel/printk/printk.c:1791 [inline]
console_unlock+0xb7a/0xf00 kernel/printk/printk.c:2473
vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
printk+0xba/0xed kernel/printk/printk.c:2056
register_console+0x745/0xb50 kernel/printk/printk.c:2798
univ8250_console_init+0x3e/0x4b drivers/tty/serial/8250/8250_core.c:682
console_init+0x461/0x67b kernel/printk/printk.c:2884
start_kernel+0x653/0x943 init/main.c:712
x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:490
x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:471
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242
-> #0 (console_owner){-.-.}:
check_prev_add kernel/locking/lockdep.c:2476 [inline]
check_prevs_add kernel/locking/lockdep.c:2581 [inline]
validate_chain kernel/locking/lockdep.c:2971 [inline]
__lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4485
console_lock_spinning_enable kernel/printk/printk.c:1654 [inline]
console_unlock+0x47f/0xf00 kernel/printk/printk.c:2470
vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
printk+0xba/0xed kernel/printk/printk.c:2056
show_pwq+0x154/0x7cb kernel/workqueue.c:4673
show_workqueue_state kernel/workqueue.c:4769 [inline]
show_workqueue_state.cold+0x1a6/0x802 kernel/workqueue.c:4740
wq_watchdog_timer_fn+0x511/0x590 kernel/workqueue.c:5783
call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1404
expire_timers kernel/time/timer.c:1449 [inline]
__run_timers kernel/time/timer.c:1773 [inline]
__run_timers kernel/time/timer.c:1740 [inline]
run_timer_softirq+0xdca/0x1790 kernel/time/timer.c:1788
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
rep_nop arch/x86/include/asm/processor.h:674 [inline]
cpu_relax arch/x86/include/asm/processor.h:679 [inline]
irq_work_sync+0xd3/0x1d0 kernel/irq_work.c:194
_free_event+0x89/0x13b0 kernel/events/core.c:4584
put_event+0x47/0x60 kernel/events/core.c:4709
perf_event_release_kernel+0x6d5/0xd70 kernel/events/core.c:4824
perf_release+0x37/0x50 kernel/events/core.c:4834
__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x904/0x2e60 kernel/exit.c:817
do_group_exit+0x135/0x360 kernel/exit.c:921
get_signal+0x47c/0x24f0 kernel/signal.c:2734
do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
entry_SYSCALL_64_after_hwframe+0x49/0xbe
other info that might help us debug this:
Chain exists of:
console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&pool->lock)->rlock);
lock(&pool->lock/1);
lock(&(&pool->lock)->rlock);
lock(console_owner);
*** DEADLOCK ***
4 locks held by syz-executor.1/11333:
#0: ffff8880ae809d50 ((&wq_watchdog_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:172 [inline]
#0: ffff8880ae809d50 ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0xe0/0x780 kernel/time/timer.c:1394
#1: ffffffff88faccc0 (rcu_read_lock){....}, at: show_workqueue_state+0x0/0x120 kernel/workqueue.c:4638
#2: ffff8880ae836b58 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state kernel/workqueue.c:4767 [inline]
#2: ffff8880ae836b58 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold+0x156/0x802 kernel/workqueue.c:4740
#3: ffffffff88fa22e0 (console_lock){+.+.}, at: console_trylock_spinning kernel/printk/printk.c:1716 [inline]
#3: ffffffff88fa22e0 (console_lock){+.+.}, at: vprintk_emit+0x283/0x700 kernel/printk/printk.c:1995
stack backtrace:
CPU: 0 PID: 11333 Comm: syz-executor.1 Not tainted 5.4.0-rc7-next-20191115 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
print_circular_bug.isra.0.cold+0x163/0x172 kernel/locking/lockdep.c:1685
check_noncircular+0x32e/0x3e0 kernel/locking/lockdep.c:1809
check_prev_add kernel/locking/lockdep.c:2476 [inline]
check_prevs_add kernel/locking/lockdep.c:2581 [inline]
validate_chain kernel/locking/lockdep.c:2971 [inline]
__lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4485
console_lock_spinning_enable kernel/printk/printk.c:1654 [inline]
console_unlock+0x47f/0xf00 kernel/printk/printk.c:2470
vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
printk+0xba/0xed kernel/printk/printk.c:2056
show_pwq+0x154/0x7cb kernel/workqueue.c:4673
show_workqueue_state kernel/workqueue.c:4769 [inline]
show_workqueue_state.cold+0x1a6/0x802 kernel/workqueue.c:4740
wq_watchdog_timer_fn+0x511/0x590 kernel/workqueue.c:5783
call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1404
expire_timers kernel/time/timer.c:1449 [inline]
__run_timers kernel/time/timer.c:1773 [inline]
__run_timers kernel/time/timer.c:1740 [inline]
run_timer_softirq+0xdca/0x1790 kernel/time/timer.c:1788
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
RIP: 0010:atomic_read include/asm-generic/atomic-instrumented.h:26 [inline]
RIP: 0010:irq_work_sync+0xd3/0x1d0 kernel/irq_work.c:193
Code: f5 ff 4d 89 e6 4d 89 e5 48 b8 00 00 00 00 00 fc ff df 49 c1 ee 03 41 83 e5 07 49 01 c6 41 83 c5 03 eb 07 e8 7f 87 f5 ff f3 90 78 87 f5 ff be 04 00 00 00 4c 89 e7 e8 ab a6 31 00 41 0f b6 06
RSP: 0018:ffff888055f67858 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: ffff888055e0e0c0 RBX: 0000000000000002 RCX: ffffffff817e51f6
RDX: 0000000000000000 RSI: ffffffff817e51c1 RDI: 0000000000000005
RBP: ffff888055f67878 R08: ffff888055e0e0c0 R09: ffffed101299bc82
R10: ffffed101299bc81 R11: ffff888094cde40b R12: ffff888094cde408
R13: 0000000000000003 R14: ffffed101299bc81 R15: ffff888055f67918
_free_event+0x89/0x13b0 kernel/events/core.c:4584
put_event+0x47/0x60 kernel/events/core.c:4709
perf_event_release_kernel+0x6d5/0xd70 kernel/events/core.c:4824
perf_release+0x37/0x50 kernel/events/core.c:4834
__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16
Lost 33 message(s)!
workqueue events_highpri: flags=0x10
pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 refcnt=3
pending: flush_backlog BAR(2862)
workqueue events_power_efficient: flags=0x80
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=4/256 refcnt=5
pending: gc_worker, do_cache_clean, device_uncache_fw_images_work, neigh_periodic_work
workqueue mm_percpu_wq: flags=0x8
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
pending: vmstat_update
workqueue netns: flags=0xe000a
pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=4
in-flight: 2862:cleanup_net
workqueue ipv6_addrconf: flags=0x40008
pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2
pending: addrconf_verify_work
pool 4: cpus=0-1 flags=0x4 nice=0 hung=0s workers=8 idle: 271 10705 11342 7 100 21 388