BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 183s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 160s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 in-flight: 5715:cache_reap ====================================================== WARNING: possible circular locking dependency detected 4.19.0-rc5-next-20180924+ #78 Not tainted ------------------------------------------------------ ksoftirqd/0/9 is trying to acquire lock: 00000000728635e3 (console_owner){-.-.}, at: log_next kernel/printk/printk.c:496 [inline] 00000000728635e3 (console_owner){-.-.}, at: console_unlock+0x55d/0x1150 kernel/printk/printk.c:2391 but task is already holding lock: 00000000515650c9 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.51+0xad0/0x159a kernel/workqueue.c:4557 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #4 (&(&pool->lock)->rlock){-.-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x34c/0x1440 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] schedule_work include/linux/workqueue.h:570 [inline] put_pwq+0x175/0x1c0 kernel/workqueue.c:1090 put_pwq_unlocked.part.28+0x34/0x70 kernel/workqueue.c:1107 put_pwq_unlocked kernel/workqueue.c:1101 [inline] destroy_workqueue+0x868/0x9c0 kernel/workqueue.c:4217 ucma_close+0x276/0x310 drivers/infiniband/core/ucma.c:1776 __fput+0x3bc/0xa70 fs/file_table.c:279 ____fput+0x15/0x20 fs/file_table.c:312 task_work_run+0x1e8/0x2a0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:188 [inline] exit_to_usermode_loop+0x318/0x380 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x6be/0x820 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #3 (&pool->lock/1){..-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x34c/0x1440 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] tty_schedule_flip+0x14c/0x1d0 drivers/tty/tty_buffer.c:413 tty_flip_buffer_push+0x15/0x20 drivers/tty/tty_buffer.c:552 pty_write+0x19d/0x1f0 drivers/tty/pty.c:124 n_tty_write+0xc5b/0x11a0 drivers/tty/n_tty.c:2340 do_tty_write drivers/tty/tty_io.c:958 [inline] tty_write+0x3f1/0x880 drivers/tty/tty_io.c:1042 __vfs_write+0x119/0x9f0 fs/read_write.c:485 vfs_write+0x1fc/0x560 fs/read_write.c:549 ksys_write+0x101/0x260 fs/read_write.c:598 __do_sys_write fs/read_write.c:610 [inline] __se_sys_write fs/read_write.c:607 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:607 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #2 (&(&port->lock)->rlock){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x99/0xd0 kernel/locking/spinlock.c:152 tty_port_tty_get+0x20/0x80 drivers/tty/tty_port.c:287 tty_port_default_wakeup+0x15/0x40 drivers/tty/tty_port.c:47 tty_port_tty_wakeup+0x5d/0x70 drivers/tty/tty_port.c:387 uart_write_wakeup+0x44/0x60 drivers/tty/serial/serial_core.c:103 serial8250_tx_chars+0x4be/0xb60 drivers/tty/serial/8250/8250_port.c:1825 serial8250_handle_irq.part.23+0x1ee/0x280 drivers/tty/serial/8250/8250_port.c:1898 serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1884 [inline] serial8250_default_handle_irq+0xc8/0x150 drivers/tty/serial/8250/8250_port.c:1914 serial8250_interrupt+0x104/0x200 drivers/tty/serial/8250/8250_core.c:125 __handle_irq_event_percpu+0x1e1/0xaf0 kernel/irq/handle.c:149 handle_irq_event_percpu+0xa0/0x1d0 kernel/irq/handle.c:189 handle_irq_event+0xa7/0x135 kernel/irq/handle.c:206 handle_edge_irq+0x227/0x880 kernel/irq/chip.c:791 generic_handle_irq_desc include/linux/irqdesc.h:154 [inline] handle_irq+0x252/0x3d8 arch/x86/kernel/irq_64.c:78 do_IRQ+0x98/0x1c0 arch/x86/kernel/irq.c:246 ret_from_intr+0x0/0x1e arch_local_irq_restore arch/x86/include/asm/paravirt.h:761 [inline] __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline] _raw_spin_unlock_irqrestore+0xaf/0xd0 kernel/locking/spinlock.c:184 spin_unlock_irqrestore include/linux/spinlock.h:384 [inline] uart_write+0x4b2/0x740 drivers/tty/serial/serial_core.c:598 process_output_block drivers/tty/n_tty.c:580 [inline] n_tty_write+0x6c1/0x11a0 drivers/tty/n_tty.c:2317 do_tty_write drivers/tty/tty_io.c:958 [inline] tty_write+0x3f1/0x880 drivers/tty/tty_io.c:1042 redirected_tty_write+0xaf/0xc0 drivers/tty/tty_io.c:1063 __vfs_write+0x119/0x9f0 fs/read_write.c:485 vfs_write+0x1fc/0x560 fs/read_write.c:549 ksys_write+0x101/0x260 fs/read_write.c:598 __do_sys_write fs/read_write.c:610 [inline] __se_sys_write fs/read_write.c:607 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:607 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #1 (&port_lock_key){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x99/0xd0 kernel/locking/spinlock.c:152 serial8250_console_write+0x8dd/0xb10 drivers/tty/serial/8250/8250_port.c:3266 univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:586 call_console_drivers kernel/printk/printk.c:1723 [inline] console_unlock+0xaa5/0x1150 kernel/printk/printk.c:2404 vprintk_emit+0x33d/0x930 kernel/printk/printk.c:1921 vprintk_default+0x28/0x30 kernel/printk/printk.c:1962 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1995 register_console+0x7e6/0xc00 kernel/printk/printk.c:2719 univ8250_console_init+0x3f/0x4b drivers/tty/serial/8250/8250_core.c:681 console_init+0x6a2/0x9cd kernel/printk/printk.c:2805 start_kernel+0x5db/0x8f7 init/main.c:661 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:470 x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:451 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 -> #0 (console_owner){-.-.}: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3900 console_lock_spinning_enable kernel/printk/printk.c:1586 [inline] console_unlock+0x5ca/0x1150 kernel/printk/printk.c:2401 vprintk_emit+0x33d/0x930 kernel/printk/printk.c:1921 vprintk_default+0x28/0x30 kernel/printk/printk.c:1962 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1995 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.51+0xc6c/0x159a kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x6ea/0x810 kernel/workqueue.c:5571 call_timer_fn+0x26d/0x920 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7e0/0xc60 kernel/time/timer.c:1682 run_timer_softirq+0x88/0xb0 kernel/time/timer.c:1697 __do_softirq+0x30b/0xb03 kernel/softirq.c:292 run_ksoftirqd+0x94/0x100 kernel/softirq.c:654 smpboot_thread_fn+0x68b/0xa00 kernel/smpboot.c:164 kthread+0x35a/0x440 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 other info that might help us debug this: Chain exists of: console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&(&pool->lock)->rlock); lock(&pool->lock/1); lock(&(&pool->lock)->rlock); lock(console_owner); *** DEADLOCK *** 4 locks held by ksoftirqd/0/9: #0: 000000004020ac7e ((&wq_watchdog_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:178 [inline] #0: 000000004020ac7e ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0x1db/0x920 kernel/time/timer.c:1316 #1: 000000005a1fce84 (rcu_read_lock_sched){....}, at: show_workqueue_state+0x0/0x1d0 kernel/workqueue.c:4423 #2: 00000000515650c9 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.51+0xad0/0x159a kernel/workqueue.c:4557 #3: 00000000d1a767a5 (console_lock){+.+.}, at: console_trylock_spinning kernel/printk/printk.c:1648 [inline] #3: 00000000d1a767a5 (console_lock){+.+.}, at: vprintk_emit+0x322/0x930 kernel/printk/printk.c:1920 stack backtrace: CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.0-rc5-next-20180924+ #78 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1d3/0x2c4 lib/dump_stack.c:113 print_circular_bug.isra.33.cold.54+0x1bd/0x27d kernel/locking/lockdep.c:1221 check_prev_add kernel/locking/lockdep.c:1861 [inline] check_prevs_add kernel/locking/lockdep.c:1974 [inline] validate_chain kernel/locking/lockdep.c:2415 [inline] __lock_acquire+0x33e4/0x4ec0 kernel/locking/lockdep.c:3411 lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3900 console_lock_spinning_enable kernel/printk/printk.c:1586 [inline] console_unlock+0x5ca/0x1150 kernel/printk/printk.c:2401 vprintk_emit+0x33d/0x930 kernel/printk/printk.c:1921 vprintk_default+0x28/0x30 kernel/printk/printk.c:1962 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1995 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.51+0xc6c/0x159a kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x6ea/0x810 kernel/workqueue.c:5571 call_timer_fn+0x26d/0x920 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7e0/0xc60 kernel/time/timer.c:1682 ? __sanitizer_cov_trace_const_cmp4+0x1 Lost 30 message(s)! pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=9/256 pending: kvmclock_update_fn, kvmclock_update_fn, vmstat_shepherd, cache_reap, defense_work_handler, defense_work_handler, defense_work_handler, kvmclock_update_fn, rht_deferred_worker workqueue events_power_efficient: flags=0x80 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 pending: neigh_periodic_work pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 pending: gc_worker, do_cache_clean, neigh_periodic_work workqueue mm_percpu_wq: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 pending: vmstat_update workqueue writeback: flags=0x4e pwq 4: cpus=0-1 flags=0x4 nice=0 active=7/256 in-flight: 7:wb_workfn pending: wb_workfn, wb_workfn, wb_workfn, wb_workfn, wb_workfn, wb_workfn workqueue kblockd: flags=0x18 pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 pending: blk_mq_timeout_work workqueue dm_bufio_cache: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 pending: work_fn pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=161s workers=4 idle: 17 22 5856 pool 4: cpus=0-1 flags=0x4 nice=0 hung=1s workers=6 idle: 29 523 21 7098 823 kobject: 'rx-0' (000000002480250e): kobject_cleanup, parent 00000000be4f46f4 kobject: 'rx-0' (000000002480250e): auto cleanup 'remove' event kobject: 'rx-0' (000000002480250e): kobject_uevent_env kobject: 'rx-0' (000000002480250e): fill_kobj_path: path = '/devices/virtual/net/syz_tun/queues/rx-0' kobject: 'rx-0' (000000002480250e): auto cleanup kobject_del kobject: 'rx-0' (000000002480250e): calling ktype release kobject: 'rx-0': free name kobject: 'tx-0' (000000005bf368ee): kobject_cleanup, parent 00000000be4f46f4 kobject: 'tx-0' (000000005bf368ee): auto cleanup 'remove' event kobject: 'tx-0' (000000005bf368ee): kobject_uevent_env kobject: 'tx-0' (000000005bf368ee): fill_kobj_path: path = '/devices/virtual/net/syz_tun/queues/tx-0' kobject: 'tx-0' (000000005bf368ee): auto cleanup kobject_del kobject: 'tx-0' (000000005bf368ee): calling ktype release kobject: 'tx-0': free name kobject: 'queues' (00000000be4f46f4): kobject_cleanup, parent (null) kobject: 'queues' (00000000be4f46f4): calling ktype release kobject: 'queues' (00000000be4f46f4): kset_release kobject: 'queues': free name kobject: 'syz_tun' (00000000dda1933a): kobject_uevent_env kobject: 'syz_tun' (00000000dda1933a): fill_kobj_path: path = '/devices/virtual/net