BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 273s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for 224s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=5/256 pending: free_obj_work, defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler ====================================================== WARNING: possible circular locking dependency detected 4.19.0-rc2+ #1 Not tainted ------------------------------------------------------ swapper/0/0 is trying to acquire lock: 000000004177d799 (console_owner){-.-.}, at: log_next kernel/printk/printk.c:498 [inline] 000000004177d799 (console_owner){-.-.}, at: console_unlock+0x7a7/0x10d0 kernel/printk/printk.c:2394 but task is already holding lock: 00000000123738d3 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.50+0xb16/0x15ec kernel/workqueue.c:4557 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #4 (&(&pool->lock)->rlock){-.-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x352/0x12f0 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] schedule_work include/linux/workqueue.h:570 [inline] put_pwq+0x175/0x1c0 kernel/workqueue.c:1090 put_pwq_unlocked.part.30+0x34/0x70 kernel/workqueue.c:1107 put_pwq_unlocked kernel/workqueue.c:1101 [inline] destroy_workqueue+0x880/0x9d0 kernel/workqueue.c:4217 tipc_topsrv_work_stop net/tipc/topsrv.c:641 [inline] tipc_topsrv_stop+0x4b1/0x6ce net/tipc/topsrv.c:700 tipc_exit_net+0x15/0x40 net/tipc/core.c:100 ops_exit_list.isra.7+0xb0/0x160 net/core/net_namespace.c:153 setup_net+0x56a/0x980 net/core/net_namespace.c:331 copy_net_ns+0x2b8/0x4d0 net/core/net_namespace.c:437 create_new_namespaces+0x6ad/0x900 kernel/nsproxy.c:107 unshare_nsproxy_namespaces+0xc3/0x1f0 kernel/nsproxy.c:206 ksys_unshare+0x723/0xfb0 kernel/fork.c:2490 __do_sys_unshare kernel/fork.c:2558 [inline] __se_sys_unshare kernel/fork.c:2556 [inline] __x64_sys_unshare+0x31/0x40 kernel/fork.c:2556 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #3 (&pool->lock/1){..-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x352/0x12f0 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] tty_schedule_flip+0x14c/0x1d0 drivers/tty/tty_buffer.c:408 tty_flip_buffer_push+0x15/0x20 drivers/tty/tty_buffer.c:547 pty_write+0x19d/0x1f0 drivers/tty/pty.c:124 n_tty_write+0xc5b/0x11a0 drivers/tty/n_tty.c:2340 do_tty_write drivers/tty/tty_io.c:958 [inline] tty_write+0x3f1/0x880 drivers/tty/tty_io.c:1042 __vfs_write+0x117/0x9d0 fs/read_write.c:485 vfs_write+0x1fc/0x560 fs/read_write.c:549 ksys_write+0x101/0x260 fs/read_write.c:598 __do_sys_write fs/read_write.c:610 [inline] __se_sys_write fs/read_write.c:607 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:607 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #2 (&(&port->lock)->rlock){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152 tty_port_tty_get+0x20/0x80 drivers/tty/tty_port.c:288 tty_port_default_wakeup+0x15/0x40 drivers/tty/tty_port.c:47 tty_port_tty_wakeup+0x5d/0x70 drivers/tty/tty_port.c:390 uart_write_wakeup+0x44/0x60 drivers/tty/serial/serial_core.c:103 serial8250_tx_chars+0x4be/0xb60 drivers/tty/serial/8250/8250_port.c:1806 serial8250_handle_irq.part.23+0x1ee/0x280 drivers/tty/serial/8250/8250_port.c:1879 serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1865 [inline] serial8250_default_handle_irq+0xc8/0x150 drivers/tty/serial/8250/8250_port.c:1895 serial8250_interrupt+0xfa/0x1d0 drivers/tty/serial/8250/8250_core.c:125 __handle_irq_event_percpu+0x1c8/0xa50 kernel/irq/handle.c:149 handle_irq_event_percpu+0xa0/0x1d0 kernel/irq/handle.c:189 handle_irq_event+0xa7/0x135 kernel/irq/handle.c:206 handle_edge_irq+0x20f/0x870 kernel/irq/chip.c:791 generic_handle_irq_desc include/linux/irqdesc.h:154 [inline] handle_irq+0x18c/0x2e7 arch/x86/kernel/irq_64.c:78 do_IRQ+0x80/0x1a0 arch/x86/kernel/irq.c:246 ret_from_intr+0x0/0x1e arch_local_irq_restore arch/x86/include/asm/paravirt.h:788 [inline] __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline] _raw_spin_unlock_irqrestore+0xa1/0xc0 kernel/locking/spinlock.c:184 spin_unlock_irqrestore include/linux/spinlock.h:384 [inline] uart_write+0x4b2/0x740 drivers/tty/serial/serial_core.c:598 process_output_block drivers/tty/n_tty.c:580 [inline] n_tty_write+0x6c1/0x11a0 drivers/tty/n_tty.c:2317 do_tty_write drivers/tty/tty_io.c:958 [inline] tty_write+0x3f1/0x880 drivers/tty/tty_io.c:1042 redirected_tty_write+0xaf/0xc0 drivers/tty/tty_io.c:1063 __vfs_write+0x117/0x9d0 fs/read_write.c:485 vfs_write+0x1fc/0x560 fs/read_write.c:549 ksys_write+0x101/0x260 fs/read_write.c:598 __do_sys_write fs/read_write.c:610 [inline] __se_sys_write fs/read_write.c:607 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:607 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #1 (&port_lock_key){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152 serial8250_console_write+0x8e5/0xb20 drivers/tty/serial/8250/8250_port.c:3247 univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:590 call_console_drivers kernel/printk/printk.c:1725 [inline] console_unlock+0xace/0x10d0 kernel/printk/printk.c:2407 vprintk_emit+0x33a/0x910 kernel/printk/printk.c:1926 vprintk_default+0x28/0x30 kernel/printk/printk.c:1967 vprintk_func+0x7a/0x117 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:2000 register_console+0x7e7/0xc00 kernel/printk/printk.c:2722 univ8250_console_init+0x3f/0x4b drivers/tty/serial/8250/8250_core.c:685 console_init+0x5d4/0x891 kernel/printk/printk.c:2808 start_kernel+0x610/0x94e init/main.c:661 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:452 x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:433 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242 -> #0 (console_owner){-.-.}: lock_acquire+0x1e4/0x4f0 kernel/locking/lockdep.c:3901 console_lock_spinning_enable kernel/printk/printk.c:1588 [inline] console_unlock+0x814/0x10d0 kernel/printk/printk.c:2404 vprintk_emit+0x33a/0x910 kernel/printk/printk.c:1926 vprintk_default+0x28/0x30 kernel/printk/printk.c:1967 vprintk_func+0x7a/0x117 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:2000 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.50+0xcb8/0x15ec kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x709/0x830 kernel/workqueue.c:5571 call_timer_fn+0x254/0x810 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7d2/0xc60 kernel/time/timer.c:1682 run_timer_softirq+0x31/0x40 kernel/time/timer.c:1697 __do_softirq+0x2eb/0xa74 kernel/softirq.c:292 invoke_softirq kernel/softirq.c:372 [inline] irq_exit+0x1d6/0x210 kernel/softirq.c:412 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x18e/0x6a0 arch/x86/kernel/apic/apic.c:1056 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864 native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline] default_idle+0xc2/0x410 arch/x86/kernel/process.c:498 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:489 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x3aa/0x580 kernel/sched/idle.c:262 cpu_startup_entry+0x10c/0x120 kernel/sched/idle.c:368 rest_init+0xe1/0xe4 init/main.c:442 start_kernel+0x913/0x94e init/main.c:739 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:452 x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:433 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242 other info that might help us debug this: Chain exists of: console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&(&pool->lock)->rlock); lock(&pool->lock/1); lock(&(&pool->lock)->rlock); lock(console_owner); *** DEADLOCK *** 4 locks held by swapper/0/0: #0: 00000000a9f13156 ((&wq_watchdog_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:178 [inline] #0: 00000000a9f13156 ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0x1db/0x810 kernel/time/timer.c:1316 #1: 0000000093b7f950 (rcu_read_lock_sched){....}, at: show_workqueue_state+0x0/0x1d0 kernel/workqueue.c:4423 #2: 00000000123738d3 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.50+0xb16/0x15ec kernel/workqueue.c:4557 #3: 00000000b428d4ad (console_lock){+.+.}, at: console_trylock_spinning kernel/printk/printk.c:1650 [inline] #3: 00000000b428d4ad (console_lock){+.+.}, at: vprintk_emit+0x31f/0x910 kernel/printk/printk.c:1925 stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.0-rc2+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113 print_circular_bug.isra.34.cold.55+0x1bd/0x27d kernel/locking/lockdep.c:1222 check_prev_add kernel/locking/lockdep.c:1862 [inline] check_prevs_add kernel/locking/lockdep.c:1975 [inline] validate_chain kernel/locking/lockdep.c:2416 [inline] __lock_acquire+0x3449/0x5020 kernel/locking/lockdep.c:3412 lock_acquire+0x1e4/0x4f0 kernel/locking/lockdep.c:3901 console_lock_spinning_enable kernel/printk/printk.c:1588 [inline] console_unlock+0x814/0x10d0 kernel/printk/printk.c:2404 vprintk_emit+0x33a/0x910 kernel/printk/printk.c:1926 vprintk_default+0x28/0x30 kernel/printk/printk.c:1967 vprintk_func+0x7a/0x117 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:2000 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.50+0xcb8/0x15ec kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x709/0x830 kernel/workqueue.c:5571 call_timer_fn+0x254/0x810 kernel/time/timer.c:1326 expire_timers kernel/time/timer.c:1363 [inline] __run_timers+0x7d2/0xc60 kernel/time/timer.c:1682 ? trace_hardirqs_on_cal Lost 70 message(s)! pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=7/256 in-flight: 5:linkwatch_event, 5606:switchdev_deferred_process_work pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler workqueue events_highpri: flags=0x10 pwq 3: cpus=1 node=0 flags=0x0 nice=-20 active=1/256 pending: flush_backlog BAR(7113) workqueue events_unbound: flags=0x2 pwq 4: cpus=0-1 flags=0x4 nice=0 active=4/512 in-flight: 7:fsnotify_mark_destroy_workfn fsnotify_mark_destroy_workfn, 121:fsnotify_connector_destroy_workfn fsnotify_connector_destroy_workfn workqueue events_power_efficient: flags=0x80 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 in-flight: 18:neigh_periodic_work workqueue rcu_gp: flags=0x8 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 pending: process_srcu, process_srcu workqueue ipv6_addrconf: flags=0x40008 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 in-flight: 2126:addrconf_verify_work pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=1s workers=5 idle: 13 9381 pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=274s workers=4 idle: 2062 5618 25 pool 4: cpus=0-1 flags=0x4 nice=0 hung=225s workers=4 idle: 22 52