BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 167s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=128/256
    in-flight: 17:rtc_timer_do_work
    pending: kernfs_notify_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, vmpressure_work_fn, bpf_prog_free_deferred, bpf_prog_free_deferred, defense_work_handler, defense_work_handler, psi_update_work, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, cache_reap, bpf_prog_free_deferred, bpf_prog_free_deferred, psi_update_work, psi_update_work, psi_update_work, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, nfc_urelease_event_work, pwq_unbound_release_workfn, pwq_unbound_release_workfn, nfc_urelease_event_work, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred
, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, free_obj_work, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, nfc_urelease_event_work, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred
, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, nfc_urelease_event_work, bpf_prog_free_deferred, bpf_prog_free_deferred, free_ioctx, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_map_free_deferred, pwq_unbound_release_workfn, bpf_map_free_deferred
, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, pwq_unbound_release_workfn, bpf_prog_free_deferred, bpf_prog_free_deferred, pwq_unbound_release_workfn, key_garbage_collector, kvmclock_sync_fn

======================================================
WARNING: possible circular locking dependency detected
5.1.0-rc1-next-20190322 #9 Not tainted
------------------------------------------------------
swapper/0/0 is trying to acquire lock:
00000000a1b9bce0 (console_owner){-.-.}, at: log_next kernel/printk/printk.c:499 [inline]
00000000a1b9bce0 (console_owner){-.-.}, at: console_unlock+0x3fc/0xeb0 kernel/printk/printk.c:2439

but task is already holding lock:
0000000093aee507 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold+0x879/0x11dc kernel/workqueue.c:4724

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (&(&pool->lock)->rlock){-.-.}:
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:144
       spin_lock include/linux/spinlock.h:329 [inline]
       __queue_work+0x23d/0x1180 kernel/workqueue.c:1450
       queue_work_on+0x192/0x200 kernel/workqueue.c:1519
       queue_work include/linux/workqueue.h:494 [inline]
       schedule_work include/linux/workqueue.h:552 [inline]
       put_pwq kernel/workqueue.c:1123 [inline]
       put_pwq+0x178/0x1d0 kernel/workqueue.c:1108
       put_pwq_unlocked.part.0+0x34/0x70 kernel/workqueue.c:1140
       put_pwq_unlocked kernel/workqueue.c:1134 [inline]
       destroy_workqueue+0x623/0x700 kernel/workqueue.c:4384
       do_floppy_init drivers/block/floppy.c:4699 [inline]
       floppy_async_init+0x2031/0x2183 drivers/block/floppy.c:4717
       async_run_entry_fn+0x126/0x570 kernel/async.c:127
       process_one_work+0x98e/0x1790 kernel/workqueue.c:2270
       worker_thread+0x98/0xe40 kernel/workqueue.c:2416
       kthread+0x357/0x430 kernel/kthread.c:253
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352

-> #3 (&pool->lock/1){..-.}:
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:144
       spin_lock include/linux/spinlock.h:329 [inline]
       __queue_work+0x23d/0x1180 kernel/workqueue.c:1450
       queue_work_on+0x192/0x200 kernel/workqueue.c:1519
       queue_work include/linux/workqueue.h:494 [inline]
       tty_schedule_flip drivers/tty/tty_buffer.c:413 [inline]
       tty_flip_buffer_push+0xc5/0x100 drivers/tty/tty_buffer.c:556
       pty_write+0x1a6/0x200 drivers/tty/pty.c:125
       n_tty_write+0xb06/0x1150 drivers/tty/n_tty.c:2356
       do_tty_write drivers/tty/tty_io.c:961 [inline]
       tty_write+0x45b/0x7a0 drivers/tty/tty_io.c:1045
       __vfs_write+0x8d/0x110 fs/read_write.c:485
       vfs_write+0x20c/0x580 fs/read_write.c:549
       ksys_write+0xea/0x1f0 fs/read_write.c:598
       __do_sys_write fs/read_write.c:610 [inline]
       __se_sys_write fs/read_write.c:607 [inline]
       __x64_sys_write+0x73/0xb0 fs/read_write.c:607
       do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (&(&port->lock)->rlock){-.-.}:
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x95/0xcd kernel/locking/spinlock.c:152
       tty_port_tty_get+0x22/0x80 drivers/tty/tty_port.c:287
       tty_port_default_wakeup+0x16/0x40 drivers/tty/tty_port.c:47
       tty_port_tty_wakeup+0x5d/0x70 drivers/tty/tty_port.c:387
       uart_write_wakeup+0x46/0x70 drivers/tty/serial/serial_core.c:103
       serial8250_tx_chars+0x4a4/0xb20 drivers/tty/serial/8250/8250_port.c:1806
       serial8250_handle_irq.part.0+0x1be/0x2e0 drivers/tty/serial/8250/8250_port.c:1879
       serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1865 [inline]
       serial8250_default_handle_irq+0xc5/0x150 drivers/tty/serial/8250/8250_port.c:1895
       serial8250_interrupt+0xfb/0x1a0 drivers/tty/serial/8250/8250_core.c:125
       __handle_irq_event_percpu+0x146/0x900 kernel/irq/handle.c:149
       handle_irq_event_percpu+0x74/0x160 kernel/irq/handle.c:189
       handle_irq_event+0xa7/0x134 kernel/irq/handle.c:206
       handle_edge_irq+0x264/0x8e0 kernel/irq/chip.c:822
       generic_handle_irq_desc include/linux/irqdesc.h:156 [inline]
       handle_irq+0x252/0x3d8 arch/x86/kernel/irq_64.c:78
       do_IRQ+0x99/0x1d0 arch/x86/kernel/irq.c:246
       ret_from_intr+0x0/0x1e
       arch_local_irq_restore arch/x86/include/asm/paravirt.h:767 [inline]
       __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:160 [inline]
       _raw_spin_unlock_irqrestore+0x95/0xe0 kernel/locking/spinlock.c:184
       spin_unlock_irqrestore include/linux/spinlock.h:384 [inline]
       uart_write+0x3b6/0x6f0 drivers/tty/serial/serial_core.c:615
       process_output_block drivers/tty/n_tty.c:595 [inline]
       n_tty_write+0x3ff/0x1150 drivers/tty/n_tty.c:2333
       do_tty_write drivers/tty/tty_io.c:961 [inline]
       tty_write+0x45b/0x7a0 drivers/tty/tty_io.c:1045
       redirected_tty_write+0xb2/0xc0 drivers/tty/tty_io.c:1066
       __vfs_write+0x8d/0x110 fs/read_write.c:485
       vfs_write+0x20c/0x580 fs/read_write.c:549
       ksys_write+0xea/0x1f0 fs/read_write.c:598
       __do_sys_write fs/read_write.c:610 [inline]
       __se_sys_write fs/read_write.c:607 [inline]
       __x64_sys_write+0x73/0xb0 fs/read_write.c:607
       do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&port_lock_key){-.-.}:
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x95/0xcd kernel/locking/spinlock.c:152
       serial8250_console_write+0x253/0x9c0 drivers/tty/serial/8250/8250_port.c:3245
       univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:586
       call_console_drivers kernel/printk/printk.c:1774 [inline]
       console_unlock+0xb1e/0xeb0 kernel/printk/printk.c:2452
       vprintk_emit+0x280/0x6d0 kernel/printk/printk.c:1975
       vprintk_default+0x28/0x30 kernel/printk/printk.c:2002
       vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:398
       printk+0xba/0xed kernel/printk/printk.c:2035
       register_console+0x74d/0xb50 kernel/printk/printk.c:2767
       univ8250_console_init+0x3e/0x4b drivers/tty/serial/8250/8250_core.c:681
       console_init+0x4f7/0x761 kernel/printk/printk.c:2853
       start_kernel+0x574/0x84f init/main.c:675
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:470
       x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:451
       secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243

-> #0 (console_owner){-.-.}:
       check_prevs_add kernel/locking/lockdep.c:2333 [inline]
       validate_chain kernel/locking/lockdep.c:2714 [inline]
       __lock_acquire+0x239c/0x3fb0 kernel/locking/lockdep.c:3701
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
       console_lock_spinning_enable kernel/printk/printk.c:1637 [inline]
       console_unlock+0x466/0xeb0 kernel/printk/printk.c:2449
       vprintk_emit+0x280/0x6d0 kernel/printk/printk.c:1975
       vprintk_default+0x28/0x30 kernel/printk/printk.c:2002
       vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:398
       printk+0xba/0xed kernel/printk/printk.c:2035
       show_pwq kernel/workqueue.c:4631 [inline]
       show_workqueue_state.cold+0x9e4/0x11dc kernel/workqueue.c:4726
       wq_watchdog_timer_fn+0x516/0x5a0 kernel/workqueue.c:5738
       call_timer_fn+0x190/0x720 kernel/time/timer.c:1325
       expire_timers kernel/time/timer.c:1362 [inline]
       __run_timers kernel/time/timer.c:1681 [inline]
       __run_timers kernel/time/timer.c:1649 [inline]
       run_timer_softirq+0xd03/0x1700 kernel/time/timer.c:1696
       __do_softirq+0x266/0x95a kernel/softirq.c:293
       invoke_softirq kernel/softirq.c:374 [inline]
       irq_exit+0x180/0x1d0 kernel/softirq.c:414
       exiting_irq arch/x86/include/asm/apic.h:536 [inline]
       smp_apic_timer_interrupt+0x14a/0x570 arch/x86/kernel/apic/apic.c:1062
       apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807
       native_safe_halt+0x2/0x10 arch/x86/include/asm/irqflags.h:57
       arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:567
       default_idle_call+0x36/0x90 kernel/sched/idle.c:93
       cpuidle_idle_call kernel/sched/idle.c:153 [inline]
       do_idle+0x386/0x570 kernel/sched/idle.c:262
       cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:353
       rest_init+0x245/0x37b init/main.c:450
       arch_call_rest_init+0xe/0x1b
       start_kernel+0x816/0x84f init/main.c:747
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:470
       x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:451
       secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243

other info that might help us debug this:

Chain exists of:
  console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&(&pool->lock)->rlock);
                               lock(&pool->lock/1);
                               lock(&(&pool->lock)->rlock);
  lock(console_owner);

 *** DEADLOCK ***

4 locks held by swapper/0/0:
 #0: 000000002cffb0b7 ((&wq_watchdog_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:170 [inline]
 #0: 000000002cffb0b7 ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0xda/0x720 kernel/time/timer.c:1315
 #1: 0000000007f52aed (rcu_read_lock_sched){....}, at: show_workqueue_state+0x0/0x120 kernel/workqueue.c:4596
 #2: 0000000093aee507 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold+0x879/0x11dc kernel/workqueue.c:4724
 #3: 00000000ba7294f8 (console_lock){+.+.}, at: console_trylock_spinning kernel/printk/printk.c:1699 [inline]
 #3: 00000000ba7294f8 (console_lock){+.+.}, at: vprintk_emit+0x267/0x6d0 kernel/printk/printk.c:1974

stack backtrace:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.1.0-rc1-next-20190322 #9
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1571
 check_prev_add.constprop.0+0xf11/0x23c0 kernel/locking/lockdep.c:2220
 check_prevs_add kernel/locking/lockdep.c:2333 [inline]
 validate_chain kernel/locking/lockdep.c:2714 [inline]
 __lock_acquire+0x239c/0x3fb0 kernel/locking/lockdep.c:3701
 lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
 console_lock_spinning_enable kernel/printk/printk.c:1637 [inline]
 console_unlock+0x466/0xeb0 kernel/printk/printk.c:2449
 vprintk_emit+0x280/0x6d0 kernel/printk/printk.c:1975
 vprintk_default+0x28/0x30 kernel/printk/printk.c:2002
 vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:398
 printk+0xba/0xed kernel/printk/printk.c:2035
 show_pwq kernel/workqueue.c:4631 [inline]
 show_workqueue_state.cold+0x9e4/0x11dc kernel/workqueue.c:4726
 wq_watchdog_timer_fn+0x516/0x5a0 kernel/workqueue.c:5738
 call_timer_fn+0x190/0x720 kernel/time/timer.c:1325
 expire_timers kernel/time/timer.c:1362 [inline]
 __run_timers kernel/time/timer.c:1681 [inline]
 __run_timers kernel/time/timer.c:1649 [inline]
 run_timer_softirq+0xd03/0x1700 kernel/time/timer.c:1696
 __do_softirq+0x266/0x95a kernel/softirq.c:293
 invoke_softirq kernel/softirq.c:374 [inline]
 irq_exit+0x180/0x1d0 kernel/softirq.c:414
 exiting_irq arch/x86/include/asm/apic.h:536 [inline]
 smp_apic_timer_interrupt+0x14a/0x570 arch/x86/kernel/apic/apic.c:1062
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807
 </IRQ>
RIP: 0010:native_safe_halt+0x2/0x10 arch/x86/include/asm/irqflags.h:58
Code: ff ff ff 48 89 c7 48 89 45 d8 e8 99 18 97 fa 48 8b 45 d8 e9 ce fe ff ff 48 89 df e8 88 18 97 fa eb 82 90 90 90 90 90 90 fb f4 <c3> 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 f4 c3 90 90 90 90 90 90
RSP: 0018:ffffffff88807d08 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff11252d9 RBX: ffffffff8887e080 RCX: 0000000000000000
RDX: dffffc0000000000 RSI: 0000000000000006 RDI: ffffffff8887e8fc
RBP: ffffffff88807d38 R08: ffffffff8887e080 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff889296b8 R14: 0000000000000000 R15: 0000000000000000
 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:567
 default_idle_call+0x36/0x90 kernel/sched/idle.c:93
 cpuidle_idle_call kernel/sched/idle.c:153 [inline]
 do_idle+0x386/0x570 kernel/sched/idle.c:262
 cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:353
 rest_init+0x245/0x37b init/main.c:450
 arch_call_rest_init+0xe/0x1b
 start_kernel+0x816/0x84f init/main.c:747
 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:470
 x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:451
 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243
workqueue events_long: flags=0x0
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
    pending: br_fdb_cleanup, br_fdb_cleanup
workqueue events_unbound: flags=0x2
  pwq 4: cpus=0-1 flags=0x4 nice=0 active=4/512
    in-flight: 9347:fsnotify_mark_destroy_workfn fsnotify_mark_destroy_workfn BAR(9263) BAR(15457) BAR(9820) BAR(10387) BAR(10612) BAR(10578) BAR(10066) BAR(10528) BAR(11618), 21:fsnotify_connector_destroy_workfn fsnotify_connector_destroy_workfn
workqueue events_power_efficient: flags=0x80
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
    pending: gc_worker, neigh_periodic_work
workqueue rcu_gp: flags=0x8
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
    pending: process_srcu
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
    pending: vmstat_update
workqueue netns: flags=0xe000a
  pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1
    in-flight: 7842:cleanup_net
workqueue memcg_kmem_cache: flags=0x0
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1
    pending: memcg_kmem_cache_create_func
    delayed: memcg_kmem_cache_create_func, memcg_kmem_cache_create_func
workqueue krxrpcd: flags=0x0
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1
    pending: rxrpc_peer_keepalive_worker
workqueue bat_events: flags=0xe000a
  pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1
    pending: batadv_nc_worker
    delayed: batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_mcast_mla_update, batadv_mcast_mla_update
pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=169s workers=6 idle: 7828 13012 9004 8249 22
pool 4: cpus=0-1 flags=0x4 nice=0 hung=1s workers=6 idle: 2538 15796 21349