BUG: workqueue lockup - pool[ 716.239891] INFO: task kworker/u4:1:28346 blocked for more than 140 seconds. Not tainted 4.9.182+ #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:1 D 25600 28346 2 0x80000000 Workqueue: netns cleanup_net 0000000000000083 ffff8801a2355f00 0000000000000000 ffff8801db621000 ffff8801da680000 ffff8801db621018 ffff8801a66ef788 ffffffff8280a19e ffffffff82819ea8 ffff8801a2355f00 00ff8801a66ef6d8 ffff8801db6218f0Call Trace: [<00000000e6c7f5c1>] schedule+0x92/0x1c0 kernel/sched/core.c:3546 [<0000000018677fee>] schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3579 [<000000002e687cd0>] __mutex_lock_common kernel/locking/mutex.c:582 [inline] [<000000002e687cd0>] mutex_lock_nested+0x38d/0x920 kernel/locking/mutex.c:621 [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 [<0000000050785dd9>] netdev_run_todo+0x212/0x780 net/core/dev.c:7553 [<00000000aa773038>] rtnl_unlock+0xe/0x10 net/core/rtnetlink.c:104 [<0000000032cac980>] sit_exit_net+0x437/0x610 net/ipv6/sit.c:1871 [<000000004b944637>] ops_exit_list.isra.0+0xb0/0x160 net/core/net_namespace.c:136 [<000000007af5dc80>] cleanup_net+0x3d6/0x8a0 net/core/net_namespace.c:474 [<00000000f300ff7a>] process_one_work+0x88b/0x1600 kernel/workqueue.c:2114 [<00000000baa929fe>] worker_thread+0x5df/0x11d0 kernel/workqueue.c:2251 [<00000000b0ff1b47>] kthread+0x278/0x310 kernel/kthread.c:211 [<000000003b8a9fbd>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Showing all locks held in the system: 2 locks held by khungtaskd/24: #0: (rcu_read_lock){......}, at: [<00000000a228415f>] check_hung_uninterruptible_tasks kernel/hung_task.c:169 [inline] (rcu_read_lock){......}, at: [<00000000a228415f>] watchdog+0x14b/0xaf0 kernel/hung_task.c:263 (tasklist_lock){.+.+..}, at: [<00000000071d2cab>] debug_show_all_locks+0x7f/0x21f kernel/locking/lockdep.c:4336 #0: (&f->f_pos_lock){+.+.+.}, at: [<00000000b00e273b>] __fdget_pos+0xa8/0xd0 fs/file.c:782 #0: (&tty->ldisc_sem){++++++}, at: [<000000002a9d9cdc>] ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:377 (&ldata->atomic_read_lock){+.+...}, at: [<000000000587bf73>] n_tty_read+0x1fe/0x1820 drivers/tty/n_tty.c:2156 #0: ("%s"("ipv6_addrconf")){.+.+..}, at: [<0000000054c4a837>] process_one_work+0x790/0x1600 kernel/workqueue.c:2107 ((addr_chk_work).work){+.+...}, at: [<0000000060a11c39>] process_one_work+0x7ce/0x1600 kernel/workqueue.c:2111 (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: ("%s""netns"){.+.+.+}, at: [<0000000054c4a837>] process_one_work+0x790/0x1600 kernel/workqueue.c:2107 (net_cleanup_work){+.+.+.}, at: [<0000000060a11c39>] process_one_work+0x7ce/0x1600 kernel/workqueue.c:2111 (net_mutex){+.+.+.}, at: [<000000004c6894b2>] cleanup_net+0x131/0x8a0 net/core/net_namespace.c:440 (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 (rcu_preempt_state.exp_mutex){+.+...}, at: [<00000000bfb4b2bb>] exp_funnel_lock kernel/rcu/tree_exp.h:256 [inline] (rcu_preempt_state.exp_mutex){+.+...}, at: [<00000000bfb4b2bb>] _synchronize_rcu_expedited+0x339/0x850 kernel/rcu/tree_exp.h:569 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000b5e6a517>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [<00000000e7a0b5a9>] rtnl_lock net/core/rtnetlink.c:70 [inline] (rtnl_mutex){+.+.+.}, at: [<00000000e7a0b5a9>] rtnetlink_rcv+0x1c/0x40 net/core/rtnetlink.c:4086 ============================================= NMI backtrace for cpu 1 CPU: 1 PID: 24 Comm: khungtaskd Not tainted 4.9.182+ #1 ffff8801d98efcc8 ffffffff81b57e21 0000000000000001 0000000000000000 0000000000000001 ffffffff81099901 dffffc0000000000 ffff8801d98efd00 ffffffff81b630bc 0000000000000001 0000000000000000 0000000000000001Call Trace: [<00000000e2363da9>] __dump_stack lib/dump_stack.c:15 [inline] [<00000000e2363da9>] dump_stack+0xc1/0x120 lib/dump_stack.c:51 [<0000000047963056>] nmi_cpu_backtrace.cold+0x47/0x87 lib/nmi_backtrace.c:99 [<0000000068af0a51>] nmi_trigger_cpumask_backtrace+0x124/0x155 lib/nmi_backtrace.c:60 [<0000000067f591f7>] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [<00000000b5e767b4>] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [<00000000b5e767b4>] check_hung_task kernel/hung_task.c:126 [inline] [<00000000b5e767b4>] check_hung_uninterruptible_tasks kernel/hung_task.c:183 [inline] [<00000000b5e767b4>] watchdog+0x670/0xaf0 kernel/hung_task.c:263 [<00000000b0ff1b47>] kthread+0x278/0x310 kernel/kthread.c:211 [<000000003b8a9fbd>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 28848 Comm: kworker/0:0 Not tainted 4.9.182+ #1 Workqueue: events rtc_timer_do_workc task: 000000008786c034 task.stack: 00000000d15133a3 RIP: 0010:[] c [<00000000e3e3ec25>] io_serial_in+0x6b/0x90 drivers/tty/serial/8250/8250_port.c:414 RSP: 0000:ffff8801db6077b0 EFLAGS: 00000002 RAX: dffffc0000000000 RBX: 00000000000003fd RCX: 0000000000000000 RDX: 00000000000003fd RSI: ffffffff81d77951 RDI: ffffffff84b65d18 RBP: ffff8801db6077c0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: ffffffff841d456a R12: ffffffff84b65ce0 R13: 0000000000000020 R14: fffffbfff096cbe3 R15: fffffbfff096cba5 FS: 0000000000000000(0000) GS:ffff8801db600000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c4315e3000 CR3: 00000001d6a60000 CR4: 00000000001606b0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 Stack: ffffffff84b65ce0c 0000000000002707c ffff8801db607810c ffffffff81d79bf4c ffff8801a9655f00c ffffffff84b65d28c ffffffff84b65f1ac ffffffff841d454cc ffffffff84b65ce0c 0000000000000037c dffffc0000000000c 0000000000000037c Call Trace: d [<00000000a17bda66>] serial_in drivers/tty/serial/8250/8250.h:111 [inline] d [<00000000a17bda66>] wait_for_xmitr+0x94/0x1e0 drivers/tty/serial/8250/8250_port.c:1997 [<000000006f273013>] serial8250_console_putchar+0x20/0x60 drivers/tty/serial/8250/8250_port.c:3103 [<00000000065be0d4>] uart_console_write+0x56/0xe0 drivers/tty/serial/serial_core.c:1880 [<00000000a5f22d9f>] serial8250_console_write+0x2fb/0x860 drivers/tty/serial/8250/8250_port.c:3169 [<0000000044e9dae5>] univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:594 [<00000000fccc1a31>] call_console_drivers.isra.0.constprop.0+0x1ac/0x360 kernel/printk/printk.c:1594 [<0000000051adb12c>] console_cont_flush kernel/printk/printk.c:2321 [inline] [<0000000051adb12c>] console_unlock+0x17b/0xb20 kernel/printk/printk.c:2387 [<00000000e4c914d2>] vprintk_emit+0x43f/0x6f0 kernel/printk/printk.c:1908 [<0000000007000739>] vprintk+0x28/0x30 kernel/printk/printk.c:1918 [<00000000312799d1>] vprintk_func kernel/printk/internal.h:36 [inline] [<00000000312799d1>] printk+0xc2/0xf5 kernel/printk/printk.c:1980 [<0000000082564e64>] wq_watchdog_timer_fn.cold+0x11/0x66 kernel/workqueue.c:5432 [<00000000f8393933>] call_timer_fn+0x167/0x6d0 kernel/time/timer.c:1319 [<00000000fb772c67>] expire_timers+0x25b/0x5c0 kernel/time/timer.c:1359 [<0000000030a0406e>] __run_timers kernel/time/timer.c:1674 [inline] [<0000000030a0406e>] run_timer_softirq+0x404/0x620 kernel/time/timer.c:1689 [<00000000dc729b88>] __do_softirq+0x22d/0x964 kernel/softirq.c:288 [<000000006e9f4a92>] invoke_softirq kernel/softirq.c:368 [inline] [<000000006e9f4a92>] irq_exit+0x119/0x160 kernel/softirq.c:409 [<0000000010bf36ff>] exiting_irq arch/x86/include/asm/apic.h:669 [inline] [<0000000010bf36ff>] smp_apic_timer_interrupt+0x7e/0xb0 arch/x86/kernel/apic/apic.c:962 [<000000003d8972cb>] apic_timer_interrupt+0x9d/0xb0 arch/x86/entry/entry_64.S:650 d [<0000000078b4dc5e>] ? rtc_aie_update_irq+0x30/0x30 drivers/rtc/interface.c:538 [<00000000068076d0>] spin_unlock_irqrestore include/linux/spinlock.h:362 [inline] [<00000000068076d0>] rtc_handle_legacy_irq+0x81/0x190 drivers/rtc/interface.c:515 [<0000000001890cae>] rtc_uie_update_irq+0x20/0x30 drivers/rtc/interface.c:550 [<00000000156ba54b>] rtc_timer_do_work+0x1fe/0x600 drivers/rtc/interface.c:881 [<00000000f300ff7a>] process_one_work+0x88b/0x1600 kernel/workqueue.c:2114 [<00000000baa929fe>] worker_thread+0x5df/0x11d0 kernel/workqueue.c:2251 [<00000000b0ff1b47>] kthread+0x278/0x310 kernel/kthread.c:211 [<000000003b8a9fbd>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Code: c24 cc9 c00 c00 c00 c49 c8d c7c c24 c38 c48 cb8 c00 c00 c00 c00 c00 cfc cff cdf c48 c89 cfa c48 cc1 cea c03 cd3 ce3 c80 c3c c02 c00 c75 c17 c41 c03 c5c c24 c38 c89 cda cec c<5b> c0f cb6 cc0 c41 c5c c5d cc3 ce8 c18 c55 c78 cff ceb cc2 ce8 c71 c55 c78 cff ceb c