BUG: workqueue lockup - pool[ 716.103818] INFO: task kworker/1:1:25 blocked for more than 140 seconds. Not tainted 4.9.113-g47bbcd6 #10 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/1:1 D 26968 25 2 0x00000000 Workqueue: ipv6_addrconf addrconf_verify_work ffff8801d9aec800 ffff8801a1d164c0 ffff8801a1d14000 ffff880199343000 ffff8801db321c18 ffff8801d942fae8 ffffffff839e909d ffff8801d9aed0c8 ffffed003b35da18 ffff8801d9aec800 00fffc0000000000 ffff8801db3224e8Call Trace: [] schedule+0x7f/0x1b0 kernel/sched/core.c:3553 [] schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3586 [] __mutex_lock_common kernel/locking/mutex.c:582 [inline] [] mutex_lock_nested+0x326/0x870 kernel/locking/mutex.c:621 [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 [] addrconf_verify_work+0xe/0x20 net/ipv6/addrconf.c:4366 [] process_one_work+0x7e1/0x1500 kernel/workqueue.c:2092 [] worker_thread+0xd6/0x10a0 kernel/workqueue.c:2226 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Showing all locks held in the system: 3 locks held by kworker/1:1/25: #0: ("%s"("ipv6_addrconf")){.+.+..}, at: [] work_static include/linux/workqueue.h:186 [inline] ("%s"("ipv6_addrconf")){.+.+..}, at: [] set_work_data kernel/workqueue.c:617 [inline] ("%s"("ipv6_addrconf")){.+.+..}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] ("%s"("ipv6_addrconf")){.+.+..}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 ((addr_chk_work).work){+.+...}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: ("%s""netns"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] ("%s""netns"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] ("%s""netns"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] ("%s""netns"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 (net_mutex){+.+.+.}, at: [] cleanup_net+0x13f/0x630 net/core/net_namespace.c:439 (rcu_preempt_state.exp_mutex){+.+...}, at: [] exp_funnel_lock kernel/rcu/tree_exp.h:289 [inline] (rcu_preempt_state.exp_mutex){+.+...}, at: [] _synchronize_rcu_expedited+0x3a7/0x840 kernel/rcu/tree_exp.h:569 #0: (rcu_read_lock){......}, at: [] check_hung_uninterruptible_tasks kernel/hung_task.c:168 [inline] (rcu_read_lock){......}, at: [] watchdog+0x11c/0xa20 kernel/hung_task.c:239 (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x79/0x218 kernel/locking/lockdep.c:4336 #0: (&f->f_pos_lock){+.+.+.}, at: [] __fdget_pos+0xac/0xd0 fs/file.c:781 #0: (&tty->ldisc_sem){++++++}, at: [] ldsem_down_read+0x32/0x40 drivers/tty/tty_ldsem.c:367 (&ldata->atomic_read_lock){+.+...}, at: [] n_tty_read+0x202/0x16e0 drivers/tty/n_tty.c:2142 #0: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 (rcu_preempt_state.exp_mutex){+.+...}, at: [] exp_funnel_lock kernel/rcu/tree_exp.h:256 [inline] (rcu_preempt_state.exp_mutex){+.+...}, at: [] _synchronize_rcu_expedited+0x339/0x840 kernel/rcu/tree_exp.h:569 #0: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (event_mutex){+.+.+.}, at: [] perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234 (rcu_sched_state.exp_mutex){+.+...}, at: [] exp_funnel_lock kernel/rcu/tree_exp.h:256 [inline] (rcu_sched_state.exp_mutex){+.+...}, at: [] _synchronize_rcu_expedited+0x339/0x840 kernel/rcu/tree_exp.h:569 #0: (&type->s_umount_key#59/1){+.+.+.}, at: [] alloc_super fs/super.c:241 [inline] (&type->s_umount_key#59/1){+.+.+.}, at: [] sget_userns+0x552/0xba0 fs/super.c:503 (lock#2){+.+...}, at: [] lru_add_drain_all+0x4a/0x400 mm/swap.c:695 (cpu_hotplug.dep_map){++++++}, at: [] get_online_cpus+0x32/0x90 kernel/cpu.c:252 #0: (rtnl_mutex){+.+.+.}, at: [] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:70 #0: (lock#2){+.+...}, at: [] lru_add_drain_all+0x4a/0x400 mm/swap.c:695 #0: (lock#2){+.+...}, at: [] lru_add_drain_all+0x4a/0x400 mm/swap.c:695 ============================================= NMI backtrace for cpu 1 CPU: 1 PID: 519 Comm: khungtaskd Not tainted 4.9.113-g47bbcd6 #10 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d8527d08 ffffffff81eb32a9 0000000000000000 0000000000000001 0000000000000001 0000000000000001 ffffffff810b9bd0 ffff8801d8527d40 ffffffff81ebe5a7 0000000000000001 0000000000000000 0000000000000003Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] nmi_cpu_backtrace.cold.2+0x48/0x87 lib/nmi_backtrace.c:99 [] nmi_trigger_cpumask_backtrace+0x12a/0x14f lib/nmi_backtrace.c:60 [] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [] check_hung_task kernel/hung_task.c:125 [inline] [] check_hung_uninterruptible_tasks kernel/hung_task.c:182 [inline] [] watchdog+0x6b4/0xa20 kernel/hung_task.c:239 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.113-g47bbcd6 #10 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events rtc_timer_do_workc task: ffff8801d99b4800 task.stack: ffff8801d99f8000 RIP: 0010:[] c [] inb arch/x86/include/asm/io.h:316 [inline] RIP: 0010:[] c [] io_serial_in+0x6b/0x90 drivers/tty/serial/8250/8250_port.c:414 RSP: 0000:ffff8801db2077c8 EFLAGS: 00000002 RAX: dffffc0000000000 RBX: 00000000000003fd RCX: 0000000000000000 RDX: 00000000000003fd RSI: ffffffff8214fb01 RDI: ffffffff864aae78 RBP: ffff8801db2077d8 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff864aae40 R13: 0000000000000020 R14: fffffbfff0c9560f R15: fffffbfff0c955d1 FS: 0000000000000000(0000) GS:ffff8801db200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000607534 CR3: 000000000461e000 CR4: 00000000001606f0 DR0: 00000000200001c0 DR1: 00000000200001c0 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 Stack: ffffffff864aae40c 000000000000270bc ffff8801db207828c ffffffff82151724c 1ffff1003b640f00c ffffffff864aae88c ffffffff864ab07ac ffffffff864aae40c 0000000000000034c ffffffff82151870c dffffc0000000000c 0000000000000034c Call Trace: d [] serial_in drivers/tty/serial/8250/8250.h:111 [inline] d [] wait_for_xmitr+0x94/0x1e0 drivers/tty/serial/8250/8250_port.c:1998 [] serial8250_console_putchar+0x1f/0x60 drivers/tty/serial/8250/8250_port.c:3104 [] uart_console_write+0x59/0xf0 drivers/tty/serial/serial_core.c:1859 [] serial8250_console_write+0x528/0x820 drivers/tty/serial/8250/8250_port.c:3170 [] univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:594 [] call_console_drivers.isra.18.constprop.28+0x1ad/0x360 kernel/printk/printk.c:1589 [] console_cont_flush kernel/printk/printk.c:2316 [inline] [] console_unlock+0x788/0xb40 kernel/printk/printk.c:2382 [] vprintk_emit+0x448/0x790 kernel/printk/printk.c:1903 [] vprintk+0x28/0x30 kernel/printk/printk.c:1913 [] vprintk_default+0x1d/0x30 kernel/printk/printk.c:1914 [] vprintk_func kernel/printk/internal.h:36 [inline] [] printk+0xaf/0xd7 kernel/printk/printk.c:1975 [] wq_watchdog_timer_fn.cold.49+0x11/0x65 kernel/workqueue.c:5393 [] call_timer_fn+0x163/0x6e0 kernel/time/timer.c:1319 [] expire_timers kernel/time/timer.c:1359 [inline] [] __run_timers kernel/time/timer.c:1658 [inline] [] run_timer_softirq+0xd47/0x1590 kernel/time/timer.c:1686 [] __do_softirq+0x20b/0x937 kernel/softirq.c:284 [] invoke_softirq kernel/softirq.c:364 [inline] [] irq_exit+0x147/0x190 kernel/softirq.c:405 [] exiting_irq arch/x86/include/asm/apic.h:659 [inline] [] smp_apic_timer_interrupt+0x81/0xa0 arch/x86/kernel/apic/apic.c:960 [] apic_timer_interrupt+0xa0/0xb0 arch/x86/entry/entry_64.S:648 d [] ? arch_local_irq_restore arch/x86/include/asm/paravirt.h:768 [inline] d [] ? __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:162 [inline] d [] ? _raw_spin_unlock_irqrestore+0x5f/0x70 kernel/locking/spinlock.c:191 [] spin_unlock_irqrestore include/linux/spinlock.h:362 [inline] [] __wake_up+0x3f/0x50 kernel/sched/wait.c:96 [] rtc_handle_legacy_irq+0x12c/0x190 drivers/rtc/interface.c:518 [] rtc_uie_update_irq+0x1f/0x30 drivers/rtc/interface.c:545 [] rtc_timer_do_work+0x1fe/0x5d0 drivers/rtc/interface.c:876 [] process_one_work+0x7e1/0x1500 kernel/workqueue.c:2092 [] worker_thread+0xd6/0x10a0 kernel/workqueue.c:2226 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Code: c24 cc9 c00 c00 c00 c49 c8d c7c c24 c38 c48 cb8 c00 c00 c00 c00 c00 cfc cff cdf c48 c89 cfa c48 cc1 cea c03 cd3 ce3 c80 c3c c02 c00 c75 c17 c41 c03 c5c c24 c38 c89 cda cec c<5b> c0f cb6 cc0 c41 c5c c5d cc3 ce8 c58 cc0 c3e cff ceb cc2 ce8 cb1 cc0 c3e cff ceb c