IPVS: ftp: loaded support on port[0] = 21 INFO: task kworker/1:7:7194 blocked for more than 120 seconds. Not tainted 4.16.0-rc7+ #3 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/1:7 D20864 7194 2 0x80000000 Workqueue: events cgwb_release_workfn Call Trace: context_switch kernel/sched/core.c:2862 [inline] __schedule+0x8fb/0x1ec0 kernel/sched/core.c:3440 schedule+0xf5/0x430 kernel/sched/core.c:3499 bit_wait+0x18/0x90 kernel/sched/wait_bit.c:250 __wait_on_bit+0x88/0x130 kernel/sched/wait_bit.c:51 out_of_line_wait_on_bit+0x204/0x3a0 kernel/sched/wait_bit.c:64 IPv6: ADDRCONF(NETDEV_UP): bridge0: link is not ready wait_on_bit include/linux/wait_bit.h:84 [inline] wb_shutdown+0x335/0x430 mm/backing-dev.c:377 cgwb_release_workfn+0x8b/0x61d mm/backing-dev.c:520 process_one_work+0xc47/0x1bb0 kernel/workqueue.c:2113 worker_thread+0x223/0x1990 kernel/workqueue.c:2247 kthread+0x33c/0x400 kernel/kthread.c:238 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:406 Showing all locks held in the system: 4 locks held by kworker/1:0/17: #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] work_static include/linux/workqueue.h:198 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] set_work_data kernel/workqueue.c:619 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] process_one_work+0xb12/0x1bb0 kernel/workqueue.c:2084 #1: ((work_completion)(&cw->work)){+.+.}, at: [<000000002b905731>] process_one_work+0xb89/0x1bb0 kernel/workqueue.c:2088 #2: (cpu_hotplug_lock.rw_sem){++++}, at: [<00000000b0119068>] get_online_cpus include/linux/cpu.h:124 [inline] #2: (cpu_hotplug_lock.rw_sem){++++}, at: [<00000000b0119068>] memcg_create_kmem_cache+0x16/0x170 mm/slab_common.c:619 #3: (slab_mutex){+.+.}, at: [<000000009d1cf1d5>] memcg_create_kmem_cache+0x24/0x170 mm/slab_common.c:622 2 locks held by khungtaskd/869: #0: (rcu_read_lock){....}, at: [<000000005cf5b296>] check_hung_uninterruptible_tasks kernel/hung_task.c:175 [inline] #0: (rcu_read_lock){....}, at: [<000000005cf5b296>] watchdog+0x1c5/0xd60 kernel/hung_task.c:249 #1: (tasklist_lock){.+.+}, at: [<00000000762774f7>] debug_show_all_locks+0xd3/0x3d0 kernel/locking/lockdep.c:4470 2 locks held by rs:main Q:Reg/4331: #0: (&f->f_pos_lock){+.+.}, at: [<000000003560fecc>] __fdget_pos+0x12b/0x190 fs/file.c:765 #1: (sb_writers#4){.+.+}, at: [<00000000bfabb1b7>] file_start_write include/linux/fs.h:2709 [inline] #1: (sb_writers#4){.+.+}, at: [<00000000bfabb1b7>] vfs_write+0x407/0x510 fs/read_write.c:543 2 locks held by getty/4423: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4424: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4425: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4426: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4427: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4428: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 2 locks held by getty/4429: #0: (&tty->ldisc_sem){++++}, at: [<000000006c3379bd>] ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365 #1: (&ldata->atomic_read_lock){+.+.}, at: [<000000007f5c978f>] n_tty_read+0x2ef/0x1a40 drivers/tty/n_tty.c:2131 5 locks held by kworker/0:2/4856: #0: ((wq_completion)"cgroup_destroy"){+.+.}, at: [<00000000b2bdee7c>] work_static include/linux/workqueue.h:198 [inline] #0: ((wq_completion)"cgroup_destroy"){+.+.}, at: [<00000000b2bdee7c>] set_work_data kernel/workqueue.c:619 [inline] #0: ((wq_completion)"cgroup_destroy"){+.+.}, at: [<00000000b2bdee7c>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ((wq_completion)"cgroup_destroy"){+.+.}, at: [<00000000b2bdee7c>] process_one_work+0xb12/0x1bb0 kernel/workqueue.c:2084 #1: ((work_completion)(&css->destroy_work)){+.+.}, at: [<000000002b905731>] process_one_work+0xb89/0x1bb0 kernel/workqueue.c:2088 #2: (cgroup_mutex){+.+.}, at: [<00000000c71f4f72>] css_killed_work_fn+0x93/0x5c0 kernel/cgroup/cgroup.c:4967 #3: (cpu_hotplug_lock.rw_sem){++++}, at: [<0000000032f69eb1>] get_online_cpus include/linux/cpu.h:124 [inline] #3: (cpu_hotplug_lock.rw_sem){++++}, at: [<0000000032f69eb1>] memcg_deactivate_kmem_caches+0x21/0xf0 mm/slab_common.c:747 #4: (slab_mutex){+.+.}, at: [<00000000b141acb3>] memcg_deactivate_kmem_caches+0x2f/0xf0 mm/slab_common.c:750 3 locks held by kworker/1:6/6448: #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] work_static include/linux/workqueue.h:198 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] set_work_data kernel/workqueue.c:619 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] process_one_work+0xb12/0x1bb0 kernel/workqueue.c:2084 #1: (slab_caches_to_rcu_destroy_work){+.+.}, at: [<000000002b905731>] process_one_work+0xb89/0x1bb0 kernel/workqueue.c:2088 #2: (slab_mutex){+.+.}, at: [<0000000014aca233>] slab_caches_to_rcu_destroy_workfn+0x25/0xc0 mm/slab_common.c:556 4 locks held by kworker/0:8/7018: #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] work_static include/linux/workqueue.h:198 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] set_work_data kernel/workqueue.c:619 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ((wq_completion)"memcg_kmem_cache"){+.+.}, at: [<00000000b2bdee7c>] process_one_work+0xb12/0x1bb0 kernel/workqueue.c:2084 #1: ((work_completion)(&cw->work)){+.+.}, at: [<000000002b905731>] process_one_work+0xb89/0x1bb0 kernel/workqueue.c:2088 #2: (cpu_hotplug_lock.rw_sem){++++}, at: [<00000000b0119068>] get_online_cpus include/linux/cpu.h:124 [inline] #2: (cpu_hotplug_lock.rw_sem){++++}, at: [<00000000b0119068>] memcg_create_kmem_cache+0x16/0x170 mm/slab_common.c:619 #3: (slab_mutex){+.+.}, at: [<000000009d1cf1d5>] memcg_create_kmem_cache+0x24/0x170 mm/slab_common.c:622 2 locks held by kworker/1:7/7194: #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] work_static include/linux/workqueue.h:198 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] set_work_data kernel/workqueue.c:619 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline] #0: ((wq_completion)"events"){+.+.}, at: [<00000000b2bdee7c>] process_one_work+0xb12/0x1bb0 kernel/workqueue.c:2084 #1: ((work_completion)(&wb->release_work)){+.+.}, at: [<000000002b905731>] process_one_work+0xb89/0x1bb0 kernel/workqueue.c:2088 ============================================= NMI backtrace for cpu 1 CPU: 1 PID: 869 Comm: khungtaskd Not tainted 4.16.0-rc7+ #3 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x24d lib/dump_stack.c:53 nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103 nmi_trigger_cpumask_backtrace+0x123/0x180 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_all_cpu_backtrace include/linux/nmi.h:138 [inline] check_hung_task kernel/hung_task.c:132 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:190 [inline] watchdog+0x90c/0xd60 kernel/hung_task.c:249 kthread+0x33c/0x400 kernel/kthread.c:238 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:406 Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 5846 Comm: kworker/0:4 Not tainted 4.16.0-rc7+ #3 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events rht_deferred_worker RIP: 0010:separate_irq_context kernel/locking/lockdep.c:3074 [inline] RIP: 0010:__lock_acquire+0x7a5/0x3e00 kernel/locking/lockdep.c:3422 RSP: 0018:ffff8801b0bdeb30 EFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff8801af67ac88 RCX: 1ffffffff11bbc56 RDX: 0000000000000001 RSI: 1ffff10035ecf595 RDI: ffff8801af67aca9 RBP: ffff8801b0bdeec0 R08: 1ffff1003617bd43 R09: 000000000000000c R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000031 R13: 3fbf39e76707aa99 R14: 0000000000000003 R15: ffff8801af67a3c0 FS: 0000000000000000(0000) GS:ffff8801db200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f59223aba8c CR3: 0000000007a22002 CR4: 00000000001606f0 DR0: 0008000000000000 DR1: 0008000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 Call Trace: lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:310 [inline] __queue_work+0x2c4/0x1230 kernel/workqueue.c:1419 queue_work_on+0x16a/0x1c0 kernel/workqueue.c:1488 queue_work include/linux/workqueue.h:488 [inline] schedule_work include/linux/workqueue.h:546 [inline] rht_deferred_worker+0x2ba/0x1cd0 lib/rhashtable.c:436 process_one_work+0xc47/0x1bb0 kernel/workqueue.c:2113 worker_thread+0x223/0x1990 kernel/workqueue.c:2247 kthread+0x33c/0x400 kernel/kthread.c:238 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:406 Code: b8 00 00 00 00 00 fc ff df 45 0f b6 76 21 48 89 fa 48 c1 ea 03 0f b6 04 02 48 89 fa 83 e2 07 38 d0 7f 08 84 c0 0f 85 f1 33 00 00 <44> 32 73 21 41 83 e6 60 0f 84 11 05 00 00 44 89 e6 31 c9 31 c0