IPVS: length: 71 != 24 sched: DL replenish lagged too much INFO: task kworker/u4:3:1615 blocked for more than 120 seconds. Not tainted 4.9.109-ga4230be #48 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:3 D25688 1615 2 0x00000000 Workqueue: netns cleanup_net ffff8801cf856000 0000000000000000 ffff8801b610aa00 ffff8801d9a6b000 ffff8801db321c18 ffff8801cf967770 ffffffff839e8a8d ffffffff85597a90 0000000000000046 0000000000000168 00ff8801cf856958 ffff8801db3224e8 Call Trace: [] schedule+0x7f/0x1b0 kernel/sched/core.c:3557 [] schedule_timeout+0x861/0xf70 kernel/time/timer.c:1768 [] do_wait_for_common kernel/sched/completion.c:75 [inline] [] __wait_for_common kernel/sched/completion.c:93 [inline] [] wait_for_common kernel/sched/completion.c:101 [inline] [] wait_for_completion+0x203/0x2e0 kernel/sched/completion.c:122 [] kthread_stop+0xa6/0x5d0 kernel/kthread.c:513 [] pg_net_exit+0x3de/0x6e0 net/core/pktgen.c:3923 [] ops_exit_list.isra.6+0xb0/0x160 net/core/net_namespace.c:136 [] cleanup_net+0x321/0x630 net/core/net_namespace.c:473 [] process_one_work+0x7e1/0x1500 kernel/workqueue.c:2092 [] worker_thread+0xd6/0x10a0 kernel/workqueue.c:2226 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Showing all locks held in the system: 2 locks held by khungtaskd/519: #0: (rcu_read_lock){......}, at: [] check_hung_uninterruptible_tasks kernel/hung_task.c:168 [inline] #0: (rcu_read_lock){......}, at: [] watchdog+0x11c/0xa20 kernel/hung_task.c:239 #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x79/0x218 kernel/locking/lockdep.c:4336 3 locks held by kworker/u4:3/1615: #0: ("%s""netns"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("%s""netns"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 #2: (net_mutex){+.+.+.}, at: [] cleanup_net+0x13f/0x630 net/core/net_namespace.c:439 1 lock held by rsyslogd/3682: #0: (&f->f_pos_lock){+.+.+.}, at: [] __fdget_pos+0xac/0xd0 fs/file.c:781 2 locks held by getty/3778: #0: (&tty->ldisc_sem){++++++}, at: [] ldsem_down_read+0x32/0x40 drivers/tty/tty_ldsem.c:367 #1: (&ldata->atomic_read_lock){+.+...}, at: [] n_tty_read+0x202/0x16e0 drivers/tty/n_tty.c:2133 1 lock held by syz-executor2/3832: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/3836: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by udevd/7617: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by udevd/8399: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by udevd/10021: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by udevd/10027: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by kpktgend_0/10763: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by kpktgend_1/10766: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/12159: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/12238: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/12239: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/12172: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/12178: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/12240: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/12241: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12190: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12197: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12198: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12210: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12214: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/12216: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/12242: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 519 Comm: khungtaskd Not tainted 4.9.109-ga4230be #48 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d8b97d08 ffffffff81eb3e29 0000000000000000 0000000000000000 0000000000000000 0000000000000001 ffffffff810b9580 ffff8801d8b97d40 ffffffff81ebf127 0000000000000000 0000000000000000 0000000000000003 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] nmi_cpu_backtrace.cold.2+0x48/0x87 lib/nmi_backtrace.c:99 [] nmi_trigger_cpumask_backtrace+0x12a/0x14f lib/nmi_backtrace.c:60 [] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [] check_hung_task kernel/hung_task.c:125 [inline] [] check_hung_uninterruptible_tasks kernel/hung_task.c:182 [inline] [] watchdog+0x6b4/0xa20 kernel/hung_task.c:239 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 skipped: idling at pc 0xffffffff839f8596