sched: DL replenish lagged too much INFO: task kworker/u4:7:5146 blocked for more than 140 seconds. Not tainted 4.9.111-g03c70fe #6 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:7 D25736 5146 2 0x00000000 Workqueue: netns cleanup_net ffff8801cb4e4800 0000000000000000 ffff8801cd430000 ffffffff84629800 ffff8801db221c18 ffff8801d88a7770 ffffffff839e7ded ffffffff85597a90 ffff8801cb4e4800 0000000000007f49 00ff8801cb4e5158 ffff8801db2224e8 Call Trace: [] schedule+0x7f/0x1b0 kernel/sched/core.c:3557 [] schedule_timeout+0x861/0xf70 kernel/time/timer.c:1768 [] do_wait_for_common kernel/sched/completion.c:75 [inline] [] __wait_for_common kernel/sched/completion.c:93 [inline] [] wait_for_common kernel/sched/completion.c:101 [inline] [] wait_for_completion+0x203/0x2e0 kernel/sched/completion.c:122 [] kthread_stop+0xa6/0x5d0 kernel/kthread.c:513 [] pg_net_exit+0x3de/0x6e0 net/core/pktgen.c:3923 [] ops_exit_list.isra.6+0xb0/0x160 net/core/net_namespace.c:136 [] cleanup_net+0x321/0x630 net/core/net_namespace.c:473 [] process_one_work+0x7e1/0x1500 kernel/workqueue.c:2092 [] worker_thread+0xd6/0x10a0 kernel/workqueue.c:2226 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Showing all locks held in the system: 2 locks held by khungtaskd/519: #0: (rcu_read_lock){......}, at: [] check_hung_uninterruptible_tasks kernel/hung_task.c:168 [inline] #0: (rcu_read_lock){......}, at: [] watchdog+0x11c/0xa20 kernel/hung_task.c:239 #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x79/0x218 kernel/locking/lockdep.c:4336 1 lock held by rsyslogd/3664: #0: (&f->f_pos_lock){+.+.+.}, at: [] __fdget_pos+0xac/0xd0 fs/file.c:781 2 locks held by getty/3760: #0: (&tty->ldisc_sem){++++++}, at: [] ldsem_down_read+0x32/0x40 drivers/tty/tty_ldsem.c:367 #1: (&ldata->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x202/0x16e0 drivers/tty/n_tty.c:2133 1 lock held by kworker/dying/4984: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 3 locks held by kworker/u4:7/5146: #0: ("%s""netns"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("%s""netns"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 #2: (net_mutex){+.+.+.}, at: [] cleanup_net+0x13f/0x630 net/core/net_namespace.c:439 1 lock held by syz-executor4/943: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by kpktgend_0/17302: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by kpktgend_1/17303: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/20160: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by kworker/dying/22654: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/26725: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/30693: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/31679: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/32215: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/32274: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/32281: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/32232: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/32234: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor2/32280: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/32250: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/32253: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/32288: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32263: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32267: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32283: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32284: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32285: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32286: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/32287: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/32264: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/32266: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/32289: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor4/32290: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by blkid/32282: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32291: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32292: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32293: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32294: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32295: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/32296: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by run-parts/32299: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 519 Comm: khungtaskd Not tainted 4.9.111-g03c70fe #6 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d96dfd08 ffffffff81eb2729 0000000000000000 0000000000000000 0000000000000000 0000000000000001 ffffffff810b9af0 ffff8801d96dfd40 ffffffff81ebda27 0000000000000000 0000000000000000 0000000000000002 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] nmi_cpu_backtrace.cold.2+0x48/0x87 lib/nmi_backtrace.c:99 [] nmi_trigger_cpumask_backtrace+0x12a/0x14f lib/nmi_backtrace.c:60 [] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [] check_hung_task kernel/hung_task.c:125 [inline] [] check_hung_uninterruptible_tasks kernel/hung_task.c:182 [inline] [] watchdog+0x6b4/0xa20 kernel/hung_task.c:239 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 skipped: idling at pc 0xffffffff839f78f6