sched: DL replenish lagged too much INFO: task kworker/0:3:8199 blocked for more than 140 seconds. Not tainted 5.0.0-rc3+ #42 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/0:3 D23768 8199 2 0x80000000 Workqueue: events linkwatch_event Call Trace: context_switch kernel/sched/core.c:2831 [inline] __schedule+0x897/0x1e60 kernel/sched/core.c:3472 schedule+0xfe/0x350 kernel/sched/core.c:3516 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:3574 __mutex_lock_common kernel/locking/mutex.c:1002 [inline] __mutex_lock+0xa3b/0x1670 kernel/locking/mutex.c:1072 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1087 rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77 linkwatch_event+0x5f/0xf0 net/core/link_watch.c:236 process_one_work+0xd0c/0x1ce0 kernel/workqueue.c:2153 worker_thread+0x143/0x14a0 kernel/workqueue.c:2296 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 Showing all locks held in the system: 1 lock held by khungtaskd/1040: #0: 00000000a8380460 (rcu_read_lock){....}, at: debug_show_all_locks+0xc6/0x41d kernel/locking/lockdep.c:4389 1 lock held by rsyslogd/8029: #0: 000000000a35944d (&f->f_pos_lock){+.+.}, at: __fdget_pos+0x1b3/0x1f0 fs/file.c:795 2 locks held by getty/8151: #0: 00000000211fbce5 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 0000000093c58b4d (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8152: #0: 00000000159c084c (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 000000008d213079 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8153: #0: 00000000c44ced70 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 00000000a0883d8a (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8154: #0: 0000000017bbcf4f (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 00000000e36ff03f (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8155: #0: 00000000bd146299 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 00000000a1b2ded4 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8156: #0: 000000000bfb2345 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 00000000f1c83c8d (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 2 locks held by getty/8157: #0: 0000000006f3f7f3 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:341 #1: 0000000017ba2ae4 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x30a/0x1eb0 drivers/tty/n_tty.c:2154 3 locks held by kworker/0:3/8199: #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: process_one_work+0xbc7/0x1ce0 kernel/workqueue.c:2124 #1: 00000000fb51bb27 ((linkwatch_work).work){+.+.}, at: process_one_work+0xc1d/0x1ce0 kernel/workqueue.c:2128 #2: 00000000bcf8defe (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77 3 locks held by kworker/1:3/8205: #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 00000000dea7b702 ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: process_one_work+0xbc7/0x1ce0 kernel/workqueue.c:2124 #1: 000000003aabf0b1 ((addr_chk_work).work){+.+.}, at: process_one_work+0xc1d/0x1ce0 kernel/workqueue.c:2128 #2: 00000000bcf8defe (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77 2 locks held by kworker/u4:5/8330: 3 locks held by kworker/1:0/8433: #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: __write_once_size include/linux/compiler.h:220 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:40 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:59 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: set_work_data kernel/workqueue.c:617 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: 0000000016b3f5af ((wq_completion)"events"){+.+.}, at: process_one_work+0xbc7/0x1ce0 kernel/workqueue.c:2124 #1: 0000000094692a09 (deferred_process_work){+.+.}, at: process_one_work+0xc1d/0x1ce0 kernel/workqueue.c:2128 #2: 00000000bcf8defe (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:77 1 lock held by syz-executor4/13332: ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 1040 Comm: khungtaskd Not tainted 5.0.0-rc3+ #42 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1db/0x2d0 lib/dump_stack.c:113 nmi_cpu_backtrace.cold+0x63/0xa4 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x1be/0x236 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:203 [inline] watchdog+0xbbb/0x1170 kernel/hung_task.c:287 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 11119 Comm: kworker/1:5 Not tainted 5.0.0-rc3+ #42 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: events_power_efficient gc_worker RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:766 [inline] RIP: 0010:lock_acquire+0x259/0x570 kernel/locking/lockdep.c:3844 Code: 00 00 00 00 00 48 c1 e8 03 80 3c 10 00 0f 85 64 02 00 00 48 83 3d 76 31 2e 08 00 0f 84 d0 01 00 00 48 8b bd 48 ff ff ff 57 9d <0f> 1f 44 00 00 48 b8 00 00 00 00 00 fc ff df 48 03 85 40 ff ff ff RSP: 0018:ffff88804fcef7a8 EFLAGS: 00000286 RAX: 1ffffffff132502e RBX: ffff8880a8388380 RCX: 0000000000000000 RDX: dffffc0000000000 RSI: 0000000000000000 RDI: 0000000000000286 RBP: ffff88804fcef878 R08: 0000000000000002 R09: ffff8880a8388c70 R10: ffff8880a8388c50 R11: 0000000000000001 R12: ffffffff899a4180 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c435157010 CR3: 00000000935d6000 CR4: 00000000001426e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: rcu_lock_acquire include/linux/rcupdate.h:223 [inline] rcu_read_lock include/linux/rcupdate.h:607 [inline] gc_worker+0x1bf/0x1100 net/netfilter/nf_conntrack_core.c:1158 process_one_work+0xd0c/0x1ce0 kernel/workqueue.c:2153 worker_thread+0x143/0x14a0 kernel/workqueue.c:2296 kthread+0x357/0x430 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352