IPVS: stopping master sync thread 7355 ... INFO: task kworker/u4:1:22 blocked for more than 140 seconds. Not tainted 4.9.124-g09eb2ba #31 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:1 D27096 22 2 0x00000000 Workqueue: netns cleanup_net ffff8801d9aeb000 0000000000000000 ffff8801b6613480 ffff8801d9a6b000 ffff8801db321c18 ffff8801d9bdf7a0 ffffffff839f0afd ffffffff85599a50 ffff8801d9aeb000 000000000000363d 00ff8801d9aeb980 ffff8801db3224e8 Call Trace: [] schedule+0x7f/0x1b0 kernel/sched/core.c:3553 [] schedule_timeout+0x861/0xf70 kernel/time/timer.c:1768 [] do_wait_for_common kernel/sched/completion.c:75 [inline] [] __wait_for_common kernel/sched/completion.c:93 [inline] [] wait_for_common kernel/sched/completion.c:101 [inline] [] wait_for_completion+0x203/0x2e0 kernel/sched/completion.c:122 [] kthread_stop+0xa6/0x5d0 kernel/kthread.c:519 [] stop_sync_thread.cold.10+0xcc/0x363 net/netfilter/ipvs/ip_vs_sync.c:1985 [] ip_vs_sync_net_cleanup+0x2f/0x80 net/netfilter/ipvs/ip_vs_sync.c:2033 [] __ip_vs_dev_cleanup+0x6b/0x80 net/netfilter/ipvs/ip_vs_core.c:2285 [] ops_exit_list.isra.6+0xb0/0x160 net/core/net_namespace.c:136 [] cleanup_net+0x321/0x630 net/core/net_namespace.c:473 [] process_one_work+0x7e1/0x1500 kernel/workqueue.c:2092 [] worker_thread+0xd6/0x10a0 kernel/workqueue.c:2226 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Showing all locks held in the system: 1 lock held by kworker/dying/4: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 4 locks held by kworker/u4:1/22: #0: ("%s""netns"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("%s""netns"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 #2: (net_mutex){+.+.+.}, at: [] cleanup_net+0x13f/0x630 net/core/net_namespace.c:439 #3: (ipvs->sync_mutex){+.+.+.}, at: [] ip_vs_sync_net_cleanup+0x22/0x80 net/netfilter/ipvs/ip_vs_sync.c:2032 2 locks held by khungtaskd/519: #0: (rcu_read_lock){......}, at: [] check_hung_uninterruptible_tasks kernel/hung_task.c:168 [inline] #0: (rcu_read_lock){......}, at: [] watchdog+0x11c/0xa20 kernel/hung_task.c:239 #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x79/0x218 kernel/locking/lockdep.c:4336 2 locks held by rs:main Q:Reg/3680: #0: (&f->f_pos_lock){+.+.+.}, at: [] __fdget_pos+0xac/0xd0 fs/file.c:781 #1: (sb_writers#3){.+.+.+}, at: [] file_start_write include/linux/fs.h:2640 [inline] #1: (sb_writers#3){.+.+.+}, at: [] vfs_write+0x3ae/0x530 fs/read_write.c:556 2 locks held by getty/3780: #0: (&tty->ldisc_sem){++++++}, at: [] ldsem_down_read+0x32/0x40 drivers/tty/tty_ldsem.c:367 #1: (&ldata->atomic_read_lock){+.+...}, at: [] n_tty_read+0x202/0x16e0 drivers/tty/n_tty.c:2142 1 lock held by syz-executor0/3868: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor6/3876: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 2 locks held by kworker/u4:4/4126: #0: ("events_unbound"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: ((&sub_info->work)){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 2 locks held by kworker/u4:5/4214: #0: ("events_unbound"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: ((&sub_info->work)){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 2 locks held by kworker/u4:8/5809: #0: ("events_unbound"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: ((&sub_info->work)){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 1 lock held by ipvs-m:3:0/7355: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/7946: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 2 locks held by kworker/u4:9/9405: #0: ("events_unbound"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: ((&sub_info->work)){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 2 locks held by kworker/u4:10/9417: #0: ("events_unbound"){.+.+.+}, at: [] work_static include/linux/workqueue.h:186 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("events_unbound"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x6ee/0x1500 kernel/workqueue.c:2085 #1: ((&sub_info->work)){+.+.+.}, at: [] process_one_work+0x728/0x1500 kernel/workqueue.c:2089 1 lock held by syz-executor1/13207: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/15599: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor3/16062: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 3 locks held by syz-executor3/16068: #0: (&bdev->bd_mutex){+.+.+.}, at: [] __blkdev_put+0xbb/0x840 fs/block_dev.c:1579 #1: (loop_index_mutex){+.+.+.}, at: [] lo_release+0x1f/0x1a0 drivers/block/loop.c:1606 #2: (&lo->lo_ctl_mutex#2){+.+.+.}, at: [] __lo_release drivers/block/loop.c:1584 [inline] #2: (&lo->lo_ctl_mutex#2){+.+.+.}, at: [] lo_release+0x85/0x1a0 drivers/block/loop.c:1607 1 lock held by syz-executor3/16089: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by loop3/16069: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor6/16120: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor6/16124: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor6/16133: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor6/16141: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/16122: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/16129: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/16137: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor5/16145: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/16134: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/16136: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor0/16139: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by modprobe/16138: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by blkid/16140: #0: (&bdev->bd_mutex){+.+.+.}, at: [] __blkdev_get+0x10c/0xd60 fs/block_dev.c:1268 1 lock held by syz-executor1/16142: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor1/16143: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor1/16146: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor1/16147: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor1/16149: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by syz-executor1/16150: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by modprobe/16144: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by modprobe/16148: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by modprobe/16151: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by modprobe/16152: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16153: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16154: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16155: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16156: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16157: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 1 lock held by getty/16158: #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] __blocking_notifier_call_chain kernel/notifier.c:316 [inline] #0: ((task_exit_notifier).rwsem){.+.+..}, at: [] blocking_notifier_call_chain+0x68/0xa0 kernel/notifier.c:328 ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 519 Comm: khungtaskd Not tainted 4.9.124-g09eb2ba #31 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d8417d08 ffffffff81eb95e9 0000000000000000 0000000000000000 0000000000000000 0000000000000001 ffffffff810b9fa0 ffff8801d8417d40 ffffffff81ec4927 0000000000000000 0000000000000000 0000000000000003 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] nmi_cpu_backtrace.cold.2+0x48/0x87 lib/nmi_backtrace.c:99 [] nmi_trigger_cpumask_backtrace+0x12a/0x14f lib/nmi_backtrace.c:60 [] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [] check_hung_task kernel/hung_task.c:125 [inline] [] check_hung_uninterruptible_tasks kernel/hung_task.c:182 [inline] [] watchdog+0x6b4/0xa20 kernel/hung_task.c:239 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 skipped: idling at pc 0xffffffff83a00606