INFO: task kworker/u4:1:21 blocked for more than 143 seconds. Not tainted 5.3.0-next-20190920 #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:1 D24896 21 2 0x80004000 Workqueue: netns cleanup_net Call Trace: context_switch kernel/sched/core.c:3384 [inline] __schedule+0x828/0x1c20 kernel/sched/core.c:4065 schedule+0xd9/0x260 kernel/sched/core.c:4132 rxrpc_destroy_all_calls+0x1fa/0x60e net/rxrpc/call_object.c:631 rxrpc_exit_net+0x1ba/0x310 net/rxrpc/net_ns.c:121 ops_exit_list.isra.0+0xaa/0x150 net/core/net_namespace.c:172 cleanup_net+0x4e2/0xa60 net/core/net_namespace.c:594 process_one_work+0x9af/0x1740 kernel/workqueue.c:2269 worker_thread+0x98/0xe40 kernel/workqueue.c:2415 kthread+0x361/0x430 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Showing all locks held in the system: 3 locks held by kworker/u4:1/21: #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: set_work_data kernel/workqueue.c:620 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:647 [inline] #0: ffff8880a99c8d28 ((wq_completion)netns){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2240 #1: ffff8880a9a1fdc0 (net_cleanup_work){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2244 #2: ffffffff8998b688 (pernet_ops_rwsem){++++}, at: cleanup_net+0xae/0xa60 net/core/net_namespace.c:556 3 locks held by kworker/1:1/22: #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: set_work_data kernel/workqueue.c:620 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:647 [inline] #0: ffff8880aa427d28 ((wq_completion)events){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2240 #1: ffff8880a9a27dc0 ((linkwatch_work).work){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2244 #2: ffffffff89998860 (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72 1 lock held by khungtaskd/1066: #0: ffffffff88fa9e80 (rcu_read_lock){....}, at: debug_show_all_locks+0x5f/0x27e kernel/locking/lockdep.c:5337 1 lock held by rsyslogd/8607: #0: ffff888095ac80e0 (&f->f_pos_lock){+.+.}, at: __fdget_pos+0xee/0x110 fs/file.c:801 2 locks held by getty/8701: #0: ffff8880a8935090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005f112e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8702: #0: ffff888090ac5090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005f012e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8703: #0: ffff8880922a8090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005f0d2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8704: #0: ffff8880a4e88090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005ef12e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8705: #0: ffff8880a6184090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005efd2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8706: #0: ffff88808f80c090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005f052e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by getty/8707: #0: ffff8880a752a090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340 #1: ffffc90005ed92e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156 2 locks held by syz-executor.3/8748: #0: ffff8880a66bd7b8 (&bdev->bd_mutex){+.+.}, at: __blkdev_get+0x19b/0x1660 fs/block_dev.c:1553 #1: ffffffff8950d0a0 (loop_ctl_mutex){+.+.}, at: lo_open+0x1d/0xd0 drivers/block/loop.c:1774 1 lock held by syz-executor.4/8752: #0: ffff888090427238 (&bdev->bd_mutex){+.+.}, at: __blkdev_get+0x19b/0x1660 fs/block_dev.c:1553 2 locks held by kworker/0:4/8759: #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: set_work_data kernel/workqueue.c:620 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:647 [inline] #0: ffff8880aa436928 ((wq_completion)rcu_gp){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2240 #1: ffff88805c6dfdc0 ((work_completion)(&rew.rew_work)){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2244 3 locks held by kworker/u4:7/8790: 2 locks held by syz-executor.1/20804: #0: ffff8880a29737f8 (&bdev->bd_mutex){+.+.}, at: __blkdev_get+0x19b/0x1660 fs/block_dev.c:1553 #1: ffffffff8950d0a0 (loop_ctl_mutex){+.+.}, at: lo_open+0x1d/0xd0 drivers/block/loop.c:1774 1 lock held by syz-executor.2/23515: #0: ffffffff88feea40 (event_mutex){+.+.}, at: perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:236 1 lock held by syz-executor.2/23527: #0: ffffffff8950d0a0 (loop_ctl_mutex){+.+.}, at: __loop_clr_fd+0x88/0xd60 drivers/block/loop.c:1083 2 locks held by syz-executor.0/23517: #0: ffffffff89998860 (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72 #1: ffffffff88fadc38 (rcu_state.exp_mutex){+.+.}, at: exp_funnel_lock kernel/rcu/tree_exp.h:285 [inline] #1: ffffffff88fadc38 (rcu_state.exp_mutex){+.+.}, at: synchronize_rcu_expedited+0x4d6/0x5f0 kernel/rcu/tree_exp.h:817 2 locks held by blkid/23536: #0: ffff888090427238 (&bdev->bd_mutex){+.+.}, at: __blkdev_get+0x19b/0x1660 fs/block_dev.c:1553 #1: ffffffff8950d0a0 (loop_ctl_mutex){+.+.}, at: lo_open+0x1d/0xd0 drivers/block/loop.c:1774 ============================================= NMI backtrace for cpu 0 CPU: 0 PID: 1066 Comm: khungtaskd Not tainted 5.3.0-next-20190920 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline] check_hung_uninterruptible_tasks kernel/hung_task.c:269 [inline] watchdog+0xc99/0x1360 kernel/hung_task.c:353 kthread+0x361/0x430 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 3893 Comm: udevd Not tainted 5.3.0-next-20190920 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:26 [inline] RIP: 0010:check_kcov_mode kernel/kcov.c:68 [inline] RIP: 0010:write_comp_data+0x9/0x70 kernel/kcov.c:123 Code: 13 00 00 8b 80 04 13 00 00 48 8b 11 48 83 c2 01 48 39 d0 76 07 48 89 34 d1 48 89 11 5d c3 0f 1f 00 65 4c 8b 04 25 40 fe 01 00 <65> 8b 05 e8 27 8f 7e a9 00 01 1f 00 75 51 41 8b 80 00 13 00 00 83 RSP: 0018:ffff88809653f588 EFLAGS: 00000093 RAX: ffff888096536080 RBX: 0000000000000000 RCX: ffffffff87426fa0 RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000001 RBP: ffff88809653f590 R08: ffff888096536080 R09: ffffed1012ca7f17 R10: ffffed1012ca7f16 R11: ffff88809653f8b6 R12: ffff88809653f6d0 R13: ffff88809653f8b7 R14: ffff88809653f628 R15: ffff88809653f710 FS: 00007f788284a7a0(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f62d1ae0000 CR3: 0000000096835000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: format_decode+0x100/0xbc0 lib/vsprintf.c:2234 vsnprintf+0x17e/0x19a0 lib/vsprintf.c:2462 sprintf+0xc0/0x100 lib/vsprintf.c:2712 print_time kernel/printk/printk.c:1271 [inline] print_prefix kernel/printk/printk.c:1297 [inline] msg_print_text+0x535/0x5c0 kernel/printk/printk.c:1316 console_unlock+0x38d/0xf10 kernel/printk/printk.c:2448 vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996 vprintk_default+0x28/0x30 kernel/printk/printk.c:2023 vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386 printk+0xba/0xed kernel/printk/printk.c:2056 kobject_uevent_env+0xe2/0x101d lib/kobject_uevent.c:476 kobject_synth_uevent.cold+0xa0/0x100 lib/kobject_uevent.c:208 uevent_store+0x25/0x60 drivers/base/core.c:1280 dev_attr_store+0x57/0x80 drivers/base/core.c:983 sysfs_kf_write+0x116/0x170 fs/sysfs/file.c:138 kernfs_fop_write+0x2b8/0x480 fs/kernfs/file.c:315 __vfs_write+0x8a/0x110 fs/read_write.c:494 vfs_write+0x268/0x5d0 fs/read_write.c:558 ksys_write+0x14f/0x290 fs/read_write.c:611 __do_sys_write fs/read_write.c:623 [inline] __se_sys_write fs/read_write.c:620 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:620 do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x7f7881f52370 Code: 73 01 c3 48 8b 0d c8 4a 2b 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90 90 83 3d 85 a2 2b 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 0e 8a 01 00 48 89 04 24 RSP: 002b:00007ffd3558f8b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000001530690 RCX: 00007f7881f52370 RDX: 0000000000000006 RSI: 000000000041f4f9 RDI: 000000000000000b RBP: 000000000152f4c0 R08: 000000000041f4f1 R09: 00007f7881fa87d0 R10: 7269762f73656369 R11: 0000000000000246 R12: 000000000000000b R13: 0000000000000000 R14: 0000000001530690 R15: 000000000151e250