xs_tcp_setup_socket: connect returned unhandled error -113 xs_tcp_setup_socket: connect returned unhandled error -113 ====================================================== WARNING: possible circular locking dependency detected 4.13.0-next-20170913+ #21 Not tainted ------------------------------------------------------ kworker/1:1/23 is trying to acquire lock: ("xprtiod"){+.+.}, at: [] start_flush_work kernel/workqueue.c:2857 [inline] ("xprtiod"){+.+.}, at: [] flush_work+0x5a2/0x8f0 kernel/workqueue.c:2889 but task is already holding lock: ((&task->u.tk_work)){+.+.}, at: [] process_one_work+0xb2c/0x1bd0 kernel/workqueue.c:2094 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 ((&task->u.tk_work)){+.+.}: check_prevs_add kernel/locking/lockdep.c:2020 [inline] validate_chain kernel/locking/lockdep.c:2469 [inline] __lock_acquire+0x328f/0x4620 kernel/locking/lockdep.c:3498 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002 process_one_work+0xba2/0x1bd0 kernel/workqueue.c:2095 worker_thread+0x223/0x1860 kernel/workqueue.c:2253 kthread+0x39c/0x470 kernel/kthread.c:231 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 -> #0 ("xprtiod"){+.+.}: check_prev_add+0x865/0x1520 kernel/locking/lockdep.c:1894 check_prevs_add kernel/locking/lockdep.c:2020 [inline] validate_chain kernel/locking/lockdep.c:2469 [inline] __lock_acquire+0x328f/0x4620 kernel/locking/lockdep.c:3498 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002 start_flush_work kernel/workqueue.c:2858 [inline] flush_work+0x5c7/0x8f0 kernel/workqueue.c:2889 __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2961 cancel_work_sync+0x17/0x20 kernel/workqueue.c:2997 xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] kref_put include/linux/kref.h:70 [inline] xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 rpc_release_task net/sunrpc/sched.c:1059 [inline] __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 process_one_work+0xbfa/0x1bd0 kernel/workqueue.c:2119 worker_thread+0x223/0x1860 kernel/workqueue.c:2253 kthread+0x39c/0x470 kernel/kthread.c:231 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock((&task->u.tk_work)); lock("xprtiod"); lock((&task->u.tk_work)); lock("xprtiod"); *** DEADLOCK *** 2 locks held by kworker/1:1/23: #0: ("rpciod"){+.+.}, at: [] __write_once_size include/linux/compiler.h:305 [inline] #0: ("rpciod"){+.+.}, at: [] atomic64_set arch/x86/include/asm/atomic64_64.h:33 [inline] #0: ("rpciod"){+.+.}, at: [] atomic_long_set include/asm-generic/atomic-long.h:56 [inline] #0: ("rpciod"){+.+.}, at: [] set_work_data kernel/workqueue.c:617 [inline] #0: ("rpciod"){+.+.}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline] #0: ("rpciod"){+.+.}, at: [] process_one_work+0xad4/0x1bd0 kernel/workqueue.c:2090 #1: ((&task->u.tk_work)){+.+.}, at: [] process_one_work+0xb2c/0x1bd0 kernel/workqueue.c:2094 stack backtrace: CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 4.13.0-next-20170913+ #21 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: rpciod rpc_async_schedule Call Trace: __dump_stack lib/dump_stack.c:16 [inline] dump_stack+0x194/0x257 lib/dump_stack.c:52 print_circular_bug+0x503/0x710 kernel/locking/lockdep.c:1259 check_prev_add+0x865/0x1520 kernel/locking/lockdep.c:1894 check_prevs_add kernel/locking/lockdep.c:2020 [inline] validate_chain kernel/locking/lockdep.c:2469 [inline] __lock_acquire+0x328f/0x4620 kernel/locking/lockdep.c:3498 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002 start_flush_work kernel/workqueue.c:2858 [inline] flush_work+0x5c7/0x8f0 kernel/workqueue.c:2889 __cancel_work_timer+0x30a/0x7e0 kernel/workqueue.c:2961 cancel_work_sync+0x17/0x20 kernel/workqueue.c:2997 xprt_destroy+0xa1/0x130 net/sunrpc/xprt.c:1467 xprt_destroy_kref net/sunrpc/xprt.c:1477 [inline] kref_put include/linux/kref.h:70 [inline] xprt_put+0x38/0x40 net/sunrpc/xprt.c:1501 rpc_task_release_client+0x299/0x430 net/sunrpc/clnt.c:986 rpc_release_resources_task+0x7f/0xa0 net/sunrpc/sched.c:1020 rpc_release_task net/sunrpc/sched.c:1059 [inline] __rpc_execute+0x4d9/0xe70 net/sunrpc/sched.c:824 rpc_async_schedule+0x16/0x20 net/sunrpc/sched.c:848 process_one_work+0xbfa/0x1bd0 kernel/workqueue.c:2119 worker_thread+0x223/0x1860 kernel/workqueue.c:2253 kthread+0x39c/0x470 kernel/kthread.c:231 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431 Buffer I/O error on dev loop0, logical block 0, async page read