================================= [ INFO: inconsistent lock state ] IPVS: stopping master sync thread 25198 ... 4.9.111-g03c70fe #58 Not tainted --------------------------------- inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage. ksoftirqd/0/3 [HC0[0]:SC1[1]:HE1:SE0] takes: IPVS: stopping master sync thread 25201 ... (&(&rnp->exp_lock)->rlock){+.?...}, at: [] spin_lock include/linux/spinlock.h:302 [inline] (&(&rnp->exp_lock)->rlock){+.?...}, at: [] exp_funnel_lock kernel/rcu/tree_exp.h:271 [inline] (&(&rnp->exp_lock)->rlock){+.?...}, at: [] _synchronize_rcu_expedited+0x1fb/0x840 kernel/rcu/tree_exp.h:569 IPVS: stopping master sync thread 25205 ... {SOFTIRQ-ON-W} state was registered at: mark_irqflags kernel/locking/lockdep.c:2941 [inline] __lock_acquire+0xdd0/0x4070 kernel/locking/lockdep.c:3302 lock_acquire+0x130/0x3e0 kernel/locking/lockdep.c:3756 __raw_spin_lock include/linux/spinlock_api_smp.h:144 [inline] _raw_spin_lock+0x36/0x50 kernel/locking/spinlock.c:151 spin_lock include/linux/spinlock.h:302 [inline] rcu_exp_wait_wake+0x2d2/0x5e0 kernel/rcu/tree_exp.h:504 rcu_exp_sel_wait_wake kernel/rcu/tree_exp.h:535 [inline] _synchronize_rcu_expedited+0x73c/0x840 kernel/rcu/tree_exp.h:575 synchronize_rcu_expedited kernel/rcu/tree_exp.h:687 [inline] synchronize_rcu.part.62+0xd3/0x110 kernel/rcu/tree_plugin.h:682 synchronize_rcu+0x27/0x90 kernel/rcu/tree_plugin.h:685 rcu_test_sync_prims+0x9/0x30 kernel/rcu/update.c:829 rcu_scheduler_starting+0x51/0x80 kernel/rcu/tree.c:4016 rest_init+0xf/0x189 init/main.c:388 start_kernel+0x67e/0x6b2 init/main.c:664 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:196 x86_64_start_kernel+0x13f/0x162 arch/x86/kernel/head64.c:177 irq event stamp: 9983290 hardirqs last enabled at (9983290): [] __local_bh_enable_ip+0x6a/0xd0 kernel/softirq.c:186 hardirqs last disabled at (9983289): [] __local_bh_enable_ip+0x32/0xd0 kernel/softirq.c:163 softirqs last enabled at (9982906): [] __do_softirq+0x46c/0x937 kernel/softirq.c:310 softirqs last disabled at (9982911): [] run_ksoftirqd+0x2e/0x60 kernel/softirq.c:676 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&rnp->exp_lock)->rlock); lock(&(&rnp->exp_lock)->rlock); *** DEADLOCK *** 1 lock held by ksoftirqd/0/3: #0: (rcu_callback){......}, at: [] __rcu_reclaim kernel/rcu/rcu.h:108 [inline] #0: (rcu_callback){......}, at: [] rcu_do_batch kernel/rcu/tree.c:2789 [inline] #0: (rcu_callback){......}, at: [] invoke_rcu_callbacks kernel/rcu/tree.c:3053 [inline] #0: (rcu_callback){......}, at: [] __rcu_process_callbacks kernel/rcu/tree.c:3020 [inline] #0: (rcu_callback){......}, at: [] rcu_process_callbacks+0x98e/0x12b0 kernel/rcu/tree.c:3037 stack backtrace: CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.9.111-g03c70fe #58 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d99ef498 ffffffff81eb2729 ffff8801d99b3000 ffffffff85595210 ffff8801d99b38f0 ffff8801d99b3910 0000000000000000 ffff8801d99ef508 ffffffff81427762 0000000000000001 0000000000000001 ffff880100000000 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] print_usage_bug.cold.56+0x327/0x421 kernel/locking/lockdep.c:2387 [] valid_state kernel/locking/lockdep.c:2400 [inline] [] mark_lock_irq kernel/locking/lockdep.c:2602 [inline] [] mark_lock+0xcc6/0x1280 kernel/locking/lockdep.c:3065 [] mark_irqflags kernel/locking/lockdep.c:2923 [inline] [] __lock_acquire+0xd40/0x4070 kernel/locking/lockdep.c:3302 [] lock_acquire+0x130/0x3e0 kernel/locking/lockdep.c:3756 [] __raw_spin_lock include/linux/spinlock_api_smp.h:144 [inline] [] _raw_spin_lock+0x36/0x50 kernel/locking/spinlock.c:151 [] spin_lock include/linux/spinlock.h:302 [inline] [] exp_funnel_lock kernel/rcu/tree_exp.h:271 [inline] [] _synchronize_rcu_expedited+0x1fb/0x840 kernel/rcu/tree_exp.h:569 [] synchronize_rcu_expedited kernel/rcu/tree_exp.h:687 [inline] [] synchronize_rcu.part.62+0xd3/0x110 kernel/rcu/tree_plugin.h:682 [] synchronize_rcu+0x27/0x90 kernel/rcu/tree_plugin.h:685 [] __l2tp_session_unhash+0x38a/0x520 net/l2tp/l2tp_core.c:1796 [] l2tp_tunnel_closeall+0x1f7/0x350 net/l2tp/l2tp_core.c:1364 [] l2tp_tunnel_destruct+0x2f2/0x590 net/l2tp/l2tp_core.c:1324 [] __sk_destruct+0x55/0x590 net/core/sock.c:1428 [] __rcu_reclaim kernel/rcu/rcu.h:118 [inline] [] rcu_do_batch kernel/rcu/tree.c:2789 [inline] [] invoke_rcu_callbacks kernel/rcu/tree.c:3053 [inline] [] __rcu_process_callbacks kernel/rcu/tree.c:3020 [inline] [] rcu_process_callbacks+0x8ae/0x12b0 kernel/rcu/tree.c:3037 [] __do_softirq+0x20b/0x937 kernel/softirq.c:284 [] run_ksoftirqd+0x2e/0x60 kernel/softirq.c:676 [] smpboot_thread_fn+0x5c9/0x900 kernel/smpboot.c:163 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 BUG: sleeping function called from invalid context at kernel/rcu/tree_exp.h:281 in_atomic(): 1, irqs_disabled(): 0, pid: 3, name: ksoftirqd/0 INFO: lockdep is turned off. Preemption disabled at:[ 461.900632] [] __do_softirq+0xdb/0x937 kernel/softirq.c:261 CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.9.111-g03c70fe #58 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 ffff8801d99ef738 ffffffff81eb2729 ffffffff839ff78b 0000000000000000 0000000000000100 ffff8801d99b3000 ffff8801d99b3000 ffff8801d99ef770 ffffffff81424390 ffff8801d99b3000 ffffffff83c6d460 0000000000000119 Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0xc1/0x128 lib/dump_stack.c:51 [] ___might_sleep.cold.123+0x1bc/0x1f5 kernel/sched/core.c:7992 [] __might_sleep+0x95/0x1a0 kernel/sched/core.c:7949 [] exp_funnel_lock kernel/rcu/tree_exp.h:279 [inline] [] _synchronize_rcu_expedited+0x600/0x840 kernel/rcu/tree_exp.h:569 [] synchronize_rcu_expedited kernel/rcu/tree_exp.h:687 [inline] [] synchronize_rcu.part.62+0xd3/0x110 kernel/rcu/tree_plugin.h:682 [] synchronize_rcu+0x27/0x90 kernel/rcu/tree_plugin.h:685 [] __l2tp_session_unhash+0x38a/0x520 net/l2tp/l2tp_core.c:1796 [] l2tp_tunnel_closeall+0x1f7/0x350 net/l2tp/l2tp_core.c:1364 [] l2tp_tunnel_destruct+0x2f2/0x590 net/l2tp/l2tp_core.c:1324 [] __sk_destruct+0x55/0x590 net/core/sock.c:1428 [] __rcu_reclaim kernel/rcu/rcu.h:118 [inline] [] rcu_do_batch kernel/rcu/tree.c:2789 [inline] [] invoke_rcu_callbacks kernel/rcu/tree.c:3053 [inline] [] __rcu_process_callbacks kernel/rcu/tree.c:3020 [inline] [] rcu_process_callbacks+0x8ae/0x12b0 kernel/rcu/tree.c:3037 [] __do_softirq+0x20b/0x937 kernel/softirq.c:284 [] run_ksoftirqd+0x2e/0x60 kernel/softirq.c:676 [] smpboot_thread_fn+0x5c9/0x900 kernel/smpboot.c:163 [] kthread+0x26d/0x300 kernel/kthread.c:211 [] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:373 BUG: scheduling while atomic: ksoftirqd/0/3/0x00000101 INFO: lockdep is turned off. Modules linked in: Preemption disabled at:[ 462.237803] [] __do_softirq+0xdb/0x937 kernel/softirq.c:261