bisecting cause commit starting from 26e0105550862a137eba701e2f4e3eeb343759e9 building syzkaller on f3f7d9c8c3dfa1406a5f5a5f5f45a351f533139f testing commit 26e0105550862a137eba701e2f4e3eeb343759e9 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: general protection fault in cbq_dequeue run #4: crashed: KASAN: use-after-free Read in cbq_enqueue run #5: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: use-after-free Read in cbq_enqueue run #8: crashed: general protection fault in cbq_dequeue run #9: crashed: KASAN: use-after-free Read in cbq_enqueue testing release v5.3 testing commit 4d856f72c10ecb060868ed10ff1b1453943fc6c8 with gcc (GCC) 8.1.0 run #0: crashed: general protection fault in cbq_dequeue run #1: crashed: KASAN: use-after-free Read in cbq_enqueue run #2: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #3: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #4: crashed: general protection fault in cbq_dequeue run #5: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #8: crashed: KASAN: use-after-free Read in cbq_enqueue run #9: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: KASAN: use-after-free Read in cbq_enqueue run #4: crashed: general protection fault in cbq_dequeue run #5: crashed: KASAN: use-after-free Read in cbq_enqueue run #6: crashed: general protection fault in cbq_dequeue run #7: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #8: crashed: KASAN: use-after-free Read in cbq_enqueue run #9: crashed: KASAN: use-after-free Read in cbq_enqueue testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in cbq_enqueue run #1: crashed: general protection fault in cbq_dequeue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: KASAN: use-after-free Read in cbq_enqueue run #4: crashed: KASAN: use-after-free Read in cbq_enqueue run #5: crashed: KASAN: use-after-free Read in cbq_enqueue run #6: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #7: crashed: KASAN: use-after-free Read in cbq_enqueue run #8: crashed: KASAN: use-after-free Read in cbq_enqueue run #9: crashed: KASAN: use-after-free Read in cbq_enqueue testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #1: crashed: KASAN: use-after-free Read in cbq_enqueue run #2: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #3: crashed: KASAN: use-after-free Read in cbq_enqueue run #4: crashed: KASAN: use-after-free Read in cbq_enqueue run #5: crashed: KASAN: use-after-free Read in cbq_enqueue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: use-after-free Read in cbq_enqueue run #8: crashed: KASAN: use-after-free Read in cbq_enqueue run #9: crashed: general protection fault in cbq_enqueue testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: KASAN: use-after-free Read in cbq_enqueue run #4: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #5: crashed: KASAN: use-after-free Read in cbq_enqueue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #8: crashed: general protection fault in cbq_dequeue run #9: crashed: KASAN: use-after-free Read in cbq_enqueue testing release v4.19 testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0 run #0: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: KASAN: use-after-free Read in cbq_enqueue run #4: crashed: KASAN: use-after-free Read in cbq_enqueue run #5: crashed: KASAN: use-after-free Read in cbq_enqueue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: use-after-free Read in cbq_enqueue run #8: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #9: crashed: KASAN: use-after-free Read in cbq_enqueue testing release v4.18 testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0 run #0: crashed: KASAN: use-after-free Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: use-after-free Read in cbq_enqueue run #3: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #4: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #5: crashed: general protection fault in cbq_dequeue run #6: crashed: KASAN: use-after-free Read in cbq_enqueue run #7: crashed: KASAN: use-after-free Read in cbq_enqueue run #8: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #9: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.17 testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.16 testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.15 testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.14 testing commit bebc6082da0a9f5d47a1ea2edc099bf671058bd4 with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.13 testing commit 569dbb88e80deb68974ef6fdd6a13edb9d686261 with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.12 testing commit 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c with gcc (GCC) 8.1.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.11 testing commit a351e9b9fc24e982ec2f0e76379a49826036da12 with gcc (GCC) 7.3.0 run #0: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #1: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #2: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #3: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #4: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #5: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #6: crashed: KASAN: slab-out-of-bounds in cbq_enqueue at addr ADDR run #7: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #8: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue run #9: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.10 testing commit c470abd4fde40ea6a0846a2beab642a578c0b8cd with gcc (GCC) 5.5.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.9 testing commit 69973b830859bc6529a7a0468ba0d80ee5117826 with gcc (GCC) 5.5.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.8 testing commit c8d2bc9bc39ebea8437fd974fdbc21847bb897a3 with gcc (GCC) 5.5.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.7 testing commit 523d939ef98fd712632d93a5a2b588e477a7565e with gcc (GCC) 5.5.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.6 testing commit 2dcd0af568b0cf583645c8a317dd12e344b1c72a with gcc (GCC) 5.5.0 all runs: crashed: KASAN: slab-out-of-bounds Read in cbq_enqueue testing release v4.5 testing commit b562e44f507e863c6792946e4e1b1449fbbac85d with gcc (GCC) 5.5.0 all runs: crashed: general protection fault in cbq_dequeue testing release v4.4 testing commit afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc with gcc (GCC) 5.5.0 all runs: OK # git bisect start v4.5 v4.4 Bisecting: 6608 revisions left to test after this (roughly 13 steps) [e535d74bc50df2357d3253f8f3ca48c66d0d892a] Merge tag 'docs-4.5' of git://git.lwn.net/linux testing commit e535d74bc50df2357d3253f8f3ca48c66d0d892a with gcc (GCC) 5.5.0 all runs: OK # git bisect good e535d74bc50df2357d3253f8f3ca48c66d0d892a Bisecting: 3304 revisions left to test after this (roughly 12 steps) [62c79bb3a99fb46a8624f9c7e86fa5ee2f936360] Merge tag 'armsoc-dt64' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc testing commit 62c79bb3a99fb46a8624f9c7e86fa5ee2f936360 with gcc (GCC) 5.5.0 all runs: OK # git bisect good 62c79bb3a99fb46a8624f9c7e86fa5ee2f936360 Bisecting: 1651 revisions left to test after this (roughly 11 steps) [d5bfb96bdad3588961f49a6eff89a625fbaa12bf] Merge tag 'for-linus-4.5-2' of git://git.code.sf.net/p/openipmi/linux-ipmi testing commit d5bfb96bdad3588961f49a6eff89a625fbaa12bf with gcc (GCC) 5.5.0 run #0: crashed: possible deadlock in nf_conntrack_lock run #1: crashed: possible deadlock in nf_conntrack_lock run #2: crashed: possible deadlock in nf_conntrack_lock run #3: crashed: possible deadlock in nf_conntrack_lock run #4: crashed: possible deadlock in nf_conntrack_lock run #5: crashed: possible deadlock in nf_conntrack_lock run #6: crashed: possible deadlock in nf_conntrack_lock run #7: crashed: possible deadlock in nf_conntrack_lock run #8: crashed: possible deadlock in nf_conntrack_lock run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor793763697" "root@10.128.1.34:./syz-executor793763697"]: exit status 1 ssh: connect to host 10.128.1.34 port 22: Connection timed out lost connection # git bisect bad d5bfb96bdad3588961f49a6eff89a625fbaa12bf Bisecting: 828 revisions left to test after this (roughly 10 steps) [d0e5fbb01a67e400e82fefe4896ea40c6447ab98] block: fix bio splitting on max sectors testing commit d0e5fbb01a67e400e82fefe4896ea40c6447ab98 with gcc (GCC) 5.5.0 all runs: OK # git bisect good d0e5fbb01a67e400e82fefe4896ea40c6447ab98 Bisecting: 405 revisions left to test after this (roughly 9 steps) [b943d0b9c7c5162d81a1dc7e83315550131c7164] Merge branch 'drm-fixes' of git://people.freedesktop.org/~airlied/linux testing commit b943d0b9c7c5162d81a1dc7e83315550131c7164 with gcc (GCC) 5.5.0 all runs: OK # git bisect good b943d0b9c7c5162d81a1dc7e83315550131c7164 Bisecting: 202 revisions left to test after this (roughly 8 steps) [510ae0c994cb38fcf5033ad69eb8103552e335e8] Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus testing commit 510ae0c994cb38fcf5033ad69eb8103552e335e8 with gcc (GCC) 5.5.0 all runs: OK # git bisect good 510ae0c994cb38fcf5033ad69eb8103552e335e8 Bisecting: 101 revisions left to test after this (roughly 7 steps) [74608fc98d2856fa7201a498b61c9dd9455b504a] bnxt_en: Ring free response from close path should use completion ring testing commit 74608fc98d2856fa7201a498b61c9dd9455b504a with gcc (GCC) 5.5.0 run #0: crashed: possible deadlock in nf_conntrack_lock run #1: crashed: possible deadlock in nf_conntrack_lock run #2: crashed: possible deadlock in nf_conntrack_lock run #3: crashed: possible deadlock in nf_conntrack_lock run #4: crashed: possible deadlock in nf_conntrack_lock run #5: crashed: possible deadlock in nf_conntrack_lock run #6: crashed: possible deadlock in nf_conntrack_lock run #7: crashed: possible deadlock in nf_conntrack_lock run #8: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor065710926" "root@10.128.1.33:./syz-executor065710926"]: exit status 1 ssh: connect to host 10.128.1.33 port 22: Connection timed out lost connection run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor395377749" "root@10.128.10.14:./syz-executor395377749"]: exit status 1 ssh: connect to host 10.128.10.14 port 22: Connection timed out lost connection # git bisect bad 74608fc98d2856fa7201a498b61c9dd9455b504a Bisecting: 50 revisions left to test after this (roughly 6 steps) [4dc236c31733f5d2a3872c88cf12f607d7d36905] mlxsw: spectrum: Handle port leaving LAG while bridged testing commit 4dc236c31733f5d2a3872c88cf12f607d7d36905 with gcc (GCC) 5.5.0 run #0: crashed: possible deadlock in nf_conntrack_lock run #1: crashed: possible deadlock in nf_conntrack_lock run #2: crashed: possible deadlock in nf_conntrack_lock run #3: crashed: possible deadlock in nf_conntrack_lock run #4: crashed: possible deadlock in nf_conntrack_lock run #5: crashed: possible deadlock in nf_conntrack_lock run #6: crashed: possible deadlock in nf_conntrack_lock run #7: crashed: possible deadlock in nf_conntrack_lock run #8: crashed: possible deadlock in nf_conntrack_lock run #9: crashed: BUG: spinlock lockup suspected in nf_conntrack_lock # git bisect bad 4dc236c31733f5d2a3872c88cf12f607d7d36905 Bisecting: 24 revisions left to test after this (roughly 5 steps) [0e03f563a04207cc8e5db6afe63309a585995de7] net: mvneta: sort the headers in alphabetic order testing commit 0e03f563a04207cc8e5db6afe63309a585995de7 with gcc (GCC) 5.5.0 all runs: crashed: possible deadlock in nf_conntrack_lock # git bisect bad 0e03f563a04207cc8e5db6afe63309a585995de7 Bisecting: 9 revisions left to test after this (roughly 4 steps) [8034e1efcb330d2aecef8cbf8a83f206270c1775] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf testing commit 8034e1efcb330d2aecef8cbf8a83f206270c1775 with gcc (GCC) 5.5.0 run #0: crashed: possible deadlock in nf_conntrack_lock run #1: crashed: possible deadlock in nf_conntrack_lock run #2: crashed: possible deadlock in nf_conntrack_lock run #3: crashed: possible deadlock in nf_conntrack_lock run #4: crashed: possible deadlock in nf_conntrack_lock run #5: crashed: possible deadlock in nf_conntrack_lock run #6: crashed: possible deadlock in nf_conntrack_lock run #7: crashed: possible deadlock in nf_conntrack_lock run #8: crashed: possible deadlock in nf_conntrack_lock run #9: crashed: BUG: spinlock lockup suspected in nf_conntrack_lock # git bisect bad 8034e1efcb330d2aecef8cbf8a83f206270c1775 Bisecting: 7 revisions left to test after this (roughly 3 steps) [81e8f2e930fe76b9814c71b9d87c30760b5eb705] net: dp83640: Fix tx timestamp overflow handling. testing commit 81e8f2e930fe76b9814c71b9d87c30760b5eb705 with gcc (GCC) 5.5.0 all runs: OK # git bisect good 81e8f2e930fe76b9814c71b9d87c30760b5eb705 Bisecting: 3 revisions left to test after this (roughly 2 steps) [d6b3347bf178266259af64b1f27b5cf54acf62c8] netfilter: xt_TCPMSS: handle CHECKSUM_COMPLETE in tcpmss_tg6() testing commit d6b3347bf178266259af64b1f27b5cf54acf62c8 with gcc (GCC) 5.5.0 all runs: OK # git bisect good d6b3347bf178266259af64b1f27b5cf54acf62c8 Bisecting: 1 revision left to test after this (roughly 1 step) [b16c29191dc89bd877af99a7b04ce4866728a3e0] netfilter: nf_conntrack: use safer way to lock all buckets testing commit b16c29191dc89bd877af99a7b04ce4866728a3e0 with gcc (GCC) 5.5.0 run #0: crashed: possible deadlock in nf_conntrack_lock run #1: crashed: possible deadlock in nf_conntrack_lock run #2: crashed: possible deadlock in nf_conntrack_lock run #3: crashed: possible deadlock in nf_conntrack_lock run #4: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/syzkaller/jobs/linux/workdir/image/key" "/tmp/syz-executor517338906" "root@10.128.10.24:./syz-executor517338906"]: exit status 1 ssh: connect to host 10.128.10.24 port 22: Connection timed out lost connection run #5: crashed: possible deadlock in nf_conntrack_lock run #6: crashed: possible deadlock in nf_conntrack_lock run #7: crashed: possible deadlock in nf_conntrack_lock run #8: crashed: possible deadlock in nf_conntrack_lock run #9: crashed: possible deadlock in nf_conntrack_lock # git bisect bad b16c29191dc89bd877af99a7b04ce4866728a3e0 Bisecting: 0 revisions left to test after this (roughly 0 steps) [35b815392a6b6c268baf3b63d7f2ba350597024f] netfilter: nf_tables_netdev: fix error path in module initialization testing commit 35b815392a6b6c268baf3b63d7f2ba350597024f with gcc (GCC) 5.5.0 all runs: OK # git bisect good 35b815392a6b6c268baf3b63d7f2ba350597024f b16c29191dc89bd877af99a7b04ce4866728a3e0 is the first bad commit commit b16c29191dc89bd877af99a7b04ce4866728a3e0 Author: Sasha Levin Date: Mon Jan 18 19:23:51 2016 -0500 netfilter: nf_conntrack: use safer way to lock all buckets When we need to lock all buckets in the connection hashtable we'd attempt to lock 1024 spinlocks, which is way more preemption levels than supported by the kernel. Furthermore, this behavior was hidden by checking if lockdep is enabled, and if it was - use only 8 buckets(!). Fix this by using a global lock and synchronize all buckets on it when we need to lock them all. This is pretty heavyweight, but is only done when we need to resize the hashtable, and that doesn't happen often enough (or at all). Signed-off-by: Sasha Levin Acked-by: Jesper Dangaard Brouer Reviewed-by: Florian Westphal Signed-off-by: Pablo Neira Ayuso :040000 040000 9c648128d8818ac7a2bc407fab9bf14198a34d41 5fd5658cad39bf023288229f876a86fb1944bc42 M include :040000 040000 f2f3b3615b89bd8e02536ddce37e27f1e4046b65 570e405c083c650c78a7764026eefb42dc4f079b M net revisions tested: 36, total time: 5h39m16.533768739s (build: 2h19m33.629633493s, test: 3h10m55.530535854s) first bad commit: b16c29191dc89bd877af99a7b04ce4866728a3e0 netfilter: nf_conntrack: use safer way to lock all buckets cc: ["brouer@redhat.com" "fw@strlen.de" "pablo@netfilter.org" "sasha.levin@oracle.com"] crash: possible deadlock in nf_conntrack_lock ============================================= [ INFO: possible recursive locking detected ] 4.4.0+ #0 Not tainted --------------------------------------------- kworker/u4:4/2315 is trying to acquire lock: (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] spin_lock include/linux/spinlock.h:302 [inline] (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] nf_conntrack_lock+0xd/0x50 net/netfilter/nf_conntrack_core.c:74 but task is already holding lock: (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] spin_lock include/linux/spinlock.h:302 [inline] (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] nf_conntrack_lock+0xd/0x50 net/netfilter/nf_conntrack_core.c:74 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&nf_conntrack_locks[i])->rlock); lock(&(&nf_conntrack_locks[i])->rlock); *** DEADLOCK *** May be due to missing lock nesting notation 4 locks held by kworker/u4:4/2315: #0: ("%s""netns"){.+.+.+}, at: [] set_work_data kernel/workqueue.c:615 [inline] #0: ("%s""netns"){.+.+.+}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline] #0: ("%s""netns"){.+.+.+}, at: [] process_one_work+0x17e/0x690 kernel/workqueue.c:2029 #1: (net_cleanup_work){+.+.+.}, at: [] set_work_data kernel/workqueue.c:615 [inline] #1: (net_cleanup_work){+.+.+.}, at: [] set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline] #1: (net_cleanup_work){+.+.+.}, at: [] process_one_work+0x17e/0x690 kernel/workqueue.c:2029 #2: (net_mutex){+.+.+.}, at: [] cleanup_net+0x7a/0x270 net/core/net_namespace.c:397 #3: (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] spin_lock include/linux/spinlock.h:302 [inline] #3: (&(&nf_conntrack_locks[i])->rlock){+.....}, at: [] nf_conntrack_lock+0xd/0x50 net/netfilter/nf_conntrack_core.c:74 stack backtrace: CPU: 1 PID: 2315 Comm: kworker/u4:4 Not tainted 4.4.0+ #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: netns cleanup_net ffffffff83bf3f90 ffff8800baa9bb20 ffffffff8185e0cc ffffffff83bf3f90 ffff8800baa9bbf8 ffffffff811bf55c 00000000000b9f40 ffff8800baa9bbd8 0000000000000003 0000000000000000 ffff8800baa9bb60 ffffffff811da3bd Call Trace: [] __dump_stack lib/dump_stack.c:15 [inline] [] dump_stack+0x4e/0x82 lib/dump_stack.c:50 [] print_deadlock_bug kernel/locking/lockdep.c:1752 [inline] [] check_deadlock kernel/locking/lockdep.c:1796 [inline] [] validate_chain kernel/locking/lockdep.c:2128 [inline] [] __lock_acquire+0x1a5c/0x1ba0 kernel/locking/lockdep.c:3206 [] lock_acquire+0xed/0x210 kernel/locking/lockdep.c:3585 [] __raw_spin_lock include/linux/spinlock_api_smp.h:144 [inline] [] _raw_spin_lock+0x36/0x50 kernel/locking/spinlock.c:151 [] spin_lock include/linux/spinlock.h:302 [inline] [] nf_conntrack_lock+0xd/0x50 net/netfilter/nf_conntrack_core.c:74 [] ctnl_untimeout.isra.4+0xdc/0x110 net/netfilter/nfnetlink_cttimeout.c:315 [] cttimeout_net_exit+0x28/0x80 net/netfilter/nfnetlink_cttimeout.c:581 [] ops_exit_list.isra.4+0x33/0x60 net/core/net_namespace.c:134 [] cleanup_net+0x1a9/0x270 net/core/net_namespace.c:431 [] process_one_work+0x21b/0x690 kernel/workqueue.c:2036 [] worker_thread+0x49/0x490 kernel/workqueue.c:2170 [] kthread+0xea/0x100 drivers/block/aoe/aoecmd.c:1312 [] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:468 IPv6: ADDRCONF(NETDEV_CHANGE): nr2: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): nr0: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): nr1: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): nr4: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): nr3: link becomes ready IPv6: ADDRCONF(NETDEV_CHANGE): nr5: link becomes ready