bisecting cause commit starting from fde50b96be821ac9673a7e00847cc4605bd88f34 building syzkaller on c85e1c5be98819e885698b690ac15a8d05ae38a6 testing commit fde50b96be821ac9673a7e00847cc4605bd88f34 with gcc (GCC) 8.1.0 run #0: crashed: general protection fault in tls_trim_both_msgs run #1: crashed: general protection fault in tls_trim_both_msgs run #2: crashed: general protection fault in tls_trim_both_msgs run #3: crashed: general protection fault in tls_trim_both_msgs run #4: crashed: general protection fault in tls_trim_both_msgs run #5: crashed: general protection fault in tls_trim_both_msgs run #6: crashed: general protection fault in tls_trim_both_msgs run #7: crashed: general protection fault in tls_trim_both_msgs run #8: crashed: INFO: rcu detected stall in ipv6_rcv run #9: crashed: INFO: rcu detected stall in ipv6_rcv testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 all runs: OK # git bisect start fde50b96be821ac9673a7e00847cc4605bd88f34 v5.2 Bisecting: 8265 revisions left to test after this (roughly 13 steps) [aea01a2234d26ffa9d9ee01e43705824c0c7b08a] net/rds: Fix NULL/ERR_PTR inconsistency testing commit aea01a2234d26ffa9d9ee01e43705824c0c7b08a with gcc (GCC) 8.1.0 all runs: OK # git bisect good aea01a2234d26ffa9d9ee01e43705824c0c7b08a Bisecting: 4171 revisions left to test after this (roughly 12 steps) [24e44913aa746098349370a0f279733c0cadcba7] Merge tag 'armsoc-soc' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 24e44913aa746098349370a0f279733c0cadcba7 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 24e44913aa746098349370a0f279733c0cadcba7 Bisecting: 2079 revisions left to test after this (roughly 11 steps) [ae6262d1a0704a92804b549f86ca24aeb7f523e6] Merge remote-tracking branch 'hwmon-staging/hwmon-next' testing commit ae6262d1a0704a92804b549f86ca24aeb7f523e6 with gcc (GCC) 8.1.0 all runs: crashed: general protection fault in tls_trim_both_msgs # git bisect bad ae6262d1a0704a92804b549f86ca24aeb7f523e6 Bisecting: 1045 revisions left to test after this (roughly 10 steps) [40ca0ce56d4bb889dc43b455c55398468115569a] arm64: entry: SP Alignment Fault doesn't write to FAR_EL1 testing commit 40ca0ce56d4bb889dc43b455c55398468115569a with gcc (GCC) 8.1.0 run #0: boot failed: can't ssh into the instance run #1: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #2: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #3: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #4: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #5: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #6: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #7: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #8: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect run #9: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect # git bisect skip 40ca0ce56d4bb889dc43b455c55398468115569a Bisecting: 1045 revisions left to test after this (roughly 10 steps) [0a5c3c2f47667a14cd1a3127160af709e64e67b2] s390/bitops: make test functions return bool testing commit 0a5c3c2f47667a14cd1a3127160af709e64e67b2 with gcc (GCC) 8.1.0 all runs: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect # git bisect skip 0a5c3c2f47667a14cd1a3127160af709e64e67b2 Bisecting: 1045 revisions left to test after this (roughly 10 steps) [53c580c1bdbd6332947fdfa6634b61048762b9b5] arm64: dts: uniphier: update to new Denali NAND binding testing commit 53c580c1bdbd6332947fdfa6634b61048762b9b5 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 53c580c1bdbd6332947fdfa6634b61048762b9b5 Bisecting: 1044 revisions left to test after this (roughly 10 steps) [09d8492381132c6df60a54b6eb619ab245cd9aa2] arm64: dts: imx8mm-evk: Remove invalid properties testing commit 09d8492381132c6df60a54b6eb619ab245cd9aa2 with gcc (GCC) 8.1.0 all runs: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect # git bisect skip 09d8492381132c6df60a54b6eb619ab245cd9aa2 Bisecting: 1044 revisions left to test after this (roughly 10 steps) [a444d1aabe6b0cf5bc0814f3029001c0089efd41] xtensa: add exclusive atomics support testing commit a444d1aabe6b0cf5bc0814f3029001c0089efd41 with gcc (GCC) 8.1.0 all runs: OK # git bisect good a444d1aabe6b0cf5bc0814f3029001c0089efd41 Bisecting: 1043 revisions left to test after this (roughly 10 steps) [cc216dfd5615f9ba1abe14c1d5e19b21ac1e1e95] arm64: dts: mt8183: add capacity-dmips-mhz testing commit cc216dfd5615f9ba1abe14c1d5e19b21ac1e1e95 with gcc (GCC) 8.1.0 all runs: OK # git bisect good cc216dfd5615f9ba1abe14c1d5e19b21ac1e1e95 Bisecting: 1042 revisions left to test after this (roughly 10 steps) [fc2e6f03a16574849a6cb8010498236ef40bc716] Merge remote-tracking branch 'renesas/next' testing commit fc2e6f03a16574849a6cb8010498236ef40bc716 with gcc (GCC) 8.1.0 run #0: crashed: general protection fault in tls_trim_both_msgs run #1: crashed: general protection fault in tls_trim_both_msgs run #2: crashed: general protection fault in tls_trim_both_msgs run #3: crashed: general protection fault in tls_trim_both_msgs run #4: crashed: general protection fault in tls_trim_both_msgs run #5: crashed: general protection fault in tls_trim_both_msgs run #6: crashed: general protection fault in tls_trim_both_msgs run #7: crashed: general protection fault in tls_trim_both_msgs run #8: crashed: general protection fault in tls_trim_both_msgs run #9: crashed: INFO: rcu detected stall in ipv6_rcv # git bisect bad fc2e6f03a16574849a6cb8010498236ef40bc716 Bisecting: 811 revisions left to test after this (roughly 10 steps) [168c79971b4a7be7011e73bf488b740a8e1135c8] Merge tag 'kbuild-v5.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild testing commit 168c79971b4a7be7011e73bf488b740a8e1135c8 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 168c79971b4a7be7011e73bf488b740a8e1135c8 Bisecting: 403 revisions left to test after this (roughly 9 steps) [5438f428bb1cf1f564f2db323540c97d79b16d12] Merge remote-tracking branch 'bpf/master' testing commit 5438f428bb1cf1f564f2db323540c97d79b16d12 with gcc (GCC) 8.1.0 all runs: crashed: general protection fault in tls_trim_both_msgs # git bisect bad 5438f428bb1cf1f564f2db323540c97d79b16d12 Bisecting: 203 revisions left to test after this (roughly 8 steps) [02a0242549aaec4490e9280cebbb1474b36281ca] ARM: defconfig: u8500: Add new drivers testing commit 02a0242549aaec4490e9280cebbb1474b36281ca with gcc (GCC) 8.1.0 all runs: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect # git bisect skip 02a0242549aaec4490e9280cebbb1474b36281ca Bisecting: 203 revisions left to test after this (roughly 8 steps) [d64b212ea960db4276a1d8372bd98cb861dfcbb0] ARM: davinci: fix sleep.S build error on ARMv4 testing commit d64b212ea960db4276a1d8372bd98cb861dfcbb0 with gcc (GCC) 8.1.0 all runs: boot failed: WARNING: workqueue cpumask: online intersect > possible intersect # git bisect skip d64b212ea960db4276a1d8372bd98cb861dfcbb0 Bisecting: 203 revisions left to test after this (roughly 8 steps) [c8eee4135a456bc031d67cadc454e76880d1afd8] selftests/bpf: fix sendmsg6_prog on s390 testing commit c8eee4135a456bc031d67cadc454e76880d1afd8 with gcc (GCC) 8.1.0 all runs: crashed: general protection fault in tls_trim_both_msgs # git bisect bad c8eee4135a456bc031d67cadc454e76880d1afd8 Bisecting: 9 revisions left to test after this (roughly 3 steps) [0e858739c2d2eedeeac1d35bfa0ec3cc2a7190d8] bpf: sockmap, only create entry if ulp is not already enabled testing commit 0e858739c2d2eedeeac1d35bfa0ec3cc2a7190d8 with gcc (GCC) 8.1.0 run #0: crashed: general protection fault in tls_sk_proto_close run #1: crashed: general protection fault in tls_trim_both_msgs run #2: crashed: general protection fault in tls_trim_both_msgs run #3: crashed: general protection fault in tls_trim_both_msgs run #4: crashed: general protection fault in tls_trim_both_msgs run #5: crashed: general protection fault in tls_sk_proto_close run #6: crashed: general protection fault in tls_sk_proto_close run #7: crashed: general protection fault in tls_trim_both_msgs run #8: crashed: general protection fault in tls_sk_proto_close run #9: crashed: INFO: rcu detected stall in tcp_delack_timer # git bisect bad 0e858739c2d2eedeeac1d35bfa0ec3cc2a7190d8 Bisecting: 4 revisions left to test after this (roughly 2 steps) [f87e62d45e51b12d48d2cb46b5cde8f83b866bc4] net/tls: remove close callback sock unlock/lock around TX work flush testing commit f87e62d45e51b12d48d2cb46b5cde8f83b866bc4 with gcc (GCC) 8.1.0 all runs: OK # git bisect good f87e62d45e51b12d48d2cb46b5cde8f83b866bc4 Bisecting: 2 revisions left to test after this (roughly 1 step) [32857cf57f920cdc03b5095f08febec94cf9c36b] net/tls: fix transition through disconnect with close testing commit 32857cf57f920cdc03b5095f08febec94cf9c36b with gcc (GCC) 8.1.0 run #0: crashed: general protection fault in tls_sk_proto_close run #1: crashed: general protection fault in tls_sk_proto_close run #2: crashed: general protection fault in tls_trim_both_msgs run #3: crashed: general protection fault in tls_sk_proto_close run #4: crashed: general protection fault in tls_sk_proto_close run #5: crashed: general protection fault in tls_trim_both_msgs run #6: crashed: general protection fault in tls_trim_both_msgs run #7: crashed: general protection fault in tls_trim_both_msgs run #8: crashed: general protection fault in tls_trim_both_msgs run #9: crashed: INFO: rcu detected stall in ipv6_rcv # git bisect bad 32857cf57f920cdc03b5095f08febec94cf9c36b Bisecting: 0 revisions left to test after this (roughly 0 steps) [313ab004805cf52a42673b15852b3842474ccd87] net/tls: remove sock unlock/lock around strp_done() testing commit 313ab004805cf52a42673b15852b3842474ccd87 with gcc (GCC) 8.1.0 all runs: OK # git bisect good 313ab004805cf52a42673b15852b3842474ccd87 32857cf57f920cdc03b5095f08febec94cf9c36b is the first bad commit commit 32857cf57f920cdc03b5095f08febec94cf9c36b Author: John Fastabend Date: Fri Jul 19 10:29:18 2019 -0700 net/tls: fix transition through disconnect with close It is possible (via shutdown()) for TCP socks to go through TCP_CLOSE state via tcp_disconnect() without actually calling tcp_close which would then call the tls close callback. Because of this a user could disconnect a socket then put it in a LISTEN state which would break our assumptions about sockets always being ESTABLISHED state. More directly because close() can call unhash() and unhash is implemented by sockmap if a sockmap socket has TLS enabled we can incorrectly destroy the psock from unhash() and then call its close handler again. But because the psock (sockmap socket representation) is already destroyed we call close handler in sk->prot. However, in some cases (TLS BASE/BASE case) this will still point at the sockmap close handler resulting in a circular call and crash reported by syzbot. To fix both above issues implement the unhash() routine for TLS. v4: - add note about tls offload still needing the fix; - move sk_proto to the cold cache line; - split TX context free into "release" and "free", otherwise the GC work itself is in already freed memory; - more TX before RX for consistency; - reuse tls_ctx_free(); - schedule the GC work after we're done with context to avoid UAF; - don't set the unhash in all modes, all modes "inherit" TLS_BASE's callbacks anyway; - disable the unhash hook for TLS_HW. Fixes: 3c4d7559159bf ("tls: kernel TLS support") Reported-by: Eric Dumazet Signed-off-by: John Fastabend Signed-off-by: Jakub Kicinski Signed-off-by: Daniel Borkmann :040000 040000 223558f8dad48de97cc767914402214939a59200 a9899507d88e63dc4a106ab4a33856a2f9c6835e M Documentation :040000 040000 5ab078ab55a48f203552efb9798517cfd04acb1a f4f84ff0a870164b457e3d8b7fa3ceea97dde7f0 M include :040000 040000 4244655432d7a13da7386516135a3287394e32f4 6bc0e2fa57cb7360306caedb67ea0ec697eaf9ee M net revisions tested: 21, total time: 5h31m48.883072996s (build: 2h4m3.686647863s, test: 3h19m41.678495249s) first bad commit: 32857cf57f920cdc03b5095f08febec94cf9c36b net/tls: fix transition through disconnect with close cc: ["aviadye@mellanox.com" "borisp@mellanox.com" "corbet@lwn.net" "daniel@iogearbox.net" "davejwatson@fb.com" "davem@davemloft.net" "jakub.kicinski@netronome.com" "john.fastabend@gmail.com" "linux-doc@vger.kernel.org" "linux-kernel@vger.kernel.org" "netdev@vger.kernel.org"] crash: INFO: rcu detected stall in ipv6_rcv rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 1-...!: (10500 ticks this GP) idle=d2a/1/0x4000000000000002 softirq=9914/9914 fqs=0 (t=10500 jiffies g=7437 q=73) rcu: rcu_preempt kthread starved for 10500 jiffies! g7437 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 rcu: RCU grace-period kthread stack dump: rcu_preempt I29792 10 2 0x80004000 Call Trace: context_switch kernel/sched/core.c:3252 [inline] __schedule+0x713/0x1520 kernel/sched/core.c:3878 schedule+0x8f/0x260 kernel/sched/core.c:3942 schedule_timeout+0x38b/0xad0 kernel/time/timer.c:1807 rcu_gp_fqs_loop kernel/rcu/tree.c:1611 [inline] rcu_gp_kthread+0x997/0x1820 kernel/rcu/tree.c:1768 kthread+0x331/0x3f0 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 NMI backtrace for cpu 1 CPU: 1 PID: 16 Comm: ksoftirqd/1 Not tainted 5.2.0+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x113/0x167 lib/dump_stack.c:113 nmi_cpu_backtrace.cold.7+0x4b/0x84 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x18b/0x1b7 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x16f/0x1bc kernel/rcu/tree_stall.h:254 print_cpu_stall kernel/rcu/tree_stall.h:455 [inline] check_cpu_stall kernel/rcu/tree_stall.h:529 [inline] rcu_pending kernel/rcu/tree.c:2736 [inline] rcu_sched_clock_irq.cold.83+0x4dc/0xc55 kernel/rcu/tree.c:2183 update_process_times+0x2a/0x60 kernel/time/timer.c:1639 tick_sched_handle+0x77/0x140 kernel/time/tick-sched.c:167 tick_sched_timer+0x43/0x100 kernel/time/tick-sched.c:1296 __run_hrtimer kernel/time/hrtimer.c:1389 [inline] __hrtimer_run_queues+0x32f/0xb50 kernel/time/hrtimer.c:1451 hrtimer_interrupt+0x2df/0x760 kernel/time/hrtimer.c:1509 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1068 [inline] smp_apic_timer_interrupt+0x163/0x5f0 arch/x86/kernel/apic/apic.c:1093 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:828 RIP: 0010:check_memory_region+0x144/0x1d0 mm/kasan/generic.c:193 Code: 00 00 00 fc ff df 49 01 d9 49 01 c0 eb 0d 41 80 38 00 49 8d 40 01 75 8e 49 89 c0 4d 39 c8 75 ee 5b b8 01 00 00 00 41 5c 41 5d <5d> c3 4d 89 c8 4d 39 c2 74 5c e8 8d e4 ff ff 31 c0 5b 41 5c 41 5d RSP: 0018:ffff8880a99170b8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000001 RBX: ffff8880a8cb0cc8 RCX: ffffffff8151de3e RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8880a8cb0cc8 RBP: ffff8880a99170b8 R08: ffffed101519619a R09: ffffed101519619a R10: ffffed1015196199 R11: ffff8880a8cb0ccb R12: 0000000000000001 R13: 0000000000000003 R14: ffffed1015196199 R15: 0000000000000001 __kasan_check_read+0x11/0x20 mm/kasan/common.c:92 atomic_read include/asm-generic/atomic-instrumented.h:26 [inline] virt_spin_lock arch/x86/include/asm/qspinlock.h:83 [inline] native_queued_spin_lock_slowpath+0xbe/0xa70 kernel/locking/qspinlock.c:325 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:654 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline] queued_spin_lock include/asm-generic/qspinlock.h:81 [inline] do_raw_spin_lock+0x206/0x2d0 kernel/locking/spinlock_debug.c:113 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:136 [inline] _raw_spin_lock_bh+0x39/0x40 kernel/locking/spinlock.c:175 spin_lock_bh include/linux/spinlock.h:343 [inline] release_sock+0x1b/0x180 net/core/sock.c:2932 wait_on_pending_writer+0x1c7/0x3b0 net/tls/tls_main.c:91 tls_sk_proto_cleanup+0x232/0x320 net/tls/tls_main.c:295 tls_sk_proto_unhash+0x7b/0x350 net/tls/tls_main.c:330 tcp_set_state+0x29b/0x6e0 net/ipv4/tcp.c:2233 tcp_done+0xac/0x270 net/ipv4/tcp.c:3822 tcp_reset+0xdc/0x380 net/ipv4/tcp_input.c:4080 tcp_validate_incoming+0x891/0x16d0 net/ipv4/tcp_input.c:5440 tcp_rcv_established+0x818/0x2250 net/ipv4/tcp_input.c:5648 tcp_v6_do_rcv+0x397/0x11c0 net/ipv6/tcp_ipv6.c:1356 tcp_v6_rcv+0x2798/0x3310 net/ipv6/tcp_ipv6.c:1588 ip6_protocol_deliver_rcu+0x263/0x1590 net/ipv6/ip6_input.c:397 ip6_input_finish+0x54/0xf0 net/ipv6/ip6_input.c:438 NF_HOOK include/linux/netfilter.h:305 [inline] ip6_input+0xd0/0x2e0 net/ipv6/ip6_input.c:447 dst_input include/net/dst.h:442 [inline] ip6_rcv_finish+0x184/0x290 net/ipv6/ip6_input.c:76 NF_HOOK include/linux/netfilter.h:305 [inline] ipv6_rcv+0xf3/0x310 net/ipv6/ip6_input.c:272 __netif_receive_skb_one_core+0xe9/0x170 net/core/dev.c:4999 __netif_receive_skb+0x1f/0x1b0 net/core/dev.c:5113 process_backlog+0x1cb/0x670 net/core/dev.c:5924 napi_poll net/core/dev.c:6347 [inline] net_rx_action+0x458/0xe00 net/core/dev.c:6413 __do_softirq+0x262/0x9a8 kernel/softirq.c:292 run_ksoftirqd+0x94/0x100 kernel/softirq.c:603 smpboot_thread_fn+0x55f/0x8b0 kernel/smpboot.c:165 kthread+0x331/0x3f0 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352