syzbot


possible deadlock in do_ipv6_setsockopt (2)

Status: fixed on 2018/02/26 20:04
Subsystems: netfilter
[Documentation on labels]
Reported-by: syzbot+886c94d913dfba6f328a@syzkaller.appspotmail.com
Fix commit: 01ea306f2ac2 netfilter: drop outermost socket lock in getsockopt()
First crash: 2330d, last: 2311d
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in do_ipv6_setsockopt (3) net 1 1183d 1182d 0/27 auto-closed as invalid on 2021/07/26 21:07
upstream possible deadlock in do_ipv6_setsockopt netfilter C 109 2332d 2339d 22/27 closed as dup on 2018/01/30 13:59
android-44 possible deadlock in do_ipv6_setsockopt 7 2051d 1897d 0/2 auto-closed as invalid on 2019/05/11 00:02

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
4.16.0-rc1+ #315 Not tainted
------------------------------------------------------
syz-executor3/5565 is trying to acquire lock:
 (sk_lock-AF_INET6){+.+.}, at: [<00000000e91b50b9>] lock_sock include/net/sock.h:1463 [inline]
 (sk_lock-AF_INET6){+.+.}, at: [<00000000e91b50b9>] do_ipv6_setsockopt.isra.8+0x3c5/0x39d0 net/ipv6/ipv6_sockglue.c:167

but task is already holding lock:
 (rtnl_mutex){+.+.}, at: [<000000005a74981e>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (rtnl_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74
       unregister_netdevice_notifier+0x91/0x4e0 net/core/dev.c:1673
       tee_tg_destroy+0x61/0xc0 net/netfilter/xt_TEE.c:123
       cleanup_entry+0x218/0x350 net/ipv4/netfilter/ip_tables.c:654
       __do_replace+0x79d/0xa50 net/ipv4/netfilter/ip_tables.c:1089
       do_replace net/ipv4/netfilter/ip_tables.c:1145 [inline]
       do_ipt_set_ctl+0x40f/0x5f0 net/ipv4/netfilter/ip_tables.c:1675
       nf_sockopt net/netfilter/nf_sockopt.c:106 [inline]
       nf_setsockopt+0x67/0xc0 net/netfilter/nf_sockopt.c:115
       ip_setsockopt+0x97/0xa0 net/ipv4/ip_sockglue.c:1259
       udp_setsockopt+0x45/0x80 net/ipv4/udp.c:2401
       ipv6_setsockopt+0xa0/0x130 net/ipv6/ipv6_sockglue.c:917
       sctp_setsockopt+0x2b6/0x61d0 net/sctp/socket.c:4104
       sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2975
       SYSC_setsockopt net/socket.c:1849 [inline]
       SyS_setsockopt+0x189/0x360 net/socket.c:1828
       do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (&xt[i].mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       xt_find_table_lock+0x3e/0x3e0 net/netfilter/x_tables.c:1046
       xt_request_find_table_lock+0x28/0xc0 net/netfilter/x_tables.c:1093
       get_info+0x154/0x690 net/ipv6/netfilter/ip6_tables.c:989
       do_ip6t_get_ctl+0x159/0xaf0 net/ipv6/netfilter/ip6_tables.c:1710
       nf_sockopt net/netfilter/nf_sockopt.c:104 [inline]
       nf_getsockopt+0x6a/0xc0 net/netfilter/nf_sockopt.c:122
       ipv6_getsockopt+0x1df/0x2e0 net/ipv6/ipv6_sockglue.c:1371
       tcp_getsockopt+0x82/0xd0 net/ipv4/tcp.c:3359
       sock_common_getsockopt+0x95/0xd0 net/core/sock.c:2934
       SYSC_getsockopt net/socket.c:1880 [inline]
       SyS_getsockopt+0x178/0x340 net/socket.c:1862
       do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (sk_lock-AF_INET6){+.+.}:
       lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
       lock_sock_nested+0xc2/0x110 net/core/sock.c:2777
       lock_sock include/net/sock.h:1463 [inline]
       do_ipv6_setsockopt.isra.8+0x3c5/0x39d0 net/ipv6/ipv6_sockglue.c:167
       ipv6_setsockopt+0xd7/0x130 net/ipv6/ipv6_sockglue.c:922
       sctp_setsockopt+0x2b6/0x61d0 net/sctp/socket.c:4104
       sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2975
       SYSC_setsockopt net/socket.c:1849 [inline]
       SyS_setsockopt+0x189/0x360 net/socket.c:1828
       do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
  sk_lock-AF_INET6 --> &xt[i].mutex --> rtnl_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(rtnl_mutex);
                               lock(&xt[i].mutex);
                               lock(rtnl_mutex);
  lock(sk_lock-AF_INET6);

 *** DEADLOCK ***

1 lock held by syz-executor3/5565:
 #0:  (rtnl_mutex){+.+.}, at: [<000000005a74981e>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74

stack backtrace:
CPU: 1 PID: 5565 Comm: syz-executor3 Not tainted 4.16.0-rc1+ #315
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
 check_prev_add kernel/locking/lockdep.c:1863 [inline]
 check_prevs_add kernel/locking/lockdep.c:1976 [inline]
 validate_chain kernel/locking/lockdep.c:2417 [inline]
 __lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
 lock_sock_nested+0xc2/0x110 net/core/sock.c:2777
 lock_sock include/net/sock.h:1463 [inline]
 do_ipv6_setsockopt.isra.8+0x3c5/0x39d0 net/ipv6/ipv6_sockglue.c:167
 ipv6_setsockopt+0xd7/0x130 net/ipv6/ipv6_sockglue.c:922
 sctp_setsockopt+0x2b6/0x61d0 net/sctp/socket.c:4104
 sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2975
 SYSC_setsockopt net/socket.c:1849 [inline]
 SyS_setsockopt+0x189/0x360 net/socket.c:1828
 do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x453a59
RSP: 002b:00007ff7a59d0c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
RAX: ffffffffffffffda RBX: 00007ff7a59d16d4 RCX: 0000000000453a59
RDX: 000000000000002a RSI: 0000000000000029 RDI: 0000000000000013
RBP: 000000000071bea0 R08: 0000000000000088 R09: 0000000000000000
R10: 0000000020058000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000004fb R14: 00000000006f7828 R15: 0000000000000000
x_tables: ip_tables: icmp match: only valid for protocol 1
audit: type=1400 audit(1518842662.564:27): avc:  denied  { map } for  pid=5629 comm="syz-executor7" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=14327 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:hugetlbfs_t:s0 tclass=file permissive=1
audit: type=1400 audit(1518842662.605:28): avc:  denied  { setfcap } for  pid=5620 comm="syz-executor3" capability=31  scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=cap_userns permissive=1
QAT: Invalid ioctl
QAT: Invalid ioctl
device lo entered promiscuous mode
device lo left promiscuous mode
audit: type=1400 audit(1518842663.296:29): avc:  denied  { name_bind } for  pid=5832 comm="syz-executor5" src=20020 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=system_u:object_r:port_t:s0 tclass=dccp_socket permissive=1
device eql entered promiscuous mode
audit: type=1400 audit(1518842663.296:30): avc:  denied  { node_bind } for  pid=5832 comm="syz-executor5" saddr=::1 src=20020 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=system_u:object_r:node_t:s0 tclass=dccp_socket permissive=1
audit: type=1400 audit(1518842663.298:31): avc:  denied  { name_connect } for  pid=5832 comm="syz-executor5" dest=20020 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=system_u:object_r:port_t:s0 tclass=dccp_socket permissive=1
QAT: Invalid ioctl
audit: type=1400 audit(1518842663.457:32): avc:  denied  { read } for  pid=5879 comm="syz-executor4" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
QAT: Invalid ioctl
syz1: refused to change device tx_queue_len
syz1: refused to change device tx_queue_len
SELinux: failed to load policy
SELinux: failed to load policy
QAT: Invalid ioctl
QAT: Invalid ioctl
netlink: 20 bytes leftover after parsing attributes in process `syz-executor5'.
netlink: 20 bytes leftover after parsing attributes in process `syz-executor5'.
9pnet_virtio: no channels available for device ./file0
9pnet_virtio: no channels available for device ./file0
mmap: syz-executor3 (6105) uses deprecated remap_file_pages() syscall. See Documentation/vm/remap_file_pages.txt.
can: request_module (can-proto-0) failed.
can: request_module (can-proto-0) failed.
device eql entered promiscuous mode
xt_connbytes: Forcing CT accounting to be enabled
capability: warning: `syz-executor1' uses deprecated v2 capabilities in a way that may be insecure
arp_tables: arp_tables: error: 'ξΟ ‘˜™«arY,G΅ν>+Χt€σ_Γm '
QAT: Invalid ioctl
QAT: Invalid ioctl
netlink: 'syz-executor6': attribute type 1 has an invalid length.
netlink: 'syz-executor6': attribute type 1 has an invalid length.
rfkill: input handler disabled
rfkill: input handler enabled
device lo entered promiscuous mode
device lo left promiscuous mode
device lo entered promiscuous mode
device lo left promiscuous mode
device eql entered promiscuous mode
kauditd_printk_skb: 9 callbacks suppressed
audit: type=1400 audit(1518842666.275:42): avc:  denied  { setuid } for  pid=6808 comm="syz-executor6" capability=7  scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=cap_userns permissive=1
audit: type=1400 audit(1518842666.357:43): avc:  denied  { setgid } for  pid=6828 comm="syz-executor7" capability=6  scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=cap_userns permissive=1
audit: type=1400 audit(1518842666.511:44): avc:  denied  { getopt } for  pid=6846 comm="syz-executor6" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
device bridge0 entered promiscuous mode
audit: type=1400 audit(1518842666.539:45): avc:  denied  { bind } for  pid=6846 comm="syz-executor6" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
device bridge0 left promiscuous mode
device bridge0 entered promiscuous mode
device bridge0 left promiscuous mode
xt_CT: netfilter: NOTRACK target is deprecated, use CT instead or upgrade iptables
audit: type=1400 audit(1518842666.761:46): avc:  denied  { ioctl } for  pid=6911 comm="syz-executor4" path="socket:[18745]" dev="sockfs" ino=18745 ioctlcmd=0x8918 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
x_tables: ip6_tables: eui64 match: used from hooks PREROUTING/FORWARD/OUTPUT, but only valid from PREROUTING/INPUT/FORWARD
x_tables: ip6_tables: eui64 match: used from hooks PREROUTING/FORWARD/OUTPUT, but only valid from PREROUTING/INPUT/FORWARD
audit: type=1400 audit(1518842666.981:47): avc:  denied  { accept } for  pid=6981 comm="syz-executor1" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_netfilter_socket permissive=1
encrypted_key: insufficient parameters specified
encrypted_key: insufficient parameters specified
netlink: 3 bytes leftover after parsing attributes in process `syz-executor5'.
netlink: 3 bytes leftover after parsing attributes in process `syz-executor5'.
netlink: 3 bytes leftover after parsing attributes in process `syz-executor5'.
netlink: 3 bytes leftover after parsing attributes in process `syz-executor5'.
ipt_REJECT: TCP_RESET invalid for non-tcp
ipt_REJECT: TCP_RESET invalid for non-tcp
syz-executor5 uses obsolete (PF_INET,SOCK_PACKET)
SELinux: failed to load policy
SELinux: failed to load policy
audit: type=1326 audit(1518842667.915:48): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=7215 comm="syz-executor5" exe="/root/syz-executor5" sig=9 arch=c000003e syscall=202 compat=0 ip=0x453a59 code=0x0
audit: type=1326 audit(1518842667.963:49): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 pid=7215 comm="syz-executor5" exe="/root/syz-executor5" sig=9 arch=c000003e syscall=202 compat=0 ip=0x453a59 code=0x0
audit: type=1400 audit(1518842668.029:50): avc:  denied  { setopt } for  pid=7232 comm="syz-executor3" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_generic_socket permissive=1
audit: type=1400 audit(1518842668.893:51): avc:  denied  { prog_run } for  pid=7513 comm="syz-executor4" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=bpf permissive=1
*** Guest State ***
CR0: actual=0x0000000000000030, shadow=0x0000000060000010, gh_mask=fffffffffffffff7
CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffe871
CR3 = 0x00000000fffbc000
RSP = 0x0000000000000000  RIP = 0x000000000000fff0
RFLAGS=0x00000002         DR7 = 0x0000000000000400
Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
CS:   sel=0xf000, attr=0x0009b, limit=0x0000ffff, base=0x00000000ffff0000
DS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
SS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
ES:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
FS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
GS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
GDTR:                           limit=0x0000ffff, base=0x0000000000000000
LDTR: sel=0x0000, attr=0x00082, limit=0x0000ffff, base=0x0000000000000000
IDTR:                           limit=0x0000ffff, base=0x0000000000000000
TR:   sel=0x0000, attr=0x0008b, limit=0x0000ffff, base=0x0000000000000000
EFER =     0x0000000000000000  PAT = 0x0007040600070406
DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000
Interruptibility = 00000001  ActivityState = 00000000
*** Host State ***
RIP = 0xffffffff811cd915  RSP = 0xffff8801b232f3b8
CS=0010 SS=0018 DS=0000 ES=0000 FS=0000 GS=0000 TR=0040
FSBase=00007f80779fd700 GSBase=ffff8801db500000 TRBase=fffffe0000034000
GDTBase=fffffe0000032000 IDTBase=fffffe0000000000
CR0=0000000080050033 CR3=00000001b0596001 CR4=00000000001626e0
Sysenter RSP=fffffe0000033200 CS:RIP=0010:ffffffff85a01e70
EFER = 0x0000000000000d01  PAT = 0x0000000000000000
*** Control State ***
PinBased=0000003f CPUBased=b599edfe SecondaryExec=000000c2
EntryControls=0000d1ff ExitControls=0023efff
ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000
VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
        reason=80000021 qualification=0000000000000000
IDTVectoring: info=00000000 errcode=00000000
TSC Offset = 0xffffffe545a0ae3f
EPT pointer = 0x00000001a96f501e
device eql entered promiscuous mode
RDS: rds_bind could not find a transport for 172.20.5.23, load rds_tcp or rds_rdma?
RDS: rds_bind could not find a transport for 172.20.5.23, load rds_tcp or rds_rdma?
netlink: 'syz-executor4': attribute type 21 has an invalid length.
netlink: 'syz-executor4': attribute type 21 has an invalid length.
encrypted_key: insufficient parameters specified
netlink: 'syz-executor4': attribute type 21 has an invalid length.
device eql entered promiscuous mode
binder: 8189:8193 BC_INCREFS_DONE u0000000000000000 no match
binder: BINDER_SET_CONTEXT_MGR already set
binder: 8189:8208 ioctl 40046207 0 returned -16
binder: 8189:8208 BC_INCREFS_DONE u0000000000000000 no match
RDS: rds_bind could not find a transport for 0.0.0.1, load rds_tcp or rds_rdma?

Crashes (3642):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/02/17 04:44 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 04:43 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 04:41 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 04:33 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 04:19 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 02:32 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 02:18 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 02:15 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 01:56 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 01:51 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 01:28 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 00:57 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 00:42 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 00:42 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 00:21 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/17 00:07 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:35 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:26 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:15 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:04 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:04 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 23:02 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 22:39 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 22:33 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 22:09 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 22:05 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 22:02 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 21:55 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 21:45 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 21:18 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 21:02 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 20:56 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 20:09 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 19:38 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 19:36 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 18:23 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 18:16 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 17:59 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 17:39 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/16 17:38 upstream 1a2a7d3ee659 c8b3f7c1 .config console log report ci-upstream-kasan-gce
2018/02/07 15:10 upstream a2e5790d8416 9fb5ec43 .config console log report ci-upstream-kasan-gce
2018/02/05 08:46 net-next-old 617aebe6a97e a1bc9d40 .config console log report ci-upstream-net-kasan-gce
* Struck through repros no longer work on HEAD.