SNI, WorxWeb, Netscaler and slate.adobe.com

Citrix XenMobile
1

Recently I’ve come across an issue with users being unable to connect to https://slate.adobe.com using WorxWeb on both iOS and Android. Connection seems to terminate instantly as you enter the URL.

Digging further into this problem came up with some interesting results.

Doing a dig on slate.adobe.com returns a number of A records

; <<>> DiG 9.8.3-P1 <<>> slate.adobe.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39445
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;slate.adobe.com.		IN	A

;; ANSWER SECTION:
slate.adobe.com.	0	IN	CNAME	slate.adobestandout.com.
slate.adobestandout.com. 60	IN	A	54.230.246.81
slate.adobestandout.com. 60	IN	A	54.230.246.229
slate.adobestandout.com. 60	IN	A	54.230.246.242
slate.adobestandout.com. 60	IN	A	54.230.246.244
slate.adobestandout.com. 60	IN	A	54.230.246.203
slate.adobestandout.com. 60	IN	A	54.230.246.88
slate.adobestandout.com. 60	IN	A	54.230.246.195
slate.adobestandout.com. 60	IN	A	54.230.246.40

The first record of the answer will change randomly, this seems to be a type of load balancing implemented by adobe.

An SSL scan on slate.adobe.com SSL Server Test: slate.adobe.com (Powered by Qualys SSL Labs) reveals “This site works only in browsers with SNI support.”

Now we are using SecureBrowse for WorxWeb as a preferred VPN method.
SecureBrowse causes the netscaler to acts like a reverse proxy for the said traffic with the device traffic terminating on the netscaler and netscaler establishing a new SSL tunnel in order to fetch the requested traffic.

Due to this we can run a traffic capture on the netscaler in order to capture the SSL handshake (if we were to use Full Tunnel for VPN it would be a bit trickier as we would have to decrypt the traffic in order to see the handshake between the device and the target website).

Looking the SSL handshake traffic capture between netscaler and slate.adobe.com we can see that SSL handshake is failing.

However looking at the Client Hello we can see that the protocol being offered by the client and ciphers are actually supported by slate.adobe.com according to the SSL scan.

Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 139
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 135
            Version: TLS 1.0 (0x0301)
            Random
                GMT Unix Time: Jan 14, 2016 11:46:26.000000000 NZDT
                Random Bytes: a562d85be83e6aa825121dd960976ec17353b5f7b433f284...
            Session ID Length: 32
            Session ID: 040a5645c6c2ea8abcaedd6b35b8013efc6a788b4134d6b3...
            Cipher Suites Length: 64
            Cipher Suites (32 suites)
                Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
                Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
                Cipher Suite: TLS_RSA_EXPORT_WITH_RC4_40_MD5 (0x0003)
                Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
                Cipher Suite: TLS_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0008)
                Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_SHA (0x0064)
                Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_MD5 (0x0060)
                Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5 (0x0061)
                Cipher Suite: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 (0x0006)
                Cipher Suite: TLS_DHE_DSS_WITH_RC4_128_SHA (0x0066)
                Cipher Suite: TLS_DHE_DSS_EXPORT1024_WITH_RC4_56_SHA (0x0065)
                Cipher Suite: TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA (0x0063)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_DHE_RSA_WITH_DES_CBC_SHA (0x0015)
                Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
                Cipher Suite: TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0014)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_DHE_DSS_WITH_DES_CBC_SHA (0x0012)
                Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
                Cipher Suite: TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA (0x0011)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
                Cipher Suite: TLS_DH_anon_WITH_RC4_128_MD5 (0x0018)
                Cipher Suite: TLS_DH_anon_WITH_DES_CBC_SHA (0x001a)
                Cipher Suite: TLS_DH_anon_WITH_3DES_EDE_CBC_SHA (0x001b)
                Cipher Suite: TLS_DH_anon_EXPORT_WITH_RC4_40_MD5 (0x0017)
                Cipher Suite: TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA (0x0019)
                Cipher Suite: TLS_DH_anon_WITH_AES_128_CBC_SHA (0x0034)
                Cipher Suite: TLS_DH_anon_WITH_AES_256_CBC_SHA (0x003a)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)

This was a bit puzzling to me until I’ve read more about SNI.
SNI is a method of hosting multiple certificates on the same web server IP & port.
SNI relies on the hostname to be provided in the client hello in order to be able to answer the SSL handshake.
You can read more about SNI here. Server Name Indication - Wikipedia

I’ve ran wireshark on my machine while attempting to access the same site and I could see a few additional lines in the Client Hello, specifically:

Extension: server_name
                Type: server_name (0x0000)
                Length: 20
                Server Name Indication extension
                    Server Name list length: 18
                    Server Name Type: host_name (0)
                    Server Name length: 15
                    Server Name: slate.adobe.com

Now it all makes sense, as we are not presenting the above in the Client Hello sent by the netscaler the web server in question has no idea which SSL certificate is being requested therefore closing the connection with ssl handshake error.

This can be replicated in OpenSSL

openssl s_client -connect slate.adobe.com:443

will return a handshake error

openssl s_client -servername slate.adobe.com -connect slate.adobe.com:443

will connect successfully

It does not seem like the Netscaler Gateway module has support for SNI therefore slate.adobe.com cannot be reached. I currently have a ticket open with Citrix in order to see if this is something that can be enabled or something that exists in other versions of the netscaler otherwise slate.adobe.com will not be accessible from WorxWeb.

2

Excellent piece of fault finding work @daniil_michine, handy to know!

3

It seems that Amazon load balancing service Cloudfront (https://aws.amazon.com/cloudfront/) relies heavily on SNI therefore any websites that use cloudfront will have the same issue

4

Did you find any workaround for this WorxWeb (SecureBrowse) connection to the backend server that requires SNI support?

Citrix Netscaler supports SNI to backend LB Services since version 11.1 however what about Cloud servers that you dont Load Balance and you dont have Service created to enable SNI under it?

5

I’ve not tried with the netscaler 11.1 yet, Netscaler Gateway might handle the SNI part differently from the loadbalancer.

You can look into using the new Reverse Split Tunnel setting in SecureWeb which allows you to send some traffic direct (i.e. for sites that require SNI) rather then going via SecureBrowse through netscaler.

Alternatively the Full Tunnel option should work since the SSL handshake would then be between the device (which supports SNI) and the website.

1 Like
6

That’s correct the Netscaler Gateway handles SNI differently, the SNI can be enabled only for front-end traffic under SSL Parameters of NG VIP, however this is not good enough for backend server connection as there is no way to enable SNI (if not doing any Load Balancing).

The MicroVPN in Full VPN Tunnel mode works fine, but we have to use Secure Browse (SSL Offload) to initiate SSO for this backend server connection. This is the whole purpose to proxy this Cloud application traffic via NetScaler

Also we don’t use Citrix Secure Web but custom in-house wrapped app.

7

There is a SSL service created for the NSG when you run through the XenMobile wizards, however from my understanding this SSL service is used for communicating with the XMS (STA)

You could potentially create a load balancer on the netscaler for this cloud service with SNI configured on the backend and send the application via this load balancer

Netscaler Gateway -> Load balancer - > Cloud Service

8

This particular Cloud Service is hosted using Amazon CloudFront so it’s not Load Balanced on our NetScaler and we don’t have access to the SSL Certificate to import it to NetScaler.

I was thinking about this flow: Netscaler Gateway -> Load balancer - > Cloud Service
but without SSL cert that we can import on NS LB VIP we wont be able to enable SNI under Service.

It looks like there is no hope for us on NetScaler 11.1?

9

Netscaler Gateway does not verify SSL certificates, you can access sites with expired and invalid certificates. You can verify the behaviour by accessing https://badssl.com via WorxWeb and running through their tests.

So in theory you could issue a self signed cert for the cloud service and install it on your load balancer.

Netscaler gateway → Load balancer (with self signed cert) → Cloud Service (with public cert)

10

Citrix have released netscaler v12.0.53.6 that adds SNI support on the netscaler gateway

NetScaler Gateway
SNI Support for NetScaler Gateway

A NetScaler Gateway appliance can now be configured to include a server name indication (SNI) extension in the SSL “client hello” packet sent to the backend server. The SNI extension helps the backend server identify the FQDN being requested during the SSL handshake and respond with the respective certificates.
Note: Enable SNI support when multiple SSL domains are hosted on same server.

For more information, see Product Documentation.

[# 624091]

After upgrading netscaler you may need to enable SNI

set vpn parameter backendServerSni ENABLED

Reference docs:

11

From our testing update to netscaler v12.0.53.6 enabling backend server SNI has resolved issues accessing sites fronted by cloudfront