SSL Debugging - Part - I

We've been involved in many cases solving issues related to SSL.

The latest one I came across was at a customer site, where WSO2 ESB wants to communicate with an SSL end point(WCF) hosted on IIS 7.

The only thing here what we have to do is, importing the CA certificate of the WCF end point to the ESB's client-trustore.jks [which is under ESB_HOME\resources\security].

This worked well at the start - but in one machine it started to fail - with no clue at all..

This is where we need some handy SSL debugging tools - and the easiest one is setting the system property javax.net.debug=all. For example you need to start the WSO2 ESB as,

:\> sh wso2server.sh -Djavax.net.debug=all

Once you set this, it will print the entire SSL handshake.

By going through the logs, we could figure out the issue - where by mistake in this particular machine, in IIS, for this end point - 'Require SSL' being set to Accept, instead of Ignore.

What does that mean.. and why did that fail..?

When we set the above parameter to Accept, the server validates the client certificate only if it's been sent in the request from the client.

In our case at the ESB end we set following two system properties,

System.setProperty("javax.net.ssl.keyStore", "keyStorePath");
System.setProperty("javax.net.ssl.keyStorePassword", "password");


When you set these two, the client will automatically attach the client certificate to the SSL handshake - in our case it failed because we were not expecting mutual authentication, so IIS didn't trust ESB as a client.

Another tool comes in handy while SSL debugging is openssl s_client.

There was a case, where WSO2 ESB talking to an SSL end point behind an Apache server - in that case all the configuration options we provided didn't work - so the option left behind was to test the SSL setup of the Apache server in an independent manner - and proved us Apache server had issues in SSL setup. There we used openssl s_client. For example if you want to verify the SSL handshake with the end point, localhost:9443, you can use the following command.

:\>openssl s_client -connect localhost:9443 -state -nbio 2>&1 | grep "^SSL"

The above will result in the following out put.
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL handshake has read 1149 bytes and written 293 bytes
SSL-Session:
To see all the options available with s_client, type the following,

:\>openssl s_client --help

Let's finish off the first part of the SSL debugging series with ssldump.

ssldump is an SSL/TLS network protocol analyzer. It identifies TCP connections on the chosen network interface and attempts to interpret them as SSL/TLS traffic. When it identifies SSL/TLS traffic, it decodes the records and displays them in a textual form to stdout.

:\>sudo ssldump -i en1 port 443

The above prints following [only a part displayed], when I visit https://cloud.wso2.com.
New TCP connection #1: 192.168.1.3(49986) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
1 1  0.3614 (0.3614)  C>S  Handshake
      ClientHello
        Version 3.1 
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc009
        Unknown value 0xc007
        Unknown value 0xc008
        Unknown value 0xc013
        Unknown value 0xc014
        Unknown value 0xc011
        Unknown value 0xc012
        Unknown value 0xc004
        Unknown value 0xc005
        Unknown value 0xc002
        Unknown value 0xc003
        Unknown value 0xc00e
        Unknown value 0xc00f
        Unknown value 0xc00c
        Unknown value 0xc00d
        Unknown value 0x2f
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        Unknown value 0x35
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC4_40_MD5
        TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        Unknown value 0x32
        Unknown value 0x33
        Unknown value 0x38
        Unknown value 0x39
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_DES_CBC_SHA
        TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_DES_CBC_SHA
        TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
        compression methods
                  NULL
1 2  0.7638 (0.4023)  S>C  Handshake
      ServerHello
        Version 3.1 
        session_id[32]=
          22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97 
          ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81 
        cipherSuite         Unknown value 0x2f
        compressionMethod                   NULL
1 3  1.1691 (0.4053)  S>C  Handshake
      Certificate
1 4  1.1691 (0.0000)  S>C  Handshake
      ServerHelloDone
1 5  1.1749 (0.0058)  C>S  Handshake
      ClientKeyExchange
1 6  1.1749 (0.0000)  C>S  ChangeCipherSpec
1 7  1.1750 (0.0000)  C>S  Handshake
1 8  1.5787 (0.4037)  S>C  ChangeCipherSpec
1 9  1.5787 (0.0000)  S>C  Handshake
1 10 1.5794 (0.0006)  C>S  application_data
1 11 2.1889 (0.6095)  S>C  application_data
1 12 2.1889 (0.0000)  S>C  application_data
1 13 2.1889 (0.0000)  S>C  application_data
1 14 2.1889 (0.0000)  S>C  application_data
1 15 2.1889 (0.0000)  S>C  application_data
1 16 2.1889 (0.0000)  S>C  application_data
1 17 2.1889 (0.0000)  S>C  application_data
1 18 2.1889 (0.0000)  S>C  application_data
1 19 2.1896 (0.0007)  C>S  Alert
1    2.1902 (0.0005)  C>S  TCP FIN
New TCP connection #2: 192.168.1.3(49987) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
2 1  0.3662 (0.3662)  C>S  Handshake
      ClientHello
        Version 3.1 
        resume [32]=
          22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97 
          ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81 
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc009
        Unknown value 0xc007
        Unknown value 0xc008
        Unknown value 0xc013
        Unknown value 0xc014
        Unknown value 0xc011
        Unknown value 0xc012
        Unknown value 0xc004
        Unknown value 0xc005
        Unknown value 0xc002
        Unknown value 0xc003
        Unknown value 0xc00e
        Unknown value 0xc00f
        Unknown value 0xc00c
        Unknown value 0xc00d
        Unknown value 0x2f
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        Unknown value 0x35
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC4_40_MD5
        TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        Unknown value 0x32
        Unknown value 0x33
        Unknown value 0x38
        Unknown value 0x39
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_DES_CBC_SHA
        TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_DES_CBC_SHA
        TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
        compression methods
                  NULL
2 2  0.9732 (0.6070)  S>C  Handshake
      ServerHello
        Version 3.1 
        session_id[32]=
          22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97 
          ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81 
        cipherSuite         Unknown value 0x2f
        compressionMethod                   NULL
2 3  0.9732 (0.0000)  S>C  ChangeCipherSpec
2 4  0.9732 (0.0000)  S>C  Handshake
2 5  0.9735 (0.0003)  C>S  ChangeCipherSpec
2 6  0.9736 (0.0000)  C>S  Handshake
2 7  0.9737 (0.0001)  C>S  application_data
2 8  1.6900 (0.7162)  S>C  application_data
2 9  1.6900 (0.0000)  S>C  application_data
2 10 1.6900 (0.0000)  S>C  application_data
2 11 1.6900 (0.0000)  S>C  application_data
2 12 1.6900 (0.0000)  S>C  application_data
2 13 1.6900 (0.0000)  S>C  application_data
2 14 1.6900 (0.0000)  S>C  application_data
2 15 1.6900 (0.0000)  S>C  application_data
2 16 1.6903 (0.0002)  C>S  Alert
2    1.6909 (0.0006)  C>S  TCP FIN
New TCP connection #3: 192.168.1.3(49988) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
3 1  0.3674 (0.3674)  C>S  Handshake
      ClientHello
        Version 3.1 
        resume [32]=
          22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97 
          ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81 
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc009
        Unknown value 0xc007
        Unknown value 0xc008
        Unknown value 0xc013
        Unknown value 0xc014
        Unknown value 0xc011
        Unknown value 0xc012
        Unknown value 0xc004
        Unknown value 0xc005
        Unknown value 0xc002
        Unknown value 0xc003
        Unknown value 0xc00e
        Unknown value 0xc00f
        Unknown value 0xc00c
        Unknown value 0xc00d
        Unknown value 0x2f
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        Unknown value 0x35
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC4_40_MD5
        TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        Unknown value 0x32
        Unknown value 0x33
        Unknown value 0x38
        Unknown value 0x39
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_DES_CBC_SHA
        TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_DES_CBC_SHA
        TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
        compression methods
                  NULL