The latest one I came across was at a customer site, where WSO2 ESB wants to communicate with an SSL end point(WCF) hosted on IIS 7.
The only thing here what we have to do is, importing the CA certificate of the WCF end point to the ESB's client-trustore.jks [which is under ESB_HOME\resources\security].
This worked well at the start - but in one machine it started to fail - with no clue at all..
This is where we need some handy SSL debugging tools - and the easiest one is setting the system property javax.net.debug=all. For example you need to start the WSO2 ESB as,
:\> sh wso2server.sh -Djavax.net.debug=all
Once you set this, it will print the entire SSL handshake.
By going through the logs, we could figure out the issue - where by mistake in this particular machine, in IIS, for this end point - 'Require SSL' being set to Accept, instead of Ignore.
What does that mean.. and why did that fail..?
When we set the above parameter to Accept, the server validates the client certificate only if it's been sent in the request from the client.
In our case at the ESB end we set following two system properties,
System.setProperty("javax.net.ssl.keyStore", "keyStorePath");
System.setProperty("javax.net.ssl.keyStorePassword", "password");
When you set these two, the client will automatically attach the client certificate to the SSL handshake - in our case it failed because we were not expecting mutual authentication, so IIS didn't trust ESB as a client.
Another tool comes in handy while SSL debugging is openssl s_client.
There was a case, where WSO2 ESB talking to an SSL end point behind an Apache server - in that case all the configuration options we provided didn't work - so the option left behind was to test the SSL setup of the Apache server in an independent manner - and proved us Apache server had issues in SSL setup. There we used openssl s_client. For example if you want to verify the SSL handshake with the end point, localhost:9443, you can use the following command.
:\>openssl s_client -connect localhost:9443 -state -nbio 2>&1 | grep "^SSL"
The above will result in the following out put.
SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:error in SSLv2/v3 read server hello A SSL_connect:SSLv3 read server hello A SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server key exchange A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A SSL handshake has read 1149 bytes and written 293 bytes SSL-Session:To see all the options available with s_client, type the following,
:\>openssl s_client --help
Let's finish off the first part of the SSL debugging series with ssldump.
ssldump is an SSL/TLS network protocol analyzer. It identifies TCP connections on the chosen network interface and attempts to interpret them as SSL/TLS traffic. When it identifies SSL/TLS traffic, it decodes the records and displays them in a textual form to stdout.
:\>sudo ssldump -i en1 port 443
The above prints following [only a part displayed], when I visit https://cloud.wso2.com.
New TCP connection #1: 192.168.1.3(49986) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
1 1 0.3614 (0.3614) C>S Handshake
ClientHello
Version 3.1
cipher suites
Unknown value 0xc00a
Unknown value 0xc009
Unknown value 0xc007
Unknown value 0xc008
Unknown value 0xc013
Unknown value 0xc014
Unknown value 0xc011
Unknown value 0xc012
Unknown value 0xc004
Unknown value 0xc005
Unknown value 0xc002
Unknown value 0xc003
Unknown value 0xc00e
Unknown value 0xc00f
Unknown value 0xc00c
Unknown value 0xc00d
Unknown value 0x2f
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
Unknown value 0x35
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_DES_CBC_SHA
TLS_RSA_EXPORT_WITH_RC4_40_MD5
TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
Unknown value 0x32
Unknown value 0x33
Unknown value 0x38
Unknown value 0x39
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_DES_CBC_SHA
TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_DES_CBC_SHA
TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
compression methods
NULL
1 2 0.7638 (0.4023) S>C Handshake
ServerHello
Version 3.1
session_id[32]=
22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97
ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81
cipherSuite Unknown value 0x2f
compressionMethod NULL
1 3 1.1691 (0.4053) S>C Handshake
Certificate
1 4 1.1691 (0.0000) S>C Handshake
ServerHelloDone
1 5 1.1749 (0.0058) C>S Handshake
ClientKeyExchange
1 6 1.1749 (0.0000) C>S ChangeCipherSpec
1 7 1.1750 (0.0000) C>S Handshake
1 8 1.5787 (0.4037) S>C ChangeCipherSpec
1 9 1.5787 (0.0000) S>C Handshake
1 10 1.5794 (0.0006) C>S application_data
1 11 2.1889 (0.6095) S>C application_data
1 12 2.1889 (0.0000) S>C application_data
1 13 2.1889 (0.0000) S>C application_data
1 14 2.1889 (0.0000) S>C application_data
1 15 2.1889 (0.0000) S>C application_data
1 16 2.1889 (0.0000) S>C application_data
1 17 2.1889 (0.0000) S>C application_data
1 18 2.1889 (0.0000) S>C application_data
1 19 2.1896 (0.0007) C>S Alert
1 2.1902 (0.0005) C>S TCP FIN
New TCP connection #2: 192.168.1.3(49987) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
2 1 0.3662 (0.3662) C>S Handshake
ClientHello
Version 3.1
resume [32]=
22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97
ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81
cipher suites
Unknown value 0xc00a
Unknown value 0xc009
Unknown value 0xc007
Unknown value 0xc008
Unknown value 0xc013
Unknown value 0xc014
Unknown value 0xc011
Unknown value 0xc012
Unknown value 0xc004
Unknown value 0xc005
Unknown value 0xc002
Unknown value 0xc003
Unknown value 0xc00e
Unknown value 0xc00f
Unknown value 0xc00c
Unknown value 0xc00d
Unknown value 0x2f
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
Unknown value 0x35
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_DES_CBC_SHA
TLS_RSA_EXPORT_WITH_RC4_40_MD5
TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
Unknown value 0x32
Unknown value 0x33
Unknown value 0x38
Unknown value 0x39
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_DES_CBC_SHA
TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_DES_CBC_SHA
TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
compression methods
NULL
2 2 0.9732 (0.6070) S>C Handshake
ServerHello
Version 3.1
session_id[32]=
22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97
ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81
cipherSuite Unknown value 0x2f
compressionMethod NULL
2 3 0.9732 (0.0000) S>C ChangeCipherSpec
2 4 0.9732 (0.0000) S>C Handshake
2 5 0.9735 (0.0003) C>S ChangeCipherSpec
2 6 0.9736 (0.0000) C>S Handshake
2 7 0.9737 (0.0001) C>S application_data
2 8 1.6900 (0.7162) S>C application_data
2 9 1.6900 (0.0000) S>C application_data
2 10 1.6900 (0.0000) S>C application_data
2 11 1.6900 (0.0000) S>C application_data
2 12 1.6900 (0.0000) S>C application_data
2 13 1.6900 (0.0000) S>C application_data
2 14 1.6900 (0.0000) S>C application_data
2 15 1.6900 (0.0000) S>C application_data
2 16 1.6903 (0.0002) C>S Alert
2 1.6909 (0.0006) C>S TCP FIN
New TCP connection #3: 192.168.1.3(49988) <-> ec2-184-73-175-181.compute-1.amazonaws.com(443)
3 1 0.3674 (0.3674) C>S Handshake
ClientHello
Version 3.1
resume [32]=
22 24 f7 6c 43 84 ba 39 6f b9 02 5c 4d 76 cf 97
ad b4 39 1b 82 fe c1 cf d7 5b 14 41 87 bd 6a 81
cipher suites
Unknown value 0xc00a
Unknown value 0xc009
Unknown value 0xc007
Unknown value 0xc008
Unknown value 0xc013
Unknown value 0xc014
Unknown value 0xc011
Unknown value 0xc012
Unknown value 0xc004
Unknown value 0xc005
Unknown value 0xc002
Unknown value 0xc003
Unknown value 0xc00e
Unknown value 0xc00f
Unknown value 0xc00c
Unknown value 0xc00d
Unknown value 0x2f
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
Unknown value 0x35
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_DES_CBC_SHA
TLS_RSA_EXPORT_WITH_RC4_40_MD5
TLS_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
Unknown value 0x32
Unknown value 0x33
Unknown value 0x38
Unknown value 0x39
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_DES_CBC_SHA
TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_DES_CBC_SHA
TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
compression methods
NULL