Wrote a blog sometime back on "How to invoke a web service call with curl ?" in
RampartFAQ.com - but that doesn't explain how to work with Curl with HTTPS.
Today morning I got a question from one of my colleagues and thought of writing a blog on the $subject in detail..
Let's go with few examples..
1. Invoke https://www.amazon.com
$ curl https://www.amazon.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://www.amazon.com/">here</a>.</p>
</body></html>
2. Above won't print details about the SSL handshake. To view SSL handshake let's use -v as an argument
$ curl -v https://www.amazon.com
* About to connect() to www.amazon.com port 443 (#0)
* Trying 72.21.214.128... connected
* Connected to www.amazon.com (72.21.214.128) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=www.amazon.com
* start date: 2010-07-15 00:00:00 GMT
* expire date: 2013-07-14 23:59:59 GMT
* common name: www.amazon.com (matched)
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.21.2 (x86_64-apple-darwin10.5.0) libcurl/7.21.2 OpenSSL/1.0.0b zlib/1.2.5 libidn/1.19
> Host: www.amazon.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 02 Dec 2010 07:22:22 GMT
< Server: Server
< Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Thu, 02-Dec-2010 07:22:22 GMT
< Location: http://www.amazon.com/
< Content-Length: 230
< nnCoection: close
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://www.amazon.com/">here</a>.</p>
</body></html>
* Connection #0 to host www.amazon.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
This HTTPS connection works fine, because, the client trusts the CA of the Amazon's SSL certificate.
By going through the logs you will see, the CA certificates are verified against the curl-ca-bundle.crt file located at /opt/local/share/curl/curl-ca-bundle.crt
If you have a look at the above file, you will see all the trusted CAs listed there..
Verisign Class 1 Public Primary Certification Authority
=======================================================
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCED9pHoGc8JpK83P/uUii5N0wDQYJKoZIhvcNAQEFBQAwXzELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAxIFB1YmxpYyBQcmltYXJ5
IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVow
XzELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAx
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDlGb9to1ZhLZlIcfZn3rmN67eehoAKkQ76OCWvRoiC5XOooJskXQ0fzGVuDLDQ
VoQYh5oGmxChc9+0WDlrbsH2FdWoqD+qEgaNMax/sDTXjzRniAnNFBHiTkVWaR94AoDa3EeRKbs2
yWNcxeDXLYd7obcysHswuiovMaruo2fa2wIDAQABMA0GCSqGSIb3DQEBBQUAA4GBAFgVKTk8d6Pa
XCUDfGD67gmZPCcQcMgMCeazh88K4hiWNWLMv5sneYlfycQJ9M61Hd8qveXbhpxoJeUwfLaJFf5n
0a3hUKw8fGJLj7qE1xIVGx/KXQ/BUpQqEZnae88MNhPVNdwQGVnqlMEAv3WP2fr9dgTbYruQagPZ
RjXZ+Hxb
-----END CERTIFICATE-----
Verisign Class 3 Public Primary Certification Authority
=======================================================
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCEDyRMcsf9tAbDpq40ES/Er4wDQYJKoZIhvcNAQEFBQAwXzELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmltYXJ5
IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVow
XzELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAz
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhEBarsAx94
f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/isI19wKTakyYbnsZogy1Ol
hec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBABByUqkFFBky
CEHwxWsKzH4PIRnN5GfcX6kb5sroc50i2JhucwNhkcV8sEVAbkSdjbCxlnRhLQ2pRdKkkirWmnWX
bj9T/UWZYB2oK0z5XqcJ2HUw19JlYD1n1khVdWk/kfVIC0dpImmClr7JyDiGSnoscxlIaU5rfGW/
D/xwzoiQ
-----END CERTIFICATE-----
In the case of Amazon, it's Verisign who the CA is..
3. Let's try to invoke https://amazon.com [Not www.amazon.com]
$ curl -v https://amazon.com
* About to connect() to amazon.com port 443 (#0)
* Trying 72.21.210.250... connected
* Connected to amazon.com (72.21.210.250) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=www.amazon.com
* start date: 2010-07-15 00:00:00 GMT
* expire date: 2013-07-14 23:59:59 GMT
* SSL: certificate subject name 'www.amazon.com' does not match target host name 'amazon.com'
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
curl: (51) SSL: certificate subject name 'www.amazon.com' does not match target host name 'amazon.com'
This fails.. Why ? It's the same CA which is trusted by the Curl client - but it fails due to the host name verification. The certificate was issued to www.amazon.com - not for amazon.com
4. Let's try to invoke https://amazon.com [Not www.amazon.com] - by disabling SSL validations with the argument -k
curl -v -k https://amazon.com
* About to connect() to amazon.com port 443 (#0)
* Trying 72.21.210.250... connected
* Connected to amazon.com (72.21.210.250) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=www.amazon.com
* start date: 2010-07-15 00:00:00 GMT
* expire date: 2013-07-14 23:59:59 GMT
* common name: www.amazon.com (does not match 'amazon.com')
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.21.2 (x86_64-apple-darwin10.5.0) libcurl/7.21.2 OpenSSL/1.0.0b zlib/1.2.5 libidn/1.19
> Host: amazon.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 02 Dec 2010 07:38:04 GMT
< Server: Server
< Location: https://www.amazon.com/
< Content-Length: 231
< Cneonction: close
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.amazon.com/">here</a>.</p>
</body></html>
* Connection #0 to host amazon.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
This succeeds.. since we asked Curl to ignore SSL validation issues..
5. Let's try to invoke a service with a certificate issued from a CA not trusted by Curl. I am using WSO2 ESB here - which ships with a self-signed certificate - so, obviously not trusted by Curl
$ curl -v https://localhost:9447/carbon
* About to connect() to localhost port 9447 (#0)
* Trying ::1... connected
* Connected to localhost (::1) port 9447 (#0)
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html
You will see, above fails since server certification is not trusted.
6. Let's improve step-5 - by asking Curl to trust our CA certificate.
$ curl -v --cacert esbcert.pem https://localhost:9447/carbon
Here we specify CA certificate of the ESB - with the argument --cacert
Notes :
1. To find the path of the trusted CA bundle
$
curl-config --ca
2. To set your own trusted CA bundle
$
export CURL_CA_BUNDLE=my-ca-bundle.crt