cURL is an incredibly flexible and powerful tool data transferring tool, that supports a wide variety of protocols, being the HTTP (and HTTPS) the most used. cURL library is available for many programming languages, but in this example we will see how to use it from the command line.
For this tutorial, Mint 17.3 has been used.
1. Installation
cURL can be installed easily via apt-get:
sudo apt-get update sudo apt-get install curl
2. Basic usage
The simplest way to use cURL is just executing it, and specifying a site:
curl <site>
For example
curl google.com
And the terminal would show:
<HTML>
<HEAD>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE>
</HEAD>
<BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.es/">here</A>.
</BODY>
</HTML>Which is not what we were expecting. This is because, by default, cURL doesn’t follow 3XX redirections. To make cURL follow redirections, we have to pass the -L option:
curl -L google.com
To save the HTTP response returned by cURL, we can always make a shell redirect with >, or use the -o option:
curl -o google.html -L google.com
When used, data about the receiving response is shown:
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 219 100 219 0 0 2259 0 --:--:-- --:--:-- --:--:-- 2281 0 0 0 11675 0 0 50630 0 --:--:-- --:--:-- --:--:-- 10.5M
To see more information, including the HTTP headers, we have available the -v (--verbose) option:
curl -v google.com
Which will show something similar to the following before the HTTP response:
* Rebuilt URL to: google.com/ * Hostname was NOT found in DNS cache * Trying 130.206.193.53... * Connected to google.com (130.206.193.53) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.35.0 > Host: google.com > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: http://www.google.es/?gfe_rd=cr&ei=OulzV836N9Op8weYzoxY < Content-Length: 256 < Date: Wed, 29 Jun 2016 15:28:58 GMT < # HTML
3. Requests
3.1. GET
Making GET request with cURL is so simple. Actually, is as same as making it in the browser. For example:
curl -o systemcodegeeks.html https://www.systemcodegeeks.com?s=linux
We just have to add the query string in field=value& format after ?.
3.2. POST
To make a POST request, the usage is slightly different. The syntax is the following:
curl -d "field1=value1&field2=value2&fieldN=valueN" <site>
So, for making a search as the one done above, but supposing that the form method is POST, would be:
curl -d "s=linux" systemcodegeeks.com -o systemcodegeeks.html
Of course, we can make a POST request for uploading files:
curl -F "field=@/path/to/file.txt" awebsite.com/uploadfile
In this case, we have to specify the -F option for the file. And, the leading @ before the path to the file is necessary.
For making a POST request that combines both files and normal data, the data has to bet set also with -F, and not with -d. This is because the forms for file uploads sets the data encoding to -multipart/form-data, and this is what -F option does; whereas -d option sets the encoding to application/x-www-form-urlencoded. And all the data in a form has to be encoded under the same encoding. So, we have to specify also the data with -F option:
curl -F "field=value" -F "file=@/path/to/file.txt" awebsite.com/uploadfile
3.3. Custom requests: PUT and DELETE
PUT and DELETE are considered custom-methods. These requests are mostly used in CRUDs (Create, Read, Update, Delete), in RESTful services. To specify these commands, we have to use -X option.
For example:
curl -X PUT -d "data=stuff" http://myserver.com/rows/1 curl -X DELETE http://myserver.com/rows/1
3.4. Adding extra headers
For adding extra headers to a request, we just have to use the -H option (--header), specifying the header, and the value:
curl --header "X-ExtraHeader: myheader" http://myserver.com
4. FTP
We can also manage FTP server with cURL:
curl -u username:password ftp://myftp.com # Or curl fpt://username:password@myftp.com
We must take into account that, unless SSL is being used, the data will travel in plain text. If SSL is available, ftps should be use, instead of ftp.
We can upload files pretty simply, just specifying the local file with -T option:
curl -T file.txt ftps://username:password@myftp.com/path
Several files can be uploaded at once:
curl -T {file1.txt, file2.txt} ftps://username:password@myftp.com/pathDownloading files from FTP server is the same as downloading a simple HTML:
curl ftps://username:password@myftp.com/path/file.txt -o file.txt
5. Other interesting options
Apart from the common uses of cURL, let’s see other features that may result useful in some occasions.
5.1. HTTP authentication
With cURL is also possible to make HTTP authentication (that login prompts shown by the browser, not the authentication forms). For that, we have to use the -u (--user) option and specify the credentials:
curl -u username:password https://mylogin.com
5.2. Ignoring SSL errors
You will probably ever have encountered a site for which the browser shows a SSL certificate error, that it can be caused because it’s expired, because the certificate authority is not trusted by the browser, or any other reason. If we want to ignore the warning and access the site, we have to give explicit consent to the browser. By default, cURL does not accept the connections with SSL errors, but we can make it ignore them with -k (--insecure) option:
curl -k https://insecure-site.com
5.3. Cookies
Setting custom cookies with cURL is as easy as specifying them in name=value format, with -b (--cookie) option:
curl -b "mycookie=value" systemcodegeeks.com
We can even specify the cookies from a file!
curl -b cookies.txt systemcodegeeks.com
And, of course, we can save the cookies to a file, with -c (--cookie-jar) option:
curl -c cookies.txt systemcodegeeks.com
5.4. Making requests through proxy
Yes, cURL also allows to make requests through proxies. We just have to use the -x option, specifying the proxy:
curl -x https://aproxy.com systemcodegeeks.com
If the proxy server requires authentication, we have to specify as same as with HTTP authentication, but with --proxy-user option:
curl -x https://aproxy.com --proxy-user username:password systemcodegeeks.com
5.5. Download depending on the modification time
This is nice. We can make cURL download a file only if it has suffered modifications after the specified time, with -z (--time-cond) option:
curl -z 25 -Jun-16 systemcodegeeks.com
This feature can be specially useful for automating the task of having up-to-date documentation or software, for example.
6. Summary
This example has shown how to use cURL, from its simplest uses, which are downloading pages or making simple requests, to more advanced topics (which are actually easy with cURL), such us FTP connection, setting and saving cookies, or using proxies.
