Looking for a powerful command-line tool to transfer data from or to a server without any user interaction? Look no further than curl! This versatile utility supports a range of protocols, including HTTP, HTTPS, SCP, SFTP, and FTP, and comes packed with features like resume transfers, bandwidth limiting, proxy support, and user authentication.
In this tutorial, we’ll guide you through the basics of using curl, providing practical examples and detailed explanations of its most commonly used options. Get ready to take your data transfer game to the next level with curl!
Installing Curl
Most Linux distributions come with the Curl package pre-installed. To verify if curl is installed on your system, launch your console and enter the command curl. If the package is already installed, the console should display a message that says curl: try 'curl --help' or 'curl --manual' for more information. However, if curl is not installed, you will receive an error message stating curl command not found.
In case curl is not installed on your system, you can simply install it using the package manager that comes with your distribution.
Install Curl on Ubuntu and Debian
sudo apt update
sudo apt install curlInstall Curl on CentOS and Fedora
sudo yum install curlHow to Use Curl
The curl command has the following syntax:
curl [options] [URL...]When called without any flags, curl presents the designated resource on the standard output in its most basic configuration.
To fetch the example.com home page, you could execute the following command:
curl example.comThis command will display the source code of the homepage for example.com in your terminal window.
If the protocol is not specified, curl will attempt to determine the protocol you intend to use and will use HTTP by default.
Save the Output to a File
You can use either the -o or -O option to store the output of the curl command.
If you use the lowercase -o option, the file will be saved with a default filename, such as vue-v2.6.10.js in the example below:
curl -o vue-v2.6.10.js https://cdn.jsdelivr.net/npm/vue/dist/vue.jsThe file can be saved with its original filename by using the uppercase -O:
curl -O https://cdn.jsdelivr.net/npm/vue/dist/vue.jsDownload Multiple files
To download several files simultaneously, utilize numerous -O selections, proceeded by the web address of the desired files for download.
For instance, we can download the iso files for Arch Linux and Debian as demonstrated below:
curl -O http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso \
-O https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.isoResume a Download
The option -C - can be employed to resume a download in case of a connection failure while downloading a large file. This approach is beneficial as it allows you to avoid restarting the download from the beginning and continue from where it was interrupted.
To illustrate, suppose you are in the process of downloading the Ubuntu 18.04 iso file, then you can use the command:
curl -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.isoIn the event of an abrupt disconnection, you can resume the download by:
curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.isoGet the HTTP Headers of a URL
HTTP headers consist of key-value pairs separated by colons, which carry information such as the user agent, content type, and encoding. These headers are exchanged between the client and the server along with the request or response.
To retrieve only the HTTP headers for a specific resource, you can utilize the -I option:
curl -I --http2 https://www.ubuntu.com/
Test if a Website Supports HTTP/2
To verify if a specific URL is compatible with the new HTTP/2 protocol, use the combination of the -I flag and the --http2 option to retrieve the HTTP Headers.
curl -I --http2 -s https://trioguide.com/ | grep HTTPWhen utilizing the -s flag, curl operates in a muted manner, concealing both the progress meter and any error messages.
In case the distant server has HTTP/2 capability, curl will show HTTP/2.0 200:
Output:
HTTP/2 200In case of all other scenarios, the reply shall be HTTP/1.1 200:
Output:
HTTP/1.1 200 OKHTTP/2 is automatically enabled for all HTTPS connections if you’re using curl version 7.47.0 or later, so there’s no need to use the --http2 option.
Follow Redirects
The curl command doesn’t automatically follow HTTP Location headers, which means that when attempting to access the non-www version of google.com, you will be redirected to the www version instead of receiving the page source.
curl google.com
With the -L option enabled, curl is directed to track and adhere to any redirects until it arrives at the ultimate destination:
curl -L google.comChange the User-Agent
On occasions during file downloads, the Curl User-Agent could be blocked by the remote server or the contents may differ based on the device and browser of the visitor.
To address such scenarios, the -A option can be utilized to simulate a different browser.
As an instance, to mimic Firefox 60, the following can be used:
curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https://getfedora.org/Specify a Maximum Transfer Rate
You can restrict the data transfer rate using the --limit-rate parameter, which allows you to set a value in bytes, kilobytes (k), megabytes (m), or gigabytes (g).
For instance, the below example demonstrates how to download the Go binary with the curl command and restrict the download speed to 1 MB per second:
curl --limit-rate 1m -O https://dl.google.com/go/go1.10.3.linux-amd64.tar.gzThis feature can be advantageous in limiting curl’s utilization of the entire bandwidth.
Transfer Files via FTP
To use curl for accessing a secured FTP server, you need to use the -u flag followed by the FTP username and password. Here’s an example command:
curl -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/This command enables you to view all files and directories available in the user’s home directory once you log in.
If you wish to download a specific file from the FTP server, use the following syntax:
curl -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/file.tar.gzOn the other hand, if you want to upload a file to the FTP server, use the -T flag followed by the filename you want to upload, like this:
curl -T newfile.tar.gz -u FTP_USERNAME:FTP_PASSWORD ftp://ftp.example.com/Send Cookies
At times, it may be necessary to use specific cookies to make an HTTP request, either for accessing a remote resource or for troubleshooting purposes.
By default, curl does not send or store any cookies when requesting a resource.
To transmit cookies to the server, utilize the -b option, followed by either a file name containing the cookies or a string.
For instance, to download the jdk-10.0.2_linux-x64_bin.rpm file for Oracle Java JDK, you will require a cookie named oraclelicense with the value a. To accomplish this, execute the following command:
curl -L -b "oraclelicense=a" -O http://download.oracle.com/otn-pub/java/jdk/10.0.2+13/19aef61b38124481863b1413dce1855f/jdk-10.0.2_linux-x64_bin.rpmUsing Proxies
To utilize various types of proxies such as HTTP, HTTPS, and SOCKS, curl provides the -x (--proxy) option, which requires the proxy URL to transfer data via a proxy server.
For instance, the following curl command downloads a particular resource through a proxy server at IP address 192.168.44.1 and port 8888:
curl -x 192.168.44.1:8888 http://linux.com/In the event that the proxy server mandates authentication, the -U (--proxy-user) option can be utilized followed by the username and password separated by a colon (user:password):
curl -U username:password -x 192.168.44.1:8888 http://linux.com/Conclusion
Curl is a powerful command-line utility that enables you to transfer data to or from a remote server effortlessly. It serves as a useful tool for a wide range of tasks, including resolving issues, downloading files, and much more.
The examples provided in this tutorial are straightforward and highlight the most commonly used curl options, offering a basic understanding of how the curl command functions. By using this guide, you can quickly get started with curl and explore its capabilities.
If you’re looking for additional information on curl, you can visit the Curl Documentation page. And if you have any queries or feedback, please leave a comment below. Our team is always ready to assist you
