curl command examples

Transferring files using the curl

"CURL" is a computer software project that provides a library (libcurl) and a command line tool called curl. Curl is a tool to transfer data to or from a server using one of the many supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). Support for authentication , Proxy servers, transfer resume, SSL are all catered for. For the full list of functionality see the "man pages" for curl.

Below are examples of how to install curl on popular Linux distributions along with some useful examples of the curl tool in action.

Installing cURL

On many distributions you will find that curl is already included. If not, it can easily be installed:

Debian/Ubuntu Installing Curl

To install curl on a Debian or Ubuntu/Mint system you need to issue the following commands:

First, lets check to see if curl is installed:


sudo apt show curl

If curl is not installed, it can be installed with the following command:


sudo apt update
sudo apt install curl

Red Hat/CentOS Installing curl

In the majority of instances curl will be automatically installed with most server builds. We can verify that curl is installed by either using the "rpm -qa" command or by issuing the relevant "dnf info curl" command:

Below is an example taken from a CentOS server


# rpm -qa | grep curl
libcurl-7.61.1-14.el8_3.1.x86_64
curl-7.61.1-14.el8_3.1.x86_64

# dnf info curl
Last metadata expiration check: 0:36:26 ago on Tue 25 May 2021 20:40:00 BST.
Installed Packages
Name         : curl
Version      : 7.61.1
Release      : 14.el8_3.1
Architecture : x86_64
Size         : 693 k
Source       : curl-7.61.1-14.el8_3.1.src.rpm
Repository   : @System
From repo    : baseos
Summary      : A utility for getting files from remote servers (FTP, HTTP, and
             : others)
URL          : https://curl.haxx.se/
License      : MIT

If curlis not installed, it can be installed with the following command: dnf install curl -y

openSUSE/SLES

If you are working with an openSUSE or SLES (SUSE Linux Enterprise Server) you can use the "zypper se curl" command to verify that the relevant packages have been installed already:


# zypper se curl
Loading repository data...
Reading installed packages...

S | Name                | Summary                                   | Type     
--+---------------------+-------------------------------------------+-----------
i | curl                | A Tool for Transferring Data from URLs    | package  
  | curl                | A Tool for Transferring Data from URLs    | srcpackage
  | curlftpfs           | Filesystem for mounting FTP hosts using-> | package  
  | flickcurl           | Command-Line Tools for the Flickr Web S-> | package  
  | flickcurl-doc       | C Library API to the Flickr Web Service-> | package  
  | libcurl-devel       | A Tool for Transferring Data from URLs    | package  
i | libcurl4            | Version 4 of cURL shared library          | package  
  | libcurl4-32bit      | Version 4 of cURL shared library          | package  
  | libflickcurl-devel  | C Library API to the Flickr Web Service   | package  
  | libflickcurl0       | C Library API to the Flickr Web Service   | package  
  | libflickcurl0-32bit | C Library API to the Flickr Web Service   | package  
  | perl-WWW-Curl       | Perl extension interface for libcurl      | package  
  | php5-curl           | PHP5 Extension Module                     | package  
i | python-pycurl       | PycURL -- cURL library module             | package  
  | python-pycurl-doc   | Documentation for python-curl             | package  
  | python3-pycurl      | PycURL -- cURL library module             | package  
  | python3-pycurl-doc  | Documentation for python-curl             | package  
  | tclcurl             | Tcl Binding to libcurl                    | package  
  | xmms2-plugin-curl   | Curl Support for xmms2                    | package 

From the above output we can see that the "curl, libcurl and python-pycurl" packages are installed. This is indicated by an "i in the first column.

If "curl" is not installed, it can be installed by issuing the command "zypper in curl". This command will install curl along with any dependencies.

Retrieving a web page


$ curl https://landoflinux.com/linux_wget_command.html

By issuing the "curl" command followed by a web address, we can retrieve the page source information. This output is displayed directly to your terminal.

If you wanted to send the output from this command to a file we can use re-direction:


$ curl https://landoflinux.com/linux_wget_command.html > test.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26003  100 26003    0     0  71241      0 --:--:-- --:--:-- --:--:-- 71241

Now our output will be sent to a file called "test.txt". Information relating to the file transfer is indicated in the output.

Another option for retrieving a file(s) to a local destination is to use the Output options flags "-o" or "-O".

Specifying a file name with the curl command option -o

The "-o" specifies that the specified name is to be used for the retrieved file:


$ curl -o myfile.html https://landoflinux.com/linux_wget_command.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26003  100 26003    0     0  84152      0 --:--:-- --:--:-- --:--:-- 84152

To verify your file, you can check the first 5 lines of the retrieved file "myfile.html" by issuing the command "head -n 5 myfile.html":


$ head -n 5 myfile.html 
<!DOCTYPE HTML>
<html>
	<head>
		<title>Linux wget Command Examples</title>
		<meta http-equiv="content-type" content="text/html; charset=utf-8" />

Using the retrieved file name with the curl option -O

The "-O" option specifies that the saved file will use the same name as the file on the remote server.


$ curl -O https://landoflinux.com/linux_wget_command.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26003  100 26003    0     0  60331      0 --:--:-- --:--:-- --:--:-- 60331

$ ls -l linux_wget_command.html 
-rw-rw-r-- 1 john john 26003 May 25 21:37 linux_wget_command.html

Downloading Multiple Files

As we have seen from the above examples we can specify a single file to retrieve, however it is also possible to retrieve multiple files in a similar way. Below is an example of two files being retrieved with the "-O" option specified:


$ curl -O https://landoflinux.com/linux_books.html -O https://landoflinux.com/linux_wget_command.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10145  100 10145    0     0  29577      0 --:--:-- --:--:-- --:--:-- 29577
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26003  100 26003    0     0   159k      0 --:--:-- --:--:-- --:--:--  159k

The above output shows the two specified files being retrieved successfully. This is shown by the "100 %" in the first column.

Continuing - Resuming a file transfer using -C -

The option "-C" can be used to continue/resume a file transfer that has been disrupted. This is a very useful feature if you are downloading a large file and you want to interrupt the transfer or the transfer gets disrupted. The "-C -" tells curl to automatically find out where/how to resume the file transfer. The output/input files are then used to calculate the starting position (resume position).

To demonstrate this feature we can initiate a large download and then interrupt this by hitting "CTRL + C". We should then be able to resume the download by specifying the "-C" option:


$ curl -O https://mirror.bytemark.co.uk/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 1780M    3 65.4M    0     0  4209k      0  0:07:12  0:00:15  0:06:57 5094k^C

Now we can resume the download by issuing the command: curl -C - -O https://mirror.bytemark.co.uk/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso


$ curl -C - -O https://mirror.bytemark.co.uk/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso
** Resuming transfer from byte position 73859072
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1709M    0  368k    0     0  49021      0 10:09:28  0:00:07 10:09:21     0^C

We can see that from the above output that the file transfer has been resumed. This is indicated by the message "** Resuming transfer from byte position 73859072".

Summary

Although we have shown several examples of curl in use. There are many more features/functions available within this tool. More information can be found at the main homepage for curl:

Curl Homepage

curl --help

For a quick overview of some of the parameters available you can issue the command "curl --help" or consult the man pages with the command "man curl".