Linux wget Command Guide [With Examples]

Linux wget Command

If you followed our Magic Mirror tutorial, you’d see the wget command was used to download some files.

This tutorial explains how to use wget, a command-line tool for downloading (getting) files from the web, be it via HTTP, HTTPS, FTP, or FTPS.

Use it to download files from the internet from the Linux shell, call it from Bash scripts – it’s simple and versatile and doesn’t require user interference once started so that it can run in the background.

Syntax

wget [OPTIONS]... [ URLS ]...

Note that:

  • OPTIONS is a list of options that can be passed to the application from the below table, separated by a space
  • URLS is a list of URLs to download, separated by a space

Here are the wget options, straight from the docs:

Common Options
-o logfile Log all messages to logfile. The messages are normally reported to standard error.
-a logfile Append to logfile. This is the same as -o, only it appends to logfile instead of overwriting the old log file. If logfile does not exist, a new file is created.
-q Turn off Wget’s output.
-i file Read URLs from a local or external file. If this function is used, no URLs need be present on the command line.
-t number Set number of retries to number.
-c Continue getting a partially-downloaded file.
-T seconds Set the network timeout to seconds seconds.
-w seconds Wait the specified number of seconds between the retrievals.
–user=user Set the HTTP or FTP authentication username.
–password=password Set the HTTP or FTP authentication password.
–post-data=string Make a POST request instead of GET and send data. string should be in the format “key1=value1&key2=value2”
-r Turn on recursive retrieving.
-l depth Specify recursion maximum depth level depth. The default maximum depth is 5.
-m Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings.
-p This option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets.

Examples

Download a File from an HTTPS Server

Download a single file, basic usage:

wget https://www.example.com/file.zip

Continue Downloading a File

If a download only partially completed, continue/resume downloading it with the -c option:

wget -c https://www.example.com/file.zip

Download From a List of Files, Appending to Log

If you have a text file containing a list of URLs to download, you can pass it directly to wget and write a log of the results for later inspection:

wget -a log.txt -i url-list.txt

You could also use -o to write out the log file, and it will overwrite rather than append an existing log file if it’s already there.

Download a File, Retry 5 Times, Quietly

Retry downloading a file and don’t’ print progress to the terminal:

wget -t 5 -q https://www.example.com/file.zip

Download From a List of Files, Waiting 6 Seconds Between Each Download, with a 12 Second Timeout

Wait between downloads to reduce server load, and abort if the server fails to respond within 12 seconds:

wget -w 6 -T 12 -i url-list.txt

Download a File from an FTPS Server which Requires a Username and Password

Download from an FTPS server with the username bob and the password boat:

wget --user=bob --password=boat ftps://ftp.example.com/file.zip

Download a File with a POST Request

Make an HTTP POST request instead of the default GET request, and send data. A blank string can be sent with –post-data:

wget --post-data="postcode=2000&country=Australia" https://www.example.com/file.zip

In this example, we’re sending two pieces of POST data – postcode and country.

If making POST requests, cURL can be more versatile.

Download Directory Recursively via FTP with a Depth Limit

Downloading recursively will download the contents of a folder and the contents of the folders in that folder. A depth limit of 3 is defined in this example – meaning that if a folder is nested within 3 other folders, it won’t be downloaded:

wget -r -l 3 ftps://ftp.example.com/path/to/folder

Downloading a Whole Directory including ALL Contents via FTP

wget -m ftps://ftp.example.com/path/to/folder

Cloning a Whole Web Page Using Wget

If you want to try and grab a whole webpage – including all images, styles, and scripts, you can use

wget -p https://www.example.com/page.html

Your success will vary – some modern web pages don’t really work well when ripped out of their native habitat.

SHARE:
nv-author-image

Brad Morton

I'm Brad, and I'm nearing 20 years of experience with Linux. I've worked in just about every IT role there is before taking the leap into software development. Currently, I'm building desktop and web-based solutions with NodeJS and PHP hosted on Linux infrastructure. Visit my blog or find me on Twitter to see what I'm up to.

Leave a Reply

Your email address will not be published. Required fields are marked *