Editing Wget

{{Short description|Computer command line program}}
{{Use dmy dates|date=October 2020}}
{{Use American English|date=March 2023}}
{{More citations needed|date=March 2023}}
{{Infobox software
| name = Wget
| screenshot = File:Wget 1.13.4.png
| caption = Screenshot of Wget running on [[Ubuntu (operating system)|Ubuntu]].
| author = Hrvoje Nikšić
| developer = Giuseppe Scrivano, Tim Rühsen, Darshit Shah
| released = {{Start date and age|1996|01|df=yes}}
| latest release version = {{wikidata|property|preferred|references|edit|Q535461|P348|P548=Q2804309}}
| latest release date = {{wikidata|qualifier|preferred|single|Q535461|P348|P548=Q2804309|P577}}
| latest preview version = {{wikidata|property|preferred|references|edit|Q535461|P348|P548=Q51930650}}
| latest preview date = {{wikidata|qualifier|preferred|single|Q535461|P348|P548=Q51930650|P577}}
| platform = [[Cross-platform]]
| programming language = [[C (programming language)|C]]
| genre = [[FTP client]] / [[HTTP client]]
| license = [[GNU General Public License|GPL-3.0-or-later]]{{efn|GPL-3.0-or-later with OpenSSL exception.}}<ref name="license">{{cite web |url= https://git.savannah.gnu.org/cgit/wget.git/tree/README |title= README file |access-date= 2014-12-01}}</ref>
| website = {{URL|https://www.gnu.org/software/wget/}}
}}

'''GNU Wget''' (or just '''Wget''', formerly '''Geturl''', also written as its package name, '''wget''') is a [[computer program]] that retrieves content from [[web servers]]. It is part of the [[GNU Project]]. Its name derives from "[[World Wide Web]]" and "[[HTTP GET|''get'']]", a HTTP request method. It supports downloading via [[HTTP]], [[HTTPS]], and [[File Transfer Protocol|FTP]].

Its features include recursive download, conversion of links for offline viewing of local HTML, and support for proxies. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among [[Unix]] users and distribution with most major [[Linux distribution]]s. Wget is written in [[C (programming language)|C]], and can be easily installed on any Unix-like system. Wget has been ported to [[Microsoft Windows]], [[macOS]], [[OpenVMS]], [[HP-UX]], [[AmigaOS]], [[MorphOS]], and [[Solaris_(operating_system)|Solaris]]. Since version 1.14, Wget has been able to save its output in the web archiving standard [[Web ARChive|WARC]] format.<ref name="FSF2012">
{{Cite web
| url = https://lists.gnu.org/archive/html/info-gnu/2012-08/msg00002.html
| title = GNU wget 1.14 released
| last = Scrivano
| first = Giuseppe
| date = 6 August 2012
| website = GNU wget 1.14 released
| publisher = Free Software Foundation, Inc.
| access-date = 25 February 2016
}}
</ref>

==History==
Wget descends from an earlier program named Geturl by the same author,<ref name="2007-newsfile" /> the development of which commenced in late 1995. The name changed to Wget after the author became aware of an earlier [[Amiga]] program named GetURL, written by James Burton in [[AREXX]].

Wget filled a gap in the inconsistent web-downloading software available in the mid-1990s. No single program could reliably use both [[HTTP]] and [[File Transfer Protocol|FTP]] to download files. Existing programs either supported FTP (such as [[NcFTP]] and dl) or were written in [[Perl]], which was not yet ubiquitous. While Wget was inspired by features of some of the existing programs, it supported both HTTP and FTP and could be built using only the standard development tools found on every Unix system.

At that time many Unix users struggled behind extremely slow university and [[Dial-up access|dial-up]] [[Internet]] connections, leading to a growing need for a downloading agent that could deal with transient network failures without assistance from the human operator.

==Features==
===Robustness===
Wget has been designed for robustness over slow or unstable network connections. If a [[download]] does not complete due to a [[computer network|network]] problem, Wget will automatically try to continue the download from where it left off, and repeat this until the whole file has been retrieved. It was one of the first clients to make use of the then-new <code>Range</code> [[List of HTTP headers|HTTP header]] to support this feature.

===Recursive download===
Wget can optionally work like a [[web crawler]] by extracting resources [[Hyperlink|linked]] from [[HTML]] [[Web page|pages]] and downloading them in sequence, repeating the process [[Recursion|recursively]] until all the pages have been downloaded or a maximum recursion depth specified by the user has been reached. The downloaded pages are saved in a directory structure resembling that on the remote server. This "recursive download" enables partial or complete mirroring of [[web site]]s via HTTP. Links in downloaded HTML pages can be adjusted to point to locally downloaded material for [[offline]] viewing. When performing this kind of automatic [[mirror website|mirroring]] of web sites, Wget supports the [[Robots Exclusion Standard]] (unless the option <code>-e robots=off</code> is used).

Recursive download works with [[File Transfer Protocol|FTP]] as well, where Wget issues the <code>LIST</code> command to find which additional files to download, repeating this process for directories and files under the one specified in the top [[Uniform Resource Locator|URL]]. Shell-like [[Wildcard character|wildcards]] are supported when the download of FTP URLs is requested.

When downloading recursively over either [[HTTP]] or [[File Transfer Protocol|FTP]], Wget can be instructed to inspect the [[timestamps]] of local and remote files, and download only the remote files newer than the corresponding local ones. This allows easy mirroring of [[HTTP]] and [[File Transfer Protocol|FTP]] sites, but is considered inefficient and more error-prone when compared to programs designed for mirroring from the ground up, such as [[rsync]]. On the other hand, Wget does not require special server-side software for this task.

===Non-interactiveness===
Wget is non-interactive in the sense that, once started, it does not require user interaction and does not need to control a [[tty (Unix)|TTY]], being able to log its progress to a separate file for later inspection. Users can start Wget and [[log out|log off]], leaving the program unattended. By contrast, most [[Graphical user interface|graphical]] or [[text user interface]] [[web browser]]s require the user to remain logged in and to manually restart failed downloads, which can be a great hindrance when transferring a lot of data.

===Portability===
Written in a highly portable style of [[C (programming language)|C]] with minimal dependencies on third-party libraries, Wget requires little more than a C compiler and a BSD-like interface to [[Transmission Control Protocol|TCP/IP]] networking.{{cn|date=December 2022}} Designed as a Unix program invoked from the [[Unix shell]], the program has been ported to numerous Unix-like environments and systems, including [[Microsoft Windows]] via [[Cygwin]], and [[Homebrew (package management software)|macOS]]. It is also available as a native [[Microsoft Windows]] program as one of the [[GnuWin32|GnuWin]] packages.

===Other features===
*Wget supports download through [[Proxy server|proxies]], which are widely deployed to provide web access inside company [[Firewall (networking)|firewalls]] and to cache and quickly deliver frequently accessed content.
*It makes use of [[HTTP persistent connection]]s where available.
*[[IPv6]] is supported on systems that include the appropriate interfaces.
*[[Transport Layer Security|SSL/TLS]] is supported for encrypted downloads using the [[OpenSSL]] or [[GnuTLS]] library.
*Files larger than 2 [[gibibyte|GiB]] are [[Large File Support|supported]] on 32-bit systems that include the appropriate interfaces.
*Download speed may be [[Bandwidth Throttling|throttled]] to avoid using up all of the available [[Bandwidth (computing)|bandwidth]].
*Can save its output in the web archiving standard [[Web ARChive|WARC]] format, deduplicating from an associated CDX file as required.<ref name="FSF2012" />

==Authors and copyright==
GNU Wget was written by Hrvoje Nikšić with contributions by many other people, including Dan Harkless, Ian Abbott, and Mauro Tortonesi. Significant contributions are credited in the ''AUTHORS'' file included in the distribution, and all remaining ones are documented in the [[changelog]]s, also included with the program. Wget is currently maintained by Giuseppe Scrivano, Tim Rühsen and Darshit Shah.<ref>{{cite web|url=https://www.gnu.org/software/wget/|title=Wget - GNU Project|date=30 November 2018|access-date=30 November 2018}}</ref>
<!--The previous maintainer was Micah Cowan. -->

The copyright to Wget belongs to the [[Free Software Foundation]], whose policy is to require copyright assignments for all non-trivial contributions to GNU software.<ref>{{cite web|url=https://www.gnu.org/licenses/why-assign.html |title=Why the FSF gets copyright assignments from contributors - GNU Project - Free Software Foundation (FSF) |publisher=Gnu.org |access-date=2012-12-08}}</ref>

===License===
GNU Wget is distributed under the terms of the [[GNU General Public License]], version 3 or later, with a special [[OpenSSL exception|exception]] that allows distribution of binaries [[Linker (computing)|linked]] against the [[OpenSSL]] library. The text of the exception follows:<ref name="license"/>

<blockquote>
Additional permission under GNU GPL version 3 section 7

If you modify this program, or any covered work, by linking or combining it with the OpenSSL project's OpenSSL library (or a modified version of that library), containing parts covered by the terms of the OpenSSL or SSLeay licenses, the Free Software Foundation grants you additional permission to convey the resulting work. Corresponding Source for a non-source form of such a combination shall include the source code for the parts of OpenSSL used as well as that of the covered work.
</blockquote>

It is expected{{by whom |date=December 2014}} that the exception clause will be removed once Wget is modified to also [[Linker (computing)|link]] with the [[GnuTLS]] library.

Wget's [[documentation]], in the form of a [[Texinfo]] reference manual, is distributed under the terms of the [[GNU Free Documentation License]], version 1.2 or later. The [[man page]] usually distributed on Unix-like systems is automatically generated from a subset of the Texinfo manual and falls under the terms of the same license.

==Development==
Wget is developed in an open fashion, most of the design decisions typically being discussed on the public mailing list<ref name="Gmane">{{cite web |url=http://news.gmane.org/gmane.comp.web.wget.general |title=Gmane Loom |publisher=News.gmane.org |access-date=2012-12-08 |archive-date=9 January 2013 |archive-url=https://web.archive.org/web/20130109011631/http://news.gmane.org/gmane.comp.web.wget.general |url-status=dead }}</ref> followed by users and developers. Bug reports and patches are relayed to the same list.

===Source contribution===
The preferred method of contributing to Wget's code and documentation is through source updates in the form of textual [[Patch (Unix)|patches]] generated by the [[diff]] utility. Patches intended for inclusion in Wget are submitted to the mailing list<ref name="Gmane"/> where they are reviewed by the maintainers. Patches that pass the maintainers' scrutiny are installed in the sources. Instructions on patch creation as well as style guidelines are outlined on the project's wiki.<ref>{{cite web|url=http://wget.addictivecode.org/PatchGuidelines |title=PatchGuidelines - The Wget Wgiki |publisher=Wget.addictivecode.org |date=2009-09-22 |access-date=2012-12-08}}</ref>

The source code can also be tracked via a remote [[version control]] repository that hosts revision history beginning with the 1.5.3 release. The repository is currently running [[Git (software)|Git]].<ref>{{cite web |url=http://wget.addictivecode.org/RepositoryAccess |title=RepositoryAccess |date=31 July 2012 |access-date=7 June 2013}}</ref> Prior to that, the source code had been hosted on (in reverse order): [[Bazaar (software)|Bazaar]],<ref>{{cite web |url=http://wget.addictivecode.org/RepositoryAccess |title=RepositoryAccess |date=22 May 2010 |access-date=20 June 2010}}</ref> [[Mercurial (software)|Mercurial]], [[Subversion (software)|Subversion]], and via [[Concurrent Versions System|CVS]].

====Release====
When a sufficient number of features or bug fixes accumulate during development, Wget is released to the general public via the GNU FTP site and its mirrors. Being entirely run by volunteers, there is no external pressure to issue a release nor are there enforceable release deadlines.

Releases are numbered as [[Software versioning|versions]] of the form of ''major.minor[.revision]'', such as ''Wget 1.11'' or ''Wget 1.8.2''. An increase of the major version number represents large and possibly incompatible changes in Wget's behavior or a radical redesign of the code base. An increase of the minor version number designates addition of new features and bug fixes. A new revision indicates a release that, compared to the previous revision, only contains bug fixes. Revision zero is omitted, meaning that for example Wget 1.11 is the same as 1.11.0. Wget does not use the [[Software versioning#Odd-numbered versions for development releases|odd-even release number convention]] popularized by Linux.

====Popular references====
Wget makes an appearance in the 2010 [[Columbia Pictures]] motion picture release, ''[[The Social Network]]''. The lead character, a somewhat fictionalized version of [[Facebook]] co-founder [[Mark Zuckerberg]], uses Wget to aggregate student photos from various [[Harvard University]] housing-facility directories.

===Notable releases===
The following releases represent notable milestones in Wget's development. Features listed next to each release are edited for brevity and do not constitute comprehensive information about the release, which is available in the ''NEWS'' file distributed with Wget.<ref name="2007-newsfile">{{cite web|url=http://svn.dotsrc.org/repo/wget/tags/WGET_1_10/NEWS |title=GNU Wget NEWS – history of user-visible changes |publisher=Svn.dotsrc.org |date=2005-03-20 |access-date=2012-12-08 |archive-url=https://web.archive.org/web/20070313111309/http://svn.dotsrc.org/repo/wget/tags/WGET_1_10/NEWS |archive-date=13 March 2007 |quote=Wget 1.4.0 [formerly known as Geturl] is an extensive rewrite of Geturl.}}</ref>
*Geturl 1.0, released January 1996, was the first publicly available release. The first English-language announcement can be traced to a Usenet news posting, which probably refers to Geturl 1.3.4 released in June.<ref>{{cite web |url=https://groups-beta.google.com/group/comp.infosystems.www.announce/msg/4268334d269d42ce?hl=en |title=Geturl: Software for non-interactive downloading |date=24 June 1996 |first=Hrvoje |last=Niksic |publisher=comp.infosystems.www.announce |access-date=17 November 2016}}</ref>
*Wget 1.4.0, released November 1996, was the first version to use the name ''Wget''. It was also the first release distributed under the terms of the [[GNU GPL]], Geturl having been distributed under an ad hoc no-[[warranty]] [[Software license|license]].
*Wget 1.4.3, released February 1997, was the first version released as part of [[GNU|the GNU project]] with the copyright assigned to the [[Free Software Foundation|FSF]].
*Wget 1.5.3, released September 1998, was a milestone in the program's popularity. This version was bundled with many [[Linux]] based distributions, which exposed the program to a much wider audience.
*Wget 1.6, released December 1999, incorporated many bug fixes for the (by then stale) 1.5.3 release, largely thanks to the effort of Dan Harkless.
*Wget 1.7, released June 2001, introduced [[Transport Layer Security|SSL]] support, [[HTTP cookie|cookies]], and [[HTTP persistent connection|persistent connections]].
*Wget 1.8, released December 2001, added [[bandwidth throttling]], new progress indicators, and the breadth-first traversal of the hyperlink [[Graph (discrete mathematics)|graph]].
*Wget 1.9, released October 2003, included experimental [[IPv6]] support, and ability to POST data to HTTP servers.
*Wget 1.10, released June 2005, introduced [[large file support]], [[IPv6]] support on dual-family systems, [[NTLM]] authorization, and [[Transport Layer Security|SSL]] improvements. The maintainership was picked up by Mauro Tortonesi.
*Wget 1.11, released January 2008, moved to version 3 of the [[GNU General Public License]], and added preliminary support for the <code>Content-Disposition</code> header, which is often used by [[Common Gateway Interface|CGI]] scripts to indicate the name of a file for downloading. Security-related improvements were also made to the HTTP authentication code. Micah Cowan took over maintainership of the project.
*Wget 1.12, released September 2009, added support for parsing URLs from [[Cascading Style Sheets|CSS]] content on the web, and for handling [[Internationalized Resource Identifier]]s.
*Wget 1.13, released August 2011, supports HTTP/1.1, fixed some portability issues, and used the [[GnuTLS]] library by default for secure connections.<ref>{{Cite web|url=https://bzr.savannah.gnu.org/lh/wget/trunk/annotate/head:/NEWS|title=/wget/trunk : contents of NEWS at revision 2608|website=bzr.savannah.gnu.org}}</ref>
*Wget 1.14, released August 2012, improved support for TLS and added support for {{IETF RFC|2617}} Digest Access Authentication.
*Wget 1.15, released January 2014, added—https-only and support for Perfect-Forward Secrecy.
*Wget 1.16, released October 2014, changed the default progress bar output, closed {{CVE|2014-4877}}, added support for libpsl to verify cookie domains, and introduced—start-pos to allow starting downloads from a specified position.
*Wget 1.17, released November 2015, removed FTP passive to active fallback due to privacy concerns, added support for FTPS and for—if-modified-since.
*Wget 1.18, released June 2016, resolved the {{CVE|2016-4971}} issue, and added the "--bind-dns-address" and "--dns-servers" options.
*Wget 1.19, released February 2017, added new options for processing a Metalink file; version 1.19.1 added the—retry-on-http-error option to retry a download if the Web server responds with a given HTTP status code.
*Wget 1.20, released November 2018, added <code>--retry-on-host-error</code> for more reliability and <code>--accept-regex</code>, <code>--reject-regex</code> options for recursive FTP retrievals.

==Wget2==
{{Infobox software
| name = Wget2
| released = {{Start date and age|2021|09|26|df=yes}}
| latest release version = {{wikidata|property|preferred|references|edit|Q97458426|P348|P548=Q2804309}}
| latest release date = {{wikidata|qualifier|preferred|single|Q97458426|P348|P548=Q2804309|P577}}
| latest preview version = {{wikidata|property|preferred|references|edit|Q97458426|P348|P548=Q51930650}}
| latest preview date = {{wikidata|qualifier|preferred|single|Q97458426|P348|P548=Q51930650|P577}}
| repo = {{URL|gitlab.com/gnuwget/wget2}}
| license = [[GNU General Public License|GPL-3.0-or-later]]<ref name="GNU Wget2 2.0.0" />
| website = {{URL|www.gnu.org/software/wget/}}
}}

GNU Wget2 2.0.0 was released on 26 September 2021. It is licensed under the [[GNU General Public License|GPL-3.0-or-later]] license, and is wrapped around Libwget which is under the [[GNU Lesser General Public License|LGPL-3.0-or-later]] license.<ref name="GNU Wget2 2.0.0">{{Cite web|url=https://lists.gnu.org/archive/html/info-gnu/2021-09/msg00011.html|website=gnu.org|title=GNU Wget2 2.0.0 released|date=2021-09-26}}</ref> It has many improvements in comparison to Wget, particularly, in many cases Wget2 downloads much faster than Wget1.x due to support of the following protocols and technologies:<ref>{{Cite web|url=https://gitlab.com/gnuwget/wget2|website=GitLab|title=wget2}}</ref>

* [[HTTP/2]],
* [[HTTP compression]],
* parallel connections,
* use of [[List_of_HTTP_header_fields|If-Modified-Since]] HTTP header,
* [[TCP Fast Open]].

==Related works==
{{Multiple images
|size = 220
|direction = vertical

|image1 = Gwget-1.0.4.png
|caption1 = Screenshot of GWget 1.0.4 in [[Fedora (operating system)|Fedora]] v12 with [[GNOME]] v2.28.2 installed
}}

===GWget===
GWget is a free software [[graphical user interface]] for Wget. It is developed by David Sedeño Fernández based on the [[GNOME]] software stack. GWget supports all of the main features that Wget does, as well as parallel downloads.<ref>{{Cite web|url=https://sourceforge.net/projects/gwget/|title=GWget Home Page|date=22 March 2013 }}</ref>

===Cliget===
Cliget is an open source [[Firefox]] addon downloader that uses [[CURL|Curl]], Wget and Aria2. It is developed by Zaid Abdulla.<ref>{{Cite web|url=https://github.com/zaidka/cliget|title=zaidka/cliget|website=GitHub|access-date=2016-08-25}}</ref><ref>{{Cite web|url=https://addons.mozilla.org/en-US/firefox/addon/cliget/developers|title=Meet the cliget Developer :: Add-ons for Firefox|website=addons.mozilla.org|access-date=2016-08-25|archive-date=21 September 2016|archive-url=https://web.archive.org/web/20160921114045/https://addons.mozilla.org/en-US/firefox/addon/cliget/developers|url-status=dead}}</ref><ref>{{Cite web|url=https://addons.mozilla.org/en-US/firefox/addon/cliget/|title=cliget|website=addons.mozilla.org|access-date=2016-08-25}}</ref>

== Clones ==
There exist clones of GNU Wget intended for [[Embedded system|embedded systems]], which are often limited in memory and storage. They support its most basic options, usually limited to downloading.
* [[OpenWrt]] uclient-fetch<ref>{{Cite web|url=https://git.openwrt.org/?p=project%2Fuclient.git%3Ba%3Dsummary|title=git.openwrt.org Git - project/uclient.git/summary|website=git.openwrt.org|access-date=6 October 2021|archive-date=6 October 2021|archive-url=https://web.archive.org/web/20211006232052/https://git.openwrt.org/?p=project%2Fuclient.git%3Ba%3Dsummary|url-status=dead}}</ref>
* [[BusyBox]] wget<ref name="auto">{{Cite web|url=https://github.com/landley/toybox/blob/1986075975db4cdab08133f8299ad72dd5af4b23/toys/pending/wget.c|title=landley/toybox|date=6 October 2021|via=GitHub}}</ref>
* [[ToyBox]] wget<ref name="auto"/>

==See also==
{{Portal|Free and open-source software}}
*[[cURL]]
*[[HTTrack]]
*[[lftp]]
*[[Web crawler]]
*[[PowerShell]] iwr Invoke-WebRequest command

==Notes==
{{notelist}}

==References==
{{Reflist|30em}}

==External links==
{{Commons category|GNU Wget}}
* {{official website}}
* [https://gitlab.com/gnuwget/wget2/-/wikis/Home#differing-cli-options-wgetwget2 Differing CLI options Wget/Wget2]
* {{cite web |url= https://daniel.haxx.se/docs/curl-vs-wget.html |title= Comparison of cURL vs wget |author-link= Daniel Stenberg |first= Daniel |last= Stenberg |date= Nov 9, 2021 |orig-year= 2007 }}


{{Download managers}}
{{GNU}}
{{Web crawlers}}

[[Category:1996 software]]
[[Category:Command-line software]]
[[Category:Cross-platform free software]]
[[Category:Download managers]]
[[Category:Free FTP clients]]
[[Category:Free web crawlers]]
[[Category:GNU Project software]]
[[Category:Hypertext Transfer Protocol clients]]
[[Category:Portable software]]
[[Category:Text mode]]
[[Category:Web archiving]]
[[Category:Web scraping]]