Editing Large-file support

{{Lead too short|date=February 2020}}
{{Use dmy dates|date=January 2020|cs1-dates=y}}
{{Use list-defined references|date=January 2022}}
'''Large-file support''' ('''LFS''') is the term frequently applied to the ability to create files larger than either 2 or 4&nbsp;[[Gibibyte|GiB]] on [[32-bit]] [[filesystem]]s.

==Details==
Traditionally, many operating systems and their underlying [[file system]] implementations used [[32-bit]] [[integer]]s to represent [[computer file|file]] sizes and positions. Consequently, no file could be larger than 2<sup>32</sup> − 1 bytes (4 GiB − 1). In many implementations, the problem was exacerbated by treating the sizes as [[Sign (mathematics)|signed]] numbers, which further lowered the limit to 2<sup>31</sup> − 1 bytes (2 GiB − 1). Files that were too large for 32-bit operating systems to handle came to be known as ''large files''.

While the limit was quite acceptable at a time when [[hard disk]]s were smaller, the general increase in storage capacity combined with increased server and desktop file usage, especially for [[database]] and [[multimedia]] files, led to intense pressure for OS vendors to overcome the limitation.

In 1996, multiple vendors responded by forming an industry initiative known as the '''Large File Summit''' to support large files on POSIX (at the time Windows NT already supported large files on NTFS), an obvious [[backronym]] of "LFS". The summit was tasked to define a standardized way to switch to [[64-bit]] numbers to represent file sizes.<ref name="Solaris_1996"/>

This switch caused deployment issues and required design modifications, the consequences of which can still be seen:
* The change to 64-bit file sizes frequently required incompatible changes to file system layout, which meant that large-file support sometimes necessitated a file system change.  For example, the [[FAT32]] file system does not support files larger than 4&nbsp;GiB−1 (with older applications even only 2&nbsp;GiB−1); the variant [[FAT32+]] does support larger files (up to 256&nbsp;GiB−1), but (so far) is only supported in some versions of [[DR-DOS]],<ref name="Kuhnt-Georgiev-Davis_2007_FAT+"/><ref name="Kuhnt_2011_EDR"/> so users of [[Microsoft Windows]] have to use [[NTFS]] or [[exFAT]] instead.
* To support binary compatibility with old [[application software|application]]s, operating system [[application programming interface|interface]]s had to retain their use of 32-bit file sizes and new interfaces had to be designed specifically for large-file support.
* To support writing [[porting|portable]] code that makes use of LFS where possible, [[C standard library]] authors devised mechanisms that, depending on [[C preprocessor|preprocessor]] constants, transparently redefined the functions to the 64-bit large-file aware ones.
* Many old interfaces, especially [[C (programming language)|C]]-based ones, explicitly specified argument types in a way that did not allow straightforward or transparent transition to 64-bit types.  For example, the C functions <code>[[fseek]]</code> and <code>ftell</code> operate on file positions of type <code>long int</code>, which is typically 32 bits wide on 32-bit platforms, and cannot be made larger without sacrificing backward compatibility.  (This was resolved by introducing new functions <code>fseeko</code> and <code>ftello</code> in [[POSIX]].<ref name="Unix_1996_LFS"/> On Windows machines, under Visual C++, functions <code>_fseeki64</code> and <code>_ftelli64</code> are used.)

==Adoption==
The usage of the large-file API in 32-bit programs had been incomplete for a long time. An analysis did show in 2002 that many base libraries of operating systems were still shipped without large-file support thereby limiting applications using them.<ref name="Largefile_Distros"/> The much-used [[zlib]] library started to support 64-bit large-files on 32-bit platform not before 2006.<ref name="ZLib_Changelog"/>

The problem disappeared slowly with PCs and workstations moving completely to [[64-bit computing]]. Microsoft Windows Server 2008 has been the last server version to be shipped in 32-bit.<ref name="Kolokythas_2007"/> [[Red Hat Enterprise Linux#Red Hat Enterprise Linux 7.x|Redhat Enterprise Linux 7]] was published in 2014 only as a 64-bit operating system.<ref name="RHEL_2014_32bit"/> Ubuntu Linux stopped delivering a 32-bit variant in 2019.<ref name="Cooke_2019_32bit"/> Nvidia stopped to develop 32-bit drivers in 2018 and deliver updates after January 2019.<ref name="Addams_2018_Nvidia"/> Apple stopped developing 32-bit Mac OS versions in 2018 delivering [[macOS Mojave]] only as a 64-bit operating system.<ref name="Silver_2018_Apple"/> The [[End-of-life (product)|end-of-life]] for Windows 10 has been set to 2025 on the desktop which is related to the latest upgrades from old systems like Windows 7 & Windows 8 in January 2020 as some of those system ran on old computers built on the i386 architecture.<ref name="Microsoft_Windows7"/> [[Windows 11]] however will ship only as a 64-bit operating system since its first version in 2021.

A similar development can be seen in the mobile area. Google required to support 64-bit versions of applications in their app store by August 2019,<ref name="Sebayang_2019_Android"/> which allows to discontinue 32-bit support for [[Android (operating system)|Android]] later.<ref name="MW_2014"/> The shift towards 64-bit started in 2014 when all new processors were designed to a 64-bit architecture and [[Android Lollipop|Android 5]] ("Lollipop") was published in that year providing a fitting 64-bit variant of the operating system.<ref name="Android_User_2014"/><ref name="MW_2014"/> Apple had made shift in the year before starting to produce the 64-Bit [[Apple A7]] by 2013. Google started to deliver the development environment for Linux only in 64-bit by 2015.<ref name="APT_2015"/> In May 2019 the share of Android versions below 5 had fallen to ten percent.<ref name="Tenzer_2019"/> As [[Mobile app|app]] developers concentrate on a single [[executable|compilation]] variant, many manufacturers started to require Android 5 as the minimum version by mid 2019, for example Niantic.<ref name="Favero_2019"/> Subsequently the 32-bit versions were hard to get.<ref name="Reddit_2019"/>

Except for [[embedded systems]] with their special programs, the consideration of varying large-file support becomes obsolete in program code after 2020.

== Related problems ==
The [[year 2038 problem]] is well known for another case where a 32-bit "long" on 32-bit platforms will lead into problems. Just like the large-file limitation it will get obsolete when systems move to 64-bit only. In the meantime a 64-bit timestamp was introduced. In the Win32 API it is visible in functions having a "64" suffix along the earlier "32" suffix. When large-file support was added to the Win32 API it has led to functions having an additional "i64" suffix which sometimes makes for four combinations.(findfirst32, findfirst64, findfirst32i64, findfirst64i32).<ref name="Microsoft_2019_CRT"/> By comparison the UNIX98 API introduces functions with a "64" suffix when "_LARGEFILE64_SOURCE" is used.

Related to the large-file API there is a limitation of block numbers for [[mass storage]] media. With a common size of 512 bytes per [[Block (data storage)|data block]] the barrier resulting from 32-bit numbers did occur later. When [[hard disk drive]]s reached a size of 2 terabyte (around 2010) the [[master boot record]] had to be replaced by the [[GUID Partition Table]] which uses 64-bit for the LBA numbers ([[logical block addressing|logical block address]]). On [[Unix-like]] operating systems it did also require to enlarge the [[inode]] numbers which are used in some functions (stat64, setrlimit64). The [[Linux kernel]] introduced that in 2001 leading to version 2.4 which was picked up by the glibc in that year.<ref name="lfs_2015"/> As the large-file support and large-disk support was introduced at the same time the [[GNU C Library]] exports 64-bit inode structures on 32-bit architectures at the same time when the Unix LFS API is activated in program code.<ref name="Linux_stat.h"/>

When the kernel moved to 64-bit inodes the file system [[ext3]] used them internally in the driver by 2001. However the inode format on the storage media itself was stuck at 32-bit numbers.<ref name="lfs_2015"/> As mass storage devices moved to the [[Advanced Format]] of  4 kilobyte per block the actual limit of that file system format is at 8 or 16 terabyte.<ref name="lfs_2015"/> Handling larger disk partitions requires the usage of a different file system like [[XFS]] which was designed with 64-bit inodes from the start allowing for exabyte files and partitions.<ref name="Rutter_2020_inode"/><ref name="ext4_2019"/> The first 16 terabyte magnetic disk drives were delivered by mid 2019. [[Solid-state drive]] with 32 TiB for data centers were available as early as 2016 with some manufacturers forecasting 100 TiB SSD by 2020.<ref name="Scherer_2016"/>

==See also==
* [[2 GB limit]]
* [[RF64]] – 64-bit support for [[Broadcast Wave Format|BWF WAV]] audio files
* [[Comparison of text editors#Extra features|Comparison of large-file support in text editors]]
* [[FAT32+]]
* [[File size]]
* [[Long filename support]] (LFN)
* [[Year 2038 problem]]

==References==
{{Reflist|refs=
<ref name="Solaris_1996">{{cite web |author=Solaris OS group |date=March 1996 |title=Large Files in Solaris: A White Paper |publisher=[[Sun Microsystems]] |url=http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf |archive-url=https://web.archive.org/web/20070228204423/http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf |archive-date=2007-02-28}}</ref>
<ref name="Unix_1996_LFS">{{cite web |date=1996-08-14 |title=Adding Large File Support to the Single UNIX Specification |publisher=X/Open Base Working Group |url=http://opengroup.org/platform/lfs.html |access-date=2006-09-10}}</ref>
<ref name="Kuhnt-Georgiev-Davis_2007_FAT+">{{cite web |title=FAT+ draft revision 2 |author-first1=Udo |author-last1=Kuhnt |author-first2=Luchezar I. |author-last2=Georgiev |author-first3=Jeremy |author-last3=Davis |date=2007 |format=FATPLUS.TXT |edition=2 |url=http://www.fdos.org/kernel/fatplus.txt |access-date=2015-08-05 |url-status=dead |archive-url=https://web.archive.org/web/20150219123449/http://www.fdos.org/kernel/fatplus.txt |archive-date=2015-02-19}}</ref>
<ref name="Kuhnt_2011_EDR">{{cite web |title=DR-DOS/OpenDOS Enhancement Project |author-first=Udo |author-last=Kuhnt |date=2011-07-21 |url=http://www.drdosprojects.de/ |access-date=2015-04-20 |url-status=dead |archive-url=https://web.archive.org/web/20160604014846/http://www.drdosprojects.de/ |archive-date=2016-06-04}}</ref>
<ref name="Largefile_Distros">{{cite web |url=http://ac-archive.sourceforge.net/largefile/distros.html |title=Distro Makers |date=2002-01-13}}</ref>
<ref name="ZLib_Changelog">https://www.zlib.net/ChangeLog.txt {{Bare URL plain text|date=March 2022}}</ref>
<ref name="Kolokythas_2007">{{cite web |title=Windows Server 2008: Microsofts letztes 32-Bit-Betriebssystem für Server |language=de |author-first=Panagiotis |author-last=Kolokythas |publisher=[[PC Welt]] |date=2007-05-28 |url=https://www.pcwelt.de/news/Windows-Server-2008-Microsofts-letztes-32-Bit-Betriebssystem-fuer-Server-334946.html}}</ref>
<ref name="RHEL_2014_32bit">{{cite web |title=Are 32-bit applications supported in RHEL 7 or later releases? |publisher=[[Red Hat]] |date=February 2014 |url=https://access.redhat.com/solutions/509373}}</ref>
<ref name="Cooke_2019_32bit">{{cite web |title=Intel 32bit packages on Ubuntu from 19.10 onwards |author-first=Will |author-last=Cooke |date=2019-06-02 |publisher=Canonical |url=https://discourse.ubuntu.com/t/intel-32bit-packages-on-ubuntu-from-19-10-onwards/11263}}</ref>
<ref name="Addams_2018_Nvidia">{{cite web |title=Nvidia discontinues support for 32-bit Windows platforms |author-first=Matthew |author-last=Addams |publisher=Windows Report |date=2018-04-12 |url=https://windowsreport.com/nvidia-32-bit-windows-end-support/}}</ref>
<ref name="Silver_2018_Apple">{{cite web |title=Mojave is Apple's last version of macOS to support 32-bit apps |publisher=[[Apple Insider]] |author-first=Steven |author-last=Silver |date=2018-06-05 |url=https://appleinsider.com/articles/18/06/05/mojave-is-apples-last-version-of-macos-to-support-32-bit-apps}}</ref>
<ref name="Microsoft_Windows7">{{cite web |title=Der Support für Windows 7 endet am 14. Januar 2020 |language=de |publisher=[[Microsoft]] |url=https://support.microsoft.com/de-de/help/4057281/windows-7-support-ended-on-january-14-2020 |access-date=2020-02-09}}</ref>
<ref name="Sebayang_2019_Android">{{cite web |title=Auf dem Weg zu reinen 64-Bit-Android-Apps |language=de |publisher=Golem |author-first=Andreas |author-last=Sebayang |date=2019-01-17 |url=https://www.golem.de/news/google-auf-dem-weg-zum-reinen-64-bit-android-1901-138792.html}}</ref>
<ref name="MW_2014">{{cite web |title=Google kündigt Ende von 32-Bit-Android-Apps per 2021 an |language=de |publisher=IT Magazin |author=mw |date=2019-01-17 |url=https://www.itmagazine.ch/Artikel/68830/Google_kuendigt_Ende_von_32-Bit-Android-Apps_per_2021_an.html}}</ref>
<ref name="Android_User_2014">{{cite web |title=64-Bit-Android: Diese Prozessoren gibt es, diese Veränderungen kommen |language=de |date=2014-08-26 |publisher=Android User |url=https://www.android-user.de/64-bit-android-diese-prozessoren-gibt-es-diese-veraenderungen-kommen/}}</ref>
<ref name="APT_2015">{{cite web |title=Platform-tools 23.1.0 Linux changed to 64-bit without notice |publisher=Android Public Tracker |date=2015-12-11 |url=https://issuetracker.google.com/issues/37074522 |quote=It turns out the android-sdk-linux/platform-tools content is 32-bit ELF in 23.0.1 but 64-bit ELF in 23.1_rc1 and 23.1.0. […] I set ANDROID_EMULATOR_FORCE_32BIT=true […] 23.0.1 is the last 32-bit Linux build.}}</ref>
<ref name="Tenzer_2019">{{cite web |title=Anteile der verschiedenen Android-Versionen an allen Geräten mit Android OS weltweit im Zeitraum 01. bis 07. Mai 2019 |language=de |publisher=Statista |author-first=F. |author-last=Tenzer |date=2019-11-14 |url=https://de.statista.com/statistik/daten/studie/180113/umfrage/anteil-der-verschiedenen-android-versionen-auf-geraeten-mit-android-os/}}</ref>
<ref name="Favero_2019">{{cite web |title=Ingress und Pokémon Go brauchen bald mindestens Android 5 |author-first=Elia |author-last=Del Favero |date=2019-06-10 |url=https://www.nau.ch/news/games/ingress-und-pokemon-go-brauchen-bald-mindestens-android-5-65541520}}</ref>
<ref name="Reddit_2019">{{cite web |title=Why is 32bit 0.159.0 version apk still not available? |publisher=Reddit |work=TheSilphRoad/ |date=December 2019 |url=https://www.reddit.com/r/TheSilphRoad/comments/dm6c51/why_is_32bit_01590_version_apk_still_not_available/}}</ref>
<ref name="Microsoft_2019_CRT">{{cite web |url=https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/findfirst-functions?view=vs-2019 |title=C Run-time library (CRT) reference: findfirst |publisher=[[Microsoft]] |access-date=2020-02-17}}</ref>
<ref name="lfs_2015">{{cite web |url=https://users.suse.com/~aj/linux_lfs.html |title=Large File Support in Linux |publisher=[[SUSE S.A.|SuSE GmbH]] |date=2015-02-15 |author-first=Andreas |author-last=Jaeger}}</ref>
<ref name="Linux_stat.h">linux/bits/stat.h: /* Note stat64 has the same shape as stat for x86-64. */</ref>
<ref name="Rutter_2020_inode">{{cite web |url=https://www.mjr19.org.uk/sw/inodes64.html |title=The 64 bit inode problem |author-first=M. J. |author-last=Rutter |access-date=2020-02-10}}</ref>
<ref name="ext4_2019">{{cite web |url=https://ext4.wiki.kernel.org/index.php/Ext4_Howto |title=Ext4 Howto |website=kernel.org |date=2019-02-11 |quote=Although very large fileystems are on ext4's feature list, current e2fsprogs currently still limits the filesystem size to 2^32 blocks (16TiB for a 4KiB block filesystem). Allowing filesystems larger than 16T is one of the very next high-priority features to complete for ext4.}}</ref>
<ref name="Scherer_2016">{{cite web |url=https://www.elektormagazine.de/news/samsungs-32-tb-ssd-der-anfang-vom-ende-der-festplatte |title=Samsungs 32-TB-SSD: Der Anfang vom Ende der Festplatte |language=de |publisher=[[Elektor]] |date=2016-08-15 |author-first=Thomas |author-last=Scherer}}</ref>
}}

==External links==
* {{cite web |author-first=Andreas |author-last=Jaeger |date=2005-02-15 |title=Large File Support in Linux |publisher=[[SUSE S.A.|SuSE GmbH]] |url=http://www.suse.de/~aj/linux_lfs.html |access-date=2006-09-10}}

[[Category:Computer file systems]]