Editing Google File System

{{Short description|Proprietary file system}}
{{more citations needed|date=July 2016}}

{{Infobox software
| name                   = 
| title                  = Google File System
| logo                   = <!-- Image name is enough -->
| logo caption           = 
| logo_size              = 
| logo_alt               = 
| screenshot             = <!-- Image name is enough -->
| caption                = 
| screenshot_size        = 
| screenshot_alt         = 
| collapsible            = 
| author                 = 
| developer              = [[Google]]
| released               = <!-- {{Start date and age|YYYY|MM|DD|df=yes/no}} -->
| discontinued           = 
| latest release version = 
| latest release date    = <!-- {{Start date and age|YYYY|MM|DD|df=yes/no}} -->
| latest preview version = 
| latest preview date    = <!-- {{Start date and age|YYYY|MM|DD|df=yes/no}} -->
| programming language   = 
| operating system       = [[Linux kernel]]
| platform               = 
| size                   = 
| genre                  = [[Distributed file system]]
| license                = [[Proprietary software|Proprietary]]
| website                = <!-- {{URL|example.org}} -->
}}

'''Google File System''' ('''GFS''' or '''GoogleFS''', not to be confused with the [[GFS2|GFS]] Linux file system) is a [[Proprietary software|proprietary]] [[distributed file system]] developed by [[Google]] to provide efficient, reliable access to data using large clusters of [[commodity hardware]]. Google file system was replaced by Colossus in 2010.<ref name="ma">{{cite web | url = https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/ | title = Colossus: Successor to the Google File System (GFS) | publisher = SysTutorials | date = 2012-11-29 | first = Eric | last = Ma | access-date = 2016-05-10 | archive-url = https://web.archive.org/web/20190412112114/https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/ | archive-date = 2019-04-12 | url-status = live}}</ref>

==Design==
[[File:GoogleFileSystemGFS.svg|thumb|300px|Google File System is designed for system-to-system interaction, and not for user-to-system interaction. The chunk servers replicate the data automatically.]]

GFS is enhanced for Google's core data storage and usage needs (primarily the [[Google (search engine)|search engine]]), which can generate enormous amounts of data that must be retained; Google File System grew out of an earlier Google effort, "BigFiles", developed by [[Larry Page]] and [[Sergey Brin]] in the early days of Google, while it was still located in [[Stanford]]. Files are divided into fixed-size ''chunks'' of 64 [[megabyte]]s, similar to clusters or sectors in regular file systems, which are only extremely rarely overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google's computing clusters, dense nodes which consist of cheap "commodity" computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data [[throughput]]s, even when it comes at the cost of [[Latency (engineering)#Computer hardware and operating systems|latency]].

A GFS cluster consists of multiple nodes. These nodes are divided into two types: one ''Master'' node and multiple ''Chunkservers''. Each file is divided into fixed-size chunks. Chunkservers store these chunks. Each chunk is assigned a globally unique 64-bit label by the master node at the time of creation, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated several times throughout the network. At default, it is replicated three times, but this is configurable.{{Sfn | Ghemawat | Gobioff | Leung | 2003}} Files which are in high demand may have a higher replication factor, while files for which the application client uses strict storage optimizations may be replicated less than three times - in order to cope with quick garbage cleaning policies.{{Sfn | Ghemawat | Gobioff | Leung | 2003}}

The Master server does not usually store the actual chunks, but rather all the [[metadata]] associated with the chunks, such as the tables mapping the 64-bit labels to chunk locations and the files they make up (mapping from files to chunks), the locations of the copies of the chunks, what processes are reading or writing to a particular chunk, or taking a "snapshot" of the chunk pursuant to replicate it (usually at the instigation of the Master server, when, due to node failures, the number of copies of a chunk has fallen beneath the set number). All this metadata is kept current by the Master server periodically receiving updates from each chunk server ("Heart-beat messages").

Permissions for modifications are handled by a system of time-limited, expiring "leases", where the Master server grants permission to a process for a finite period of time during which no other process will be granted permission by the Master server to modify the chunk. The modifying chunkserver, which is always the primary chunk holder, then propagates the changes to the chunkservers with the backup copies. The changes are not saved until all chunkservers acknowledge, thus guaranteeing the completion and [[Atomicity (database systems)|atomicity]] of the operation.

Programs access the chunks by first querying the Master server for the locations of the desired chunks; if the chunks are not being operated on (i.e. no outstanding leases exist), the Master replies with the locations, and the program then contacts and receives the data from the chunkserver directly (similar to [[Kazaa]] and its [[supernode (networking)|supernode]]s).

Unlike most other file systems, GFS is not implemented in the [[kernel (operating system)|kernel]] of an [[operating system]], but is instead provided as a [[userspace]] library.<ref>{{Cite book| last=Kyriazis|first=Dimosthenis| title=Data Intensive Storage Services for Cloud Environments|publisher=IGI Global| year=2013|isbn=9781466639355|pages=13| url=https://books.google.com/books?id=mNCeBQAAQBAJ&pg=PA13}}</ref>

== Interface ==
The Google File System does not provide a [[POSIX]] interface.<ref name="convo">{{cite journal |author1=Marshall Kirk McKusick |author2=Sean Quinlan |title=GFS: Evolution on Fast-forward |journal=ACM Queue |date=August 2009 |volume=7 |issue=7 |pages=10–20 |doi=10.1145/1594204.1594206 |url=https://queue.acm.org/detail.cfm?id=1594206 |access-date=21 December 2019|doi-access=free |url-access=subscription }}</ref> Files are organized hierarchically in directories and identified by pathnames. The file operations such as create, delete, open, close, read, write are supported. It supports Record Append which allows multiple clients to append data to the same file concurrently and atomicity is guaranteed.

== Performance ==
Deciding from benchmarking results,{{Sfn | Ghemawat | Gobioff | Leung | 2003}} when used with relatively small number of servers (15), the file system achieves reading performance comparable to that of a single disk (80–100 MB/s), but has a reduced write performance (30 MB/s), and is relatively slow (5 MB/s) in appending data to existing files. The authors present no results on random seek time. As the master node is not directly involved in data reading (the data are passed from the chunk server directly to the reading client), the read rate increases significantly with the number of chunk servers, achieving 583 MB/s for 342 nodes. Aggregating multiple servers also allows big capacity, while it is somewhat reduced by storing data in three independent locations (to provide redundancy).

==See also==

* [[Bigtable]]
* [[Cloud storage]]
* [[CloudStore]]
* [[Fossil (file system)|Fossil]], the native file system of [[Plan 9 from Bell Labs|Plan 9]]
* [[GPFS]] IBM's General Parallel File System
* [[GFS2]] Red Hat's Global File System 2
* [[Apache Hadoop]] and its "Hadoop Distributed File System" (HDFS), an open source Java product similar to GFS
* [[List of Google products]]
* [[MapReduce]]
* [[Moose File System]]
* [[LizardFS]]

==References==
{{reflist}}

==Bibliography==
{{refbegin}}
* {{Cite book | doi = 10.1145/945445.945450| chapter = The Google file system| title = Proceedings of the nineteenth ACM Symposium on Operating Systems Principles - SOSP '03| pages = 29| year = 2003| last1 = Ghemawat | first1 = S. | last2 = Gobioff | first2 = H. | last3 = Leung | first3 = S. T. | citeseerx = 10.1.1.125.789| isbn = 1581137575| s2cid = 221261373| url = http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf}}
{{refend}}

==External links==
* {{Citation | url = http://queue.acm.org/detail.cfm?id=1594206 | contribution = GFS: Evolution on Fast-forward | title = Queue | publisher = ACM}}. <!-- http://cacm.acm.org/magazines/2010/3/76283-gfs-evolution-on-fast-forward/fulltext -->
* {{Citation | url = https://storagemojo.com/google-file-system-eval-part-i/ | contribution = Google File System Eval, Part I | title = Storage mojo}}.

{{File systems}}
{{Google LLC}}

[[Category:Distributed file systems supported by the Linux kernel]]
[[Category:Google|File System]]
[[Category:Parallel computing]]
[[Category:Distributed file systems]]