Editing Data corruption

{{Short description|Errors in computer data that introduce unintended changes to the original data}}
{{redirect|Corrupted|the Japanese metal band|Corrupted (band)}}

[[Image:Data loss of image file.JPG|thumb|Photo data corruption; in this case, a result of a failed data recovery from a hard disk drive]]
'''Data corruption''' refers to errors in [[computer data]] that occur during writing, reading, storage, transmission, or processing, which introduce unintended changes to the original data. Computer, transmission, and storage systems use a number of measures to provide end-to-end [[data integrity]], or lack of errors.

In general, when data corruption occurs, a [[Computer file|file]] containing that data will produce unexpected results when accessed by the system or the related application. Results could range from a minor loss of data to a system crash. For example, if a [[Document file format|document file]] is corrupted, when a person tries to open that file with a document editor they may get an [[error message]], thus the file might not be opened or might open with some of the data corrupted (or in some cases, completely corrupted, leaving the document unintelligible). The adjacent image is a corrupted image file in which most of the information has been lost.

Some types of [[malware]] may intentionally corrupt files as part of their [[Payload (computing)|payloads]], usually by overwriting them with inoperative or garbage code, while a non-malicious virus may also unintentionally corrupt files when it accesses them. If a virus or [[Trojan horse (computing)|trojan]] with this payload method manages to alter files critical to the running of the computer's operating system software or physical hardware, the entire system may be rendered unusable.

Some programs can give a suggestion to repair the file automatically (after the error), and some programs cannot repair it. It depends on the level of corruption, and the built-in functionality of the application to handle the error. There are various causes of the corruption.

== Overview ==
[[File:Atari 2600 with corrupted ram.jpg|thumb|Photo of an Atari 2600 with corrupted RAM.]]
[[File:Fourteen Second Clip of Corrupted Video.ogg|thumb|right|A video that has been corrupted. ''Epilepsy warning: This video contains bright, flashing images.'']]
There are two types of data corruption associated with computer systems: undetected and detected.  Undetected data corruption, also known as ''silent data corruption'', results in the most dangerous errors as there is no indication that the data is incorrect.  Detected
data corruption may be permanent with the loss of data, or may be temporary when some part of the system is able to detect and correct the error; there is no data corruption in the latter case.

Data corruption can occur at any level in a system, from the host to the storage medium.  Modern systems attempt to detect corruption at many layers and then recover or correct the corruption; this is almost always successful but very rarely the information arriving in the systems memory is corrupted and can cause unpredictable results.

<!--- Add Data corruption at the system level -->
Data corruption during transmission has a variety of causes. Interruption of data transmission causes [[Data loss|information loss]]. Environmental conditions can interfere with data transmission, especially when dealing with wireless transmission methods. Heavy clouds can block satellite transmissions. Wireless networks are susceptible to interference from devices such as microwave ovens.
<!--- Add Data corruption at the storage level -->

Hardware and software failure are the two main causes for [[data loss]].  [[Background radiation]], [[head crash]]es, and [[Mean time between failures|aging]] or wear of the storage device fall into the former category, while software failure typically occurs due to [[Software bug|bugs]] in the code.
[[Cosmic ray]]s cause most [[soft error]]s in DRAM.<ref>{{cite web|author = Scientific American|date = 2008-07-21|title = Solar Storms: Fast Facts|url = http://www.scientificamerican.com/article.cfm?id=solar-storms-fast-facts|publisher = [[Nature Publishing Group]]|access-date = 2009-12-08|url-status = live|archive-url = https://web.archive.org/web/20101226165751/http://www.scientificamerican.com/article.cfm?id=solar-storms-fast-facts|archive-date = 2010-12-26|author-link = Scientific American}}</ref>

== {{Anchor|SILENT}}Silent ==
{{See also|Hard disk drive error rates and handling}}

Some errors go unnoticed, without being detected by the disk firmware or the host operating system; these errors are known as ''silent data corruption''.<ref>{{cite web
 |url=https://support.google.com/cloud/answer/10759085?hl=en#:~:text=Silent%20Data%20Corruption%20(SDC)%2C,to%20data%20loss%20and%20corruption.
 |title=Silent Data Corruption |date=2023 |publisher=Google Inc. |access-date=January 30, 2023
 |quote=Silent Data Corruption (SDC), sometimes referred to as Silent Data Error (SDE), is an industry-wide issue impacting not only long-protected memory, storage, and networking, but also computer CPUs.}}</ref>

There are many error sources beyond the disk storage subsystem itself. For instance, cables might be slightly loose, the power supply might be unreliable,<ref>{{cite web|title=ZFS saves the day(-ta)!|url=http://blogs.oracle.com/elowe/entry/zfs_saves_the_day_ta|work=Oracle – Core Dumps of a Kernel Hacker's Brain – Eric Lowe's Blog|publisher=Oracle|access-date=9 June 2012|author=Eric Lowe|format=Blog|date=16 November 2005|url-status=dead|archive-url=https://web.archive.org/web/20120205040345/http://blogs.oracle.com/elowe/entry/zfs_saves_the_day_ta|archive-date=5 February 2012}}</ref> external vibrations such as a loud sound,<ref>{{cite web|title=Shouting in the Datacenter|url=https://www.youtube.com/watch?v=tDacjrSCeq4|work=YouTube|access-date=9 June 2012|author=bcantrill|format=Video file|date=31 December 2008|url-status=live|archive-url=https://web.archive.org/web/20120703132341/http://www.youtube.com/watch?v=tDacjrSCeq4|archive-date=3 July 2012}}</ref> the network might introduce undetected corruption,<ref>{{cite web|title=Faulty FC port meets ZFS|url=http://jforonda.blogspot.com/2007/01/faulty-fc-port-meets-zfs.html|work=Blogger – Outside the Box|access-date=9 June 2012|author=jforonda|format=Blog|date=31 January 2007|url-status=live|archive-url=https://web.archive.org/web/20120426055112/http://jforonda.blogspot.com/2007/01/faulty-fc-port-meets-zfs.html|archive-date=26 April 2012}}</ref> [[Cosmic ray#Effect on electronics|cosmic radiation]] and many other causes of [[soft error|soft memory errors]], etc. In 39,000 storage systems that were analyzed, firmware bugs accounted for 5–10% of storage failures.<ref>{{cite web |url=http://www.usenix.org/event/fast08/tech/full_papers/jiang/jiang.pdf |title=Are Disks the Dominant Contributor for Storage Failures? A Comprehensive Study of Storage Subsystem Failure Characteristics |publisher=USENIX |access-date=2014-01-18 |archive-date=2022-01-25 |archive-url=https://web.archive.org/web/20220125061938/https://www.usenix.org/legacy/event/fast08/tech/full_papers/jiang/jiang.pdf |url-status=live }}</ref> All in all, the error rates as observed by a [[CERN]] study on silent corruption are far higher than one in every 10<sup>16</sup> bits.<ref name="CERN2007">{{cite web|title=Draft 1.3|url=http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=0&resId=1&materialId=paper&confId=13797|work=Data integrity|publisher=CERN|access-date=9 June 2012|author=Bernd Panzer-Steindel|date=8 April 2007|url-status=live|archive-url=https://web.archive.org/web/20121027083405/http://indico.cern.ch/getFile.py/access?contribId=3&sessionId=0&resId=1&materialId=paper&confId=13797|archive-date=27 October 2012}}</ref> Webshop [[Amazon.com]] has acknowledged similar high data corruption rates in their systems.<ref>{{cite web| url = http://perspectives.mvdirona.com/2012/02/26/ObservationsOnErrorsCorrectionsTrustOfDependentSystems.aspx| title = Observations on Errors, Corrections, & Trust of Dependent Systems| url-status = live| archive-url = https://web.archive.org/web/20131029192337/http://perspectives.mvdirona.com/2012/02/26/ObservationsOnErrorsCorrectionsTrustOfDependentSystems.aspx| archive-date = 2013-10-29}}</ref> In 2021, faulty processor cores were identified as an additional cause in publications by Google and Facebook; cores were found to be faulty at a rate of several in thousands of cores.<ref>{{Cite book|last1=Hochschild|first1=Peter H.|last2=Turner|first2=Paul Jack|last3=Mogul|first3=Jeffrey C.|last4=Govindaraju|first4=Rama Krishna|last5=Ranganathan|first5=Parthasarathy|last6=Culler|first6=David E.|last7=Vahdat|first7=Amin|title=Proceedings of the Workshop on Hot Topics in Operating Systems |chapter=Cores that don't count |date=2021|chapter-url=https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf|pages=9–16|doi=10.1145/3458336.3465297|isbn=9781450384384|s2cid=235311320|access-date=2021-06-02|archive-date=2021-06-03|archive-url=https://web.archive.org/web/20210603055415/https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf|url-status=live}}</ref><ref>{{Citation|title=HotOS 2021: Cores That Don't Count (Fun Hardware)| date=27 May 2021 |url=https://www.youtube.com/watch?v=QMF3rqhjYuM |archive-url=https://ghostarchive.org/varchive/youtube/20211222/QMF3rqhjYuM |archive-date=2021-12-22 |url-status=live|language=en|access-date=2021-06-02}}{{cbignore}}</ref>

One problem is that hard disk drive capacities have increased substantially, but their error rates remain unchanged. The data corruption rate has always been roughly constant in time, meaning that modern disks are not much safer than old disks. In old disks the probability of data corruption was very small because they stored tiny amounts of data. In modern disks the probability is much larger because they store much more data, whilst not being safer. That way, silent data corruption has not been a serious concern while storage devices remained relatively small and slow. In modern times and with the advent of larger drives and very fast RAID setups, users are capable of transferring 10<sup>16</sup> bits in a reasonably short time, thus easily reaching the data corruption thresholds.<ref>{{cite web
 |url         = http://www.necam.com/docs/?id=54157ff5-5de8-4966-a99d-341cf2cb27d3
 |title       = Silent data corruption in disk arrays: A solution
 |year        = 2009
 |access-date  = 14 December 2020
 |format      = PDF
 |publisher   = NEC
 |archive-url  = https://web.archive.org/web/20131029210013/http://www.necam.com/docs/?id=54157ff5-5de8-4966-a99d-341cf2cb27d3
 |archive-date = 29 October 2013
}}</ref>

As an example, [[ZFS]] creator Jeff Bonwick stated that the fast database at [[Greenplum]], which is a database software company specializing in large-scale data warehousing and analytics, faces silent corruption every 15 minutes.<ref>{{cite web
 |url         = http://queue.acm.org/detail.cfm?id=1317400
 |title       = A Conversation with Jeff Bonwick and Bill Moore
 |date        = November 15, 2007
 |publisher   = Association for Computing Machinery
 |access-date  = 14 December 2020
 |url-status  = live
 |archive-url  = https://web.archive.org/web/20110716221142/http://queue.acm.org/detail.cfm?id=1317400
 |archive-date = 16 July 2011
}}</ref>  As another example, a real-life study performed by [[NetApp]] on more than 1.5 million HDDs over 41 months found more than 400,000 silent data corruptions, out of which more than 30,000 were not detected by the hardware RAID controller (only detected during [[data scrubbing|scrubbing]]).<ref>{{Cite news |title= Keeping Bits Safe: How Hard Can It Be? |work= ACM Queue |date= October 1, 2010 |author= David S. H. Rosenthal |url= http://queue.acm.org/detail.cfm?id=1866298 |access-date= 2014-01-02 |url-status= live |archive-url= https://web.archive.org/web/20131217020947/http://queue.acm.org/detail.cfm?id=1866298 |archive-date= December 17, 2013 |author-link= David S. H. Rosenthal }};  Bairavasundaram, L., Goodson, G., Schroeder, B., Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H. 2008. An analysis of data corruption in the storage stack. In Proceedings of 6th Usenix Conference on File and Storage Technologies.</ref>  Another study, performed by [[CERN]] over six months and involving about 97&nbsp;[[petabytes]] of data, found that about 128&nbsp;[[megabytes]] of data became permanently corrupted silently somewhere in the pathway from network to disk.<ref>{{cite conference |conference=8th Annual Workshop on Linux Clusters for Super Computing |last1=Kelemen |first1=P |title=Silent corruptions |url=https://indico.desy.de/event/257/contributions/58082/attachments/37574/46878/kelemen-2007-HEPiX-Silent_Corruptions.pdf}}</ref>

Silent data corruption may result in [[cascading failure]]s, in which the system may run for a period of time with undetected initial error causing increasingly more problems until it is ultimately detected.<ref>{{cite web | url = http://www.fiala.me/pubs/papers/sc12-redmpi.pdf | title = Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing | date = November 2012 | access-date = 2015-01-26 | author1 = David Fiala | author2 = Frank Mueller | author3 = Christian Engelmann | author4 = Rolf Riesen | author5 = Kurt Ferreira | author6 = Ron Brightwell | website = fiala.me | publisher = [[IEEE]] | url-status = live | archive-url = https://web.archive.org/web/20141107074511/http://www.fiala.me/pubs/papers/sc12-redmpi.pdf | archive-date = 2014-11-07 }}</ref>  For example, a failure affecting file system [[metadata]] can result in multiple files being partially damaged or made completely inaccessible as the file system is used in its corrupted state.

== {{Anchor|END-TO-END-PROTECTION|INTEL-REPLAY}}Countermeasures ==
{{See also|Error detection and correction}}

When data corruption behaves as a [[Poisson process]], where each [[bit]] of data has an independently low probability of being changed, data corruption can generally be detected by the use of [[checksum]]s, and can often be [[Error detection and correction|corrected]] by the use of [[error correcting code]]s (ECC).

If an uncorrectable data corruption is detected, procedures such as automatic retransmission or restoration from [[backup]]s can be applied. Certain levels of [[Redundant array of independent disks|RAID]] disk arrays have the ability to store and evaluate [[parity bit]]s for data across a set of hard disks and can reconstruct corrupted data upon the failure of a single or multiple disks, depending on the level of RAID implemented.  Some [[CPU]] architectures employ various transparent checks to detect and mitigate data corruption in [[CPU cache]]s, [[Data buffer|CPU buffers]] and [[instruction pipeline]]s; an example is ''Intel Instruction Replay'' technology, which is available on [[Intel Itanium]] processors.<ref>{{cite web
 |url         = http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/itanium-9500-reliability-mission-critical-applications-paper.pdf
 |title       = Rachet Up Reliability for Mission-Critical Applications: Intel Instruction Replay Technology
 |year        = 2012
 |access-date  = 2016-01-27
 |author      = Steve Bostian
 |publisher   = [[Intel]]
 |url-status     = live
 |archive-url  = https://web.archive.org/web/20160202125833/http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/itanium-9500-reliability-mission-critical-applications-paper.pdf
 |archive-date = 2016-02-02
}}</ref>

Many errors are detected and corrected by the hard disk drives using the ECC codes<ref>{{cite web|title=Read Error Severities and Error Management Logic|url=http://pcguide.com/ref/hdd/geom/errorRead-c.html|access-date=4 April 2012|url-status=live|archive-url=https://web.archive.org/web/20120407181624/http://www.pcguide.com/ref/hdd/geom/errorRead-c.html|archive-date=7 April 2012}}</ref> which are stored on disk for each sector.  If the disk drive detects multiple read errors on a sector it may make a copy of the failing sector on another part of the disk, by remapping the failed sector of the disk to a spare sector without the involvement of the operating system (though this may be delayed until the next write to the sector).  This "silent correction" can be monitored using [[Self-Monitoring, Analysis and Reporting Technology|S.M.A.R.T.]] and tools available for most operating systems to automatically check the disk drive for impending failures by watching for deteriorating SMART parameters.

Some [[file systems]], such as [[Btrfs]], [[HAMMER (file system)|HAMMER]], [[ReFS]], and [[ZFS]], use internal data and [[metadata]] checksumming to detect silent data corruption.  In addition, if a corruption is detected and the file system uses integrated RAID mechanisms that provide [[data redundancy]], such file systems can also reconstruct corrupted data in a transparent way.<ref>{{cite web
 |url         = http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html
 |title       = How I Use the Advanced Capabilities of Btrfs
 |date        = August 2012
 |access-date  = 2014-01-02
 |author1     = Margaret Bierman
 |author2     = Lenz Grimmer
 |publisher   = [[Oracle Corporation]]
 |url-status     = live
 |archive-url  = https://web.archive.org/web/20140102193726/http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html
 |archive-date = 2014-01-02
}}</ref>  This approach allows improved data integrity protection covering the entire data paths, which is usually known as ''end-to-end data protection'', compared with other data integrity approaches that do not span different layers in the storage stack and allow data corruption to occur while the data passes boundaries between the different layers.<ref name="Zhang2010">{{cite q | Q111972797 
| access-date = 2014-08-12
}}</ref>

''[[Data scrubbing]]'' is another method to reduce the likelihood of data corruption, as disk errors are caught and recovered from before multiple errors accumulate and overwhelm the number of parity bits.  Instead of parity being checked on each read, the parity is checked during a regular scan of the disk, often done as a low priority background process. The "data scrubbing" operation activates a parity check.  If a user simply runs a normal program that reads data from the disk, then the parity would not be checked unless parity-check-on-read was both supported and enabled on the disk subsystem.

If appropriate mechanisms are employed to detect and remedy data corruption, data integrity can be maintained. This is particularly important in commercial applications (e.g. [[banking]]), where an undetected error could either corrupt a database index or change data to drastically affect an account balance, and in the use of [[encryption|encrypted]] or [[data compression|compressed]] data, where a small error can make an extensive dataset unusable.<ref name="CERN2007" />

== See also ==

{{colbegin}}
* Various resources:
** [[Data degradation]], also called data rot
** [[Computer science]]
** [[Data integrity]]
** [[Database integrity]]
** [[Radiation hardening]]
** [[Software rot]]
* Countermeasures:
** [[Data Integrity Field]]
** [[ECC memory]]
** [[Forward error correction]]
** [[List of data recovery software]]
** [[Parchive]]
** [[RAID]]
** [[Reed–Solomon error correction]]
{{colend}}

== References ==
{{Reflist|30em}}

== External links ==
* [http://pdos.csail.mit.edu/papers/softecc:ddopson-meng/softecc_ddopson-meng.pdf SoftECC: A System for Software Memory Integrity Checking]
* [http://www.fiala.me/pubs/papers/libsdc11.pdf A Tunable, Software-based DRAM Error Detection and Correction Library for HPC]
* [http://www.fiala.me/pubs/papers/sc12-redmpi.pdf Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing]
* [https://www.usenix.org/legacy/events/fast10/tech/full_papers/zhang.pdf End-to-end Data Integrity for File Systems: A ZFS Case Study]
* [http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf DRAM Errors in the Wild: A Large-Scale Field Study]
* [https://www.nsc.liu.se/lcsc2007/presentations/LCSC_2007-kelemen.pdf A study on silent corruptions], and an associated paper on [https://web.archive.org/web/20141220141819/http://indico.cern.ch/event/13797/session/0/material/paper/1?contribId=3 data integrity] (CERN, 2007)
* [http://www.hgst.com/tech/techlib.nsf/techdocs/328DB53FC65C4DF18625742C00562B92/$file/End-to-end_Data_Protection.pdf End-to-end Data Protection in SAS and Fibre Channel Hard Disk Drives] (HGST)

{{data}}

[[Category:Data quality]]
[[Category:Product expiration]]