Editing Data integrity

{{Short description|Maintenance of data over its entire life-cycle}}
'''Data integrity''' is the maintenance of, and the assurance of, data accuracy and consistency over its entire [[Information Lifecycle Management|life-cycle]].<ref>{{cite web|last=Boritz |first=J. |title=IS Practitioners' Views on Core Concepts of Information Integrity |url=http://www.fdewb.unimaas.nl/marc/ecais_new/files/boritz.doc |work=International Journal of Accounting Information Systems |publisher=Elsevier |access-date=12 August 2011 |url-status=dead |archive-url=https://web.archive.org/web/20111005085820/http://www.fdewb.unimaas.nl/marc/ecais_new/files/boritz.doc |archive-date=5 October 2011 }}</ref> It is a critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data. The term is broad in scope and may have widely different meanings depending on the specific context even under the same general umbrella of [[computing]]. It is at times used as a proxy term for [[data quality]],<ref>[https://www.veracode.com/blog/2012/05/what-is-data-integrity What is Data Integrity? Learn How to Ensure Database Data Integrity via Checks, Tests, & Best Practices]</ref> while [[data validation]] is a prerequisite for data integrity.<ref>[https://digitalguardian.com/blog/what-data-integrity-data-protection-101 What is Data Integrity? Data Protection 101]</ref>

== Definition ==
Data integrity is the opposite of [[data corruption]].<ref>[https://books.google.com/books?id=oI2GAgAAQBAJ&dq=data+integrity+opposite+data+corruption&pg=PA40 From the book: Uberveillance and the Social Implications of Microchip Implants: Emerging Page 40]</ref> The overall intent of any data integrity technique is the same: ensure data is recorded exactly as intended (such as a database correctly rejecting mutually exclusive possibilities). Moreover, upon later [[Data retrieval|retrieval]], ensure the data is the same as when it was originally recorded. In short, data integrity aims to prevent unintentional changes to information. Data integrity is not to be confused with [[data security]], the discipline of protecting data from unauthorized parties.

Any unintended changes to data as the result of a storage, retrieval or processing operation, including malicious intent, unexpected hardware failure, and [[human error]], is failure of data integrity. If the changes are the result of unauthorized access, it may also be a failure of data security. Depending on the data involved this could manifest itself as benign as a single pixel in an image appearing a different color than was originally recorded, to the loss of vacation pictures or a business-critical database, to even catastrophic loss of human life in a [[life-critical system]].

== Integrity types ==

=== Physical integrity ===
Physical integrity deals with challenges which are associated with correctly storing and fetching the data itself. Challenges with physical integrity may include [[electromechanics|electromechanical]] faults, design flaws, material [[fatigue (material)|fatigue]], [[corrosion]], [[Power outage|power outages]], natural disasters, and other special environmental hazards such as [[ionizing radiation]], extreme temperatures, pressures and [[g-force]]s. Ensuring physical integrity includes methods such as [[Redundancy (engineering)|redundant]] hardware, an [[uninterruptible power supply]], certain types of [[RAID]] arrays, [[radiation hardened]] chips, [[ECC memory|error-correcting memory]], use of a [[clustered file system]], using file systems that employ block level [[checksum]]s such as [[ZFS]], storage arrays that compute parity calculations such as [[exclusive or]] or use a [[cryptographic hash function]] and even having a [[watchdog timer]] on critical subsystems.

Physical integrity often makes extensive use of error detecting algorithms known as [[error-correcting codes]]. Human-induced data integrity errors are often detected through the use of simpler checks and algorithms, such as the [[Damm algorithm]] or [[Luhn algorithm]]. These are used to maintain data integrity after manual transcription from one computer system to another by a human intermediary (e.g. credit card or bank routing numbers). Computer-induced transcription errors can be detected through [[hash functions]].

In production systems, these techniques are used together to ensure various degrees of data integrity. For example, a computer [[file system]] may be configured on a fault-tolerant RAID array, but might not provide block-level checksums to detect and prevent [[silent data corruption]]. As another example, a database management system might be compliant with the [[ACID]] properties, but the RAID controller or hard disk drive's internal write cache might not be.

=== Logical integrity ===
{{See also|Mutex|Copy-on-write}}

This type of integrity is concerned with the [[correctness (computer science)|correctness]] or [[rationality]] of a piece of data, given a particular context. This includes topics such as [[referential integrity]] and [[entity integrity]] in a [[relational database]] or correctly ignoring impossible sensor data in robotic systems. These concerns involve ensuring that the data "makes sense" given its environment. Challenges include [[software bugs]], design flaws, and human errors. Common methods of ensuring logical integrity include things such as [[check constraint]]s, [[foreign key constraint]]s, program [[assertion (computing)|assertion]]s, and other run-time sanity checks.

Physical and logical integrity often share many challenges such as human errors and design flaws, and both must appropriately deal with concurrent requests to record and retrieve data, the latter of which is entirely a subject on its own.

If a data sector only has a logical error, it can be reused by overwriting it with new data. In case of a physical error, the affected data sector is permanently unusable.

== Databases ==
Data integrity contains guidelines for [[data retention]], specifying or guaranteeing the length of time data can be retained in a particular database (typically a [[relational database]]). To achieve data integrity, these rules are consistently and routinely applied to all data entering the system, and any relaxation of enforcement could cause errors in the data. Implementing checks on the data as close as possible to the source of input (such as human data entry), causes less erroneous data to enter the system. Strict enforcement of data integrity rules results in lower error rates, and time saved troubleshooting and tracing erroneous data and the errors it causes to algorithms.

Data integrity also includes rules defining the relations a piece of data can have to other pieces of data, such as a ''Customer'' record being allowed to link to purchased ''Products'', but not to unrelated data such as ''Corporate Assets''. Data integrity often includes checks and correction for invalid data, based on a fixed [[database schema|schema]] or a predefined set of rules. An example being textual data entered where a date-time value is required. Rules for data derivation are also applicable, specifying how a data value is derived based on algorithm, contributors and conditions. It also specifies the conditions on how the data value could be re-derived.

=== Types of integrity constraints ===
Data integrity is normally enforced in a [[database system]] by a series of integrity constraints or rules. Three types of integrity constraints are an inherent part of the [[Relational model|relational data model]]: entity integrity, referential integrity and domain integrity.

* ''[[Entity integrity]]'' concerns the concept of a [[primary key]]. Entity integrity is an integrity rule which states that every table must have a primary key and that the column or columns chosen to be the primary key should be unique and not null.
* ''[[Referential integrity]]'' concerns the concept of a [[foreign key]]. The referential integrity rule states that any foreign-key value can only be in one of two states. The usual state of affairs is that the foreign-key value refers to a primary key value of some table in the database. Occasionally, and this will depend on the rules of the data owner, a foreign-key value can be [[Null (SQL)|null]]. In this case, we are explicitly saying that either there is no relationship between the objects represented in the database or that this relationship is unknown.
* ''Domain integrity'' specifies that all columns in a relational database must be declared upon a defined domain. The primary unit of data in the relational data model is the data item. Such data items are said to be non-decomposable or atomic. A domain is a set of values of the same type. Domains are therefore pools of values from which actual values appearing in the columns of a table are drawn.
* ''User-defined integrity'' refers to a set of rules specified by a user, which do not belong to the entity, domain and referential integrity categories.

If a database supports these features, it is the responsibility of the database to ensure data integrity as well as the [[consistency model]] for the data storage and retrieval. If a database does not support these features, it is the responsibility of the applications to ensure data integrity while the database supports the [[consistency model]] for the data storage and retrieval.

Having a single, well-controlled, and well-defined data-integrity system increases:

* stability (one centralized system performs all data integrity operations)
* performance (all data integrity operations are performed in the same tier as the consistency model)
* re-usability (all applications benefit from a single centralized data integrity system)
* maintainability (one centralized system for all data integrity administration).

Modern [[database]]s support these features (see [[Comparison of relational database management systems]]), and it has become the de facto responsibility of the database to ensure data integrity. Companies, and indeed many database systems, offer products and services to migrate legacy systems to modern databases.

=== Examples ===
An example of a data-integrity mechanism is the parent-and-child relationship of related records. If a parent record owns one or more related child records all of the referential integrity processes are handled by the database itself, which automatically ensures the accuracy and integrity of the data so that no child record can exist without a parent (also called being orphaned) and that no parent loses their child records. It also ensures that no parent record can be deleted while the parent record owns any child records. All of this is handled at the database level and does not require coding integrity checks into each application.

== File systems ==
Various research results show that neither widespread [[filesystem]]s (including [[Unix File System|UFS]], [[Extended file system|Ext]], [[XFS]], [[JFS (file system)|JFS]] and [[NTFS]]) nor [[RAID#Weaknesses|hardware RAID]] solutions provide sufficient protection against data integrity problems.<ref>
{{cite web |title=IRON FILE SYSTEMS |url=http://pages.cs.wisc.edu/~vijayan/vijayan-thesis.pdf |archive-url=https://ghostarchive.org/archive/20221009/http://pages.cs.wisc.edu/~vijayan/vijayan-thesis.pdf |archive-date=2022-10-09 |url-status=live|work=Doctor of Philosophy in Computer Sciences |publisher=University of Wisconsin-Madison |access-date=9 June 2012 |author=Vijayan Prabhakaran |year=2006 }}</ref><ref>{{cite web |title=Parity Lost and Parity Regained |url=http://www.cs.wisc.edu/adsl/Publications/parity-fast08.html}}</ref><ref>
{{cite web |title=An Analysis of Data Corruption in the Storage Stack |url=http://www.cs.wisc.edu/adsl/Publications/corruption-fast08.pdf |archive-url=https://ghostarchive.org/archive/20221009/http://www.cs.wisc.edu/adsl/Publications/corruption-fast08.pdf |archive-date=2022-10-09 |url-status=live}}</ref><ref>
{{cite web |title=Impact of Disk Corruption on Open-Source DBMS |url=http://www.cs.wisc.edu/adsl/Publications/corrupt-mysql-icde10.pdf |archive-url=https://ghostarchive.org/archive/20221009/http://www.cs.wisc.edu/adsl/Publications/corrupt-mysql-icde10.pdf |archive-date=2022-10-09 |url-status=live}}</ref><ref>
{{cite web |url=http://www.baarf.com/  |title=Baarf.com |publisher=Baarf.com |access-date=4 November 2011}}</ref>

Some filesystems (including [[Btrfs]] and [[ZFS]]) provide internal data and [[metadata]] checksumming that is used for detecting [[silent data corruption]] and improving data integrity.  If a corruption is detected that way and internal RAID mechanisms provided by those filesystems are also used, such filesystems can additionally reconstruct corrupted data in a transparent way.<ref>{{cite web | url = http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html | title = How I Use the Advanced Capabilities of Btrfs | date = August 2012 | access-date = 2014-01-02 | first1 = Margaret | last1 = Bierman | first2 = Lenz | last2 = Grimmer}}</ref>  This approach allows improved data integrity protection covering the entire data paths, which is usually known as [[end-to-end data protection]].<ref name= "Zhang2010">{{cite q | Q111972797 | access-date = 2014-01-02}}</ref>

== Data integrity as applied to various industries ==

* The U.S. [[Food and Drug Administration]] has created draft guidance on data integrity for the pharmaceutical manufacturers required to adhere to U.S. Code of Federal Regulations 21 CFR Parts 210–212.<ref name="HHSData16">{{cite web |url=https://www.fda.gov/downloads/drugs/guidances/ucm495891.pdf |archive-url=https://ghostarchive.org/archive/20221009/https://www.fda.gov/downloads/drugs/guidances/ucm495891.pdf |archive-date=2022-10-09 |url-status=dead |title=Data Integrity and Compliance with CGMP: Guidance for Industry |publisher=U.S. Food and Drug Administration |date=April 2016 |access-date=20 January 2018}}</ref> Outside the U.S., similar data integrity guidance has been issued by the United Kingdom (2015), Switzerland (2016), and Australia (2017).<ref name="DavidsonData17">{{cite web |url=https://www.contractpharma.com/issues/2017-07-01/view_fda-watch/data-integrity-guidance-around-the-world/ |title=Data Integrity Guidance Around the World |author=Davidson, J. |work=Contract Pharma |publisher=Rodman Media |date=18 July 2017 |access-date=20 January 2018}}</ref>
* Various standards for the manufacture of medical devices address data integrity either directly or indirectly, including [[ISO 13485]], [[ISO 14155]], and ISO 5840.<ref name="PDAData15">{{cite web |url=https://www.pda.org/docs/default-source/website-document-library/chapters/presentations/ireland/pda-data-integrity-seminar-presentations.pdf?sfvrsn=4 |format=PDF |title=Data Integrity: A perspective from the medical device regulatory and standards framework |work=Data Integrity Seminar |author=Scannel, P. |publisher=Parenteral Drug Association |pages=10–57 |date=12 May 2015 |access-date=20 January 2018 |archive-date=20 January 2018 |archive-url=https://web.archive.org/web/20180120221035/https://www.pda.org/docs/default-source/website-document-library/chapters/presentations/ireland/pda-data-integrity-seminar-presentations.pdf?sfvrsn=4 |url-status=dead }}</ref>
* In early 2017, the [[Financial Industry Regulatory Authority]] (FINRA), noting data integrity problems with automated trading and money movement surveillance systems, stated it would make "the development of a data integrity program to monitor the accuracy of the submitted data" a priority.<ref name="Cook2017_17">{{cite web |url=http://www.finra.org/industry/2017-regulatory-and-examination-priorities-letter |title=2017 Regulatory and Examination Priorities Letter |author=Cook, R. |publisher=Financial Industry Regulatory Authority |date=4 January 2017 |access-date=20 January 2018}}</ref> In early 2018, FINRA said it would expand its approach on data integrity to firms' "technology change management policies and procedures" and Treasury securities reviews.<ref name="Cook2018_18">{{cite web |url=http://www.finra.org/industry/2018-regulatory-and-examination-priorities-letter |title=2018 Regulatory and Examination Priorities Letter |author=Cook, R. |publisher=Financial Industry Regulatory Authority |date=8 January 2018 |access-date=20 January 2018}}</ref>
* Other sectors such as mining<ref name="AccentureDataInt16">{{cite web |url=https://www.accenture.com/t20160610T050736Z__w__/us-en/_acnmedia/PDF-21/Accenture-Data-Integrity-Mining-Productivity.pdf |archive-url=https://ghostarchive.org/archive/20221009/https://www.accenture.com/t20160610T050736Z__w__/us-en/_acnmedia/PDF-21/Accenture-Data-Integrity-Mining-Productivity.pdf |archive-date=2022-10-09 |url-status=live |title=Data Integrity: Enabling Effective Decisions in Mining Operations |publisher=Accenture |date=2016 |access-date=20 January 2018}}</ref> and product manufacturing<ref name="NymiIndustry17">{{cite web |url=https://nymi.com/blog/industry-40-and-cyber-physical-systems-raise-data-integrity-imperative |archive-url=https://web.archive.org/web/20180120222300/https://nymi.com/blog/industry-40-and-cyber-physical-systems-raise-data-integrity-imperative |url-status=dead |archive-date=January 20, 2018 |title=Industry 4.0 and Cyber-Physical Systems Raise the Data Integrity Imperative |work=Nymi Blog |publisher=Nymi, Inc |date=24 October 2017 |access-date=20 January 2018 }}</ref> are increasingly focusing on the importance of data integrity in associated automation and production monitoring assets.
* Cloud storage providers have long faced significant challenges ensuring the integrity or provenance of customer data and tracking violations.<ref name="PriyadharshiniData12">{{cite journal |title=Data integrity in cloud storage |journal=Proceedings from the 2012 International Conference on Advances in Engineering, Science and Management |author=Priyadharshini, B. |author2=Parvathi, P. |year=2012 |isbn=9788190904223}}</ref><ref name="ZafarASurv17">{{cite journal |title=A survey of cloud computing data integrity schemes: Design challenges, taxonomy and future trends |journal=Computers & Security |author=Zafar, F. |author2=Khan, A. |author3=Malik, S.U.R.|display-authors=et al |volume=65 |issue=3 |pages=29–49 |year=2017 |doi=10.1016/j.cose.2016.10.006}}</ref><ref name="ImranProvenance17">{{cite journal |title=Provenance based data integrity checking and verification in cloud environments |journal=PLOS ONE |author=Imran, M. |author2=Hlavacs, H. |author3=Haq, I.U.I.|display-authors=et al |volume=12 |issue=5 |pages=e0177576 |year=2017 |doi=10.1371/journal.pone.0177576 |pmid=28545151 |pmc=5435237|bibcode=2017PLoSO..1277576I |doi-access=free }}</ref>

== See also ==
* [[End-to-end data integrity]]
* [[Message authentication]]
* [[National Information Assurance Glossary]]
* [[Single version of the truth]]

== References ==
{{Reflist|30em}}

{{Authority control}}

{{data}}

{{DEFAULTSORT:Data Integrity}}
[[Category:Data quality]]
[[Category:Relational model]]
[[Category:Transaction processing]]