Editing Durability (database systems) (section)

==Mechanisms==
[[File:Database failure states and relations.png|thumb|A simplified [[Finite-state machine|finite state automaton]] showing possible DBMS after-failure (in red) states and the transitions (in black) that are necessary to return to a running system to achieve durability.|350x350px]]
In transaction-based systems, the mechanisms that assure durability are historically associated with the concept of [[Reliability engineering|reliability]] of systems, as proposed by [[Jim Gray (computer scientist)|Jim Gray]] in 1981.<ref name=":0" /> This concept includes durability, but it also relies on aspects of the [[Atomicity (database systems)|atomicity]] and [[Consistency (database systems)|consistency]] properties.<ref name=":2">{{Cite book |title=Database systems: concepts, languages & architectures |date=1999 |publisher=McGraw-Hill |isbn=978-0-07-709500-0 |editor-last=Atzeni |editor-first=Paolo |location=New York |pages=311–320}}</ref> Specifically, a reliability mechanism requires [[Statement (computer science)|primitives]] that explicitly state the beginning, the end, and the [[Rollback (data management)|rollback]] of transactions,<ref name=":0" /> which are also implied for the other two aforementioned properties. In this article, only the mechanisms strictly related to durability have been considered. These mechanisms are divided into three levels: transaction, system, and media level.  This can be seen as well for scenarios where failures could happen and that have to be considered in the design of database systems to address durability.<ref name=":1" /> 

=== Transaction level ===
Durability against failures that occur at transaction level, such as canceled calls and inconsistent actions that may be blocked before committing by [[Constraint (database)|constraints]] and [[Database trigger|triggers]], is guaranteed by the [[serializability]] property of the execution of transactions. The state generated by the effects of precedently committed transactions is available in main memory and, thus, is resilient, while the changes carried by non-committed transactions can be undone. In fact, thanks to serializability, they can be discerned from other transactions and, therefore, their changes are discarded.<ref name=":1" /> In addition, it is relevant to consider that in-place changes, which overwrite old values without keeping any kind of history are discouraged.<ref name=":0" /> There exist multiple approaches that keep track of the history of changes, such as [[Timestamp-based concurrency control|timestamp]]-based solutions<ref>{{Cite journal |last=Svobodova |first=L. |date=1980 |title=MANAGEMENT OF OBJECT HISTORIES IN THE SWALLOW REPOSITORY |url=https://dl.acm.org/doi/10.5555/889850 |journal=Mit/LCS Tr-243 |location=USA |doi=}}</ref> or [[Logging (computing)|logging]] and [[Locking (computer science)|locking]].<ref name=":0" />

=== System level ===
At system level, failures happen, by definition,<ref name=":1" /> when the contents of the volatile storage are lost. This can occur in events like system crashes or [[Power outage|power outages]]. Existing database systems use volatile storage (i.e. the [[Computer memory|main memory]] of the system) for different purposes: some store their whole state and data in it, even without any durability guarantee; others keep the state and the data, or part of them, in memory, but also use the [[non-volatile storage]] for data; other systems only keep the state in main memory, while keeping all the data on disk.<ref>{{Cite book |last=Petrov |first=Oleksandr |title=Database internals: a deep dive into how distributed data systems work |date=2019 |publisher=O'Reilly |isbn=978-1-4920-4034-7 |edition=1st |location=Beijing Boston Farnham Sebastopol Tokyo |pages=40–42}}</ref> The reason behind the choice of having volatile storage, which is subject to this type of failure, and non-volatile storage, is found in the performance differences of the existing technologies that are used to implement these kinds of storage. However, the situation is likely to evolve as the popularity of [[Non Volatile Memory Express|non-volatile memories (NVM)]] technologies grows.<ref>{{Cite book |last1=Arulraj |first1=Joy |last2=Pavlo |first2=Andrew |title=Proceedings of the 2017 ACM International Conference on Management of Data |chapter=How to Build a Non-Volatile Memory Database Management System |date=2017-05-09 |chapter-url=https://dl.acm.org/doi/10.1145/3035918.3054780 |series=SIGMOD '17 |location=New York, NY, USA |publisher=Association for Computing Machinery |pages=1753–1758 |doi=10.1145/3035918.3054780 |isbn=978-1-4503-4197-4|s2cid=648876 }}</ref>

In systems that include non-volatile storage, durability can be achieved by keeping and flushing an immutable sequential [[Transaction log|log of the transactions]] to such non-volatile storage before acknowledging commitment. Thanks to their atomicity property, the transactions can be considered the unit of work in the [[Data recovery|recovery]] process that guarantees durability while exploiting the log. In particular, the logging mechanism is called [[Write-ahead logging|write-ahead log (WAL)]] and allows durability by buffering changes to the disk before they are synchronized from the main memory. In this way, by reconstruction from the log file, all committed transactions are resilient to system-level failures, because they can be redone. Non-committed transactions, instead, are recoverable, since their operations are logged to non-volatile storage before they effectively modify the state of the database.<ref name=":3">{{Cite book |last=Petrov |first=Oleksandr |title=Database internals: a deep dive into how distributed data systems work |date=2019 |publisher=O'Reilly |isbn=978-1-4920-4034-7 |edition=1st |location=Beijing Boston Farnham Sebastopol Tokyo |pages=185–195}}</ref> In this way, the partially executed operations can be undone without affecting the state of the system. After that, those transactions that were incomplete can be redone. Therefore, the transaction log from non-volatile storage can be reprocessed to recreate the system state right before any later system-level failure. Logging is done as a combination of tracking data and operations (i.e. transactions) for performance reasons.<ref>{{Cite journal |last1=Mohan |first1=C. |last2=Haderle |first2=Don |last3=Lindsay |first3=Bruce |last4=Pirahesh |first4=Hamid |last5=Schwarz |first5=Peter |date=1992-03-01 |title=ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging |journal=ACM Transactions on Database Systems |volume=17 |issue=1 |pages=94–162 |doi=10.1145/128765.128770 |s2cid=8759704 |issn=0362-5915|doi-access=free }}</ref>

=== Media level ===
At media level, failure scenarios affect non-volatile storage, like [[Hard disk drive|hard disk drives]], [[Solid-state drive|solid-state drives]], and other types of [[Computer data storage|storage hardware components]].<ref name=":3" /> To guarantee durability at this level, the database system shall rely on stable memory, which is a memory that is completely and ideally failure-resistant. This kind of memory can be achieved with mechanisms of [[Replication (computing)|replication]] and robust writing protocols.<ref name=":2" />  

Many tools and technologies are available to provide a logical stable memory, such as the [[Mirroring RAID|mirroring]] of disks, and their choice depends on the [[Requirement|requirements]] of the specific applications.<ref name=":2" /> In general, replication and [[Redundancy (engineering)|redundancy]] strategies and architectures that behave like stable memory are available at different levels of the technology stack. In this way, even in case of catastrophic events where the storage hardware is damaged, [[data loss]] can be prevented.<ref>{{Cite book |last=Eich |first=Margaret H. |title=1987 IEEE Third International Conference on Data Engineering |chapter=A classification and comparison of main memory database recovery techniques |date=1987-02-01 |chapter-url=https://ieeexplore.ieee.org/document/7272398 |publisher=IEEE |pages=332–339 |doi=10.1109/ICDE.1987.7272398 |isbn=978-0-8186-0762-2|s2cid=207773738 }}</ref> At this level, there is a strong bond between durability and [[Data recovery|system and data recovery]], in the sense that the main goal is to preserve the data, not necessarily in online replicas, but also as offline copies.<ref name=":2" /> These last techniques fall into the categories of [[backup]], [[Data loss prevention software|data loss prevention]], and [[IT disaster recovery]].<ref>{{Cite journal |last1=Choy |first1=Manhoi |last2=Leong |first2=Hong Va |last3=Wong |first3=Man Hon |date=2000 |title=Disaster recovery techniques for database systems |url=https://dl.acm.org/doi/10.1145/352515.352521 |journal=Communications of the ACM |language=en |volume=43 |issue=11es |pages=6 |doi=10.1145/352515.352521 |s2cid=14781378 |issn=0001-0782|url-access=subscription }}</ref> 

Therefore, in case of media failure, the durability of transactions is guaranteed by the ability to reconstruct the state of the database from the log files stored in the stable memory, in any way it was implemented in the database system.<ref name=":3" /> There exist several mechanisms to store and reconstruct the state of a database system that improves the performance, both in terms of space and time, compared to managing all the log files created from the beginning of the database system. These mechanisms often include incremental [[Database dump|dumping]], [[Differential backup|differential files]], and [[Database checkpoint|checkpoints]].<ref>{{Cite journal |last=Verhofstad |first=Joost S. M. |date=1978-06-01 |title=Recovery Techniques for Database Systems |url=http://portal.acm.org/citation.cfm?doid=356725.356730 |journal=ACM Computing Surveys |volume=10 |issue=2 |pages=167–195 |doi=10.1145/356725.356730|s2cid=8847522 |url-access=subscription }}</ref>

===Distributed databases===
In [[distributed transaction]]s, ensuring durability requires additional mechanisms to preserve a consistent state sequence across all database nodes. This means, for example, that a single node may not be enough to decide to conclude a transaction by committing it. In fact, the resources used in that transaction may be on other nodes, where other transactions are occurring concurrently. Otherwise, in case of failure, if consistency could not be guaranteed, it would be impossible to acknowledge a safe state of the database for recovery.  For this reason, all participating nodes must coordinate before a commit can be acknowledged. This is usually done by a [[two-phase commit protocol]].<ref name=":4">{{Cite journal |last1=Mohan |first1=C. |last2=Haderle |first2=Don |last3=Lindsay |first3=Bruce |last4=Pirahesh |first4=Hamid |last5=Schwarz |first5=Peter |date=1992-03-01 |title=ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging |journal=ACM Transactions on Database Systems |volume=17 |issue=1 |pages=94–162 |doi=10.1145/128765.128770 |s2cid=8759704 |issn=0362-5915|doi-access=free }}</ref>

In addition, in [[Distributed database|distributed databases]], even the protocols for logging and recovery shall address the issues of [[Distributed computing|distributed environments]], such as [[deadlock (computer science)|deadlock]]s, that could prevent the resilience and recoverability of transactions and, thus, durability.<ref name=":4" /> A widely adopted family of algorithms that ensures these properties is [[Algorithms for Recovery and Isolation Exploiting Semantics|Algorithms for Recovery and Isolation Exploiting Semantics (ARIES)]].<ref name=":3" />