Editing Extract, transform, load (section)

=== Extract ===

ETL processing involves extracting the data from the source system(s). In many cases, this represents the most important aspect of ETL, since extracting data correctly sets the stage for the success of subsequent processes. Most data-warehousing projects combine data from different source systems. Each separate system may also use a different data organization and/or [[File format|format]]. Common data-source formats include [[relational database]]s, [[flat-file database]]s, [[XML]], and [[JSON]], but may also include non-relational database structures such as [[IBM Information Management System]] or other data structures such as [[Virtual Storage Access Method|Virtual Storage Access Method (VSAM)]] or [[ISAM|Indexed Sequential Access Method (ISAM)]], or even formats fetched from outside sources by means such as a [[web crawler]] or [[data scraping]]. The streaming of the extracted data source and loading on-the-fly to the destination database is another way of performing ETL when no intermediate data storage is required.

An intrinsic part of the extraction involves data validation to confirm whether the data pulled from the sources has the correct/expected values in a given domain (such as a pattern/default or list of values). If the data fails the validation rules, it is rejected entirely or in part. The rejected data is ideally reported back to the source system for further analysis to identify and to rectify incorrect records or perform [[data wrangling]].