Editing Exploratory data analysis

{{short description|Approach of analyzing data sets in statistics}}{{Data Visualization}}

In [[statistics]], '''exploratory data analysis''' (EDA) is an approach of [[data analysis|analyzing]] [[data set]]s to summarize their main characteristics, often using [[statistical graphics]] and other [[data visualization]] methods. A [[statistical model]] can be used or not, but primarily EDA is for seeing what the data can tell beyond the formal modeling and thereby contrasts with traditional hypothesis testing, in which a model is supposed to be selected before the data is seen. Exploratory data analysis has been promoted by [[John Tukey]] since 1970 to encourage statisticians to explore the data, and possibly formulate hypotheses that could lead to new data collection and experiments. EDA is different from [[Data analysis#Initial data analysis|initial data analysis (IDA)]],<ref>{{cite book |last=Chatfield |first=C. |year=1995 |title=Problem Solving: A Statistician's Guide |publisher=Chapman and Hall |isbn=978-0412606304 |edition=2nd }}</ref><ref>{{cite journal |doi=10.1371/journal.pcbi.1009819|title=Ten simple rules for initial data analysis|year=2022|last1=Baillie|first1=Mark|last2=Le Cessie|first2=Saskia|last3=Schmidt|first3=Carsten Oliver|last4=Lusa|first4=Lara|last5=Huebner|first5=Marianne|author6=Topic Group "Initial Data Analysis" of the STRATOS Initiative|journal=PLOS Computational Biology|volume=18|issue=2|pages=e1009819|pmid=35202399|pmc=8870512|bibcode=2022PLSCB..18E9819B |doi-access=free }}</ref> which focuses more narrowly on checking assumptions required for model fitting and hypothesis testing, and handling missing values and making transformations of variables as needed. EDA encompasses IDA.

==Overview==

Tukey defined data analysis in 1961 as: "Procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data."<ref>[http://projecteuclid.org/download/pdf_1/euclid.aoms/1177704711 John Tukey-The Future of Data Analysis-July 1961]</ref>

Exploratory data analysis is a technique to analyze and investigate a dataset and summarize its main characteristics. A main advantage of EDA is providing the visualization of data after conducting analysis. 

Tukey's championing of EDA encouraged the development of [[Computational statistics|statistical computing]] packages, especially [[S (programming language)|S]] at [[Bell Labs]].<ref>{{Citation |last=Becker |first=Richard A. |title=A Brief History of S |publisher=AT&T Bell Laboratories |place=Murray Hill, New Jersey |access-date=2015-07-23 |url=http://www2.research.att.com/areas/stat/doc/94.11.ps |format=PS |archive-url=https://web.archive.org/web/20150723044213/http://www2.research.att.com/areas/stat/doc/94.11.ps |archive-date=2015-07-23 |quotation="... we wanted to be able to interact with our data, using Exploratory Data Analysis (Tukey, 1971) techniques."}}</ref> The S programming language inspired the systems [[S-PLUS]] and [[R (programming language)|R]]. This family of statistical-computing environments featured vastly improved dynamic visualization capabilities, which allowed statisticians to identify [[outlier]]s, [[trend estimation|trends]] and [[pattern recognition|patterns]] in data that merited further study.

Tukey's EDA was related to two other developments in [[statistical theory]]: [[robust statistics]] and [[nonparametric statistics]], both of which tried to reduce the sensitivity of statistical inferences to errors in formulating [[statistical model]]s. Tukey promoted the use of [[five number summary]] of numerical data—the two [[extreme value|extreme]]s ([[maximum]] and [[minimum]]), the [[median]], and the [[quartile]]s—because these median and quartiles, being functions of the [[empirical distribution function|empirical distribution]] <!-- [[statistical functional]]s  (and the related [[interquartile range]] and [[range]]) -->are defined for all distributions, unlike the [[mean value|mean]] and [[standard deviation]]. Moreover, the quartiles and median are more robust to [[skewness|skewed]] or [[heavy-tailed distribution]]s than traditional summaries (the mean and standard deviation). The packages [[S (programming language)|S]], [[S-PLUS]], and [[R (programming language)|R]] included routines using [[resampling (statistics)|resampling statistics]], such as Quenouille and Tukey's [[resampling (statistics)#Jackknife|jackknife]] and [[Bradley Efron|Efron]]{{'s}} [[bootstrapping (statistics)|bootstrap]], which are nonparametric and robust (for many problems).

Exploratory data analysis, robust statistics, nonparametric statistics, and the development of statistical programming languages facilitated statisticians' work on scientific and engineering problems. Such problems included the fabrication of semiconductors and the understanding of communications networks, both of which were of interest to Bell Labs. These statistical developments, all championed by Tukey, were designed to complement the [[analytic function|analytic]] theory of [[statistical hypothesis testing|testing statistical hypotheses]], particularly the [[Pierre-Simon Laplace|Laplacian]] tradition's emphasis on [[exponential family|exponential families]].<ref>{{cite journal |title=Conversation with John W. Tukey and Elizabeth Tukey, Luisa T. Fernholz and Stephan Morgenthaler |journal=Statistical Science |volume=15 |issue=1 |year=2000 |pages=79–94 |doi=10.1214/ss/1009212675|last1=Morgenthaler |first1=Stephan |last2=Fernholz |first2=Luisa T. |doi-access=free }}</ref>

== Development ==
[[File:Data visualization process v1.png|right|350px|thumb|Data science process flowchart]]
[[John W. Tukey]] wrote the book ''Exploratory Data Analysis'' in 1977.<ref name="Tukey1977">{{cite book
| last = Tukey
| first = John W.
| year = 1977
| title = Exploratory Data Analysis | publisher = Pearson
| isbn = 978-0201076165| title-link = Exploratory Data Analysis
}}</ref> Tukey held that too much emphasis in statistics was placed on [[statistical hypothesis testing]] (confirmatory data analysis); more emphasis needed to be placed on using [[data]] to suggest hypotheses to test. In particular, he held that confusing the two types of analyses and employing them on the same set of data can lead to [[Systematic error|systematic bias]] owing to the issues inherent in [[testing hypotheses suggested by the data]].

The objectives of EDA are to:
*Enable unexpected discoveries in the data
*Suggest hypotheses about the [[causality|causes]] of observed [[phenomenon|phenomena]]
*Assess assumptions on which [[statistical inference]] will be based
*Support the selection of appropriate statistical tools and techniques
*Provide a basis for further data collection through [[survey sampling|surveys]] or [[design of experiments|experiments]]<ref>[https://web.archive.org/web/20170808064326/cll.stanford.edu/~willb/course/behrens97pm.pdf Behrens-Principles and Procedures of Exploratory Data Analysis-American Psychological Association-1997]</ref>

Many EDA techniques have been adopted into [[data mining]]. They are also being taught to young students as a way to introduce them to statistical thinking.<ref>{{cite journal |last=Konold |first=C. |year=1999 |title=Statistics goes to school |journal=Contemporary Psychology |volume=44 |issue=1 |pages=81–82 |doi= 10.1037/001949}}</ref>

== Techniques and tools ==
There are a number of tools that are useful for EDA, but EDA is characterized more by the attitude taken than by particular techniques.<ref>{{cite journal |title=We need both exploratory and confirmatory |first=John W. |last=Tukey |journal=The American Statistician |volume=34 |issue=1 |year=1980 |pages=23–25 |doi=10.1080/00031305.1980.10482706 }}</ref>

Typical [[Statistical graphics|graphical techniques]] used in EDA are:

*[[Box plot]]
*[[Histogram]]
*[[Multi-vari chart]]
*[[Run chart]]
*[[Pareto chart]]
*[[Scatter plot]] (2D/3D)
*[[stemplot|Stem-and-leaf plot]]
*[[Parallel coordinates]]
*[[Odds ratio]]
*[[Targeted projection pursuit]]
*[[Heat map]]
*[[Bar chart]]
*Horizon graph
*Glyph-based visualization methods such as PhenoPlot<ref>{{Cite journal|last1=Sailem|first1=Heba Z.|last2=Sero|first2=Julia E.|last3=Bakal|first3=Chris|date=2015-01-08|title=Visualizing cellular imaging data using PhenoPlot|journal=Nature Communications|language=En|volume=6|issue=1|pages=5825|doi=10.1038/ncomms6825|pmid=25569359|pmc=4354266|bibcode=2015NatCo...6.5825S|issn=2041-1723}}</ref> and [[Chernoff face]]s
* Projection methods such as grand tour, guided tour and manual tour
* Interactive versions of these plots

[[Dimensionality reduction]]:
*[[Multidimensional scaling]]
*[[Principal component analysis]] (PCA)
*[[Multilinear principal component analysis|Multilinear PCA]]
*[[Nonlinear dimensionality reduction]] (NLDR)
*[[Iconography of correlations]]

Typical [[quantity|quantitative]] techniques are:

*[[Median polish]]
*[[Trimean]]
*[[Ordination (statistics)|Ordination]]

== History ==
Many EDA ideas can be traced back to earlier authors, for example:
* [[Francis Galton]] emphasized [[order statistic]]s and [[quantile]]s.
* [[Arthur Lyon Bowley]] used precursors of the stemplot and [[five-number summary]] (Bowley actually used a "[[seven-number summary|seven-figure summary]]", including the extremes, [[decile]]s and [[quartile]]s, along with the median&mdash;see his ''Elementary Manual of Statistics'' (3rd edn., 1920), p.&nbsp;62<ref>Elementary Manual of Statistics (3rd edn., 1920)https://archive.org/details/cu31924013702968/page/n5</ref>– he defines "the maximum and minimum, median, quartiles and two deciles" as the "seven positions").
* [[Andrew S. C. Ehrenberg|Andrew Ehrenberg]] articulated a philosophy of [[data reduction]] (see his book of the same name).

The [[Open University]] course ''Statistics in Society'' (MDST 242), took the above ideas and merged them with [[Gottfried Noether]]'s work, which introduced [[statistical inference]] via coin-tossing and the [[median test]].

== Example ==

Findings from EDA are orthogonal to the primary analysis task. To illustrate, consider an example from Cook et al. where the analysis task is to find the variables which best predict the tip that a dining party will give to the waiter.<ref>[[Dianne Cook (statistician)|Cook, D.]] and [[Deborah F. Swayne|Swayne, D.F.]] (with A. Buja, D. Temple Lang, H. Hofmann, H. Wickham, M. Lawrence) (2007) "Interactive and Dynamic Graphics for Data Analysis: With R and GGobi" Springer, 978-0387717616</ref> The variables available in the data collected for this task are: the tip amount, total bill, payer gender, smoking/non-smoking section, time of day, day of the week, and size of the party. The primary analysis task is approached by fitting a regression model where the tip rate is the response variable. The fitted model is

: ([[tip rate]]) = 0.18 - 0.01 × (party size)

which says that as the size of the dining party increases by one person (leading to a higher bill), the tip rate will decrease by 1%, on average.

However, exploring the data reveals other interesting features not described by this model.
<gallery mode="packed" heights="200px." style="text-align:left">
Tips-hist1.png|Histogram of tip amounts where the bins cover $1 increments. The distribution of values is skewed right and unimodal, as is common in distributions of small, non-negative quantities.
Tips-hist2.png|Histogram of tip amounts where the bins cover $0.10 increments. An interesting phenomenon is visible: peaks occur at the whole-dollar and half-dollar amounts, which is caused by customers picking round numbers as tips. This behavior is common to other types of purchases too, like gasoline.
Tips-scat1.png|Scatterplot of tips vs. bill. Points below the line correspond to tips that are lower than expected (for that bill amount), and points above the line are higher than expected. We might expect to see a tight, positive linear association, but instead see [[heteroscedasticity|variation that increases with tip amount]]. In particular, there are more points far away from the line in the lower right than in the upper left, indicating that more customers are very cheap than very generous.
Tips-scat2.png|Scatterplot of tips vs. bill separated by payer gender and smoking section status. Smoking parties have a lot more variability in the tips that they give. Males tend to pay the (few) higher bills, and the female non-smokers tend to be very consistent tippers (with three conspicuous exceptions shown in the sample).
</gallery>

What is learned from the plots is different from what is illustrated by the regression model, even though the experiment was not designed to investigate any of these other trends. The patterns found by exploring the data suggest hypotheses about tipping that may not have been anticipated in advance, and which could lead to interesting follow-up experiments where the hypotheses are formally stated and tested by collecting new data.

== Software ==
* [[JMP (statistical software)|JMP]], an EDA package from [[SAS Institute]].
* [[KNIME]], Konstanz Information Miner – Open-Source data exploration platform based on Eclipse.
*[[Minitab]], an EDA and general statistics package widely used in industrial and corporate settings.
* [[Orange (software)|Orange]], an [[open-source software|open-source]] [[data mining]] and [[machine learning]] software suite.
* [[Python (programming language)|Python]], an open-source programming language widely used in data mining and machine learning.
* Matplotlib & Seaborn are the Python libraries used in todays world for EDA and Plotting/Data Visualization.(point updated: 2025)
* [[R (programming language)|R]], an open-source programming language for statistical computing and graphics. Together with Python one of the most popular languages for data science.
* [[TinkerPlots]] an EDA software for upper elementary and middle school students.
* [[Weka (machine learning)|Weka]] an open source data mining package that includes visualization and EDA tools such as [[targeted projection pursuit]].

== See also ==
*[[Anscombe's quartet]], on importance of exploration
*[[Data dredging]]
*[[Predictive analytics]]
*[[Structured data analysis (statistics)]]
*[[Configural frequency analysis]]
*[[Descriptive statistics]]

== References ==
{{reflist}}

== Bibliography ==
*[[Natalia Andrienko|Andrienko, N]] & Andrienko, G (2005) ''Exploratory Analysis of Spatial and Temporal Data. A Systematic Approach''. Springer. {{ISBN|3-540-25994-5}}
*{{cite book |author=[[Dianne Cook (statistician)|Cook, D.]] and [[Deborah F. Swayne|Swayne, D.F.]] (with A. Buja, D. Temple Lang, H. Hofmann, H. Wickham, M. Lawrence)| title=Interactive and Dynamic Graphics for Data Analysis: With R and GGobi| publisher=Springer| isbn=9780387717616| date=2007-12-12}}
*Cook, D. and Swayne, D.F. (with A. Buja, D. Temple Lang, H. Hofmann, H. Wickham, M. Lawrence) (2007-12-12). Interactive and Dynamic Graphics for Data Analysis: With R and GGobi. Springer. ISBN 9780387717616.
*Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 978-0-471-09776-1.
*Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 978-0-471-09777-8.
*Young, F. W. Valero-Mora, P. and Friendly M. (2006) Visual Statistics: Seeing your data with Dynamic Interactive Graphics. Wiley ISBN 978-0-471-68160-1 Jambu M. (1991) Exploratory and Multivariate Data Analysis. Academic Press ISBN 0123800900
* S. H. C. DuToit, A. G. W. Steyn, R. H. Stumpf (1986) Graphical Exploratory Data Analysis. Springer ISBN 978-1-4612-9371-2
*{{cite book |last=Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) |title=Exploring Data Tables, Trends and Shapes |year=1985 |publisher=Wiley |isbn=978-0-471-09776-1 |url-access=registration |url=https://archive.org/details/exploringdatatab0000unse }}
*{{cite book |last=Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) |title=Understanding Robust and Exploratory Data Analysis |year=1983 |publisher=Wiley |isbn=978-0-471-09777-8 }}
*{{cite book |last=Inselberg |first= Alfred |title=Parallel Coordinates:Visual Multidimensional Geometry and its Applications |year=2009 |publisher=Springer |location= London New York|isbn=978-0-387-68628-8 }}
*Leinhardt, G., Leinhardt, S., ''[https://journals.sagepub.com/doi/pdf/10.3102/0091732X008001085 Exploratory Data Analysis: New Tools for the Analysis of Empirical Data]'', Review of Research in Education, Vol. 8, 1980 (1980), pp.&nbsp;85–157.
*{{cite book|author1=Martinez, W. L.|author1-link= Wendy L. Martinez |author2=Martinez, A. R. |author3= Solka, J. |name-list-style=amp |year=2010|title=Exploratory Data Analysis with MATLAB, second edition|publisher=Chapman & Hall/CRC|isbn= 9781439812204}}
*Theus, M., Urbanek, S. (2008), Interactive Graphics for Data Analysis: Principles and Examples, CRC Press, Boca Raton, FL, {{ISBN|978-1-58488-594-8}}
*{{cite book |author1=Tucker, L |author2=MacCallum, R. |title=Exploratory Factor Analysis |year=1993 |url= http://www.unc.edu/~rcm/book/factornew.htm}}
*{{cite book |last=Tukey |first=John Wilder |title=Exploratory Data Analysis |year=1977 |publisher=Addison-Wesley |isbn=978-0-201-07616-5 |url=https://archive.org/details/exploratorydataa00tuke_0 |url-access=registration }}
*{{cite book |title=Applications, Basics and Computing of Exploratory Data Analysis |last1=Velleman |first1=P. F. |last2=Hoaglin |first2=D. C. |year=1981 |publisher=Duxbury Press |isbn=978-0-87150-409-8 |url-access=registration |url=https://archive.org/details/applicationsbasi00vell }}
* Young, F. W. Valero-Mora, P. and Friendly M. (2006) [http://www.uv.es/visualstats/Book ''Visual Statistics: Seeing your data with Dynamic Interactive Graphics'']. Wiley {{ISBN|978-0-471-68160-1}}
*Jambu M. (1991) [http://www.sciencedirect.com/science/book/9780123800909 ''Exploratory and Multivariate Data Analysis'']. Academic Press {{ISBN|0123800900}}
*S. H. C. DuToit, A. G. W. Steyn, R. H. Stumpf (1986) [https://link.springer.com/book/10.1007%2F978-1-4612-4950-4 ''Graphical Exploratory Data Analysis''].  Springer {{ISBN|978-1-4612-9371-2}}

<!-- unclear why these are repeated here when they are listed above
Andrienko, N & Andrienko, G (2005) Exploratory Analysis of Spatial and Temporal Data. A Systematic Approach. Springer. ISBN 3-540-25994-5
Cook, D. and Swayne, D.F. (with A. Buja, D. Temple Lang, H. Hofmann, H. Wickham, M. Lawrence) (2007-12-12). Interactive and Dynamic Graphics for Data Analysis: With R and GGobi. Springer. ISBN 9780387717616.
Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 978-0-471-09776-1.
Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 978-0-471-09777-8.
Young, F. W. Valero-Mora, P. and Friendly M. (2006) Visual Statistics: Seeing your data with Dynamic Interactive Graphics. Wiley ISBN 978-0-471-68160-1 Jambu M. (1991) Exploratory and Multivariate Data Analysis. Academic Press ISBN 0123800900 S. H. C. DuToit, A. G. W. Steyn, R. H. Stumpf (1986) Graphical Exploratory Data Analysis. Springer ISBN 978-1-4612-9371-2 -->

== External links ==
* [https://oli.cmu.edu/courses/free-open/statistics-course-details/ Carnegie Mellon University – free online course on Probability and Statistics, with a module on EDA]
* [https://www.itl.nist.gov/div898/handbook/eda/eda.htm • Exploratory data analysis chapter: engineering statistics handbook]

{{Social surveys}}
{{Authority control}}

[[Category:Exploratory data analysis| ]]