Plain-language explanation.
Exploratory data analysis (EDA) is the first step in any data project — getting to know your data before formal analysis. It involves summarising, visualising, and checking data to understand its structure, find patterns, detect anomalies, and identify problems. EDA prevents costly mistakes that come from jumping straight to modelling with poorly understood data.
Core concepts and standard treatment.
EDA workflow: data acquisition and loading (CSV, JSON, SQL, API — pandas, tidyverse); shape and structure inspection (dimensions, dtypes, sample rows, memory usage); summary statistics (mean, median, std, min, max, quartiles — df.describe()); missing data analysis (missingness patterns — MAR, MCAR, MNAR; heatmaps, msno.matrix); and duplicate detection. Univariate analysis: histograms and KDE plots for continuous variables (normality assessment — Q-Q plots, Shapiro-Wilk test, skewness and kurtosis); bar charts for categorical variables (value counts, cardinality issues); and box plots (IQR-based outlier detection — Tukey fences).
Deeper theory, debates and edge cases.
Bivariate and multivariate analysis: scatter plots with regression lines (correlation, non-linear relationships, heteroscedasticity); correlation matrices and heatmaps (Pearson for linear, Spearman for monotonic, Kendall for ordinal); pair plots (seaborn pairplot — diagonal KDE, off-diagonal scatter); and grouped analysis (split by categorical variable using groupby, facet grids). Outlier analysis: Z-score method (> 3 SD), IQR method, isolation forests (unsupervised — identify anomalous data points), and the distinction between outliers as errors vs. outliers as extreme true values. Feature engineering during EDA: log transforms for skewed distributions, binning continuous variables, creating interaction features, and datetime feature extraction.
How it is applied in practice.
Tools and workflows: Jupyter notebooks (reproducible EDA with code, outputs, and narrative); pandas-profiling and ydata-profiling (automated EDA reports); sweetviz (comparative EDA between train/test splits); and D-Tale (interactive pandas exploration). EDA for data quality: identifying incorrect data types, implausible values (ages > 150, negative prices), inconsistent categorical values (UK, United Kingdom, Uk, GB), and join key anomalies (many-to-many joins, missing foreign keys). Communication of EDA findings: the EDA report documents data provenance, quality issues, key distributions, correlations, and hypotheses for modelling — a critical hand-off document in team data science projects. Version control for data: DVC (Data Version Control), Delta Lake, and data catalogues track data changes alongside code changes for reproducibility.