Plain-language explanation.
Big data refers to datasets so large and complex that traditional database tools cannot handle them. Data engineering is the work of building the systems that collect, store, process, and serve data so that analysts and scientists can use it. Without good data engineering, data science is impossible — clean, reliable, well-organised data is the foundation of all analysis.
Core concepts and standard treatment.
Data engineering concepts: data sources (transactional databases, APIs, log files, IoT sensors, web scraping, third-party data providers); ETL/ELT pipelines (Extract, Transform, Load — or Extract, Load, Transform in modern cloud architectures); data warehouses (structured, query-optimised: Snowflake, BigQuery, Redshift — OLAP vs. transactional OLTP databases); data lakes (raw storage: S3, Azure Data Lake, GCS — schema-on-read flexibility); and data lakehouses (Delta Lake, Apache Iceberg — combining lake storage with warehouse reliability).
Deeper theory, debates and edge cases.
Data modelling: entity-relationship diagrams (ERDs), normalisation (1NF, 2NF, 3NF, BCNF — reducing redundancy) vs. denormalisation (star and snowflake schemas for analytics — dimensional modelling, Kimball approach); and graph databases (Neo4j: nodes and edges for network data). SQL mastery: window functions (ROW_NUMBER, RANK, LAG, LEAD, SUM OVER, running totals); CTEs (Common Table Expressions for readable complex queries); set operations (UNION, INTERSECT, EXCEPT); and performance optimisation (index strategy, query execution plan analysis). Workflow orchestration: Apache Airflow (DAGs for scheduling and monitoring data pipelines), dbt (data build tool: SQL-based transformation, version control, testing, documentation).
How it is applied in practice.
Modern data stack: Fivetran/Stitch (EL connectors), Snowflake/BigQuery (warehouse), dbt (transformation), Looker/Tableau/Power BI (visualisation) — enabling analytics engineering as a discipline bridging data engineering and analytics. Data quality: Great Expectations, dbt tests — schema validation, null checks, uniqueness, referential integrity, and distribution monitoring. Real-time data: Apache Kafka (event streaming), Apache Spark Streaming, Flink (stream processing) for low-latency analytics on live data. Data governance: data catalogues (Alation, Atlan, DataHub — metadata management, data lineage, business glossary), data access controls, GDPR/CCPA data subject rights implementation, and the tension between data accessibility and privacy protection in modern organisations.