A data lake is a large, low-cost store that holds raw data in its original format, structured or not, until someone needs to process or analyse it.
Apache Spark is an open-source engine for processing large datasets in parallel across a cluster of machines. Teams use it for batch jobs, streaming and machine learning.
A data lake keeps raw data of any format cheaply for later use; a data warehouse holds cleaned, structured data modelled for fast queries and reporting. Many teams run both, or a lakehouse that combines them.
A data lakehouse combines the cheap, flexible storage of a data lake with warehouse features such as tables, transactions and fast SQL. Both kinds of work run on the same copy of the data.
Big data refers to datasets too large, fast-moving or varied to store and process efficiently on a single traditional database or server. A retailer analysing clickstream events from millions of visitors to tune product recommendations is a typical case, using tools that spread the work across many machines.
A clear proven process, that delivers
You've seen our ideas - now go deeper.
Discover more insights, guides, and engineering practice.