Big data refers to datasets too large, fast-moving or varied to store and process efficiently on a single traditional database or server. A retailer analysing clickstream events from millions of visitors to tune product recommendations is a typical case, using tools that spread the work across many machines.
Batch processing is handling data in scheduled groups, such as hourly or nightly runs, rather than processing each record the moment it arrives. Payroll runs, overnight warehouse loads and monthly billing are common examples, where completeness matters more than instant results.
A data lake is a large, low-cost store that holds raw data in its original format, structured or not, until someone needs to process or analyse it.
Apache Spark is an open-source engine for processing large datasets in parallel across a cluster of machines. Teams use it for batch jobs, streaming and machine learning.
A clear proven process, that delivers
You've seen our ideas - now go deeper.
Discover more insights, guides, and engineering practice.