Batch processing is handling data in scheduled groups, such as hourly or nightly runs, rather than processing each record the moment it arrives. Payroll runs, overnight warehouse loads and monthly billing are common examples, where completeness matters more than instant results.
Data integration is combining data from different sources into a consistent, unified view. Sources include applications, databases, files and partners, and the result serves people and systems alike.
A data pipeline is an automated series of steps that moves data from where it is created to where it is stored, analysed or used. Steps along the way may extract, clean, reshape or enrich it.
ETL (extract, transform, load) cleans and reshapes data before loading it into a target system. ELT (extract, load, transform) loads raw data first and does the transformation inside the target, usually a warehouse.
Stream processing is computing on data continuously as each event arrives, so results update within seconds. It is the engine behind real-time data streaming, replacing waits for scheduled batch runs.
Apache Spark is an open-source engine for processing large datasets in parallel across a cluster of machines. Teams use it for batch jobs, streaming and machine learning.
Big data refers to datasets too large, fast-moving or varied to store and process efficiently on a single traditional database or server. A retailer analysing clickstream events from millions of visitors to tune product recommendations is a typical case, using tools that spread the work across many machines.
A clear proven process, that delivers
You've seen our ideas - now go deeper.
Discover more insights, guides, and engineering practice.