Big Data Processing Frameworks — Streaming and Graphs
Stub. Stream processing frameworks (Storm, Spark Streaming) for continuous, low-latency computation over unbounded data, and graph-processing frameworks (Pregel's vertex-centric, bulk-synchronous model) for large-scale graph algorithms and machine learning.
Two generalizations of MapReduce’s batch model for workloads it fits poorly. Stream processing (Storm’s topology of spouts/bolts, Spark Streaming’s micro-batch model built on Spark’s RDD abstraction) trades batch latency for continuous, low-latency processing of unbounded data. Graph processing (Pregel) instead changes the computation model: a vertex-centric, bulk-synchronous-parallel program where each vertex exchanges messages with its neighbors each superstep, which fits graph/machine-learning algorithms (PageRank, label propagation, and similar iterative fixpoint computations) far more naturally than MapReduce’s stateless per-record transform.
Outline (TODO — flesh out each)
- Stream processing — Storm’s topology model
- Spark Streaming and the RDD abstraction
- Graph processing — Pregel’s vertex-centric, bulk-synchronous model
- Machine learning on top of graph-processing frameworks