Naiad: Incremental And Iterative Data-Parallel Computation


Is there a way to shave years off of the trial and error implementing Agile?
Find Out Now.


Naiad is a distributed system that supports efficient incremental and iterative data-parallel computation. It extends standard batch data-parallel processing models like MapReduce, Hadoop and Dryad/DryadLINQ to support efficient incremental updates in the manner of a stream processing system, while at the same time enabling arbitrarily nested fixed-point iteration. In this talk I describe the design and implementation of Naiad, and show that complex analyses like strongly connected components, requiring multiple nested loops, run on streams of input with subsecond update times.