Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Matthew Rocklin · Proceedings of the Python in Science Conferences · 2015
Dask enables parallel and out-of-core computation.We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone.We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.