Dask: Parallel Computation with Blocked algorithms and Task Scheduling

Rocklin, Matthew

doi:10.25080/majora-7b98e3ed-013

Public

Dask: Parallel Computation with Blocked algorithms and Task Scheduling

Published in Proceedings of the Python in Science Conferences • Jan 1, 2015

Authors:

Matthew Rocklin

Abstract

Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more ...

Subject

Computer science

Parallel computing

Computation

Research Assistant

AI chat, annotations, notes & similar papers

Finding related papers...

Discussions

(0)

No comments yet

Be the first to share your thoughts!