scida: scalable analysis for scientific big data
Chris Byrohl, Dylan Nelson · The Journal of Open Source Software · 2024
scida is a Python package for reading and analyzing large scientific data sets.Data access is provided through a hierarchical dictionary-like data structure after a simple load() function.Using the dask library for scalable, parallel and out-of-core computation (Dask Development Team, 2016), all computation requests from a user session are first collected in a task graph.Arbitrary custom analysis, as well as all available dask (array) operations, can be performed.The subsequent computation is executed only upon request, on a target resource (e.g. a HPC cluster, see Figure 1).