The open connectome project data cluster
Randal C. Burns, Kunal Lillaney, Daniel Raimund Berger, Logan Grosenick, Karl A. Deisseroth, R. Clay Reid, William Gray Roncal, Priya Manavalan, Davi D. Bock, Narayanan Kasthuri, Michael Kazhdan, Stephen J Smith, Dean M. Kleissas, Eric Perlman, Kwanghun Chung, Nicholas Collins Weiler, Jeff William Lichtman, Alexander S. Szalay, Joshua T Vogelstein, R. Jacob Vogelstein · 2013
- neural connectivity maps of the brain-using the parallel execution of computer vision algorithms on high-performance compute clusters. These services and open-science data sets are publicly available at openconnecto.me. The system design inherits much from NoSQL scale-out and data-intensive computing architectures. We distribute data to cluster nodes by partitioning a spatial index. We direct I/O to different systems-reads to parallel disk arrays and writes to solid-state storage-to avoid I/O interference and maximize throughput. All programming interfaces are RESTful Web services, which are simple and stateless, improving scalability and usability. We include a performance evaluation of the production system, highlighting the effec-tiveness of spatial data organization.