The OPeNDAP and Remote NetCDF Invocation (RNI) middleware platform for Scientific Data Fusion
Stephan Zednik, Jose L Hernandez Garcia, Peter Fox, West, Patrick · AGUFM · 2008
Poster presented at the Fall 2008 AGU meeting by Stephan Zednik ([email protected]), Jose Garcia ([email protected]), Peter Fox ([email protected]), Patrick West ([email protected])Abstract: Across geosciences, there are large data holdings being made available via the DAP protocol by means of OPeNDAP software. A lot of the underlying data are in the NetCDF format. Often, each individual dataset is a combination of hundreds of individual NetCDF files. Requesting such datasets for analysis is an expensive data fusion transaction, especially as the number and size of datasets increase. We present a set of solutions that instead request needed portions of the data fused just-in-time. The fusion includes both subsetting and aggregation operations as well as analysis and data manipulation steps. We have modified the NetCDF C library for Remote NetCDF Invocation (RNI) that is, to operate on remote dataset, over HTTP, HTTPS or GSIFTP (or any) protocols, individual NetCDF Application Programming Interface (API) calls as if they were local. This Invocation model can also be applied to OPeNDAP data streams, and local files. This mechanism resembles the well known Remote Procedure Call (RPC) yet it radically differs on the binding between local and remote remote operations. We describe our current approach, implementation and benefits obtained from this approach and indicate how it aids data fusion.