The Remote NetCDF Invocation (RNI) middleware platform. Making Scientific Datasets Available for Ubiquitous Computing

Printer-friendly version

Presented at the AGU Fall Meeting 2007


Large holding of NetCDF data, such as in the Earth System Grid (ESG) or the Community Spectro-Polarimetric Analysis Center (CSAC) are vast repositories of data, making it if not impossible, but impractical for users to download and replicate the complete database. Furthermore, each individual dataset is a combination of hundreds of individual NetCDF files. Therefore requesting such dataset for analysis is an expensive transaction for individuals seeking ubiquitous computing. Since the current state of networks can provide for access to individual pieces of the dataset with enough reliability and speed, we seek a solution that will avoid the bulk download of the dataset required a priori, and will instead request needed portions of the dataset just-in-time. In order to achieve this, we modify the NetCDF C library to execute Remote NetCDF Invocation (RNI), that is, to operate on remote dataset, over HTTPS and gsiFTP protocols, individual NetCDF Application Programming Interface (API) calls as if they were local. This mechanism resembles the well known Remote Procedure Call (RPC) yet it radically differs on the binding between local and remote operations. Our design is based on the extensibility mechanism provided by the popular OPeNDAP Back-End Server (BES) middleware platform with Globus GridFTP and Apache modules acting as the proxy transport mechanism (binding) between the local and remote transactions. This paper describes the architecture as well as how we address the technical challenges for the complete system.


DateCreated ByLink
July 13, 2011
Stephan ZednikDownload

Related Projects:

Earth System Grid - Center for Enabling Technologies (ESG-CET)
Principal Investigator: Peter Fox and Dean Williams
Description: The Earth System Grid (ESG) integrates supercomputers with large-scale data and analysis servers located at numerous national labs and research centers to create a powerful environment for next generation climate research. Access to ESG is provided through a system of federated Data Gateways, that collectively allow access to massive data and services for Climate Global and Regional Models, IPCC research, and analysis and visualization software. The Earth System Grid - Center for Enabling Technologies (ESG-CET) is funded by the U.S. Department of Energy as part of the SciDAC (Scientific Discovery through Advanced Computing) program. At the heart of the Earth System Grid is the need for access and transport of various data products. These products are produced and consumed in many and varied parts of the ESG architecture, including on data nodes, gateway nodes, clients and federated sites. The RPI effort involves secure, high-performance access and transport of data within the ESG-CET framework.

Related Research Areas:

Data Science
Lead Professor: Peter Fox
Description: Science has fully entered a new mode of operation. Data science is advancing inductive conduct of science driven by the greater volumes, complexity and heterogeneity of data being made available over the Internet. Data science combines of aspects of data management, library science, computer science, and physical science using supporting cyberinfrastructure and information technology. As such it is changing the way all of these disciplines do both their individual and collaborative work.

Data science is helping scienists face new global problems of a magnitude, complexity and interdisciplinary nature whose progress is presently limited by lack of available tools and a fully trained and agile workforce.

At present, there is a lack formal training in the key cognitive and skill areas that would enable graduates to become key participants in escience collaborations. The need is to teach key methodologies in application areas based on real research experience and build a skill-set.

At the heart of this new way of doing science, especially experimental and observational science but also increasingly computational science, is the generation of data.

Lead Professor: Peter Fox
Description: In the last 2-3 years, Informatics has attained greater visibility across a broad range of disciplines, especially in light of great successes in bio- and biomedical-informatics and significant challenges in the explosion of data and information resources. Xinformatics is intended to provide both the common informatics knowledge as well as how it is implemented in specific disciplines, e.g. X=astro, geo, chem, etc. Informatics' theoretical basis arises from information science, cognitive science, social science, library science as well as computer science. As such, it aggregates these studies and adds both the practice of information processing, and the engineering of information systems.
Concepts: ,