OPeNDAP Hyrax: An extensible data access framework within the Earth System Grid Federation

Printer-friendly version

Presented at the AGU Fall Meeting 2011

Abstract:

There is an ever-growing need for the researcher to not only have access to research data, but to also execute aggregation and server-side analysis functionality against this data remotely, and to have this new data product available for further analysis and manipulation. This reduces the burden on the researcher of retrieving and maintaining large data files and streamlines common, repetitive pre-processing tasks.  It also helps to standardize common pre-processing tasks by providing them as a service, maintained and tested by the data publisher. OPeNDAP Hyrax is a multi-tier software framework and data access server that implements not only the DAP (Data Access Protocol) specification, but is also an extensible, modular framework that provides the data provider and researcher with the ability to perform on-demand server-side analysis, aggregation and manipulation of the data. The framework supports the installation of dynamically loaded modules that may be developed to add support for new data formats, data product responses, or server-side analysis operations. This presentation covers the use of OPeNDAP Hyrax in the Earth System Grid Federation for access to climate research models and observations in order to meet the needs of the World Climate Research Programme (WCRP).

History

DateCreated ByLink
December 9, 2011
14:46:11
Patrick WestDownload

Related Projects:

Earth System Grid - Center for Enabling Technologies (ESG-CET)
Principal Investigator: Peter Fox and Dean Williams
Description: The Earth System Grid (ESG) integrates supercomputers with large-scale data and analysis servers located at numerous national labs and research centers to create a powerful environment for next generation climate research. Access to ESG is provided through a system of federated Data Gateways, that collectively allow access to massive data and services for Climate Global and Regional Models, IPCC research, and analysis and visualization software. The Earth System Grid - Center for Enabling Technologies (ESG-CET) is funded by the U.S. Department of Energy as part of the SciDAC (Scientific Discovery through Advanced Computing) program. At the heart of the Earth System Grid is the need for access and transport of various data products. These products are produced and consumed in many and varied parts of the ESG architecture, including on data nodes, gateway nodes, clients and federated sites. The RPI effort involves secure, high-performance access and transport of data within the ESG-CET framework.
SeSF Project LogoSemantic eScience Framework (SeSF)
Principal Investigator: Peter Fox
Co Investigator: Jim Hendler and Deborah L. McGuinness
Description: Over the past few years, semantic technologies have evolved and new tools are appearing. Part of the effort in this project will be to accommodate these advances in the new framework and lay out a sustainable software path for the (certain) technical advances. In addition to a generalization of the current data science interface, we will include an upper-level interface suitable for use by clearinghouses, and/or educational portals, digital libraries, and other disciplines.

Related Research Areas:

Data Frameworks
Lead Professor: Peter Fox
Description: None.
Concepts: eScience
Data Science
Lead Professor: Peter Fox
Description: Science has fully entered a new mode of operation. Data science is advancing inductive conduct of science driven by the greater volumes, complexity and heterogeneity of data being made available over the Internet. Data science combines of aspects of data management, library science, computer science, and physical science using supporting cyberinfrastructure and information technology. As such it is changing the way all of these disciplines do both their individual and collaborative work.

Data science is helping scienists face new global problems of a magnitude, complexity and interdisciplinary nature whose progress is presently limited by lack of available tools and a fully trained and agile workforce.

At present, there is a lack formal training in the key cognitive and skill areas that would enable graduates to become key participants in escience collaborations. The need is to teach key methodologies in application areas based on real research experience and build a skill-set.

At the heart of this new way of doing science, especially experimental and observational science but also increasingly computational science, is the generation of data.

Concepts: eScience
Semantic eScience
Lead Professor: Peter Fox
Description:
Science has fully entered a new mode of operation. E-science, defined as a combination of science, informatics, computer science, cyberinfrastructure and information technology is changing the way all of these disciplines do both their individual and collaborative work.
As semantic technologies have been gaining momentum in various e-Science areas (for example, W3C's new interest group for semantic web health care and life science), it is important to offer semantic-based methodologies, tools, middleware to facilitate scientific knowledge modeling, logical-based hypothesis checking, semantic data integration and application composition, integrated knowledge discovery and data analyzing for different e-Science applications.
Partially influenced by the Artificial Intelligence community, the Semantic Web researchers have largely focused on formal aspects of semantic representation languages or general-purpose semantic application development, with inadequate consideration of requirements from specific science areas. On the other hand, general science researchers are growing ever more dependent on the web, but they have no coherent agenda for exploring the emerging trends on the semantic web technologies. It urgently requires the development of a multi-disciplinary field to foster the growth and development of e-Science applications based on the semantic technologies and related knowledge-based approaches.

Concepts: eScience
X-informatics
Lead Professor: Peter Fox
Description: In the last 2-3 years, Informatics has attained greater visibility across a broad range of disciplines, especially in light of great successes in bio- and biomedical-informatics and significant challenges in the explosion of data and information resources. Xinformatics is intended to provide both the common informatics knowledge as well as how it is implemented in specific disciplines, e.g. X=astro, geo, chem, etc. Informatics' theoretical basis arises from information science, cognitive science, social science, library science as well as computer science. As such, it aggregates these studies and adds both the practice of information processing, and the engineering of information systems.
Concepts: , eScience