Developing service-oriented applications in a grid environment - Experiences using the OPeNDAP back-end-server

Printer-friendly version

Concepts:Data Science


The Open-source Project for a Network Data Access Protocol (OPeNDAP) software framework has evolved over the last 10 years to become a robust, high performance, service oriented architecture for the access and transport of scientific data from a broad variety of disciplines, over the Internet. Starting with version 4.0 of the server release, Hyrax, has at its core, the Back-End Server (BES). The BES offers ease of programming, extensibility and reliability allowing added functionality especially related to high- performance data Grids, that goes beyond its original goal of distributed data access. We present the fundamentals of the BES server as a component based architecture, as well as our experiences using the BES to distribute data processing in Grid-oriented intensive parallel computational tasks.


DateCreated ByLink
December 17, 2011
Patrick WestDownload
December 17, 2011
Patrick WestDownload

Related Projects:

Earth System Grid - Center for Enabling Technologies (ESG-CET)
Principal Investigator: Peter Fox and Dean Williams
Description: The Earth System Grid (ESG) integrates supercomputers with large-scale data and analysis servers located at numerous national labs and research centers to create a powerful environment for next generation climate research. Access to ESG is provided through a system of federated Data Gateways, that collectively allow access to massive data and services for Climate Global and Regional Models, IPCC research, and analysis and visualization software. The Earth System Grid - Center for Enabling Technologies (ESG-CET) is funded by the U.S. Department of Energy as part of the SciDAC (Scientific Discovery through Advanced Computing) program. At the heart of the Earth System Grid is the need for access and transport of various data products. These products are produced and consumed in many and varied parts of the ESG architecture, including on data nodes, gateway nodes, clients and federated sites. The RPI effort involves secure, high-performance access and transport of data within the ESG-CET framework.
Description: Tasks for various TWC projects related to data access and the OPeNDAP software products.

Related Research Areas:

Data Science
Lead Professor: Peter Fox
Description: Science has fully entered a new mode of operation. Data science is advancing inductive conduct of science driven by the greater volumes, complexity and heterogeneity of data being made available over the Internet. Data science combines of aspects of data management, library science, computer science, and physical science using supporting cyberinfrastructure and information technology. As such it is changing the way all of these disciplines do both their individual and collaborative work.

Data science is helping scienists face new global problems of a magnitude, complexity and interdisciplinary nature whose progress is presently limited by lack of available tools and a fully trained and agile workforce.

At present, there is a lack formal training in the key cognitive and skill areas that would enable graduates to become key participants in escience collaborations. The need is to teach key methodologies in application areas based on real research experience and build a skill-set.

At the heart of this new way of doing science, especially experimental and observational science but also increasingly computational science, is the generation of data.