SESF Working Group: Ontology Development and Evaluation

Printer-friendly version

Documents

The slideshare presentation from 2008 is here
This has 3 examples of how the evaluation is applied.

Activities

Working Group Meetings

Evaluation Approach

The aim is to examine SeSF ontologies for terms, concepts and relationships. One of the intended outcomes of the evaluation is to determine readiness for reuse and help ensure that terms used are well-used within the community.

SeSF candidate sources are:

Adoption of Gruber's approach http://tomgruber.org/writing/onto-design.pdf, Toward Principles for the Design of Ontologies Used for Knowledge Sharing (1993)

  1. Clarity:An ontology should effectively communicate the intended meaning of defined terms. Definitions should be objective. While the motivation for defining a concept might arise from social situations or computational requirements, the definition should be independent of social or computational context. Formalism is a means to this end. When a definition can be stated in logical axioms, it should be. Where possible, a complete definition (a predicate defined by necessary and sufficient conditions) is preferred over a partial definition (defined by only necessary or sufficient conditions). All definitions should be documented with natural language.
  2. Coherence: An ontology should be coherent: that is, it should sanction inferences that are consistent with the definitions. At the least, the defining axioms should be logically consistent. Coherence should also apply to the concepts that are defined informally, such as those described in natural language documentation and examples. If a sentence that can be inferred from the axioms contradicts a definition or example given informally, then the ontology is incoherent.
  3. Extendibility: An ontology should be designed to anticipate the uses of the shared vocabulary. It should offer a conceptual foundation for a range of anticipated tasks, and the representation should be crafted so that one can extend and specialize the ontology monotonically. In other words, one should be able to define new terms for special uses based on the existing vocabulary, in a way that does not require the revision of the existing definitions.
  4. Minimal encoding bias: The conceptualization should be specified at the knowledge level without depending on a particular symbol-level encoding. An encoding bias results when a representation choices are made purely for the convenience of notation or implementation. Encoding bias should be minimized, because knowledge sharing agents may be implemented in different representation systems and styles of representation.
  5. Minimal ontological commitment: An ontology should require the minimal ontological commitment sufficient to support the intended knowledge sharing activities. An ontology should make as few claims as possible about the world being modeled, allowing the parties committed to the ontology freedom to specialize and instantiate the ontology as needed. Since ontological commitment is based on consistent use of vocabulary, ontological commitment can be minimized by specifying the weakest theory (allowing the most models) and defining only those terms that are essential to the communication of knowledge consistent with that theory.

Added columns (Peter Fox and Chris Lynnes)

  1. CntxtRelev - Contextual Relevance to SSIII, context can be assigned by number and degree to which key concepts/ terms are defined in the SSIII context
  2. Maturity - Maturity in relation to SSIII need and perhaps in comparison to other vocabularies, number of years, number of users, applications, revision level, community convention or standard, national or international standard, etc.
  3. IntendUse - Degree to which SSIII use is aligned or not with original intended use, an estimate of how 'ready' the vocabulary is and some indication of degree of alignment
  4. Fit4Use - Estimate of current fitness for purpose for SSIII (before modification or extension), especially related to accuracy of vocabulary and minimal alteration, extension required.

Evaluation

  • 1 to 9 scale (a TRL-like scale), notes below

How to apply the principles to derive a VRL

  • VRL 1-3 - little to no implementation in application or service
  • VRL 4-6 - demonstrated application
  • VRL 7-9 - widely available and used via application or service

Stage One

Cols 2-8, 10 are VRL, col 9 is text.

Ontology Clarity Coherence Extendibility MinEncodingBias MinOntCommit CntxtRelev Maturity IntendUse Fit4Use
VSTO 1.0 4 5 6 6 4 6 6 Overall framework into which domain-specific terms will be fitted... TBD. 7
VSTO 2.0 txt
spacetime 1.0 txt

Stage Two

CRITERIA FOR EVALUATING USE ASPECTS OF Ontologies

Ref: (based on recent experience of NSF+NOAA folks jointly designing cruise-level metadata for R2R program; draws from MMI Guides)

          

  • (*= potential deal breaker/minimum requirement)
  • Is it available online, programmatically accessible? In modern RDF/XML format? *
  • In a stable namespace (HTTP style URI) *
  • Formally versioned? (i.e. earlier versions are programmatically available)
  • Is it a Discovery level vocabulary or a Use vocabulary?
  • Does it link to upper and/or lower level vocabs (synonyms/aliases and larger context)
  • Does the vocabulary discretize types vs. instances?
  • Is there a governance structure? (Simple/individual email request? Formal committee/review structure?) + Is there a well-documented system for suggesting new terms and have them adopted for the larger community? *
  • Is it flat (relatively simpler to use) or hierarchical?
  • Do mappings exist to other vocabularies? (Are they maintained?)
  • Do terms have definitions? (See above criteria)
  • Is there higher level documentation/cookbook/userguide? (See above criteria)
  • Is there a science community that has already adopted it?

References

http://marinemetadata.org/community/teams/presence/reports/iodejcommcocoreport

there is a link from that report summary to the presentation by John Graybeal that includes an overview of metadata, vocabularies and ontologies http://marinemetadata.org/files/mmi/OntologiesforIODE.pdf

MMI has a Guide for Choosing and Implementing Established Controlled Vocabularies http://www.marinemetadata.org/guides/vocabs/cvchooseimplement

There is additional information available from Marine Metadata Interoperability (MMI) Project Guides:
John Graybeal's IODE presentation from January 2008 available from the MMI site. The MMI site offers a summary of the IODE/JCOMM Forum on Oceanographic Data Management and Exchange Standards Meeting that was held 21-25 January, 2008 at the IOC Project Office for IODE in Oostende, Belgium