SPCDIS Meeting Notes May 11, 2012

Printer-friendly version

Welcome to TitanPad!

General Meeting Information


  • Status and Action Items


  • Patrick
  • Stephan
  • Naveen
  • Sapan

Past Action Items

Action Items

  • See SPCDIS trac tickets
  • PW: sftp issues from draco to orion
  • PW: smtp on leo vserver and aquarius vserver
  • PW: https put of files instead of sftp or scp
  • PW: converter working group
    • move internal page content to this working group
  • PW: clean up working group pages
    • there are none for COMP
  • JM: latest idl code in svn
  • JM: latest converter code in svn
  • Sapan: Merge James Code with the converter and update in subversion
  • Sapan: Run draco code


Ticket 878

  • https://scm.escience.rpi.edu/trac/ticket/878
  • Everything is in subversion except what Sapan has
  • By end of today (May 11, 2012)

Ticket 887

  • https://scm.escience.rpi.edu/trac/ticket/887

Ticket 1095

  • https://scm.escience.rpi.edu/trac/ticket/1095

Ticket 876

  • https://scm.escience.rpi.edu/trac/ticket/876

Ticket 877

  • https://scm.escience.rpi.edu/trac/ticket/877


Pellint presentation in provenance meeting - May 25

This did not happen on May 25 - could we get a link to slides for the presentation?

May 15, 2012


  • COMP IDL Pipeline is working
  • Logging system is working
  • converter is working
  • RDF is being generated

What's left to do

  • start the cron using the batch script James set up
  • We do not currently know where the raw data is
  • update script to accomodate scenario when no raw data is available

Pipeline is run manually by Leonard
Our Pipeline cron will run nightly, 11pm Mountain

Any complete day's log files should be handled by the crons that move files from draco to leo


  • Still having problems transferring problems between Draco and Orion
  • Wondering if we should got directly from Draco to Aquarius (Could use a HTTPS PUT)
    • Patrick will look into HTTPS and supporting PUT (via apache?) for aquarius


  • James went through code and added comments, additional parameters
  • Commented out processing of fits headers? (FUTURE)
    • possibly implement system on Draco to read FITS files and generate dataset-level RDF from record header information
    • open issue - associating provenance RDF with FITS metadata RDF
  • Format of the logs are in the code, comments
    • Possible to generate a schema for csv files?
    • James says yes, would be in a separate csv file from the actual data
    • This needs to be put into the working group page for converter
  • Exception handling
    • If there is an error in the log file then the rdf should not be generated
    • would be nice to be able to email when there is a problem
    • ISSUE - need to setup SMTP server on leo/aquarius (whichever we end up using)

SPARQL to generate xsl transformations

  • RDF is based on OPM, STOM
  • Could create conversion to PML
  • ACTION ITEM - load new RDF into SPCDIS triplestore (virtuoso on leo vserver)
  • ACTION ITEM - virtuoso instalation on leo vserver (same as above?)