- To instruct future scientists how to sustainably generate/ collect and use data for their research as well as for others: data science.
- To instruct future technologists how to understand and support essential data and information needs of a wide variety of producers and consumers
- For both to know tools, and requirements to properly handle data and information
- Will learn and be evaluated on the full life-cycle of data and relevant methods, technologies and best practices.
Instructor: Prof. Peter Fox, pfox at cs dot rpi dot edu
TA: Sneha Das, dass4 at rpi dot edu
Meeting times: Tuesday 0900-1150
Office Hours: Monday 1500-1600 (Winslow but sometimes Lally)
phone: x 4862 (Winslow 2120), or x 2108 (Lally 207A)
TA Office Hours: By appointment
Class Listing: CSCI/ERTH/ITWS 4350/ 6350
Class Location: Lally 102
Syllabus/ Calendar
Refer to Reading/ Assignment/ Reference list for each week (see below).
- Week 1 (Aug. 26): History of Data and Information, Data, Information, Knowledge Concepts and State-of-the-Art, Data life-cycle for Science; Data acquisition, curation, preservation, metadata Week 1 slides [Download] Week 1 in-class notes
- Week 2 (Sep. 2): Data and information acquisition (curation) and metadata/ provenance - management Week 2 slides [Download] Week 2 in-class notes
- Week 3 (Sep. 9): Data formats, metadata standards, conventions, reading and writing data and information Week 3 slides [Download] Week 3 in-class notes
- Week 4 (Sep. 16): Data Analysis I Week 4 slides [Download] Week 4 in-class notes
- Week 5 (Sep. 23): Class exercise - collecting data - individual Week 5 notes [Download]
- Week 6 (Sep. 30) : Class Presentations: present your data (4 groups) - start in Lally 102
- Week 7 (Oct. 7): Data Analysis II Week 7 slides [Download] Week 7 in-class notes and Class exercise - group project definitions - working with someone else's data
- Oct. 14 - no classes (Tuesday follows Monday schedule)
- Week 8 (Oct. 21): Intro to Data Mining for Data Science Week 8 slides [Download]
- Week 9 (Oct. 28): Data Workflow Management, Preservation and Data Stewardship Week 9 slides [Download]
- Week 10 (Nov. 4): Data Quality, Uncertainty and Bias Week 10 slides [Download]
- Week 11 (Nov. 11): Academic basis for Data Science, Data Models, Schema, Markup Languages Week 11 slides [Download]
- Week 12 (Nov. 18): Webs of Data and Data on the Web, the Deep Web, Data Infrastructures, Data Discovery, Data Citation Week 12 slides [Download]
- Nov. 25: No lecture - continue project and assignment work
- Week 13 (Dec. 2): Final Project Presentations
Reading/ Assignment/ Reference List
Class 1 Reading Assignment (choose 5-6 and at least 2-3 in depth):
- Changing Science: Chris Anderson: [1]
- Rise of the Data Scientist [2]
- Where to draw the line? [3]
- Career of the Future [4]
- What is Data Science (I) [5]
- What is Data Science (II) [6]
- Data Scientist: The Hottest Job You've Never Heard Of [7]
- What Is a Data Scientist? [8]
- Data Scientist - sexiest job of the 21st C? [9]
- An example of data science [10]
- Big Data Science [11]
- A Very Short History of Data Science [12]
- Data Science Programs on the Increase [13]
Reference
Class 2: Reading Assignment:
- MIT Libraries: [16]
- Earth Science Information Partners Data Management Workshop: [17]
- Earth Science Information Partners: Course Outline [18]
- Univ. Minnesota [19]
- Moore et al., Data Management Systems for Scientific Applications, IFIP Conference Proceedings; Vol. 188, Proceedings of the IFIP TC2/WG2.5 Working Conference on the Architecture of Scientific Software, pp. 273 – 284 (2000) [20]
- Data Management and Workflows [21]
- Metadata and Provenance Management [2]
- Provenance Management in Astronomy (case study) [23]
- Web Data Provenance for QA [24]
- W3 PROV Overview [25]
- W3 PROV Data Model [26]
- ISO Lineage Model (NOAA Environmental Data Management) [18]
Assignment 1 - Data Science 2014 Assignment 1 [Download] Preparing for Data Collection (10% of grade) due week 3 on Sept. 9, 2014
Class 3: Reading Assignment:
- Data formats: netCDF [18]
- Spatial Data Transfer Standard GIS format [19]
- Metadata resources [20]
- Metadata Encoding and Transfer Standard - METS [21]
- Open Archives Initiative - Protocol for Metadata Harvesting - OAI-PMH [22]
- Keyhole Markup Languge - KML Tutorial [23]
- Earth Science Markup Language - ESML [24]
- Climate Science Markup Language - CSML [25]
- Climate and Forecast (CF) conventions [26]
Assignment 2: Data Science 2014 Assignment 2 [Download] Presenting your Data (20% of grade) due in week 6, Sep. 30, 2014.
Class 4: Reading Assignment:
- Brief Introduction to Data Mining [27]
- Longer Introduction to Data Mining and slide sets [28]
- See the software resources list [29]
- Data Analysis - Introduction
- Example: Data Mining
Class 5: Reading Assignment: None.
Class 6: Reading Assignment: None.
Assignment 3: Data Science 2014 Assignment 3 [Download] Reformatting and Submitting Your Data (20% of grade) due in week 8, October 21, 2014
Class 7: Reading Assignment: preview government and other (science) data repositories
Some of these have no single "entry point" to their data; you can find them fairly easily by searching for the name of the agency:
- Department of Energy EIA [30]
- Humanities - Digging into Data [31]
- Environmental Protection Agency (EPA)
- US Geological Survey (and state surveys) (USGS)
- NASA Earth Observing System (EOS) and ECHO
- National Oceanic and Atmospheric Administration (NOAA) NODC, NGDC, NCDC
- Department of Energy (DoE): [32]
- National Library of Medicine (NLM): [32a]
- Cancer Grid (CaBIG)
- OneGeology
- data.gov [33]
- Find one of your own
Assignment 4: Data Science 2014 Assignment 4 [Download] Working with someone else's data (40% of grade) writeup due November 25, 2014, final presentations December 2, 2014
Class 8: Reading Assignment:
- None
Class 9: Reading Assignment:
- Introduction to Data Management [44]
- Changing software, hardware a nightmare for tracking scientific data [45] (and Parts I, II and III)
- Overview of Scientific Workflow Systems, Gil (AAAI08 Tutorial) [46]
- Comparison of workflow software products, Krasimira Stoilova ,Todor Stoilov [47]
- Scientific Workflow Systems for 21st Century, New Bottle or New Wine? Yong Zhao, Ioan Raicu, Ian Foster [48]
- NITRD report: [38]
- OCLC Sustainable Digital Preservation and Access [39]
- National Science Founcation Cyberinfrastructure Plan chapter on Data [40]
Class 10: Reading Assignment:
- Another Look at Data (Mealy 1967)!
- Identifying Content and Levels of Representation in Scientific Data (Wickett et al. 2012)
Assignment - final: Data Science 2014 Final Assignment [Download] Stewardship: Workflow construction for Preservation (10% of grade) due in week 13, December 2, 2014
Class 11: Reading Assignment:
- The Deep Web (Internet Tutorials) [50]
- Digital Image Resources on the Deep Web [51]
- Tom Heath Linked Data Tutorial (2009)[53]
- Relational Databases on the Semantic Web, Tim Berners-Lee, Design Issue Note, 1998-2009. [42]
- A Survey of Current Approaches for Mapping of Relational Databases to RDF (PDF), Satya S. Sahoo, Wolfgang Halb, Sebastian Hellmann, Kingsley Idehen, Ted Thibodeau Jr, Sören Auer, Juan Sequeda, Ahmed Ezzat, 2009-01-31. [43]
Class 12: Reading Assignment: none
Reference material (purchase not required - please ask instructor if you are interested in any of these):
- Scientific data management: [54]
- BRDI activities: [55]
- Data policy [56]
- Self-directed study (answer the quiz): [57]
- To instruct future scientists how to sustainably generate/ collect and use data for their research as well as for others: data science.
- To instruct future technologists how to understand and support essential data and information needs of a wide variety of producers and consumers
- For both to know tools, and requirements to properly handle data and information
- Will learn and be evaluated on the full life-cycle of data and relevant methods, technologies and best practices.
Learning Objective:
Successful Completion of this course will lead to the following (note additional outcomes for graduate level) – Through class lectures, practical sessions, written and oral presentation assignments and projects, students should:
- Develop and demonstrate skill in Data Collection and Data Management
- Demonstrate proficiency in Data and Information Product Generation
- Demonstrate science-driven Analysis and Presentation of Integrated Datasets from the Web
- Demonstrate the development and application of Data Models
- Convey knowledge of and apply Data and Metadata Standards and explaining Provenance
- Apply Data Life-Cycle principles and construct Data Workflows
- Develop and demonstrate skill in Data Tool Use and Evaluation
- Via written assignments with specific percentage of grade allocation provided with each assignment
- Via oral presentations with specific percentage of grade allocation provided
- Via group presentations
- Via participation in class (not to exceed 10% of total)
Assignment 1 - Preparing for Data Collection (10% of grade) due week 3, Assignment 2: Presenting your Data (20% of grade) due in week 5, Assignment 3: Curating your Data (20% of grade) due in week 8, Assignment 4: Working with someone else's data (40% of grade) due in week 12/13, Assignment - final: Stewardship: Workflow construction for Preservation (10% of grade) due in week 13
Late submission policy: first time with valid reason – no penalty, otherwise 20% of score deducted each late day
Course: Data Science
Date: to