Preparing the poster helped me to rethink the water quality portal. Thank Evan for his poster for ISWC 2011, which was a very good starting point for my AGU poster! Having the poster session was even more fun and rewarding! I presented the portal to researchers from various fields and countries. Most of the people I talked to said that the portal is a nice and interesting project. Some researchers gave me very helpful comments like:
1. bring in crowd sourcing, e.g. let users report problem
2. help farmers to identify polluted wells
3. we should have an approach for pulling new data from USGS and EPA, e.g. some subscription
4. regulation management for users (insert/upload/delete)
5. consider allergic as use cases, possible conditions for allergic alert: wind + time, a combination of pollutants

I went to several talks during AGU and got to know the cool projects that researchers from different organizations (EPA, Standford, UMD, Google, NASA) have been doing. It was impressive to see that how computer science has been widely and deeply used in geophysical research. And I felt that scientists from geophysical fields expect more cooperation with people from computer science.

I went to the exhibits twice and spent quite some time there. I used the wired network provided by Google to do my assignments for the AI course. I also listened the talk about Google earth engine, a very cool platform for geophysical scientists!

Attending a conference as huge as AGU indeed requires some energy but after all this is worthwhile.

Tips about travel reimbursement that Carol gave me today:

1. Keep boarding pass to show if you sit in economic class
2. check out at the hotel and get the folio to show that you actually
stayed at the hotel for how many nights
3. get itemized receipts at restaurants

Thank Carol!

AGU 2011 Fall Meeting was the first time I went to an academic conference. I was very excited when I learned I’ve got such an opportunity. My goals were to present our poster, to check out what it is like in such a conference, and to have an idea about what other people are doing in the Informatics area.

My poster was about the work with Eric Rozell on the temporal metadata modeling in VSTO. I presented its motivation and methodology to several people, and it certainly drew some interest. Our approach has been viewed as an effective way to deal with a large amount of data and to improve reasoning and searching capacities. It was suggested that a similar technique (in the sense of including the temporal range for a dataset to a granularity of days using time:DateTimeInterval) has been used for data indexing in relational databases in NASA. In terms of the presentation, I think putting our posters, publications, and demos into flash drives and distributing them to people was a very good idea. It greatly helped the interested audiences to understand our work more afterwards.

There were many other interesting work across a couple of sessions. For example, Nicholas Del Rios etc. from University of Texas at El Paso presented a semantic and provenance aware visualization framework (VisKo) that links data with visualization processes. It has been used to visualize data on behalf of Giovanni. It is able to capture data processing provenance and visualization provenance in PML. Besides posters, I also went several talks from different sessions. Though I failed to connect most of them to my research work, I thought it was nice to hear about what other people have been working on.

Another output for me was to meet people in the Earth Science and Informatics areas. Although the names I could remember were limited, what I saw was they are a group of people who show enthusiasm about their work. They believe in what they are doing and have the confidence in the accomplishment their work will bring. I really look forward to working with many of them.

To sum up, this was a great experience for me in the beginning stage of my Ph.D. career.  Next time I will try to meet and talk to more people, and get more feedback about my own work.

I had the pleasure to work with Greg on the Multi-Sensor Data Synergy Advisor (MDSA) a prototype semantic extension to the already successful Giovanni online data anaylsis tool.  Giovanni has proven to be a successful tool for reducing the technical barriers in science data processing, analysis, and visualization and information provided through Giovanni has played a role in over 400 science publications to date.  With MDSA, Greg intended to show how Giovanni could be instrumented to provide provenance, quality, and expert knowledge about data to interested users.  Greg was extremely enthusiastic about the potential of semantic technologies to power these enhancements; ontologies to describe concepts important to data generation and use and rules to expose and explain scenarios that may lead to misunderstood analysis results.  I will always admire Greg’s enthusiasm for what we had been able to accomplish, and what we would be able to accomplish in future projects.  Greg clearly saw what were were doing as a means to empower scientists, a noble goal if ever there was.

I am thankful for having had the opportunity to have worked with Greg, and incredibly sad he was not able to see the fruition of this work.

You will be missed friend.

Science product quality is hard to define, characterize, and act upon. Product quality reflects a comparison against standard products of a similiar kind, but it is also reflective of the fitness-for-use of the product for the end-user. Users weigh quality characteristics (e.g. accuracy, completeness, coverage, consistency, representativeness) based on their intended use for the data, and therefore quality of a product can be different based on different users’ needs and interests.  Despite the subjective nature of quality assertions, and their sensitivity to users fitness-for-use, most quality information is provided by the product producer and the subjective criteria used to determine quality is opaque, if available at all.

If users are given product quality information at all, this information usually comes in one of two forms:

• tech reports where extensive statistical analysis is reported on very specific characteristics of the product
• in the form of subjective and unexplained statements such as ‘good’, ‘marginal’, ‘bad’.

This is either information overload that is not easy for the user to quickly assess or a near lack of the type of information that a user needs to make their own subjective quality assessment.

Is there a smilar scenario in common-day life where users are presented with quality information that they can readily understand and act upon?

There is, and you see it every day in the supermarket.

a common application of information used to make subjective quality assessments

Nutrition Facts labels provide nutrition per serving information (e.g. amount of Total Fat, Total Carbohydrates, Protein) and how the the listed amounts per serving compare to a perspective daily diet.

The comparison to a standard 2,000 calorie diet provides the user with a simple assessment tool for the usefulness of food item in their unique diet. Quality assertions, such as whether this food is ‘good’, or ‘bad’ for the consumer’s diet are left to the consumer – but are relatively easy to make with the available information.

A ‘quality facts’ label for a scientific data product, showing computed values for community-recognized quality indicators, would go a long way towards enabling a nutrition label-like presentation of quality that is easy for science users to consume and act upon.

an early mockup of a presentation of quality information for a science data product

We have begun working on mockups of what such a presentation of quality could look like, and have constructed a basic quality model that would allow us to express in RDF the information that would be used to construct a quality facts label.

Our quality model primer presents our high-level quality model and its application to an aerosol satellite data product in detail.

Our poster presentation was a hit at AGU, where we received a great deal of positive feedback on it.  This nutrition label-like presentation is immediately familiar, and supports the metaphor of science users ‘shopping’ for the best data product to fit their needs.

We still have a long way to go on developing our presentation, but the feedback from discussions at AGU tells me that our message resonated with our intended audience.

Addressing the first type of feedback is not difficult. I want everyone to be able to deploy an S2S interface for their data. However, I often have to hold myself back, because I know that the software is not to a point that it can be easily reused without a significant amount of hand-holding on my part. The basic problems are documentation and complexity of installation. While the documentation problem can be easily fixed, the problem of installation will remain until the S2S back-end architecture is updated. The back-end architecture depends on a triplestore deployed on one of TWC’s machines for indexing metadata about S2S services. I plan to move the back-end to a linked data crawler approach next spring, removing the dependencies on TWC triplestores and enabling wider installation.

The second type of feedback was more interesting to address. It’s always good to hear constructive criticism about a project. The argument was, because S2S uses its own vocabulary to describe, i.a., Web services, “widgets”, and parameters, it is not interoperable because existing tools will not understand those vocabularies. I have two primary defenses to that. The first is that S2S allows you to define virtually any term so that they can be used by old tools and new tools. For instance, S2S allows you to define each of the OpenSearch vocabulary terms including “results”, “searchTerms”, “startIndex”, and “count”. Each of these have in fact been implemented by our OpenSearch services for S2S, so when a traditional OpenSearch tool finds an S2S OpenSearch service, it should still be able to use it. The second defense is, if you do not agree with the S2S vocabulary, find a vocabulary with as much tool support as S2S for developing faceted browse or advanced search interfaces. At the time the S2S project started, we found no vocabularies for defining the “extensibility” aspects of OpenSearch (i.e., the fact that URIs can be used in place of any of the OpenSearch terms). So we did define those vocabularies, and we specifically designed them for S2S’s purpose. I’d be happy to collaborate with anyone who has a broader or different purpose from S2S to extend the vocabulary to their needs, or map S2S terms to their terms.

The last type of feedback was why the S2S framework has so much complexity. I’m not sure there is one good response to that inquiry, I think the complexity is useful when you look at the big picture for S2S. For one, S2S was never explicitly designed to be a framework for faceted browsing interfaces. Rather, it was designed to develop configurable user interfaces, with a heavy emphasis on reusability of user interface components. Faceted browsing became the focus because we had two use cases that were best implemented with faceted browse. Another complexity issue was in the number of queries made by an S2S faceted browser compared to something like Apache Solr. For instance, a browser with 6 facets could potentially require 7 queries to populate the browser with data in S2S (1 per facet plus 1 for the results). In Solr, there is a single query that can return all facets and facet values. The design decision in S2S was that a data manager may need to query a remote source to determine what its facet values are. Alternatively, the data manager may have a single input that it does not wish to facet (say, for performance reasons). In either case, we designed S2S to be as flexible as possible, which in some cases means it takes a little more effort to set up when compared to something more rigid, such as Apache Solr.

First, a bit of context (from the ORGPedia site):

ORGPedia: The Open Organizational Data Project, led by NYLS Professor and former United State Deputy CTO Beth Noveck (Project Lead) and TWC Senior Constellation Professor Jim Hendler (Tech Lead) explores how to create the legal, policy and technology framework for a data exchange to facilitate efficient comparison of organizational data across regulatory schemes as well as public reuse and annotation of that data. By designing a universal exchange rather than a new numbering scheme, OrgPedia aims to achieve goals like improving corporate transparency and efficiency, organizational performance, risk management, and data-driven regulatory policy–without having to wait until legislation is enacted for a single, legal entity identifier.

To date, TWC’s contribution to ORGPedia has been to aggregate data from a variety of sources, develop an experimental site to serve as a platform for integrating the data and prototyping ORGPedia concepts, and develop data visualizations and mashups that demonstrate the potential of an open system of canonical identifiers for corporate entities. Led by TWC Ph.D. student Xian Li, undergrads Alexei Bulazel and Bharath Santosh teamed together to create interesting visualizations based on the data aggregated.

Bharath first describes a visualization he created that allows users to analyze various financial properties of the financial sectors in the US using our aggregated data:

The visualization itself is through Google Motion Charts which is in Google’s Visualization API. It is an interactive multidimensional graph of a dataset of sectors and the mean of various financial properties across the sector’s companies. The data shown above is represented is represented in millions USD. The Motion Chart allows for really neat temporal analysis of data in various forms. Clicking the play button shows the change in properties from 2008 to 2011. There are also three different styles you can view the data: bubbles(shown above), bar charts, line graphs. These can be switched in the top corner.

The dataset behind the visualization was created in R. I made a sparql query that would access Orgpedia’s datasets and pull out sector of the US and the companies and their stock tickers within the sectors. Then I took these companies and pulled in their income statements from Google Finance and went through each sector and averaged various properties from the sector’s companies’ financial statements. The data manipulation in R took some getting used to, but now its very easy for me to transform data frames, matrices, and other objects in R. After the dataset was created and cleaned for non-existent values its just defining properties of the Motion Chart and running it. It generates a html file with the graph and data represented in javascript. All the data processing and manipulation takes around 15 minutes mostly due to the large amount of data to be downloaded.

Bharath then goes on to describe the compelling visualization he and Alexei created of the “social network” of corporate board members:

…The visualization utilizes data from LittleSis.org and gathers data about board members of various companies in the US and shows the members in a force graph that shows which board members are on multiple boards (Board Members Network):

The graph visualization is done using the D3 visualization toolkit’s Forced Graph. Each node represents a board member. The clustered colored nodes are a group of members on the same board. The multicolored nodes represent board members that are on multiple boards. Mousing over a node shows you their name and the companies they work for. Clicking a node takes you to their LittleSis.org page. The graph shows many interesting relationships between various companies and board members. Especially Steven S Reinemund who resides on 5 different boards.

On his blog, Alexei provides additional detail about the work they did to prepare the data for the visualization:

The project involved creating an interactive graph visualization of connections between members of corporate boards (the final product can be found here). Given a list of a few hundred stock tickers and access to the LittleSis API, the goal was to ultimately produce a JSON file of board members that could be use by the D3.js force-directed graph framework. I started by looking up each ticker symbol, yielding a JSON file with a unique ID number for each company. My script then queried the API for actual company page associated with that ID and stored the names, company associations, and URIs of each board member. Finally, a JSON file for the D3.js graph was output describing the ~2800 board members and the links between each of them.

While I had used Python a bit for command line scripting, I hadn’t really dug into it before this project. The work gave me a better taste for the language and its capabilities. I made extensive use of the “urllib” library for accessing web content, and worked with opening up the data in JSON files. Bharath helped me with the syntax of program and some of the graph construction. While I was aware of Python’s reputation for ease of use and high level abstraction, working with it let me experience this abstraction first hand, I was very impressed. The ease with which complex multistep operations could be completed let me focus more on the flow of the data through the process rather than the specifics of handling it. The project also gave me a bit more hands on experience with JSON.

The reader is encouraged to read both Alexei‘s and Bharath‘s blogs for more details on these great contributions by a couple of our TWC undergrads!

Before I continue, though, allow me to disclaim a bit. I am not a representative of the entire semantic web community, although I do consider myself a member of it. Additionally, I am not officially associated with the W3C. I write this blog entry simply in the capacity of a semantic web enthusiast (henceforth, semwebber), and not even as a member of the Tetherless World Constellation. I invite, nay, urge other semwebbers to contribute comments to this blog post in any capacity (agree, disagree, amend, etc.).

1. “One ontology to rule them all”

To my knowledge, nobody has ever claimed that there should be “one ontology to rule them all.” Instead, what is regularly promoted is ontology reuse and/or integration. For example, the FOAF ontology is widely used in the semantic web to describe persons; why create your own ontology when you can reuse a well-established one? Integration of ontologies allows for conciliation of perspectives, causing data that use these ontologies to become meaningfully related. Admittedly, there are some rather large, comprehensive ontologies out there, and there are some very popular and pervasive ones, too. However, there is no standard or recommendation that requires publishers of RDF data to comply with any particular ontology. You could even ignore the RDF vocabulary if you so please (yes, even rdf:type).

The primary purpose of an ontology (in my view) is to attach explicit semantics to your data. Just as the participant had stated (although he meant it in contrast to the semantic web), there are many ontologies. They compete in the ecosystem of the World Wide Web and evolve accordingly (or become extinct).

2. “Triples all the way down”

(First, let me say, this is not an affront to Planet RDF.)

This is a bit of a pet peeve of mine, and perhaps what I say here will offend some semwebbers (I hope not). The semantic web (in my view) is not about “triples all the way down.” What do I mean by that? Let me explain.

RDF brings primarily two things to the table when it comes to publishing and integrating data on the web: names in the form of URIs, and a simple data model that is flexible enough for (arguably) nearly any kind of data. (I would like to add a third, meaningful links, but I will avoid that for now.) So when data is published to the web, publishing it as RDF allows you: (1) to identify the things in your data across the World Wide Web, and (2) to structurally (and possibly semantically) integrate your data with other data on the World Wide Web. (I emphasize “World Wide” here to bring to attention the vast scope of publication, identification, and integration that is being achieved.) Fantastic.

Does this mean that everything can be efficiently (or rather, ideally) represented in RDF? No. Then why would you ever want to handle triples? You probably don’t. Let me explain.

RDF is meant to solve the problem of meaningfully publishing data (not just documents) on the World Wide Web. Beyond that, do what you want. More specifically, when you crawl and/or aggregate data from the World Wide Web, you don’t have to keep the RDF data as triples in your system. It is no longer on the global stage of the World Wide Web; rather, it is now in your system where you are king. So optimize away! Store it or process it however you like! Relational databases? Sure! Rewrite URIs as shorter terms? Whatever floats your boat! Ignore the explicit semantics and treat it like an unlabeled graph? I wouldn’t recommend it, but you’re the king! Do whatever it takes to meet your use case, and if your use case has something to do with RDF data, then fine, leave it as triples if you want. My point is, it’s not necessarily “RDF all the way down,” but it is “RDF at the top” where “top” is the place of publication, the World Wide Web. The universal naming mechanism of URIs and the generic data model enables data publishers to get data out there in a way that can be explicitly understood by machines (for example, when I say “Beast is furry,” am I talking about Mark Zuckerberg’s dog or the fictional X-Man Dr. Henry Philip “Hank” McCoy?), but as the creator of that machine, it’s up to you how to utilize those explicit semantics.

(They both look furry to me.)

To be clear, though, I am promoting RDF as a way to publish structured, semantic data as opposed to not publishing structured, semantic data.  In the future, it is conceivable that there may exist other good ways to publish structured, semantic data, but RDF exists today and is widely used.

So I will leave it at that. Again, I invite comments, rebuttals, accolades, disparagements, etc.

Jesse Weaver

• LexEVS is available as a AMI: https://cabig-kc.nci.nih.gov/Vocab/KC/index.php/Cloud_Computing_Services
• A Virtuoso Universal Server is available as an AMI: http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtInstallationEC2
• I’m pretty sure NCBO is doing something here, but I can’t find any specifics.
• Will Stardog be available as an AMI or similar?

So that’s on the “semantics using the cloud” side, but I really think that there’s a lot of potential going the other way: using semantics to discover data and services in the cloud. SADI has the ability to discover and link services through ontologies. It’s similar to SAWSDL (in fact, they wrap SAWSDL services), but they don’t bother with the extra layer, and just let the service process RDF directly. When SADI services are deployed to the cloud, it’ll solve a big problem for people who want others to use their services/algorithms without the overhead of maintaining those servers themselves. In fact, with the Amazon DevPay structure, it’s possible for small labs to release datasets, databases, and algorithms to the world and not have to pay to support it.

I say when, not if, because my implementation of SADI in Python is almost ready for deployment through Google App Engine (which can be deployed in AWS or other systems using AppScale), and from what I hear, it won’t take much work to do the same with the Java implementation. Between this and the extreme portability of python SADI services (it’s just a script), use in the cloud and redeployment to private clouds is going to be trivial.

So I’m asking folks, am I full of it? Also, what else is there out there? Please help me out so that we all get some good exposure!

The current state of OGD

David Eaves, an activist who advises the city of Vancouver, Canada in issues about Open Data, gave a keynote in which he described his views on the current state of Open Data movement. First, it is striking that the success stories are not just a few anymore (as Data.gov or Data.gov.uk) but there are dozens (perhaps hundreds), both at national, regional and local levels. Similarly, the term Open Government Data is becoming increasingly popular, which is good because it is easier to stop explaining the ‘what’ and start focusing in the ‘how’.

Another interesting point is how the movement of Open Government Data already passed an inflection point, where it is no longer seen as people demanding from the outside, but being increasingly being invited to help working on these initiatives from within the government. For many, this change in perspective can be confusing and may create some concerns of Open Data being absorbed in a bureaucratic system that makes impossible to implement Open Data initiatives. However, it is clear that in order for these changes to occur, the movement can not reject to collaborate with governments.

Local initiatives, by locals

A talk that I really liked was by Ton Zylstra, who lives in the city of Enschede, the Netherlands. This city has only 150,000 inhabitants. He wanted an Open Data initiative there, however, it was difficult to convince the authorities, so he with a group of people decided to start working on their own. Inviting a handful of hackers to a bar, they created their first application that used data from Twitter, Foursquare, and the venues of a local festival. Eventually they convinced the municipal government that the default option for local data ought to be open.

From this experience, Ton showed several important lessons: You have to create something concrete, no matter if it is small: This implies something that requires little funding (the first beers at the bar were free) and short-term (no more than a couple of weeks). It does not matter if it is something original or not, there are some great ideas out there that deserve to be copied and are very useful for the local community.

How the Open Data died

Another very interesting keynote was by Chris Taggart, founder of OpenCorporates, who warned of the risks that the Open Data movement is facing today. His main concern is the lack of relevance in terms of impact Open Data has on society. For example, he mentioned that so far no one’s business depends on Open Data (although this is not true, there are a few out there, but I have to concede they are rare examples). In general, making data available is not enough, it is necessary for it to be used either in applications, by data journalists, etc. Also, it is fundamental to link different sites with Open Data (something quite uncommon in the movement), so that people can find out more information. Finally, I liked his idea that if the Open Data does not cause problems to its incumbents, then it is not working.

Redefining what is public

Finally another talk that I found interesting was the idea of ​​Dave Rasiej, founder of Personal Democracy, and Nigel Shaldbolt, professor at University of Southampton, to redefine “the public” in terms of data that “is available on the Web in machine-processable formats.” That is, uploading a bunch of PDFs with scanned tables does not make that information public, because it is not easily accessible. This initiative raises the bar of what public data is, especially when compared to the FOIA (Freedom of Information Act) that allows you to request information from government. Note that this applies to all information, as Rasiej so vehemently described it.

So… what did you talked about at OGDCamp?

In my case, I presented a system for publishing Linked Data called LODSPeaKr, which can be used for the rapid publication of government data and to create applications based on Linked Data. In the near future I will be writing more about this framework, but for now you can see my presentation here.