The Tetherless World Weblog

Scaling Up at the Tetherless World Constellation in 2009

Jesse Weaver — Fri, 20 Nov 2009 22:15:50 +0000

Since this is my first post to the Tetherless World blog, perhaps a brief introduction is in order. I’m Jesse Weaver, one of Jim Hendler’s Ph.D. students in the Tetherless World Constellation (TWC) at Rensselaer Polytechnic Institute (RPI). My general research interest is in high-performance computing for the semantic web. Specifically, I have been looking at employing parallelism on cluster architectures for rule-based reasoning and RDF query. Since joining TWC in Fall 2008, I have been working with colleagues toward this end, and it is that work that I would like to share in this blog post.

Jim and I recently published a paper at ISWC 2009 entitled Parallel Materialization of the Finite RDFS Closure for Hundreds of Millions of Triples. Since the time that paper was accepted (and as presented at ISWC), we have actually scaled to billions of triples. We show in this paper that the RDFS rules can be applied to independent partitions of data to produce the RDFS closure for all of the data, as long as each partition has the ontologies. In parallel computing terms, the RDFS closure can be computed in an embarrassingly parallel fashion. “Embarrassingly parallel” is a technical term from parallel computing describing a computation that can be divided into completely independent parts. Such computations are considered ideal for parallelism because there is no need for communication between processes and hence there is essentially no overhead for parallelization. Peter Patel-Schneider had some good questions and comments after the presentation. I have made my responses publicly available in a brief note.

Gregory Todd Williams and I published a paper at SSWS 2009 entitled Scalable RDF query processing on clusters and supercomputers. This paper shows how parallel hash joins can be used on high-performance clusters to efficiently query large RDF datasets. It seemed to get a lot of attention at the SSWS workshop as well as stir up a little bit of controversy. The interesting thing about our approach is that no global indexes are created. Each process in the cluster gets a portion of the data and indexes it locally, but no global indexes are maintained (e.g., we do not globally dictionary encode RDF terms). This allows us to load data extremely quickly with some cost to query time. In many cases, though, the decrease in loading time outweighs the added cost in query time. (The added cost in query time comes from communicating full string values instead of global IDs during the parallel hash join.) This allows for exploratory querying and easy handling of dynamically changing data. Whereas many previous query systems depend heavily on global indexes (for which loading can take on the order of hours or days), we can load large datasets on the order of seconds and minutes. Therefore, if the data changes, it can just be reloaded instead of updating indexes.

Finally, Greg, Medha Atre, Jim, and I submitted a paper to the Billion Triples Challenge (BTC), which we won!

We composed together three systems for our submission. First, we created a simple upper ontology of 31 triples for our domain of interest, linking established concepts of Person to our concept of Person (by subclass), and we did the same for many relevant properties (name, email, etc.) (by subproperty). Then, we used the aforementioned parallel materialization work to produce inferences on the BTC dataset, inferring triples that use our terms from the upper ontology. Using the aforementioned work on scalable query, we then extracted only our triples of interest. This reduced dataset is almost 800K triples, much more manageable than the original 900M triples, and it can now be used by existing tools without much concern of dataset size. As a finishing touch, we compressed the reduced dataset down into a BitMat RDF data structure, resulting in a final disk space of 8 MB for the triples and 25 MB for the dictionary encoding. Simple basic graph pattern queries can be executed against the BitMat. The entire process took roughly 22 minutes. See more about the submission at our BTC website which contains the datasets and some statistics about the datasets.

That being said, the future holds much work to be done for scalability in the semantic web domain.

At present, I have been looking at formalizing a more general notion of “abox partitioning” for the purpose of classifying rules that fit such a paradigm, and then explore its application to OWL2RL. Some parts of OWL2RL—like symmetric properties and inverse properties—clearly fit in the inferencing scheme from the parallel materialization paper. However, many of the much desired features—like inverse functional properties and owl:sameAs—do not. For such rules, parallel hash joins may be needed, or perhaps a more clever partitioning scheme.

We could also improve loading time of these systems (and perhaps communication time during parallel hash joins) by using an RDF syntax that is less verbose than N-Triples, but not as complex as Turtle. (Remember, we are concerned about parallel I/O.) To that end, we are exploring defining a subset of Turtle that would be helpful for I/O purposes without trading off the inherent simplicity of N-Triples (one triple per line).

We would also like to start employing more memory-efficient RDF storage data structures (like BitMat or Parliament) directly in our systems. This is particularly important for the Blue Gene/L architecture which has at most 1 GB of memory per node.

And speaking of the Blue Gene/L, I have been doing all my work at RPI’s fabulous Computational Center for Nanotechnology Innovations (CCNI). The CCNI is really a great computation facility having parallel file systems, high performance clusters, large SMP machines, and—of course—a Blue Gene/L. Such a resource is a great enabler for our research.

Jesse Weaver
Ph.D. Student, Patroon Fellow
Tetherless World Constellation
Rensselaer Polytechnic Institute
http://www.cs.rpi.edu/~weavej3/

Parliament, storage density, and napkin math

greg — Sun, 08 Nov 2009 08:53:41 +0000

What follows is some napkin math on the storage requirements of the Parliament triplestore (as described in their SSWS paper) and its implications for our clustered RDF query engine (any errors are presumably with my understanding of the described system).

The triple store has three data structures: a resource dictionary, statement list, and resource table. The resource dictionary is a mapping from node values (literals/IRIs/blank node identifiers) to an integer identifier and can be implemented with a search tree or hash (Parliament uses a BerkeleyDB B-tree). The statement list and resource table are the two interesting structures.

The statement list is an array of fixed width records containing an ID for the subject, predicate, and object in the triple, and corresponding offsets to the next triple record with the same subject, predicate, and object, respectively. Making some assumptions about the storage sizes involved (related to my specific needs), a statement list record’s fields are:

Statement List Record
field	type
subject	4 byte Offset into Resource Table
predicate	4 byte Offset into Resource Table
object	4 byte Offset into Resource Table
next subject	4 byte Offset into Statement List
next predicate	4 byte Offset into Statement List
next object	4 byte Offset into Statement List

The resource table is an array of fixed width records for each resource containing three offsets into the statement list for the first statement with the specific resource in subject, predicate, and object position, respectively, three counts representing the length of these three lists, and the resoure value. The storage requirements for a resource table record’s fields are:

Resource Table Record
field	type
first subject	4 byte Offset into Statement List
first predicate	4 byte Offset into Statement List
first object	4 byte Offset into Statement List
subject count	4 byte Integer
predicate count	4 byte Integer
object count	4 byte Integer
value	8 byte pointer to n byte resource value

Puting aside the storage cost for the resource dictionary (which would obviously depend on the specific structure used, whether tree or hash), the storage requirements for the statement list and resource table are then:

(24 bytes * |triples|) + ((32+n) bytes * |nodes|)

where n is the average resource value size.

To get some real world numbers, I looked at a subset of the Billion Triples Challenge dataset (BTC chunks 000-009). This subset has 100M triples, ~34M nodes, and occupies 3.5GB with nodes serialized using N-Triples syntax (I’ll assume here that the N-Triples serialization gives a rough approximation of the node’s representation in memory). This averages out to ~341 unique nodes per thousand triples and ~109 bytes per node.

Using these numbers, the statement list and resource table for 100M triples would take (24*100M) + ((32+109)*34144466) bytes = 7214369706 bytes = 6.7GB. The upper limit on a single triplestore with these structures (using 32-bit offsets) is 4B statements, with a storage estimate of 288GB.

I’d like to try implementing this and using it on our clustered RDF query engine (part of the system that won the Billion Triples Challenge last month). It seems like a very natural fit since we’d like high storage density but haven’t spent any effort on that front yet (and our current code is awful in this respect) and our parallel query answering code doesn’t need the sorting that a traditional tree-based index would provide. Some more (very rough) napkin math suggests that we’d be able to get over 100 billion triples loaded onto the Blue Gene/L. It’ll be interesting to see if the numbers actually pan out, and if the query algorithm can handle anything close to that much data.

Gregory Todd Williams

Web Accessibility in an Educational Context

Evan — Sun, 25 Oct 2009 15:32:44 +0000

I’m currently at the annual meeting for the Human Factors and Ergonomics Society and a few hours ago I was attending a set of presentations by the Internet TG on Input and Output on the Web. One of the talks that caught my attention was by Wayne Shebilske of Wright State University on the use of screen readers to help impaired students complete tasks within educational programs such as WebCT, student status management software, and the like. There seem to be many problems with such tools as Dr. Shebilske pointed out based on his study. I think this could be a prime candidate for SW technologies to step in to improve the end user experience. I foresee using such tools along with technologies like RDFa to give the user a better representation of the content being displayed and improving the overall quality of life for these individuals.

Evan

Probing the SPARQL endpoint of data.gov.uk

li — Fri, 23 Oct 2009 04:13:10 +0000

We just ran across the preview SPARQL endpoint for UK’s Data.gov (powered by Talis) following Harry Metcalfe’s blog . In order to understand what data is hosted by the triple store, we use a series of SPARQL queries to probe the content in data.gov. We leverage a web service http://data-gov.tw.rpi.edu/ws/sparqlproxy.php to convert SPARQL/XMl result into HTML and JSON.

First, let’s do some warm up exercises

Q: show me some triples!

SPARQL:

SELECT ?s ?p ?o WHERE {?s ?p ?o} LIMIT 5

Result:

“2007-05-02″^^http://www.w3.org/2001/XMLSchema#date .

A: ok, it has some gazette dataset about lond (see http://www.london-gazette.co.uk/), and it uses FOAF and DC vocabulary.

Q: show me some classes and their instances?

SPARQL:

SELECT DISTINCT ?c WHERE { [] a ?c. } LIMIT 5

Result:

A: same observation as above.

Q: Does it host any named graphs?

SPARQL:

SELECT ?g WHERE {GRAPH ?g { ?s ?p ?o } } LIMIT 10

Result: “0″ ^^

A: no named graph found, and there is only one big default graph

Now let’s run several expensive aggregation queries (note aggregation queries are not part of the current SPARQL specification)

Q: How many triples?

SPARQL:

SELECT count(*) WHERE {?s ?p ?o}

Result: “5529380″^^

A: alright! Aggregation query is support, and there are 5 million triples. Note “count” is a non-standard aggregation function, it may be support differently be different SPARQL endpoints.

Q: How many graphs (and the number of triples in each graph)?

SPARQL:

SELECT ?g count(*) WHERE {GRAPH ?g { ?s ?p ?o } }

Result: “0″ ^^

A: no named graph.

Q: How many populated classes?

SPARQL:

SELECT count(distinct ?c) WHERE {[] a ?c}

Result: “99″^^

A: there are 99 different classes having direct instances in this triple store.

Q: How many populated properties?

SPARQL:

SELECT count(distinct ?p) WHERE {[] ?p ?o}

Result: “86″ ^^

A: There are 86 unique properties being used as predicate in this dataset. Each property has used by 64K (5529380/86) triples in average. There must be some very popular properties, and we will do that survey later.

Q: How many typed individuals?

SPARQL:

SELECT count(distinct ?s) WHERE {?s a ?c}

Result: 504 Gateway Time-out

A: Opps, really expensive. Let’s try something else

Q: How many defined classes?

SPARQL:

SELECT count(distinct ?s) WHERE {{?s a } UNION {?s a }}

Result: 0

A: no class defined, and the triple store is full of individuals

Q: How many individuals (again)?

SPARQL:

SELECT count(*) WHERE {[] a ?c}

Result: 995694

A: There are nearly 1 millions of typed individuals, so we can easily see every invidual has 5 (5M/1M) triples in average.

Now, let’s do some knowledge discovery

Q: Show me the 3 most/least used classes in this dataset?

SPARQL:

1) select ?c ( count(?s) AS ?count ) where {?s a ?c} group by ?c order by desc(?count) limit 3

2) select ?c ( count(?s) AS ?count ) where {?s a ?c} group by ?c order by ?count limit 3

Result:

1) most used classes

“156452″ ^^

“106934″ ^^

“87798″ ^^

2) least used classes

“1″ ^^

“2″ ^^

A: Again, the SPAQL query is “safe” because Talis support these SPARQL extensions (c.f. http://n2.talis.com/wiki/SPARQL_Extensions). The class that has the most number of instances in this dataset is http://www.gazettes-online.co.uk/ontology#Notice (which has 156,452 instances.)

Q: what about property usage (top 5)?

SPARQL:

1) select ?p ( count(?s) AS ?count ) where {?s ?p ?o} group by ?p order by DESC(?count) limit 5

2) select ?p ( count(?s) AS ?count ) where {?s ?p ?o} group by ?p order by ?count limit 5

Result:

1) 5 most used properties:

“995694″ ^^

“312476″ ^^

“310170″ ^^

“181940″ ^^

“181335″ ^^

2) 5 least used properties:

“1″ ^^

“2″ ^^

“3″ ^^

“49″ ^^

A: Some properties are really used (e.g., http://www.gazettes-online.co.uk/ontology/corp-insolvency#dateAdministrationOrderMade has been used only once) while some are heavily used (like rdf:type, which is the most frequently used predicate).

Conclusion

We need to stop probing now. A number of complex queries ended up with a timeout error because (i) “LIMIT” only control the final results, so that we cannot just get statistical results on 1000 triples; (ii) “GROUP BY” may produce too many intermediate results, (iii) the statistics queries does not leverage the index structure of triple store, or index structure are not designed for handling such queries, and (iv) many other issues.

Updates

We also delivered a web service running this probe job, see http://data-gov.tw.rpi.edu/ws/sparqlprobe.php. This service is focusing on default graph, ant we welcome input on measuring named graph.

presented by Li Ding and Zhengning Shangguan

OWL 2 Reference Card released

Jie Bao — Sun, 18 Oct 2009 04:39:22 +0000

We’re pleased to announce the OWL 2 Reference Card [1]. The Card is meant to be a “cheat sheet” of OWL 2 features printable on a single piece of paper (on both sides). It is based on the OWL 2 Quick Reference Guide [1], which is now a Proposed Recommendation [2] in the OWL 2 Web Ontology Language document set.

Background: OWL 2 [4] is an extension to OWL 1 with a few new functionalities. Some of the new features are syntactic sugar (e.g., disjoint union of classes) while others offer new expressivity, including:

* keys;
* property chains;
* richer datatypes, data ranges;
* qualified cardinality restrictions;
* asymmetric, reflexive, and disjoint properties; and
* enhanced annotation capabilities

Comments and suggestions to the Card are welcome (please send to public-owl-comments@w3.org)

[1] http://www.w3.org/2007/OWL/refcard

[2] http://www.w3.org/2007/OWL/wiki/Quick_Reference_Guide

[3] http://www.w3.org/TR/2009/PR-owl2-quick-reference-20090922/

[4] http://www.w3.org/TR/owl2-overview/

Jie Bao

My Personal (unofficial) Semantic Web FAQ — a pointer

hendler — Wed, 02 Sep 2009 00:32:59 +0000

The joy of multiple blog sites is having to post pointers to one blog entry from another.

My blog at nature.com now has an entry entitled “The Semantic Web: My personal (unofficial) FAQ” which lives at http://network.nature.com/people/jhendler/blog/2009/08/03/the-semantic-web-my-personal-unofficial-faq. Comments, and especially your suggestions for Qs and As are more than welcome there or here (or anywhere else for that matter)

Cheers,

Jim H.

Current Issues in data.gov

li — Fri, 31 Jul 2009 22:54:34 +0000

While translating data.gov data into RDF, we have discovered some issues with the published datasets. These issues can be roughly categorized as follows:

Duplicated Datasets- Some datasets are part of another dataset, e.g. Dataset 140 (2005 Toxics Release Inventory data for the state of California (Environmental Protection Agency)) is a subset of Dataset 191 (2005 Toxics Release Inventory National data file of all US States and Territories (Environmental Protection Agency)).
Formatting Issues - The format of some datasets is not friendly to machine processing. Not all datasets offer CSV format data, and parsing table data from them requires non-trivial efforts. Example: Dataset 37 (Lower Colorado River Daily Average Water Elevations and Releases (US Bureau of Reclamation)). Some websites, meanwhile, have no data at all: Dataset 335 (National Longitudinal Surveys (US Bureau of Labor Statistics)), for example, tells you how to order data from the government.

screen shot of the text file from dataset 37 (Lower Colorado River Daily Average Water Elevations and Releases) by US Bureau of Reclamation

Access Point Issues - The access points for some datasets do not point to pages friendly to machine access. Instead of pointing to a downloadable file covering the entire dataset, some lead to an interactive website where only partial data can be returned by a web-based query. Example: Dataset 330 (Local Area Unemployment Statistics (US Bureau of Labor Statistics)) and Dataset 96 (National Water Information System (NWIS) (US Geological Survey)).
screen shot of the query interface for accessing dataset 330 (Local Area Unemployment Statistics) by US Bureau of Labor Statistics

For more details, please visit http://data-gov.tw.rpi.edu/wiki/Current_Issues_in_data.gov .

Sarah Magidson, Li Ding, Dominic DiFranzo, and Jim Hendler

Data.gov Datasets Translated in RDF!

li — Thu, 23 Jul 2009 00:45:21 +0000

We have created 16 RDF datasets covering 187 of the datasets published at data.gov (171 EPA datasets are subsets of three larger EPA datasets). The original datasets were published by EPA, US Census Bureau, USGS and Office of Management and Budget in CSV compatible format, and they contributed 13,532,250 table entries. The translated RDF datasets includes a total of 2,927,398,352 triples involving 2,526 properties.

We publish the RDF data in two alternative ways: (i) a collection of linked partition files in RDF/XML for users to browse the dataset and dereference the URIs using semantic web browsers, and (ii) one big N-TRIPLE file (data.nt) concatenating the partition files for machines, especially triple stores, to download and import. The largest dataset is Dataset_91, which contributed 2.11 billion triples.

To access the RDF datasets, users may go to Data.gov_Catalog with the following options:

follow links in the “rdf(index file)” column to access the index file in RDF/XML which contains the property list, statistics, and links of the RDF dataset. e.g. http://data-gov.tw.rpi.edu/raw/401/index.rdf
follow links in the “rdf(partition files)” column to start an RDF browser (e.g. tabulator) to surf the RDF/XML partition files. e.g. http://data-gov.tw.rpi.edu/raw/401/link00001.rdf
follow links in “the rdf(complete file)” column to download the complete RDF dataset in N-TRIPLE format (gzipped). e.g. http://data-gov.tw.rpi.edu/raw/401/data-401.nt.gz
follow links in the “url(data.gov)” column to see the original metadata at data.gov
follow links in the “wiki page” column to see enhanced metadata about data.gov datasets

More datasets are coming, so please stay tuned and come back to http://data-gov.tw.rpi.edu/.

Tilting at the NSF windmill

hendler — Mon, 13 Jul 2009 18:05:49 +0000

Colleagues - one of my blog entries at Nature seems to have hit a nerve - been zinging around the “twittersphere” and I’ve received a number of responses in private not just commiserating, but agreeing with the major points. I want to make it clear that this is solely my own opinion, and it has not been carefully researched, but given that so many US Semantic Web researchers have shared the frustration that I express here, I thought I’d share it on planetRDF as well (Europeans, believe it or not, on this side of the ocean it is hard to get funding for Semantic Web research - you have no idea how lucky you are!)

-Jim H

from blog entry: “Why NSF cannot fund high-risk, high-reward research”

I just got turned down for a grant. That’s nothing new, you win some and you lose some, and every senior professor has gotten used to that over time. This time, however, I cannot find it in myself to just say “oh well” and let it go at that. This time, I think I need to go public, because I think what happened shows an endemic problem with the US National Science Foundation and, I hope, points out some things they could do to fix it.

Click here for the blog entry at Nature.com

What’s in data.gov

li — Thu, 25 Jun 2009 14:05:24 +0000

A recent article by Tim Berners-Lee, “Putting Government Data online“, has attracted significant interest to the datasets published at the US data.gov website. As Berners-Lee discusses the Semantic Web techniques that can be used to get those data into RDF space (something we are now working on), we would like to share our initial investigation of the contents of these government datasets.

updates:

* we have not published 5 billions triples from hundreds of datasets at http://data.gov. see http://data-gov.tw.rpi.edu/wiki/Data.gov_Catalog

I. Translate dataset into RDF

The catalog of the datasets in data.gov,http://www.data.gov/details/92, is published in CSV format as part of data.gov. We converted it into RDF using simple CSV parsing. We kept the translation minimal: (i) the properties are directly created from thecolumn names; (ii) each table row is mapped to an instance of pmlp:Dataset; (iii) all non-header cells are mapped to a literal - we don’t create new URIs at this point. The output of our work is published on tw website at:

http://data-gov.tw.rpi.edu/raw/92/data-92.rdf

(We are now starting to do more integration work, extracting multiple objects from single tables, linking into the linked open data cloud, etc. and will publish new version when that is done - the purpose of this first work was simply to make the catalog more available to the RDF community)

II. Browse and query the RDF graph

As an example, we can browse the dataset in tabulator, and then use a SPARQL web service to query the dataset. For example, we use a sparql query to list datasets published in CSV format:

http://onto.rpi.edu/sw4j/sparql?queryURL=http://data-gov.tw.rpi.edu/sparql/select-csv-dataset.sparql

III. Observations on the RDF graph

Using this service we can answer some basic questions about the data.gov datatsets:

1. How many datasets are published, and how many among them can be easily converted into RDF?

There are 332 datasets which can be partitioned by type: raw data catalog(301); tool catalog (31).

Not all of the datasets have a link to downloadable data because some offer only browseable data via their own websites, Others publish datasets in multiple formats. As of today, the online static files associated with the datasets are distributed as follows: 204 datasets offer a CSV format dump, 10 datasets offer an XML format dump, and 21 datasets offer an XLS format dump.

2. How are the datasets categorized?

Category	number of datasets
Geography and Environment	227
Labor Force, Employment, and Earnings	30
Social Insurance and Human Services	30
Health and Nutrition	11
Law Enforcement, Courts, and Prisons	7
Population	4
Other	3
Prices	3
Business Enterprise	2
Education	2
Energy and Utilities	2
Federal Government Finances and Employment	2
Income, Expenditures, Poverty, and Wealth	2
Science and Technology	2
Transportation	2
Construction and Housing	1
International Statistics	1
National Security and Veterans Affairs	1

3. What are some of the key items in the dataset?

4. What are the sources of the datasets?

The majority of the datasets are published by the EPA, and they contain environmental data partitioned by the states of the US in three individual years. Others come from other govt agencies - the distribution is as follows:

IV. Getting Datasets linked

Although the datasets are not explicily linked, we see a number of opportunities for connecting these datasets to others (and into the Linked Open Data datasets):

A large percentage of files have some sort of geo-tagging, thus they can be linked to DBpedia or Geo-names (and then presented via Map services).
Some datasets are subsets of other datasets, e.g. EPA data “2005 Toxics Release Inventory data for the state of Georgia” is a subset of “2005 Toxics Release Inventory National data file of all US States and Territories” making for easier “internal” linking of the datasets.
A number of the datasets contain temporal information, e.g. IRS’s “Tax Year 1992 Private Foundations Study”,…”Tax Year 2005 Private Foundations Study” which provides an opportunity for mashups using timelines and such.

V. Conclusions

We are committed to getting more of the data.gov data online soon (in RDF), and then investigating data integration and knowledge discovery. In order to get our datasets linked to the linked data cloud, we will use SPARQL for extracting entities and our Semantic Mediawiki as a platform to capture the owl:sameAs mappings. Scalable dataset publishing is also challenging as some of these are very large datasets, e.g. “2005-2007 American Community Survey Three-Year PUMS Population File” has a 1.1 g zipped csv file. Moreover, some datasets are not directly available in one file but via a web service. Our current plan is to produce RDF documents available for download soon, and to work on bringing more of these datasets into live, SPARQLable forms as we can.

Li Ding, Dominic DiFranzo and Jim Hendler