Research
Publications
Below is a technical report given as a part of the final presentation for the Ontology Engineering class.
ANNO Technical Report Draft in docx format
ANNO Technical Report Draft in PDF format
Iqbal, MD Ridwan, Klawonn, Matthew, Norris, Spencer. An Ontology for Automated Neural Network Initialization with Heuristic Knowledge Graph, Rensselaer Polytechnic Institute Technical Report, Ontology Engineering 2016, May 6th, 2016.
Presentations
Here are presentations given in the Ontology Engineering class which may be useful
Assignment 11 Presentation
Assignment 8 Presentation
Assignment 7 Presentation
Assignment 3 Presentation
Related Work and References
On the use of ontologies for recommendation systems
Middleton, Stuart E., David De Roure, and Nigel R. Shadbolt. "Ontology-based recommender systems." Handbook on ontologies. Springer Berlin Heidelberg, 2009. 779-796.
Staab, Steffen, and Rudi Studer, eds. Handbook on ontologies. Springer Science & Business Media, 2013.
Costa, A., R. Guizzardi, and G. Guizzardi Filho. "COReS: Context-aware, Ontology-based Recommender system for Service recommendation." Proceedings of the 19th international conference on advanced information systems engineering (CAiSE’07). http://www. inf. ufes. br/~ rguizzardi/publications/costa-guizz-guizz-gonc-UMICS07-cameraready. pdf. pdf. 2007.
On neural network weight initialization
K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
X. Glorot, Y Bengio, “Understanding the difficulty of training deep feedforward neural networks,” International conference on artificial intelligence and statistics, pp. 249–256.
Y. Lecun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient BackProp,” Lecture Notes in Computer Science Neural Networks: Tricks of the Trade, pp. 9–50, 1998.
J. Y. F. Yam and T. W. S. Chow, “A weight initialization method for improving training speed in feedforward neural network,” A weight initialization method for improving training speed in feedforward neural networks, vol. 30, no. 1, pp. 219–232, Jan. 2000.
Montavon Grégoire, Orr Geneviève B., Müller Klaus-Robert, and Y. Bengio, Neural networks tricks of the trade. Berlin: Springer, 2012. pp 437-478
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580, Jul. 2012.
Matsugu, Masakazu, Katsuhiko Mori, Yusuke Mitari, and Yuji Kaneda. "Subject independent facial expression recognition with robust face detection using a convolutional neural network." Neural Networks 16, no. 5 (2003): 555-559.
Beyer, W. H. CRC Standard Mathematical Tables, 28th ed. Boca Raton, FL: CRC Press, pp. 531 and 533, 1987.
Class Content
Here are links to the ANNO team submissions for assignments of the 2016 Ontology Engineering class.
Assignment 13
Assignment 12
Assignment 11
Assignment 10
Assignment 9
Assignment 8
Assignment 7
Assignment 6
Assignment 5
Assignment 4
Assignment 3
Design
Use Cases
You have found the current use case documents for ANNO.
ANNO Weight Initialization Use Case
Artificial Neural Network Conceptual Model and Ontology Diagram
Conceptual Model
Our recommendation system works by examining which weight initialization algorithms are compatible with various aspects of a neural network's structure, and provides access to the accompanying provenance. Here you can see the relevant classes and relations.
There are a few classes which are of particular importance to how the ontology can be used to recommend weight initialization algorithms for a certain network structure.
WeightInitializationAlgorithm
The weight initialization algorithm class captures knowledge from the literature about which algorithms are appropriate with which network structures. It does this through the isCompatibleWithActivationFunction, isCompatibleWithLayerType, and hasMaximumCompatibleDepth object properties. These three properties capture the aspects of neural networks that we have deemed to be important when considering weight initialization choice: activation functions, layer types, and network depth. Assertions which include these properties have their associated provenance, that is the papers which they come from, stored via nanopublications. The nanopublications link PROV-O objects (like Agents) to the assertions via TriG subgraphs.
ActivationFunction
The activation function class contains information about activation functions which is relevant to weight initialization algorithms. For our ontology, the relevant factors are linearity and smoothness, for which each activation function class has two booleans: isLinear and isSmooth. These booleans are what the recommendation engine will query for.
Layer
Layers indicate which function will be performed on input data, with specialized layers often containing complex functions. In our ontology, the functions are not necessarily described in the classes. Rather, initialization algorithms are linked to layers via name only. In the future this may change, with some properties of the layer potentially being taken into consideration during the recommendation process.
ArtificialNeuralNetwork
The artificial neural network class serves to tie everything together, in the sense that it references specific layers, which in turn reference activation functions and weight initializations. Further, this class provides a hasDepth data property, indicating the depth of the corresponding network. This depth property is linked to weight initialization algorithms via the hasMaximumCompatibleDepth object property discussed earlier.
Ontologies
Current Ontologies
Here are the current ontologies, updated as of May 6th, 2016. These ontologies may be imported using their URLs.
Demonstration and Queries
Intended Use
Currently the ANNO ontology contains only one use case, with the primary purpose of the ontology being to recommend a weight initialization for a network based on the network's structure. The use of the ontology is best explained by an example. Consider the case where an agent is attempting to query the ontology for a weight initialization based on the agent's neural network. This agent could be a human user of the ontology, or a software library with a model loaded into memory. In essence, the agent will distill the network into instances of classes which ANNO provides, such as Artificial Neural Network, Layer, Unit, etc. The figure here shows this process.
Figure showing conversion from network to ontology instances.
Once the translation between the agent's neural network and ANNO classes is complete, the agent can construct queries which use these classes to recommend an ideal network. Presented below is a query searching for initializations compatible with a network of depth greater than 10 which contains non-linear activation functions.
Example Query 1
-
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
-
PREFIX owl: <http://www.w3.org/2002/07/owl#>
-
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
-
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
-
PREFIX anno: <http://tw.rpi.edu/web/Courses/Ontologies/2016/OE_11_Team_Cerebral/>
-
-
select distinct ?initialization where {
-
?initialization rdfs:subClassOf anno:WeightInitializationAlgorithm.
-
?initialization rdfs:subClassOf ?b. ?initialization rdfs:subClassOf ?c.
-
?b a owl:Restriction. ?c a owl:Restriction.
-
?b owl:onProperty anno:hasMaximumCompatibleDepth.
-
?b owl:hasValue ?depth.
-
?c owl:onProperty anno:isCompatibleWithActivationFunction.
-
?c owl:someValuesFrom ?function.
-
{anno:NonlinearActivationFunction rdfs:subClassOf ?function.} UNION {?function a anno:NonlinearActivationFunction.}
-
FILTER (?depth >= 10)
-
}
There are two criteria to be met for weight initialization function lookup. First, the system must query for weight initialization functions for networks of depth > 10. Second, the system must query for all functions which are appropriate as activation functions and which are subclasses of the non-linear activation function class. If no weight functions were to meet all the criteria, then perhaps we would go for one which met the most, or return none. The diagram below shows the relevant classes and how they relate such that the above query returns the correct answer of HeInitializationAlgorithm.
Competency Question Reasoning.
The provenance of the assertions which link initialization algorithms to network structure can also prove extremely valuable. In the next example query we look for papers which indicate that the XavierInitializationAlgorithm is compatible with networks of depth greater than 10.
Example Query 2
-
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
-
PREFIX owl: <http://www.w3.org/2002/07/owl#>
-
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
-
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
-
PREFIX anno: <http://tw.rpi.edu/web/Courses/Ontologies/2016/OE_11_Team_Cerebral/>
-
select distinct ?pub ?paper ?p ?o where {
-
graph ?assertion {
-
anno:XavierInitializationAlgorithm rdfs:subClassOf ?target .
-
?target a owl:Restriction ;
-
owl:onProperty anno:hasMaximumCompatibleDepth ;
-
owl:hasValue ?val .
-
}
-
FILTER(?val > 10) .
-
graph ?head {
-
?pub a np:Nanopublication ;
-
np:hasAssertion ?assertion ;
-
np:hasProvenance ?provenance .
-
}
-
graph ?provenance {
-
?assertion prov:wasDerivedFrom ?paper .
-
?paper ?p ?o .
-
}
-
}
This question is answered by the ontology by first searching for graphs containing an assertion that the XavierInitializationAlgorithm class is a subclass of an owl:Restriction limiting the value of hasMaximumCompatibleDepth to integers greater than 10. This graph will match to a Nanopublication Assertion graph containing these triples. This graph can then be tied to a Nanopublication Head graph, which in turn references a Nanopublication Provenance graph. The Provenance graph is then searched for the prov:derivedFrom predicate, which finally connects the aforementioned paper back to the original question.
In the future the ontology will be incorporated into a recommendation engine, which will likely take the form of a python library. The library will be called by the user, and rely on a knowledge base for reasoning over an input neural network. This knowledge base relies on the ontology for structure and inference. This flow is illustrated below.
Our ontology will be evaluated by testing the recommendations of our system on various open source neural network models. The goal will be to improve the test accuracy of these models, with our expectation being that many open source models have weight initializations which can be improved upon. In a brief survey, as much as 75% of models found have naive weight initializations*.
For further reading on the technical details of the use case and the ontology in general, check the relevant parts of the project page.
Getting Involved
We gladly welcome feedback and contributions to ANNO. Please feel free to fork our github repository, or raise issues and request features. You may also contact us directly at klawom@rpi.edu. Our goal is to involve both machine learning and semantic researchers in this project, so collaboration of any kind is encouraged. Our ontology maintenance policies can be found in PDF form at ANNO Maintenance and Evolution Policies [Upload]. They are also displayed below.
Thanks for your interest!
-The ANNO Team
Current Maintenance Policy
The goal of our maintenance policy is to prevent loss of code and data from unforseen circumstances. We maintain a strict rule set such that no unwanted changes may happen in the system. These maintenance policies will likely change once the Ontology Engineering course has finished, as many of them exist specifically for the purpose of completing assignments.
- Only one person can edit a document at one time. Each team member must inform before editing and also after finishing an edit.
- One person handles the header metadata changes and those changes are made before a homework submission.
- All documents must be named with v1,v2, and so on for each new revision before the final homework submission.
- All documents must be uploaded to the team google drive after making changes.
- Each new homework assignment represents a new revision of all the documents and are named accordingly
Alternative Approach
If we were to start over, we would use a versioning system such as Github to manage changes. This would have made editing and tracking changes much simpler and more efficient.
Future Maintenance Policy
Right now we are developing the code for the recommendation system, but after the first release of our code we will follow a much stricter maintenance policy.
- There will be no deletion of any entities or restrictions in a minor revision
- Restrictions can be loosened or removed but no new restriction can be added in the minor revision
- New subclasses can be created and a class can be subclassed under a parent, but a parent child relationship cannot be removed or changed.