Research
Publications
Presentations
- OE13 Final Presentation
- OE11 Conceptual Model
- OE8 Conceptual Model
- OE7 Conceptual Model
- OE5 Conceptual Model
- Project Pitch
References
Here is a list of references used by the FRMA team to build and document their ontology.
-
[1] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” in Workshop on faces in’Real-Life’Images: detection, alignment, and recognition, 2008.
-
[2] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 815–823.
-
[3] N. Kumar, A. Berg, P. N. Belhumeur, and S. Nayar, “Describable visual attributes for face verification and image search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 10, pp. 1962–1977, 2011.
-
[4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. Ieee, 2009, pp. 248–255.
-
[5] M. Johnson and A. Savakis, “L1-grassmann manifolds for robust face recognition,” in Image Processing (ICIP), 2015 IEEE International Conference on. IEEE, 2015, pp. 482–486.
-
[6] Imagesnippets. [Online]. Available: www.imagesnippets.com/
-
[7] D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learning Research, vol. 10, no. Jul, pp. 1755–1758, 2009.
-
[8] E. Learned-Miller, G. B. Huang, A. RoyChowdhury, H. Li, and G. Hua, “Labeled faces in the wild: A survey,” in Advances in face detection and facial image analysis. Springer, 2016, pp. 189–248.
-
[9] D. Silverstein and J. Farrell, “The relationship between image fidelity and image quality,” Proceedings of 3rd IEEE International Conference on Image Processing.
-
[10] “Hard and soft light,” Wikipedia, 06-Sep-2018. [Online]. Available: https://en.wikipedia.org/wiki/Hard_and_soft_light. [Accessed: 19-Oct-2018].
-
[11] “Pose,” Merriam-Webster. [Online]. Available: https://www.merriam-webster.com/dictionary/pose. [Accessed: 19-Oct-2018].
-
[12] “Candid,” Merriam-Webster. [Online]. Available: https://www.merriam-webster.com/dictionary/candid. [Accessed: 19-Oct-2018].
License
Copyright 2018 Tetherless World Constellation
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Maintenance Policy
Maintenance Policy
In order to ensure system availability, we have developed a maintenance policy for the FRMA Ontology. All changes to the ontology should be submitted as a pull request on GitHub, with documentation explaining what the change is and why it should be changed. FRMA team members will review the pull request and merge the changes if the requests contain merit and are in line with the FRMA use cases. New versions along with all legacy version will be made available on the project website and all documentation will be kept up to date. A strict hierarchy for changes will be enforced with the following order, any changes made to artifacts higher in the hierarchy the change will cascade to all lower artifacts:
- Use Case
- Ontologies
- Concept Map
- LFW Scripts
- SPARQL queries
Suggestions are welcome and can be sent to:
-
Matthew Johnson johnsm21@rpi.edu
-
Alexander Schwartzberg aeksco@gmail.com
-
Jordan Faas-Bush jmanfb@gmail.com
-
John Angel Angelson1992@gmail.com
-
Andrew Gaudet amg0508@gmail.com
Design
- Use Case
- Conceptual Model and Documentation
FRMA specifically outlines the concepts of “occlusions” and “wearable objects.” Wearable objects are items recognized by facial recognition algorithms (e.g., eyeglasses, earrings, hats). FRMA breaks these items into two categories; headwear (e.g., eyeglasses, hats) and ornaments (e.g., jewelry, neckwear, and makeup). Occlusions model the concept of something that blocks the view of the subject within an image. FRMA breaks occlusions into two further categories; cervical occlusions (anything obstructing the subject’s neck), and facial occlusions (modeling the upper and lower portions of the subject’s face separately).
Hair Ontology
The Hair Ontology aims to more precisely describe the hair on the subject’s head and/or face. Because human hair is incredibly varied, the hair ontology focuses on multiple traits to more completely describe a person’s head and/or facial hair.
The Hair Ontology includes subclasses for hair location, color, texture, and haircut. Location is broken into further subclasses like facial hair and balding. This allows for more detailed description of facial hair and other traits that are usually a binary option in traditional models. This gives our queries the requisite vocabulary to answer competency questions like, ““Which of these two models is better at classifying people with long hair?"
Image Ontology
Image Ontology enables the FRMA system to classify the properties of individual image files. This metadata includes semantic descriptors of the visual image and its subject rather than native image metadata describing camera settings and file information. FRMA is interested in semantically defining the dearth of content within the image rather than merely interpreting the image itself as a flat photograph.
The Image Ontology encapsulates the metadata required to describe images in the context of the FRMA project. Specifically, the Image Ontology defines an Image class and numerous subclasses that describe descriptors like fidelity, color, lighting conditions, and pose.
Machine Learning Model Ontology
The Machine Learning Model Ontology allows users to describe the learning process, the structure of the learned model, and the evaluation of the model from a data-centric perspective.
This ontology defines the provenance used throughout the testing data. It is also used to model the concepts of testing and training ML models, each of which is a subclass of an activity. Many machine learning ontologies describe models using simple text fields or overly simplified Latex equations. However, the Machine Learning Model Ontology allows users to describe their model in RDF allowing developers to describe their model layer by layer. This allows future users to more easily understand the model's architecture and more confidently reuse their systems.
Person, Face, and Demographic Ontology
This sub-ontology focuses specifically on a person’s demographic and facial features that appear in their images, including facial expression, age range, and nose shape.
A “person” is the subject or face that is analyzed by FRMA. A “face” is defined by visual descriptors including face shape, facial expression, nose size, face size, and skin tone. “Demographic” is used to define things like age range, ethnicity, gender expression, and weight range.
Wearable Things Ontology
When dealing with facial recognition, people in the images may be wearing some type of clothing or accessory. Those pieces of clothing may effectively block a part of or occlude someone’s face. The Wearable Things Ontology was created to keep track of various clothing and accessories that could disrupt a facial recognition algorithm. This ontology acts as a place to hold potential sources of occlusion that also happen to be wearable things.
It holds a simple tree like structure of is-a relationships between the different concepts that we currently have in our dataset, all of which span from the root concept of being some “wearable thing.” When viewed in conjunction with the rest of the project, the different concepts within Wearable Things Ontology (WTO) are connected to the FRMA concept of occlusions through the idea that the objects within WTO will typically be occlusive. Expressed concisely, (Forall WTO:SomeObject (Exist FRMA:Occlusion [WTO:SomeObject FRMA:isOcclusionSourceOf FRMA:Occlusion])).
Ontology
"FRMA Latest"
"Individuals Latest"
"Hair Ontology Latest"
"Image Ontology Latest"
"Machine Learning Model Ontology Latest"
"Person Face and Demographic Ontology Latest"
"Wearable Things Ontology Latest"
"OE_12_FRMA"
"OE_12_Individuals"
"OE_12_HairOntology"
"OE_12_ImageOntology"
"OE_12_MachineLearningModelOntology"
"OE_12_PersonFaceAndDemographicOntology"
"OE_12_WearableThingsOntology"
"OE_11_FRMA"
"OE_11_Individuals"
"OE_11_HairOntology"
"OE_11_ImageOntology"
"OE_11_MachineLearningModelOntology"
"OE_11_PersonFaceAndDemographicOntology"
"OE_11_WearableThingsOntology"
"OE_10_FRMA"
"OE_10_Individuals"
"OE_10_HairOntology"
"OE_10_ImageOntology"
"OE_10_MachineLearningModelOntology"
"OE_10_PersonFaceAndDemographicOntology"
"OE_10_WearableThingsOntology"
FRMA Ontology Terms
Hair Ontology Terms
Image Ontology Terms
Machine Learning Model Ontology Terms
Person, Face, and Demographic Ontology Terms
Wearable Things Ontology Terms
Competency Questions
Q: What type of image attributes does the model have the most trouble classifying?
A: 45% of misclassification occurs on images that contain facial hair.
Prior to the triggers the system will consist of several components: LFW[1] smart images which semantically describe who the image depicts and what Kumar [2] features the image in a similar to Image Snippet[6] and a visual concept ontology that describes a taxonomy of Kumar [2] features. Model test results will be loaded into the system and will be lifted to RDF. The system will then use an inference rule and the smart images to determine if an images within the test result are classified correctly or misclassified. For example if we get a test result that says image 203 is Steve Irwin, we can perform a query to get the ground truth data on image 203 and if that image depicts Steve Irwin, we will assert a new triple saying that the classification result is correct.Another inference rule will calculate the accuracy of classified images with an attribute by counting the number of correctly classified images with the attribute and dividing by all images with the attribute. This process will be repeated for each attribute and upper level term within the ontology of visual concepts to calculate accuracy. Afterwards another inferencing rule will examine the accuracies and provide a rating of great, good, ok, fair, bad. Finally we will display these results to the end user through the hierarchical view that shows the hierarchy of attributes/upper terms and the accuracy, ratings. The hierarchy view will start by displaying the results for the highest levels of the taxonomy tree and show every image and the accuracy overall, but the user will be able to select a lower level of the tree such as whether the photo used a flash, whether the subject is smiling or not, or, as listed in this competency question, whether or not the subject has facial hair, and the view will only show the results for photos that belong to that section of the hierarchy tree.
Q: What type of image attributes does the model most associate with George W. Bush?
A: ‘White’, 'Wearing tie', and 'Appears masculine" are most associated with George W. Bush.
This is the query for the competency question, "What type of image attributes does the model most associate with George W. Bush?" Since the model test results are known, we can link them to our “smart” images that contain attribute tagging. From this we can look at all the images of George W. Bush that were classified properly using the LFW ground truth and count which attributes occur more frequently by running a SPARQL query counting the number of occurrences of each attribute in images that were classified as "George W. Bush" by the user’s model - for instance, “wearing necktie”. Then, using our ontology’s taxonomy structure for attributes, we can determine that this attribute is a type of “wearable.” We then display the results of these queries using the hierarchy view as mentioned before, only this time instead of showing the accuracy statistics and percentages we show the accumulated number of images that the model has classified as George W. Bush and contains one or more of the associated attributes (or sub attributes). The user is then able to look at the hierarchy to determine which attributes are most associated with George W. Bush at the level of detail that they are interested in. This query could work for any attribute we (using the Kumar features) or the model have tagged, be it the person’s name or an image attribute.
Q: Which of these two models is better at classifying people with long hair?
A: FaceNet is 10% better than dlib at classifying people with long hair.
The system will load both models and compare the accuracy results generated using the inferencing rules for the long hair which is an upper ontology term which consists of images with the following attributes: Curly Hair, Wavy Hair, and Straight Hair. The model which produces the higher accuracy will be declared better at classifying people with long hair and the difference between the accuracies will be used to qualify the statement.
Q: How well would my model work at classifying mugshot photos?
A: The model has 90% accuracy on images that match labels that the user associated with mugshots.
Here the user wishes to evaluate the model’s use in a particular case. While the LFW images do not contain mugshots, the attributes do contain several attributes which appear in mugshot photos as well. The user enters in a list of attributes (some of which may be higher-level attributes in our taxonomy tree like “facial hair.” In the mugshot case these attributes would likely be “Harsh lighting”, “Posed photo”, and “Eyes open.” The system would then, using our ontological taxonomy tree, find all of the photos that are tagged with these attributes or fall into a higher level attribute such as “facial hair”. Then using an inferencing rule we will identify misclassification and calculate the models accuray on this subset of images. After calculating this accuracy another inferencing rule will rate the model great, good, ok, fair, or bad depending on the accuracy. Finally the system will display those results to the user in our UI.
Q: What type of occlusions does my facial recognition model misclassify the most?
A: DLib and FaceNet are most likely to fail with cranial occlusions.
We consider the above question to be semantically similar to, “What part of the face does my model have trouble handling when it is occluded?” In our model, we have defined Occlusion as a class. In addition, we have defined various subclasses of Occlusion that are differentiated by what part of the face they are blocking. When running a model’s result set with this question in mind, we will have access to the ground truth of what is truly correct, and access to what the model believes is correct. By comparing these, we will now have access to what classification the model in question got incorrect. Because we have specified that the models being used with our system must have been trained on the LFW training set, we also have access to what accessories, attributes, and — with a bit of inferencing— sources of occlusion are in an image. [Note that images are pre-tagged with what accessories a person is hearing. With that, we can do “insert-aforementioned-accessory-here” isOcclusionSourceOf “resultant-Occlusion”] Because the occlusions are associated with a specific region of the face, this means that you can now find the associated facial region. [“resultant-Occlusion” isOccludingBodyRegion “facial-region”] Throw in a couple sprinkles of math and now you have the answer.
Demo and SPARQL Queries
Generate LFW Test Results using your model and generate a result file with the following schema: | Test Image Name | Img Num | Prediction | |-----------------|----------|------------| | Aaron_Peirsol | 2 | Naji_Sabri | | Aaron_Peirsol | 4 | Colin_Powell | | Aaron_Peirsol | 1 | Aaron_Peirsol | python generateTestResults.py ALGORITHM_NAME TEST_NAME PATH_TO_RESULTS_FILE PATH_TO_OUTPUT_RDF_FILE.ttl This generates a turtle file the test results aligned with to the FRMA ontology, which can be loaded into FRMA Blazegraph client. From here FRMA Queries can be run against the results to answer a variety of competency questions:
**Q: How well would my model work at classifying mugshot-like images?** **A:** DLib has 49.7% and FaceNet has 7.7% accuracy on images that have mugshot-like properties. Since the LFW images do not contain mugshots, this query looks for several image attributes that mugshots exhibit. The query calculates and returns the accuracy of the model on images that are posed, indoors, have no facial occlusions, and aren’t blurry.
**Q: Which of these two models is better at classifying people with long hair?** **A:** DLib is 87.7% better than FaceNet at classifying people with long hair. Similar to the mugshot query, the images aren’t tagged with long hair. We are able to build a collection of photos of people with longer hair by removing the photos tagged with features related to short hair such as baldness or a receding hairline. The query then calculates the accuracy of the model on these images.
**Q: What type of occlusions does my facial recognition model misclassify most?** **A:** DLib and FaceNet are most likely to fail with cranial occlusions The SPARQL query uses our ontology’s hierarchy to determine the percentage of occluded images that fail for each type of occlusion and returns the results.
| **Incorrectly Classified Photo with Cranial Occlusion (DLib)** | **Incorrectly Classified Photo with Cranial Occlusion (FaceNet)** | |-------------|---------------| | 
| 
|
**Q: What type of image attributes does the model have the most trouble classifying?** **A:** Most misclassification occurs on images that contain people who are bald or blurry images. This SPARQL query calculates the percentage of misclassifications on each element of our ontology’s photo feature hierarchy.
Getting Involved
Contributing to the FRMA Ontology
This document will walk you through the steps of setting up your computer for contributing to the FRMA Ontology. Before you start, be sure to check our open issues or open a new issue.
First Steps
Development Environment Setup
-
Fork the FRMA-Ontology/Ontology repository on GitHub.
-
Rename your fork of the repository on GitHub to
FRMA-Ontology(otherwise it will just be calledyourusername/Ontology, which doesn't adequately describe the repository.) -
Clone your fork of the repository, (i.e. 'yourusername/FRMA-Ontology
). UseClone with SSHfrom theClone or download` dropdown on GitHub with the following terminal command:
git clone git@github.com:yourusername/FRMA-Ontology.git
- Navigate into the
FRMA-Ontologydirectory (i.e.cd FRMA-Ontology) and add theupstreamremote to the repository:
git remote add upstream git@github.com:FRMA-Ontology/Ontology.git
This enables you to pull the latest upstream changes. You can ensure this worked correclty by running git remote show. You should see entries for both origin and upstream.
- Create and checkout a
devbranch:
git branch devgit checkout dev
Now you can start developing features!
Commiting Changes
To commit your changes, do the following steps:
- Check the status of your changes
You can view the status of your latest changes using the following command: git status
- Stage your changes for the next commit
You can stage all your changes for the next commit with the following command: git add .
- Commit your changes
Commit your changes with an inline message with the following command: git commit -m "Fixed type in ImageOntology.rdf"
- Push your changes
You can push your commited changes to your dev branch with the following command: git push origin dev
Merging Changes
-
Navigate here to open a new pull request into the FRMA-Ontology/Ontology GitHub repository
-
Click
Compare across forksand select you fork (i.e.yourusername/FRMA-Ontology) from the dropdown -
Make sure that you're opening your PR from your
devbranch into thercos/observatorydevbranch -
Reference the GitHub issue inside the pull request comment with the GitHub issue number
-
Click
Create Pull Requestand your code will be reviewed by an administrator.
Class Content
Assignments
This is where Team FRMA maintains a record of all deliverables for the Fall 2018 semester.
Assignment 13
- OE13_FRMA_Ontology
- OE13_FRMA_CuratedTermList
- OE13_FRMA_UseCase
- OE13_FRMA_ConceptualMap
- OE13_FRMA_FinalReportDraft
- OE13_FRMA_SPARQL_Queries
- OE_13_FRMA_Final_Presentation
Assignment 12
- OE12_FRMA_Ontology
- OE12_FRMA_CuratedTermList
- OE12_FRMA_UseCase
- OE12_FRMA_ConceptualMap
- OE12_FRMA_ConceptualMapReport
- OE12_FRMA_Report
- OE12_FRMA_FinalReportOutline
- OE12_FRMA_SPARQL_Queries
Assignment 11
- OE11_FRMA_Ontology
- OE11_FRMA_CuratedTermList
- OE11_FRMA_UseCase
- OE11_FRMA_ConceptualModel
- OE11_FRMA_Report
- OE11_FRMA_SPARQL_Queries
Assignment 10
- OE10_FRMA_Ontology
- OE10_FRMA_CuratedTermList
- OE10_FRMA_UseCase
- OE10_FRMA_ConceptualModel
- OE10_FRMA_Report
- OE10_FRMA_SPARQL_Queries
Assignment 9
- OE9_FRMA_Ontology
- OE9_FRMA_CuratedTermList
- OE9_FRMA_UseCase
- OE9_FRMA_ConceptualModel
- OE9_FRMA_Report
Assignment 8
- OE8_FRMA_Ontology
- OE8_FRMA_CuratedTermList
- OE8_FRMA_UseCase
- OE8_FRMA_ConceptualModel
- OE8_FRMA_Report
Assignment 7
- OE7_FRMA_Ontology
- OE7_FRMA_CuratedTermList
- OE7_FRMA_UseCase
- OE7_FRMA_ConceptualModel
- OE7_FRMA_Report
Assignment 6
- OE6_FRMA_Ontology
- OE6_FRMA_CuratedTermList
- OE6_FRMA_UseCase
- OE6_FRMA_ConceptualModel
- OE6_FRMA_Report