Commit e4502758 authored by Robert Schaffenrath's avatar Robert Schaffenrath
Browse files

updated the project evaluation draft

parent 7e81d525
......@@ -6,6 +6,7 @@
\usepackage{lmodern}
\usepackage{titling}
\usepackage{amsmath}
\usepackage{hyperref}
\usepackage{float}
\usepackage{cite}
\usepackage{url}
......@@ -30,19 +31,20 @@
\section{The research question}
The objective of the research `Empirical Evaluation of SHACL Implementations in Graph Databases' is to evaluate what Graph Databases support SHACL, how it's implemented and how it performs. The research questions are therefore: What Graph Databases support SHACL? How does the SHACL implementation perform? How good is the documentation for using the SHACL implementations.
The objective of the research `Empirical Evaluation of SHACL Implementations in Graph Databases' is to evaluate which graph databases support SHACL and how they perform. The research questions are therefore: Which graph databases support SHACL? How does the SHACL implementation perform? How good is the documentation for using the SHACL implementations.
\section{Solution}
The research was conducted by first evaluating which graph databases, among all those available at the time of writing, support SHACL. The databases were identified by searching for references to the support of this function on the homepage or in the documentation. Search results led to the selection of Stardog, Apache Jena, RDF4J, GraphDB and AllegroGraph.
The research group `STI Innsbruck' provided us with the Tyrolean Graph for evaluation. From the two possible options: one of 5 GiB and another of 13 GiB, both containing one triple or quad per line we first tried to use the 5 GiB but ran into problems and had to restrict the number of tuples to 1 million of the 30 million that were present in the 5 GiB file.
Subsequently, the SHACL constraints were defined. The shapes have been defined considering the domains present in the graph and with the guidance of the Tourism Scheme domain specification. The constructed shapes consider class type, propriety type, cardinality, value pattern and value range.
Thereafter, the shapes have been validated on all selected databases. The methodology to have a database report on the constraint violation defined in the SHACL file is different between databases. In some it was enough to write and execute a bash script, in others it was necessary to use the database API and dig deeper into the provided documentation.
Finally, the speed and usability of the SHACL implementation was evaluated. With regard to speed, the time needed to obtain a SHACL report was measured and compared. The test was performed eight times in order to obtain reliable measurements. Usability, on the other hand, was evaluated by reporting the difficulties encountered in implementing SHACL, considering whether the documentation provided was sufficient or whether it was necessary to consult external sources.
The research was conducted by first evaluating which graph databases support SHACL at the time of writing. The databases were identified by searching for references of mentioned SHACL support on the homepages of the providers, in the documentations and in the source code if it was available. Search results led to the selection of Stardog, Apache Jena, RDF4J, GraphDB and AllegroGraph.
The research group `STI Innsbruck' provided us with the Tyrolean Graph, which was used as the main dataset for the evaluation. From two feasible dataset options: one with the size of 5 GB and another of 13 GB, both containing one quad per line. Because we had limited memory available on test machine, we decided to use the 5 GB version. Due to unexpected high memory consumption of RDF4J, we had to limit the dataset even further to 160 MB, which corresponds to 1 million quads in total.
For the Tyrolean Graph we defined our own SHACL constraints, which have been constructed regarding the tourism domain and with the guidance of the Tourism Scheme domain specification from \href{https://ds.sti2.org/}{https://ds.sti2.org/}. The constructed shapes consider class type, propriety type, cardinality, value pattern and value range.
The methodology to obtain a validation report differs between the used databases. For Stardog and AllegroGraph it was only necessary to execute the client program and specify the SHACL shapes as arguments. Because RDF4J and Jena are mainly frameworks, it was necessary to implement an application in Java that validates the dataset from a provided triplestore.
Finally, the speed and usability of the SHACL implementation was evaluated. With regard to speed, the time needed to obtain a SHACL report was measured and compared. The test was performed eight times in order to obtain reliable measurements. Usability, on the other hand, was evaluated by reporting the difficulties encountered in using SHACL with a specific graph database, considering whether the provided documentation was sufficient or whether it was necessary to consult external sources.
\section{Evaluation of the project}
The study was conducted by installing the databases on a single machine. The machine has an Intel Core i7-6700K @ 4.00GHz processor, 16GB PC 3200 DDR4 RAM and a Kingston A400 240GB SSD.
The criteria used to assess performance were the measurement of SHACL implementation time in seconds. Because the report output of the constraint violations differs dramatically between databases, we opted out on comparing the results. We also don't have a ground truth for violations we could compare single outputs to. The method used in the evaluation was to measure the implementation eight times on the same database under the same conditions to increase the accuracy of the results.
In addition, information was reported regarding the difficulty in implementing SHACL, the usefulness of the documentation and the quality of the SHACL report produced by the databeses. Unlike previous measurements, these are of a qualitative nature and consist of a textual description.
The criterion used to assess performance was the time measurement of generating a SHACL report. Because the structure and provided information of the generated reports differ between databases and size of the reports was around 1.8 GB, we were not able to compare them.
The method used in the evaluation was to measure the SHACL implementation eight times on the same database under the same conditions to increase the accuracy of the results.
In addition, information was reported regarding the difficulty of using SHACL, the usefulness of the documentation and the quality of the SHACL report produced by the databases. Unlike previous measurements, these are of a qualitative nature and consist of a textual description.
\end{document}
\ No newline at end of file
\end{document}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment