7 min read
Introduction
This benchmark report provides a comprehensive analysis of the performance of various ontology generation models, with a specific focus on the GPT and Sonnet models. The ontologies were generated using our Ontology-Toolkit. By evaluating these models across diverse use cases (UC) and configurations, this report aims to offer insights into their effectiveness in producing accurate and relevant ontologies.
Ontology. We evaluate our approach using a real-world use case by comparing the ontology generated by Ontology-Toolkit with a reference ontology manually developed by professional knowledge engineers. This reference ontology belongs to the financial domain and enables to provide information about companies as well as specific financial events. It comprises 37 classes, 54 object properties, and 48 data properties. It also follows an event-based design pattern, primarily created to capture information on events affecting companies such as mergers, acquisitions, settlements, and legal disputes. The Event superclass plays a central role, with no fewer than 15 subclasses, including MergerEvent, SettlementEvent, and LegalDisputeEvent. The ontology also includes 20 object properties indicating the participation of companies and organizations in these events, such as biddingParticipant, regulatorParticipant, and victimParticipant.
Experiments. We used the Ontology-Toolkit to generate an ontology from a corpus of 30 documents on financial events from BBC and Reuters, averaging 900 tokens each. The domain is defined as market-moving events, and the use case (UC) is Assess and analyze the impact of market-moving events, the parties involved, and their subsequent effects.
In this quantitative evaluation, we have set the number of classes and competency questions to generate to 50 and focus the evaluation on the question: is specifying a use case when modeling an ontology useful or not? Both configurations will be named with_use_case_50 and no_use_case_50.
These configurations are based on the results of a previous evaluation, in which 12 ontologies were generated by considering various factors: the number of classes and questions in the prompt (10, 30, 50), the presence or absence of a UC, and the use of a prompt chaining step. The best results were achieved when 50 questions were asked with and without use case and no prompt chaining step.
Six ontologies were generated for this experiment, specifying a use case or not and using three LLMs: Claude 3.5 Sonnet, GPT-4o, and GPT-4o-mini. This setup allowed for a comparative analysis of the models’ performance based on the presence or absence of an explicit use case.
Generated Ontologies
- Claude 3.5 Sonnet_with_use_case_50
- Claude 3.5 Sonnet_no_use_case_50
- GPT-4o_with_use_case _50
- GPT-4o_no_use_case _50_
- GPT-4o-mini_with_use_case _50
- GPT-4o-mini_no_use_case _50
Evaluation results. For each generated ontology, a manual analysis was conducted, comparing the presence of classes and properties between the generated and reference ontologies. We also assessed the presence of hallucinations. We had a degree of tolerance in the comparison: missing classes or properties may correspond to elements with broader or narrower meanings, which were noted separately. For example, the object property hasLocation is broader than country, while the data property usesCryptocurrency is narrower than currency. Two sets of evaluations were conducted using different metrics: (1) Accuracy, and (2) Precision, Recall, and F1 Score.
- Accuracy: The overall proportion of matching classes and properties between the generated and reference ontologies.
- Precision: The ratio of concepts or relations in the generated ontology that are present in the reference ontology.
- Recall: The ratio of concepts or relations in the reference ontology that are also present in the generated ontology.
- F1 Score: The harmonic mean of precision and recall.
Accuracy
As seen in Table 1, the Claude 3.5 Sonnet shows stable results both for ontologies with and without UC. GPT-4o-Mini comes second, particularly for classes, and although more limited in the precision of its properties, also shows an improvement with UC. Finally, GPT-4o shows weaker results, particularly in the generation of properties with UC. These results suggest that the integration of a UC enhances the consistency of ontologies generated by Claude 3.5 Sonnet and GPT-4o-Mini, but less so for those generated with GPT-4o.
Comparing the three models, Claude 3.5 Sonnet stands out for its balanced performance, both with and without a use case, offering stable accuracy in the generation of classes and properties.
(Table 1 : Accuracy comparison of 6 ontologies generated using 3 LLMs with respect to a reference ontology)
Precision, Recall, and F1 Score
Based on F1 Scores (Table 2), the Claude 3.5 Sonnet model offers the best overall results, with a clear improvement when UCs are taken into account. It outperforms the other models in class generation (0.76 with UC), although its performance on properties is limited. GPT-4o shows a good F1 Score for classes with UC (0.70), but drops drastically without this option. Finally, GPT-4o-Mini shows stable but weaker results than the other two models, particularly when it comes to property generation. As a result, Claude 3.5 Sonnet stands out especially for ontologies with UC.
(Table 2 : Precision, Recall, and F1 score comparison of 6 ontologies using 3 LLMs with respect to a reference ontology)
Conclusion
To sum up, our work shows that Claude 3.5 Sonnet offers stable performance whether or not the use case is specified. GPT-4o-mini ranks second, ahead of GPT-4o. But we also note that adding the use case is not always beneficial: in this case, it improves the consistency of ontologies generated by Claude 3.5 Sonnet and GPT-4o-mini, but has less impact on GPT-4o.