Introduction to Knowledge Graph Completion with LLMs

Introduction

Knowledge Graph Completion (KGC) is the process of predicting missing entities or relationships in a Knowledge Graph (KG) to make it more comprehensive. This task is crucial because most real-world knowledge graphs suffer from incompleteness and sparsity, meaning they lack crucial facts.

KGs serve as the backbone of many artificial intelligence applications, providing structured representations of entities and their relationships. They enable AI systems to perform semantic search, recommendation, question answering, and reasoning by organizing information into a network of interconnected nodes. KGs have been proven to be more reliable for example in GraphRAG settings but despite their advantages, KGs remain incomplete, leading to gaps in knowledge representation and limiting AI’s reasoning capabilities

For years, KGC has relied on structural models to fill in missing information. Translational models like TransE and TransH map entities and relations into vector spaces, while semantic matching models such as DistMult analyze entity similarities. CNN-based approaches add complexity by capturing relational patterns. But these methods have a major flaw, they depend solely on existing triples, struggling when data is sparse or incomplete. Without external context or deeper reasoning, they miss subtle connections. As knowledge graphs grow, it’s clear that a shift toward language-driven approaches, powered by large language models, is the next evolution

Key Challenges in KGC:

1. Data Sparsity and Incompleteness:
   
   • Many entities and relationships are underrepresented in KGs, making it difficult for models to generalize.
   • Rare or emerging entities have limited data, impacting model performance.
2. Structural Limitations:
   
   • Traditional knowledge graph embedding models rely heavily on structural relationships, ignoring textual descriptions that could provide additional context.
   • Existing embedding-based methods struggle with inductive reasoning, meaning they cannot easily infer new facts for unseen entities.
3. Scalability Issues:
   
   • As KGs grow, computational complexity increases, making it difficult for traditional models to process vast amounts of interconnected data.
   • Large-scale KGs require efficient algorithms to handle billions of nodes and edges.
4. Generalization Beyond Training Data:
   
   • Many KGC models are task-specific and fail to generalize beyond the training dataset.
   • A robust KGC model should be able to transfer knowledge to unseen triples without requiring extensive retraining.

Leveraging Large Language Models (LLMs)

Recent advancements in LLMs have gained significant attention for their ability to perform complex reasoning and language understanding tasks.  Research has established that scaling pre-trained language models often enhances their capacity for downstream tasks such as Knowledge Graph Completion (KGC). Unlike smaller pre-trained models like BERT, LLMs demonstrate emergent abilities, such as in-context learning, instruction following, and step-by-step reasoning, which are particularly beneficial for KGC tasks.

Vendors like Neo4j ,TigerGraph, and Triplex have made notable advancements in the field of KGC and graph-based AI. Neo4j,  has introduced enhanced algorithms for link prediction and node classification, facilitating more accurate and efficient knowledge graph completions. Their focus on integrating LLM capabilities for natural language-based queries has also improved accessibility for users who are not well-versed in Cypher. TigerGraph CoPilot combines two of the most powerful technologies for extracting value from a knowledge base , LLMs and graph databases. Their recent developments include optimized query execution and the integration of graph neural networks (GNNs) to better capture and infer missing relationships within knowledge graphs. Triplex on another hand is a model that addresses the challenges of converting unstructured data into Knowledge Graphs. Triplex aims to enhance the Knowledge Graph Completion (KGC) process, by converting unstructured text into semantic triples. The model claims to accurately extract entities, relationships, and attributes to fill gaps in existing Knowledge Graphs. The variety of solutions in KGC’s are in response to the demand around Reliable AI , which has become essential in major sectors such as finance, legal and healthcare.

KG-LLM Framework

Traditional Knowledge Graph KGC methods rely on structured embeddings, but KG-LLM transforms the problem into a text-based task. Instead of treating knowledge graphs as node-edge structures, entities, relations, and triples are represented as natural language sequences. This enables LLMs to process and complete missing knowledge using contextual reasoning. By reformulating KG tasks into prompts, LLMs can classify triples, predict missing relations, and infer entities. This approach leverages the language model’s ability to understand and generate human-like responses, making knowledge graph completion more flexible and scalable.

Rather than treating knowledge graphs as rigid structures of nodes and edges, KG-LLM represents entities, relations, and triples as textual sequences. This transformation allows LLMs to process knowledge graph data similarly to natural language, enabling them to understand contextual relationships between entities.

Key KGC Tasks

KG-LLM addresses three primary tasks in Knowledge Graph Completion by leveraging natural language processing and prompt-based learning.

1. Triple Classification

This task determines whether a given triple is true or false. A knowledge graph triple, such as (Steve Jobs, founded, Apple Inc.), is converted into a question: "Is this true: Steve Jobs founded Apple Inc.?" The model then evaluates the plausibility of the statement and provides a binary response. This approach helps filter incorrect or incomplete triples from a knowledge graph.

2. Relation Prediction

In cases where two entities are known, but their relationship is missing, the model predicts the appropriate connection. For example, given "Steve Jobs" and "Apple Inc.", the system answers: "What is the relationship between Steve Jobs and Apple Inc.?" and selects the correct relation, such as "founded." This method is particularly useful for expanding knowledge graphs with accurate relational links.

3. Entity (Link) Prediction

Entity prediction focuses on inferring missing entities when either the head or tail entity of a triple is unknown. For example:

• Given "Steve Jobs founded ___?", the model predicts "Apple Inc."
• Given "Who founded Apple Inc.?", the model identifies "Steve Jobs."

A note on Ontologies

Ontologies are formal representations of a set of concepts within a domain and the relationships between those concepts. In the context of knowledge graphs, they provide a structured framework that ensures entities and relationships are represented accurately and consistently. While KGs capture entities and their interrelations, ontologies define the types of entities, permissible relationships, and constraints on those relationships.

For KGC, ontologies are essential as they guide the model to predict relationships that are both accurate and domain-compliant. For example, an ontology might define the relationship between "product" and "category" in an e-commerce knowledge graph or between "employee" and "department" in a corporate knowledge graph.

When combined with large language models (LLMs), ontologies enhance the process by ensuring predictions remain consistent with domain-specific rules. By respecting these ontologies, LLMs can make predictions that adhere to structured knowledge representations, resulting in a more reliable and semantically rich knowledge graph.

Integrating ontologies with models at Lettria enhances the precision and consistency of knowledge graph completion, particularly in areas where complex domain knowledge and relationships are involved.

Evaluation of Text2Graph for KGC

At Lettria, we evaluated the performance of our models in KGC using the Open Source Text to Knowledge Graph Benchmark tool (Text2KGBench). This tool measures how well language models generate knowledge graphs from natural language text while adhering to a specific ontology. We utilized our Text2Graph pipeline to create the ontologies for this evaluation.

Text2KGBench is designed to test how models extract facts from text while ensuring compliance with the given ontology (concepts, relations, domain/range constraints), and ensuring accuracy in representing the input sentences. By using this benchmark, we assessed how effectively Lettria's models generate and complete knowledge graphs, ensuring that the results are not only accurate but also aligned with specific use cases and domain knowledge.

For this evaluation, we tested a JSON-based method with three different prompting techniques for large language models (LLMs):

• Zero-shot
• Few-shot
• Few-shot (Gen)

Additionally, we applied these methods to twelve different LLMs:

• GPT-4
• GPT-4 Turbo
• GPT-4o
• GPT-4o mini
• Gemini 1.5 Pro
• Gemini 1.5 Flash
• Claude 3.5 Sonnet
• Claude 3 Opus
• Mistral Large
• CodeStral
• DeepSeek-Coder-V2
• Triplex

Shot Definitions:

We ran 84 configurations (7 different shots × 12 LLMs) with 19 datasets from Text2KGBench to evaluate the ability of these models to generate and complete knowledge graphs for KGC tasks.

What Does Text2KGBench Measure in KGC?

Text2KGBench evaluates several key metrics to assess the performance of models in completing knowledge graphs:

Through this rigorous evaluation process, Lettria tested 84 different configurations of our KGC models using Text2KGBench, ensuring that we not only meet high standards of accuracy but also address the complex requirements of domain-specific knowledge graph completion.

To streamline our analysis, we summarize the global averages for each of the 84 configurations across the 12 LLMs, resulting in a total of 189 models evaluated.

F1 Evaluation in KGC with Text2Graph

The F1 score provides a combined measure of precision and recall, offering a single value that makes it easier to compare our 7 models across the 12 different LLMs. This metric is particularly useful for evaluating how well the models balance accuracy (precision) and completeness (recall) in their knowledge graph completion tasks.

Hallucinations in KGC

Hallucinations occur when an LLM model generates information that is incorrect or misleading, which is especially critical in data models. Reducing hallucinations is vital for ensuring the reliability of knowledge graph completion. In our analysis, we specifically tested hallucinations for all three components of the knowledge graph triple: subject, relation, and object.

‍

‍

‍

Conclusion

In this evaluation of Knowledge Graph Completion using Lettria's Text2Graph pipeline, we have demonstrated the power and flexibility of LLMs in completing knowledge graphs based on domain-specific ontologies. By leveraging the Text2KGBench benchmark, we thoroughly tested various models and methods, showcasing how Lettria’s KGC solutions can effectively fill gaps in knowledge while maintaining high precision and minimal hallucinations.

Our results indicate that, while models performed well in generating accurate relations, there is still room for improvement in minimizing hallucinations in subjects and objects. As the field of KGC continues to evolve, Lettria remains committed to refining our models, ensuring that they deliver precise, relevant, and reliable knowledge graph completions.

By embracing language-driven approaches and utilizing cutting-edge LLMs, Lettria is positioned to lead advancements in KGC, making AI-driven knowledge more accessible and actionable across industries.