In this talk John Erickson discusses work related to ChatBS-NexGen, as presented in the recent Journal of Web Semantics article, "LLM Experimentation through Knowledge Graphs: Towards Improved Management, Repeatability, and Verification." John reviews the experiment described in JoWS and discusses current work to extend ChatBS to more precisely implement the ChatBS-NexGen architecture described.
VIDEO: https://youtu.be/x9P4csZTpx0
SLIDES: https://bit.ly/chatbs-slides-feb25
JoWS LINK: https://www.sciencedirect.com/science/article/pii/S1570826824000398
ABSTRACT: Generative large language models (LLMs) have transformed AI by enabling rapid, human-like text generation, but they face challenges, including managing inaccurate information generation. Strategies such as prompt engineering, Retrieval-Augmented Generation (RAG), and incorporating domain-specific Knowledge Graphs (KGs) aim to address their issues. However, challenges remain in achieving the desired levels of management, repeatability, and verification of experiments, especially for developers using closed-access LLMs via web APIs, complicating integration with external tools. To tackle this, we are exploring a software architecture to enhance LLM workflows by prioritizing flexibility and traceability while promoting more accurate and explainable outputs. We describe our approach and provide a nutrition case study demonstrating its ability to integrate LLMs with RAG and KGs for more robust AI solutions.
CITATION: Erickson, J. S., Santos, H., Pinheiro, V., McCusker, J. P., & McGuinness, D. L. (2025). LLM experimentation through knowledge graphs: Towards improved management, repeatability, and verification. In Journal of Web Semantics (Vol. 85, p. 100853). Elsevier BV. https://doi.org/10.1016/j.websem.2024.100853