Amos Eaton 214
WHAT: "The Large Language Model for Mixed Reality (LLMR)"
LEADER: Bill Huang
VIDEO: https://youtu.be/efLGnsV4LIE
EVENT PAGE: https://tw.rpi.edu/media/foci-genai-users-group-large-language-model-mi…
WHEN: 6p, 31 Jan
CONTACT: Aaron Green <greena12@rpi.edu>
DESCRIPTION: We present the Large Language Model for Mixed Reality (LLMR), an innovative framework for the real-time creation and modification of interactive virtual experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR's cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. In addition, LLMR explores a novel approach for real-time animation generation on rigged models via natural language inputs. With few-shot demonstrations, LLMR can output structured texts that are parsed into diverse and realistic animations. We showcase the robustness of our approach through qualitative results on various actuated models and motions.
BIO: Bill Huang is a fifth year PhD student in the mathematics department of RPI. His main research area lies in the intersection between generative modeling and mean-field game theory. In particular, he is interested in applying deep generative architectures to solve mean-field systems efficiently and leveraging the mean-field game framework to improve fundamental aspects of generative modeling..
A recording of this talk is available here...
Recordings of previous FOCI GenAI Users Group sessions: