- AI systems often incorporate training data sourced from human feedback.
- These “Human in the Loop,” or HITL, methods vary in their effectiveness.
- Researchers are comparing four common HITL methods to determine which, if any, are the most effective
Artificial intelligence modalities like large language models and machine learning derive their insights from repetitive, iterative training on massive models of even more massive datasets. Yet these complex computing systems also benefit greatly from human intervention.
Researchers from the University of Florida, the New Jersey Institute of Technology and the University of Texas at Dallas are set to publish a paper in “ACM Transactions on Interactive Intelligent Systems” that examines common methods for including human feedback in large language models like ChatGPT to determine how effective those methods are.
Approaches to machine learning that include humans in the feedback loop, referred to as HITL, specifically incorporate human feedback into the results generated by AI systems to improve their accuracy and performance. The better the models perform under the hood, the more accurate and meaningful the results are for end users.
In their paper, “Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models,” the authors focused on one specific task for humans: text-based labeling and verification of AI-generated video descriptions. In this task, users reviewed a video clip alongside a textual description produced by a vision-language model and provided feedback indicating whether the description was correct and, if not, how it should be revised.
PhD student, CISE
“The participants looked at a video clip, just a few seconds, where a person was doing some activities and they see a text description of what’s happening generated from AI models,” described Reza Shahriari, doctoral candidate in UF’s Department of Computer & Information Science & Engineering, known as CISE, and lead author on the paper. “The goal was to say whether the description was correct or not.”
The participants were divided into four groups according to the type of feedback they would provide for the AI model. One group simply provided binary feedback (yes/no), one selected the text that was incorrect, one corrected the problematic text and one group used free-response text using natural language. The researchers then analyzed the relationship between human feedback and machine learning outcomes.
The results? Models that were trained with rich feedback (free-response or error correction) yielded more accurate models. But there was a tradeoff. The human attention and effort required for rich responses was tiring for participants. Performance dropped over time as humans became bored or fatigued.
In the end, Shahriari and his co-authors found that when it comes to keeping a human in the loop with AI training, results vary. The authors suggest that the feedback method employed might depend on the specific type of task and the level of accuracy that is required.
“These models are not perfect. People trust them a lot when they Google something, or search on ChatGPT; they think, ‘Oh, of course this is correct.’” Shahriari noted. “But this is not true in many cases. That’s why we want to understand which types of human feedback are most effective at improving these models.”
Co-authors include Amal Hashky, Shivvrat Arya, Tyler Audino, Eric D. Ragan, Ph.D., Vibhav Gogate and Jaime Ruiz.