Instruction-guided visual embeddings and feedback-based learning in large vision-language models

Inventors

Chen, Yangyi • Sikka, Karan • COGSWELL, Michael A. • Divakaran, Ajay

Assignees

SRI International Inc

Abstract

In an example, a method for fine-tuning a Large Visual Language Model (LVLM) includes providing visual queries, each of the visual queries comprises at least an image and a textual query related to the image; processing, by the LVLM, the visual queries to extract visual embeddings from the visual queries, wherein the LVLM comprises a Visual Language Model (VLM), a first Large Language Model (LLM), and a linear projection layer interconnecting the VLM and the LLM; for visual queries: i) generating, by the LVLM, a response to the corresponding visual query based on the corresponding visual embedding; ii) evaluating, by a second LLM, the generated response to verify that the generated response satisfies predefined criteria; and iii) providing, by the second LLM, a feedback to the LVLM, in response to the evaluating the generated response; and fine-tuning the LVLM using aggregated feedback provided by the second LLM for the visual queries.

Core Innovation

The disclosure describes instruction-guided training of a Large Visual Language Model (LVLM) that connects a pretrained Large Language Model (LLM) with a Visual Language Model (VLM) using a trainable linear projection layer to produce instruction-aware visual embeddings. The LVLM processes visual queries that each comprise at least an image and a textual query related to the image, and the VLM portion extracts one or more visual embeddings for generating responses.

After supervised fine-tuning, the LVLM is fine-tuned further using feedback-based reinforcement learning in which a second LLM evaluates generated responses. For each visual query, the LVLM generates a response based on the visual embeddings, and the second LLM evaluates the generated response to verify that it satisfies predefined criteria.

The second LLM provides aggregated feedback to the LVLM, where the predefined criteria include helpfulness, honesty, and harmlessness, and the feedback includes natural-language feedback comprising a numerical score, critique feedback, and refinement feedback. The disclosure states that fine-tuning uses aggregated second-LLM feedback across multiple visual queries, combining conditional reinforcement learning with rejection sampling, and is described using RLAIF and natural-language feedback conditioning into subsequent turns.

Claims Coverage

The partial content includes two independent claims, a method and a computing system, that cover generating responses for image-and-text visual queries with an LVLM, using a second LLM to evaluate responses against predefined criteria, and fine-tuning the LVLM with aggregated feedback. The inventive features across the independent claims align on the evaluation-and-feedback loop and the aggregated fine-tuning based on second-LLM feedback.

Across the independent claims, the central coverage is generating responses from an LVLM to image-and-text visual queries, evaluating those responses with a second LLM against one or more predefined criteria, and fine-tuning the LVLM using aggregated feedback provided by the second LLM.

Stated Advantages

Documented Applications