Publications

2024

1. NIST TRECVID

   

   PolySmart @ TRECVid 2024 Medical Video Question Answering

   Jiaxin Wu, Yiyang Jiang, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , 2024

   Abs Bib Website

   Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical question based on the similarity of video transcript and answers generated by GPT4. For the visual answer localization, the start and end timestamps of the answer are predicted by the alignments on both visual content and subtitles with queries. For the Query-Focused Instructional Step Captioning (QFISC) task, the step captions are generated by GPT4. Specifically, we provide the video captions generated by the LLaVA-Next-Video model and the video subtitles with timestamps as context, and ask GPT4 to generate step captions for the given medical query. We only submit one run for evaluation and it obtains a F-score of 11.92 and mean IoU of 9.6527.

   @inproceedings{wu2024polysmarttrecvid2024,
     title = {PolySmart @ TRECVid 2024 Medical Video Question Answering},
     author = {Wu, Jiaxin and Jiang, Yiyang and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

2. NIST TRECVID

   

   PolySmart @ TRECVid 2024 Video-To-Text

   Jiaxin Wu, Wengyu Zhang, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , 2024

   Abs Bib Website

   In this paper, we present our methods and results for the Video-To-Text (VTT) task at TRECVid 2024, exploring the capabilities of Vision-Language Models (VLMs) like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content. We investigate the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency. Our analysis reveals that fine-tuning substantially improves the model’s ability to produce more detailed and domain-aligned text, bridging the gap between generic VLM tasks and the specialized needs of VTT. Experimental results demonstrate that our fine-tuned model outperforms baseline VLMs across various evaluation metrics, underscoring the importance of domain-specific tuning for complex VTT tasks.

   @inproceedings{wu2024polysmarttrecvid2025,
     title = {PolySmart @ TRECVid 2024 Video-To-Text},
     author = {Wu, Jiaxin and Zhang, Wengyu and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

3. NIST TRECVID

   

   PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search

   Jiaxin Wu, Chong-Wah Ngo, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , 2024

   Abs Bib Website

   This year, we explore generation-augmented retrieval for the TRECVid AVS task. Specifically, the understanding of textual query is enhanced by three generations, including Text2Text, Text2Image, and Image2Text, to address the out-of-vocabulary problem. Using different combinations of them and the rank list retrieved by the original query, we submitted four automatic runs. For manual runs, we use a large language model (LLM) (i.e., GPT4) to rephrase test queries based on the concept bank of the search engine, and we manually check again to ensure all the concepts used in the rephrased queries are in the bank. The result shows that the fusion of the original and generated queries outperforms the original query on TV24 query sets. The generated queries retrieve different rank lists from the original query.

   @inproceedings{wu2024polysmartvireotrecvid,
     title = {PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search},
     author = {Wu, Jiaxin and Ngo, Chong-Wah and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

4. ACM MM

   

   Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval

   Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiaoyong Wei, Chang Wen Chen, and Qing Li

   In ACM Multimedia 2024 , 2024

   Abs arXiv Bib Code Website

   In this paper, we investigate the feasibility of leveraging large language models (LLMs) for integrating general knowledge and incorporating pseudo-events as priors for temporal content distribution in video moment retrieval (VMR) models. The motivation behind this study arises from the limitations of using LLMs as decoders for generating discrete textual descriptions, which hinders their direct application to continuous outputs like salience scores and inter-frame embeddings that capture inter-frame relations. To overcome these limitations, we propose utilizing LLM encoders instead of decoders. Through a feasibility study, we demonstrate that LLM encoders effectively refine inter-concept relations in multimodal embeddings, even without being trained on textual embeddings. We also show that the refinement capability of LLM encoders can be transferred to other embeddings, such as BLIP and T5, as long as these embeddings exhibit similar inter-concept similarity patterns to CLIP embeddings. We present a general framework for integrating LLM encoders into existing VMR architectures, specifically within the fusion module. The LLM encoder’s ability to refine concept relation can help the model to achieve a balanced understanding of the foreground concepts (e.g., persons, faces) and background concepts (e.g., street, mountains) rather focusing only on the visually dominant foreground concepts. Additionally, we introduce the concept of pseudo-events, obtained through event detection techniques, to guide the prediction of moments within event boundaries instead of crossing them, which can effectively avoid the distractions from adjacent moments. The integration of semantic refinement using LLM encoders and pseudo-event regulation is designed as plug-in components that can be incorporated into existing VMR methods within the general framework. Through experimental validation, we demonstrate the effectiveness of our proposed methods by achieving state-of-the-art performance in VMR.

   @inproceedings{jiang2024prior,
     title = {Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval},
     author = {Jiang, Yiyang and Zhang, Wengyu and Zhang, Xulu and Wei, Xiaoyong and Chen, Chang Wen and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

5. ACM MM

   

   A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning

   Changmeng Zheng, DaYong Liang, Wengyu Zhang, Xiaoyong Wei, Tat-Seng Chua, and Qing Li

   In ACM Multimedia 2024 , 2024

   Abs arXiv Bib Code Website

   This paper presents a pilot study aimed at introducing multi-agent debate into multimodal reasoning. The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images. These challenges stem from the inductive (bottom-up) nature of existing debating schemes. To address the issue, we propose a deductive (top-down) debating approach called Blueprint Debate on Graphs (BDoG). In BDoG, debates are confined to a blueprint graph to prevent opinion trivialization through world-level summarization. Moreover, by storing evidence in branches within the graph, BDoG mitigates distractions caused by frequent but irrelevant concepts. Extensive experiments validate BDoG, achieving state-of-the-art results in Science QA and MMBench with significant improvements over previous methods.

   @inproceedings{zheng2024a,
     title = {A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning},
     author = {Zheng, Changmeng and Liang, DaYong and Zhang, Wengyu and Wei, Xiaoyong and Chua, Tat-Seng and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

6. ACM MM

   

   Generative Active Learning for Image Synthesis Personalization

   Xulu Zhang, Wengyu Zhang, Xiaoyong Wei, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li

   In ACM Multimedia 2024 , 2024

   Abs arXiv Bib Code Website

   This paper presents a pilot study that explores the application of active learning, traditionally studied in the context of discriminative models, to generative models. We specifically focus on image synthesis personalization tasks. The primary challenge in conducting active learning on generative models lies in the open-ended nature of querying, which differs from the closed form of querying in discriminative models that typically target a single concept. We introduce the concept of anchor directions to transform the querying process into a semi-open problem. We propose a direction-based uncertainty sampling strategy to enable generative active learning and tackle the exploitation-exploration dilemma. Extensive experiments are conducted to validate the effectiveness of our approach, demonstrating that an open-source model can achieve superior performance compared to closed-source models developed by large companies, such as Google’s StyleDrop. The source code is available at https://github.com/zhangxulu1996/GAL4Personalization

   @inproceedings{zhang2024generative,
     title = {Generative Active Learning for Image Synthesis Personalization},
     author = {Zhang, Xulu and Zhang, Wengyu and Wei, Xiaoyong and Wu, Jinlin and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

7. AAAI

   

   Compositional inversion for stable diffusion models

   Xulu Zhang, Xiao-Yong Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, and Qing Li

   In Proceedings of the AAAI Conference on Artificial Intelligence , 2024

   Abs Bib Paper Code

   Inversion methods, such as Textual Inversion, generate personalized images by incorporating concepts of interest provided by user images. However, existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. It stems from the fact that during inversion, the irrelevant semantics in the user images are also encoded, forcing the inverted concepts to occupy locations far from the core distribution in the embedding space. To address this issue, we propose a method that guides the inversion process towards the core distribution for compositional embeddings. Additionally, we introduce a spatial regularization approach to balance the attention on the concepts being composed. Our method is designed as a post-training approach and can be seamlessly integrated with other inversion methods. Experimental results demonstrate the effectiveness of our proposed approach in mitigating the overfitting problem and generating more diverse and balanced compositions of concepts in the synthesized images. The source code is available at https://github.com/zhangxulu1996/Compositional-Inversion.

   @inproceedings{zhang2024compositional,
     title = {Compositional inversion for stable diffusion models},
     author = {Zhang, Xulu and Wei, Xiao-Yong and Wu, Jinlin and Zhang, Tianyi and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
     paper = {https://ojs.aaai.org/index.php/AAAI/article/view/28565},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     volume = {38},
     number = {7},
     pages = {7350--7358},
     year = {2024},
     publisher = {AAAI},
   }