Publications

2026

1. VLDB

   

   Towards robust and interactive text-to-visualization translation: enhancing model robustness and mitigating ambiguity in natural language queries

   Jinwei Lu, Wenxin Xu, Yuanfeng Song, Haodi Zhang, Chen Jason Zhang, Kaishun Wu, Xiaoyong Wei, and Raymond Chi-Wing Wong

   The VLDB Journal, 2026

   Abs Bib Website

   Text-to-Vis is an emerging task in the data engineering and mining area that aims to automatically generate data visualizations from natural language questions (NLQs). Despite their progress, existing text-to-vis models often heavily rely on lexical matching between words in the questions and tokens in data schemas. This overreliance on lexical matching may lead to a diminished level of model robustness against input variations. In this study, we thoroughly examine the robustness of current text-to-vis models, an area that has not previously been explored. In particular, we construct the first robustness dataset nvBench-Rob, which contains diverse lexical and phrasal variations based on the original text-to-vis benchmark nvBench. Then, we found that the performance of existing text-to-vis models on this new dataset dramatically drops, implying that these methods exhibit inadequate robustness overall. To this end, we propose a novel framework based on Retrieval-Augmented Generation (RAG) technique, named GRED, specifically designed to address input perturbations in these two variants. The framework consists of three parts: NLQ-Retrieval Generator, Visualization Query-Retrieval Retuner and Annotation-based Debugger, which are used to tackle the challenges posed by natural language variants, programming style differences and data schema variants, respectively. Furthermore, we extend the framework, named C&C, which further mitigates the uncertainty in visualization caused by ambiguous NLQ. It chooses the clause with the largest expected reduction of uncertainty under an elicited heuristic candidate-confidence measure and interacts with users for clarification to resolve ambiguity. This targeted interaction ensures that the resulting visualization more accurately reflects the user’s actual intent. Extensive experimental evaluations show that, compared to the state-of-the-art model Prompt4Vis in the Text-to-Vis field, GRED and C&C perform better in terms of model robustness, with more than 40% increase in accuracy on the proposed nvBench-Rob dataset.

   @article{lu2026towards,
     title = {Towards robust and interactive text-to-visualization translation: enhancing model robustness and mitigating ambiguity in natural language queries},
     author = {Lu, Jinwei and Xu, Wenxin and Song, Yuanfeng and Zhang, Haodi and Zhang, Chen Jason and Wu, Kaishun and Wei, Xiaoyong and Wong, Raymond Chi-Wing},
     journal = {The VLDB Journal},
     volume = {35},
     number = {5},
     pages = {49},
     year = {2026},
     publisher = {Springer},
   }

2. ICDE

   

   Osmt: Bridging openstreetmap queries and natural language with open-source tag-aware language models

   Zhuoyue Wan, Wentao Hu, Chen Jason Zhang, Yuanfeng Song, Shuaimin Li, Ruiqiang Xiao, Xiao-Yong Wei, and Raymond Chi-Wing Wong

   In 2026 IEEE 42nd International Conference on Data Engineering (ICDE) , 2026

   Abs Bib Website

   Bridging natural language and structured query languages is a long-standing challenge in the database community. While recent advances in language models have shown promise in this direction, existing solutions often rely on large-scale closed-source models that suffer from high inference costs, limited transparency, and lack of adaptability for lightweight deployment. In this paper, we present OsmT, an open-source tag-aware language model specifically designed to bridge natural language and Overpass Query Language (OverpassQL), a structured query language for accessing large-scale OpenStreetMap (OSM) data. To enhance the accuracy and structural validity of generated queries, we introduce a Tag Retrieval Augmentation (TRA) mechanism that incorporates contextually relevant tag knowledge into the generation process. This mechanism is designed to capture the hierarchical and relational dependencies present in the OSM database, addressing the topological complexity inherent in geospatial query formulation. In addition, we define a reverse task, OverpassQL-to-Text, which translates structured queries into natural language explanations to support query interpretation and improve user accessibility. We evaluate OsmT on a public benchmark against strong baselines and observe consistent improvements in both query generation and interpretation. Despite using significantly fewer parameters, our model achieves competitive accuracy, demonstrating the effectiveness of open-source pre-trained language models in bridging natural language and structured query languages within schema-rich geospatial environments.

   @inproceedings{wan2026osmt,
     title = {Osmt: Bridging openstreetmap queries and natural language with open-source tag-aware language models},
     author = {Wan, Zhuoyue and Hu, Wentao and Zhang, Chen Jason and Song, Yuanfeng and Li, Shuaimin and Xiao, Ruiqiang and Wei, Xiao-Yong and Wong, Raymond Chi-Wing},
     booktitle = {2026 IEEE 42nd International Conference on Data Engineering (ICDE)},
     pages = {517--530},
     year = {2026},
     organization = {IEEE},
   }

3. ECCV

   

   OmniColor: A Unified Framework for Multi-modal Lineart Colorization

   Xulu Zhang, Haoqian Du, Xiaoyong Wei, and Qing Li

   ECCV 2026, 2026

   Abs Bib Code Website

   Lineart colorization is a critical stage in professional content creation, yet achieving precise and flexible results under diverse user constraints remains a significant challenge. To address this, we propose OmniColor, a unified framework for multi-modal lineart colorization that supports arbitrary combinations of control signals. Specifically, we systematically categorize guidance signals into two types: spatially-aligned conditions and semantic-reference conditions. For spatially-aligned inputs, we employ a dual-path encoding strategy paired with a Dense Feature Alignment loss to ensure rigorous boundary preservation and precise color restoration. For semantic-reference inputs, we utilize a VLM-only encoding scheme integrated with a Temporal Redundancy Elimination mechanism to filter repetitive information and enhance inference efficiency. To resolve potential input conflicts, we introduce an Adaptive Spatial-Semantic Gating module that dynamically balances multi-modal constraints. Experimental results demonstrate that OmniColor achieves superior controllability, visual quality, and temporal stability, providing a robust and practical solution for lineart colorization.

   @article{zhang2026omnicolor,
     title = {OmniColor: A Unified Framework for Multi-modal Lineart Colorization},
     author = {Zhang, Xulu and Du, Haoqian and Wei, Xiaoyong and Li, Qing},
     journal = {ECCV 2026},
     year = {2026},
   }

4. Bioinformatics

   

   RelAgent: A multi-agent solution for molecular relationship grounding

   Rubing Chen, Jiaxin Wu, Chen Jason Zhang, Xiao-Yong Wei, and Qing Li

   Bioinformatics, 2026

   Abs Bib Code Website

   Molecular captions, patents, and medicinal-chemistry notes describe substructures and their relations in natural language, whereas computational models operate on formal representations such as SMILES. Bridging this semantic–structural gap is important for patent interpretation, structural relationship analysis, and controllable molecular editing, yet current large language models struggle to ground textual references to precise molecular components.

   @article{chen2026relagent,
     title = {RelAgent: A multi-agent solution for molecular relationship grounding},
     author = {Chen, Rubing and Wu, Jiaxin and Zhang, Chen Jason and Wei, Xiao-Yong and Li, Qing},
     journal = {Bioinformatics},
     pages = {btag541},
     year = {2026},
     publisher = {Oxford University Press},
   }

5. ACL (Oral)

   

   Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

   Yiyang Jiang, Li Zhang, Xiao-Yong Wei, and Qing Li

   In ACL 2026 Main , 2026

   Abs arXiv Bib Code Website

   Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods.

   @inproceedings{jiang2026think,
     title = {Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation},
     author = {Jiang, Yiyang and Zhang, Li and Wei, Xiao-Yong and Li, Qing},
     booktitle = {ACL 2026 Main},
     year = {2026},
   }

6. KDD

   

   Chem-r: Learning to reason as a chemist

   Weida Wang, Benteng Chen, Di Zhang, Wanhao Liu, Shuchen Pu, Ben Gao, Jin Zeng, Xiaoyong Wei, Tianshu Yu, Shuzhou Sun, Tianfan Fu, Wanli Ouyang, Lei Bai, Jiatong Li, Zifu Wang, Yuqiang Li, and Shufei Zhang

   In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2026

   Abs arXiv Bib Code Website

   Although large language models (LLMs) have significant potential to advance chemical discovery, current LLMs lack core chemical knowledge, produce unreliable reasoning trajectories, and exhibit suboptimal performance across diverse chemical tasks. To address these challenges, we propose Chem-R, a generalizable Chemical Reasoning model designed to emulate the deliberative processes of chemists. Chem-R is trained through a three-phase framework that progressively builds advanced reasoning capabilities, including: 1) Chemical Foundation Training, which establishes core chemical knowledge. 2) Chemical Reasoning Protocol Distillation, incorporating structured, expert-like reasoning traces to guide systematic and reliable problem solving. 3) Multi-task Group Relative Policy Optimization that optimizes the model for balanced performance across diverse molecular- and reaction-level tasks. This structured pipeline enables Chem-R to achieve state-of-the-art performance on comprehensive benchmarks, surpassing leading large language models, including Gemini-2.5-Pro and DeepSeek-R1, by up to 32% on molecular tasks and 48% on reaction tasks. Meanwhile, Chem-R also consistently outperforms the existing chemical foundation models across both molecular and reaction level tasks. These results highlight Chem-R’s robust generalization, interpretability, and potential as a foundation for next-generation AI-driven chemical discovery.

   @inproceedings{wang2025chem,
     title = {Chem-r: Learning to reason as a chemist},
     author = {Wang, Weida and Chen, Benteng and Zhang, Di and Liu, Wanhao and Pu, Shuchen and Gao, Ben and Zeng, Jin and Wei, Xiaoyong and Yu, Tianshu and Sun, Shuzhou and Fu, Tianfan and Ouyang, Wanli and Bai, Lei and Li, Jiatong and Wang, Zifu and Li, Yuqiang and Zhang, Shufei},
     booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
     year = {2026},
   }

7. KDD

   

   Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation

   Jiatong Li, Junxian Li, Weida Wang, Yunqing Liu, Changmeng Zheng, Dongzhan Zhou, Xiao-yong Wei, and Qing Li

   In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2026

   Abs arXiv Bib Code Website

   Recently, Large Language Models (LLMs) have demonstrated great potential in natural language-driven molecule discovery. However, existing datasets and benchmarks for molecule-text alignment are predominantly built on one-to-one mappings, measuring LLMs’ ability to retrieve a single, pre-defined answer, rather than their creative potential to generate diverse, yet equally valid, molecular candidates. To address this critical gap, we propose Speak-to-Structure (S^2-Bench), the first benchmark to evaluate LLMs in open-domain natural language-driven molecule generation. S^2-Bench is specifically designed for one-to-many relationships, challenging LLMs to exhibit genuine molecular understanding and open-ended generation capabilities. Our benchmark includes three key tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and customized molecule generation (MolCustom), each probing a different aspect of molecule discovery. We also introduce OpenMolIns, a large-scale instruction tuning dataset that enables Llama3.1-8B to surpass the most powerful LLMs like GPT-4o and Claude-3.5 on S^2-Bench. Our comprehensive evaluation of 31 LLMs shifts the focus from simple pattern recall to realistic molecular design, paving the way for more capable LLMs in natural language-driven molecule discovery.

   @inproceedings{li2024speak,
     title = {Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation},
     author = {Li, Jiatong and Li, Junxian and Wang, Weida and Liu, Yunqing and Zheng, Changmeng and Zhou, Dongzhan and Wei, Xiao-yong and Li, Qing},
     booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
     year = {2026},
   }

8. AAAI

   

   OptScale: Probabilistic Optimality for Inference-time Scaling

   Youkang Wang, Jian Wang, Rubing Chen, and Xiao-Yong Wei

   Proceedings of the AAAI Conference on Artificial Intelligence, Mar 2026

   arXiv Bib Code Website

   @article{Wang_Wang_Chen_Wei_2026,
     title = {OptScale: Probabilistic Optimality for Inference-time Scaling},
     volume = {40},
     url = {https://ojs.aaai.org/index.php/AAAI/article/view/40661},
     doi = {10.1609/aaai.v40i40.40661},
     abstractnote = {Inference-time scaling has emerged as a powerful technique for enhancing the reasoning performance of Large Language Models (LLMs). However, existing approaches often rely on heuristic strategies for parallel sampling, lacking a principled foundation. 
         To address this gap, we propose a probabilistic framework that formalizes the optimality of inference-time scaling under the assumption that parallel samples are independently and identically distributed (i.i.d.), and where the Best-of-N selection strategy follows a probability distribution that can be estimated. 
         Within this framework, we derive a theoretical lower bound on the required number of samples to achieve a target performance level, providing the first principled guidance for compute-efficient scaling. Leveraging this insight, we develop OptScale, a practical algorithm that dynamically determines the optimal number of sampled responses. OptScale employs a language model-based predictor to estimate probabilistic prior parameters, enabling the decision of the minimal number of samples needed that satisfy predefined performance thresholds and confidence levels. Extensive experiments on representative reasoning benchmarks (including MATH-500, GSM8K, AIME, and AMC) demonstrate that OptScale significantly reduces sampling overhead while remaining better or on par with state-of-the-art reasoning performance. Our work offers both a theoretical foundation and a practical solution for principled inference-time scaling, addressing a critical gap in the efficient deployment of LLMs for complex reasoning.},
     number = {40},
     journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
     author = {Wang, Youkang and Wang, Jian and Chen, Rubing and Wei, Xiao-Yong},
     year = {2026},
     month = mar,
     pages = {33710-33718}
   }

2025

1. CVPR

   

   Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues

   Sihong Huang, Jiaxin Wu, Xiaoyong Wei, Yi Cai, Dongmei Jiang, and Yaowei Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2025

   Bib Code Website

   @inproceedings{Huang_2025_CVPR,
     author = {Huang, Sihong and Wu, Jiaxin and Wei, Xiaoyong and Cai, Yi and Jiang, Dongmei and Wang, Yaowei},
     title = {Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     month = jun,
     year = {2025},
     pages = {28942-28951},
   }

2. IEEE Transactions on Medical Imaging

   

   Self-paced learning for images of antinuclear antibodies

   Yiyang Jiang, Guangwu Qian, Jiaxin Wu, Qi Huang, Qing Li, Yongkang Wu, and Xiao-Yong Wei

   IEEE Transactions on Medical Imaging, Jun 2025

   arXiv Bib Code Website

   @article{jiang2025self,
     title = {Self-paced learning for images of antinuclear antibodies},
     author = {Jiang, Yiyang and Qian, Guangwu and Wu, Jiaxin and Huang, Qi and Li, Qing and Wu, Yongkang and Wei, Xiao-Yong},
     journal = {IEEE Transactions on Medical Imaging},
     year = {2025},
     publisher = {IEEE},
   }

3. Machine Intelligence Research

   

   A survey on personalized content synthesis with diffusion models

   Xulu Zhang, Xiaoyong Wei, Wentao Hu, Jinlin Wu, Jiaxin Wu, Wengyu Zhang, Zhaoxiang Zhang, Zhen Lei, and Qing Li

   Machine Intelligence Research, Jun 2025

   arXiv Bib Code Website

   @article{zhang2025survey,
     title = {A survey on personalized content synthesis with diffusion models},
     author = {Zhang, Xulu and Wei, Xiaoyong and Hu, Wentao and Wu, Jinlin and Wu, Jiaxin and Zhang, Wengyu and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
     journal = {Machine Intelligence Research},
     volume = {22},
     number = {5},
     pages = {817--848},
     doi = {10.1007/s11633-025-1563-3},
     year = {2025},
     publisher = {Springer}
   }

4. npj Materials Sustainability

   

   Machine learning for efficient CO2 sequestration in cementitious materials: a data-driven method

   Yanjie Sun, Chen Zhang, Yuan-Hao Wei, Haoliang Jin, Peiliang Shen, Chi Sun Poon, He Yan, and Xiao-Yong Wei

   npj Materials Sustainability, Jun 2025

   Bib Website

   @article{sun2025machine,
     title = {Machine learning for efficient CO2 sequestration in cementitious materials: a data-driven method},
     author = {Sun, Yanjie and Zhang, Chen and Wei, Yuan-Hao and Jin, Haoliang and Shen, Peiliang and Poon, Chi Sun and Yan, He and Wei, Xiao-Yong},
     journal = {npj Materials Sustainability},
     volume = {3},
     number = {1},
     pages = {9},
     year = {2025},
     publisher = {Nature Publishing Group UK London},
   }

5. EMNLP Findings

   

   Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond

   Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiaoyong Wei, and Qing Li

   In Findings of the Association for Computational Linguistics: EMNLP 2025 , Nov 2025

   Abs arXiv Bib Code

   The increasing demand for domain-specific evaluation of large language models (LLMs) has led to the development of numerous benchmarks. These efforts often adhere to the principle of data scaling, relying on large corpora or extensive question-answer (QA) sets to ensure broad coverage. However, the impact of corpus and QA set design on the precision and recall of domain-specific LLM performance remains poorly understood. In this paper, we argue that data scaling is not always the optimal principle for domain-specific benchmark construction. Instead, we introduce Comp-Comp, an iterative benchmarking framework grounded in the principle of comprehensiveness and compactness. Comprehensiveness ensures semantic recall by covering the full breadth of the domain, while compactness improves precision by reducing redundancy and noise. To demonstrate the effectiveness of our approach, we present a case study conducted at a well-renowned university, resulting in the creation of PolyBench, a large-scale, high-quality academic benchmark. Although this study focuses on academia, the Comp-Comp framework is domain-agnostic and readily adaptable to a wide range of specialized fields. The source code and datasets can be accessed at https://github.com/Anya-RB-Chen/COMP-COMP.

   @inproceedings{chen-etal-2025-benchmarking-domain,
     title = {Benchmarking for Domain-Specific {LLM}s: A Case Study on Academia and Beyond},
     author = {Chen, Rubing and Wu, Jiaxin and Wang, Jian and Zhang, Xulu and Fan, Wenqi and Lin, Chenghua and Wei, Xiaoyong and Li, Qing},
     editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
     month = nov,
     year = {2025},
     address = {Suzhou, China},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.findings-emnlp.622/},
     doi = {10.18653/v1/2025.findings-emnlp.622},
     pages = {11606--11619},
     isbn = {979-8-89176-335-7},
   }

6. EMNLP Findings

   

   GLProtein: Global-and-Local Structure Aware Protein Representation Learning

   Yunqing Liu, Wenqi Fan, Xiaoyong Wei, and Qing Li

   In Findings of the Association for Computational Linguistics: EMNLP 2025 , Nov 2025

   Abs arXiv Bib Code Website

   analysis, there remains potential for further exploration in integrating protein structural information. We argue that the structural information of proteins is not only limited to their 3D information but also encompasses information from amino acid molecules (local information) to protein-protein structure similarity (global information). To address this, we propose \textbfGLProtein, the first framework in protein pre-training that incorporates both global structural similarity and local amino acid details to enhance prediction accuracy and functional insights. GLProtein innovatively combines protein-masked modelling with triplet structure similarity scoring, protein 3D distance encoding and substructure-based amino acid molecule encoding. Experimental results demonstrate that GLProtein outperforms previous methods in several bioinformatics tasks, including predicting protein-protein interaction, contact prediction, and so on.

   @inproceedings{liu2025glprotein,
     title = {GLProtein: Global-and-Local Structure Aware Protein Representation Learning},
     author = {Liu, Yunqing and Fan, Wenqi and Wei, Xiaoyong and Li, Qing},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
     pages = {4355--4372},
     year = {2025},
   }

7. IEEE Transactions on Mobile Computing

   

   Hierarchical and heterogeneous federated learning via a learning-on-model paradigm

   Leming Shen, Qiang Yang, Kaiyan Cui, Yuanqing Zheng, Xiao-Yong Wei, Jianwei Liu, and Jinsong Han

   IEEE Transactions on Mobile Computing, Nov 2025

   Abs Bib Website

   Federated Learning (FL) collaboratively trains a shared global model without exposing clients’ private data. In practical FL systems, clients (e.g., smartphones and wearables) typically have disparate system resources. Traditional FL, however, adopts a one-size-fits-all solution, where a homogeneous large model is sent to and trained on each client. This method results in an overwhelming workload for less capable clients and starvation for others. To tackle this, we propose FedConv, a client-friendly FL framework, minimizing the system overhead on resource-constrained clients by providing heterogeneous customized sub-models. FedConv features a novel learning-on-model paradigm that learns the parameters of heterogeneous sub-models via convolutional compression. To aggregate heterogeneous sub-models, we propose transposed convolutional dilation to convert them back to large models with a unified size while retaining personalized information. The compression and dilation processes, transparent to clients, are tuned on the server using a small public dataset. We further propose a hierarchical and clustering-based local training strategy for enhanced performance. Extensive experiments on six datasets show that FedConv outperforms state-of-the-art FL systems in terms of model accuracy (by more than 35% on average), computation and communication overhead (with 33% and 25% reduction, respectively).

   @article{shen2025hierarchical,
     title = {Hierarchical and heterogeneous federated learning via a learning-on-model paradigm},
     author = {Shen, Leming and Yang, Qiang and Cui, Kaiyan and Zheng, Yuanqing and Wei, Xiao-Yong and Liu, Jianwei and Han, Jinsong},
     journal = {IEEE Transactions on Mobile Computing},
     year = {2025},
     publisher = {IEEE},
   }

8. ACL

   

   Removal of Hallucination on Hallucination: Debate-Augmented RAG

   Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, and Qing Li

   In ACL 2025 Main , Nov 2025

   Abs arXiv Bib Code Website

   Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external knowledge, yet it introduces a critical issue: erroneous or biased retrieval can mislead generation, compounding hallucinations, a phenomenon we term Hallucination on Hallucination. To address this, we propose Debate Augmented RAG (DRAG), a training-free framework that integrates Multi-Agent Debate (MAD) mechanisms into both retrieval and generation stages. In retrieval, DRAG employs structured debates among proponents, opponents, and judges to refine retrieval quality and ensure factual reliability. In generation, DRAG introduces asymmetric information roles and adversarial debates, enhancing reasoning robustness and mitigating factual inconsistencies. Evaluations across multiple tasks demonstrate that DRAG improves retrieval reliability, reduces RAG-induced hallucinations, and significantly enhances overall factual accuracy. Our code is available at https://github.com/Huenao/Debate-Augmented-RAG.

   @inproceedings{hu2025removal,
     title = {Removal of Hallucination on Hallucination: Debate-Augmented RAG},
     author = {Hu, Wentao and Zhang, Wengyu and Jiang, Yiyang and Zhang, Chen Jason and Wei, Xiaoyong and Li, Qing},
     booktitle = {ACL 2025 Main},
     year = {2025},
   }

9. Briefings in Bioinformatics

   

   GraphATC: advancing multilevel and multi-label anatomical therapeutic chemical classification via atom-level graph learning

   Wengyu Zhang, Qi Tian, Yi Cao, Wenqi Fan, Dongmei Jiang, Yaowei Wang, Qing Li, and Xiao-Yong Wei

   Briefings in Bioinformatics, Apr 2025

   Abs Bib Code Website

   The accurate categorization of compounds within the anatomical therapeutic chemical (ATC) system is fundamental for drug development and fundamental research. Although this area has garnered significant research focus for over a decade, the majority of prior studies have concentrated solely on the Level 1 labels defined by the World Health Organization (WHO), neglecting the labels of the remaining four levels. This narrow focus fails to address the true nature of the task as a multilevel, multi-label classification challenge. Moreover, existing benchmarks like Chen-2012 and ATC-SMILES have become outdated, lacking the incorporation of new drugs or updated properties of existing ones that have emerged in recent years and have been integrated into the WHO ATC system. To tackle these shortcomings, we present a comprehensive approach in this paper. Firstly, we systematically cleanse and enhance the drug dataset, expanding it to encompass all five levels through a rigorous cross-resource validation process involving KEGG, PubChem, ChEMBL, ChemSpider, and ChemicalBook. This effort culminates in the creation of a novel benchmark termed ATC-GRAPH. Secondly, we extend the classification task to encompass Level 2 and introduce graph-based learning techniques to provide more accurate representations of drug molecular structures. This approach not only facilitates the modeling of Polymers, Macromolecules, and Multi-Component drugs more precisely but also enhances the overall fidelity of the classification process. The efficacy of our proposed framework is validated through extensive experiments, establishing a new state-of-the-art methodology. To facilitate the replication of this study, we have made the benchmark dataset, source code, and web server openly accessible.

   @article{zhang2025graphatc,
     author = {Zhang, Wengyu and Tian, Qi and Cao, Yi and Fan, Wenqi and Jiang, Dongmei and Wang, Yaowei and Li, Qing and Wei, Xiao-Yong},
     title = {GraphATC: advancing multilevel and multi-label anatomical therapeutic chemical classification via atom-level graph learning},
     journal = {Briefings in Bioinformatics},
     volume = {26},
     number = {2},
     pages = {bbaf194},
     year = {2025},
     month = apr,
     issn = {1477-4054},
     doi = {10.1093/bib/bbaf194},
     eprint = {https://academic.oup.com/bib/article-pdf/26/2/bbaf194/63012495/bbaf194.pdf},
   }

2024

1. NIST TRECVID

   

   PolySmart @ TRECVid 2024 Medical Video Question Answering

   Jiaxin Wu, Yiyang Jiang, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , Apr 2024

   Abs Bib Website

   Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical question based on the similarity of video transcript and answers generated by GPT4. For the visual answer localization, the start and end timestamps of the answer are predicted by the alignments on both visual content and subtitles with queries. For the Query-Focused Instructional Step Captioning (QFISC) task, the step captions are generated by GPT4. Specifically, we provide the video captions generated by the LLaVA-Next-Video model and the video subtitles with timestamps as context, and ask GPT4 to generate step captions for the given medical query. We only submit one run for evaluation and it obtains a F-score of 11.92 and mean IoU of 9.6527.

   @inproceedings{wu2024polysmarttrecvid2024,
     title = {PolySmart @ TRECVid 2024 Medical Video Question Answering},
     author = {Wu, Jiaxin and Jiang, Yiyang and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

2. NIST TRECVID

   

   PolySmart @ TRECVid 2024 Video-To-Text

   Jiaxin Wu, Wengyu Zhang, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , Apr 2024

   Abs Bib Website

   In this paper, we present our methods and results for the Video-To-Text (VTT) task at TRECVid 2024, exploring the capabilities of Vision-Language Models (VLMs) like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content. We investigate the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency. Our analysis reveals that fine-tuning substantially improves the model’s ability to produce more detailed and domain-aligned text, bridging the gap between generic VLM tasks and the specialized needs of VTT. Experimental results demonstrate that our fine-tuned model outperforms baseline VLMs across various evaluation metrics, underscoring the importance of domain-specific tuning for complex VTT tasks.

   @inproceedings{wu2024polysmarttrecvid2025,
     title = {PolySmart @ TRECVid 2024 Video-To-Text},
     author = {Wu, Jiaxin and Zhang, Wengyu and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

3. NIST TRECVID

   

   PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search

   Jiaxin Wu, Chong-Wah Ngo, Xiao-Yong Wei, and Qing Li

   In NIST TRECVID Notebook 2024 , Apr 2024

   Abs Bib Website

   This year, we explore generation-augmented retrieval for the TRECVid AVS task. Specifically, the understanding of textual query is enhanced by three generations, including Text2Text, Text2Image, and Image2Text, to address the out-of-vocabulary problem. Using different combinations of them and the rank list retrieved by the original query, we submitted four automatic runs. For manual runs, we use a large language model (LLM) (i.e., GPT4) to rephrase test queries based on the concept bank of the search engine, and we manually check again to ensure all the concepts used in the rephrased queries are in the bank. The result shows that the fusion of the original and generated queries outperforms the original query on TV24 query sets. The generated queries retrieve different rank lists from the original query.

   @inproceedings{wu2024polysmartvireotrecvid,
     title = {PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search},
     author = {Wu, Jiaxin and Ngo, Chong-Wah and Wei, Xiao-Yong and Li, Qing},
     booktitle = {NIST TRECVID Notebook 2024},
     year = {2024},
   }

4. ACM MM

   

   Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval

   Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiaoyong Wei, Chang Wen Chen, and Qing Li

   In ACM Multimedia 2024 , Apr 2024

   Abs arXiv Bib Code Website

   In this paper, we investigate the feasibility of leveraging large language models (LLMs) for integrating general knowledge and incorporating pseudo-events as priors for temporal content distribution in video moment retrieval (VMR) models. The motivation behind this study arises from the limitations of using LLMs as decoders for generating discrete textual descriptions, which hinders their direct application to continuous outputs like salience scores and inter-frame embeddings that capture inter-frame relations. To overcome these limitations, we propose utilizing LLM encoders instead of decoders. Through a feasibility study, we demonstrate that LLM encoders effectively refine inter-concept relations in multimodal embeddings, even without being trained on textual embeddings. We also show that the refinement capability of LLM encoders can be transferred to other embeddings, such as BLIP and T5, as long as these embeddings exhibit similar inter-concept similarity patterns to CLIP embeddings. We present a general framework for integrating LLM encoders into existing VMR architectures, specifically within the fusion module. The LLM encoder’s ability to refine concept relation can help the model to achieve a balanced understanding of the foreground concepts (e.g., persons, faces) and background concepts (e.g., street, mountains) rather focusing only on the visually dominant foreground concepts. Additionally, we introduce the concept of pseudo-events, obtained through event detection techniques, to guide the prediction of moments within event boundaries instead of crossing them, which can effectively avoid the distractions from adjacent moments. The integration of semantic refinement using LLM encoders and pseudo-event regulation is designed as plug-in components that can be incorporated into existing VMR methods within the general framework. Through experimental validation, we demonstrate the effectiveness of our proposed methods by achieving state-of-the-art performance in VMR.

   @inproceedings{jiang2024prior,
     title = {Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval},
     author = {Jiang, Yiyang and Zhang, Wengyu and Zhang, Xulu and Wei, Xiaoyong and Chen, Chang Wen and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

5. ACM MM

   

   A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning

   Changmeng Zheng, DaYong Liang, Wengyu Zhang, Xiaoyong Wei, Tat-Seng Chua, and Qing Li

   In ACM Multimedia 2024 , Apr 2024

   Abs arXiv Bib Code Website

   This paper presents a pilot study aimed at introducing multi-agent debate into multimodal reasoning. The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images. These challenges stem from the inductive (bottom-up) nature of existing debating schemes. To address the issue, we propose a deductive (top-down) debating approach called Blueprint Debate on Graphs (BDoG). In BDoG, debates are confined to a blueprint graph to prevent opinion trivialization through world-level summarization. Moreover, by storing evidence in branches within the graph, BDoG mitigates distractions caused by frequent but irrelevant concepts. Extensive experiments validate BDoG, achieving state-of-the-art results in Science QA and MMBench with significant improvements over previous methods.

   @inproceedings{zheng2024a,
     title = {A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning},
     author = {Zheng, Changmeng and Liang, DaYong and Zhang, Wengyu and Wei, Xiaoyong and Chua, Tat-Seng and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

6. ACM MM

   

   Generative Active Learning for Image Synthesis Personalization

   Xulu Zhang, Wengyu Zhang, Xiaoyong Wei, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li

   In ACM Multimedia 2024 , Apr 2024

   Abs arXiv Bib Code Website

   This paper presents a pilot study that explores the application of active learning, traditionally studied in the context of discriminative models, to generative models. We specifically focus on image synthesis personalization tasks. The primary challenge in conducting active learning on generative models lies in the open-ended nature of querying, which differs from the closed form of querying in discriminative models that typically target a single concept. We introduce the concept of anchor directions to transform the querying process into a semi-open problem. We propose a direction-based uncertainty sampling strategy to enable generative active learning and tackle the exploitation-exploration dilemma. Extensive experiments are conducted to validate the effectiveness of our approach, demonstrating that an open-source model can achieve superior performance compared to closed-source models developed by large companies, such as Google’s StyleDrop. The source code is available at https://github.com/zhangxulu1996/GAL4Personalization

   @inproceedings{zhang2024generative,
     title = {Generative Active Learning for Image Synthesis Personalization},
     author = {Zhang, Xulu and Zhang, Wengyu and Wei, Xiaoyong and Wu, Jinlin and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
     booktitle = {ACM Multimedia 2024},
     year = {2024},
   }

7. AAAI

   

   Compositional inversion for stable diffusion models

   Xulu Zhang, Xiao-Yong Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, and Qing Li

   In Proceedings of the AAAI Conference on Artificial Intelligence , Apr 2024

   Abs Bib Paper Code

   Inversion methods, such as Textual Inversion, generate personalized images by incorporating concepts of interest provided by user images. However, existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. It stems from the fact that during inversion, the irrelevant semantics in the user images are also encoded, forcing the inverted concepts to occupy locations far from the core distribution in the embedding space. To address this issue, we propose a method that guides the inversion process towards the core distribution for compositional embeddings. Additionally, we introduce a spatial regularization approach to balance the attention on the concepts being composed. Our method is designed as a post-training approach and can be seamlessly integrated with other inversion methods. Experimental results demonstrate the effectiveness of our proposed approach in mitigating the overfitting problem and generating more diverse and balanced compositions of concepts in the synthesized images. The source code is available at https://github.com/zhangxulu1996/Compositional-Inversion.

   @inproceedings{zhang2024compositional,
     title = {Compositional inversion for stable diffusion models},
     author = {Zhang, Xulu and Wei, Xiao-Yong and Wu, Jinlin and Zhang, Tianyi and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},
     paper = {https://ojs.aaai.org/index.php/AAAI/article/view/28565},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     volume = {38},
     number = {7},
     pages = {7350--7358},
     year = {2024},
     publisher = {AAAI},
   }