Publications

2024

1. EMNLP

   mDPO: Conditional Preference Optimization for Multimodal Large Language Models

   Fei Wang,  Wenxuan Zhou ,  James Y Huang ,  Nan Xu ,  Sheng Zhang ,  Hoifung Poon , and  Muhao Chen

   In Proceedings of EMNLP , 2024

   Website
2. EMNLP

   Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models

   Fei Wang,  Ninareh Mehrabi ,  Palash Goyal ,  Rahul Gupta ,  Kai-Wei Chang , and  Aram Galstyan

   In Proceedings of EMNLP , 2024

   Website
3. EMNLP

   Llms assist nlp researchers: Critique paper (meta-) reviewing

   Jiangshu Du ,  Yibo Wang ,  Wenting Zhao ,  Zhongfen Deng ,  Shuaiqi Liu ,  Renze Lou ,  Henry Peng Zou ,  Pranav Narayanan Venkit ,  Nan Zhang ,  Mukund Srinath ,  Haoran Ranran Zhang ,  Vipul Gupta ,  Yinghui Li ,  Tao Li , Fei Wang, and 1 more author

   In Proceedings of EMNLP , 2024
4. EMNLP

   Monotonic Paraphrasing Improves Generalization of Language Model Prompting

   Qin Liu , Fei Wang,  Nan Xu ,  Tianyi Yan ,  Tao Meng , and  Muhao Chen

   In Findings of EMNLP , 2024

   Code
5. ACL

   Contrastive Instruction Tuning

   Tianyi Yan , Fei Wang,  James Y Huang ,  Wenxuan Zhou ,  Fan Yin ,  Aram Galstyan ,  Wenpeng Yin , and  Muhao Chen

   In Findings of ACL , 2024

   Code
6. NAACL

   From Shortcuts to Triggers: Backdoor Defense with Denoised PoE

   Qin Liu , Fei Wang,  Chaowei Xiao , and  Muhao Chen

   In Proceedings of NAACL , 2024

   Code
7. NAACL

   Rethinking Tabular Data Understanding with Large Language Models

   Tianyang Liu , Fei Wang, and  Muhao Chen

   In Proceedings of NAACL , 2024

   Code
8. NAACL

   Instructional Fingerprinting of Large Language Models

   Jiashu Xu , Fei Wang*,  Mingyu Derek Ma* ,  Pang Wei Koh ,  Chaowei Xiao , and  Muhao Chen

   In Proceedings of NAACL , 2024

   Code
9. NAACL

   Deceiving Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?

   Bangzheng Li ,  Ben Zhou , Fei Wang,  Xingyu Fu ,  Dan Roth , and  Muhao Chen

   In Proceedings of NAACL , 2024
10. NAACL

    Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

    Jiashu Xu ,  Mingyu Derek Ma , Fei Wang,  Chaowei Xiao , and  Muhao Chen

    In Proceedings of NAACL , 2024
11. NAACL

    Cognitive overload: Jailbreaking large language models with overloaded logical thinking

    Nan Xu , Fei Wang,  Ben Zhou ,  Bang Zheng Li ,  Chaowei Xiao , and  Muhao Chen

    In Findings of NAACL , 2024

    Code
12. Allerton

    Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges

    Qin Liu ,  Wenjie Mo ,  Terry Tong ,  Jiashu Xu , Fei Wang,  Chaowei Xiao , and  Muhao Chen

    In Allerton Conference , 2024
13. statement

    Developing Robust and Controllable Generative AI

    Fei Wang

    In , 2024
14. preprint

    Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models

    Fei Wang,  Xingchen Wan ,  Ruoxi Sun ,  Jiefeng Chen , and  Sercan Ö. Arık

    In arXiv preprint , 2024
15. preprint

    MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

    Fei Wang*,  Xingyu Fu* ,  James Y Huang ,  Zekun Li ,  Qin Liu ,  Xiaogeng Liu ,  Mingyu Derek Ma ,  Nan Xu ,  Wenxuan Zhou ,  Kai Zhang , and  others

    In arXiv preprint , 2024

    Website
16. preprint

    From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

    Fei Wang,  Chao Shang ,  Sarthak Jain ,  Shuai Wang ,  Qiang Ning ,  Bonan Min ,  Vittorio Castelli ,  Yassine Benajiba , and  Dan Roth

    In arXiv preprint , 2024
17. preprint

    Unraveling Cross-Modality Knowledge Conflict in Large Vision-Language Models

    Tinghui Zhu ,  Qin Liu , Fei Wang,  Zhengzhong Tu , and  Muhao Chen

    In arXiv preprint , 2024

    Website
18. preprint

    From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning

    Nan Xu , Fei Wang,  Sheng Zhang ,  Hoifung Poon , and  Muhao Chen

    In arXiv preprint , 2024
19. preprint

    Privacy-Preserving Language Model Inference with Instance Obfuscation

    Yixiang Yao , Fei Wang,  Srivatsan Ravi , and  Muhao Chen

    In arXiv preprint , 2024
20. preprint

    Offset Unlearning for Large Language Models

    James Y Huang ,  Wenxuan Zhou , Fei Wang,  Fred Morstatter ,  Sheng Zhang ,  Hoifung Poon , and  Muhao Chen

    In arXiv preprint , 2024

    Code
21. preprint

    FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation

    Bangzheng Li ,  Ben Zhou ,  Xingyu Fu , Fei Wang,  Dan Roth , and  Muhao Chen

    In arXiv preprint , 2024
22. preprint

    Training Small Multimodal Models to Bridge Biomedical Competency Gap: A Case Study in Radiology Imaging

    Juan Manuel Zambrano Chaves^* ,  Shih-Cheng Huang^* ,  Yanbo Xu^* ,  Hanwen Xu^* ,  Naoto Usuyama^* ,  Sheng Zhang^* , Fei Wang,  Yujia Xie ,  Mahmoud Khademi ,  Ziyi Yang ,  Hany Awadalla ,  Julia Gong ,  Houdong Hu ,  Jianwei Yang ,  Chunyuan Li , and 11 more authors

    In arXiv preprint , 2024

2023

1. EMNLP

   A Causal View of Entity Bias in (Large) Language Models

   Fei Wang,  Wenjie Mo ,  Yiwei Wang ,  Wenxuan Zhou , and  Muhao Chen

   In Findings of EMNLP , 2023

   Code
2. EMNLP

   Dense Retrieval as Indirect Supervision for Large-space Decision Making

   Nan Xu , Fei Wang,  Mingtao Dong , and  Muhao Chen

   In Findings of EMNLP , 2023

   Code
3. CoNLL

   How Fragile is Relation Extraction under Entity Replacements?

   Yiwei Wang ,  Bryan Hooi , Fei Wang,  Yujun Cai ,  Yuxuan Liang ,  Wenxuan Zhou ,  Jing Tang ,  Manjuan Duan , and  Muhao Chen

   In Proceedings of CoNLL , 2023

   Code
4. AACL

   Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

   Fei Wang,  Kuan-Hao Huang ,  Kai-Wei Chang , and  Muhao Chen

   In Proceedings of AACL , 2023

   Abstract Code

   Zero-shot cross-lingual transfer is a central task in multilingual NLP, allowing models trained in languages with more sufficient training resources to generalize to other low-resource languages. Earlier efforts on this task use parallel corpora, bilingual dictionaries, or other annotated alignment data to improve cross-lingual transferability, which are typically expensive to obtain. In this paper, we propose a simple yet effective method, SALT, to improve the zero-shot cross-lingual transfer of the multilingual pretrained language models without the help of such external data. By incorporating code-switching and embedding mixup with self-augmentation, SALT effectively distills cross-lingual knowledge from the multilingual PLM and enhances its transferability on downstream tasks. Experimental results on XNLI and PAWS-X show that our method is able to improve zero-shot cross-lingual transferability without external data.

5. ACL

   Robust Natural Language Understanding with Residual Attention Debiasing

   Fei Wang*,  James Y. Huang* ,  Tianyi Yan ,  Wenxuan Zhou , and  Muhao Chen

   In Findings of ACL , 2023

   Abstract Code

   Natural language understanding (NLU) models often suffer from unintended dataset biases. Among bias mitigation methods, ensemble-based debiasing methods, especially product-of-experts (PoE), have stood out for their impressive empirical success. However, previous ensemble-based debiasing methods typically apply debiasing on top-level logits without directly addressing biased attention patterns. Attention serves as the main media of feature interaction and aggregation in PLMs and plays a crucial role in providing robust prediction. In this paper, we propose REsidual Attention Debiasing (READ), an end-to-end debiasing method that mitigates unintended biases from attention. Experiments on three NLU tasks show that READ significantly improves the performance of BERT-based models on OOD data with shortcuts removed, including +12.9% accuracy on HANS, +11.0% accuracy on FEVER-Symmetric, and +2.7% F1 on PAWS. Detailed analyses demonstrate the crucial role of unbiased attention in robust NLU models and that READ effectively mitigates biases in attention.

6. ACL

   Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality

   Tanay Dixit , Fei Wang, and  Muhao Chen

   In Proceedings of ACL , 2023

   Abstract Code

   Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality. We propose EFACTSUM (i.e., Effective Factual Summarization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics. Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training. Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.

2022

1. EMNLP

   Salience Allocation as Guidance for Abstractive Summarization

   Fei Wang*,  Kaiqiang Song* ,  Hongming Zhang ,  Lifeng Jin ,  Sangwoo Cho ,  Wenlin Yao ,  Xiaoyang Wang ,  Muhao Chen , and  Dong Yu

   In Proceedings of EMNLP , 2022

   Abstract Code

   Abstractive summarization models typically learn to capture the salient information from scratch implicitly. Recent literature adds extractive summaries as guidance for abstractive summarization models to provide hints of salient content and achieves better performance. However, extractive summaries as guidance could be over strict, leading to information loss or noisy signals. Furthermore, it cannot easily adapt to documents with various abstractiveness. As the number and allocation of salience content pieces vary, it is hard to find a fixed threshold deciding which content should be included in the guidance. In this paper, we propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON). SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness. Automatic and human evaluations on two benchmark datasets show that the proposed method is effective and reliable. Empirical results on more than one million news articles demonstrate a natural fifteen-fifty salience split for news article sentences, providing a useful insight for composing news articles.

2. EMNLP

   Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing

   Nan Xu , Fei Wang,  Bangzheng Li ,  Mingtao Dong , and  Muhao Chen

   In Proceedings of EMNLP , 2022

   Abstract Code

   The entity typing task aims at predicting one or more words or phrases that describe the type(s) of a specific mention in a sentence. Due to shortcuts from surface patterns to annotated entity labels and biased training, existing entity typing models are subject to the problem of spurious correlations. To comprehensively investigate the faithfulness and reliability of entity typing methods, we first systematically define distinct kinds of model biases that are reflected mainly from spurious correlations. Particularly, we identify six types of existing model biases, including mention-context bias, lexical overlapping bias, named entity bias, pronoun bias, dependency bias, and overgeneralization bias. To mitigate these model biases, we then introduce a counterfactual data augmentation method. By augmenting the original training set with their bias-free counterparts, models are forced to fully comprehend the sentences and discover the fundamental cues for entity typing, rather than relying on spurious correlations for shortcuts. Experimental results on the UFET dataset show that our counterfactual data augmentation approach helps improve generalization of different entity typing models with consistently better performance on both in- and out-of-distribution test sets.

3. NAACL

   Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning

   Fei Wang,  Zhewei Xu ,  Pedro Szekely , and  Muhao Chen

   In Proceedings of NAACL , 2022

   Abstract Code

   Controlled table-to-text generation seeks to generate natural language descriptions for highlighted subparts of a table. Previous SOTA systems still employ a sequence-to-sequence generation method, which merely captures the table as a linear structure and is brittle when table layouts change. We seek to go beyond this paradigm by (1) effectively expressing the relations of content pieces in the table, and (2) making our model robust to content-invariant structural transformations. Accordingly, we propose an equivariance learning framework, which encodes tables with a structure-aware self-attention mechanism. This prunes the full self-attention structure into an order-invariant graph attention that captures the connected graph structure of cells belonging to the same row or column, and it differentiates between relevant cells and irrelevant cells from the structural perspective. Our framework also modifies the positional encoding mechanism to preserve the relative position of tokens in the same cell but enforce position invariance among different cells. Our technology is free to be plugged into existing table-to-text generation models, and has improved T5-based models to offer better performance on ToTTo and HiTab. Moreover, on a harder version of ToTTo, we preserve promising performance, while previous SOTA systems, even with transformation-based data augmentation, have seen significant performance drops.

4. EMNLP MMNLU

   Zero-shot cross-lingual sequence tagging as Seq2Seq generation for joint intent classification and slot filling

   Fei Wang,  Kuan-Hao Huang ,  Anoop Kumar ,  Aram Galstyan ,  Greg Ver Steeg , and  Kai-Wei Chang

   In Proceedings of the MMNLU Workshop at EMNLP , 2022

   Abstract

   The joint intent classification and slot filling task seeks to detect the intent of an utterance and extract its semantic concepts. In the zero-shot cross-lingual setting, a model is trained on a source language and then transferred to other target languages through multi-lingual representations without additional training data. While prior studies show that pre-trained multilingual sequence-to-sequence (Seq2Seq) models can facilitate zero-shot transfer, there is little understanding on how to design the output template for the joint prediction tasks. In this paper, we examine three aspects of the output template – (1) label mapping, (2) task dependency, and (3) word order. Experiments on the MASSIVE dataset consisting of 51 languages show that our output template significantly improves the performance of pre-trained cross-lingual language models.

2021

1. EMNLP

   Table-based Fact Verification With Salience-aware Learning

   Fei Wang,  Kexuan Sun ,  Jay Pujara ,  Pedro Szekely , and  Muhao Chen

   In Findings of EMNLP , 2021

   Abstract Code

   Tables provide valuable knowledge that can be used to verify textual statements. While a number of works have considered table-based fact verification, direct alignments of tabular data with tokens in textual statements are rarely available. Moreover, training a generalized fact verification model requires abundant labeled training data. In this paper, we propose a novel system to address these problems. Inspired by counterfactual causality, our system identifies token-level salience in the statement with probing-based salience estimation. Salience estimation allows enhanced learning of fact verification from two perspectives. From one perspective, our system conducts masked salient token prediction to enhance the model for alignment and reasoning between the table and the statement. From the other perspective, our system applies salience-aware data augmentation to generate a more diverse set of training instances by replacing non-salient terms. Experimental results on TabFact show the effective improvement by the proposed salience-aware learning techniques, leading to the new SOTA performance on the benchmark.

2. CIKM

   Tabular Functional Block Detection with Embedding-based Agglomerative Cell Clustering

   Kexuan Sun , Fei Wang,  Muhao Chen , and  Jay Pujara

   In Proceedings of CIKM , 2021

   Abstract

   Tables are a widely-used format for data curation. The diversity of domains, layouts, and content of tables makes knowledge extraction challenging. Understanding table layouts is an important step for automatically harvesting knowledge from tabular data. Since table cells are spatially organized into regions, correctly identifying such regions and inferring their functional roles, referred to as functional block detection, is a critical part of understanding table layouts. Earlier functional block detection approaches fail to leverage spatial relationships and higher-level structure, either depending on cell-level predictions or relying on data types as signals for identifying blocks. In this paper, we introduce a flexible functional block detection method by applying agglomerative clustering techniques which merge smaller blocks into larger blocks using two merging strategies. Our proposed method uses cell embeddings with a customized dissimilarity function which utilizes local and margin distances, as well as block coherence metrics to capture cell, block, and table scoped features. Given the diversity of tables in real-world corpora, we also introduce a sampling-based approach for automatically tuning distance thresholds for each table. Experimental results show that our method improves over the earlier state-of-the-art method in terms of several evaluation metrics.

3. SIGIR

   Retrieving Complex Tables with Multi-Granular Graph Representation Learning

   Fei Wang,  Kexuan Sun ,  Muhao Chen ,  Jay Pujara , and  Pedro Szekely

   In Proceedings of SIGIR , 2021

   Abstract Code

   The task of natural language table retrieval (NLTR) seeks to retrieve semantically relevant tables based on natural language queries. Existing learning systems for this task often treat tables as plain text based on the assumption that tables are structured as dataframes. However, tables can have complex layouts which indicate diverse dependencies between subtable structures, such as nested headers. As a result, queries may refer to different spans of relevant content that is distributed across these structures. Moreover, such systems fail to generalize to novel scenarios beyond those seen in the training set. Prior methods are still distant from a generalizable solution to the NLTR problem, as they fall short in handling complex table layouts or queries over multiple granularities. To address these issues, we propose Graph-based Table Retrieval (GTR), a generalizable NLTR framework with multi-granular graph representation learning. In our framework, a table is first converted into a tabular graph, with cell nodes, row nodes and column nodes to capture content at different granularities. Then the tabular graph is input to a Graph Transformer model that can capture both table cell content and the layout structures. To enhance the robustness and generalizability of the model, we further incorporate a self-supervised pre-training task based on graph-context matching. Experimental results on two benchmarks show that our method leads to significant improvements over the current state-of-the-art systems. Further experiments demonstrate promising performance of our method on cross-dataset generalization, and enhanced capability of handling complex tables and fulfilling diverse query intents.

2020

1. ACL

   CorefQA: Coreference resolution as query-based span prediction

   Wei Wu , Fei Wang,  Arianna Yuan ,  Fei Wu , and  Jiwei Li

   In Proceedings of ACL , 2020

   Abstract Code

   In this paper, we present an accurate and extensible approach for the coreference resolution task. We formulate the problem as a span prediction task, like in machine reading comprehension (MRC): A query is generated for each candidate mention using its surrounding context, and a span prediction module is employed to extract the text spans of the coreferences within the document using the generated query. This formulation comes with the following key advantages: (1) The span prediction strategy provides the flexibility of retrieving mentions left out at the mention proposal stage; (2) In the MRC framework, encoding the mention and its context explicitly in a query makes it possible to have a deep and thorough examination of cues embedded in the context of coreferent mentions; and (3) A plethora of existing MRC datasets can be used for data augmentation to improve the model’s generalization capability. Experiments demonstrate significant performance boost over previous models, with 87.5 (+2.5) F1 score on the GAP benchmark and 83.1 (+3.5) F1 score on the CoNLL-2012 benchmark.

2019

1. NeurIPS

   Glyce: Glyph-vectors for chinese character representations

   Yuxian Meng* ,  Wei Wu* , Fei Wang*,  Xiaoya Li* ,  Ping Nie ,  Fan Yin ,  Muyu Li ,  Qinghong Han ,  Xiaofei Sun , and  Jiwei Li

   In Proceedings of NeurIPS , 2019

   Abstract

   It is intuitive that NLP tasks for logographic languages like Chinese should benefit from the use of the glyph information in those languages. However, due to the lack of rich pictographic evidence in glyphs and the weak generalization ability of standard computer vision models on character data, an effective way to utilize the glyph information remains to be found. In this paper, we address this gap by presenting Glyce, the glyph-vectors for Chinese character representations. We make three major innovations: (1) We use historical Chinese scripts (e.g., bronzeware script, seal script, traditional Chinese, etc) to enrich the pictographic evidence in characters; (2) We design CNN structures (called tianzege-CNN) tailored to Chinese character image processing; and (3) We use image-classification as an auxiliary task in a multi-task learning setup to increase the model’s ability to generalize. We show that glyph-based models are able to consistently outperform word/char ID-based models in a wide range of Chinese NLP tasks. We are able to set new state-of-the-art results for a variety of Chinese NLP tasks, including tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks, dependency parsing, and semantic role labeling. For example, the proposed model achieves an F1 score of 80.6 on the OntoNotes dataset of NER, +1.5 over BERT; it achieves an almost perfect accuracy of 99.8% on the Fudan corpus for text classification.