Data are crucial element in large language model (LLM) alignment. Recent studies have explored using LLMs for efficient data collection. However, LLM-generated data often suffers from quality issues, with underrepresented or absent aspects and low-quality datapoints. To address these problems, we propose Data Advisor, an enhanced LLM-based method for generating data that takes into account the characteristics of the desired dataset. Starting from a set of pre-defined principles in hand, Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. Data Advisor can be easily integrated into existing data generation methods to enhance data quality and coverage. Experiments on safety alignment of three representative LLMs (i.e., Mistral, Llama2, and Falcon) demonstrate the effectiveness of Data Advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility.
we propose Data Advisor, which enhances LLM-based data generation by dynamically and proactively incorporating guiding principles of the target dataset. Data Advisor instructs the data generator to create alignment data with predefined principles, involving both quality and directional control of an independent prompt, as well as the overall statistics of the dataset. With a set of principles in hand, Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. At the monitor stage, it summarizes the current dataset iteratively, with the last data summary and the newly generated instance as input. At the advise stage, it identifies the current data weaknesses based on the summary, which is sent to the data generator later to guide the generation of the next instance. Data Advisor can be easily integrated into existing data generation methods to enhance data quality and coverage.
Data Advisor:
@inproceedings{wang2024data,
title={DATA ADVISOR: Dynamic Data Curation for Safety Alignment of Large Language Models},
author={Wang, Fei and Mehrabi, Ninareh and Goyal, Palash and Gupta, Rahul and Chang, Kai-Wei and Galstyan, Aram},
journal={Proceedings of EMNLP 2024},
year={2024}
}