Holistic and Generalizable Evaluation of Generative Models
Files
TR Number
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The rapid advancement of generative models, e.g., language models and multimodal models, has created a critical gap between their sophisticated capabilities and the evaluation methodologies used to assess them. While these models now excel at complex creative and interactive tasks, evaluation approaches remain anchored to task-specific metrics designed for constrained generation. This dissertation addresses the fundamental challenge of developing comprehensive, scalable, and generalizable evaluation methodologies that match the complexity of modern generative models. First, I introduce our efforts to advance the evaluation paradigms for assessing emerging generative capabilities. I develop instruction-tuned evaluators that generalize to unseen evaluation aspects without retraining, enabling adaptive assessment with new criteria. I then extend evaluation to the multimodal scenario, where I introduce a holistic framework for unified text-and-image generation and address the unique challenges of cross-modality assessment. Next, I extend the evaluation of generative models to scientific domains and propose the structured ideation space that decomposes scientific ideas into orthogonal conceptual dimensions, enabling fine-grained novelty assessments grounded in retrieved evidence rather than surface-level similarity. Beyond assessing generative capabilities, I present our work in investigating generative models' safety and how to deploy them responsibly. I develop a systematic framework for evaluating safety risks in goal-driven interactions, revealing critical misalignments between models' refusal behaviors and their deployment of unethical strategies. I also introduce a generalizable guardrail training methodology that enables AI guardrails to adapt to unseen policies and domains. Together, these contributions establish new evaluation paradigms that enable more principled development of generative models while ensuring alignment with human values and societal needs.