Understanding and Enhancing Sequential Decision-Making and Alignment in Foundation Models

Abstract

Foundation models are used as sequential decision-makers, in tasks whose outputs unfold as dependent steps: reasoning, planning, decisions affecting several parties, and generation under safety requirements. This dissertation asks which factors govern the performance of foundation models in such settings and how that performance can be improved. The answer locates those factors in training data and training procedure. Pretraining text preserves finished solutions far more often than the failed attempts behind them, so a model trained on it defaults to confident, linearly coherent continuation; exploring alternative paths and retracting mistaken steps are underrepresented behaviors, not absent capabilities. The contributions elicit these behaviors in context and then teach them by supervised and reinforcement learning. One observation recurs: a model given room to explore diverse solution paths and to back out of its own mistakes makes better decisions, and in safety-critical settings the same capacity lets it recover from unsafe trajectories.

The body has three parts. Part I treats foundation models in reinforcement learning: a meta-learning method for sequences of derivative-free optimization tasks, with task-averaged regret guarantees; policy optimization under several reward objectives and hard safety constraints, with a rectification step that restores feasibility after a detected violation; and an analysis tracing in-context reinforcement learning to the diversity of the pretraining task distribution. Part II treats large language models in sequential decision-making: a tool-use framework that translates natural-language energy-management requests into solver-ready optimization programs; the Algorithm of Thoughts, a prompting strategy whose exemplars record a search process so that the model explores, prunes, and backtracks within a single generation; an extension to autonomous long-horizon planning; and a training pipeline that makes the behavior a concise default. Part III turns the same capacity to alignment and safety: a prompting framework that surveys a decision's consequences for every affected stakeholder before answering, and a reinforcement-learning method that trains the backtracking step as a safety signal, so that the model retracts an emerging violation and continues from the safe prefix. Recovery complements avoidance rather than replacing it; an integrated red-team study of a static classifier defense motivates judging safety on the generated trajectory.