LLM-Guided Run-Time Parameter Optimization for Energy-Efficient Model Inference

Abstract

The scale of Large Language Models (LLMs) has been increasing over time, and is predicted to continue to increase as LLMs become an integral part of many real world workflows. However, LLMs consume a tremendous amount of energy, which becomes a large concern in the scale of the demand for these tools. Inference engines and serving systems, like vLLM and PyTorch, are used to perform efficient LLM inference. These tools include runtime parameters that have the potential to improve the energy efficiency of inference. The task is choosing the values for those parameters that actually result in lower energy consumption. However, these parameters have complex relationships and trade offs that make this choice difficult and sometimes non intuitive. This choice can also require deep knowledge of the application or inefficient traditional optimization methods. In this work, we created a human-in-the-loop flow with LLM assisted runtime parameter optimization in order to solve this issue. With human created, specific feedback prompting methods, chat based LLMs can iteratively find energy efficient inference parameters faster than traditional search methods. We evaluated LLM as an optimization method on a text prompt workload with vLLM as the inference engine and on an image prompt workload with PyTorch as the inference engine. Across these two systems, we compared the performance of LLM optimization when prompting with baseline prompts that only gave the minimum amount of information and enhanced prompts which included different prompting strategies with the goal of improving performance. LLM optimization that uses the enhnaced prompt strategy, was compared to a more traditional optimization method: Sobol sampling. The enhanced prompting strategy outperforms the baseline across all experiment runs. It achieves an average of 45% reduction in energy-per-token for vLLM inference and an average of 90.7% reduction for PyTorch multimodal inference relative to the default configurations. Compared to Sobol sampling, the LLM guided approach reaches lower energy-per-token values more efficiently in the vLLM inference and achives lower energy-per-image values in the PyTorch setting. Sobol sampling also consistently failed to avoid out of memory configurations in the PyTorch configuration. These results suggest that LLMs may serve as effective optimization agents for reducing the energy consumption of AI inference, without requiring expertise or exhaustive search.