Toward Transformer-based Large Energy Models for Smart Energy Management
| dc.contributor.author | Gu, Yueyan | en |
| dc.contributor.committeechair | Wang, Xuan | en |
| dc.contributor.committeecochair | Jazizadeh, Farrokh | en |
| dc.contributor.committeemember | Zhou, Dawei | en |
| dc.contributor.department | Computer Science | en |
| dc.date.accessioned | 2025-02-21T13:45:45Z | en |
| dc.date.available | 2025-02-21T13:45:45Z | en |
| dc.date.issued | 2024-11-01 | en |
| dc.description.abstract | Buildings contribute significantly to global energy demand and emissions, highlighting the need for precise energy forecasting for effective management. Existing research tends to focus on specific target problems, such as individual buildings or small groups of buildings, leading to current challenges in data-driven energy forecasting, including dependence on data quality and quantity, limited generalizability, and computational inefficiency. To address these challenges, Generalized Energy Models (GEMs) for energy forecasting can potentially be developed using large-scale datasets. Transformer architectures, known for their scalability, ability to capture long-term dependencies, and efficiency in parallel processing of large datasets, are considered good candidates for GEMs. In this study, we tested the hypothesis that GEMs can be efficiently developed to outperform in-situ models trained on individual buildings. To this end, we investigated and compared three candidate multi-variate Transformer architectures, utilizing both zero-shot and fine-tuning strategies, with data from 1,014 buildings. The results, evaluated across three prediction horizons (24, 72, and 168 hours), confirm that GEMs significantly outperform Transformer-based in-situ (i.e., building-specific) models. Fine-tuned GEMs showed performance improvements of up to 28% and reduced training time by 55%. Besides Transformer-based in-situ models, GEMs outperformed several state-of-the-art non-Transformer deep learning baseline models in efficiency and efficiency. We further explored the answer to a number of questions including the required data size for effective fine-tuning, as well as the impact of input sub-sequence length and pre-training dataset size on GEM performance. The findings show a significant performance boost by using larger pre-training datasets, highlighting the potential for larger GEMs using web-scale global data to move toward Large Energy Models (LEM). | en |
| dc.description.abstractgeneral | Buildings account for a large share of global energy use and emissions, which makes predicting their energy needs critical for better management. However, most research focuses on creating energy models for specific buildings or small groups, which limits their usefulness for larger-scale applications. Additionally, these models often face challenges such as relying on high-quality data, limited adaptability to different buildings, and inefficiencies when dealing with large amounts of data. This study aims to address these issues by developing Generalized Energy Models (GEMs), which use data from a large number of buildings to create more versatile and efficient energy forecasting tools. To achieve this, we used Transformer models, a type of machine learning approach known for handling large datasets efficiently and recognizing long-term patterns. We tested whether GEMs could provide better predictions than traditional models designed for individual buildings. Our analysis included data from over 1,000 buildings and used two strategies: zero-shot (using the model without further adjustments) and fine-tuning (adapting the model to specific data). The results showed that GEMs were more accurate than traditional models, improving prediction accuracy by up to 28% and reducing the time needed for training by over 50%. Additionally, GEMs outperformed other advanced methods of energy forecasting. We also examined how different factors, such as the amount of data and the length of the data sequences, influenced the model’s performance. The findings suggest that using even larger datasets could lead to further improvements, opening the possibility of creating Large Energy Models (LEMs) that can make predictions on a global scale. | en |
| dc.description.degree | Master of Science | en |
| dc.format.medium | ETD | en |
| dc.format.mimetype | application/pdf | en |
| dc.identifier.uri | https://hdl.handle.net/10919/124674 | en |
| dc.publisher | Virginia Tech | en |
| dc.rights | In Copyright | en |
| dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | en |
| dc.subject | Building Energy Management | en |
| dc.subject | Time Series Forecasting | en |
| dc.subject | Transformers | en |
| dc.subject | Scalable AI | en |
| dc.subject | Foundation Models | en |
| dc.title | Toward Transformer-based Large Energy Models for Smart Energy Management | en |
| dc.type | Thesis | en |
| dc.type.dcmitype | Text | en |
| thesis.degree.discipline | Computer Science and Application | en |
| thesis.degree.grantor | Virginia Polytechnic Institute and State University | en |
| thesis.degree.level | masters | en |
| thesis.degree.name | Master of Science | en |