Description:

The advent of large foundation models has shifted the machine learning paradigm from “one model per task” to “one large model, many tasks.” On the one hand, the prevalent LLM-based foundation models excel at many tasks that can be described by natural language, programming language and mathematical language. On the other hand, they still struggle with tasks of other modalities such as DNA data and real-valued time-series data. This limitation has led researchers to pursue specialized foundation models for other data domains. This thesis is to study time series foundation models(TSFMs) and their applications to scientific discoveries. Attempts at TSFMs have demonstrated efficacy across various benchmark datasets, outperforming traditional one-model-one-task approaches in forecasting tasks. However, there still exist limitations and challenges to give TSFMs more comprehensive capabilities. Besides, their potential contribution to scientific discoveries remains largely unexplored.

This thesis focuses on TSFMs from three perspectives. (i) Starting from LLM-based foundation models, I study the adaptive batch size technique for pre-training large models. (ii) I study methodologies such as the support for heterogeneous covariates that extend the abilities of existing TSFMs. (iii) I study the applications of TSFMs to accelerate scientific discoveries in astrophysics.