Foundations of Model Selection

Title: Foundations of Model Selection

Research Question: How can we determine the best model for explaining a given set of data, considering the complexity of the model and the amount of data?

Methodology: The researchers proposed a method called "model selection," which involves minimizing a two-part code consisting of a model subject to complexity constraints, and a data-to-model code. This method is based on Kolmogorov complexity, which measures the minimal length of a program that can generate a given data set.

Results: The researchers found that minimizing the two-part code produces a model of best fit, meaning the data is maximally "typical." They also showed that both the structure function and the minimum randomness deficiency function can assume all shapes over their full domains, improving on previous results. They provided an explicit realization of optimal two-part codes at all levels of model complexity.

Implications: This research provides a foundation for Model-Based Coding (MBC) and related methods in model selection. It shows that the minimal randomness deficiency function, which measures maximal "typicality," cannot be monotonically approximated, but the shortest two-part code can. This has implications for the field of machine learning and data analysis, as it offers a new approach to model selection that takes into account both the complexity of the model and the amount of data.

Link to Article: https://arxiv.org/abs/0204037v1 Authors: arXiv ID: 0204037v1