Foundations of Model Selection

Title: Foundations of Model Selection

Research Question: How can we determine the best model for explaining a given set of data, especially when considering model complexity?

Methodology: The authors propose a new approach to model selection called Kolmogorov's structure function. This function measures the relationship between the individual data and its explanation (model), and can be expressed as a two-part code consisting of a model description and a data-to-model code. The authors also consider a one-part code consisting of just the data-to-model code.

Results: The main result of this study is that minimizing the two-part code or the one-part code always selects a model that is a "best explanation" of the data within given model-complexity constraints. This means that the best fit cannot be computationally monotonically approximated, but the two-part code or the one-part code can be monotonically minimized, allowing for an approximation of the best-fitting model.

Implications: This research has significant implications for the field of statistics and learning theory. It suggests that the Kolmogorov structure function can be used to determine the best model for explaining a given set of data, especially when considering model complexity. This approach is particularly relevant in situations where average relations are irrelevant, such as in complex video and sound analysis.

Link to Article: https://arxiv.org/abs/0204037v4 Authors: arXiv ID: 0204037v4