Performance Profiles for Benchmarking and Comparing Optimization Software

Title: Performance Profiles for Benchmarking and Comparing Optimization Software

Abstract: This research proposes the use of performance profiles, distribution functions of a performance metric, as a tool for benchmarking and comparing optimization software. It discusses the benefits of this approach and compares it to other methods.

Main Research Question: How effective are performance profiles as a tool for benchmarking and comparing optimization software?

Methodology: The study uses a ratio of a solver's computing time to the best time of all solvers as the performance metric. It compares this approach to other methods, such as averages, cumulative totals, and ranking solvers.

Results: The research found that performance profiles provide valuable information about the performance of optimization software and are more effective than other methods in certain aspects. For instance, they eliminate the negative effects of a few difficult problems dominating the conclusions and provide information on the percentage improvement.

Implications: The use of performance profiles can lead to more accurate and meaningful comparisons of optimization software. It can help researchers and users make better-informed decisions about which software to use for their specific needs.

Conclusion: Performance profiles are a useful tool for benchmarking and comparing optimization software. They provide a comprehensive view of the software's performance and help eliminate biases that may arise from other methods.

Link to Article: https://arxiv.org/abs/0102001v2 Authors: arXiv ID: 0102001v2