_        __  __ _      
    /\        | |      |  \/  | |     
   /  \  _   _| |_ ___ | \  / | |     
  / /\ \| | | | __/ _ \| |\/| | |     
 / ____ \ |_| | || (_) | |  | | |____ 
/_/    \_\__,_|\__\___/|_|  |_|______|

NOTE

AutoML.org is no longer updated. We moved all information over to ml4aad.org/automl.

Overview

HPOlib is a hyperparameter optimization library. It provides a common interface to three state of the art hyperparameter optimization packages: SMAC, Spearmint and Hyperopt. Moreover, the library provides optimization benchmarks which can be used to compare different hyperparameter optimization packages and to establish standard test routines for hyperparameter optimization packages.

News

• 15th May 2014: HPOlib 0.1 released. This release mainly contains refactoring and many small changes which make it easier to use the HPOlib for practical applications. We also created a HPOlib github repository, from where you can download the latest changes, file issues and contribute to the development of the HPOlib.
• 9th December 2013: Our paper introducing HPOlib has appeared at the NIPS Workshop on Bayesian Optimization in Theory and Practice (BayesOpt '13):
  Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters [pdf] [bib] [poster]
  This includes results for SMAC, spearmint and TPE on the benchmarks we provide.

Documentation

• Installing HPOlib
• Run other benchmarks than the Branin function
• Other included benchmarks
• Create your own benchmarks
• Adjust settings in the config file
• Add your own optimizer

Help

Please post questions in our google group.

Contribute

At the time of writing, HPOlib contains three hyperparameter optimization algorithms and six different benchmarks. In order to make HPOlib a standard library for testing hyperparameter optimization software for machine learning, we want HPOlib to include more optimization packages and benchmarks.

Contribute a benchmark

If you have written a machine learning algorithm or work on a machine learning problem which you think is worth including into HPOlib, please look here to find out how to add it to the library. If you have a new benchmark, please

• contact us via e-mail (address can be found at the bottom of these page) if it is a code-heavy benchmark
• send us a pull request on github if the benchmark consists only of a few files and does not introduce any new dependencies.

Ideally, the benchmark falls into one of these categories:

• The benchmark was previously used in work about hyperparameter optimization.
• The benchmark demonstrates strengths and/or weaknesses of a hyperparameter optimization algorithms.
• The benchmark is a lookup of conducted experiments like the Online LDA/Latent Structured SVM benchmarks.
• The benchmark provides a problem which is not yet included in HPOlib.

and is available on github or a different open source collaboration platform.

Contribute a hyperparameter optimization package

If you have written a hyperparemeter optimization package and want to:

• Make it available with the same interface as three very popular hyperparameter optimization packages are available
• Compare your hyperparameter optimization package with other hyperparameter optimization packages on standardized benchmarks

then you can find an explanation of the interface here. If you have an optimizer, please send us an e-mail (you can find the address at the bottom of this page) so we can discuss the further integration into the HPOlib.

Comments

We would like to hear your comments, remarks, advice or feature requests. Please feel free to contact us via e-mail (you can find the address at the bottom of this page) or post in the google group.

Google Group

Visit our google group on google.com