The loss landscape of an HPO problem is typically unknown (e.g., we need to solve a black-box function) and expensive to evaluate. Bayesian Optimization (BO) is designed as a global optimization strategy for expensive black-box functions. BO first estimates the shape of the target loss landscape with a surrogate model and then suggests the configuration to be evaluated in the next iteration. By trading off exploitation and exploration based on the surrogate model, it is well known for its sample efficiency.
- SMAC is a versatile tool for optimizing algorithm hyperparameters and implementing different surrogate models, acquisition functions, and model transformations.
- BOHB implements a variant of TPE as a BO approach.
- JES (Joint Entropy Search) is a new information theory-based acquisition function for BO.
- Extensions to Tree Parzen Estimators (TPEs)