Benchmarking is often challenging anyway and comes with many pitfalls. In DAC this is amplified by how closely target algorithm and DAC controller have to be interfaced for controlling hyperparameters during runtime. Many design decisions are involved in designing such an interface, including defining the cost function, hyperparameter ranges, runtime statistics, interaction points and more. Therefore there are several ways to control even a single target algorithm that are hardly comparable, let alone the difficulty of comparing DAC approaches across target algorithms and domains.
DACBench alleviates this issue of comparability by standardizing interaction dynamics and an interface between target algorithm and DAC controller in addition to providing an easy way to adapt, document and share design decisions. It also provides cheap surrogate problems that provide testbeds for algorithms and are highly configurable to get insights into DAC controllers that would be impossible on complex and opaque target algorithms. These include discrete and continuous hyperparameter spaces, many possibilities of instance manipulation as well as benchmarks that target specific attributes of DAC problems.
Interaction between DAC controllers and DACBench
The real-world benchmarks in DACBench include different domains like Deep Learning, AI Planning or Evolutionary Computation. With a wide range of difficulties and runtimes, these benchmarks present both open challenges and tools to further improve existing solution methods.
In summary, DACBench facilitates fair comparisons of methods across domains, ensures reproducibility of previous work and provides a template for future benchmarks.
If you’re curious to try solving some of these benchmarks, you can find DACBench on GitHub. For more insights, check out our blog post, youtube channel or the IJCAI 2021 paper.
References
- T. Eimer and A. Biedenkapp and M. Reimer and S. Adriaensen and F. Hutter and M. Lindauer
DACBench: A Benchmark Library for Dynamic Algorithm Configuration
In: Proceedings of the International Joint Conferences on Aritificial Intelligence (IJCAI’21)