Run Bayesian optimization – Valohai Help

Using interactive hyperparameter optimization can make hyperparameter tuning faster and more efficient than for example using a random search or an exhaustive grid search.

Valohai supports

Optuna - a hyperparameter optimization framework
the open-source Hyperopt-library’s Tree Parzen Estimator algorithm

to use the hyperparameters and outputs of the previous executions to suggest future execution hyperparameters.

Under the hood, Bayesian optimization (of which TPE is an implementation) works in the following steps:

Create startup executions using random search
Based on these executions, create a simplified function to model the relationship between the hyperparameters and the target metric value (for example “loss”)
Based on this simplification of their relationship, find the optimal values for the hyperparameter to make the target metric as close to the target value as possible
Run the next batch of executions and repeat the process from step 2.

Creating a bayesian optimization task

Prequisites

A Valohai project that is connected to a Git-repository.
At least one Step with parameters defined. Follow our Define Valohai parameters tutorial to create a step with parameters.

Go to your Project and open the Task page
Click on Create task
Choose the step where you defined your parameters
Scroll down to the Parameters section
Select Bayesian optimization as the Task type

Now set your preferred settings:

Setting	Description
Early stopping	Allows you to set early stopping criteria based on the Tasks Collect metadata. When one of the executions from the Task meets this criteria, the whole Task will be stopped.
Optimization engine	By default Valohai will use Optuna but you can also choose to use HyperOpt as the engine.
Maximum execution count	Defines how many executions can Valohai launch in total for this Task. We recommend the execution count is over 30.
Execution batch size	Valohai will run the executions in batches. This defines how many executions will be run in a single batch.
Optimization target metric	Defines which Distribution metadata metric you’re looking to optimize.
Optimization target value	The target value for your Distribution metadata metric

Use Bayesian optimization only when the execution count is over 30

We recommend using Bayesian optimization when creating more than 30 executions to ensure the optimizer has enough base values to use TPE effectively.

Valohai follows the Hyperopt recommendation and executes the first 20 runs with Random Search before using the TPE to ensure the best results. If you have less than 20 runs, the executions will be based on Random Search instead of TPE optimization.

Creating a bayesian optimization task

Related articles