Using interactive hyperparameter optimization can make hyperparameter tuning faster and more efficient than for example using a random search or an exhaustive grid search.
Valohai supports
 Optuna  a hyperparameter optimization framework
 the opensource Hyperoptlibrary’s Tree Parzen Estimator algorithm
to use the hyperparameters and outputs of the previous executions to suggest future execution hyperparameters.
Under the hood, Bayesian optimization (of which TPE is an implementation) works in the following steps:

Create startup executions using random search

Based on these executions, create a simplified function to model the relationship between the hyperparameters and the target metric value (for example “loss”)

Based on this simplification of their relationship, find the optimal values for the hyperparameter to make the target metric as close to the target value as possible

Run the next batch of executions and repeat the process from step 2.
Creating a bayesian optimization task
 A Valohai project that is connected to a Gitrepository.
 At least one Step with parameters defined. Follow our Define Valohai parameters tutorial to create a step with parameters.
 Go to your Project and open the Task page
 Click on Create task
 Choose the step where you defined your parameters
 Scroll down to the Parameters section
 Select Bayesian optimization as the Task type
Now set your preferred settings:
Setting 
Description 

Early stopping 
Allows you to set early stopping criteria based on the Tasks Collect metadata. When one of the executions from the Task meets this criteria, the whole Task will be stopped. 
Optimization engine 
By default Valohai will use Optuna but you can also choose to use HyperOpt as the engine. 
Maximum execution count 
Defines how many executions can Valohai launch in total for this Task. We recommend the execution count is over 30. 
Execution batch size 
Valohai will run the executions in batches. This defines how many executions will be run in a single batch. 
Optimization target metric 
Defines which Distribution metadata metric you’re looking to optimize. 
Optimization target value 
The target value for your Distribution metadata metric 
We recommend using Bayesian optimization when creating more than 30 executions to ensure the optimizer has enough base values to use TPE effectively.
Valohai follows the Hyperopt recommendation and executes the first 20 runs with Random Search before using the TPE to ensure the best results. If you have less than 20 runs, the executions will be based on Random Search instead of TPE optimization.
Comments
0 comments
Please sign in to leave a comment.