Using interactive hyperparameter optimization can make hyperparameter tuning faster and more efficient than for example using a random search or an exhaustive grid search.
Valohai supports
- Optuna - a hyperparameter optimization framework
- the open-source Hyperopt-library’s Tree Parzen Estimator algorithm
to use the hyperparameters and outputs of the previous executions to suggest future execution hyperparameters.
Under the hood, Bayesian optimization (of which TPE is an implementation) works in the following steps:
-
Create startup executions using random search
-
Based on these executions, create a simplified function to model the relationship between the hyperparameters and the target metric value (for example “loss”)
-
Based on this simplification of their relationship, find the optimal values for the hyperparameter to make the target metric as close to the target value as possible
-
Run the next batch of executions and repeat the process from step 2.
Creating a bayesian optimization task
- A Valohai project that is connected to a Git-repository.
- At least one Step with parameters defined. Follow our Define Valohai parameters tutorial to create a step with parameters.
- Go to your Project and open the Task page
- Click on Create task
- Choose the step where you defined your parameters
- Scroll down to the Parameters section
- Select Bayesian optimization as the Task type
Now set your preferred settings:
Setting |
Description |
---|---|
Early stopping |
Allows you to set early stopping criteria based on the Tasks Collect metadata. When one of the executions from the Task meets this criteria, the whole Task will be stopped. |
Optimization engine |
By default Valohai will use Optuna but you can also choose to use HyperOpt as the engine. |
Maximum execution count |
Defines how many executions can Valohai launch in total for this Task. We recommend the execution count is over 30. |
Execution batch size |
Valohai will run the executions in batches. This defines how many executions will be run in a single batch. |
Optimization target metric |
Defines which Distribution metadata metric you’re looking to optimize. |
Optimization target value |
The target value for your Distribution metadata metric |
We recommend using Bayesian optimization when creating more than 30 executions to ensure the optimizer has enough base values to use TPE effectively.
Valohai follows the Hyperopt recommendation and executes the first 20 runs with Random Search before using the TPE to ensure the best results. If you have less than 20 runs, the executions will be based on Random Search instead of TPE optimization.
Comments
0 comments
Please sign in to leave a comment.