It can be daunting to try different parameters one by one. Valohai offers a mechanism to do hyperparameter searches using parallel executions and a grid search. These are called tasks.
When starting a task, instead of a single value for a single hyperparameter, you get to define multiple values at once. There are various modes to choose from e.g. Single, Multiple, Linear, Logspace, and Random.
Single means just a single value for a hyperparameter you do not want to search for.
Multiple means a list of all the values to try for a specific hyperparameter. For example, here we are trying out 4 different values (0.81, 0.84, 0.86, and 0.91).
Logspace is a search with values inside a specific range in logarithmic space. For example, here we are trying out 4 different values between 2^2 - 2^8 (base^start - base^end)
Finally, if you want to “gamble”, you can search randomly with a specified range and distribution. For example, here we are trying out 10 different random values between 0.001 and 0.002.
When we are searching for multiple values for multiple hyperparameters, all permutations are searched. This is also called grid search.
For example, here we have 6 different values for learning_rate and 6 different values for dropout. In total this is 6*6 = 36 executions. Valohai calculates the number of permutations for you and you can see it in the right bottom corner of this screenshot.
Please sign in to leave a comment.