Introduction to Tasks

Tasks are collections of related executions, where you run the same step/code with different parameters and input data values.

The most common task is hyperparameter optimization where you execute a single step with various parameter configurations to find the most optimal neural network layout, weights, and biases.

Tasks can be created with any step that has parameters defined.

When to use Tasks?

There are a few common uses for a Task:

• Parameter sweeps with a Grid, Random or Manual search
  Tasks enable you to easily run a Grid or a Random Search. You'll just need to define which values you want to try for your parameters and let Valohai run the schedule and execute the jobs. In addition, you can also use Manual search to define the parameter combinations you want to run manually. 
• Bayesian optimization
  Valohai has native support for Optuna and HyperOpt for Bayesian optimization. You'll need to tell Valohai which metric you're optimizing, the target value, and the details of your parameters.
• Train multiple models on different datasets
  Let’s say you want to run train 100 models for 100 different merchant stores. Each store has its own data in S3, like s3://bucket/data/stores/store-335/data/* where store-335 is the ID of the store.
  
  You could define a parameter, store-id, and run a Task where you pass 100 different values for store-id and define your input data url as s3://bucket/data/stores/{parameter:store-id}/data/*.
  
  Each execution will download its own data from S3 and train its own model—all in parallel.