You can use an execution node to parametrize a downstream task node’s variations by printing out metadata and defining an edge that connects the metadata to the Task’s parameter.
List-shaped metadata from executions is automatically “spread” into a Multiple variant parameter if the destination node is a Task.
See the example below where we have two files:
preprocess.py
will generate a list of store IDstrain.py
will train a model for each store, using the store IDs that it gets
preprocess.py
import json
# Your logic to fetch the right values...
storeids = [122,154,209,916,345]
# Print out a list as metadata
print(json.dumps({
"storeids": storeids
}))
train.py
import valohai
storeid = valohai.parameters('id').value
# train model...
Here is also an example of the valohai.yaml
that defines our steps, and the pipeline. In the pipeline’s edge we pass the storeids
from preprocess
to the train
Task’s id
parameter. This will create a train
Task with a set of multiple values for the id
parameter (in our example, a Task with 5 executions, each with it’s own ID parameter value)
- step:
name: preprocess
image: python:3.7
command:
- python ./preprocess.py
- step:
name: train
image: python:3.7
command:
- pip install valohai-utils
- python ./train.py {parameters}
parameters:
- name: id
type: string
- pipeline:
name: dynamic-task
nodes:
- name: preprocess
step: preprocess
type: execution
- name: train
step: train
type: task
edges:
- [preprocess.metadata.storeids, train.parameter.id]
Comments
0 comments
Please sign in to leave a comment.